Transcript Square pegs and round holes - Welcome

Square pegs and round holes
A reflection on Class::DBI
http://popcorn.cx/
Who am I?
• Developer at Monash University
• Applications in and related to the staff
and student portal
• Built upon open source
– Perl, HTML::Mason, Apache
http://popcorn.cx/
Class::DBI
• Database abstraction layer for Perl
• Allows mapping of classes to database
tables
• Each row of the table is an instance of
the class
http://popcorn.cx/
The square peg
User directory
Learning
Management
System
Student records
Teaching staff
http://popcorn.cx/
Initial development
• Bulk written in Perl, remainder PL/SQL
and Java
• Chose Class::DBI as it gave us a leg up
due to time constraint
– But not as much as we expected
http://popcorn.cx/
Clash of standards
Music::CD->has_a(
artist => 'Music::Artist’
);
print $cd->artist->name;
http://popcorn.cx/
Clash of standards
Music::CD->has_a(
artist_id => 'Music::Artist’
);
print $cd->artist_id->name;
http://popcorn.cx/
Clash of standards
Music::CD->has_a(
artist_id => 'Music::Artist’
);
*artist = \&artist_id;
print $cd->artist->name;
http://popcorn.cx/
Clash of standards
Music::CD->has_a(
artist_id => 'Music::Artist’
);
*get_artist = \&get_artist_id;
*set_artist = \&set_artist_id;
print $cd->get_artist->name;
http://popcorn.cx/
Playing nice with friends
• Our expectation was for error at runtime
• If remote service is not available:
– Module loads
– Only when used is error indicated
• By return value or an exception
http://popcorn.cx/
Playing nice with friends
• Class::DBI connects to database, and
errors, at module load
• Which mean an error at load paradigm
• If database is unavailable
– Module will not load
– Thus stopping remaining code from loading
• We needed to be more defensive than
usual and wrap the use in an eval
http://popcorn.cx/
I canna do it captain!
• Takes XML extract of all user accounts upwards of 90,000
• Compares against data in database
• Updates accordingly
http://popcorn.cx/
First attempt
• Run through extract and call
retrieve() for each record
• 90,000 database queries
• Database bottleneck
• Runtime of over an hour
http://popcorn.cx/
Second attempt
• Load all objects into memory via
retrieve_all()
• Single database query
– no longer the bottleneck
• Now encounter memory allocation issues
• Still a runtime of over an hour
http://popcorn.cx/
Third attempt
• Avoid Class::DBI and use DBI directly
• Single database query gave all records
in an array of hash references
• Then we only call retrieve() if
a change is required
• Runtime of less than ten minutes
http://popcorn.cx/
Fourth attempt
• Did not try as third attempt gave
satisfactory results
• Would have probably loaded entire
extract and then used iterator from
retrieve_all()
http://popcorn.cx/
Overview of initial development
• A success
• Built working system in a few weeks
• Class::DBI gave significant head start.
Although we needed to:
– learn how to use it
– bend to our standards
– avoid in some cases for performance
http://popcorn.cx/
Maintenance and enhancement
• Now been running for over two years
• Huge increase in usage
– Second largest installation in the world
• Also expansion of functionality
• Can at times be frustrating
http://popcorn.cx/
Where is the object?
print $section->get_course();
•
•
•
•
Many hours wasted
A number is printed
Not the expected blessed referent
Due to stringification feature of Class::DBI
– That we had not mentioned in our documentation
http://popcorn.cx/
Unforeseen implications
• Extended update(), create() and
delete() to require a username for an
audit trail
– $object->delete($username)
• Worked well
• Until we tried to use features such as
cascade delete
http://popcorn.cx/
Unforeseen implications
• Class::DBI follows ‘has many’ relationships
and calls delete()
• Does not know to pass our username
• Which makes it fail
• Our solution to a business need limited the
features we could use
• Better knowledge of Class::DBI internals
should provide a better solution
http://popcorn.cx/
Building a round peg
Public
website
Course database
Administrators
http://popcorn.cx/
Development
• All Perl solution
– Except for Oracle database
• Perl CGI for public website
• Perl under HTML::Mason for
administration interface
• Perl for batch load process
http://popcorn.cx/
Class::DBI or not Class::DBI
• Our data model led to a matching set of
classes and database schema
• But
– Class::DBI was not available on the target
environment
– Recent issues (at the time) soured our
opinion of Class::DBI
• Decided to build our classes from scratch
http://popcorn.cx/
It lives
• Quickly became apparent that some
form of framework was needed.
• So we built our own:
http://popcorn.cx/
Basic setup
use base 'Base';
__PACKAGE__->_set_table_name( ’table' );
__PACKAGE__->_setup_attribtes(
qw{ code name ... },
);
http://popcorn.cx/
‘has_a’ relationship
__PACKAGE__->_setup_has_a_relationship(
'class'
=> 'Faculty',
'get_method' => 'get_faculty',
'set_method' => 'set_faculty',
);
http://popcorn.cx/
‘has_many’ relationship
__PACKAGE__->_setup_has_many_relationship(
'class'
=> 'OtherClass',
'table'
=> 'other_table',
'get_method'
=> 'get_others',
'add_method'
=> 'add_other',
'remove_method' => 'remove_other',
);
http://popcorn.cx/
‘has_many’ relationship
__PACKAGE__->_setup_has_many_relationship(
'class'
=> 'OtherClass',
'table'
=> 'other_table',
'get_method'
=> 'get_others',
);
http://popcorn.cx/
Compensating for ‘issues’
•
•
•
•
Built to expect our standards
Follows error at runtime paradigm
Still need to bypass in some cases
Only built what we needed/wanted
– So no stringification to primary key
• Natively requires username for audit trail
http://popcorn.cx/
Was it worth it?
• Course Finder project was a success
• Framework allowed later changes
to be implemented quickly
• We knew exactly how our framework
worked
http://popcorn.cx/
Would we do it again?
• No.
• Because we can reuse this framework
• Which we have done in one other
project and fed enhancements back
http://popcorn.cx/
Another peg
• There are other database abstraction
frameworks:
– DBIx::Class is biggest alternative
• Either didn’t exist or appear on our radar
at the time
• They are not excluded for future projects
http://popcorn.cx/
Conclusion
• I don’t really have one…
http://popcorn.cx/
Conclusion
• Any framework is beneficial
• But you need one that does what you
need
• And you will need to bypass it
http://popcorn.cx/
Questions?
http://popcorn.cx/
• Blackboard Learning System Vista
Enterprise License Release 3 Patch
something
or other.
Potato!
http://popcorn.cx/
This slide left intentionally blank
http://popcorn.cx/