Transcript Document

Supporting Persistent Objects
In Python
Jeremy Hylton
[email protected]
What is persistence?
(C) 2002 Zope Corp.
• Data lives longer than programs
– Files, pipes, relational databases, etc.
– But, discontinuity of representation
• Orthogonal persistence
– Automatic management of program state
– Independent of data type or longevity
– Allow programmer to focus on application
data model
Python approach
(C) 2002 Zope Corp.
• Goals
– Minimize changes to existing programs
– Work with standard interpreter
• Zope Object Database (ZODB)
– Support Zope application server
– Originally targeted for web development
• Separate database and storage layers
Not on today’s agenda
(C) 2002 Zope Corp.
• Benefits of persistence
• Discussion of related work
– Early work
• PS-algol, Napier88
– Much recent work on Java
• PJava, ObjectStore PSE
ZODB key ideas
(C) 2002 Zope Corp.
• Persistence by reachability
– Any reachable object is saved
• Transactions to control updates
– Safe sharing among clients
• Implement with Persistent mixin
– Provides hook for persistence
– Compromise on transparency
Persistent base class
(C) 2002 Zope Corp.
• C extension type that defines
– Four _p_ attributes
– Custom attribute accessors
– Efficient C API
• Marks serialization boundaries
• Interacts with database
– Load objects on demand
– Register modifications with TM
Persistence by reachability
(C) 2002 Zope Corp.
• Any object reachable from ZODB root
is persistent
A simple example
(C) 2002 Zope Corp.
• from Persistence import Persistent
from Transaction import get_transaction
from ZODB.FileStorage import DB
class Counter(Persistent):
_value = 0
def inc(self):
self._value += 1
def main():
fs = DB(“data.fs”)
conn = db.open(); root = conn.root()
obj = root[“myobj”] = Counter()
get_transaction().commit()
obj.inc()
get_transaction().commit()
Object serialization
(C) 2002 Zope Corp.
• Standard pickle library
– Serializes arbitrary object graph
– Raises TypeError for sockets, files, &c.
– Instance vars serialized via dictionary
• Hooks to define custom state
– __getstate__() / __setstate__()
– Persistent mixin ignores _v_ attributes
Pickling persistent objects
(C) 2002 Zope Corp.
• Stores objects in separate records
– Persistent objs pickled as oid + class
– Works with cache to maintain identity
• Handling non-persistent objects
– Copied into record of containing object
– Sharing by persistent objs is problematic
Object identity / caching
(C) 2002 Zope Corp.
• Cache maintains oid  obj mapping
– Guarantees only one copy of object
– Unpickler loads all referenced objects
• Ghost objects
– Only Persistent header initialized
– No instance state loaded
– State loaded on first object access
• LRU cache of recent objects
Attribute access handlers
(C) 2002 Zope Corp.
• Persistent implements C wrappers
– Override tp_getattro, tp_setattro slots
– Mediate access to instance variables
– Crucial Python feature
Example __getattribute__() hook
(C) 2002 Zope Corp.
class Example(object):
_p_state = False
def _p_activate(self):
print "activate"
self._p_state = True
def __getattribute__(self, attr):
print "intercept", attr
if not attr.startswith("_p_") and not self._p_state:
self._p_activate()
return super(Example, self).__getattribute__(attr)
>>> obj =
>>> print
intercept
intercept
intercept
activate
test
Example(); obj.value = "test"
obj.value
value
_p_state
_p_activate
Transactions
(C) 2002 Zope Corp.
• Supports multiple threads, processes
– Independent database connections
– Updates visible at transaction boundaries
• Optimistic concurrency control
– When conflict occurs, abort and retry
• On error, abort to restore consistency
– Reverts to last saved state
Concurrency and conflicts
(C) 2002 Zope Corp.
• Invalidations sent at commit time
– Clients process at transaction boundaries
• Conflicting transactions aborted
– Write conflict at commit time
– Read conflict on object access
• Application must retry on conflict
– Can use generic wrapper
– Can define conflict resolution method
Retrying conflicts
(C) 2002 Zope Corp.
• Example wrapper for retries
def transact(f, retries=3):
def wrapper(*args, **kwargs):
n = retries
while n:
try:
try:
return f(*args, **kwargs)
finally:
get_transaction().commit()
except ConflictError:
n -= 1
if n == 0:
raise
except:
get_transaction().abort()
raise
return wrapper
Other features
(C) 2002 Zope Corp.
• Undo support
– Storage stores multiple revisions
– Transactional undo reverts to earlier state
• BTrees: efficient persistent containers
• Storing code in database
Limitations
(C) 2002 Zope Corp.
• Schema evolution
– Must code manually in __setstate__()
• Database management
– Manual pack() to remove revisions, do GC
• Sharing of non-persistent objects
• Integration with legacy code
– Multiple inheritance helps
– Factory classes
Getting the software
(C) 2002 Zope Corp.
• http://www.zope.org/Wikis/ZODB
– info central for ZODB
• ZODB 3.1 released Oct. 2002
– Uses ExtensionClass
• ZODB4 alpha planned for Dec. 2002
– Based on Python 2.2 type support
– Fewer onions in the varnish