Transcript here
JO-SQL
Versatile Database with reusable parts
Jörg Schön 2011
1
High-level view on architecture
Various layers:
josql
josqlsh
SQL parser and query optimizer
Table driver
josqld
User programs:
Interactive shell
Server backend
josqlck
Administrative programs:
Table checker
josqldump
Backup utility
Container maintenance
dbfsadmin
Buffer pool statistics etc.
fcacheadmin
jnsqlib
josqlib
jdbapi.h
jdblib
btree
dbasefs
fcache
bigsort
2
SQL layer „josqlib“
Statement execution API either direct („josqlib“) or via network
connect to server („jnsqlib“).
Optional wrapper „josql“ for runtime switch between direct or
network connect.
Perl interface to server backend.
Table driver accessed via API „jdbapi.h“:
– can plug other table drivers easily.
– driver defined field types and functions.
Full programming language as SQL extension.
josql
User management (grant/revoke).
josqlsh
Missing features:
jnsqlib
– Subselects (planned).
josqld
josqlib
– Advanced table statistics
(for query optimizer)
jdbapi.h
– View, triggers, ...
3
Table driver „jdblib“
B-tree indexes
– Fixed length currently supported.
– Variable length and prefix compression under construction
External sorting („bigsort“) decouples memory footprint from
table size.
Supports commit, rollback, and nested transactions.
jdbapi.h
Automatic recovery from last
josqlck
jdblib
syncpoint on start.
josqldump
dbfsadmin
fcacheadmin
btree
dbasefs
fcache
bigsort
4
Disk format „dbasefs“
Resembles file system and can be used independently
of database.
Copy-on-write semantics.
Multiple writes become active simultaneously on „syncpoint“.
Supports (multiple) snapshots.
Data can be in one or multiple disk files of limited size.
„Autoextend“ of containers.
Transparent access of ordinary files (at runtime).
Conversion of normal files to containers and vice versa:
– In-place conversion not scaling with file size
– Enables simple rewrite of existing applications
dbasefs
to take advantage of atomic writes, snapshots, ...
fcache
5
Buffer pool „fcache“
• Uses System-V or POSIX shared memory and semaphores.
• Also ported to Windows (though might have room for
improvements...)
Runtime parameters:
– Multiple buffer pools possible.
– Use large allocated chunk instead of shared memory.
– Maximum number of processes.
– Page size and total cache size.
Cache may consist of multiple segments to work around shared
memory limitations.
Natively provides shared read/write access, region locking, and
sequences.
Mechanism for extensions (used by e. g.
fcache
„dbasefs“)
6
General features
•Careful layering to allow using
various parts independently.
•Various test and example programs:
– „testbigsort“, „testfcache“,
„testdbfs“, „testbtree“
– „testdblib“, „testjosql“
josql
josqlsh
jnsqlib
josqld
josqlib
jdbapi.h
jdblib
josqlck
josqldump
dbfsadmin
fcacheadmin
btree
dbasefs
fcache
bigsort
7