Transcript here

JO-SQL
Versatile Database with reusable parts
Jörg Schön 2011
1
High-level view on architecture
Various layers:
josql
josqlsh
SQL parser and query optimizer
 Table driver
josqld
 User programs:
 Interactive shell
 Server backend
josqlck
 Administrative programs:
 Table checker
josqldump
 Backup utility
 Container maintenance
dbfsadmin
 Buffer pool statistics etc.

fcacheadmin
jnsqlib
josqlib
jdbapi.h
jdblib
btree
dbasefs
fcache
bigsort
2
SQL layer „josqlib“
Statement execution API either direct („josqlib“) or via network
connect to server („jnsqlib“).
Optional wrapper „josql“ for runtime switch between direct or
network connect.
 Perl interface to server backend.
Table driver accessed via API „jdbapi.h“:
– can plug other table drivers easily.
– driver defined field types and functions.
Full programming language as SQL extension.
josql
User management (grant/revoke).
josqlsh
Missing features:
jnsqlib
– Subselects (planned).
josqld
josqlib
– Advanced table statistics
(for query optimizer)
jdbapi.h
– View, triggers, ...

3
Table driver „jdblib“
B-tree indexes
– Fixed length currently supported.
– Variable length and prefix compression under construction
External sorting („bigsort“) decouples memory footprint from
table size.
Supports commit, rollback, and nested transactions.
jdbapi.h
Automatic recovery from last
josqlck
jdblib
syncpoint on start.

josqldump
dbfsadmin
fcacheadmin
btree
dbasefs
fcache
bigsort
4
Disk format „dbasefs“
Resembles file system and can be used independently
of database.
Copy-on-write semantics.
Multiple writes become active simultaneously on „syncpoint“.
Supports (multiple) snapshots.
Data can be in one or multiple disk files of limited size.
„Autoextend“ of containers.
Transparent access of ordinary files (at runtime).
Conversion of normal files to containers and vice versa:
– In-place conversion not scaling with file size
– Enables simple rewrite of existing applications
dbasefs
to take advantage of atomic writes, snapshots, ...

fcache
5
Buffer pool „fcache“
• Uses System-V or POSIX shared memory and semaphores.
• Also ported to Windows (though might have room for
improvements...)
Runtime parameters:
– Multiple buffer pools possible.
– Use large allocated chunk instead of shared memory.
– Maximum number of processes.
– Page size and total cache size.
Cache may consist of multiple segments to work around shared
memory limitations.
Natively provides shared read/write access, region locking, and
sequences.
Mechanism for extensions (used by e. g.
fcache
„dbasefs“)
6
General features
•Careful layering to allow using
various parts independently.
•Various test and example programs:
– „testbigsort“, „testfcache“,
„testdbfs“, „testbtree“
– „testdblib“, „testjosql“
josql
josqlsh
jnsqlib
josqld
josqlib
jdbapi.h
jdblib
josqlck
josqldump
dbfsadmin
fcacheadmin
btree
dbasefs
fcache
bigsort
7