Transcript CloudDb2015Lect8MongoDbx

With thanks to Michael Grossniklaus!
Data Management in the Cloud
Lecture 8
Data Models
Document: MongoDB
“…I’ve failed over and over and over again in my life.
And that is why I succeed.” – Michael Jordan
1
Trend Towards Specialization
-no longer one size fits allNon-relational
Operational Datastores
(“NoSQL”)
New Generation OLAP
RDBMS
(vertica, aster, greenplum)
(Oracle, MySQL)
MongoDB’s definition of “NoSQL”:
non-relational, next-generation operational datastores and databases
2
Unifying Theme of NoSQL Systems:
Horizontally Scalable Architectures
• No joins + No complex multi-row transactions
– hard to implement both when scaling horizontally
– abandon the relational data model
• New data models
– key/value: memcached, Dynamo
– document-oriented: MongoDB, CouchDB, JSON stores
– tabular: BigTable
• Improved ways to develop applications?
– “easier than a relational database management system”
– data modeling: begin with normalized model and refine model by
embedding documents
3
Spectrum of Systems
scalability and performance
• memcached
• key/value
• MongoDB
• RDBMS
depth of functionality
4
MongoDB vs. RDBMS
• What is missing?
– joins
– complex multi-row transactions
– SQL (but MongoDB has document-based query “language”)
• Terminology
–
–
–
–
database → database or schema
table → collection
row → document
column → field
Database
Collection2
Doc1
Collection1
Index
Doc2
Document2
Document1
Field1
Field2
Document3
MongoDB vs. RDBMS
• “relational databases define columns at the table level
whereas a document-oriented database defines its fields at the
document level” - http://openmymind.net/mongodb.pdf (p.5)
– Table -> Collection; Row -> Document
Shoes Table
Brand
Size Color
Vans
8
Red/Gold
DC
8
White/Blue/Gree
n
Nike
8.5
YellowGreen
Shoes Collection
{“Brand”: “Vans”,
“Size”: “8”,
“Color”: “Red/Gold”}
{“Brand”: “DC”,
“Size”: “8”,
“Color”: “White/Blue/Green”}
{“Brand”: “Nike”,
“Size”: “8.5”,
“Color”: “YellowGreen”}
6
MongoDB vs. RDBMS
• “relational databases define columns at the table level
whereas a document-oriented database defines its fields at the
document level” - http://openmymind.net/mongodb.pdf (p.5)
– Table -> Collection; Row -> Document
Shoes Table
Brand
Size Color
Vans
8
Red/Gold
DC
8
White/Blue/Gree
n
Nike
8.5
YellowGreen
Shoes Collection
{“Brand”: “Vans”,
“Size”: “8”,
“Color”: “Red/Gold”,
“Note”: “Looks like mermaid”}
{“Brand”: “DC”,
“Size”: “8”,
“Color”: “White/Blue/Green”}
{“Brand”: “Nike”,
“Size”: “8.5”,
“Color”: “YellowGreen”}
7
MongoDB Data Model & Features
• Advantages of document model
– Documents (objects) correspond to native types in many programming
languages
– Embedded documents and arrays reduce need for expensive joins
• High Performance
Is this always true?
– Support for embedded data models reduces I/O activity
– Indexes + can include keys from embedded documents and arrays
• High Availability
– Automatic failover
– Data redundancy
• Automatic Scaling
– Automatic sharding
– Replica sets for eventually-consistent reads
http://docs.mongodb.org/manual/core/introduction/
8
Discussion Question 1
• Give an example of a case where embedded documents
reduce disk activity.
• Are there any cases where it increases disk activity?
9
Data Model
• MongoDB stores JSON-style documents
document size
– internally represented as BSON
field type (string)
– {“hello”: “world”}
↓
field name
\x16\x00\x00\x00\x02hello\x00
field value
\x06\x00\x00\x00world\x00\x00
– General-purpose serialization format
end of object
– light-weight and traversable
– driver converts programming language objects to BSON
– document size limited to 4 MB
• Flexible “schemas”
both documents can be
stored in the same
collection
{“author”: “mary”,
“text”: “...”,
this is a list
10
“tags”: [“mongodb”]}
– the “big difference” between a collection and a table
{“author”: “fred”,
“text”: “...”}
Source: http://bsonspec.org
Storing Documents
post = {author: “mike”,
date: new Date(),
text: “my blog post...”,
tags: [“mongodb”, “intro”]}
JavaScript object
database
tags embedded in post
vs. many-to-many join
db.posts.save(post)
collection
• Example uses JavaScript
– MongoDB supports JavaScript on the client and server side
– other language bindings exist
• BSON is the basis for a rich type system
– e.g., (binary) date values are not possible in JSON
• Collections are created implicitly on demand
– Created first time a document is saved to that collection
11
Document Identifiers
• Special key _id
– present in all documents: user or system-defined
– unique across a collection
– can be any type
• System-defined default _id is added if not specified by user
– similar to GUID/UUID, lightweight (only 12 bytes) and fast to generate
– generated at the client
– ObjectId(“4bface1a2231316e04f3c434”)
timestamp
machine id
process id
incrementing counter
Source: http://www.mongodb.org/display/DOCS/Object+IDs
12
Queries
• Query evaluation invoked by method find() on collection
– posts by author
db.posts.find({author: “mike”})
• Query language
– query-by-example plus “$ modifiers”: $gt, $lt, $gte, $lte, $ne,
$all, $in, and $nin
– age between 20 and 40
{author: “mike”, age: {$gte: 20, $lt: 40}}
• Advanced queries
– keyword $where
db.posts.find({$where: “this.author == ‘mike’ ||
this.title == ‘dbms’”})
Source: http://www.mongodb.org/display/DOCS/Advanced+Queries
13
Complex Queries
• Queries involving dates
JavaScript starts
counting months at 0
– posts since April 1
april1 = new Date(2013, 3, 1)
db.posts.find({date: {$gt: april1}})
• Sorting and limit
– last ten posts
all posts
descending
db.posts.find()
.sort({date: -1})
.limit(10)
– lazy evaluation (cursors)
– sort is defined on the cursor instance
14
Cursors
var c = db.posts.find({author: “mike”})
.skip(20)
omit first 20 results
.limit(10))
only return 10 results
c.next()
c.next()
...
next result
query
first N results and cursor id
next with cursor id
next N results and cursor id or eof
...
15
Queries over Text and Collections
• Use regular expression in find() method
– posts ending with “Tech”
db.posts.find({text: /Tech$/})
Regular Expression
• Members of collection-valued attributes are queried in the
same way as attributes with atomic values
– posts with a specific tag
db.posts.find({tags: “mongodb”})
• Use of a multi-key index speeds up collection-oriented queries
– db.posts.ensureIndex({tags: 1})
16
Aggregation Functions
• Counting the number of documents in a collection
– total posts
db.posts.count()
– total posts authored by Mike
db.posts.find({author: “mike”}).count()
• Method distinct() displays the distinct values found for a
specified key in a collection
• Method group() groups documents in a collection by the
specified key and performs simple aggregation
• More ways to compute aggregated values
– aggregation framework using aggregate command
– map/reduce aggregation for large data sets using mapReduce
command
17
Atomic Update Modifiers
var c = {author: “eliot”,
date: new Date(),
text: “Great post!”}
query selector
db.posts.update({_id: post._id},
{$push: {comments: c}})
update value
• MongoDB has no transactions
– complex functionality exposed through update modifiers
– $push adds (appends) an object to an array
• For this update, client does not need to worry about
– locking of documents
– contention of updates
18
Document Evolution
post = {author: “mike”,
date: new Date(),
text: “another blog post...”,
tags: [“mongodb”],
title: “Introduction to MongoDB”}
db.posts.save(post)
New Attribute
• Flexible schema
– documents within the same collection can have different attributes
– document attributes can simply be added or removed
– evolution performed when save() is invoked
19
Horizontally Scalable Architectures
• Remember…they said…
• No joins and no complex multi-row transactions
– hard to implement both when scaling horizontally
– abandon the relational data model
20
Parallel Database Execution - Join
SELECT brand, description, size, shelfnumber,
shelfposition
FROM shoes, shoestorage
WHERE shoes.id = shoestorage.id
AND color = ‘Green’
AND lastworn < ‘1-25-2014’
Case 1: Tuples from shoes and
shoestorage are randomly
partitioned (sharded) across the
three disks.
Bold lines indicate
transfer across
network
JOIN
JOIN
Computer 1
RePartition
SELECT
shoes
Computer 2
RePartition
SCAN
shoestorage
RePartition
SELECT
shoes
RePartition
SCAN
shoestorage
Parallel Database Execution - Join
SELECT brand, description, size, shelfnumber,
shelfposition
FROM shoes, shoestorage
WHERE shoes.id = shoestorage.id
AND color = ‘Green’
AND lastworn < ‘1-25-2014’
Case 2: Tuples from shoes and
shoestorage are partitioned
(sharded) on id.
Joins are local, no
transfer across
network.
Computer 1
JOIN
SELECT
shoes
Computer 2
JOIN
SCAN
shoestorage
SELECT
shoes
SCAN
shoestorage
Transactions Across Systems…
T1 – Update Converse Black/Red -> Black/Red/White
T2 – Update Nikes YellowGreen -> ReallyBrightYellow
T1 – Update Nike Yellow/Green -> NeonYellow
T2 – Update Nike Yellow/Green -> NeonYellow
T1 waiting on T2
Distributed Deadlock…
T2 waiting on T1
Brand
Size Color
Vans
8
DC
8
Nike
8.5
T1
T2
Brand
Size Color
Red/Gold
Converse
8
White
White/Blue/Gree
n
Converse
8
Blue
Converse
8
Black/Red
YellowGreen
23
Atomic Update Modifiers
var c = {author: “eliot”,
date: new Date(),
text: “Great post!”}
query selector
db.posts.update({_id: post._id},
{$push: {comments: c}})
update value
• MongoDB has no transactions
– complex functionality exposed through update modifiers
– $push adds (appends) an object to an array
• Client does not need to worry about
– locking of documents
– contention of updates
24
So, no transactions…
• No begin, commit, rollback
• Atomic only at the single document level…
• But, they have Write Concern …
25
Write Concern…
• Errors Ignored
– MongoDB does not acknowledge write operations.
– Client cannot detect failed write operations – including connection
errors, duplicate key exceptions…
– But fast performance…
• Unacknowledged
– MongoDB does not acknowledge the receipt of write operation.
– Similar to errors ignored; however, drivers attempt to receive and
handle network errors when possible.
– Used to be the default write concern.
• Acknowledged
– Mongodb confirms the receipt of the write operation.
– Allows clients to catch network, duplicate key, and other errors.
– Is now the default write concern.
http://docs.mongodb.org/manual/core/write-concern/
26
Write Concern…
• Journaled
– Mongodb acknowledges the write operation only after committing the
data to the journal. Write is recoverable.
– Write operations must wait for next journal commit. To reduce latency,
mongodb increases the frequency that it commits to the journal…
• Replica Acknowledged
– With replica acknowledged you can guarantee that the write operation
propagates to the members of a replica set. See
– To set replica acknowledged write concern, specify w values greater
than 1 to your driver.
http://docs.mongodb.org/manual/core/write-concern/
27
Discussion Question 2
• What would be the impact of fast (but unreliable) writes in
your application?
• How would you use the Write Concern options in your
application?
28
Query Optimizer
• No algebraic or cost-based query optimization
–
–
–
–
evaluator processes alternative plans in parallel
first plan to finish is remembers as most efficient
other plans are terminated
query plan is recorded – re-evaluate plan after 1000 writes, add, drop
or rebuild index
• Find posts by Joe about database management systems
– db.posts.find({author: “joe”, tags: “dbms”})
linear scan
terminate
index on tags
terminate
index on author
remember
Data File Allocation
• Memory-mapped storage engine
– entire files are mapped into memory
• MongoDB stores each database as a number of files
– namespace contains the catalog
– data files contain the collections and documents
– 16MB blog.ns
64MB blog.0
double in size
allocated for each database
(up to 2 GB)
128MB blog.1
16MB test.ns
...
http://docs.mongodb.org/manual/faq/storage/
30
Extent Allocation
• Extents allocated into
namespaces per collection
blog.0
blog.1
0000000000000
blog.2
0000000000000
0000000000000
0000000000000
0000000000000
0000000000000
0000000000000
0000000000000
–
–
–
–
–
000
blog.posts
blog.comments
blog.users
blog.$freelist
pre-allocated space
• Namespace metadata details
stored in blog.ns
• Empty file is kept in order to
avoid blocking on allocation
31
Supported Platforms and Languages
• Drivers
– PHP, Perl, Python, Ruby, Java, C++/C, C#, Erlang, Haskell
• Platforms
–
–
–
–
–
Window (x86, x64)
Linux (x86, x64)
OS X (x86, x64)
Solaris
FreeBSD
32
client
application
driver
writes
Replication
reads
primary
secondary
secondary
secondary
primary
secondary
primary
secondary
primary
primary
secondary
primary
• Replication provides redundancy and increases data availability
• Replica sets added in version 1.6
– “master/slave”-style replication, but roles can change
– primary receives all writes
– if primary fails, secondaries elect a new primary
http://docs.mongodb.org/manual/replication/
33
Read Preferences
Read Preference Mode
Description
primary
Default mode. All operations read from
the current replica set primary.
primaryPreferred
In most situations, operations read from
the primary but if it is unavailable,
operations read from secondary members.
secondary
All operations read from the secondary
members of the replica set.
secondaryPreferred
In most situations, operations read from
secondary members but if nosecondary
members are available, operations read
from the primary.
nearest
Operations read from the nearest member
of the replica set, irrespective of the
member’s type.
http://docs.mongodb.org/manual/core/read-preference/
34
Auto-Sharding
Stores the data
Shards
mongod
mongod
mongod
...
Configuration
Servers
mongod
mongod
mongod
Supports rangebased and hashbased sharding
based on a
“shard key”
mongod
mongod
mongod
Stores the cluster’s metadata;
contains a mapping of the
cluster’s data set to the shards.
Production – must have exactly
3 config servers.
Query Routers
mongos
client
mongos
...
Interfaces with client
applications and directs
operations to the appropriate
shard or shards
35
http://docs.mongodb.org/manual/sharding/
Other Features
• Aggregation and map/reduce
– complex aggregation functionality
– ETL/offline aggregation tasks
• Capped collections
– fixed-sized collection with wrap around (overwrite)
– e.g., for website logging
• Unique indexes
• Mongo shell
• GridFS
– file system API built on top of MongoDB
– stores large binary data by chunking it up into smaller documents
• Two-dimensional geo-spatial indexing
– e.g., foursquare
36
Use Cases
• World Wide Web
– scaling out
– high volume
– caching
• Less good at…
– highly transactional
– ad-hoc business intelligence
– problems that require SQL
37
2014 Updates
• Design Principles
– Developer Productivity (developers need good tools)
– Scalability (horizontal scalability)
– Operational scalability
• 3 Improvements in process in 2014
– Concurrency
– Storage Engine
– Automation
38
2014 Updates
• Concurrency
– Existing concurrency: “not quite there yet”
– MongoDB concurrency defn: “Cooperative time sharing at database level
using knowledge about file system cache to avoid locking around disk
access.”
– Document level locking coming in 2014
– Demo showed factor of 10 improvement in throughput
• Storage engine
–
–
–
–
Existing: general purpose storage engine
No single storage engine good for all use cases
Soln: pluggable storage API
Chose storage engine per server
• Ex: Hot data on SSDs
• Cold shards, hot shards…
• Automation
– New management service MMS – aimed at scaling to large configurations
39
References
• M. Dirolf: Introduction to MongoDB, O’Reilly Webcast, 2010,
available at: http://www.youtube.com/watch?v=w5qr4sx5Vt0.
• http://docs.mongodb.org/manual/
• http://www.mongodb.com/presentations/mongodb-world2014-keynote-eliot-horowitz
• Chapter 9 Sandage & Fowler
40