Transcript Platform for Big Data, NoSQL and Relational Data. What

PLATFORM FOR BIG DATA, NOSQL
AND RELATIONAL DATA. WHAT
MAKES SENSE FOR ME?
(+AZURE)
[email protected]
http://blogs.gartner.com/douglaney/files/2012/01/ad949-3D-Data-Management-ControllingData-Volume-Velocity-and-Variety.pdf
RoadDesignator
DrivingStatus
A1
Difficulties
Batch Processing
Interactive Analysis Stream Processing
Query runtime
Minutes to hours
Milliseconds to minutes
Never-ending
Data volume
TBs to PBs
GBs to PBs
Continuous stream
Programming model
MapReduce
Queries
DAG
Users
Developers
Analysts and developers
Developers
Originating project
Google MapReduce
Google Dremel
Twitter Storm
Open source project
Hadoop / Spark
Drill / Shark / Impala
Hbase
Storm / Apache S4 /Kafka
What’s the social
sentiment for my
brand or products
How do I better
predict future
outcomes?
How do I optimize my
fleet based on weather
and traffic patterns?
GAIN COMPETITIVE ADVANTAGE BY MOVING FIRST AND FAST IN YOUR INDUSTRY
IT infrastructure
optimization
Churn
analysis
Fraud
detection
Life sciences
research
Legal
discovery
Social network
analysis
Traffic flow
optimization
Natural resource
exploration
Weather
forecasting
Healthcare
outcomes
Advertising
analysis
Equipment
monitoring
Web app
optimization
Smart meter
monitoring
persistent | distributed
• In Memory
• Efficient at Random
Reads/Writes
• Distributed, large
scale data store
• Utilizes Hadoop for
persistence
• Both HBase and
Hadoop are
distributed
http://developer.yahoo.com/hadoop/tutorial/module4.html
MANAGE ANY DATA, ANY SIZE, ANYWHERE
010101010101010101
1010101010101010
01010101010101
101010101010
Hadoop architecture.
Distributed Processing
(Map Reduce)
Distributed Storage
(HDFS)
INSIGHTS FOR ALL USERS THROUGH FAMILIAR TOOLS
PB
TB
GB
An object contained within a user database
Defines the scheme for the federation
Represent the database being sharded
Orders_federation
Orders_federation
Database that houses the federation object
System managed SQL databases
Contain part, or “slices” of data
CREATE FEDERATION fed_name(fed_key_label fed_key_type distribution_type)
 The key used for data distribution
 int, bigint, guid, varbinary
 Represent a single instance of a federation key.
 All rows in all federated tables with the same federation key value.
Orders_federation
Orders_federation
PK=5
PK=5
PK=25
PK=25
PK=35
PK=35
Contains only atomic units for member’s key range
Non-Federated table
SPLIT members to spread workloads over to more nodes
DROP members to shrink back to fewer nodes
ALTER FEDERATION Orders_Fed SPLIT AT (tenant_id=7500)
SalesDB
Orders_federation
Orders_federation
Orders_Fed
[5000, 7500)[5000,
& [7500,10000)
10000)
Ensure apps can discover where the data is just-in-time
No “Shard Map” caching
Guaranteed member routing
USE FEDERATION Orders_Fed (tenant_id=7509)
SalesDB
Orders_federation
Orders_federation
Orders_Fed
[5000, 7500) & [7500, 10000)
Table Details
MANAGE ANY DATA, ANY SIZE ANYWHERE
1010101010101010
01010101010101
101010101010
Hadoop Connectors & ETL
Frameworks
Services
Fabric
Infrastructure
..
.
virtual machines
..
.
web sites
..
.
cloud services
Automated
Elastic
Managed Resources
Usage Based
N Central US, S Central US, N Europe, W Europe, E Asia, SE
Asia + 24 Edge CDN Locations
SQL database
..
.
noSQL database
..
.
blob storage
connect
..
.
virtual network
traffic manager