Transcript Day 1 Module 1 - Introduction to Big Data

Microsoft Big Data
Essentials
Module 1 - Introduction to Big Data
Saptak Sen, Microsoft
Bill Ramos, Advaiya
Agenda
• Why Big Data?
• Big Data Lambda Architecture
• Getting started with Windows Azure HDInsight
Service
Big Data
Lambda
Architecture
• Batch layer
•
•
Stores master dataset
Compute arbitrary views
• Speed layer
•
•
Fast, incremental algorithms
Batch layer eventually overrides
speed layer
• Serving layer
•
•
Random access to batch views
Updated by batch layer
• Stores master dataset
(in append mode)
• Unrestrained
computation
• Horizontally scalable
• High latency
• Stream processing of
data
• Stores a limited window
of data
• Dynamic computation
• Queries the batch and
real-time views
• Merges the results
Apache Hadoop
Staging Database
SQL Server Analysis Service (SSAS)
Microsoft Excel and PowerPivot
Other BI Tools and Custom
Applications
SQL Server Connector (Hadoop Hive ODBC)
SQL Server
Analysis Services
(SSAS Cube)
Hadoop Data
Third Party
Database
+
Custom
Applications
Windows Azure HDInsight
Reactive Extensions (Rx)
Microsoft Dynamics AX
SQL Server Database (InMemory OLTP)
SQL Server Analysis Services
Reactive Extensions
(Rx)
SQL Server
(In-Memory OLTP)
SQL Server Reporting Services
Data Feed from Smart Meters
Microsoft
Dynamics
AX
SQL Server
Analysis
Services
SQL Server
Reporting
Services
Windows
Azure
Storage
Azure Blob storage
Azure Storage Explorer
• Store large amounts of
unstructured text or binary
data with the fastest read
performance
• Highly scalable, durable, and
available file system
• Blobs can be exposed
publically over HTTP
• Securely lock down
permissions to blobs
Windows Azure HDInsight
Azure Blob storage
HDInsight Console
Windows Azure HDInsight
Azure Blob storage
HDInsight Console
Easy Access
to Data, Big
& Small
http://blogs.msdn.com/b/windowsazure/archive/2013/03/
19/getting-started-with-hdinsight.aspx
http://blogs.msdn.com/b/windowsazure/archive/2013/03/
21/azure-hdinsight-and-azure-storage.aspx
Questions?