Transcript intelligence in real time

Brainstorm on Smart Energy
07/07/2010
www.feedzai.com
[email protected]
twitter.com/feedzai
Founding Team
Nuno Sebastião
Product Manager at the European Space Agency
Co-founder and ex-CEO at Evolve Space Solutions
MBA London Business School
Research
Pedro Bizarro
Professor University of Coimbra
Founding Member Event Processing Technical Society
PhD University Wisconsin at Madison
Paulo Marques
Director MSE – Carnegie Mellon University | Portugal
Professor University of Coimbra
Co-founder WIT Software
Business
Domain
FeedZai Confidential
Quality
FeedZai develops FeedZai PulseTM:
High performance processing
of large scale
high throughput
data loads to produce
intelligence in real time
FeedZai Confidential
Energy – Real Time Energy Consumption
Business Objective:
•Measure in real time the energy consumption of every
household (3.500.000) in Portugal.
•Provide up to date consumption information to consumers.
Business Use:
•Real Time charting dashboards.
•Calculate nationwide energy consumption levels.
•Calculate ecologic footprint from household up to nation level.
•Create a social network for energy efficiency.
Work done under contract with ISA
iEnergy portal – Copyright ISA 2010
Business Benefits:
•Estimated savings of 12.000 tons of CO2 due to end user
behaviour change.
•Accurate determination of power grid supply needs.
FeedZai Confidential
Live at: ienergy.isa.pt
Traditional Approach to Data Processing
Databases
Data Warehouses
Reports,
ad-hoc queries,
alarms
Slow and lagging behind real-time…
FeedZai Confidential
Traditional Approach to Data Processing
Source: Paper submitted by EDP to 20th
International Conference on Electricity
Distribution (2009)
FeedZai Confidential
FeedZai PulseTM – Actionable Data Processing
Databases
Pulse Engine - real-time
actionable information
Complex Event Processing
FeedZai Confidential
Data Warehouses
Data Mining
DBMS
•
•
•
•
•
•
•
Run-to-completion queries
Finite data
SQL
No patterns
Not temporal
Multi-pass, disk-based
Queries read data
results
SQL
CEP
vs
•
•
•
•
•
•
•
Continuous Queries
Never ending data stream
CCL/StreamQL/Rules/etc
Pattern matching
Temporal queries
Single-pass, memory-based
Data fires alarms, updates
events
CCL
FeedZai Confidential
Challenge: High Volume Data Processing
Satellite Data Processing
Objective:
•Monitor 50.000 telemetry parameters/second for 4 different
spacecrafts producing dasboarding and alarms on critical
events.
Key Performance Facts:
• Observing (50.000 parameters/sec) * (4 spacecrafts) = 200.000 param/sec
• Able to calibrate and check in real-time 80.000 data points/sec
• Generation of overall global indicators based on raw and calibrated data (max, min, avg, etc.)
• Running on a Dual-Core 2.56MHz machine w/ 8GB RAM
FeedZai Confidential
Challenge: Real-Time Processing
Banking Infrastructure Monitoring
Objective:
•Detect when something is wrong in an environment of varying
user behavior, 50.000.000 transactions/month and 200.000
banking terminals.
Key Performance Facts:
• 1.000 transactions/sec in simulated time
• 60.000 peak transactions/sec in real-time
• 16 queries each processing 60k events/sec   1 Million evt/sec
• Running on a Dual-Core 2.56MHz machine w/ 8GB RAM
FeedZai Confidential
Challenge: Scalability and Processing Unit Cost
Reality Today:
•How to get all the processing power from cheap CPUs and Data Centers.
Solution:
•Graphics Processing Units (GPUs)
are many times faster and cheaper
than CPUs
•Smart software instead of expensive hardware
•Money wise:
10 cheap machines = ~1 server
•Performance wise: 10 cheap machines = ~3 servers
Patent Pending for: “An
Apparatus for Data Stream
Processing Using Massively
Parallel Graphical Processors”
•Streaming: incrementally compute answers instead of re-computing from scratch
•Column-oriented: better garbage collection, better cache usage
FeedZai Confidential
Challenge: Data Visualization
Challenges:
•How to visualize and analyze all the information being
generated by all the smart meters?
•Alert fatigue: too many false alarms, too little real alarms
Solutions:
•Improved rules and baselines generate more reliable alarms
•New ways to observe data
•Color graded maps (heat maps)
•Aggregated city-wide, neighborhood-wide
•Compare multiple time periods
•Compare across similar families
FeedZai Confidential
Summary
Disruptive product based on state-of-the-art IP
Applicability for future large-scale Smart Grids
Development with feedback from real use-cases and clients
Award winning company
Development with feedback from real use-cases and pilots
“Outputs of the project are of
significant interest for our
business activities”
Ashish Gupta,
MD Global Platforms
BT.com
FeedZai Confidential
“Pulse complements our BI
offer”
Nuno Esculcas,
Country Manager
Microstrategy Portugal
The Brain of “Smart Energy”
Thank You
www.feedzai.com
[email protected]
twitter.com/feedzai