Professional Documents
Culture Documents
to
Engineer
Architecting Big Data
Solutions
John Doe
Tr a i n i n g & P r a c t i c e B u i l d i n g f o r S o l u t i o n A r c h i t e c t s
MICROSOFT CONFIDENTIAL
Agenda
MICROSOFT CONFIDENTIAL
Engineer to
Engineer
MICROSOFT CONFIDENTIAL
Data Processing
Data without compute has limited value
Complex problems require massive
OLTP
Original
data
Transform
ETL Tool
(SSIS, etc)
Load
Transformed
data
EDW
(SQL Server, Teradata, etc)
ERP
BI Tools
Data Marts
LOB
Scale-out storage & compute
(HDFS, Blob Storage, etc)
Data
Lake(s)
Dashboard
s
Apps
Ingest (EL)
Original
data
Streaming data
MICROSOFT CONFIDENTIAL
Marketing
MICROSOFT CONFIDENTIAL
Predictive
Maintenance
Engineer to
Engineer
Demo
Web Log Processing and Big Data
MICROSOFT CONFIDENTIAL
Engineer to
Engineer
Lambda Architecture
MICROSOFT CONFIDENTIAL
Batch ingest
Partial
aggregate
Partial
aggregate
Partial
aggregate
Merged
view
Batch views
Bufered ingest
New data
stream
Precompute
views
All data
Batch layer
Real-time data
Real-time views
Process
Stream
Increment
views
MICROSOFT CONFIDENTIAL
Speed layer
Event Processing
Logic
Inbound
Data
Bufered
Ingestion
(message bus)
scoring
Hot
Store
Spooling/Archivin
g
Curation
Data Movement
/ Sync
Consumption
Analytical
Store
MICROSOFT CONFIDENTIAL
Lambda in Azure
Area
Service Offerings
Buffered Ingestion
Event Hubs
Real-Time Processing
Service Bus
Storage Queues
Storm on HDInsight
Storm on IaaS
Spark Streaming
Enrichment
Azure Machine Learning
HBase on HDInsight
Batch Processing /
Curation
Kafka
RabbitMQ
HDInsight
Consumption
Power BI
MICROSOFT CONFIDENTIAL
Engineer to
Engineer
MICROSOFT CONFIDENTIAL
INTELLIGENCE
Information
Management
ACTION
Machine Learning
and Analytics
Business
apps
Azure
Machine Learning
Azure
Data Factory
Azure
Data Lake
Custom
apps
Azure
SQL Data Warehouse
Azure
Event Hub
Power BI
Personal Digital
Assistant
Cortana
Azure
HDInsight (Hadoop)
Azure
Data Catalog
Dashboards and
Visualizations
People
Perceptual
Intelligence
Face, vision
Speech, text
Azure
Stream Analytics
Sensors
and devices
Business
Scenarios
Recommendations,
customer churn,
forecasting, etc.
MICROSOFT CONFIDENTIAL
Automate
d
Systems
Finance
and risk
Customer
and channel
Operations
and workforce
Customer Acquisition
Fraud detection
Lifetime customer
value
Credit risk
management
Personalized ofers
Operational efficiency
Product
recommendation
Smart buildings
Loyalty programs
Marketing mix
optimization
Predictive maintenance
Supply chain
management
MICROSOFT CONFIDENTIAL
Healthcare
Financial Services
Industry Overview
Industry Overview
Industry Overview
7.9% of US Economy
Industry Trends
Commerce anywhere, anytime
Personal and in context
Scenarios
Sales & Marketing: Demand Forecasting
Customer & Channel: Personalized Ofers
Industry Trends
Payments based on patient outcome
Government monitoring hospital
readmissions
and number of hospital-acquired conditions
Industry Trends
Understanding customer behavior
Cybersecurity
Taking banking mobile
Proactive regulatory compliance
Scenarios
Scenarios
MICROSOFT CONFIDENTIAL
A managed cloud service for building & operating data pipelines (aka. data flows)
MICROSOFT CONFIDENTIAL
Cloud data warehouse that can grow, shrink, and pause in seconds
processing
Independent scale of compute and storage
in seconds
Transact-SQL queries across relational and
non-relational data
Full enterprise-class SQL Server experience
Works seamlessly with Power BI, Machine
Learning, HDInsight, and Data Factory
MICROSOFT CONFIDENTIAL
Azure HDInsight
Hadoop in Azure
Name Node
Region Server
Region Server
Region Server
Region Server
Job Tracker
Data Node
Data Node
Data Node
Data Node
Task Tracker
Task Tracker
Task Tracker
Task Tracker
MICROSOFT CONFIDENTIAL
Complex
Event
Processor
Output Adapter
MICROSOFT CONFIDENTIAL
Input Adapter
Power BI
Data
sources
Power BI service
SaaS solutions
e.g. Marketo, Salesforce,
GitHub, Google analytics
Content packs
On-premises data
e.g. Analysis Services
Organizational content
packs
Corporate data sources or
external data services
Live dashboards
Visualizations
Azure services
Azure SQL, Stream Analytics
Excel files
Reports
01001
Datasets
10101
Data refresh
MICROSOFT CONFIDENTIAL
Engineer to
Engineer
Opportunity
75% of the cars shipped
globally by 2020 will be
built with the necessary
hardware to connect to the
Internet
MICROSOFT CONFIDENTIAL
Solution Architecture
Data
Ingest
Prepare
Analyze
Publish
Consume
Azure ML
Event Hub
Custom Dashboard
Application
Data in motion
Data at rest
HDInsight
HDInsight Azure ML
Blob storage
Vehicle catalog
import
Power BI
Dashboar
ds
Power BI Dashboard
MICROSOFT CONFIDENTIAL
Engineer to
Engineer
Demo
Connected Cars and Cortana Analytics Suite
MICROSOFT CONFIDENTIAL
Summary
Comprehensive ofer for our Big Data and AA cloud
portfolio
Enables a broad class of business specific scenarios
Integrates with investments in Perceptual
Intelligence & Cortana
Customers can get going in individual services today
Available to transact this fall as a simple monthly
subscription
Pricing and included quantities are still TBD
MICROSOFT CONFIDENTIAL
2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S.
and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must
respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information
provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
MICROSOFT CONFIDENTIAL
Recommendations
What should I do?
Decision support
Data
Predictions
Decision
Interactive dashboards
Why did it happen?
Static reports
What happened?
Manual process
Value
MICROSOFT CONFIDENTIAL
Action
People
Custom
apps
Cortana
Analytics
Sensors
and
devices
DATA
INTELLIGENCE
MICROSOFT CONFIDENTIAL
Automated
Systems
ACTION
Preconfigured Solutions
Dashboards and Visualizations
Machine Learning and Analytics
Big Data Store
Information Management
MICROSOFT CONFIDENTIAL
Background
Dartmouth-Hitchcock is a leading healthcare provider in New England with 1.2 million patients
DH chose Microsoft for ML-led IoT solution, over IBM, Google, Amazon, Accenture, and Cognizant
Problem
High re-admit rates, costing hospitals thousands per occurrence ($Ms/year)
Treatment plans (pathways) treat to the average and are static based on decades-old data
Poor engagement between patients and medical providers outside the hospital
Solution
ImagineCare, a 24/7 remote medical sensing solution backed by a live nurse-led clinic
Several MSFT products/services used, including ML, HDInsight, Power BI, Event Hub, ASA, Band
ML is core of solution to predict occurrences before they happen based on patients just like you
(i.e., micro-segmenting the population), laying foundation for dynamic, patient-specific pathways
SOLUTION
BENEFITS
o Empowered, healthier patients with
personalized, evidence-based treatment
plans and collaborative care
o Millions of dollars saved in readmission
costs, unnecessary ER and doctor visits,
and missed work
o Improved quality of life with 360-degree
view of patient health, mood, and
behavior
o More efective population health
management
and wellness
to the population.
Despite all of the technology involved, ImagineCare does not lose that human touch, which is so
Nathan Larson
important.
Director of Remote Medical Sensing
MICROSOFT CONFIDENTIAL
DARTMOUTHHITCHCOCK