Analyse en temps réel de BigData

16
L’analyse en temps réel de Big Data Le monitoring de flux par l’exemple Abdellatif BOUCHAMA @a_bouchama

Transcript of Analyse en temps réel de BigData

Page 1: Analyse en temps réel de BigData

L’analyse en temps réel de Big Data Le monitoring de flux par l’exempleAbdellatif BOUCHAMA@a_bouchama

Page 2: Analyse en temps réel de BigData

Abdellatif BOUCHAMA@A_BOUCHAMA

• Middleware architect, and passionate about the new technology: Big Data,IoT and Open source

• Co-founder of BusHorn.com

Page 3: Analyse en temps réel de BigData

Agenda

Big Data in 2015: Trends & Statistics

The emergence of real time

Use Case: Flow Activity Monitoring

Demo

Page 4: Analyse en temps réel de BigData

What is Big Data?

Big Data

Velocity 30KB/s --> 30GB/s· Batch · Real-Time · Streaming

Volume

1-2 Terabytes à ∞

Variety

· Structured · Semi-Structured · UnStructured

Page 5: Analyse en temps réel de BigData

Big Data Expectations:Things That Excite Executives About Big Data

It’s the “next oil.” from the Ginni Rometty, CEO of IBM

The potential to revolutionize industries, and change business models (e.g., Uber and Airbnb), with what we learn from the data.

The ability to provide real-time graphing solutions for data relationships.

Real-time operational and business data allows people to make decisions faster thereby saving significant money.

The opportunities it gives us to help clients solve real business problems.

https://dzone.com/articles/14-things-that-excite-executives-about-big-data?oid=big_data

Page 6: Analyse en temps réel de BigData

Big Data view by Gartner:

Page 7: Analyse en temps réel de BigData

Big Data in 2015:

75% of Companies Are Investing or Planning to Invest in Big Data in the Next Two Years

Goals for Big Data initiatives:1. Enhancing the customer experience2.Streamlining existing processes3.Achieving more targeted marketing and reducing costs

Last year, 37% of big data projects were initiated by the CIO, while 25% were initiated by business unit heads. In 2015, the roles are nearly tied, at 32% and 31 %, respectively.

http://www.gartner.com/newsroom/id/3130817

Page 8: Analyse en temps réel de BigData

The emergence of Real-Time

Low

Pure BatchOperationalizing

Near Real TimeOr InteractiveAnalytics

High

Real Time Analytics

• Right time

Real time

• Smart data

Big Data

Page 9: Analyse en temps réel de BigData

Use case: Flow Activity Monitoring

Page 10: Analyse en temps réel de BigData

Architecture

Page 11: Analyse en temps réel de BigData

Constraints & Requirements

Constraints• Non intrusive system• No modification on business flows.• We can plug it and unplug it easily.Requirements• System cost should be mastered

and adaptable• System automatized• Measurement

Page 12: Analyse en temps réel de BigData

Why Elasticsearch stack?Open source

Easy to deploy

Distributed

Linear scalability

REST Interface

Kibana & Logstash to complete the Landscape

Page 13: Analyse en temps réel de BigData

Data flow and constraints

Collect

JMX

Store & transport

Transform & Access

Modelize & Analyze

Visualize & Predict

JMS

Page 14: Analyse en temps réel de BigData

Demo Time!Be prepared for it to fail, because demos always do

Page 15: Analyse en temps réel de BigData

What’s Next ?• Enrichment analysis, after adding

business information in the logs

• Integration of backend and frontend applications

• Audit

Page 16: Analyse en temps réel de BigData

Thank you