Apache Storm - Introduction au traitement temps-réel avec Storm
HUGFR : Une infrastructure Kafka & Storm pour lutter contre les attaques DDoS en temps-réel par...
56
Steven Le Roux Infrastructure Engineer AntiDDoS : Threat Detection
-
Upload
hadoop-user-group-france -
Category
Internet
-
view
1.489 -
download
10
description
Presentation d'OVH lors du Hadoop User Group du 6 octobre à Paris. • Talk #4: Une infrastructure Kafka & Storm pour lutter contre les attaques DDoS en temps-réel par Steven Le Roux de la société OVH. http://hugfrance.fr/hug_paris_6_oct_cloudera/
Transcript of HUGFR : Une infrastructure Kafka & Storm pour lutter contre les attaques DDoS en temps-réel par...
Steven Le RouxInfrastructure Engineer
AntiDDoS : Threat Detection
OVH Anti-DDoS
VAC
9
3 Tbps 17 Datacenters
32 PoPs
19
Data Pipeline
Clients
Producers
Consumers
Brokers
Topics
Partitions
Replicas
/ kafka
/ kafka / topic
/ kafka / topic / replicas
/ kafka / topic / replicas / factor / 3
/ kafka / topics
/ kafka
/ kafka
/ kafka / producers
Stream Processing
Topology (DAG)
Spouts
Bolts
Cluster
Nimbus
Supervisors
Workers
/ storm
/ storm / topology
/ storm / topology / antiddos
Stream Grouping
Shuffle Grouping Field Grouping
Direct Grouping Other Grouping
/ storm
Attacks
Router Grouping
Scans
IP src Grouping
/ storm
Attacks
≈ 1s
ScoringFiltersBurst
Scans
IP
Proto
/ storm
Indexing
Prooving
Producing
/ storm / event
#lifecycle
#dataviz
Nice speech… … so what ?
False positives
Strange behaviours from customers
e.g. DB sync without connection pool
Application centric
i.e. UDP protocols
#issues
Add other sources
Application Anti-DDoSGame
Half Life/SourceCS:GOTeamSpeak / MumbleGTASA:MP…
More to come (any special need ?)
#solutions
#datalake
Nodes - Hardware
CPU 16c/32t
RAM 256GB
Disks : OS : Raid 1Data : 10 disks
per node200 MB/s ~ 1,5-2 Gbps
#hardware
Kafka
I/O bound
Bench (1node)1M+ msg/s
No compression
No ackers
80MB/s
Tuningnum.io.threadnum.network.threadsocket.*.buffer.*
Storm
CPU/RAM bound
M+ tuples/s
No ackers
Break SRP
Minimal workersAvoid transfer buffer
#config
OpenSOC
Clément Sciascia - @csciasci
Magnus Edenhill - @edenhillm
https://github.com/edenhill/librdkafka
LinkedIn - Apache Kafka
Nathan Marz - Apache Storm
#Thanks
#moreStorm basic training – Mickael G. Noll
http://fr.slideshare.net/miguno/apache-storm-09-basic-training-verisign
Kafka documentation
