Viadeo - Cost Driven Development

Click here to load reader

  • date post

    15-Feb-2017
  • Category

    Technology

  • view

    35
  • download

    2

Embed Size (px)

Transcript of Viadeo - Cost Driven Development

  • 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

    Frdric dAlessandro - Data architect

    Loc Dias Da Silva - CTO & VP Engineering

    Cost Driven Development chez ViadeoTches Spark sur EMR optimises avec des instances SPOT

  • Viadeo

    SUCCESS

    Business

    Fournisseurs, clients,

    collaborateurs,

    investisseurs...

    Networking

    Pairs, associs, collgues,

    camarades, rencontres

    pro, partenaires ...

    Carrire

    Premier job, recherche

    demploi, entreprenariat,

    dveloppement, ...

  • Migration vers le cloud AWS

    Nov. 2014

    Escape to Amazon thinking

    Dec. 2014

    First commit Infrastructure as a code

    Mar. 2015

    First industrialized instance on AWS cloud

    Dec. 2015

    Our demo environment is on AWS cloud

    July 2016

    Our live production environment is on AWS cloud

    Dec 2016

    Datacenter is closed

    demo

  • Cost-Driven Development

    - choix des instances Amazon EC2- autoscaling- optimisations Amazon RedShift- cycle de vie des donnes Amazon S3- utilisation dinstances SPOT

  • Architecture

  • Lanalytics chez Viadeo

  • Le Big Data chez Viadeo

  • Utilisation dinstances SPOT (ex: SnowPlow)- Prix c3.8xlarge la demande : 1.68$- Prix c3.8xlarge Spot : ~0.40$- Baisse du cot de ~ 1200$ ~240$ par mois

  • pop.sh : automatisation du lancement de jobs

  • Usage

    $ ./pop.sh --coreInstanceType=m4.large --coreInstanceCount=2 --ec2spot=true

    (...)

    # Pricing info = Tue Oct 11 08:00:09 CEST 2016

    # Instance type = m4.large

    # availabilityZone = us-west-1c

    # Aws Spot Price 1 min ago = 0.016700

    # Aws Spot Price 1 day ago = 0.016300

    # Our max Spot Price max proposal = 0.029

    # Aws fixed price = 0.14

    # Emr fixed price = 0.030

    # Do we take spot = 1

    $

  • Prochaines tapes

    - Amlioration de la vitesse de scaling- Passage des stacks applicatives en -services- Containerisation & orchestration