Data vault modeling et retour d'expérience

40
BÂLE BERNE BRUGG DUSSELDORF FRANCFORT S.M. FRIBOURG E. BR. GENÈVE HAMBOURG COPENHAGUE LAUSANNE MUNICH STUTTGART VIENNE ZURICH Agile Business Intelligence @ Evam

Transcript of Data vault modeling et retour d'expérience

Page 1: Data vault modeling et retour d'expérience

BÂLE BERNE BRUGG DUSSELDORF FRANCFORT S.M. FRIBOURG E.BR. GENÈVE

HAMBOURG COPENHAGUE LAUSANNE MUNICH STUTTGART VIENNE ZURICH

Agile Business Intelligence @ Evam

Page 2: Data vault modeling et retour d'expérience

Plan

• Introduction ( F. Kang à Birang)

• Pre-project (F. Kang à Birang & J-M. Delacrétaz)

• Agile project management (A. Martino)

• Agile architecture (E. Fidel)

• Data quality (A. Martino)

• EVAM Feedback (B. Albietz)

Page 3: Data vault modeling et retour d'expérience

Introduction

Fabienne Kang à Birang – Business Analyst / Product owner

Page 4: Data vault modeling et retour d'expérience

Introduction

• EVAM Presentation

• Project Sponsor

• Director

• Indicators

• 2013 – Existing B.I.

Page 5: Data vault modeling et retour d'expérience

Pre-Project Phase

Fabienne Kang à Birang – Business Analyst / Product owner

Jean-Marc Delacrétaz – Developer

Page 6: Data vault modeling et retour d'expérience

Pre-Project Phase

• Target

• Operational reporting

• Problems encountered @ EVAM

• Data interpretation

• Business rules errors

• Prerequisites

• Dictionary

• Population hierarchized

Page 7: Data vault modeling et retour d'expérience

Preexisting B.I.

• 2013

• P.O.C. to introduce B.I. «philosophy»

• Chosen Tools

• ETL : Talend

• Reporting : Tibco JasperReport

• Weaknesses

• Lack of expertise & methodology

• Bad performances

Page 8: Data vault modeling et retour d'expérience

Decision in August 2014

• Start from scratch

• With Trivadis Lausanne as a partner

• Tools

• Performances

• Architecture with « Best practices »

Page 9: Data vault modeling et retour d'expérience

Agile Project

Management

Adriano Martino – Senior B.I. Consultant

Page 10: Data vault modeling et retour d'expérience

Agility

We are uncovering better ways of developing

software by doing it and helping others do it.

Through this work we have come to value:

• Individuals and interactions over processes and tools

• Working software over comprehensive documentation

• Customer collaboration over contract negotiation

• Responding to change over following a plan

Page 11: Data vault modeling et retour d'expérience

Organisation

• Evam• Evam

• Trivadis

• Evam• Trivadis

Scrum MasterProduct

Owner

CustomerDeveloppers

Page 12: Data vault modeling et retour d'expérience

Agile Objectives

• Deliver working software frequently

• Adapt to change

Page 13: Data vault modeling et retour d'expérience

Scrum components overview

Sprint

Planning

Sprint

Backlog

Product

Backlog

Daily

Stand up

Sprint

2 to 4

weeks

Sprint

Review

Retrospective

Page 14: Data vault modeling et retour d'expérience

Normal Process for a B.I. need

Business

Analysis

Design of the

modelImplementation

Unit TestingVolume

testing

User

Acceptance

Testing

New

Need

Rework

Rework Rework

Rework

Deployment

to Validation

Deployment Production

Page 15: Data vault modeling et retour d'expérience

Normal Process for a B.I. need

Page 16: Data vault modeling et retour d'expérience

Agile Objectives

• Adapt to change

• Deliver working software frequently

• At regular intervals, the team reflects on how to become more effective

Page 17: Data vault modeling et retour d'expérience

Cadence

SCRUM

EVENT

DRIVEN

Sprint1 Sprint2 Sprint3 …

RetrospectiveReviewReleasePlanning1 2

1

3 4

2 3 4 1 2 3 4 1 2 3 4

1 2 2 2 2 213 42

Page 18: Data vault modeling et retour d'expérience

Agile Objectives

• Adapt to change

• Deliver working software frequently

• At regular intervals, the team reflects on how to become more effective

• Work close to business

Page 19: Data vault modeling et retour d'expérience

Collaborative Workshops

Business

Need

analysis

Technical

analysis

Live dev

Prototyping

Live

testing

Page 20: Data vault modeling et retour d'expérience

Agile B.I. Architecture

• Evolutive

• Easy change management

• Parallelisable development

• Business oriented

• Integration

• Possibility to automate generation

We choose Data Vault

Modelling

Page 21: Data vault modeling et retour d'expérience

Agile

Architecture

Eddie Fidel – Senior B.I. Consultant

Page 22: Data vault modeling et retour d'expérience

STAGING

DYNAMIC ETL

Enterprise

Data

Warehouse

With data vault

Modeling

Agile Bi Architecture

SOURCESVirtualized

Data Marts

Page 23: Data vault modeling et retour d'expérience

STAGING

DYNAMIC ETL

Enterprise

Data

Warehouse

With data vault

Modeling

Data Warehouse Layer

SOURCESVirtualized

Data Marts

DYNAMIC ETL

Page 24: Data vault modeling et retour d'expérience

What is Data Vault ?

• Data Modelling Method for Data Warehouses in Agile Environments

• Developed by Dan Linsted

• Suitable for

• DWH Core Layer

• Optimized for

• Agility / Integration / Historization

Page 25: Data vault modeling et retour d'expérience

Data Vault composition

• Decomposition of Source Data

• Split Data into Separate Parts

Hubs Business Entity

Links Relations

Satellites Contexts

Business Oriented

Page 26: Data vault modeling et retour d'expérience

Data Vault composition

• Elements : Hub – Link – Sat

Customer

Sat

Sat

Sat

CustomerProduct

Sat

Sat

Sat

Product

Hub = List of Unique Business Keys

Link = List of Relationships, Associations

Satellites = Descriptive DataOrder

Sat

Sat

Sat

Order

Link

Page 27: Data vault modeling et retour d'expérience

Avantages and challenges

• Standard ETL Rules to Load Data Vault

• Easy Extensibility of Data Vault Model

• Integration of Multiple Source Systems

• Traceability and Complete History

• High Number of Tables in Data Vault

Page 28: Data vault modeling et retour d'expérience

What does the Data Vault generator do ?

• Tables

• Indexes

• Surrogate keys

• Foreign keys

• Partitions

• Loading process

• SCD1 / SCD2

• Loading audits

• Handling Errors

Page 29: Data vault modeling et retour d'expérience

Generator value

29

Business spec

Technical spec

Development

Test

Deployment

Qu

ality

ass

ura

nce

Do

cum

en

tati

on

Simplify

Generator

Do

cum

en

tati

on

QS

Total Savings

Fast and short implementation cycles

Broad flexibility of change

Auto-generated quality assured components

Huge time and cost savings

On-going and recurrent with each

step of modification or enlargement!!!

Page 30: Data vault modeling et retour d'expérience

STAGING

DYNAMIC ETL

Enterprise

Data

Warehouse

With data vault

Modeling

Dynamic ETL

SOURCESVirtualized

Data Marts

Page 31: Data vault modeling et retour d'expérience

Dynamic ETL for DWH

• Parallel Loading

• HUB

• LINK et SAT

• Dynamic call to loading procedures

• No deployment of ETL needed

Page 32: Data vault modeling et retour d'expérience

STAGING

DYNAMIC ETL

Enterprise

Data

Warehouse

With data vault

Modeling

Dynamic ETL

SOURCESVirtualized

Data Marts

DYNAMIC ETL

Page 33: Data vault modeling et retour d'expérience

Data Mart

• Business Need Oriented

• Virtualized DM (materialized view)

• Can be regenerated from scratch

• Find value at a point in time

• Good perfomance

• Automatically regenerated (no deployment)

Page 34: Data vault modeling et retour d'expérience

Data Quality

Adriano Martino – B.I. Consultant

Page 35: Data vault modeling et retour d'expérience

Quality report

• Automated

• Daily execution

• Simple development

• Possible to send mail based on result

• Direction support to involve Business

Page 36: Data vault modeling et retour d'expérience

EVAM Feedback

Bruno Albietz – I.T. Manager

Page 37: Data vault modeling et retour d'expérience

Keys Learnings

• Show business value as early as possible and keep the ball rolling

• Project: December 2014 – June 2016

• Phased implementation: 1st output in June 2015, then regular outputs on a monthly basis

• Be prepared to spend most of your time on data quality

• The lifeblood of B.I. projects

Page 38: Data vault modeling et retour d'expérience

Keys Learnings

• Prepare knowledge transfer to your staff during the project

• Modelling, ETL, Reporting

• Good project management practice, from business requirements to report development

• Increase user buy-in with Scrum

• Key users and management involved from day 1

Page 39: Data vault modeling et retour d'expérience

Keys Learnings

• Learn to say “ No ”

• B.I. quality versus business process quality

• B.I. is also here to show process deficiencies, do not try to solve all business issues within the B.I. project

Page 40: Data vault modeling et retour d'expérience

Q & A