Data vault modeling et retour d'expérience

Post on 08-Jan-2017

303 views 1 download

Transcript of Data vault modeling et retour d'expérience

BÂLE BERNE BRUGG DUSSELDORF FRANCFORT S.M. FRIBOURG E.BR. GENÈVE

HAMBOURG COPENHAGUE LAUSANNE MUNICH STUTTGART VIENNE ZURICH

Agile Business Intelligence @ Evam

Plan

• Introduction ( F. Kang à Birang)

• Pre-project (F. Kang à Birang & J-M. Delacrétaz)

• Agile project management (A. Martino)

• Agile architecture (E. Fidel)

• Data quality (A. Martino)

• EVAM Feedback (B. Albietz)

Introduction

Fabienne Kang à Birang – Business Analyst / Product owner

Introduction

• EVAM Presentation

• Project Sponsor

• Director

• Indicators

• 2013 – Existing B.I.

Pre-Project Phase

Fabienne Kang à Birang – Business Analyst / Product owner

Jean-Marc Delacrétaz – Developer

Pre-Project Phase

• Target

• Operational reporting

• Problems encountered @ EVAM

• Data interpretation

• Business rules errors

• Prerequisites

• Dictionary

• Population hierarchized

Preexisting B.I.

• 2013

• P.O.C. to introduce B.I. «philosophy»

• Chosen Tools

• ETL : Talend

• Reporting : Tibco JasperReport

• Weaknesses

• Lack of expertise & methodology

• Bad performances

Decision in August 2014

• Start from scratch

• With Trivadis Lausanne as a partner

• Tools

• Performances

• Architecture with « Best practices »

Agile Project

Management

Adriano Martino – Senior B.I. Consultant

Agility

We are uncovering better ways of developing

software by doing it and helping others do it.

Through this work we have come to value:

• Individuals and interactions over processes and tools

• Working software over comprehensive documentation

• Customer collaboration over contract negotiation

• Responding to change over following a plan

Organisation

• Evam• Evam

• Trivadis

• Evam• Trivadis

Scrum MasterProduct

Owner

CustomerDeveloppers

Agile Objectives

• Deliver working software frequently

• Adapt to change

Scrum components overview

Sprint

Planning

Sprint

Backlog

Product

Backlog

Daily

Stand up

Sprint

2 to 4

weeks

Sprint

Review

Retrospective

Normal Process for a B.I. need

Business

Analysis

Design of the

modelImplementation

Unit TestingVolume

testing

User

Acceptance

Testing

New

Need

Rework

Rework Rework

Rework

Deployment

to Validation

Deployment Production

Normal Process for a B.I. need

Agile Objectives

• Adapt to change

• Deliver working software frequently

• At regular intervals, the team reflects on how to become more effective

Cadence

SCRUM

EVENT

DRIVEN

Sprint1 Sprint2 Sprint3 …

RetrospectiveReviewReleasePlanning1 2

1

3 4

2 3 4 1 2 3 4 1 2 3 4

1 2 2 2 2 213 42

Agile Objectives

• Adapt to change

• Deliver working software frequently

• At regular intervals, the team reflects on how to become more effective

• Work close to business

Collaborative Workshops

Business

Need

analysis

Technical

analysis

Live dev

Prototyping

Live

testing

Agile B.I. Architecture

• Evolutive

• Easy change management

• Parallelisable development

• Business oriented

• Integration

• Possibility to automate generation

We choose Data Vault

Modelling

Agile

Architecture

Eddie Fidel – Senior B.I. Consultant

STAGING

DYNAMIC ETL

Enterprise

Data

Warehouse

With data vault

Modeling

Agile Bi Architecture

SOURCESVirtualized

Data Marts

STAGING

DYNAMIC ETL

Enterprise

Data

Warehouse

With data vault

Modeling

Data Warehouse Layer

SOURCESVirtualized

Data Marts

DYNAMIC ETL

What is Data Vault ?

• Data Modelling Method for Data Warehouses in Agile Environments

• Developed by Dan Linsted

• Suitable for

• DWH Core Layer

• Optimized for

• Agility / Integration / Historization

Data Vault composition

• Decomposition of Source Data

• Split Data into Separate Parts

Hubs Business Entity

Links Relations

Satellites Contexts

Business Oriented

Data Vault composition

• Elements : Hub – Link – Sat

Customer

Sat

Sat

Sat

CustomerProduct

Sat

Sat

Sat

Product

Hub = List of Unique Business Keys

Link = List of Relationships, Associations

Satellites = Descriptive DataOrder

Sat

Sat

Sat

Order

Link

Avantages and challenges

• Standard ETL Rules to Load Data Vault

• Easy Extensibility of Data Vault Model

• Integration of Multiple Source Systems

• Traceability and Complete History

• High Number of Tables in Data Vault

What does the Data Vault generator do ?

• Tables

• Indexes

• Surrogate keys

• Foreign keys

• Partitions

• Loading process

• SCD1 / SCD2

• Loading audits

• Handling Errors

Generator value

29

Business spec

Technical spec

Development

Test

Deployment

Qu

ality

ass

ura

nce

Do

cum

en

tati

on

Simplify

Generator

Do

cum

en

tati

on

QS

Total Savings

Fast and short implementation cycles

Broad flexibility of change

Auto-generated quality assured components

Huge time and cost savings

On-going and recurrent with each

step of modification or enlargement!!!

STAGING

DYNAMIC ETL

Enterprise

Data

Warehouse

With data vault

Modeling

Dynamic ETL

SOURCESVirtualized

Data Marts

Dynamic ETL for DWH

• Parallel Loading

• HUB

• LINK et SAT

• Dynamic call to loading procedures

• No deployment of ETL needed

STAGING

DYNAMIC ETL

Enterprise

Data

Warehouse

With data vault

Modeling

Dynamic ETL

SOURCESVirtualized

Data Marts

DYNAMIC ETL

Data Mart

• Business Need Oriented

• Virtualized DM (materialized view)

• Can be regenerated from scratch

• Find value at a point in time

• Good perfomance

• Automatically regenerated (no deployment)

Data Quality

Adriano Martino – B.I. Consultant

Quality report

• Automated

• Daily execution

• Simple development

• Possible to send mail based on result

• Direction support to involve Business

EVAM Feedback

Bruno Albietz – I.T. Manager

Keys Learnings

• Show business value as early as possible and keep the ball rolling

• Project: December 2014 – June 2016

• Phased implementation: 1st output in June 2015, then regular outputs on a monthly basis

• Be prepared to spend most of your time on data quality

• The lifeblood of B.I. projects

Keys Learnings

• Prepare knowledge transfer to your staff during the project

• Modelling, ETL, Reporting

• Good project management practice, from business requirements to report development

• Increase user buy-in with Scrum

• Key users and management involved from day 1

Keys Learnings

• Learn to say “ No ”

• B.I. quality versus business process quality

• B.I. is also here to show process deficiencies, do not try to solve all business issues within the B.I. project

Q & A