Institut Mines-Télécom

The Analog Data Assimilation (¡AnDA!)Colloque Nationale d’Assimilation de Donnéesdu 30/11/16 au 02/12/16 à Grenoble, France

Pierre Tandeo (Télécom Bretagne, Brest)En collaboration avec :P. Ailliot, R. Fablet, R. Lguensat

Institut Mines-Télécom2

Data Assimilation: general context

2

Nonlinear state-space model:

→ hidden state → dynamical model → noisy/partial obs.

01/12/16 Colloque National d’Assimilation de Données 201630/11-02/12, Grenoble, France

■ Model issues• not enough (small scales, param.)• need to include error (bias, noise)• model badly/not known

■ AnDA idea• avoid the use of model• data-driven approach• emulate the model

Institut Mines-Télécom3

Data Assimilation: 2 points of view

3 01/12/16 Colloque National d’Assimilation de Données 201630/11-02/12, Grenoble, France

Institut Mines-Télécom4

Motivations: large datasets in oceanography

4 01/12/16 Colloque National d’Assimilation de Données 201630/11-02/12, Grenoble, France

■ Satellite data• interpolations• daily• mesoscale

■ Surface variables• Temperature

(40 years)• Height

(20 years)• Salinity

(10 years)

Institut Mines-Télécom

Catalog (analogs & successors)

5 01/12/16 Colloque National d’Assimilation de Données 201630/11-02/12, Grenoble, France

Temperature(SST)

Height(SSH)

Salinity(SSS)

Lorenz-63

Institut Mines-Télécom

Analog Forecasting (loc. const. & Gaussian)

6 01/12/16 Colloque National d’Assimilation de Données 201630/11-02/12, Grenoble, France

Lorenz-63

Find analogs and compute weights: Approximate transition distribution:

Institut Mines-Télécom7

Analog Forecasting

01/12/16 Colloque National d’Assimilation de Données 201630/11-02/12, Grenoble, France

■ Experiment• dt=0.08 (6h atmos.)• k=50 analogs, Gaussian assumpt.

■ Interests• estimate both mean and cov• low-cost procedure

Institut Mines-Télécom

AnDA = Analog Forecasting + Filtering Method

8

Step 0:

- start from last analysis step (time t)- N members with same weights (EnKF)- N particles with different weights (PF)

Step 1:

- find K nearest analogs(a1, …, aK)- compute their weights (w1, …, wK)- use appropriate distance/kernel

Step 2:

- combine the K successors (s1, ..., sK) using (w1, …, wK)- different analog forecasting strategies (constant, incremental, linear)

Step 3:

- different sampling schemes (Gaussian, multinomial)- compare forecasts to noisy observations- recombine N members (EnKF) or particles (PF)

t ← t+101/12/16 Colloque National d’Assimilation de Données 2016

30/11-02/12, Grenoble, France

Institut Mines-Télécom

Classic VS Analog Data Assimilation

9

■ Experiment• simulated data

(Lorenz-63)• 1 obs. variable

(variance R=2)• partial observations

(every 8 time steps)• K=50 analogs• perfect catalog

■ Result• equivalent

performance for large enough catalog

Classic Data AssimilationAnalog Data Assimilation

Ensemble Kalman FilterParticle FilterEnsemble Kalman Smoother

01/12/16 Colloque National d’Assimilation de Données 201630/11-02/12, Grenoble, France

Institut Mines-Télécom

Analog Data Assimilation with noisy catalog

10

■ Experiment• same as previously• 3 catalogs with different variances• K=50 analogs, AnEnKS

01/12/16 Colloque National d’Assimilation de Données 201630/11-02/12, Grenoble, France

■ Result• robust to the presence of

noise in the catalog

Var=0.5Var=1Var=2

Institut Mines-Télécom

Analog Data Assimilation for model evidence

11

■ Experiment• same as previously• 3 catalogs with different parameters• obs. generated with 1• K=50 analogs, AnPF

01/12/16 Colloque National d’Assimilation de Données 201630/11-02/12, Grenoble, France

■ Result• able to retrieve the good

parameterization:1 (61%), 2 (27%), 3 (12%)

1 = (10,28,8/3)2 = (13,28,8/3)3 = (7,28,8/3)

Institut Mines-Télécom

Local VS Global Analog Data Assimilation

12

■ Experiment• simulated data

(Lorenz-96)• 20 obs. variables

(noise R=2)• partial observations

(every 4 time steps)• analog forecasting

(k=50, 10^3 Lorenz-96 times)

■ Result• local strategy

overperforms the global one

01/12/16 Colloque National d’Assimilation de Données 201630/11-02/12, Grenoble, France

Institut Mines-Télécom13

AnDA: timeline

13

Conference paper (Climate Informatics): The Analog Ensemble Kalman Filter and Smoother, Tandeo et al.Sep2014

Journal paper (submitted): The Analog Data Assimilation, Lguensat et al.Nov2016

Book chapter (Machine Learning and Data Mining Approaches to Climate Science): Combining analog method and ensemble data assimilation: application to the Lorenz-63 chaotic system, Tandeo et al.

Jan2015

Conference paper (Climate Informatics): The analog data assimilation: application to 20 years of altimetric data, Tandeo et al.Sep2015

Journal paper (Physical Review X): The Ensemble Kalman Filtering without a Model, Hamilton et al.Mar2016

Conference paper (OCEANS): Using archived datasets for missing data interpolation in ocean remote sensing observation series, Lguensat et al.Jun2016

01/12/16 Colloque National d’Assimilation de Données 201630/11-02/12, Grenoble, France

Institut Mines-Télécom14

AnDA: collaborations

14

Télécom Bretagne, UBO, UBS, Rennes 1

Univ. Buenos Aires(climate)

Univ. ColoradoNCAR/IMAGe

Ocean Univ. of China(oceanography)

McGill Univ.(meteorology)

Methodology

Applications

IFREMER(oceanography)

Univ. Corrientes

01/12/16 Colloque National d’Assimilation de Données 201630/11-02/12, Grenoble, France

Pathfinder SST

Rainfall radar

SPEEDY simulations

AVISO/GCM SSH

Institut Mines-Télécom

Conclusions & Perspectives

15

■ Conclusions• data-driven assimilation

─ use historical datasets─ satellite data, simulations

• various implementations─ analog forecasting─ local/global analogs─ filtering method

• Pyton library on github • AnDA paper on researchgate

■ Perspectives• Methodology

─ other methods (EKF, HMM, 4D-VAR) ─ spatio-temporal dynamic kernels─ distance metric learning

• Applications─ realistic examples─ oceanography, climate, meteorology─ other fields?

• collaborations are welcomed!

01/12/16 Colloque National d’Assimilation de Données 201630/11-02/12, Grenoble, France1515

Institut Mines-Télécom

The birthplace of the Analog Data Assimilation

16

Thank you for your attention!Any questions?

01/12/16 Colloque National d’Assimilation de Données 201630/11-02/12, Grenoble, France

Data Science & Environment

3-7 July 2017, Brest, France

Workshop + Summer Schoolwebsite

Institut Mines-Télécom

Additional slides

17 Colloque National d’Assimilation de Données 201630/11-02/12, Grenoble, France

Institut Mines-Télécom18

Analog Method

■ State of the art• Lorenz (1969)“atmospheric predictability using naturally occurring analogs from past records”

• Van Den Dool (1994)“it would take order 10^30 years to find analogues that match over the entire Northern Hemisphere 500mb height field to within current observational error”

■ Revival• Amount of data

─ from satellite─ numerical simulations

• Data mining tools─ recent machine learning methods─ fast nearest neighbor search

• Scientific interest for few years─ Yiou, Climate Dynamics (2013)─ Delle Monache et al., MWR (2015)─ Dernott & Wikle, Environmetrics (2015)─ Atencia & Zawadzki, MWR (2015)─ Zhao & Giannakis, Nonlinearity (2016)

01/12/16 Colloque National d’Assimilation de Données 201630/11-02/12, Grenoble, France

Institut Mines-Télécom

AnDA: Analog Forecasting strategies

19 01/12/16 Colloque National d’Assimilation de Données 201630/11-02/12, Grenoble, France