VETESS : MDE, Testing approaches and SysML

May 2010

214/01/11

Project organization

Vérification de systèmes embarqués VEhicules par génération automatique de TESts à partir des SpécificationsChecking automotive embedded systems with automatic test case generation from specifications

• Started in 07/2008• En in 08/2010

314/01/11

Plan

Test in system engineering

Model Based Testing

The VETESS tool chain

The front wiper case study

Outlook

414/01/11

Plan

Test in system engineering

Model Based Testing

The VETESS tool chain

The front wiper case study

Outlook

Test in system engineering

514/01/11

Product specifications Product

Requirements

Specifications

Prototyping Unit testing

Fonctionnaltesting

Validation plan, integration

Integration

HIL

IntegrationDesign

Conception

Implementation

Problem: sharing specifications

614/01/11

Product specifications Product

Requirements

Specifications

Prototyping Unit testing

Fonctionnaltesting

Validation plan, integration

Integration

HIL

IntegrationDesign

Conception

Implementation

714/01/11

Project objectives

Checking automotive embedded systems with automatic test case generation from specifications• The purpose is to generate test cases directly

from the models representing system specification

814/01/11

Project objectives

Checking automotive embedded systems with automatic test case generation from specifications• Specification models:

− UML or SysML

• Test cases:− Enter test beds

914/01/11

Plan

Test in system engineering

Model Based Testing

The VETESS tool chain

The front wiper case study

Outlook

1014/01/11

Testing

Verification: checking coherence between• A model as engineered from requirements to

represent the system− Functional behavior− Non-functional properties (performance…)

• The system− At runtime− Alone or in situation

1114/01/11

Testing

Static verification• Code reviews• Model checking / proof

Dynamic verification• System stimulation and behavior control

− Can’t cover all possible cases– Most representative test cases have to be selected

− Often more expensive that the system itself– Execute tests– Produce / maintain tests– Analyze results

Testing

Questions:• Do I have enough tests ?• Are my test covering all possibilities offered

by the specification ?• Do my tests execute in a reasonable time ?

1214/01/11

1314/01/11

Test approaches

Manual approach• Scenarios describe

system stimulation and responses

• Human execute scenarios and check results

RequirementsTestplan

Testcases

Testresults

SUT

1414/01/11

Test approaches

Capture/replay approach• Stimuli and responses

are recorded and can be replayed automatically

SUT

RequirementsTestplan

Testcases Test

results

Recorder TestScripts

1514/01/11

Test approaches

Scripted approach• Scripts are written

− JUnit, TestNG,…

SUT

Requirements

Testplan

TestCases

Testresults

TestScripts

1614/01/11

Test approaches

Keyword-based approach• Scripts are written in

some pseudo-code• Concretization matches

keywords with real-life messages

SUT

Requirements

Testplan

Testcases

Testresults

TestScripts

Keywordenvironment

Adapter

Test Execution Tool

1714/01/11

Test approaches

Model Based Testing• A large number of

keyword-based tests are generated from an executable model

SUT

Requirements Testplan

TestCases

Testresults

Testscripts

Model Requirementstracability

matrixTest casesgenerator

Test scriptgenerator

Advantages

Automatically covers the specification model as much as possible

Test case mutualization• 1 test case for covering different parts of the

model

Predicts system behavior (oracle)

1814/01/11

Model Based Testing

14/01/11

SpécificationSpécification

Test caseTest casegenerationgeneration

tooltoolTest Test

RunnerRunnerInterfaceInterface

Test casesTest cases

1. Modeling 2. Generation 3. Concretization 4. Analysis

Simulation vs. test beds

2014/01/11

SpécificationSpécification

Test caseTest casegenerationgeneration

tooltoolTest Test

RunnerRunnerInterfaceInterface

Test casesTest cases

TestTestBedsBeds

SimulationSimulation

ModeMode

1. Modeling 2. Generation 3. Concretization 4. Analysis

2114/01/11

Plan

Test in system engineering

Model Based Testing

The VETESS tool chain

The front wiper case study

Outlook

2214/01/11

Test Designer

Model-based testing tool for discrete systems• The model is a behavioral specification

of the system− UML model or SysML models

• Test can be exported− Keyword− JUnit− HP Quality Center

Test-In-View

Testing continuous systems

Input and outputs are sampled signals• Expected signals are compared

to actual signals− Specific comparators

(Ignoring threshold, accepting latency…)

2314/01/11

Input

System Comparator

Actual

Expected

Result

Test may be exercised on• A system on a test bed• A simulation model

(Matlab, .NET…)

Test-In-View

2414/01/11

Input

System

ComparatorActual

Expected

Result

Model

Actual

Tool chain

2514/01/11

Test Designer

™ibd [block] PowerSubsystem [Al ternative 1 - Combined Motor Generator]

emg:ElectricMotorGener ator

trsm:Transmission

ice:InternalCombustionEngine

acl:accelerator

ecu:PowerControlUnit

ft:F uelT ankAssy

di f:Di fferential

rfw:Chass isSubsytem.FrontWheel

lfw:Chass is Subsytem.F rontWheel

Port:FuelTankFitting

Port:ICEFuelFi tting

fuelDelivery

torqueOut:T orque

torquein:T orque

spline

fuelSupply:Fuel

epc:ElectricalPowerController

bp:BatteryPack

i1:ElectricCurrent

i2 :ElectricCurrent

fp:F uelPump

fi:FuelInjector

fdistbp:BrakeSubsystem.BrakePedal

<>

<>

<><>

4

fuelReturn:F uel

<>

<>

<>

<>

g1:Torque

t2:T

orqu

e

t1:Torque

ice

ct rl

I_ICECmds

I_ICECmds

ctrl

ctrl

I_I CEData I_ICEData

trsmepc

c3

c2

c1

I_IEPCCmdI_IEPCData

I_IEPCDataI_EPCCmd

I_TRSMData

I_T RSMCmd

I_TRSMCmd

I_T RSMData

<>

<>

<>

rightHalfShaft

<>

<><>

leftHal fShaft

UML4MBTMODEL

SysML4MBTMODEL

UML MODEL(RSM, Topcased, …)

TestTestCasesCases

AdaptationAdaptationLayerLayer

+

SysML MODEL(Topcased)

Tool chain

2614/01/11

Test Designer

™ibd [block] PowerSubsystem [Al ternative 1 - Combined Motor Generator]

emg:ElectricMotorGener ator

trsm:Transmission

ice:InternalCombustionEngine

acl:accelerator

ecu:PowerControlUnit

ft:F uelT ankAssy

di f:Di fferential

rfw:Chass isSubsytem.FrontWheel

lfw:Chass is Subsytem.F rontWheel

Port:FuelTankFitting

Port:ICEFuelFi tting

fuelDelivery

torqueOut:T orque

torquein:T orque

spline

fuelSupply:Fuel

epc:ElectricalPowerController

bp:BatteryPack

i1:ElectricCurrent

i2 :ElectricCurrent

fp:F uelPump

fi:FuelInjector

fdistbp:BrakeSubsystem.BrakePedal

<>

<>

<><>

4

fuelReturn:F uel

<>

<>

<>

<>

g1:Torque

t2:T

orqu

e

t1:Torque

ice

ct rl

I_ICECmds

I_ICECmds

ctrl

ctrl

I_I CEData I_ICEData

trsmepc

c3

c2

c1

I_IEPCCmdI_IEPCData

I_IEPCDataI_EPCCmd

I_TRSMData

I_T RSMCmd

I_TRSMCmd

I_T RSMData

<>

<>

<>

rightHalfShaft

<>

<><>

leftHal fShaft

UML4MBTMODEL

SysML4MBTMODEL

UML MODEL(RSM, Topcased, …)

TestTestCasesCases

AdaptationAdaptationLayerLayer

+

SysML MODEL(Topcased)

UML4MBT / SysML4MBT

UML4MBT:Accepted input data• Formalized as a metamodel• Subset of UML• API (required interface) for Test Designer

SysML4MBT• Subset of SysML• Model transformation

from SysML4MBT to UML4MBT2714/01/11

Other languages can be supported through model transformation

2814/01/11

UML4MBTTest

DesignerModeleur UML+ TD Plug-in

TestCases

SysMLModeling Tool SysML4MBT

SysML4MBTto UML4MBT

Tool chain

2914/01/11

Test Designer

™ibd [block] PowerSubsystem [Al ternative 1 - Combined Motor Generator]

emg:ElectricMotorGener ator

trsm:Transmission

ice:InternalCombustionEngine

acl:accelerator

ecu:PowerControlUnit

ft:F uelT ankAssy

di f:Di fferential

rfw:Chass isSubsytem.FrontWheel

lfw:Chass is Subsytem.F rontWheel

Port:FuelTankFitting

Port:ICEFuelFi tting

fuelDelivery

torqueOut:T orque

torquein:T orque

spline

fuelSupply:Fuel

epc:ElectricalPowerController

bp:BatteryPack

i1:ElectricCurrent

i2 :ElectricCurrent

fp:F uelPump

fi:FuelInjector

fdistbp:BrakeSubsystem.BrakePedal

<>

<>

<><>

4

fuelReturn:F uel

<>

<>

<>

<>

g1:Torque

t2:T

orqu

e

t1:Torque

ice

ct rl

I_ICECmds

I_ICECmds

ctrl

ctrl

I_I CEData I_ICEData

trsmepc

c3

c2

c1

I_IEPCCmdI_IEPCData

I_IEPCDataI_EPCCmd

I_TRSMData

I_T RSMCmd

I_TRSMCmd

I_T RSMData

<>

<>

<>

rightHalfShaft

<>

<><>

leftHal fShaft

UML4MBTMODEL

SysML4MBTMODEL

UML MODEL(RSM, Topcased, …)

TestTestCasesCases

AdaptationAdaptationLayerLayer

+

SysML MODEL(Topcased)

Adaptation Layer

Generated test cases• Have to be concretized• Messages : discrete nature

Test bed inputs• Have to be mapped to reality• I/O Signals: continuous nature (sampled)

Automatic matching• Generate variables• Generate signals

3014/01/11

Adaptation Layer

Discrete messagesgenerate continuous signals• Example: turnLeft(moderately) message

changes the signal to -90° to be senton the SteeringWheelAngle variable

• Curves are smoothed

3114/01/11

turnLeft(moderately)SteeringWheelAngle

Testing systems in 4 steps

Create specification model in SysML• States discrete system behavior

(Could also state environment behavior)

Generate tests from model• Take the shape of sampled signals

sent to / read from variables

Map variables to reality• CAN messages, input pins…• Select comparators

Run the tests

3214/01/11

Refining expected signals

Additional step:provide a continuous mathematical model• Generated test are run against the

mathematical model• Output signals are promoted

to expected signals

3314/01/11

3414/01/11

Plan

Test in system engineering

Model Based Testing

The VETESS tool chain

The front wiper case study

Outlook

3514/01/11

Plan

Test in system engineering

Model Based Testing

The VETESS tool chain

The front wiper case study

Outlook