3.4-3.6 DO-178B and DO-254

ÉCOLE DE TECHNOLOGIE SUPÉRIEURE

1

MGA-855 Certification des systèmes embarqués

d’aéronefs Maîtrise en génie : Concentration en génie aérospatial

It’s Only Software (and Hardware)!


2

MGA-855 Certification des systèmes embarqués

d’aéronefs Maîtrise en génie : Concentration en génie aérospatial

Chapitres 3.4, 3.5,3.6 RTCA/DO-178B V&V, Traceability, Testing,

Structural Coverage, Tool Qualification and PDS, RTCA/DO-254 Design Assurance Level and AC20-152

présenté par :

Maxence Vandevivere [email protected]

Professeur responsable : René Jr. Landry Poste : 8506 Porte : 2950

Email : [email protected] Site web : www.etsmtl.ca/rlandry

mailto:[email protected]

mailto:[email protected]

http://www.etsmtl.ca/rlandry


3

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254

Certification des systèmes embarqués d’aéronefs MGA-855: Chapitre 3

This module is designed to show the application of the certification principles contained in the earlier chapters. As such, it touches upon many aspects of the certification process, but the material is not complete, comprehensive, or necessarily current with the latest regulations and guidance materials. For these reasons, the analysis contained in the following slides should not be used as the basis of a certification program for the system under investigation.


4

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


Overview

3.4 to 3.6 RTCA/DO-178B Wrap-up and DO-254 • 3.4.1 Verification & Validation • 3.4.2 Traceability • 3.4.3 Verification and Testing Methods • 3.4.4 Structural coverage • 3.4.5 Testing • 3.4.6 Conformity Review • 3.4.7 Delivery • 3.5.1 Tools • 3.5.2 Additional Considerations (tool qualification, PDS, alternative methods) • 3.6.1 DO-254 • 3.6.2 DO-254 Design Assurance Level • 3.6.3 AC20-152


5

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.4.1 – V&V

• Recap: • Verification means ensuring that what was done in a

process was correct (meetings requirements, matches process, etc.)

• Validation means ensuring that the assumptions, requirements are actually correct and complete

• Verification takes place throughout the entire life cycle • Validation takes place at the requirements level


6

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.4.1 – V&V (cont’d)

• Requirements are the foundation we build on


7

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.4.1 – V&V (cont’d)

• What happens when a requirement changes, or becomes invalid in some way?


8

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.4.1 – V&V (cont’d)

• Requirement validation is a critical process • If the requirement is wrong, there is a ripple effect that

could extend through the entire project • If a requirement changes at any level we need to:

– Update all connected requirements (up & down stream) – Update all data items connected – Verify changes


9

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.4.1 – V&V (cont’d)

• Question: How do we know what data (requirements, code, documentation, etc.) needs to change or be updated when something in our project changes?

• Answer: Traceability


10

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.4.2 – Traceability

• Traceability is: “The evidence of an association between items, such as between process outputs, between an output and its originating process, or between a requirement and its implementation.” (DO-178B, glossary)

• DO-178B: – does not prescribe any particular method of showing traceability – Does require that traceability can be demonstrated in some way


11

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


• Traceability also helps us avoid this situation:

3.4.2 – Traceability (cont’d)


12

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254



• What needs to be traceable? • Strictly speaking, DO-178B states we need to show

traceability for: – System requirements to software requirements – High-level to low-level software requirements – Low-level software requirements to code – Evidence of process outputs tracing to the process activity and its inputs


13

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254



• Generalizing, the big items that must be traceable are: – Requirements – Design – Code – Test & test results

• However, breaking it down, really everything traces:

– Reports need to trace to the process action it satisfies. Reports also need to show/trace to the artifacts that are being reported on

– Audits, reviews, checklists need to uniquely identify what was reviewed


14

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254



System requirements (System level documentation and SRD)

Software high-level requirements (SRD)

Software low-level requirements (SDD)

Code

Test

Test results


15

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254



• What does it mean if something doesn’t trace? – For code? – For requirements?


16

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.4.2 – Traceability (cont’d) Derived Requirements

• Derived requirement – Definition: “Additional requirements resulting from the software development process, which may not be directly traceable to higher level requirements.” (DO-178B glossary)

• Example of a derived requirement: – In a graphics library, that has high level requirements for drawing

primitives, in an organized way, one example might be: – “The graphics library shall externalize line styles that may be used to

render primitives.” – There is no top-level SRD requirement for a line style external definition.

However the software design may do this for flexibility, future expandability, or other benefits


17

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254



• If a requirement doesn’t trace down to a child (either another requirement, or code, or test) our traceability is broken and needs to be fixed

• If a requirement does not trace upward, we must check if it is a derived requirement

– NB: Of course, all requirements will follow the rules defined in the SRS which will show us how to label derived requirements differently than other requirements


18

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254



• If code doesn’t trace up to a requirement, we need to: – add a requirement, or – add design to support it, – or remove the code

• If code doesn’t trace down to test, we need to: – Analyze why it wasn’t tested in the first place – did we miss a

requirement in our testing? And/Or, – Add more test cases to test the code more completely


19

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.4.3 – Verification and Testing Methods

• Question: How do we ensure our requirements are implemented in our code/software?

• Answer: Testing – specifically, requirements-based testing


20

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.4.3 – Verification and Testing Methods (cont’d)

• Types of testing (from DO-178B):

– Hardware/software integration testing: To verify the correct operation of the software on the target computer environment

– Software integration testing: To verify the interrelationships between software requirements and components and to verify the implementation of the software requirements and software components within the software architecture

– Low-level testing: To verify the implementation of software low-level requirements

• Formal methods: discrete math, proofs, logic grammars checked by computers, can also provide exhaustive analysis to compliment testing

• Exhaustive input testing


21

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254



• How? – Base our test cases on the requirements – Requirements coverage analysis to verify all requirements tested – Structural coverage analysis to verify all code was tested/exercised


22

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254



• Two main types of test cases are used:

• Normal cases: Tests normal operations, or ranges of inputs

• Robustness cases: Tests that the code responds correctly to:

– Invalid input – Abnormal conditions – Failure modes – Timing/scheduling issues – Etc.


23

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254



• Robustness?


24

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254



• Data coupling: The dependence of a software component on data not exclusively under the control of that software component

• Control coupling: The manner or degree by which one software component influences the execution of another software component


25

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.4.4 – Structural Coverage

• Question: How do we know if our code represents our requirements?

• Answer: Traceability, requirements-based testing, and Structural coverage


26

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.4.4 – Structural Coverage (cont’d)

• Definition: structural coverage is an analysis of what code is executed by the requirements-based testing procedures

• Typically done during testing • Code is instrumented, and a tool reports all lines in the

code executed during an execution of the software • All lines not executed then need to be analyzed • Should confirm data and control coupling (note: this

should have been identified during design, code review, etc.)


27

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254



• The level of structural coverage detail/granularity depends on the software criticality:

– Level A: » Byte code – each part of the binary code created must be justified

and traceable to a requirement » Level A also requires MC/DC (not AC/DC) – modified condition /

decision coverage » MC/DC means: Every condition in a decision in the software has

taken all possible outcomes at least once, and the condition has been shown to independently affect the decision outcome

– All other levels (if required) : Statement coverage – or put another way – traceability to a line of code


28

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254



• Possible reasons for finding code that is not executed: – Tests were not complete – need to add more tests – Requirements were not complete – need to add more requirements and

tests – Code that is not used and never will be – “dead code”

» Does not link to requirements » Is not reachable (cannot be executed or used) in operational

configuration – Code that is not used in normal circumstances – “deactivated code”

» Like a maintenance routine, or data-loader routines used only on the ground


29

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.4.5 – Testing

• Before we start testing… what do we need to do?


30

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.4.5 – Test for score

• Definition: “test for score” : testing for certification credit • All “test for score” must be done on a representative

system or it cannot be used for credit • Question: What do we need to do before we start

testing?


31

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.4.5 – Test for score (cont’d)

• Before we start testing, we need to complete a TRR – a test readiness review

• What is a TRR? – A verification of our transition criteria to advance into the testing phase – Makes sure we have completed all items before we begin testing – If something is not complete, we are testing a moving target


32

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254



• What needs to be complete to begin testing? – All plans reviewed, audited, and complete – All requirements and design, reviewed, audited, and complete – All test procedures, test cases reviewed, audited, and complete – All traceability reviewed, audited and complete – A baseline of the software and all life-cycle data created – Is the test environment ready for testing? (calibration, configuration as

per plans, etc.) – All CRs, PRs should be closed (or at least open items contain mitigation

or have been reviewed to ensure no impact on testing)


33

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254



• What will we test? – A controlled baseline

• Where will it be tested? – in a representative target environment


34

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.4.6 – Conformity Review

• Question: What do we do when we have finished testing? • Answer: Conformity review!


35

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.4.6 – Conformity Review (cont’d)

• Definition: Conformity review: A QA (Quality Assurance) process that reviews everything we did, that we did it, and that everything has been completed as per the plans

• In general, a conformity review, ensures that: – All plans, process activities for certification credit have been completed – That all life-cycle data has been retained, and under CM – That all life-cycle data complies with the plans & standards – Traceability has been completed (especially up to the system and safety

requirements) – All problem reports have been closed, or have been dealt with according

to the CMP – Any deviations are recorded and approved – All executable object code can be re-created from what is stored under

CM using the instructions created


36

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.4.6 – Conformity Review (cont’d)

• Conformity review cont’d: – If applicable, the software can be loaded using the instructions created – Any problem reports or change requests deferred from previous

releases have been re-evaluated and reviewed to incorporate into the software or determine necessary status


37

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.4.7 – Delivery

• Now what? – Deliver to the regulatory authority concerned (TCCA or FAA) – part of

your certification liasion process – Deliver to your customer (if any) – Take a nap


38

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.5.1 – Tools

• What tools are available to help achieve our goals? • For documentation, tracking, etc:

» Excel for tracking » Word for documents » Filemaker or Access for databases » cvs, svn, git, for CM

• Some more integrated examples: – IBM Rational toolset – Rose, ClearCase, ClearQuest, etc. – MKS Integrity & Source – Polarion


39

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.5.1 – Tools (cont’d)

• Tools to help development – other than compilers, linkers, etc.

• Static and dynamic analysis tools (a few examples): – OpenSource: splint, lint, cpplint, cppcheck – PClint (Gimpel software) – Klockwork

• Code complexity: – SourceMonitor – Klockwork

• Code coverage: – Bcov, gcc/g++, – lots of compiler-specific tools


40

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.5.2 – Additional Considerations

• Tool qualification • Previously developed software (PDS) • Alternative methods

– Exhaustive testing – Multi-version dissimilar software – Product service history – equivalent level of safety


41

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.5.2 – Additional Considerations : Tool Qualification

• What if we want to use a tool to automate part of the certification process?

• We can. • DO-178B classifies the use of a tool in two ways:

– Tools we use with a human-in-the-loop – the output is verified – Purely automated, no output verification

• For the former, we need to make sure the tool is fit for the purpose


42

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.5.2 – Additional Considerations : Tool Qualification (cont’d)

• Definition: “Tool qualification” is needed when: “a processes of [DO-178B] are eliminated, reduced or automated by the use of a software tool without its output being verified as [part of the software verification process].” (section 12.2 of DO-178B)

• Tools are put into one of the following categories: – Software development tools – a tool that can introduce errors, e.g.: a

tool that automatically generates code for a developer – Software verification tools – a tool that cannot introduce errors but may

fail to detect them (analysis tools, etc.)

• Any tool that will be used needs to be controlled (CM)


43

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254



• Tool qualification for software development tools: – Typically (unless you can convince the cert authorities otherwise) you

need to certify the tool to the same criticality level as the software you’re using it with. e.g.: A certified code compiler for a Level B project would need to be Level B certified

– Need to show compliance with tool operational requirements – In addition, you need to show:

» that the operational requirements are “good” – i.e.: correct, consistent, complete – and that the requirements are all covered via analysis

» demonstrate operation in normal and robust testing conditions » Structural coverage analysis » Analysis of the potential errors produced by the tool

– Software verification tools – a tool that cannot introduce errors but may fail to detect them (analysis tools, etc.)


44

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254



• Tool qual for software verification tools: – Show that the tool complies with its operational requirements

• Quick tip: if you need a certified development tool, see if one exists on the market for purchase


45

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.5.2 – Additional Considerations: PDS

• What if we want to use pre-existing software in a new certification project?

– Code that was previously used on another certification program – Usually applies to certified code

• There are complexities that are specifically discussed in DO-178B in doing this

• For instance: – Modifying existing code for a new project (adding requirements,

features) – Changing the development environment (porting the code) – Changing the target environment (aircraft, avionics, etc.)


46

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.5.2 – Additional Considerations: PDS

• Some items and risks to be aware of: – SSA should be revised to include new

environment/target/modifications/changes – Impact analysis of new/modified requirements, design, code, etc.

(especially on data & control coupling) – Re-verification of modified areas – including areas related by data and

control – New target environment considerations need to include target processor,

memory, other hardware and software used, data busses, etc – If a different compiler is used – are different options used? The code

output will probably be different too, so need to test all over again – For COTS software, need to reverse engineer supporting data life-cycle

data – Need to show traceability from PDS to new baseline, and support

tracking changes for multiple versions of the software


47

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.5.2 – Additional Considerations: Alternative methods

• Alternative methods – can be used as alternative ways of satisfying objectives of DO-178B

• Methods outlined in DO-178B is not an exhaustive list – other methods do, and may exist in the future

• Must get early buy-in and acceptance of alternative method from certification authorities

– Exhaustive testing – Multi-version dissimilar software – Product service history – equivalent level of safety


48

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.5.2 – Additional Considerations: Alternative methods (cont’d)

• Examples of alternative methods: – Exhaustive testing – testing using logic flow, discrete math, etc. to

improve the specification and verification of the software – Multi-version dissimilar software – system design technique that uses

two or more parts of software to provide the same functionality to attempt to avoid common errors in the components. Techniques are usually a combination of some of the following:

» Using different languages to write the components » Use different compilers to generate binaries with same code » Using different processors to run the same code » Software requirements, design, code created by two separate

teams (note, system and functional requirements are identical!) Can also use different code and design standards

Note: Testing approach must take dissimilar s/w into account – Product service history – equivalent level of safety


49

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.5.2 – Additional Considerations: Alternative methods (cont’d)

• Examples of alternative methods, cont’d: – Product service history – some certification credit may be granted by

showing an equivalent level of safety with service history evidence. – This approach depends on demonstrating quality of processes used to

manage the software and attributes like: » Effectiveness of change tracking, problem reporting » Control/configuration of the software » Stability and maturity of the software » What environment the software is used in (aerospace, auto industry,

gaming?) » Error rates from service history » Impact of modifications to the software


50

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.6.1 – DO-254

• What is DO-254? – Hardware certification guideline – Also known as EUROCAE ED-80 – Released in 2000 (was not really in use – required – by the FAA until

about 2005) – Provides guidance for the design assurance of complex electronic

hardware (CEH) for airborne use in aircraft equipment and systems. – Structure of the document is based on DO-178B


51

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.6.1 – DO-254 (cont’d)

• What is Complex Electronic Hardware (CEH)? – DO-254 defines a piece of hardware as simple if: “a comprehensive

combination of deterministic tests and analyses appropriate to the design assurance level can ensure correct functional performance under all foreseeable operating conditions with no anomalous behavior.” (DO-254, section 1.6)

– Or put another way, hardware is classified as simple if you can fully test it

– If the hardware is not simple, it’s complex

Simple, right? Not quite: this distinction is still subject to heated debates on a fairly regular basis


52

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.6.1 – DO-254 (cont’d)

• A simple example: – If we have a 4-bit controller with discrete I/O (two input, two output) – This could easily be exhaustively (completely) tested

• Another example: – A 16-bit controller with discrete I/O (8 input, 8 output) – Again, we can exhaustively test all inputs and outputs

• Not so simple: – A 1000 gate FPGA with 100 pins – We can no longer easily (and deterministically) exhaustively test all

inputs and outputs


53

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.6.1 – DO-254 (cont’d)

• Document structure is basically the same DO-178B: – Need to define integral processes: Verification, CM, QA, cert. liaison. – We have a life-cycle – but it is a Hardware design life cycle instead of

software – Planning process – Hardware design process

» requirements capture, » high-level & low-level design » implementation, » production, and » testing).

– Design assurance level instead of software criticality level – Like DO-178B, there is no prescribed process you must follow


54

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.6.1 – DO-254 (cont’d)

• One difference from DO-178B is production transition: – Basically covers the transition from development to production. – Intended to ensure:

» Actual physical production of hardware has been planned » Process to procure and handle hardware is in place » Safety considerations are known and documented


55

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.6.2 – DO-254 – Design Assurance Level

Design Assurance Level

Failure Conditions Probability

Level E No Effect < 1 x10-5 Level D Minor > 1 x10-5 Level C Major 1 x 10-5 < 1 x 10-7 Level B Hazardous/Severe-Major 1 x 10-9 < 1 x 10-7 Level A Catastrophic < 1 x 10-9

* If level is Level D or below, you do not need to use DO-254

Note: DAL uses the same scale as software criticality


56

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.6.1 – DO-254 (cont’d)

• FFP analysis– functional failure path analysis – Appendix B of DO-254

– Iterative analysis to identify which components have a critical level (Level A or B DAL)

– Similar to a fault tree analysis – Only applies to critical (Level A or B) components – Used when components of different levels are connected together – Intent is to identify critical components so that they can be assigned an

appropriate DAL


57

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.6.1 – DO-254 (cont’d)

• For DO-254, need to deliver the following documents to cert authority:

– Plan for Hardware Aspects of Certification – PHAC – Hardware Verification Plan (HVP) – Top Level Drawing

» Contains the configuration information (like a configuration index with both hardware and software information)

– Hardware Accomplishment Summary (HAS)


58

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.6.1 – DO-254 (cont’d)

• COTS hardware – DO-254 is interested in their pedigree, and service history – Can use COTS chips as long as can show:

» Wide use (the wider, the better) » Manufacturer is known to have good engineering processes » At least some documentation (technical, specs, etc.) » Look for quality control, mil spec, etc.

– Don’t reinvent the wheel if you don’t need to – Examples:

» ARINC429 bus controller » MIL-STD 1553 bus controller


59

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.6.3 – AC20-152

• AC20-152 - a typical AC • Released in 2005 • A lengthy document – 2 pages long! • Purpose: Defines more specific scope and details about

the application of DO-254


60

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


3.6.3 – AC20-152 (cont’d)

• AC20-152 – highlights: – Typical scope is for “application specific integrated circuits (ASIC),

programmable logic devices (PLD), field programmable gate arrays (FPGA)” (1.a)

– “When designing level D devices, manufacturers may choose to use RTCA, Inc. document RTCA/DO-254, Design Assurance Guidance For Airborne Electronic Hardware, dated April 19, 2000, or continue to use their existing design assurance practices.” (1.b, emphasis added)

– “We don’t intend that you apply RTCA DO-254 to every type of electronic hardware” (section 2)

– “we don’t intend that you apply RTCA/DO-254 to COTS microprocessors. There are alternative methods or processes to ensure that COTS microprocessors perform their intended functions and meet airworthiness requirements. Coordinate your plans for alternative methods or processes with us early in the certification project.” (3.b, emphasis added)


61

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


Summary

• Traceability is a tool and evidence • Tools used with no human-in-the-loop must be qualified • Development tools must be “certified” & comply with ops

requirements • Verification tools must comply with ops requirements only • Before testing, must be ready (TRR) • Must test software in both normal and robustness cases • Test for score (credit) must take place:

– on target hardware, – with an identified baseline


62

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


2.3.6 Summary

• DO-254 has same basic structure as DO-178B • Don’t over-apply DO-254 • DO-254 targets complex hardware (PLDs, ASICs,

FPGAs) • COTS hardware can be used, but be careful to select

hardware with a history

• For both software and hardware: Contact cert authorities early to coordinate efforts (and decrease the amount of surprises)


63

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


Questions?


64

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


References

• RTCA/DO178B • RTCA/DO-254 • AC 20-152


65

3.4, 3.5, 3.6 DO-178B Wrap up & DO-254


Images References

• All images creative commons, unless otherwise noted.

3.4-3.6 DO-178B and DO-254

Documents

Transcript of 3.4-3.6 DO-178B and DO-254