January 16, 2012

Software quality - PA-7

Determine the rules for ensuringthe sufficient quality of software

A. Scope

The purpose of this section is to present simplified software quality requirements used to obtain a minimum Software Quality level for application software developed for CNES by industrial suppliers, by scientific laboratories, or internally by CNES, for scientific projects. The quality requirements given in "RNC‑ECSS‑Q‑ST-80 Software product assurance " can be adapted with respect to the acceptable risk level.

It is used:

  • as soon as phase A is initiated, to define the development context for future developers;
  • as a basis for the discussions with the software producer as soon as development starts (end of phase B - beginning of phase C), in order to finalise the practical measures. These measures shall be defined during the working meetings between CNES and its suppliers; they are formally established in the supplier's proposal, and may be reproduced in an application plan.

 

B. Fundamentals

1. Method

Any software development must be conducted according to pre-established rules, in response to CNES requirements.

The response to these requirements is written in an application plan (for which a standard outline is provided in appendix: Application Plan), which expresses the supplier's undertaking to take into account these requirements.

The supplier must specify the organisation and the responsibilities of the team in charge of development. In particular he must identify the Software Product Assurance Manager, the Product Assurance measures that are set up (including at least the configuration management and the management of the technical facts), and also the planning of the corresponding activities.

2. Development cycle

An analysis of needs shall be completed before any software development.

After the analysis of needs phase, the software development cycle shall include the following phases:

  • specifications,
  • design,
  • coding - unit tests,
  • validation,
  • acceptance.

 

In the case of software integrated into a piece of equipment, the software participates as a constituent to the equipment-software integration phase, and its development cycle must be consistent with that of the equipment.

2.1 Software Specification Phase

The specifications of the software are used to:

  • identify the needs expressed by the customer and then express them in terms of functions, interfaces with the outside and with respect to each other;
  • define the function sequencing;
  • state the constraints (performance, priorities, memory usage);
  • analyse the software that could be reused according to the needs to be covered, and assess the impacts of its reuse on the development (refer to paragraph 3 " reused software").

 

These elements shall be described in the software specification document (for which a standard outline is provided in appendix: Specification Document).

This phase is also used to:

  • prepare an initial version of the validation document;
  • define an initial version of a user manual;
  • prepare a mock-up of the man / machine interface, if necessary.

 

This phase ends with a Software Specification Review (RSL) to decide on the documents that will result from this phase and on the possible reuse of the software.

This technical meeting must take place with the participation of the scientists and/or users.

2.2 Design phase

The design activity consists in:

  • defining the structural breakdown of the application into software components, and then describing each one in detail;
  • defining the data flow and the interfaces;
  • estimating the resources;
  • defining the man / machine interfaces.

 

The design of the software highlights in different design elements the functions that we wish to reuse in order to be able to validate them independently.

2.3 Coding phase - Unit tests

The activities to be completed during this phase are:

  • the coding of the component parts,
  • the execution of the unit tests of the component parts and of the integration tests between components.

 

The coding rules to be used are described in the Methods section.

Wherever possible:

  • a high-level language shall be chosen rather than an assembly language,
  • the entire project shall be constructed using the same high-level language.

 

2.4 Validation phase

Validation is used to check that the software fulfils its specified functions.

The validation tests shall cover all the requirements of the software specification.

In the case of software integrated into a piece of equipment, the validation must include tests on the identification model of the corresponding equipment.

These tests shall be carried out on stable software, whose version is configuration-managed. Any modification to the current configuration during the validation phase shall be traced.

Validation of the software shall be carried out on the laboratory model.

During this phase, the validation document shall be finalised (a standard outline is provided in appendix: Validation document) and completed by the test reports and the user manual (a standard outline is provided in appendix: User manual), if necessary.

The validation phase shall be preceded by a Test Readiness Review (TRR) (see PA‑15) and closed-out by a Test Review meeting (see PA-16).

The purpose of the Technical Readiness Review (TRR), at the start of the software validation phase, is to authorise the initiation of the validation tests.

It allows:

  • The checking of the action status,
  • The checking of the completion and consistency of the test plan and procedures,
  • The identification of the configuration status and the freezing of the corresponding software release.

 

At the end of this Technical Readiness Review board, if the test start is authorised, all the tests planned are performed.

During the tests, the result of each test is recorded in a logbook, along with the non-conformances and modifications which have been identified.

As far as possible, the corrective or evolutionary actions associated with the non-conformances and to the modifications identified are only carried out at the end of the validation phase, and will result in a new release of the software.

If this is not possible, any change in configuration in the course of the tests is recorded in the logbook.

The Test Review shall take place at the end of the validation phase. It aims at:

  • Presenting a summary of the test results,
  • Reviewing the status of the non-conformance and residual modifications,
  • Identifying actions to be led, specifying the closing date and the person(s) responsible for the ones that are still open.
  • Deciding the acceptance or rejection of the end of the validation phase.

 

In case of rejection or acceptance with reservation(s), the actions to be performed and their deadlines will be explicitly mentioned in the report.

2.5 Acceptance

If there is a software acceptance, it must meet the requirements defined above for the software validation phase.

The acceptance is based on the acceptance test plan taken from the software validation document (a standard outline is provided in appendix: Acceptance log). It takes place after the validation phase has been accepted.

It shall end with aTest Review meeting during which acceptance of the developed software will be pronounced(or not).

C.Configuration management

Management of modifications

The software configuration management corresponds to all the manual or automatic activities used to identify, at any time, the elements created, used or modified by the software development process and their relations.

Its main purpose is to store each reference version of an element in order to know precisely and at all times which version is being used, so as to be able to reconstitute any earlier version, if need be.

1.Principles

The software configuration management must identify each element to be managed, in a unique way according to the project nomenclature rules.

A project team member is appointed to manage the software configuration.

His task is to:

  • define the elements to be managed;
  • define the reference version;
  • be aware of the relations between the elements of a given configuration (in order, for example, to be able to answer the question: "what are the validation tests that correspond to this version of the application software?");
  • define the configuration archiving times for a configuration element;
  • define and implement the product delivery methods,
  • produce the configuration document.

 

For each software release, configuration management must make it possible to:

  • identify the documents with their reference and issue/revision,
  • save on an identified medium with an indelible marking label specifying the name of the application software and its release (version/revision):
    • the product (sources, data, generation procedures),
    • the sets of validation tests,
    • the associated specific test means, if any (simulator, data.),
  • know the references:
    • of the back-up medium (tape No., for instance),
    • of the technical facts (non-conformances, corrections, change requests),
    • of the basic software (operating system, emulators, real-time monitors, libraries, compilers, link editors.),
    • of the hardware,
    • of the test means, if any
  • restore an archived software, if necessary

 

1.1.Configuration description file

The supplier shall, for each version, establish a Configuration Description File (CDF), to be included with the delivery of the version of the product.

This file contains at least the following files:

  • ddc.txt   item identification
  • fa-couv.txt   list of covered NCR's
  • dm-couv.txt   list of covered change requests
  • etat-fa.txt   list of open NCR's
  • ls-doc.txt   list of documents (CNES and supplier) associated with the product
  • ls-ref.txt   list of files making up the product

 

D. Methods, tools and rules

1.Principles

The methods and supporting tools (if any) adapted to the various phases of the software life cycle are selected according to the skills of the project team (training plan, past experience.) and to its financial resources.

The use of open source software is recommended (in accordance with the terms of the licence).

The choice of tools shall be justified in the application plan or equivalent. Wherever possible, standard tools shall be used in a consistent manner throughout the development.

Naming rules shall be defined, particularly for interface variables. These rules shall be consistent with respect to the usage conventions concerning the coding language used.

2.Coding rules

The interface variables shall be described in the form of comments in the code.

Each code module shall comply with the following rules:

  • It shall include a comment header containing at least the following headings:
    • project name;
    • module name;
    • release;
    • author:
    • date;
    • role and purpose of the module;
    • the files used;
    • the modules used;
    • the modules that use it;
    • the references of any modifications;
  • Each function or subroutine must specify its input, output or input/output parameters, with, for each one:
    • its name,
    • its meaning,
    • the units used, in the case of a physical quantity,
    • its precision,
    • its value domain,
  • Each conditional branch order or sequence interruption order shall be commented, explaining the purpose of thedisconnection;
  • In the main program, only write the control flow of the software;
  • Only one entry point and only one exit point for any compilable unit;
  • Every part of the code shall be accessible (no dead code).

 

E.Specific development conditions

1.Dependability of critical software

A functional analysis at system level shall be used to identify critical software modules, taking the interaction between the software and its environment into account.

The supplier must endeavour to design the critical software components as simply as possible in order to facilitate the software dependability and safety analysis and software testing.

2.Reused software

In cases where the use of existing software (market products, freeware, other products) is considered, the following elements must be taken into account for their selection:

  • the assessment of the product with respect to the needs;
  • the acceptance and warranty conditions (demonstration of correct operation);
  • the installation conditions;
  • the identification and integration by the configuration management;
  • the maintenance conditions;
  • any industrial property constraints (utilisation or modification rights).

 

For software to be adapted, it is essential, prior to any modification:

  • to assess the proportion of modifications;
  • to check the status of the documentation of existing software, in order to ensure its coherence and completeness with respect to the reused software;
  • to set up a configuration management system for this software;

 

At the end of the specification phase, the list of software that is being considered for reuse shall therefore be presented, giving details of the status of each software entity and the associated development effort as a function of the justification elements listed above.

The analysis of the modifications to be made to the reused software shall, for each software entity, identify the phase in the development cycle where the process for taking it into account is initiated.

For example:

  • the addition of a feature or a change of external interface results in a return to the software specification phase, followed by a complete "mini" development cycle in order to integrate and validate this feature;
  • a change of interface which is internal to the software, or a change of algorithm, results in a return to the design.

 

Do not forget, when assessing the consequences of a change, to include the load induced by the updating of the documentation associated with the phases to be modified.

Based on its impact, each modification also requires an analysis in order to identify the tests that are required for its validation (unit test of the modified component, functional test involving the modified component, non-regression tests to validate the entire software entity).

In the different design elements, the design of the software being developed shall establish the functions that are to be reused so that they can be validated independently.

In order to facilitate the maintenance of the software, any change to the code shall be carried out in such a way as to retain the framework of the component. If this rule cannot be followed, opt for a full reprogramming of the constituent in accordance with structured programming principles.

3.Man / machine interface

For software that has a man / machine interface, a mock-up of this interface shall be established during the specification phase.

This mock-up shall present:

  • the content of the screens,
  • the dynamics of the screen sequencing,
  • the implementation of the various commands,
  • any on-line help functions.

 

The future users shall take part in the validation of the mock-up of the man / machine interface, which shall take place during the specification phase.

The users shall take part in the software acceptance phase.

F. Documentation

The following table presents the documents to be provided during the formal meetings recommended in the case of a classic V-shaped life cycle diagram. The document delivery times will need to be modified in the case of a different type of development cycle.

Launch meeting

Software Specification Review (RSL)

End of design

Pre-validation Technical readiness review board (TRR)

Post-validation Test Review

Accept­ance

At each version

At each progress meeting

Application plan

D

F

Software Specification document

F

Design document

F

Validation document

D

C

F

Configuration description file

C

U

U

U

User manual

D

C

F

Acceptance log

D

F

Schedule

D

U

Legend:

D draft version
C complete version
F final version
U updated version

Details of the content of these documents (including standard outlines) are given it the appendices.

G. APPENDIX Standard outlines

1. Application plan

1.2 Standard outline of the application plan

This describes the management measures that are applied to the project in response to the requirements of this document. These management measures shall be defined during work meetings between the customer and the supplier.

1.2 Standard outline of the application plan Download

2. Specification document

2.1 Presentation

It describes:

  • the features of the software,
  • the manner in which the software is to be operated,
  • the external interfaces,
  • the data handled,
  • the data flow,
  • the enabling conditions, if any,
  • the data recovery or backup points, if any, to be provided for in case of contingency,
  • the constraints (resources, etc.),
  • the conditions of reuse of third-party products (COTS, open source software, etc.).

 

2.2 Specification document standard outline Download

3. Design document

3.1 Presentation

This document describes the solution retained in order to meet the software specifications:

  • the architecture of the software, with its breakdown into components (function calling graphs, data flow),
  • synchronisation between components, where applicable,
  • the strategy for processing errors and exceptions.

 

The description of each component appears as comments in the code. It includes:

  • its role, from a functional point of view,
  • its main interfaces (files, parameters, messages, etc.),
  • its "links" with other components (function call links, dependence, service, inherited, generating, etc.),
  • its activation conditions, where applicable,
  • its pseudo-code.

 

3.2 Design document standard outline Download

4. Validation document

4.1 Presentation

At the end of the software specification phase, a first version of this document describes:

  • the organisation, scheduling and sequencing of the tests,
  • the necessary resources, particularly the test facilities, stating the limits of their representativeness with respect to the real environment:

 

  1. limits of physical domains,
  2. limits in terms of performance,
  3. limits in terms of features,
  4. limits in terms of commandability and observability,

 

  • the validation plan (description of the software components to be tested from a functional point of view, list of tests in terms of objectives and required facilities).

 

Before the beginning of the software validation phase, this document is completed in order to describe the test procedures for each validation test. These procedures include the description of the implementation of the tests, of the actions to be performed before, during or after the test, of the facilities to be set up, and of the expected results.

At the end of the software validation phase, the final document must be completed by test reports which shall describe the results obtained with the references of any anomalies detected.

4.2 Validation document standard outline Download

5. Configuration description file

Example of a Configuration Description File (DDC) (text file) Download

6. User manual

6.1 Presentation

The supply of this document depends on the type of software. It is necessary for software with a man-machine interface. This document describes:

  • the different types of possible user,
  • the means of interaction (keyboard, mouse, function keys, scroll bars, buttons, etc.), their meanings and the principles of use,
  • the menus and screen grids: the types and meanings of the various fields, the checks carried out and the associated actions,
  • the messages issued by the software with an indication of the issuing function and the associated corrective actions,
  • the screen sequencing,
  • the formats / types of hard copies /print-outs.

 

The organisation and the content of this document are to be submitted to the future users for their approval.

This document is initialised right from the software specification phase, and is completed during the development for a complete version to be delivered at the end of the validation of the software.

6.2 Typical outline of the user manual Download

7. Acceptance test plan

7.1 Presentation

The acceptance tests described in this document are an excerpt from the list of tests of the software validation document. The acceptance test plan shall be approved by CNES before the beginning of the acceptance process.

7.2 Acceptance log standard outline Download

Risks

RisksLack of knowledge of software changes
Incomplete validation of software
 RecommandationsESTABLISH APPROPRIATE "SOFTWARE QUALITY" RULES
Software Quality Provisions

 

Activities / documentation

Published in: