This appendix identifies the software products that are to undergo software product evaluations, identifies the criteria to be used for each evaluation, and contains a default set of definitions for the evaluation criteria. This appendix is a mandatory part of the standard, subject to the following conditions:
1) these requirements may be tailored by the acquirer,
2) the developer may use alternate criteria or definitions if approved by the acquirer, and
3) if the development of a given software product has been tailored out of the standard, the requirement to evaluate that product does not apply.
This section is not applicable to this appendix.
Figure 6 identifies the software products that are to undergo software product evaluations and states the criteria to be applied to each one. Each software product and criterion is labelled for purposes of identification and tailoring. For convenience, they may be treated as subparagraphs of this paragraph (referring to the first criterion, for example, as D.3.1.a). The software products are expressed in lower case letters to convey generic products, not necessarily in the form of hard-copy documents. Evaluations of system-level products are to be interpreted as participation in these evaluations. Some of the criteria are subjective. Because of this, there is no requirement to prove that the criteria have been met; the requirement is to perform the evaluations using these criteria and to identify possible problems for discussion and resolution.
The following paragraphs provide definitions for the criteria in Figure 6 that may not be self-explanatory. The criteria are listed in alphabetical order, matching as closely as possible the wording used in Figure 6.
This criterion, applied to user/operator/programmer instructions and to the "as built" design and version descriptions, means that the instructions or descriptions are correct depictions of the software or other item described.
Test cases are adequate if they cover all applicable requirements or design decisions and specify the inputs to be used, the expected results, and the criteria to be used for evaluating those results. Test procedures are adequate if they specify the steps to be followed in carrying out each test case. Test data are adequate if they enable the execution of the planned test cases and test procedures. Test or dry run results are adequate if they describe the results of all test cases and show that all criteria have been met, possibly after revision and retesting.
This criterion means that:
(1) no statement or representation in one software product contradicts a statement or representation in the other software products,
(2) a given term, acronym, or abbreviation means the same thing in all of the software products, and
(3) a given item or concept is referred to by the same name or description in all of the software products.
This criterion uses the DIDs to specify the required content of software products, regardless of whether a deliverable document has been ordered. Allowances are to be made for the applicability of each DID topic. The formatting specified in the DID (required paragraphing and numbering) are not relevant to this evaluation.
A software product "covers" a given set of items if every item in the set has been dealt with in the software product. For example, a plan covers the SOW if every provision in the SOW is dealt with in the plan; a design covers a set of requirements if every requirement has been dealt with in the design; a test plan covers a set of requirements if every requirement is the subject of one or more tests. "Covers" corresponds to the downward traceability (for example, from requirements to design) in the requirement, design, and test planning/description DIDs.
This criterion means that, in the knowledge and experience of the evaluator, a given concept, set of requirements, design, test, etc. violates no known principles or lessons learned that would render it impossible to carry out.
This criterion means that the software product shows evidence of having been developed in accordance with the approach described in the software development plan. Examples include following design and coding standards described in the plan. For the software development plan itself, this criterion applies to updates to the initial plan.
This criterion means that:
(1) no two statements or representations in a software product contradict one another,
(2) a given term, acronym, or abbreviation means the same thing throughout the software product, and
(3) a given item or concept is referred to by the same name or description throughout the software product.
This criterion applies if the software product being evaluated is specified in the CDRL and has been formatted for delivery at the time of evaluation. It focuses on the format, markings, and other provisions specified in the CDRL, rather than on content, covered by other criteria.
This criterion means that the software product fulfills any Statement of Work provisions regarding it. For example, the Statement of Work may place constraints on the operational concept or the design.
This criterion means that, based on the knowledge and experience of the evaluator, a given plan represents a reasonable way to carry out the required activities.
This criterion means that recorded test results show that the item under test either passed all tests the first time or was revised and retested until the tests were passed.
A requirement or set of requirements is considered to be testable if an objective and feasible test can be designed to determine whether each requirement has been met.
This criterion means "understandable by the intended audience." For example, software products intended for programmer-to-programmer communication need not be understandable by non-programmers. A product that correctly identifies its audience (based on information in Block 3 of the corresponding DID) and is considered understandable to that audience meets this criterion.
Translator: Simon Wright simon@pogner.demon.co.uk
Last updated: 22.iii.99