Transcript Verifying a Design

What’s the Exit Strategy?
Verifying a Design
2005 MAPLD
102
Design Integrity Concepts
Agenda – Design Verification
•
•
•
•
•
•
•
Concepts and implications
The specification connection
Verification before test
Test thoroughness
Tiered verification
Concluding the process
Summary
2005 MAPLD
103
Design Integrity Concepts
Verification Concepts
• Verification – Confirmation, through the provision of
objective evidence, that the specified requirements have
been fulfilled (Q9000-2000)
• Confirmation – the act of ratifying or corroborating (AH)
2005 MAPLD
104
Design Integrity Concepts
Verification Implications
• The verification process is not intended to be a “craps
shoot”
– Verification is primarily intended to ratify correctness
– Verification is NOT primarily intended to reveal incorrectness
• The mindset is important!
– Doubt that a design will work expresses a lack of confidence in the
designer and the design process
– Waiting until “verification” to guarantee correctness is an excuse
for being sloppy
• We should expect our designs to work! Verification simply
puts the seal of confirmation on the expectation
2005 MAPLD
105
Design Integrity Concepts
Verification Implications (cont.)
• Verification addresses two processes
– Design
• Primarily a one-time, iterative process in which the developed
concept is proven sound
• Fundamental correctness can be proven analytically
– Inspection confirms logical correctness in all circumstances
» Peer reviews are a manual form of inspection
» Functional simulation is an automated form of inspection
» Inspections must be tailored to design
– Analysis verifies correctness in the presence of variability
» Reliability (WC, Derating, FMEA, etc.)
» Timing over environments and age
2005 MAPLD
106
Design Integrity Concepts
Verification Implications (cont.)
• Verification addresses two processes (cont.)
– Instantiation
• Particular instance must be sound
• Particular correctness must be demonstrated experimentally
– Inspection confirms that instructions are followed
» Correct components installed
» Correct configuration selected
» Correct processes performed
– Test and demonstration confirms that operation matches
expectations
» Functionally and parametrically
» Predicted by nominal analytical predictions
2005 MAPLD
107
Design Integrity Concepts
Verification Implications (cont.)
• Verification “after the fact” is too late
– Analysis and test are NOT equivalent
– Design analysis and inspection must proceed in lockstep with
design processes
– Implementation tests and inspections should simply confirm what
is known
• Design efforts fail because they occur in a vacuum
– Creativity takes primacy over correctness
– Functionality comes first with operability being “shoehorned” after
the fact
• The “monster” simulation syndrome
• Conclusion: Verify as you proceed
2005 MAPLD
108
Design Integrity Concepts
Verification Implications (cont.)
• Correctness is a matter of personal integrity
for the designer
– Verification is not simply externally imposed
task
– Verification is something that the designer must
want to do (a part of his/her ethos)
– Verification is confirmation that the designer is
“worth his/her salt”
2005 MAPLD
109
Design Integrity Concepts
The Specification Connection
• “Requirements” are confirmed during verification
– The designer does not have a free hand with respect to how a
design is confirmed
– Specification provides the constraint set under which verification is
prosecuted
– Therefore, verification must be defined prior to start of design
• Not simply in a general manner
• Specifically
– Since specification is a customer and vendor document
• Both customer and vendor must be involved in the verification
process
• “Trust me” is not a reasonable phrase when it comes to verification
2005 MAPLD
110
Design Integrity Concepts
The Specification Connection (cont.)
• Specification contains a complete [ideally] set of requirements for the
design
– Some requirements are very specific
• “All discrete outputs shall have a source impedance of 2 kOhms.
• An adequate test is fairly obvious
– Some requirements are not very specific
• “The unit shall provide 512 kbytes of SRAM”
• Note the implied requirement – The SRAM shall function correctly
• What is an adequate functional test for this requirement? (walking 1’s, 0’s
addressing, etc?)
– Some requirements lend themselves to quantitative analysis – “The unit
shall provide a .1° C accuracy at end of life”
– Some requirements lend themselves to simple inspection – “The unit shall
provide 4 analog output channels”
– When do we decide adequacy of the method of verification and the
individual verification cases? – Before we start designing
2005 MAPLD
111
Design Integrity Concepts
The Specification Connection (cont.)
• Therefore, verification planning must begin with
specification creation
– Each requirement must be assigned a method or
methods
– Each requirement must be assigned a test or analysis
case
• Must answer question – What is an adequate verification?
• Must establish answer to question – When are we done?
– Each requirement must be verified at one or more levels
[FPGA, board, box, sub-system, system]
– Customer must concur with decision
• Note that modern verification databases make this
relatively straightforward
2005 MAPLD
112
Design Integrity Concepts
The Specification Connection (cont.)
• Advantages to this type of verification planning
– More verifiable / verified designs
• Up-front focus on programmatically difficult verification can
be instituted
– Earlier analysis
– Simplification of overly complex circuit designs
• Guidance for lower-level verification efforts can be established
– Sub-module vs. module level
– Module vs. unit level
• Integral self-test provisions can be included in design
• Earlier simulation vector definition (FPGAs / logic)
2005 MAPLD
113
Design Integrity Concepts
The Specification Connection (cont.)
• Advantages (cont.)
– More robust test sets
• Early knowledge regarding end-item function communicated
• Capability of test set matched to need
– Precision (timing, voltage, current, etc.)
– Speed
– Test level (component, unit, system)
• Test flow design considerations included in test set design
– Knowing verification requirements early makes the
entire development process more efficient
2005 MAPLD
114
Design Integrity Concepts
The Specification Connection (cont.)
• Defining requirements and test cases at the same
time as the specification clarifies and simplifies
the FPGA verification process
– Allows early start to simulation vector design
– Creates early knowledge of additional functional
models (SRAM, peripherals, etc.) needed
– Clarifies decision regarding what can be functionally
simulated and what must be simulated with backannotation
– Defines levels at which simulations must be run
2005 MAPLD
115
Design Integrity Concepts
Verification Before Test
• It is a relatively bad thing to discover a design flaw during
test (better than nothing)
– Late in the development cycle
– Correction is more complicated / expensive
– Personal stress is higher
• Yet … a surprising lack of verification occurs early
• Types of pre-test verification
– Inspection - conformity evaluation by observation and judgment
accompanied, as appropriate by measurement or testing (ISO)
• Personal self-check
• Peer Review
• Basic functional verification (Signal audits, functional simulation,
etc.)
2005 MAPLD
116
Design Integrity Concepts
Verification Before Test (cont.)
• Types of pre-test verification (cont.)
– Analysis – Conformity evaluation of the predictions
produced by realistic models of the system
• Worst case analysis (voltage, current, frequency, time) using
models that incorporate real world degradation of function
• Derating analysis using models that reduce stress
• Other
• Note that pre-test verification is fundamentally
conceptual
– Not tool dependent
– Not design dependent
– Can be accomplished many ways
2005 MAPLD
117
Design Integrity Concepts
Verification Before Test (cont.)
• Two approaches to pre-test verification
• Verify full design at once (one big peer review, code walkthrough,
analysis, simulation)
– Benefits
•
•
•
•
Complete design reduces need to develop assumptions
When proved correct, does not need to be repeated
When design is solid, less work
Reduce final documentation for verification
– Drawbacks
• Allows small design flaws to propagate for a long time
• Complexity reduces visibility (verification more prone to mistakes and
oversights)
• When design is flawed, it increases work
• Leads to corrections that are global (hard to test effectively and predict side
effects)
• May take longer to execute than an aggregate of smaller verification activities
2005 MAPLD
118
Design Integrity Concepts
Verification Before Test (cont.)
• Two approaches to pre-test verification (cont.)
– Verify incremental designs
• Benefits
–
–
–
–
Supports development of “known good” sub-designs
Meshes with a partitioned design approach
Facilitates visibility (fewer mistakes, clearer goals)
Allows small flaws to be fixed quickly (minimizes downstream
impact)
• Drawbacks
– Is more work in the ideal case (no/few flaws)
– Increases documentation if formality is vital
• Either approach can be made to work, but one or
the other must be chosen and implemented
2005 MAPLD
119
Design Integrity Concepts
Test Thoroughness
• Principles for test thoroughness
– Every “shall” in the specification that meets the following criteria must be
tested
• Items that may be affected by the instantiation process (subject to fabrication
error)
• Items for which the only extant verification is model based
• Items for which the testing does not pose unacceptable risk (radiation,
overstress, etc.)
– Every instance of a “shall” must be tested
• 18 interfaces require 18 specific tests
– Requirements that have an exclusive character
• Must be tested for correct operation
• Should be tested at some level for abnormal operation
• Example: If you write a 5Ah to something to set it on, you should verify that
writing other values does not set it on
– Requirements must be tested over a representative range of conditions
2005 MAPLD
120
Design Integrity Concepts
Test Thoroughness (cont.)
• A thorough test is complicated and time
consuming
– Requires some level of automation to be
comprehensive
– Requires some level of compromise to be
practical
– Requires team agreement to be acceptable
2005 MAPLD
121
Design Integrity Concepts
Test Thoroughness (cont.)
• Ensuring thorough, yet practical, tests
– Define which tests require manual interaction and which can be
automated
• Output impedance or analog fine precision may need to be manual
• Signal level or analog coarse precision may be automatable
– Define which tests must be performed over environmental
conditions and which can be performed at room temperature only
• Make decision based on character of measurement
• Do not decide based purely on practicality
– Define level of complexity for all measurement sets
• Example: 24 analog inputs (3 eight channel muxes)
• Test full range of values for 24, or limited range for 7 of 8 mux inputs
and full range for remaining
– Define need for repetition of measurement at higher level of
integration (tiered verification)
• Example: voltage level / signal quality / clock jitter at board level
• Repeat at box level?
2005 MAPLD
122
Design Integrity Concepts
Test Thoroughness (cont.)
• Test thoroughness issues affect everything
– Customer relationship
– Test set design (board, box, system)
– Schedule / cost
• An adequate test program is not trivial
– Cannot wait until system level
– Must incorporate lowest level of design
– Must be a constant background activity during design process
• All programs must balance perfect and good enough
• A good test program
–
–
–
–
Will ensure that the specification is met
Will verify everything that can be affected by fabrication
Will build on the analysis and inspection effort
Will maximize automation without sacrificing test accuracy
2005 MAPLD
123
Design Integrity Concepts
Tiered Verification
• Tiered verification is the process of matching the correct type of
verification to the appropriate level of integration
• Tiered verification incorporates all types of verification (before and
during testing)
• Tiered verification applied to the test regime attempts to have
thoroughness with practicality
– Test a requirement at the lowest conceivable level
– Determine which tests must be repeated at higher levels by
• Establishing the purpose of a test at a particular level
• Determining whether the next level has the potential to change previous
measured quantities
• Determining what test set capabilities are
• Tiered verification cannot be retrofitted into the process
– Must be planned up front
– Must be implemented throughout the development process
2005 MAPLD
124
Design Integrity Concepts
Tiered Verification - Example
• Need: A set of 16 high-level pulsed discretes to control
latching relays used to change the spacecraft power
configuration
• Basic Requirements
–
–
–
–
–
–
Quantity:
Level:
Load:
Duration of pulse:
Rise / Fall time max:
Command capability:
– S/W requirements:
2005 MAPLD
16
+30V +/- 2 V
2 kOhm, 4 mH
50 ms
1 ms
Unique bit pattern, only one at a time,
arm first
Various (beyond scope of example)
125
Design Integrity Concepts
Tiered Verification – Example (cont.)
• Tier 1 – Basic Verification (functional simulation)
– Verify each channel fires for a specific bit pattern /
address
– Verify cannot be fired without an arm command
– Verify that other complementary patterns don’t
accidentally fire the pulse
– Analyze drive capability of FPGA and input / output
requirement / capability of translation chip
– Evaluate glitch hardness of logic selected
2005 MAPLD
126
Design Integrity Concepts
Tiered Verification – Example (cont.)
• Tier 2 – Board level
– Test to confirm signal level between FPGA and
driver chips is compatible
– Test rise, fall, duration, level with dummy load
– Repeat test for all 16 outputs
– Replace dummy load with relay, verify that the
relay switches for all 16 outputs
– Repeat over temperature if necessary
2005 MAPLD
127
Design Integrity Concepts
Tiered Verification – Example (cont.)
• Tier 3 – Box level
– Attach GSE (verifies with relays)
– Functionally pulse all outputs
• Verify relay switches
• Verify gross drive duration
• Tier 4 – Box and operational software level
– Verify functionality commanded through software
– GSE verifies correct relay switch
• Tier 5 – System
– Verify gross functionality as needed for system
operation
2005 MAPLD
128
Design Integrity Concepts
Tiered Verification - Notes
• Most complex and detailed functionality is verified at
lower levels
– Performance
– Error cases
– Predictive of future operation
•
•
•
•
Higher levels focus on functionality / operation
Lower level verification requires more manual touch
Higher level verification is more automated
STE and test procedure developed as appropriate
– Purpose is maximum test completeness
– As well as “bang for the buck”
2005 MAPLD
129
Design Integrity Concepts
Concluding the Process
• (“… through the provision of objective evidence…”)
• The importance of traceability
– Something is only verified (formally) when it is documented
– The tie between verification item and specification must be made
(verification matrix or database)
• The importance of planning
– Establish the links between requirement, method, level, test case
early
– Use the established structure to develop verification items
• The importance of buy in
– All responsible must complete verification and report results
– Systems engineers must track and close
– People must keep up with the process
• Only personal commitment from all involved will ensure
that the process is successful
2005 MAPLD
130
Design Integrity Concepts
Summary
• Verification is to prove success, not avoid failure
• Verification planning is an integral part of the project lifecycle
– Must be initiated with specification (including customer buy-in)
– Must be included in design effort
– Must be used to develop documentation and appropriate test hardware
• Verification is more than final test
– Incorporates analysis and inspection throughout design process
– Begins at lowest possible level
– Develops proof of compliance at all levels of operation
• A thorough test is complicated
– Requires extensive work
– Must trade perfect for practical (a tiered test approach can help)
• Verification processes must be tracked and documented
– The whole team must buy in
2005 MAPLD
131
Design Integrity Concepts