Transcript slides - Dave Whipp`s Home Page

A Multi-Paradigm Verification Flow
Dave Whipp
Multiple Paradigms
Analog
Mixed Signal
Digital
Formal
Semi-formal
Simulation
Emulation
Directed tests
Directed random
Constrained random
Self checking
Model based
Copyright © NVIDIA Corporation 2005
Cycle based
Transaction based
Unit level
System level
Plan driven
Coverage driven
Pure Verilog
SystemVerilog
VHDL
SystemC
E / Vera
HDCaml / Confluence
A Multi-Paradigm Verification Flow
What do I mean by Multiple Paradigms?
Why would we want one?
Engineering a [ Multiple Paradigm ] Flow
Was it worth the effort?
Copyright © NVIDIA Corporation 2005
Two Philosophies
Conformity
Emphasize Standardization
Diversity
Emphasize Innovation
Synergy or Conflict?
Copyright © NVIDIA Corporation 2005
Context
Large Project
Multiple Teams
Long Legacy
People move between projects
benefit of familiar environment
exposure to alternative approaches
Copyright © NVIDIA Corporation 2005
Understanding Variation
To understand what to standardize:
you need to understand what not to standardize
Personal Preferences
Technical Aspects of the Designs
Supporting Legacy
Seeking the Next Big Thing
Copyright © NVIDIA Corporation 2005
Personal Preferences
Emacs!
vi
Copyright © NVIDIA Corporation 2005
Personal Preferences
Choice of editor doesn’t affect others
At least, not much
Choice of scripting language has greater impact
But is encapsulated
A script’s users don’t see the implementation language
Choice of HVL affects whole team
Can’t write “E” tests for a “Vera” testbench!
But a unit testbench isn’t seen by other units
A good flow will allow encapsulation of preferences
I can go to any unit and build & run its tests
Enables rapid localization of infrastructure issues
Copyright © NVIDIA Corporation 2005
Technological Differences
Copyright © NVIDIA Corporation 2005
Technical Characteristics
P
C
I
E
Graphics Pipe
Video
Off-chip Memory
Copyright © NVIDIA Corporation 2005
Frame
Buffer
Unit Level System Testing
Graphics Pipeline (Transaction Model)
Stage
N-1
Stage
N
RTL
Unit
Copyright © NVIDIA Corporation 2005
Stage
N+1
DIFF
Transactions Vs Cycles
Data min_val (Addr a1, Addr a2)
Pipelined Bus
{
Data d1 = mem_read(a1);
t1
t2 t3
t4
Data d2 = mem_read(a2);
if (d1 < d2) Address
return d1;
else
Data
return d2;
}
Copyright © NVIDIA Corporation 2005
a1
t5
t6
a2
d1
d2
Reuse Vs Stagnation
Reuse considered Good
Avoid reinventing the wheel
Build on the shoulders of giants
Reuse invites Inertia
Reuse can propagate dependencies
Dependencies make things harder to change
Resistance to change is known as inertia
Inertia can lead to Stagnation
Improper reuse accumulates dependencies
Reused code that is not understood will bit-rot
To avoid stagnation, inject agitation
Copyright © NVIDIA Corporation 2005
Avoiding Stagnation
New Challenges
New Tools
New Platforms
New People
New Ideas
Refactoring
Testability
D.R.Y.
Copyright © NVIDIA Corporation 2005
Are Single Paradigm Projects Possible?
Paradigm 1
Unit A
Unit B
Unit B
Unit B
Unit C
Unit C
Paradigm 2
Paradigm 3
Unit D
time
Project 1
Copyright © NVIDIA Corporation 2005
Project 2
Project 3
A Rich Legacy
A project must support at least three approaches
Legacy
Mainstream
Future
1993
1999
GeForce 256
22 Million
NV1
1 Million
Transistors
Copyright © NVIDIA
Corporation 2005
Transistors
2002
GeForce4
63 Million
Transistors
2003
GeForce FX
130 Million
Transistors
2004
GeForce 6 Series
222 Million
Transistors
2005
GeForce 7 Series
302 Million
Transistors
A Paradigm-Agnostic Verification Flow
What is a Verification Flow?
The User Interface to Verification
Command Line Interface
Graphical Interface
Web (browser) Interface
Programmers Interface
Used By
Verification Engineers
Designers
Managers
Robots
Copyright © NVIDIA Corporation 2005
Purpose of a Flow
Conceptual Framework
Where do things live?
How do things work?
Keep out of the way
Don’t make life difficult for people
Define Metaphors
Subtly influence future directions
Anchor for Variation
Give people something to complain about
Copyright © NVIDIA Corporation 2005
Concepts
A well defined interface behaves as expected
Obvious how it works, just by looking (“affordances”)
As consistent as possible, but no more so
Options are decision points
Users don’t read the manual
Copyright © NVIDIA Corporation 2005
Designing a Flow (Interface)
Invent some Users
Figure out the important activities
Create the user model
Sketch a first draft
Iterate
Watch some real users; and iterate again
Copyright © NVIDIA Corporation 2005
Invent some Users
Copyright © NVIDIA Corporation 2005
Tim
Nalu
Dusk
Mike
Figure out the important Activities
Timbury
New Hire
Needs to get started
Checkout Tree
Run Sanity Regression
Simple Debug
Dusk
Nalu
Testing Guru
Creates Testbenches
Hacks the Flow
Intensive Debug
Setup Regressions
Uses vi
Mike
Lead Designer
Intensive Debug
Modify RTL
Quick Regressions
Analyze Coverage
Uses Emacs
Copyright © NVIDIA Corporation 2005
Manager
View Progress
Tests Written
Pass/fail counts
Coverage
Windows® User
Task Summary
build testbenches
run quick regression tests (sanity tests)
run specific test
rerun specific test with debug options
rerun specific phase(s) of a test
Add a test
Add a testbench
overnight regression testing
Summarize results (web page)
Copyright © NVIDIA Corporation 2005
User Model for Running Tests
Run
Predictor
Build
Testbench
Generate
Tests
Compare
Behaviors
Run
DUT
Copyright © NVIDIA Corporation 2005
The Implementation
“Make” driven flow
Makefiles define the build
Automated testbench creation
Compile/generate tests where appropriate
“makepp” is a better “make”
Perl scripting for the guts
Each test is a Perl object
Small number of standard classes
Individual units define additional classes
Copyright © NVIDIA Corporation 2005
An Early Sketch
rand_regress
nvmk
rdist
vcs
run_unit
stgen
tbgen
randomize
testlist
VSIM
GEN
fmgen
build_testlist
run_testgen
DIFF
CSIM
fmtbgen
compare
tjob
mod
ttgen
LSF
“grok”
Command Line Interface
nvmk
nvmk sanity
nvmk regress
nvmk all
nvmk open_bugs TESTNAME=bug42
nvmk open_bugs TESTNAME=bug42 VSIM=1 DUMP=1
nvmk open_bugs TESTNAME=bug42 CSIM=gdb
Copyright © NVIDIA Corporation 2005
Detailed Triage and Debug
cd rundir/open_bugs/bug42
ls *.log
gen.log csim.log
vsim.log diff.log
replay vsim.log +foo +bar
replay –gdb csim.log
Copyright © NVIDIA Corporation 2005
Testing the Flow
Considered a hard problem
“If it works, then it works”
Testability is a feature
Start testing early
Some tests are better than none
Create a multi purpose “Example Project”
A stable testing environment
Acts as documentation
Use the “Invented Users” to create activity scripts
Keep it up-to-date
Copyright © NVIDIA Corporation 2005
Watch some Real Users
Copyright © NVIDIA Corporation 2005
Image courtesy of DAZ Productions
How to “Watch” Users
Meetings and Discussions
Coffee-Break Grousing
Bug Reports
Keep Track of Support Requests
create FAQs
VNC (Remote Desktop)
Instrumentation
Copyright © NVIDIA Corporation 2005
Biggest Problems
Performance
make-driven user interface was subjectively slow
too much dependency checking
users know what file’s they’ve changed
how to communicate this to the tools?
Too much flexibility
Options imply decisions
Lack of sufficient understanding
So pervasive cut & paste reuse
Copyright © NVIDIA Corporation 2005
Build Time Distribution: 10,000 per sample
100%
90%
80%
> 1 hour
< 1 hour
70%
60%
50%
40%
30%
< 30 min
< 15 min
< 10 min
< 5 min
< 2 min
< 1 min
< 30 sec
20%
10%
0%
Copyright © NVIDIA Corporation 2005
< 20 sec
Build Time Distribution: 1000 per sample
100%
90%
80%
> 1 hour
< 1 hour
70%
60%
50%
40%
30%
< 30 min
< 15 min
< 10 min
< 5 min
< 2 min
< 1 min
< 30 sec
20%
10%
0%
Copyright © NVIDIA Corporation 2005
< 20 sec
Summary
Verification Engineers do more than write tests
A flow is a user interface
Encourage user feedback
Add instrumentation
A flow is an enabler, not disabler
encourage innovation
don’t tell users what not to do
Without diversity, a flow stagnates
Without standardization, a flow flounders
Copyright © NVIDIA Corporation 2005