Transcript Advancing Metrics on the Standards Track

Advancing Metrics on
the Standards Track
Equivalence Procedure & Lab Results
draft-morton-ippm-advance-metrics-02
Al Morton Nov 2010
ACK: Len Ciavattone for NetProbe
1
Outline


Continue Definition-centric metric
advancement
Procedure to determine Threshold of
Equivalence


Possible Report Template with Procedures


Key substitution for Interoperability
Supports the Protocol Action Request
Open call for participation in Lab tests
2
Definition-Centric Process (revised)
,---.
/
\
( Start )
\
/
Implementations
`-+-'
+-------+
|
/|
1
`.
+---+----+
/ +-------+ `.-----------+
,-------.
| RFC
| /
|Check for |
,' was RFC `. YES
|
| /
|Equivalence..... clause x
-------+
|
|/
+-------+ |under
|
`. clear? ,'
|
| Metric \.....|
2
....relevant
|
`---+---'
+----+---+
| Metric |\
+-------+ |identical |
No |
|Report |
| Metric | \
|network
|
+--+----+
|results+|
| ...
| \
|conditions |
|Modify |
|Advance |
|
|
\ +-------+ |
|
|Spec
+----+RFC
|
+--------+
\|
n
|.'+-----------+
+-------+
|request?|
+-------+
+--------+
3
Key Points (the sub-points)

Start with an RFC


Run test(s) with Implementations


Test plan is customized to a specific clause
Evaluate Measurements & Compare



Focus on a specific clause
Expected measurement results are Clear
Obvious place to take action if text is found to be
ambiguous
Final state is Report Dev. for Protocol Action Req.
4
Setting the Equivalence Threshold

Proposal – part of the allowable error (or
variability) is both test configuration and
implementation dependent




Maximum_Error =
Same_Implementation_Error + Systematic_Error
Only Systematic_Error need be agreed a priori
Systematic_Error should be set such that unexpected
implementation differences are detectable <<= KEY
At least, the variability in results from the Same
Implementation should be allowable.
5
Section 3.4 One-way Delay, ADK
Sample Metric (Lab)
 1. Configure a path with X ms one-way constant delay.


2. Measure a sample of one-way delay singletons with
2 or more implementations, using identical options.
3. Measure a sample of one-way delay singletons with
additional instances of the *same* implementations,
using identical options,
 noting that connectivity differences MUST be the
same as for the *cross* implementation testing.
6
Test Set-up (6 mcast streams from
MS-1 to all others, OW meas at MS-2)
NetProbe MS-1
negotiated
100baseTx-FD
10.201.0.1
eth2 10.201.0.254
eth3 10.202.0.254
NetProbe MS-2
10.202.0.1
10.203.0.1
NetProbe MS-3
Network Emulator
eth4 10.203.0.254
eth5 10.204.0.254
10.204.0.1
NetProbe MS-4
Management
LAN
Test LANs
7
Section 3.4 One-way Delay, ADK
Sample Metric (Lab) --- Results
Agate Statistical Analysis Program
k- sample Anderson Darling Test for batch equivalence
(ADK < ADC for same population)
ADK
ADC (a = 0.05)
76.109
77.332
77.438
77.438
77.319
2.479
2.479
2.479
2.479
2.479
ADC (a = 0.025)
3.050
3.050
3.050
3.050
3.050
ADC (a = 0.01)
3.830
3.830
3.830
3.830
3.830
NO
NO
NO
NO
NO
Same Population ?(a=0.025)
Modified CV Data
ADK
Same Population ?(a=0.025)
8
Section 3.4 One-way Delay, ADK
Sample Metric (Lab) --- Results
9
One-way Delay, ADK Sample Metric
(Lab) --- Normalized Results
k- sample Anderson Darling Test for batch equivalence
ADK
ADC (a = 0.05)
2.248
ADC (a = 0.025)
3.050
ADC (a = 0.01)
3.830
Same Population ?(a=0.025)
2.479
YES
Modified CV Data
ADK
Same Population ?(a=0.025)
10
Notes on Network Emulator
Correlated Loss Generation

Many emulators use this relationship:
Corr_value =
Last_value * corr_coeff + New_value * (1-corr_coeff)


Works adequately for Delay
Tests with NIST Net indicate some shortcomings



Possible that this function was lost in revisions
Also Possible that our installation has this shortcoming
Investigating these possibilities and alternative,
yet simple correlation functions
11
3.2: First-bit to Last-bit, NetProbe
1400 byte payload
32 byte payload
Delay for each mode
(one mode)
microseconds
microseconds
1001621
1002735
Delay Diff
microseconds
1000356
1000356
1265
2379
Expected Diff
microseconds
1094.4
1094.4
TxDelay(us)
1003000
1002500
1002000
1001500
1001000
1000500
1
4
7 10 13 16 19 22 25 28 31 34 37 40 43 46 49 52 55 58
12
Update on the Bi-modal Delay


Additional investigation appears to conclude that the modal
behavior is related to interrupt-to-frame arrival settings of the
specific interface board.
Various options appear to be configurable, but only when the
interface driver is compiled as a module.

Also, the board/driver does not support the "coalesce"
options of ethtool.

Until we can rebuild the Linux machine with this and other
planned modifications, confirmation will have to wait.
13
Summary




Key clauses of RFC 2679 evaluated in Lab
Proposal for Equivalence Threshold and
Procedure
Network Emulator Testing – Loss Correlation
One Implementation: NetProbe
 Other volunteers?

P/O This draft could become basis for the
Protocol Action Request for RFC 2679


draft-morton-ippm-advance-metrics-02
What other RFCs should we target first?
14
NetProbe 5.8.5




Runs on Solaris (and Linux, occasionally)
Pre-dates *WAMP, functionally similar
Software-based packet generator
Provides performance measurements
including Loss, Delay, PDV, Reordering,
Duplication, burst loss, etc. in postprocessing on stored packet records
15
Report Template – Loss Threshold

3.1. One-way Delay, Loss threshold, RFC
2679 (procedure)


3.1.1. NetProbe Lab results for Loss Threshold
3.1.2. XXX Lab Results for Loss Threshold


<your compliant implementation here>
3.1.3. Conclusions on Lab Results for Loss
Threshold
16
Section 3.1 – Loss Threshold
See Section 3.5 of [RFC2679], 3rd bullet point and also
Section 3.8.2 of [RFC2679].
 1. configure a path with 1 sec one-way constant delay
 2. measure (average) one-way delay with 2 or more
implementations, using identical waiting time thresholds for
loss set at 2 seconds
 3. configure the path with 3 sec one-way delay (or change
the delay while test is in progress, measurements in step 2)
 4. repeat measurements
 5. observe that the increase measured in step 4 caused all
packets to be declared lost, and that all packets that arrive
successfully in step 2 are assigned a valid one-way delay.
Results: 22 of 38 packets were declared lost (post-process).

17
Section 3.2: First-bit to Last-bit
See Section 3.7.2 of [RFC2679], and Section 10.2 of [RFC2330].





1. configure a path with 1000 ms one-way constant delay, and ideally
including a low-speed link (10-baseT, FD)
2. measure (average) one-way delay with 2 or more implementations,
using identical options and equal size small packets (e.g., 32 octet IP
payload)
3. maintain the same path with 1000 ms one-way delay
4. measure (average) one-way delay with 2 or more implementations,
using identical options and equal size large packets (e.g., 1400 octet IP
payload)
5. observe that the increase measured in steps 2 and 4 is equivalent to
the increase in ms expected due to the larger serialization time for each
implementation. Most of the measurement errors in each system should
cancel, if they are stationary.
18
3.2: First-bit to Last-bit, NetProbe

Bi-modal delay behavior in the Network
Emulator for UDP payload 96 octets & up




Same for Poisson, Periodic, 1 sec, 10ms spacing
Not evident in tests on the Management LAN
Delay increased with larger payload, but more
than expected (170 us for low mode)
Suggestions to investigate further are welcome
Tests on Management LAN, 100baseTx-FD
1400 byte payload
32 byte payload
Ave Delay
Ave Delay
microseconds
microseconds
443.37
143.57
Diff
Expected Diff
microseconds microseconds
299.8
109.44
(190.36)
19
Other Examples

One-way Delay, RFC 2679

This test is intended to evaluate measurements in sections 3 and 4 of
[RFC2679].
Average delays before/after 2 second increase
Average pre-increase delay, microseconds
Average post 2s additional, microseconds
Difference (should be ~= Y = 2s)

1000276.6
3000282.6
2000006
Error Calibration, RFC 2679

This is a simple check to determine if an implementation reports the error
calibration as required in Section 4.8 of [RFC2679].
20