Transcript lec13-oct13

Lecture 11
Oct 12

Floating-point numbers

Circuits for floating-point operations
Floating-point Representation - Basics




Floating-point numbers - provide a dynamic range of
representable real numbers
Want to represent very large numbers and very small
numbers
More importantly, we want to represent moderate
size numbers (like p or e) with great precision.
Representation - similar to scientific notation
Floating Point Standard
Defined by IEEE Std 754-1985
Developed in response to divergence of
representations
 Portability issues for scientific code
Now almost universally adopted
Two representations
 Single precision (32-bit)
 Double precision (64-bit)
Two parts - significand (or mantissa) M and
exponent (or characteristic) E
The floating-point number F represented by the
pair (M,E) has the value F=ME ( - base of exponent)
Base - common to all numbers in a given
system - implied - not included in the
representation of a floating point number
Example Floating Point Format
• M = largest possible exponent, m = smallest possible
exponent
• Smallest number is 1/8, Largest number is 7/4
• Smallest gap is 1/32
• Largest gap is 1/4
IEEE 754 floating-point standard



Leading “1” bit of significand is implicit
Exponent is “biased” to make sorting easier
 all 0s is smallest exponent all 1s is largest
 bias of 127 for single precision and 1023 for double
precision
sign x (1+significand) x 2exponent – bias
 summary: (–1)
Example:




decimal: -.75 = - ( ½ + ¼ )
binary: -.11 = -1.1 x 2-1
floating point: exponent = 126 = 01111110
IEEE single precision:
10111111010000000000000000000000
IEEE Floating-Point Format
single: 8 bits
double: 11 bits
S Exponent
single: 23 bits
double: 52 bits
Fraction
x  (1)S  (1 Fraction) 2(Exponent Bias)



S: sign bit (0  non-negative, 1  negative)
Normalize significand: 1.0 ≤ |significand| < 2.0
 Always has a leading pre-binary-point 1 bit, so no
need to represent it explicitly (hidden bit)
 Significand is Fraction with the “1.” restored
Exponent: excess representation: actual exponent + Bias
 Ensures exponent is unsigned
 Single: Bias = 127; Double: Bias = 1023
Precision




n bits partitioned into two parts - significand M and
exponent E
n bits – 2n different values
Range between smallest and largest representable
values increases  distance between any two
consecutive values increases
 Floating-point numbers sparser than fixed-point
numbers - lower precision
Real number between two consecutive floating-point
numbers is mapped onto one of the two
 A larger distance between two consecutive
numbers results in a lower precision of
representation
Example: What decimal representation of real
number is represented by the following 32 bit
string (in IEEE format)?
11000000101000…0
Answer: Exponent is 210 (=12910 – 12710)
Sign is negative.
Significand value is (1.01000 …)2 = 1.2510
Value of the number is –(22 x 1.25)10 = - 510
Design choices for FP format
• Recall that significand (mantissa) uses sign-
magnitude format, exponent uses excess
notation.
• Choices mainly motivated by need to
compare two FP numbers very quickly.
• Order relation is determined by comparing
the exponents first.
Normalized and denormalized numbers
If the exponent is between 1 and 254, a normal
real number is represented.
If the exponent is 0:
• if significand is 0, then value = 0.
• if significand is not zero, it represents a
denormalized number.
b1 b2 … b23 represents 0. b1 b2 … b23 rather than
1.b1b2 … b23
Why? To reduce the chance of underflow.
Nonstandard numbers
When the exponent is 255, the number
represented is
• + or – infinity (if the significand is 0)
• NaN (otherwise)
Example: What do the following numbers represent?
0 0 0 0 0 0 0 0 0 1 0 0 0 …0
1 0 0 0 0 0 0 0 0 0 1 0 0 …0
Answer: The first one represents 0.510
The second one represents -0.2510
Set of representable numbers using FP format
The number of distinct normal numbers that can be
represented using a single-precision floating point
format is 232 – 224.
Reason: There are no ambiguous representations, i.e.,
two distinct 32 bits representing normal numbers
represent two distinct real numbers.
Main advantage of double precision FP format is that it
can represent “typical” real numbers more precisely.
The expanded range of representable numbers (upto
2.0 x 10308) is less important.
Absolute and relative error in FP representation
A real number r in the range
-2127 x(2- 2-23) <= r <= 2127 x(2- 2-23)
is either representable (exactly), or there exist two
representable real numbers r1 and r2 such that r1 < r <
r2.
r should be represented as r1 (if |r – r1| < |r – r2|) or r2
(otherwise).
The (absolute) error is |r – r’| where r is the real number
being represented, and r’ is the chosen representation.
Relative error is |r – r’|/|r’|
Problem: Show the representation of the following
decimal numbers in the ANSI/IEEE short and long
floating-point formats. When the number is not exactly
representable, use the round to nearest even rule.
–6.25
Problem: Show the representation of the following
decimal numbers in the ANSI/ISSS short and long floatingpoint formats. When the number is not exactly
representable, use the round to nearest even rule.
–6.25
Answer: sign bit is 1
6.25 = 22 x 1.5625 = 1 10000001 10010000 … 0
This is exact representation.
Problem: What are the absolute and relative errors in
the representation of the real number 0.110 ? (Use the
standard rounding rule.)
FP representation of r = 0.1 is
r1 = 0 00000000 0001100110011001100 (in base 2)
r1 = .099999904632568359375 (in base 10)
Absolute error is ~ 0.00000009
Relative error is ~ 0.0000009 = 9 x 10-7
Relative error is more or less independent of the actual
value of the number.
Relative error for a large number
Example: Consider a large number, e.g. 1035.
What is its FP representation? What is the
relative error in its representation?
Relative error associated with a large
number
Example: Consider a large number, e.g. 1035. What is its
FP representation? What is the relative error in its
representation?
Solution:
Step 1: Write 1035 as 1035 = 2E y where y is in [1, 2).
Step 2: Write E in excess 127 notation.
Step 3: Write y – 1 in base ½.
Step 1: Write 1035 as 1035 = 2E y where y is in [1, 2).
We can compute E using the c++ code:
E = 1; val = 2.0;
while (val < 1.0E35) {
val *= 2;
E++;
}
E--;
y = 1.0E35/val;
The values of E and y are computed as follows:
E = 116
y = 1.203706215242022408
Step 2: Write E = 116 in excess 127 notation.
Step 3: Write y – 1 in base ½.
11110011
y – 1 = 0.203706215242022408
Converting y – 1 in binary:
0.203706215242022408 x 2 = 0. 407412430484044816 < 1 so first bit is 0
0.407412430484044816 x 2 = 0.814824860968089632 < 1 so second bit
is 0
0.814824860968089632 x 2 = 1.629649721936179264 > 1 so 3rd bit is 1
1.629649721936179264 – 1 = 0.629649721936179264
0.629649721936179264 x 2 = 1.259299443872358528 > 1 so 4th bit is 1
etc.
The final result (correct to 23 bits is): 001101000010011000010111
The FP representation of x = 1035 is
y = 0 11110011 001101000010011000010111
Relative error in the FP representation of 1035
Let x = 1035 and y be the number whose FP
representation is
0 11110011 001101000010011000010111
What is (the value of) y?
y ~ 0.999999991400877188214542197 x 1035
Relative error is |x – y|/ x ~ 0.859912281 x 10-9
Why is the integer 33554431 not exactly representable?
Slide source: UIUC, CS 232 class lecture
Effect of Loss of Precision
According to the
General
Accounting
Office of the U.S.
Government, a
loss of precision
in converting 24bit integers into
24-bit floating
point numbers
was responsible
for the failure of
a Patriot anti-
missile battery.
Effect of Loss of Precision
Slide source: UIUC, CS 232 class lecture
Floating-Point Addition

Consider a 4-digit decimal example


1. Align decimal points



9.999 × 101 + 0.016 × 101 = 10.015 × 101
3. Normalize result & check for over/underflow


Shift number with smaller exponent
9.999 × 101 + 0.016 × 101
2. Add significands


9.999 × 101 + 1.610 × 10–1
1.0015 × 102
4. Round and renormalize if necessary

1.002 × 102
Floating-Point Addition

Now consider a 4-digit binary example


1. Align binary points



1.0002 × 2–1 + –0.1112 × 2–1 = 0.0012 × 2–1
3. Normalize result & check for over/underflow


Shift number with smaller exponent
1.0002 × 2–1 + –0.1112 × 2–1
2. Add significands


1.0002 × 2–1 + –1.1102 × 2–2 (0.5 + –0.4375)
1.0002 × 2–4, with no over/underflow
4. Round and renormalize if necessary

1.0002 × 2–4 (no change) = 0.0625
FP Adder Hardware


Much more complex than integer adder
Doing it in one clock cycle would take too
long



Much longer than integer operations
Slower clock would penalize all instructions
FP adder usually takes several cycles

Can be pipelined
Floating point addition
Start

1. Compare the exponents of the two numbers.
Shift the smaller number to the right until its
exponent would match the larger exponent
2. Add the significands
3. Normalize the sum, either shifting right and
incrementing the exponent or shifting left
and decrementing the exponent
Overflow or
underflow?
Yes
No
4. Round the significand to the appropriate
number of bits
No
Still normalized?
Yes
Done
Exception
FP Adder Hardware
Step 1
Step 2
Step 3
Step 4
Addition and Subtraction




Exponents of both operands must be equal before adding or
subtracting significands
Significands aligned by shifting the significand of the smaller
operand |E1-E2| base- positions to the right, increasing its
exponent, until exponents are equal
E1E2 -
Exponent of larger number not decreased - this will result in a
significand larger than 1 - a larger significand adder required
Addition/Subtraction – post-normalization




Addition - resultant significand M (sum of two aligned
significands) is in range 1/  M < 2
If M>1 - a postnormalization step - shifting significand
to the right to yield M3 and increasing exponent by
one - is required (an exponent overflow may occur)
Subtraction - Resultant significand M is in range 0 
|M|< 1 - postnormalization step - shifting significand to
left and decreasing exponent - is required if M<1/
(an exponent underflow may occur)
In extreme cases, the postnormalization step may
require a shift left operation over all bits in significand,
yielding a zero result
Example




F1=(0.100000)16  163 ; F2=(0.FFFFFF)16  162
Short IBM format ; calculate F1 – F2
Significand of smaller number (F2) is shifted to the
right - least-significant digit lost
Shift is time consuming - result is wrong
Example - Continued




Correct result (with “unlimited" number of
significand digits)
Error (also called loss of significance) is
0.1  16-2 - 0.1 
16-3 = 0.F  16-3
Solution to problem - guard digits - additional
digits to the right of the significand to hold shiftedout digits
In example - a single (hexadecimal) guard digit is
sufficient
Steps in Addition/Subtraction of
Floating-Point Numbers





Step 1: Calculate difference d of the two exponents
- d=|E1 - E2|
Step 2: Shift significand of smaller number by d base positions to the right
Step 3: Add aligned significands and set exponent
of result to exponent of larger operand
Step 4: Normalize resultant significand and adjust
exponent if necessary
Step 5: Round resultant significand and adjust
exponent if necessary
Floating Point Complexities

Operations are somewhat more complicated (see text)

In addition to overflow we can have “underflow”

Accuracy can be a big problem



IEEE 754 keeps two extra bits, guard and round

four rounding modes

positive divided by zero yields “infinity”

zero divide by zero yields “not a number”

other complexities
Implementing the standard can be tricky
Not using the standard can be even worse
 see text for description of 80x86 and Pentium bug!
FP Arithmetic Hardware

FP multiplier is of similar complexity to FP
adder


FP arithmetic hardware usually does



But uses a multiplier for significands instead
of an adder
Addition, subtraction, multiplication, division,
reciprocal, square-root
FP  integer conversion
Operations usually takes several cycles

Can be pipelined
FP Instructions in MIPS
FP hardware is coprocessor 1

Adjunct processor that extends the ISA
Separate FP registers


32 single-precision: $f0, $f1, … $f31
Paired for double-precision: $f0/$f1, $f2/$f3, …

Release 2 of MIPs ISA supports 32 × 64-bit FP reg’s
FP instructions operate only on FP registers


Programs generally don’t do integer ops on FP
data, or vice versa
More registers with minimal code-size impact
FP load and store instructions

lwc1, ldc1, swc1, sdc1

e.g., ldc1 $f8, 32($sp)
FP Instructions in MIPS

Single-precision arithmetic

add.s, sub.s, mul.s, div.s


Double-precision arithmetic

add.d, sub.d, mul.d, div.d


e.g., mul.d $f4, $f4, $f6
Single- and double-precision comparison


c.xx.s, c.xx.d (xx is eq, lt, le, …)
Sets or clears FP condition-code bit


e.g., add.s $f0, $f1, $f6
e.g. c.lt.s $f3, $f4
Branch on FP condition code true or false

bc1t, bc1f

e.g., bc1t TargetLabel
FP Example: °F to °C

C code:
float f2c (float fahr) {
return ((5.0/9.0)*(fahr - 32.0));
}
 fahr in $f12, result in $f0, literals in global memory
space

Compiled MIPS code:
f2c: lwc1
lwc2
div.s
lwc1
sub.s
mul.s
jr
$f16,
$f18,
$f16,
$f18,
$f18,
$f0,
$ra
const5($gp)
const9($gp)
$f16, $f18
const32($gp)
$f12, $f18
$f16, $f18
FP Example: Array Multiplication

X=X+Y×Z


All 32 × 32 matrices, 64-bit double-precision
elements
C code:
void mm (double x[][],
double y[][], double z[][]) {
int i, j, k;
for (i = 0; i! = 32; i = i + 1)
for (j = 0; j! = 32; j = j + 1)
for (k = 0; k! = 32; k = k + 1)
x[i][j] = x[i][j]
+ y[i][k] * z[k][j];
}

Addresses of x, y, z in $a0, $a1, $a2, and
i, j, k in $s0, $s1, $s2
FP Example: Array Multiplication

MIPS code:
li
li
L1: li
L2: li
sll
addu
sll
addu
l.d
L3: sll
addu
sll
addu
l.d
…
$t1, 32
$s0, 0
$s1, 0
$s2, 0
$t2, $s0, 5
$t2, $t2, $s1
$t2, $t2, 3
$t2, $a0, $t2
$f4, 0($t2)
$t0, $s2, 5
$t0, $t0, $s1
$t0, $t0, 3
$t0, $a2, $t0
$f16, 0($t0)
#
#
#
#
#
#
#
#
#
#
#
#
#
#
$t1 = 32 (row size/loop end)
i = 0; initialize 1st for loop
j = 0; restart 2nd for loop
k = 0; restart 3rd for loop
$t2 = i * 32 (size of row of x)
$t2 = i * size(row) + j
$t2 = byte offset of [i][j]
$t2 = byte address of x[i][j]
$f4 = 8 bytes of x[i][j]
$t0 = k * 32 (size of row of z)
$t0 = k * size(row) + j
$t0 = byte offset of [k][j]
$t0 = byte address of z[k][j]
$f16 = 8 bytes of z[k][j]
FP Example: Array Multiplication
…
sll $t0, $s0, 5
addu $t0, $t0, $s2
sll
$t0, $t0, 3
addu $t0, $a1, $t0
l.d
$f18, 0($t0)
mul.d $f16, $f18, $f16
add.d $f4, $f4, $f16
addiu $s2, $s2, 1
bne
$s2, $t1, L3
s.d
$f4, 0($t2)
addiu $s1, $s1, 1
bne
$s1, $t1, L2
addiu $s0, $s0, 1
bne
$s0, $t1, L1
#
#
#
#
#
#
#
#
#
#
#
#
#
#
$t0 = i*32 (size of row of y)
$t0 = i*size(row) + k
$t0 = byte offset of [i][k]
$t0 = byte address of y[i][k]
$f18 = 8 bytes of y[i][k]
$f16 = y[i][k] * z[k][j]
f4=x[i][j] + y[i][k]*z[k][j]
$k k + 1
if (k != 32) go to L3
x[i][j] = $f4
$j = j + 1
if (j != 32) go to L2
$i = i + 1
if (i != 32) go to L1
Interpretation of Data
The BIG Picture

Bits have no inherent meaning


Interpretation depends on the instructions
applied
Computer representations of numbers


Finite range and precision
Need to account for this in programs

Parallel programs may interleave
operations in unexpected orders

Assumptions of associativity may fail
(x+y)+z
x+(y+z)
-1.50E+38
x -1.50E+38
y 1.50E+38 0.00E+00
z
1.0
1.0 1.50E+38
1.00E+00 0.00E+00

Need to validate parallel programs
under varying degrees of parallelism
§3.6 Parallelism and Computer Arithmetic: Associativity
Associativity

Originally based on 8087 FP coprocessor




FP values are 32-bit or 64 in memory



8 × 80-bit extended-precision registers
Used as a push-down stack
Registers indexed from TOS: ST(0), ST(1), …
Converted on load/store of memory operand
Integer operands can also be converted
on load/store
Very difficult to generate and optimize code

Result: poor FP performance
§3.7 Real Stuff: Floating Point in the x86
x86 FP Architecture
x86 FP Instructions
Data transfer
Arithmetic
Compare
Transcendent
al
FILD mem/ST(i)
FISTP mem/ST(i)
FLDPI
FLD1
FLDZ
FIADDP
FISUBRP
FIMULP
FIDIVRP
FSQRT
FABS
FRNDINT
FICOMP
FIUCOMP
FSTSW AX/mem
FPATAN
F2XMI
FCOS
FPTAN
FPREM
FPSIN
FYL2X

mem/ST(i)
mem/ST(i)
mem/ST(i)
mem/ST(i)
Optional variations




I: integer operand
P: pop operand from stack
R: reverse operand order
But not all combinations allowed
Streaming SIMD Extension 2 (SSE2)

Adds 4 × 128-bit registers


Extended to 8 registers in AMD64/EM64T
Can be used for multiple FP operands



2 × 64-bit double precision
4 × 32-bit double precision
Instructions operate on them
simultaneously

Single-Instruction Multiple-Data


Left shift by i places multiplies an
integer by 2i
Right shift divides by 2i?


Only for unsigned integers
For signed integers


Arithmetic right shift: replicate the sign bit
e.g., –5 / 4



111110112 >> 2 = 111111102 = –2
Rounds toward –∞
c.f. 111110112 >>> 2 = 001111102 = +62
§3.8 Fallacies and Pitfalls
Right Shift and Division
Who Cares About FP Accuracy?

Important for scientific code

But for everyday consumer use?


“My bank balance is out by 0.0002¢!” 
The Intel Pentium FDIV bug


The market expects accuracy
See Colwell, The Pentium Chronicles

ISAs support arithmetic



Bounded range and precision


Signed and unsigned integers
Floating-point approximation to reals
Operations can overflow and underflow
MIPS ISA

Core instructions: 54 most frequently used


100% of SPECINT, 97% of SPECFP
Other instructions: less frequent
§3.9 Concluding Remarks
Concluding Remarks
Floating-point numbers
Denormals allow graceful underflow
–
Negative numbers
max –
FLP –
min –
Sparser
Overflow
region
0
Denser
Positive numbers
min +
FLP +
Denser
Underflow
example
+
Sparser
Underflow
regions
Midway
example
max +
Overflow
region
Typical
example
Overflow
example
Distribution of floating-point numbers on the real line.

Representation for non-integral numbers


Like scientific notation




–2.34 × 1056
+0.002 × 10–4
+987.02 × 109
normalized
not normalized
In binary


Including very small and very large numbers
±1.xxxxxxx2 × 2yyyy
Types float and double in C
§3.5 Floating Point
Floating Point
Single-Precision Range


Exponents 00000000 and 11111111 reserved
Smallest value




Exponent: 00000001
 actual exponent = 1 – 127 = –126
Fraction: 000…00  significand = 1.0
±1.0 × 2–126 ≈ ±1.2 × 10–38
Largest value



exponent: 11111110
 actual exponent = 254 – 127 = +127
Fraction: 111…11  significand ≈ 2.0
±2.0 × 2+127 ≈ ±3.4 × 10+38
Double-Precision Range


Exponents 0000…00 and 1111…11 reserved
Smallest value




Exponent: 00000000001
 actual exponent = 1 – 1023 = –1022
Fraction: 000…00  significand = 1.0
±1.0 × 2–1022 ≈ ±2.2 × 10–308
Largest value



Exponent: 11111111110
 actual exponent = 2046 – 1023 = +1023
Fraction: 111…11  significand ≈ 2.0
±2.0 × 2+1023 ≈ ±1.8 × 10+308
Floating-Point Precision
Relative precision


all fraction bits are significant
Single: approx 2–23


Equivalent to 23 × log102 ≈ 23 × 0.3 ≈ 6 decimal
digits of precision
Double: approx 2–52

Equivalent to 52 × log102 ≈ 52 × 0.3 ≈ 16 decimal
digits of precision
Floating-Point Example

Represent –0.75




–0.75 = (–1)1 × 1.12 × 2–1
S=1
Fraction = 1000…002
Exponent = –1 + Bias




Single: –1 + 127 = 126 = 011111102
Double: –1 + 1023 = 1022 = 011111111102
Single: 1011111101000…00
Double: 1011111111101000…00
Floating-Point Example

What number is represented by the
single-precision float
11000000101000…00




S=1
Fraction = 01000…002
Fxponent = 100000012 = 129
x = (–1)1 × (1 + 012) × 2(129 – 127)
= (–1) × 1.25 × 22
= –5.0