Transcript Chang Gung University & National Taiwan University Motive The

IEEE 9th International Symposium on
Consumer Electronics (ISCE 2005)
Automobile Lane Detection Systemon-Chip Integrated with Mixed
Signal Mode CMOS Image Sensor
Pei-Yung Hsiao¹, Hsien-Chein Cheng¹, Chun-Wei Yeh¹,
Shih-Shinh Huang², and Li-Chen Fu²
¹ Department of Electronic Engineering
Chang Gung University
² Department of Electrical Engineering
National Taiwan University
1
Chang Gung University & National Taiwan University
OUTLINE
 The
Problem & Motive
 Brief Introduction to Algorithms
 Architecture and Circuits description
 Simulation and Results
 Conclusion
2
Chang Gung University & National Taiwan University
Original Idea for the Problem
We are interested in developing a Systemon-Chip, Soc, which can capture image as
well as produce vehicle lane map at the
same time.
Capture
Image
Land map
3
Chang Gung University & National Taiwan University
Motive

The areas of Intelligent Transportation
System, ITS, include lane detection, obstacle
recognition, vehicle detection, car following,
etc.

Our goal in this investigation is to develop a
CMOS imager to achieve real-time image
capture and lane detection, simultaneously,
for intelligent automotive driver
awareness/assistance system.
4
Chang Gung University & National Taiwan University
Automotive IC Design
Lane Departure
Prevention
Automobile Lane
Detection SoC
Built in
Vehicle Detection &
Tracking
Driver Assistance
System
The target chip can be defined as a
component device for intelligent vehicles.
5
Chang Gung University & National Taiwan University
Widespread Applications
 The
proposed imager without
demanding extra ADC circuits for
signal transformation is a single
low-cost and compact chip for used
in the thousands of consumer
electronics not limited to ITS.
6
Chang Gung University & National Taiwan University
Introduction(1/4)

From the referenced literatures, there are a lot
of vision-based lane detection algorithms
proposed in recent 10 years [1-6] (1995-2004).

In 1995, Kluge and Lakshmanan [3] proposed
the LOIS (Likelihood of Image Shape) lane
detection, which is able to detect lanes even
in situations with shadows or broken lanes by
using a stochastic optimization procedure.
7
Chang Gung University & National Taiwan University
Introduction (2/4)

In 1995, Broggi [5] proposed an edge-based
road detection algorithm, while it is effective
only for the well-painted road.

In 1999, Takahashi, etc. [4] divided the
parameter space of the lane model to generate
the lane marking patterns and then applied the
voting scheme to find the lane boundary.
8
Chang Gung University & National Taiwan University
Introduction (3/4)

In 2004, Huang, etc [1] proposed an on-board
vision system for lane recognition and frontvehicle detection to enhance driver's
awareness.

Regarding to high recognition rate and hardwired regularity, we adopted Peak-Finding
based lane detection algorithm from Huang,
etc [1], which has high recognition rate about
96%.
9
Chang Gung University & National Taiwan University
Introduction (4/4)

According to Huang’s algorithm, we also
developed an auto-regulated threshold
circuit to automatically adjust the
threshold for the lane detector to
adapted to different weather conditions.
10
Chang Gung University & National Taiwan University
Architecture and Circuit description



Our chip can be divided into three parts, such
as analogue capturing and processing circuits,
digital processing circuits and digital control
unit.
The analogous circuits include 2-D pixel cell
array, CDS module, 1-D Gaussian filter and
Peak-Finding module.
The digital processing circuits are composed
of Line Point Allocation module, column
selector and row selector.
11
Chang Gung University & National Taiwan University
SOC for real-time image capture and
lane detection
Control
Unit
Column
Selector
CDS Circuit
Lane-Point Finding Module
CMOS
Image Sensor
Array
Peak-Finding Module
Row Selector
Upper Region
Lane-Point
Output
Gaussian Filter
Module
Analogous
Processing circuits
Digital Processing
Circuits
12
Chang Gung University & National Taiwan University
Pixel Cell & Sensor Array





The developed sensor array consists of two types of
pixel cells.
Our sensor array prototype is made up of 64*64
effective pixels.
The upper region containing 16*64 pixels is ignored in
back-end processing to promote the computing
efficiency.
The other regions are horizontally partitioned into
three sub-regions. Each sub-region consists of 16
rows.
The 12th row in the sub-region or in the upper region
is defined as a 1-D sample array. Consequently, we
have four 1-D sample arrays in our sensor array.
13
Chang Gung University & National Taiwan University
1,1
i-2,1
i-1,1
i,1
i+1,1
i+2,1
64,1
Reset
1.10
i-2,10
i-1,10
i,10
i+1,10
i+2,10
64,10
X
SEL
Row Selector
Photodiode
Ipd
Ip
1,11
i-2,11
i-1,11
i,11
i+1,11
i+2,11
64,11
Normal cell
1,12
i-2,12
i-1,12
i,12
i+1,12
i+2,12
64,12
1,64
i-2,64
i-1,64
i,64
i+1,64
i+2,64
64,64
Ip(i-1,j)
Ip(i,j)
Ip(i+1,j
)
Ip(i+2,j
)
Ip(64,j)
Ip(1,j)
Ip(i-2,j)
Ips
Reset
CDS Circuit
Gaussian Filter (Current)
IG(i,j)
X
SEL
Gaussian Filter (Preivois)
IG(i-1,j)
Photodiode
Ipd
Ip
Sampling cell
CL
K
Column Selector
14
Ips
Chang Gung University & National Taiwan University
Dual 1-D Gaussian Filters

Each 1-D Gaussian filter includes 64 Gaussian
mask cells and a current divider. Each
Gaussian mask cell consists of 3 current
mirrors in 7 transistors and two OR gates.

The Gaussian filter module is used for
smoothing the selected pixel by referring to a
couple of right and left neighbors to eliminate
noisy points in the original image.
15
Chang Gung University & National Taiwan University
Dual 1-D Gaussian Filters
Current Gaussian Filter
IG(i,j)
S(64)
S(62)
S(i)
S(i+1)
Ip(64,j)
S(i+2)
S(i-1)
S(i-2)
S(1)
S(3)
S(2)
Ip(i,j)
S(63)
Ip(1,j)
Previous Gaussian Filter
S(63)
S(62)
S(61)
S(60)
S(i-1)
S(i)
S(i+1)
S(i-2)
S(i-3)
S(1)
S(2)
IG(i-1,j)
16
Chang Gung University & National Taiwan University
Peak-Finding Module (1/3)

The 1st part of the Peak-Finding Module can
accumulate and average current, Iavg, from
the aforementioned sample arrays, Ips.

The averaged current from sample array, Iavg,
was generated according to the following
equation.
1 n 1, 64
Ia vg 
Ips (12  16 * i, j )

n i 0, j 1
17
Chang Gung University & National Taiwan University
Vref
Auto-Regulated Threshold Circuit
Vbias
From
Sample Array
Vrefn
Ibias
Threshold
Mapping Circuit
Peak-Finding Module
Threshold
Mapping Circuit
Ips
ITH
Iavg
Threshold
Mapping Circuit
Threshold
Mapping Circuit
Irefn
M1
Vcn
M2
Iavg
IG(i,j)
Pp(i,j)
IG(i-2,j)
Isth
Peak-Point
Output
Threshold Mapping Circuit
Lb(i,j)
Lane-Point
Output
P1
P2
P3
P4
P5
P6
P7
P8
P9
P35
P36
P38
P37
P101
P100
P73
P72
P71
P(i-1,j)
P(I,j)
P68
P67
P66
P65
P102
P103
P129
P130
P131
P132
P133
P134
P135
P136
P137
Pp(i,j)
Line-Point Allocation
18
Chang Gung University & National Taiwan University
Peak-Finding Module (2/3)

The 2nd part of PFM is called as autoregulated threshold circuit. It compares
average current, Iavg, with four
preconfigured currents, Irefn, and then
produces the threshold current, Isth.

The total threshold current, ITH can be
noted by the following equation.
ITH  Ibias   Isth
4
, where
n 1
WM 2
Isth 
unc0(VCn  VT ) 2
LM 2
19
Chang Gung University & National Taiwan University
Peak-Finding Module (3/3)

Inside the auto-regulated threshold circuit, each
threshold mapping circuit control a threshold current.
It can be noted by the following equation:
 1, if Irefn  Iavg
VCn  
0 , otherwise
Irefn 
, where
WM 1
unc0(
LM 1
Rn
R
5
Vref

VT ) 2
m
m 1

According to the 3rd part of the PFM, If the current
pixel is a peak point, the output value will be 1,
otherwise, it should be 0.
 1, if IG (i, j )  ITH  IG (i  1, j )
Pp (i, j )  
0 , otherwise
20
Chang Gung University & National Taiwan University
Line-Point Allocation

The Lane-Point Allocation Module expressed
as the following equation is composed of two
digital functions, such as the line segment
filter and the lane point selector.
Lbi, j   Ppi  1, j   Ppi, j   [ Ppi  n, j  1   Pp(i  m, j  1)]

3
3
n  3
m  3
Lane points, Lb(i,j), are obtained, and
represented by only one pixel width in each
row.
21
Chang Gung University & National Taiwan University
Timing Diagram
Reset
clk C
clk R
i (Row)
i (Col)
Timing Cycle
Peak-Point
Pp(i,j)
Lane-Boun
Lb(i,j)
64
64
2
2
1
2
64
64
65
66
128
3969 3970
4032 4033 4044
4096 4097
1,1
63,1
64,1
1,2
63,2
64,62 1,63
63,63 64,63 1,64
63,64 64,64 1,1
--- ---
62,1
63,1
64,1
62,2
63,62 64,62
62,63 63,63 64,63
62,64 63,64 64,64
1
1
1
2
64
1
2
---
63
1
2
63
64
1
2
64
1
1
2
4098
The clock frequency of the Row Selector (clk R) is 64 times of the
Column Selector (clk C). In this case, the frequency of the Column
Selector is 25MHz and the frequency of the Row Selector is 0.78MHz.
22
Chang Gung University & National Taiwan University
HSPICE Simulation (1/2)
(a)
(b)
(c)
Peak-Point
1.5us
(a) is the output current of the current Gaussian Filter. (b) is the
output current of the previous Gaussian Filter. (c) is the results
of the Peak-Finding Module.
23
Chang Gung University & National Taiwan University
HSPICE Simulation (2/2)
(a)
(b)
(c)
(d)
5us
25us
(a) is the simulation results of the current Gaussian Filter. (b) is the
simulation results of the previous Gaussian Filter. (c) is the simulation
result of the Peak-Finding Module. (d) is the simulation results of the
Lane-Point Allocation Module.
24
Chang Gung University & National Taiwan University
Software Simulation in C
From [1], noises
are to be removed
by the followed
post processing.
(a) Original image
(512x512 > 320x240[1])
(b) Lane map points generated
by Peak-Finding algorithm.
(c) Original image
(64x64)
(d) Lane map points generated
by Peak-Finding algorithm.
25
Chang Gung University & National Taiwan University
Experimental Results
(a) Original image
in 32 * 32
(b) Result generated
by software
(c) Result generated
by our chip
26
Chang Gung University & National Taiwan University
Chip Layout in 64 * 64
27
Chang Gung University & National Taiwan University
Specification of The Proposed CMOS Imager
Item
Pixel Count
Pixel Size
Aperture Size
Fill Factor
Image Size
Chip Size
Operation Clock
Operation Voltage
Power consumption
Values
64(H) X 64(V)
18.45 um(H) X 21.8 um (V)
12.45 um(H) X 9.6 um (V)
29.7 %
1217.7 um (H) X 1455.05 um
(V)
2191.4 um (H) X 2389.8 um (V)
25MHz
3.3 V
159.4mW
28
Chang Gung University & National Taiwan University
Conclusion
Our investigation includes a 2-D image sensor
array embedded with four modularized circuits:
----- four 1-D sample array by different pixel cell
design for accumulating the sampled currents;
----- dual 1-D Gaussian filers coupling as an
analogue image smoothing module;
----- an analogue design for Peak-Finding function
associating with a novel auto-regulated
threshold operation;
----- a sophisticated digital implementation for
Lane-Point allocation.

29
Chang Gung University & National Taiwan University
Conclusion, cont.

A new current-mode mixed signal
design of CMOS image sensor
integrated with Peak-Finding based
lane detection algorithm is developed.

The proposed low-cost and one
compact chip solution can grab the
road images from the real world and
successfully detect the lane markers
simultaneously, in real time.
30
Chang Gung University & National Taiwan University
Reference (1/4)
[1] Shih-Shinh Huang, Chung-Jen Chen, Pei-Yung Hsiao, and LiChen Fu, “On-Board Vision System for Lane Recognition and
Front-Vehicle Detection to Enhance Driver's Awareness”,
IEEE International Conference on Robotics and Automation,
vol. 3, 26 April – 1 May, 2004, pp.2456–2461.
[2] Yue Wang, Eam Khwang Teoh and Dinggang She, “ Lane
detection using B-snake”, , International Conference on
Information Intelligence and Systems, 31 Oct. - 3 Nov., 1999,
pp.438 – 443.
[3] Kluge, K., and S. Lakshmanan,, “A Deformable-Template
Approach to Lane Detection”, in I. Masaky, editor,
Proceedings IEEE Intelligent Vehicle’95, Detroit, 25-26 Sept.,
1995, pp.54-59.
[4] Takahashi, A., Ninomiya, Y., Ohta, M., and Tange, K., “A
Robust Lane Detection Using Real-Time Voting Processor”,
IEEE/IEEJ/JSAI International Conference on Intelligent
Transportation Systems, 5-8 Oct., 1999, pp.577–580.
31
Chang Gung University & National Taiwan University
Reference (2/4)
[5] A. Broggi, “Robust Real-Time Lane and Road Detection in
Critical Shadow Conditions,” Computer Vision, 1995,
Proceedings., International Symposium on, 21-23 Nov. 1995
pp.353-358
[6] Li, Q., Zheng, N., and Cheng, H.,“ Springrobot: A Prototype
Autonomous Vehicle and Its Algorithms for Lane Detection”,
IEEE Transactions on Intelligent Transportation System, vol.
5, no. 4, Dec., 2004, pp.300-308.
[7] Coulombe, J., Sawan, M. and Wang, C., “Variable Resolution
CMOS Current Mode Active Sensor”, IEEE International
Symposium on Circuits and Systems, vol. 2, 28-31 May,
2000, pp.293 – 296.
[8] Tabet, M., Hornsey, R.,“ CMOS Image Sensor Camera with
Focal Plane Edge Detection”, Canadian Conference on
Electrical and Computer Engineering, vol. 2, 13-16 May,
2001, pp.1129 – 1133.
32
Chang Gung University & National Taiwan University
Reference (3/4)
[9] Pei-Yung Hsiao, Yu-Chun Hsu, Wen-Ta Lee, Chia-Chun Tsai,
and Chia-Hao Lee, ”An Embedded Analog Spatial Filter Design
of The Current-Mode CMOS Image Sensor”, IEEE Transactions
on Consumer Electronics, vol. 50, no. 3, Aug., 2004, pp.945–
951.
[10] N. Yang and G. Jianhong, “ A 256x256 Pixel Smart CMOS
Image Sensor for Line Based Stereo Vision Applications”, IEEE
Journal of solid state circuits, vol. 35, no. 7, July, 2000,
pp.1055-1061
[11] Yuan, J. and Svensson, C., “High-speed CMOS circuit
technique,” IEEE J. Solid-state Circuits, vol. 24, no. 2, 1989, pp.
62-70.
[12] Byungsoo Chang, Joonbae Park and Wonchan Kim,“A 1.2 GHz
CMOS dual-modulus prescaler using new dynamic D-type flipflops”, IEEE Journal of Solid-State Circuits, vol. 31, no. 5, May,
1996. pp.749-752
33
Chang Gung University & National Taiwan University
Reference (4/4)
[13] Fish, A and Yadid-Pecht, o.,”CMOS current/voltage mode
winner-take-all circuit with spatial filtering”, The IEEE
International Symposium on Circuits and Systems, vol. 3, 6-9
May, 2001, pp.636-639.
[14] Nakamura, J., Pain, B., Nomoto, T., Nakamura, T. and
Fossum, E.R., “On-focal-plane signal processing for currentmode active pixel sensors”, IEEE Transactions on Electron
Devices, vol. 44, no. 10, Oct., 1997, pp.1747 – 1758
34