Transcript Multi-Frame_Reference_in_H.264

Multi-Frame Reference in
H.264/AVC
卓傳育
Outline
• Introduction to Multi-Frame Reference in
H.264/AVC
• Multi-Frame Reference Problem
• Two papers propose to multi-frame
reduction and selection.
• Conclusion
Basic Macroblock Coding Structure
Multiple Reference Frames
Multiple Reference Frames
Motion Compensation Accuracy
Intra Prediction
ANALYSIS AND REDUCTION OF REFERENCE
FRAMES FOR MOTION ESTIMATION IN
MPEG-4 AVC/JVT/H.264
Yu-Wen Huang1, Bing-Yu Hsieh1, Tu-Chih Wang1, Shao-Yi Chien1,
Shyh-Yih Ma2, Chun-Fu Shen2, and Liang-Gee Chen1
1. DSP/IC Design Lab., Department of Electrical Engineering,
National Taiwan University, [email protected]
2. Vivotek Incorporation, [email protected]
Searching steps of intra/inter prediction
with multiple reference frames
H.264
Selected modes,
inter/intra prediction
residues, motion vectors
Proposed
Statistics of Reference Frames
If we can detect that the transformed and quantized coefficients are very
close to zero in the first reference frame, we can turn off the matching
process from the rest frames.
Statistics of Reference Frames
MV compactness
Statistics of Average MV Compactness
If the optimal frame or mode of a MB does not change after ME from
5 frames, we classify these MBs as type I. Otherwise, we classify
them as type II.
If the MV compactness of a MB after ME from previous frame is very
small, we should stop searching the rest 4 frames.
Texture consideration
Summarize of the analysis
• If 16x16 mode is selected, the optimal reference frame
tend to be unchanged.
• If inter-modes with smaller blocks are selected,
searching more frames tend to be helpful.
• If MVs of larger blocks are similar to MVs of smaller
blocks, it is likely that no occlusion or uncovering occurs
in MB, so one reference frame may be enough.
• If MVs of larger blocks are more different from MVs of
smaller blocks, MB often crosses object boundaries and
thus requires more reference frames.
• If the texture of a MB is very complicated, it may require
more reference frames.
Proposed algorithm
Simulation results
Simulation results
Rate distortion curves of various sequences.
Average searched frames for various sequences
A Novel Approach To Fast Multi-Frame
Selection For H.264 Video Coding
Andy Chang, Oscar C. Au, Y. M. Yeung
Department of Electrical and Electronic Engineering
The Hong Kong University of Science and Technology
Email: [email protected], [email protected], [email protected]
Observation on Multi-frame motion
estimation
• Two types of temporal redundancy that
can be captured by using multi-frame but
not traditional single frame ME/MC.
– Short-term memory
• Ex. The blinking of an eye.
– Sub-pixel movement of textures
• textures and objects with different version of subpixel movement (“integer-pixel location”, “half-pixel
location” and “quarter-pixel location”) may occur in
successive video frames.
Examples of pixel movement
Pixel movement properties
• The optimal reference frame tends to be the
one with the same “sub-pixel location” as
the current frame.
• When there is more than one frame with the
same “sub-pixel location”, the one closer to
the current frame is usually better.
Selection of Frame for motion estimation
Proposed multi-frame selection process
• Each macroblock in each reference frame will
have a “sub-pixel location” which is calculated
and updated by adding the motion vector
obtained in the previous search with the
previous “sub-pixel location”.
• When two or more macroblocks have the same
“sub-pixel location”, only one frame is enabled
for motion estimation.
• The “sub-pixel location” of two macroblocks are
considered the same if both the x and y
component of “sub-pixel location” are equal.
Proposed multi-frame selection process
• The previous reference frame has the highest
probability of being selected.
• Motion estimation is always performed in
previous reference frame and the motion vector
obtained will be used to update the “sub-pixel
location” of the current macroblock.
• If we assume frame t-3 is chose to be the
reference frame for the black (current)
macroblock in frame t, the “sub-pixel location” of
the black macroblock in frame t is obtained by
adding the motion vector information to the “subpixel position” of black macroblock in frame t-3.
Flow Chart of the proposed algorithm
Experimental Results
QCIF(176x144), QP=16