Transcript stereo-last
Introduction • 3D scene flow is the 3D motion field of points in the world. Structure is the depth of the scene. • Motivation of our work: Numerous applications including intelligent robots, human-computer interfaces, surveillance systems, dynamic rendering, dynamic scene interpretation, etc. • Challenges: • Absence of correspondences, image noises, structure ambiguities, occlusion, etc. System Block Diagram Camera 1 Image Sequence 1 Optical Flow Stereo Constraints Camera 2 Image Sequence 2 Optical Flow 3D Affine Model 3D Scene Flow 3D Correspondences Dense Scene Structure Camera N Image Sequence N Optical Flow Regularization Constraints Multiple Camera Geometry • A set of cameras C0 , C1 ,, Cn1 provide N images. A 3D point P in the world can be transformed to m i the relation, point by mi J i Wi P Ti P • Normally, one pair is used as basic stereo pair. All cameras are pre-calibrated. • Given an image point and its disparity, we can back-project it to the 3D world. Local Motion Model Selection Camera i : Frame t +1 m t 1 Frame t mt t 1 Pm 3D affine motion M t t Pm Local Motion Model Selection • To avoid overfitting and ensure convergence in each local region, we can assume the motion in consecutive S frames is similar over time. The difference is only a scaling factor. Then, a1 t t a2 M a3 0 b1 b2 c1 c2 b3 0 c3 0 d1 d 2 , d3 1 t [ , S ). Motion Model Fitting • Eliminate translation unknowns to avoid trivial solutions. • For remaining unknowns in each local region: • Non-linear model fitting by using Levenberg-Marquardt (LM) algorithm. U*t arg(min( EOF (U t ))). Available Local Constraints Constraint Discussion • The EOF function is defined based on all the available constraints. – Optical flow constraints: • The projected 2D motion of 3D affine motion should be compatible with optical flow. – Stereo constraints: • The projected image location on different image planes of the same 3D scene point should have similar intensity patterns. Cross- correlation is used to measure this similarity. EOF Function • A 3D scene point is projected to different image planes of N cameras. The intensity patterns around the projective location should be similar. So, S 1 N-1 EOFstereo t mA i , j 0,i j t t Corel (Ti M t Pm , T j M t Pm ) big i, j EOF Function • The EOF in local model fitting can be denoted as, EOF EOFoptical flow w * EOFstereo LM algorithm is then used to minimize the EOF function. Regularization Constraints • To avoid overfitting, penalty constraint is added to large motion. C p min( zi 1 zi r ,0.0) This constraint is added to EOF function and used in every iteration. Initial Guesses • The unknown vector U t need to be initialized. By assuming small motion between two adjacent frames, we have a1 1, b1 0, c1 0, a2 0, b2 1, c2 0, a3 0, b3 0, c3 1, 1 2 S 2 1. • The initial structure (depth) value can be Complete Recursive Algorithm 1. Initialize unknown vector U t . Set flag : 0 . 2. If flag 0 , carry out affine model fitting in each local region using LM algorithm. Smoothness constraint is not used. Set flag : 1 ; Else, add smoothness constraint into EOF function, then carry out affine model fitting in each local region. 3. If regularization constraints are less than a threshold or maximum number of iteration has been exceeded, end the algorithm. Else go to 2. Integrated 3D Scene Flow and Structure Recovery Experiments on Synthetic Data Integrated 3D Scene Flow and Structure Recovered Motion Fields Integrated 3D Scene Flow and Structure Ground Truth Validation Integrated 3D Scene Flow and Structure Experiments on Real Data Integrated 3D Scene Flow and Structure Recovered Motion Fields Experimental Results of RuleBased Stereo Top View Right View Segmentation Map Left View Experimental Results of RuleBased Stereo Initial Sparse Disparity Map Result After Applied Rule 1 and 2 Experimental Results of RuleBased Stereo Result by Using A Direct Method Result by Using Our Method Experimental Results of RuleBased Stereo Occlusion Map Confidence Map Experimental Results of Sequential Formulation • Sample input images (only reference views are shown). Time t Time t+1 Experimental Results of Sequential Formulation • Disparity results. Reference View Disparity Result Experimental Results of Sequential Formulation • Scene flow results. X-y projection of scene flow z motion of scene flow Experimental Results of Integrated Formulation • Disparity results. Reference View Disparity Result Experimental Results of Integrated Formulation • Scene flow results. X-y projection of scene flow z motion of scene flow Scheme Overview Local motion analysis module Even Segmentation Global motion analysis module Local Nonrigid Motion Tracking Local Nonrigid Motion Tracking Global Regularization 2D Image Sequence Local Nonrigid Motion Tracking Global Constraints Structure Nonrigid motion 3D correspondences Local Affine Motion Model yi c1 c2 c3 0 d1 d2 . d3 1 P i 1 i M i P i , P i x i a1 a2 i M a3 0 b1 b2 b3 0 zi 1 , • Affine motion model assumed to remain the same for a short period of time; • A scaling factor, ,i is incorporated in order to compensate for possible temporal deviations. Local EOF Function Ic Frame (i+1) Frame (i) I (R) i 1 P i 1 Ii (M) (R) Pi EOF I i 1 I c • Levenberg-Marquardt method is used to perform the EOF minimization. • Unknowns include affine parameters and the scaling factors. Cloud Image Acquisition GOES-8 and GOES-9 are focused on clouds; GOES-9 provides one view at approximately every minute. GOES-8 provides one view at approximately every 15 minutes; Both GOES-8 and GOES-9 have five multi-spectral channels. Experiments • Experiments have been performed on the GOES image sequences of Hurricane Luis, start from 09-06-95 at 1023 UTC to 09-06-95 at 2226 UTC. Experiments (cont.) • Although the initial mean errors are very large, they decrease very quickly after the global fluid constraints are applied. Stable results are achieved at the end of the iterations. Experiments on Simulation Images Results Validation Experiments on Real Images Reconstruction Results Jeab Jeab_render Lin Lin_render Qian Qian_render Ye Ye_render Min_Tracking Results Validation Mean Error: 0.47006 Mean Error: 0.527872 Wave Tank Experiment Experimental Setup Stereoscopic camera used to record video sequences of ice forming in the CRREL wave tank. Camera details: 15 fps B/W images at 320x240 pixel resolution 12 cm baseline with 255 pixel focal length Camera mounted on platform ~0.8 m above surface Multiple film segments captured at various stages of ice formation Several marker types (buoys, sprinkles) placed on the surface at various times Wave Tank Results Experiments Performed Visualization via Anaglyphs • Ice Bucket – 3D images of small ice surfaces • Wave Tank - 3D images of ice in CRREL wave tank Analysis • Ice Bucket - Surface reconstruction of bench-top ice • Wave Tank - Surface reconstruction of ice in CRREL wave tank Visualizations Steps to Creating an Anaglyph 1. Separate the color channels (RGB) L image R image 2. For each pixel in the anaglyph: 1.Take the Red value from the left image 2. Take the Green and Blue values from the right image 3. View the constructed image with filtered glasses. R1 G1 B1 R2 G2 B2 R1 G2 B2 Anaglyph Visualizations Ice Bucket Anaglyphs Ice pieces in small bucket Camera ~0.4 m from surface Visualizations Wave Tank Anaglyphs Wave tank motion ● Surface mostly solid ● Frames pre-aligned ● Pre-study Examples With calibration balls Without calibration balls Stereo Analysis Ice Bucket Experiment Photographs taken in lab of ice in shallow bucket Ambient lighting Stereo camera Correspondences determined manually Matching points hand selected Determining matches in specular areas still difficult Stereo Results Nearest Neighbor Surface Depths calculated at given correspondence points All other points assigned the depth of nearest known point Stereo Results Thin Plate Spline Surface Depths calculated at given correspondence points All other points assigned the depth of nearest known point Current Results: Wave Tank Wave Tank Results Photographs taken at CRREL wave tank No special lighting used Camera mounted above tank, facing down Initial correspondences determined manually Matching points hand selected Tank walls and camera support provide context Current Results: Wave Tank Thin Plate Spline Surface Depths calculated at given correspondence points All other points interpolated from smoothing spline Stereo Analysis Algorithm Thin Plate Spline Surface With Iterative Warping 1. 2. 3. 4. 5. 6. 7. Manually determine a set of correspondences Generate disparity surface using thin plate splines Warp the left image to the right image via the disparity surface Fill in any gaps in warped image Obtain dense stereo between the right and warped left images Update the disparity surface from the calculated dense stereo Iterate back to step 3 until the two images converge Stereo Analysis Algorithm Thin Plate Spline Surface With Iterative Warping 1. Fit surface 2. Warp the left image to the right Stereo Analysis Algorithm Thin Plate Spline Surface With Iterative Warping Current Results: Wave Tank Visualizations Deformable Dual Mesh --application to stereo(cont.) A 3D array is formed by the correlation values between the stereo pair. (a) A stereo pair (b) Three cross sections of a 3D array filled with the correlation values (red represents higher correlation areas) • NM starts deforming from the camera-side end of the volume V • FM starts deforming from the far-side of the volume V Deformable Dual Mesh -- application to stereo(cont.) • Coarse to Fine Scheme: A coarsely initialized 3D array V. The blue plane shows the initial position of the near mesh and the red plane shows the initial position of the far mesh Deformable Dual Mesh -- application to stereo(cont.)