On Linear Structure from Motion for Light Field Cameras

Ole Johannsen, Antonin Sulc, and Bastian Goldluecke

Contribution

Novel scheme for linear, fast and accurate relative pose estimation for light field cameras.

  • First insight: 2D affine subspace representation of 3D points in the 4D light field.
  • Second insight: light field projection in Plücker ray coordinates is projective linear.
  • Both together: linear constraints on possible rotation and translation from feature matches.
Applications  
  • 3D reconstruction with light field cameras (plenoptic cameras, camera arrays ...)
  • Refocusable light field panoramas                                                                                                                

Light field feature correspondence

4D Light field parametrization  

A light field can be interpreted as a collection of views. Cameras lie in a common focal plane Π, images in the parallel image plane Ω at focal length f. A ray is parameterized by the intersections with the two planes Ω and Π, i.e. by a four-dimensional vr vector l = (u,v,s,t)^T. We show that all rays that intersect a common scene point form an affine 2D subspace of rays. Matching rays to subspaces for multiple detected features yields linear equations for relative pose estimation.

Light field feature matches

For 2D images: feature match given by pixel coordinates corresponding to the same scene point in both images.
For 4D light fields: a single match consists of a set of rays in (u, v, s, t)-coordinates in both light fields, which corresponds to detection of the same image feature in multiple subaperture views (s, t).

The light field projection

Projections of a single scene point

Consider a scene point X = [X, Y, Z]. From the pinhole perspective projection for a camera located in (s, t, 0), one obtains an affine relationship between (s, t) and (u, v) for all rays which intersect this point,

Note that M(X,f) can be estimated from a correspondence (1), which gives a 3D point cloud for each light field.

Plücker ray coordinate

A ray r given in the coordinates of one light field (red dots) will also intersect the two planes parameterizing another light field (green dots). Given a ray in Plücker coordinates, the projection into 4D light field coordinates is a projective linear map.

Plücker coordinates of a ray r are given as a homogenous vector (q:m) with direction q ∈ R3\{0} and moment m ∈ R3. A point X ∈ R3 lies on a ray iff m = X × q. A rigid motion (rotation R followed by translation t) leads to a linear coordinate change

where E := [t]× R is called the essential matrix.

Plücker ray to light field coordinates

A ray (q:m) in Plücker coordinates projects to light field coordinates (u, v, s, t) given by

Similar as for pinhole cameras, this is a projective linear map, whose matrix we call the light field projection P(f).