OpenPano: How to write a Panorama Stitcher

OpenPano: How to write a Panorama Stitcher

This is a summary of the algorithms I used to write OpenPano: an open source panorama stitcher. You can find the source code on github.

SIFT Feature

Lowe's SIFT [1] algorithm is implemented in feature/. The procedure of the algorithm and some results are briefly described in this section.

Scale Space & DOG Space

A Scale Space consisting of S×OS×O grey-scale images is built at the beginning. The original image is resized in OOdifferent sizes (AKA. octaves), and each is then Gaussian-blurred by SS different σσ. Since features are then detected on different resized version of the original image, features will then have scale-invariant properties.

The gaussian blur here is implemented by applying two 1-D convolutions, rather than a 2-D convolution. This speeds up the computation significantly.

In each octave, calculate the differences of every two adjacent blurred images, to build a Difference-of-Gaussian space. DOG Space consists of (S1)×O(S−1)×O grey images.

OpenPano: How to write a Panorama Stitcher

Extrema Detection

In DOG Space, detect all the minimum and maximum by comparing a pixel with its 26 neighbors in three directionsx,y,σx,y,σ.

OpenPano: How to write a Panorama Stitcher OpenPano: How to write a Panorama Stitcher

Then use parabolic interpolation to look for the accurate (x,y,σ)(x,y,σ) location of the extrema. Reject the points withlow contrast (by thresholding pixel value in DOG image) or on the edge (by thresholding principle curvature), to get more distinctive features. The results are like this:

OpenPano: How to write a Panorama Stitcher

Orientation Assignment

First, we calculate gradient and orientation for every point in the Scale space. For each keypoint detected by the previous procedure, the orientations of its neighbor points will be collected and used to build an orientation histogram, weighted by the magnitude of their gradients, together with a gaussian kernel centered at the keypoint. The peak in the histogram is chosen to be the major orientation of the keypoint, as shown by the arrows below:

OpenPano: How to write a Panorama Stitcher

Descriptor Representation

Lowe suggested [1] choosing 16 points around the keypoint, to build orientation histograms for each point and concatenate them as SIFT feature. Each histogram uses 8 different bins ranging from 0 to 360 degree. Therefore the result feature is a 128-dimensional floating point vector. Since the major orientation of the keypoint is known, by using relative orientation to the major one, this feature is rotation-invariant.

Also, I followed the suggestions from [2] to use RootSIFT, which is a simple modification on SIFT, and is considered more robust.

Feature Matching

Euclidean distance of the 128-dimensional descriptor is the distance measure for feature matching between two images. A match is considered not convincing and therefore rejected, if the distances from a point to its closest neighbor and second-closest neighbor are similar. A result of matching is shown:

OpenPano: How to write a Panorama Stitcher

Feature matching turned out to be the most time-consuming step. So I use FLANN library to query 2-nearest-neighbor among feature vectors. To calculate Euclidean distance of two vectors, Intel SSE intrinsics are used to speed up.

Transformation Estimation

Estimate from Match

It's well known [3] that for any two images taken from a camera at some fixed point, the homogeneous coordinates of matching points can be related by a homography matrix HH, such that for a pair of corresponding point p=(x,y,1),q=(u,v,1)p=(x,y,1),q=(u,v,1), we have

pHq=K1R1RT2K12qp∼Hq=K1R1R2TK2−1q
The homography matrix is a 3x3 matrix up to a scale ambiguity. It has the following two possible formulation:

Homography I: H=a11a21a31a12a22a32a13a231H=[a11a12a13a21a22a23a31a321]

Homography II: H=a11a21a31a12a22a32a13a23a33H=[a11a12a13a21a22a23a31a32a33] with constraint H=1‖H‖=1

When two images are taken with only camera translation and rotation aligned with the image plane (no perspective skew), then they can be related by an affine matrix of the form: A=a11a210a12a220a13a231A=[a11a12a13a21a22a23001]

Given a set of matches, a matrix of any of the above formulations can be estimated via Direct Linear Transform. These estimation methods are implemented in lib/imgproc.cc. See [3] for details.

However these methods are not robust to noise. In practice, due to the existence of false matches, RANSAC(Random Sample Consensus) algorithm [4] is used to estimate a transformation matrix from noisy data. In every RANSAC iteration, several matched pairs of points are randomly chosen to produce a best-fit transformation matrix, and the pairs which agree with the matrix are taken as inliers. After a number of iterations, the transform that has most number of inliers are taken as the final result. It's implemented in stitch/transform_estimate.cc.

Match Validation

After each matrix estimation, I performed a health check to the matrix, to avoid malformed transformation estimated from false matches. This includes two checks: (see stitch/homography.hh)

  • The skew factor in homography (H3,1,H3,2H3,1,H3,2) cannot be too large. Their absolute values are usually less than 0.002.

  • Flipping cannot happen in image stitching. A homography is rejected if it flips either xx or yy coordinate.

In unconstrained stitching, it's necessary to decide whether two images match or not. The number of inliers estimated above could be one criteria. However, for high-resolution images, it's common that you'll actually find false matches with a considerable number of inliers, as long as enough RANSAC iterations are performed. To solve this, note that false matches are usually more randomly distributed spatially, geometry constraints of matching can also help filter out bad matches. Therefore, after RANSAC finished and returned a set of inliers, some overlapping test can be used to further validate the matching, as suggested by [5].

Specifically, with a candidate transformation matrix, I could find out the region within one image covered by another image, by calculating the convex hull of the transformed corners. Two filters can then be applied after the overlapping region is found:

  • The overlapping area within two images shouldn't differ too much.
  • Within the region, a reasonably large portion of matches should be inliers. I also used the inlier ratio as the confidence score of this match.

Since false matches are likely to be random and irregular, this technique help filter out false matches with irregular geometry.

Cylindrical Panorama

Necessity of Projection

If a single-direction rotational input is given, as most panoramas are built, using planar homography leads tovertical distortion, as following:

OpenPano: How to write a Panorama Stitcher

This is because a panorama is essentially an image taken with a cylindrical or spherical lens, not a planar lens any more. Under this setting, the white circle on the grass around the camera (as in the above picture) is expected to become a straight line. A good demo revealing the reason of this projection can be seen at this page.

The way to handle this problem is to warp images by a cylindrical or spherical projection, before or after transform estimation. These are the two modes used in this system. In this section we will introduce the pipeline based on pre-warping, which is used in cylinder mode. This pipeline is generally good, although it has more assumptions on the data and requires some tricks to work well. It is implemented in stitch/cylstitcher.cc.

Warp

We can project the image to a cylinder surface at the beginning of stitching, by the following formula:

x=arctanxxcfy=yyc(xxc)2+f2{x′=arctan⁡x−xcfy′=y−yc(x−xc)2+f2
where ff is the focal length of the camera, and xc,ycxc,yc is the center of image. See stitch/warp.cc. Some example images after warping are given below. Note that this requires focal length to be known ahead of time, otherwise it won't produce a good warping result.

OpenPano: How to write a Panorama Stitcher OpenPano: How to write a Panorama Stitcher OpenPano: How to write a Panorama Stitcher

After projecting all images to the same cylinder surface, images can be simply stitched together by estimating pairwise affine transformation, and accumulating them with respect to an arbitrarily chosen identity image. Note that the white line on the ground is more straightened.

OpenPano: How to write a Panorama Stitcher

Straightening

The cylinder projection equations described above assumes horizontal cameras (i.e. only 1 DOF is allowed for the relative rotation between cameras). This assumption could lead to distortion, then the output panorama would be bended:

OpenPano: How to write a Panorama Stitcher

Choosing the middle image as the identity help with this problem. Another method I found to effectively reduce the effect is to searching for ycyc in the warping formula.

Since the effect is caused by camera tilt, ycyc could differ from height2height2. The algorithm works by changing ycyc by bisection, and find a value which leads to a most straight result. After this process, the result is like:

OpenPano: How to write a Panorama Stitcher

The change of ycyc alone is still not enough, since we are still warping images to a vertical cylinder. If all camera shares a tilt angle, the actual projection surface should be a cone. The error can be seen at the left and right image boundary above. To account for this error, the final trick I used is a perspective transform to align the left and right boundary:

OpenPano: How to write a Panorama Stitcher

Camera Estimation Mode

Intuition

Cylinder mode assumes a known focal length and requires all cameras to have the same "up" vector, to warp them on cylinder surface at the beginning. We want to get rid of these assumptions.

However, if you simply estimate pairwise homographies, directly stitch them together, and perform a cylindrical warp after the transformation (instead of before), you'll get something like this:

OpenPano: How to write a Panorama Stitcher

Notice that images indeed are well stitched together: matched points are overlapped very well. But the overall geometry structure is broken.

This is because HH is inconsistent with the true geometric constraints, as the estimated HH is unlikely to be of the form K1R1RT2K12K1R1R2TK2−1. In other words, we estimated HH as a 3 by 3 matrix up to a scale ambiguity, but not all matrix inR3×3R3×3 is a valid transformation matrix, although they might still match the points. Or to say, it's necessary that the transformation follows the formulation of HH, but not sufficient.

Also, we noticed that HH is not a minimal parameterization: assume all adjacent image pairs, as well as the (first,last) image pair are used to estimate nn homography matrix HH. Then we are estimating 8n8n parameters. However, each camera has 3 extrinsic parameters (for rotation) and 1 intrinsic parameter (the focal length). The total degree of freedom would be 4n4n. If all cameras have the same focal length the total DOF is even smaller.

This inconsistent over-parameterization of camera parameters can lead to the kind of overfitting results above, that breaks the underlying geometry structure.

People use HH to begin with, because the estimation is just easy linear algebra. But to get true estimation of KKRRof all cameras, gradient-based non-linear iterative optimization are necessary. In geometry vision it's called bundle adjustment, which is simply an initial estimation followed by iterative LM updates.

Initial Estimation

[6] gives a method to roughly estimate the focal length of all images at first. Given all focal length and the rotation matrix of one image, the rotation matrix of a connecting image can be estimated by using the homography we already have:

R1=K11H12K2R2R1=K1−1H12K2R2

Therefore, the estimation procedure goes like a max-spanning tree algorithm: after building a pairwise-matching graph, I first choose an image with good connection property to be identity. Then, in each step a best matching image (in terms of matching confidence) not processed yet is chosen and its rotation matrix RR can then be roughly estimated. Based on the rough estimation, all cameras added so far are then globally refined by bundle adjustment to be explained in the next section. This procedure is implemented in stitch/camera_estimate.cc.

Note that, since the homography H12H12 is an inconsistent parameterization, the result RR calculated from the above equation might not be a proper rotation matrix. Therefore I actually used the closest rotation matrix in terms of the Forbenius norm. Formally speaking, given H12,Ki,R2H12,Ki,R2, we compute:

R1=minR||RK11H12K2R2||2F,s.t.R1 is rotation matrixR1=minR||R−K1−1H12K2R2||F2,s.t.R1 is rotation matrix

If no bundle adjusment is performed, due to the error in the initial estimation of both KK and RR, the stitching result will misalign and looks like this:

OpenPano: How to write a Panorama Stitcher

To further refine the parameters in KK and RR such that matched points are aligned properly, optimization over the consistent parameterization is needed.

Bundle Adjustment

We setup a consistent parameterization of KK and RR in the following ways:

K=f000f0001,R=e[u]×K=(f000f0001),R=e[u]×
Here, R=e[u]×R=e[u]× is the angle-axis parameterization of rotation matrix. Let θθ be a vector of all parameters in the system. We aim to minimize the reprojection error of all matched point pairs:
θ=minθ|xipij|2θ⋆=minθ∑|xi−pij|2
where pijpij is the 2D projection of point xjxj in image jj to image ii, under KK and RR : p˜ijKiRiRTjK1jx˜jp~ij≡KiRiRjTKj−1x~j. The above sum is over all pairwise point matches found among all images added to the bundle adjuster so far. Note that when stitching an unordered collection of photos, one image can match a number of others. These matches all contribute to the bundle adjuster and help improve the quality.

Iterative methods such as Newton's method, gradient descent, and Levenberg-Marquardt algorithm can be used to solve the optimization. Assume matrix J=rθJ=∂r∂θ, where rr is the vector of all residual terms in the sum of square optimization: r=(xipij,)r=(xi−pij,⋯), then the three optimization methods can be formalized as follows:

  • Gradient descent: Δθ=λJTrΔθ=λJTr
  • Newton's method: Δθ=(JTJ)1JTrΔθ=(JTJ)−1JTr
  • LM algorithmΔθ=(JTJ+λD)1JTrΔθ=(JTJ+λD)−1JTr, where DD is diagonal.

All iterative methods involve calculating the matrix JJ. It could be done numerically or symbolically.

  • Numeric: For each parameter θiθi, mutate it by ±ϵ±ϵ and calculate r1,r2r1,r2 respectively. Then J[:,i]=r1r22ϵJ[:,i]=r1−r22ϵ

  • Symbolic: Calculate rθ∂r∂θ by chain rule. For example, some relevant formula are:

    rkpij=1,if rk=xipij.otherwise 0pijp˜ij=[x/zy/z][xyz]=[1/z001/zx/z2y/z2]p˜ijfi=KifiRiRTjK1jxjp˜ijui=KiRiuiRTjK1jxjp˜ijuj=KiRi(Rjuj)TK1jxjp˜ijfj=KiRiRTjK1jKjfjK1jxjKf=100010000Rux=ux[u]×+[u×(IR)ex]×||u||2R∂rk∂pij=−1,if rk=xi−pij.otherwise 0∂pij∂p~ij=∂[x/zy/z]∂[xyz]=[1/z0−x/z201/z−y/z2]∂p~ij∂fi=∂Ki∂fiRiRjTKj−1xj∂p~ij∂ui=Ki∂Ri∂uiRjTKj−1xj∂p~ij∂uj=KiRi(∂Rj∂uj)TKj−1xj∂p~ij∂fj=−KiRiRjTKj−1∂Kj∂fjKj−1xj∂K∂f=[100010000]∂R∂ux=ux[u]×+[u×(I−R)ex]×||u||2R

The last equation is from the result of [7]. Bye the way, it's interesting to point out that Lowe's paper [5] actually gives an incorrect formula:

Rux=uxe[u]×=e[u]×[u]×ux=R[u]×ux∂R∂ux=∂∂uxe[u]×=e[u]×∂[u]×∂ux=R∂[u]×∂ux
This doesn't hold because the notation eAeA is not element-wise exponential, but matrix exponential.

Both methods are implemented in stitch/incremental_bundle_adjuster.cc. Symbolic method is faster because it allows us to calculate only those non-zero elements in the very-sparse matrix JJ, and also allows us to calculate JJ as well as JTJJTJ together in one pass, without the need to do large matrix multiplication.

In a collection of nn images, optimization will be run n1n−1 times, when each image is added. Each optimization ended when the error doesn't decrease in the last 5 iterations.

After optimization, the above panorama looks much better:

OpenPano: How to write a Panorama Stitcher

Straightening

Straightening is necessary as well. As suggested by [5], the result of bundle adjustment can still have wavy effect, due to the tilt angle ambiguity. By assuming all cameras have their XX vectors lying on the same plane (which is reasonable), we can estimate a YY vector perpendicular to that plane to account for the tilt and fix the wavy effect.

See the following two images and notice the straight line on the grass is corrected (it is actually a circle in the center of a soccer field).

OpenPano: How to write a Panorama Stitcher OpenPano: How to write a Panorama Stitcher

Blending

The size of the final result is determined after having all the transformations. And the pixel value in the result image is calculated by an inverse transformation and bilinear interpolation with nearby pixels, in order to reduce alias effect.

For overlapped regions, the distance from the overlapped pixel to each image center is used to calculate a weighted sum of the pixel value. I only used the distance along the xx axis to calculate the weight, to get better result in panoramic image. The result is almost seamless (see images above).

Cropping

When the option CROP is given, the program simply manage to find the largest valid rectangular within the original result.

An O(n×m)O(n×m) algorithm is used, where n,mn,m are the height and width of the original result. The algorithm works like this:

For each row ii, the update

h[j]r[j]l[j]={0,h[j]+1,if i = 0 or A[i][j] is outside the areaotherwise=maxk[0,m)N:h[t]h[j],jtk=mink[0,m)N:h[t]h[j],ktjh[j]={0,if i = 0 or A[i][j] is outside the areah[j]+1,otherwiser[j]=maxk∈[0,m)∩N:h[t]≥h[j],∀j≤t≤kl[j]=mink∈[0,m)∩N:h[t]≥h[j],∀k≤t≤j
for every j[0,n)j∈[0,n) can be done in amortized O(1)O(1) time, and the maximum possible area for the first ii rows, would be (r[j]l[j]+1)×h[j](r[j]−l[j]+1)×h[j].

OpenPano: How to write a Panorama Stitcher OpenPano: How to write a Panorama Stitcher OpenPano: How to write a Panorama Stitcher

BTW, you can download these papers with my paper downloader:

pip2 install --user sopaper
while read -r p
do
  sopaper "$p"
done <<EOM
Distinctive Image Features From Scale-invariant Keypoints
Three Things Everyone Should Know to Improve Object Retrieval
Random Sample Consensus: a Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography
Automatic Panoramic Image Stitching Using Invariant Features
Construction of Panoramic Image Mosaics with Global and Local Alignment
A Compact Formula for the Derivative of a 3-D Rotation in Exponential Coordinates
EOM