Merge pull request #10358 from catree:add_tutorial_homography_panorama_stitching

doc/tutorials/features2d/homography/homography.markdown

@@ -1,8 +1,10 @@
 Basic concepts of the homography explained with code {#tutorial_homography}
-=============================
+====================================================
 
-Introduction
+@tableofcontents
-----
+
+Introduction {#tutorial_homography_Introduction}
+============
 
 This tutorial will demonstrate the basic concepts of the homography with some codes.
 For detailed explanations about the theory, please refer to a computer vision course or a computer vision book, e.g.:
@@ -13,10 +15,10 @@ For detailed explanations about the theory, please refer to a computer vision co
 The tutorial code can be found [here](https://github.com/opencv/opencv/tree/master/samples/cpp/tutorial_code/features2D/Homography).
 The images used in this tutorial can be found [here](https://github.com/opencv/opencv/tree/master/samples/data) (`left*.jpg`).
 
-Basic theory
+Basic theory {#tutorial_homography_Basic_theory}
-----
+------------
 
-### What is the homography matrix?
+### What is the homography matrix? {#tutorial_homography_What_is_the_homography_matrix}
 
 Briefly, the planar homography relates the transformation between two planes (up to a scale factor):
 
@@ -61,7 +63,7 @@ The following examples show different kinds of transformation but all relate a t
 
 ![](images/homography_transformation_example3.jpg)
 
-### How the homography transformation can be useful?
+### How the homography transformation can be useful? {#tutorial_homography_How_the_homography_transformation_can_be_useful}
 
 *   Camera pose estimation from coplanar points for augmented reality with marker for instance (see the previous first example)
 
@@ -75,10 +77,10 @@ The following examples show different kinds of transformation but all relate a t
 
 ![](images/homography_panorama_stitching.jpg)
 
-Demonstration codes
+Demonstration codes {#tutorial_homography_Demonstration_codes}
-----
+-------------------
 
-### Demo 1: Pose estimation from coplanar points
+### Demo 1: Pose estimation from coplanar points {#tutorial_homography_Demo1}
 
 \note Please note that the code to estimate the camera pose from the homography is an example and you should use instead @ref cv::solvePnP if you want to estimate the camera pose for a planar or an arbitrary object.
 
@@ -151,11 +153,16 @@ A quick solution to retrieve the pose from the homography matrix is (see \ref po
 
 This is a quick solution (see also \ref projective_transformations "2") as this does not ensure that the resulting rotation matrix will be orthogonal and the scale is estimated roughly by normalize the first column to 1.
 
+A solution to have a proper rotation matrix (with the properties of a rotation matrix) consists to apply a polar decomposition
+(see \ref polar_decomposition "6" or \ref polar_decomposition_svd "7" for some information):
+
+@snippet pose_from_homography.cpp polar-decomposition-of-the-rotation-matrix
+
 To check the result, the object frame projected into the image with the estimated camera pose is displayed:
 
 ![](images/homography_pose.jpg)
 
-### Demo 2: Perspective correction
+### Demo 2: Perspective correction {#tutorial_homography_Demo2}
 
 In this example, a source image will be transformed into a desired perspective view by computing the homography that maps the source points into the desired points.
 The following image shows the source image (left) and the chessboard view that we want to transform into the desired chessboard view (right).
@@ -186,7 +193,7 @@ To check the correctness of the calculation, the matching lines are displayed:
 
 ![](images/homography_perspective_correction_chessboard_matches.jpg)
 
-### Demo 3: Homography from the camera displacement
+### Demo 3: Homography from the camera displacement {#tutorial_homography_Demo3}
 
 The homography relates the transformation between two planes and it is possible to retrieve the corresponding camera displacement that allows to go from the first to the second plane view (see @cite Malis for more information).
 Before going into the details that allow to compute the homography from the camera displacement, some recalls about camera pose and homogeneous transformation.
@@ -363,7 +370,7 @@ The homography matrices are similar. If we compare the image 1 warped using both
 
 Visually, it is hard to distinguish a difference between the result image from the homography computed from the camera displacement and the one estimated with @ref cv::findHomography function.
 
-### Demo 4: Decompose the homography matrix
+### Demo 4: Decompose the homography matrix {#tutorial_homography_Demo4}
 
 OpenCV 3 contains the function @ref cv::decomposeHomographyMat which allows to decompose the homography matrix to a set of rotations, translations and plane normals.
 First we will decompose the homography matrix computed from the camera displacement:
@@ -457,11 +464,66 @@ plane normal at camera 1 pose: [0.1973513139420654, -0.6283451996579068, 0.75248
 
 Again, there is also a solution that matches with the computed camera displacement.
 
-Additional references
+### Demo 5: Basic panorama stitching from a rotating camera {#tutorial_homography_Demo5}
-----
+
+\note This example is made to illustrate the concept of image stitching based on a pure rotational motion of the camera and should not be used to stitch panorama images.
+The [stitching module](@ref stitching) provides a complete pipeline to stitch images.
+
+The homography transformation applies only for planar structure. But in the case of a rotating camera (pure rotation around the camera axis of projection, no translation), an arbitrary world can be considered
+([see previously](@ref tutorial_homography_What_is_the_homography_matrix)).
+
+The homography can then be computed using the rotation transformation and the camera intrinsic parameters as (see for instance \ref homography_course "8"):
+
+\f[
+  s
+  \begin{bmatrix}
+  x^{'} \\
+  y^{'} \\
+  1
+  \end{bmatrix} =
+  \bf{K} \hspace{0.1em} \bf{R} \hspace{0.1em} \bf{K}^{-1}
+  \begin{bmatrix}
+  x \\
+  y \\
+  1
+  \end{bmatrix}
+\f]
+
+To illustrate, we used Blender, a free and open-source 3D computer graphics software, to generate two camera views with only a rotation transformation between each other.
+More information about how to retrieve the camera intrinsic parameters and the `3x4` extrinsic matrix with respect to the world can be found in \ref answer_blender "9" (an additional transformation
+is needed to get the transformation between the camera and the object frames) with Blender.
+
+The figure below shows the two generated views of the Suzanne model, with only a rotation transformation:
+
+![](images/homography_stitch_compare.jpg)
+
+With the known associated camera poses and the intrinsic parameters, the relative rotation between the two views can be computed:
+
+@snippet panorama_stitching_rotating_camera.cpp extract-rotation
+
+@snippet panorama_stitching_rotating_camera.cpp compute-rotation-displacement
+
+Here, the second image will be stitched with respect to the first image. The homography can be calculated using the formula above:
+
+@snippet panorama_stitching_rotating_camera.cpp compute-homography
+
+The stitching is made simply with:
+
+@snippet panorama_stitching_rotating_camera.cpp stitch
+
+The resulting image is:
+
+![](images/homography_stitch_Suzanne.jpg)
+
+Additional references {#tutorial_homography_Additional_references}
+---------------------
 
 *   \anchor lecture_16 1. [Lecture 16: Planar Homographies](http://www.cse.psu.edu/~rtc12/CSE486/lecture16.pdf), Robert Collins
 *   \anchor projective_transformations 2. [2D projective transformations (homographies)](https://ags.cs.uni-kl.de/fileadmin/inf_ags/3dcv-ws11-12/3DCV_WS11-12_lec04.pdf), Christiano Gava, Gabriele Bleser
 *   \anchor szeliski 3. [Computer Vision: Algorithms and Applications](http://szeliski.org/Book/drafts/SzeliskiBook_20100903_draft.pdf), Richard Szeliski
 *   \anchor answer_dsp 4. [Step by Step Camera Pose Estimation for Visual Tracking and Planar Markers](https://dsp.stackexchange.com/a/2737)
 *   \anchor pose_ar 5. [Pose from homography estimation](https://team.inria.fr/lagadic/camera_localization/tutorial-pose-dlt-planar-opencv.html)
+*   \anchor polar_decomposition 6. [Polar Decomposition (in Continuum Mechanics)](http://www.continuummechanics.org/polardecomposition.html)
+*   \anchor polar_decomposition_svd 7. [A Personal Interview with the Singular Value Decomposition](https://web.stanford.edu/~gavish/documents/SVD_ans_you.pdf), Matan Gavish
+*   \anchor homography_course 8. [Homography](http://people.scs.carleton.ca/~c_shu/Courses/comp4900d/notes/homography.pdf), Dr. Gerhard Roth
+*   \anchor answer_blender 9. [3x4 camera matrix from blender camera](https://blender.stackexchange.com/a/38210)

modules/calib3d/include/opencv2/calib3d.hpp

@@ -1885,6 +1885,12 @@ CV_EXPORTS_W cv::Mat estimateAffinePartial2D(InputArray from, InputArray to, Out
                                   size_t maxIters = 2000, double confidence = 0.99,
                                   size_t refineIters = 10);
 
+/** @example decompose_homography.cpp
+  An example program with homography decomposition.
+
+  Check @ref tutorial_homography "the corresponding tutorial" for more details.
+ */
+
 /** @brief Decompose a homography matrix to rotation(s), translation(s) and plane normal(s).
 
 @param H The input homography matrix between two images.

samples/cpp/tutorial_code/features2D/Homography/panorama_stitching_rotating_camera.cpp

@@ -0,0 +1,88 @@
+#include <iostream>
+#include <opencv2/opencv.hpp>
+
+using namespace std;
+using namespace cv;
+
+namespace
+{
+void basicPanoramaStitching(const string &img1Path, const string &img2Path)
+{
+    Mat img1 = imread(img1Path);
+    Mat img2 = imread(img2Path);
+
+    //! [camera-pose-from-Blender-at-location-1]
+    Mat c1Mo = (Mat_<double>(4,4) << 0.9659258723258972, 0.2588190734386444, 0.0, 1.5529145002365112,
+                                     0.08852133899927139, -0.3303661346435547, -0.9396926164627075, -0.10281121730804443,
+                                     -0.24321036040782928, 0.9076734185218811, -0.342020183801651, 6.130080699920654,
+                                     0, 0, 0, 1);
+    //! [camera-pose-from-Blender-at-location-1]
+
+    //! [camera-pose-from-Blender-at-location-2]
+    Mat c2Mo = (Mat_<double>(4,4) << 0.9659258723258972, -0.2588190734386444, 0.0, -1.5529145002365112,
+                                     -0.08852133899927139, -0.3303661346435547, -0.9396926164627075, -0.10281121730804443,
+                                     0.24321036040782928, 0.9076734185218811, -0.342020183801651, 6.130080699920654,
+                                     0, 0, 0, 1);
+    //! [camera-pose-from-Blender-at-location-2]
+
+    //! [camera-intrinsics-from-Blender]
+    Mat cameraMatrix = (Mat_<double>(3,3) << 700.0, 0.0, 320.0,
+                                             0.0, 700.0, 240.0,
+                                             0, 0, 1);
+    //! [camera-intrinsics-from-Blender]
+
+    //! [extract-rotation]
+    Mat R1 = c1Mo(Range(0,3), Range(0,3));
+    Mat R2 = c2Mo(Range(0,3), Range(0,3));
+    //! [extract-rotation]
+
+    //! [compute-rotation-displacement]
+    //c1Mo * oMc2
+    Mat R_2to1 = R1*R2.t();
+    //! [compute-rotation-displacement]
+
+    //! [compute-homography]
+    Mat H = cameraMatrix * R_2to1 * cameraMatrix.inv();
+    H /= H.at<double>(2,2);
+    cout << "H:\n" << H << endl;
+    //! [compute-homography]
+
+    //! [stitch]
+    Mat img_stitch;
+    warpPerspective(img2, img_stitch, H, Size(img2.cols*2, img2.rows));
+    Mat half = img_stitch(Rect(0, 0, img1.cols, img1.rows));
+    img1.copyTo(half);
+    //! [stitch]
+
+    Mat img_compare;
+    Mat img_space = Mat::zeros(Size(50, img1.rows), CV_8UC3);
+    hconcat(img1, img_space, img_compare);
+    hconcat(img_compare, img2, img_compare);
+    imshow("Compare images", img_compare);
+
+    imshow("Panorama stitching", img_stitch);
+    waitKey();
+}
+
+const char* params
+    = "{ help h         |                              | print usage }"
+      "{ image1         | ../data/Blender_Suzanne1.jpg | path to the first Blender image }"
+      "{ image2         | ../data/Blender_Suzanne2.jpg | path to the second Blender image }";
+}
+
+int main(int argc, char *argv[])
+{
+    CommandLineParser parser(argc, argv, params);
+
+    if (parser.has("help"))
+    {
+        parser.about( "Code for homography tutorial.\n"
+                      "Example 5: basic panorama stitching from a rotating camera.\n" );
+        parser.printMessage();
+        return 0;
+    }
+
+    basicPanoramaStitching(parser.get<String>("image1"), parser.get<String>("image2"));
+
+    return 0;
+}

samples/cpp/tutorial_code/features2D/Homography/pose_from_homography.cpp

@@ -107,6 +107,14 @@ void poseEstimationFromCoplanarPoints(const string &imgPath, const string &intri
     }
     //! [pose-from-homography]
 
+    //! [polar-decomposition-of-the-rotation-matrix]
+    cout << "R (before polar decomposition):\n" << R << "\ndet(R): " << determinant(R) << endl;
+    Mat W, U, Vt;
+    SVDecomp(R, W, U, Vt);
+    R = U*Vt;
+    cout << "R (after polar decomposition):\n" << R << "\ndet(R): " << determinant(R) << endl;
+    //! [polar-decomposition-of-the-rotation-matrix]
+
     //! [display-pose]
     Mat rvec;
     Rodrigues(R, rvec);