Perspective-n-Point is the problem of estimating the pose of a calibrated camera given a set of n 3D points in the world and their corresponding 2D projections in the image. The camera pose consists of 6 degrees-of-freedom (DOF) which are made up of the rotation (roll, pitch, and yaw) and 3D translation of the camera with respect to the world. This problem originates from camera calibration and has many applications in computer vision and other areas, including 3D pose estimation, robotics and augmented reality. A commonly used solution to the problem exists for n = 3 called P3P, and many solutions are available for the general case of n ≥ 3. Implementations of these solutions are also available in open source software.
Contents
Definition
Given a set of n 3D points in a world reference frame and their corresponding 2D image projections as well as the calibrated intrinsic camera parameters, determine the 6 DOF pose of the camera in the form of its rotation and translation with respect to the world. This follows the perspective project model for cameras:
where
Assumptions and Data Characteristics
There are a few preliminary aspects of the problem that are common to all solutions of PnP. The assumption made in most solutions is that the camera is already calibrated. Thus, its intrinsic properties are already known, such as the focal length, principal image point, skew parameter, and other parameters. Some methods, such as UPnP. or the Direct Linear Transform (DLT) applied to the projection model, are exceptions to this assumption as they estimate these intrinsic parameters as well as the extrinsic parameters which make up the pose of the camera that the original PnP problem is trying to find.
For each solution to PnP, the chosen point correspondences cannot be coplanar. In addition, PnP can have multiple solutions, and choosing a particular solution would require post-processing of the solution set. Furthermore, using more point correspondences can reduce the impact of noisy data when solving PnP. RANSAC is also commonly used with a PnP method to make the solution robust to outliers in the set of point correspondences. Most methods, however, assume that the data is noise-free.
Methods
This following section describes two common methods that can be used to solve the PnP problem that are also readily available in open source software and how RANSAC can be used to deal with outliers in the data set.
P3P
When n = 3, the PnP problem is in its minimal form of P3P and can be solved with three point correspondences. However, with just three point correspondences, P3P yields many solutions, so a fourth correspondence is used in practice to remove ambiguity. The setup for the problem is as follows.
Let P be the center of projection for the camera, A, B, and C be 3D world points with corresponding images points u, v, and w. Let X = |PA|, Y = |PB|, Z = |PC|,
It's common to normalize the image points before solving P3P. Solving the P3P system results in four possible solutions for R and T. The fourth world point D and its corresponding image point z are then used to find the best solution among the four. The algorithm for solving the problem as well as the complete solution classification for it is given in the 2003 IEEE Transactions on Pattern Analysis and Machine Intelligence paper by Gao, et al. An open source implementation of the P3P can be found in OpenCV's calib3d module in the solvePnP function.
EPnP
Efficient PnP (EPnP) is a method developed by Lepetit, et al. in their 2008 International Journal of Computer Vision paper that solves the general problem of PnP for n ≥ 3. This method is based on the notion that each of the n points (which are called reference points) can be expressed as a weighted sum of four virtual control points. Thus, the coordinates of these control points become the unknowns of the problem. It is from these control points that the final pose of the camera is solved for.
As an overview of the process, first note that each of the n reference points in the world frame,
From this, the derivation of the image reference points becomes
The homogeneous image control point has the form
Using these two equations for each of the n reference points, the system
where
This solution has
Using RANSAC
PnP is prone to errors if there are outliers in the set of point correspondences. Thus, RANSAC can be used in conjunction with existing solutions to make the final solution for the camera pose more robust to outliers. An open source implementation of PnP methods with RANSAC can be found in OpenCV's Camera Calibration and 3D Reconstruction module in the solvePnPRansac function