Image registration given known gravity direction

2012-05-16T15:08:27Z

I am trying to integrate gravity measurements into the estimation of the planar homography relating two images of a planar scene. The cameras are calibrated; that is, a point $X$ in space is transformed into the two images according to

$$ x_1 = K_1(R_1 X+t_1) $$ $$ x_2 = K_2(R_2 X+t_2) $$ where $K_1$ and $K_2$ are known. Without using gravity, the problem looks like

$$ \min_H d(I_1, I_2, H) $$

where $I_1$ and $I_2$ are images and $d(\cdot)$ is some measure of the quality of $H$ as the relationship between them (e.g. pixel-wise SSD).

Adding gravity, we measure $g_1$ and $g_2$ in the respective camera coordinate frames and we have $$ {R_2}^T g_2 = {R_1}^T g_1 $$

Of course there is noise in the measurements so I would like to incorporate these measurements into the cost function, something like:

$$ \min_H d(I_1, I_2, H) + h(g_1, g_2, H) $$

where $h$ measures the agreement between $H$ and the gravity constraint above. There are a number of reasonable choices for $h$, for example: $$ h(g_1, g_2, H) = \| {R_2}^T g_2 - {R_1}^T g_1 \| $$

In general I'm not too concerned about the exact form of $h$, so long as it goes to zero as the constraint on $g_1$ and $g_2$ above becomes satisfied. Of course $h$ should be derived from a noise model for the sensor but I'm willing to settle for something "reasonable".

I could simply decompose $H$ in the standard way: $$ H = K_2 (R - \frac{tn^T}{d}) {K_1}^{-1} $$

and this would be fine for evaluating the cost function but not for taking the gradients needed for optimization.

Another approach would be to represent $H$ by its rotation and translation component, and then it would be easy to evaluate $h(\cdot)$, but this would break other parts of the system that rely on an existing parametrisation.

What I really want is a way to write $h(\cdot)$ as an analytic function of $H$, without assuming any particular parametrisation of $H$.

Image registration given known gravity direction - MathOverflow

Image registration given known gravity direction