BUNDLE BLOCK ADJUSTMENT WITH CONSTRAINED RELATIVE ORIENTATIONS

This paper deals with bundle adjustment with constrained cameras, i.e. where the orientation of certain cameras is expressed relatively to others, and these relative orientations are part of the unknowns. Despite the remarkable interest for oblique multi-camera systems, an empirical study on the effect of enforcing relative orientation constraints in bundle adjustment is still missing. We provide experimental evidence that indeed these constraints improve the accuracy of the results, while reducing the computational load as well. Moreover, we report for the first time in the literature the complete derivation of the Jacobian matrix for bundle adjustment with constrained cameras, to foster other implementations.


INTRODUCTION
Over the past two decades almost all existing companies in the geospatial sector have opened up for oblique imaging technology and included multi-head oblique camera units in their portfolios. Either as stand-alone solutions (Remondino, Gerke, 2015) or more recently in combination with a LiDAR unit (Toschi et al., 2019), these oblique systems have expanded the potentialities of the area-wide mapping market towards a more complete and intuitive scene understating concept. Indeed, they offer the advantages of a slanted view geometry, that allows the potential 3D reconstruction of building facades and other urban vertical objects (Haala, Rothermel, 2015). However, this comes at the cost of tackling new challenges, that mainly consist in dealing with large variations in image scale and illumination, multiple occlusions and an increased disparity search space. With regards to image block orientation, the main issue here is how to efficiently and rigorously cope with a larger number of unknowns, while minimizing the risk of divergence for the adjustment of the bundle of image rays.
Although the use of a constrained bundle block adjustment (BBA) in oblique acquisition scenarios represents an hot research topic, few of the software applications available to the public allows to enforce relative orientation constraints. As a result, most of the literature (Wiedemann, Moré, 2012, Rupnik et al., 2013, Rupnik et al., 2014 suggests that relative constraints should be enforced without being able to produce experimental evidence of that. (Cavegn et al., 2018) implement relative constraints in a terrestrial mobile mapping application, but do not assess this feature in isolation. They also report that Pix4D is able to enforce relative constraints, but no study in the literature evaluated it so far.
Constrained relative orientations are usually recovered as part of the calibration process of multi-camera rigs (Esquivel et al., 2007, Dai et al., 2009, Schneider, Förstner, 2013. The exterior orientation of the multi-camera system is then treated as a Non-Perspective-n-Point problem, where the camera rig is modeled * Corresponding author as a generalized camera (i.e., a camera in which the bundle of rays do not intersect in a single point), and solved exploiting the relative orientations estimated by calibration.
The only work, to the best of our knowledge, that analyses an implementation of BBA with relative orientation constraints and compares the results achieved with the custom BBA is (Sun et al., 2016), that however states that this approach has a lower accuracy on the basis of experiments that report worse reprojection errors than in the case where the constraints are ignored. It must be said, though, that looking at the reprojection error without considering the degrees of freedom of the model being fitted may lead to biased conclusions.
In this paper we will argue and provide evidence that exploiting the rigidity of the system has two benefits: it reduces the number of unknowns and it improves the accuracy. As a matter of fact, disregarding a physical constraint that actually holds, artificially increases the degrees of freedom of the system, leading to over-fitting (i.e., fitting the noise).
A further contribution of this paper is to work out in full details the formula for the Jacobian matrix of BBA with constrained cameras, which was missing in the literature so far. We will use the "matrix differential calculus" formalism (Magnus, Neudecker, 1999), which allows a compact and modular derivation.

PROBLEM STATEMENT
Let us consider an oblique multi-camera system composed by k cameras, where one is taken as the reference and the remaining k − 1 have a fixed but unknown relative orientation with respect to the first one. For the sake of concreteness, one can think of the customary "Maltese cross" arrangement ( Fig. 1), that comprises a single nadir-pointing camera and four obliquepointing cameras (k = 5). Two of the oblique cameras point in opposite directions cross-track, while the remaining two oblique cameras point in opposite directions along-track. The five cameras are housed in a suitable frame that ensures rigidity and ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume V-2-2020Volume V-2- , 2020 XXIV ISPRS Congress (2020 edition) expressed relatively to others, and these relative orientations are part of the unknowns.

. INTRODUCTION
() criticise enforcing relative orientation mera systems, reporting worse re-projection where the constraints are ignored in Bunt (BBA). This is in contrast with common at a physical constraint that actually hold arded in an estimation problem, for this he degrees of freedom of the system and i.e., fitting the noise). We will show that e, if error is measured correctly. As a mate-projection error without considering the or evaluating the error on the same set of s lead to biased conclusions. In short, we ientation constraints must always be used s.
of this paper is to work out in full details acobian matrix of BBA with constrained missing in the literature so far. We will rential calculus" formalism (Magnus and ich allows a compact derivation and leads pressive formulation. Compare also the tion matrix to those reported in (Lucas, r is a geometric error corresponding to the en a projected point and a measured one.

ROBLEM STATEMENT
ticamera system composed by n cameras, the reference and the remaining n 1 have relative orientation with respect to the first oncreteness, one can think of the customrrangement, that comprises a single nadirfour oblique-pointing cameras. Two of the t in opposite directions cross-track, while oblique cameras point in opposite direche five cameras are housed in a suitable dity and their shutters are synchronized to ly. or • let Iu be one nadiral camera, whose exterior orientation is Gu = [Ru|tu]; • let Iv be one of the (four) oblique images relative to Iu, whose exterior orientation is Gv = [Rv|tv]; • let Gk = [Rk|tk] be the orientation of the oblique image relative to the nadiral image; in the case of the Maltese cross there are four such relative orientations: k = {N, S, E, W } which are fixed but unknown; • the exterior orientation of the oblique image Gv can be written as a function of those of the nadiral image and the relative orientation: (1) • let us parametrize the nadiral orientation Gu with the 6 parameters g > u = [!, , , tu], where the first three are the Euler angles; • let us parametrize the relative orientation Gk with the 6 parameters g > k = [↵, , , tk] where the first three are the Euler angles; Exploiting the rigidity of the system has two benefits: • first it reduces the number of unknowns; let m be the number of nadiral images (the total number of images is 5m).
If the oblique images are adjusted as if their orientations were independent the system has 30m + 3n unknowns, whereas, by enforcing the rigidity, the unknowns reduces to 6(m + 4) + 3n (4 is the number of relative orientations); • second, it improves the accuracy, as it enforces a physical constraint that is actually in place. We will show experimental evidence of this in Sec. 4

JACOBIAN MATRIX
The primary structure of the Jacobian of the classical BBA is composed of blocks of two types, which we will call A and B that contain respectively the derivatives of f wrt image orientation and wrt 3D point. In the case that we are examining in this paper, only the nadiral images give rise to this type of blocks, whereas oblique images (whose orientation is linked to the nadiral image) lead to three new blocks: est that relative orientation constraints should not be used when bundle adjusting blocks with oblique eed this is not the case, and provide experimental evidence for this. Moreover, we report, for the first mplete derivation of the Jacobian for bundle adjustment with constrained cameras, where the orientation sed relatively to others, and these relative orientations are part of the unknowns.

RODUCTION
ticise enforcing relative orientation ystems, reporting worse re-projection e the constraints are ignored in Bun-). This is in contrast with common ysical constraint that actually hold in an estimation problem, for this rees of freedom of the system and tting the noise). We will show that ror is measured correctly. As a matection error without considering the luating the error on the same set of to biased conclusions. In short, we on constraints must always be used s paper is to work out in full details n matrix of BBA with constrained g in the literature so far. We will calculus" formalism (Magnus and lows a compact derivation and leads ve formulation. Compare also the atrix to those reported in (Lucas, eometric error corresponding to the rojected point and a measured one.

EM STATEMENT
ra system composed by n cameras, erence and the remaining n 1 have e orientation with respect to the first teness, one can think of the customment, that comprises a single nadirlique-pointing cameras. Two of the posite directions cross-track, while ue cameras point in opposite direccameras are housed in a suitable d their shutters are synchronized to • let Iu be one nadiral camera, whose exterior orientation is Gu = [Ru|tu]; • let Iv be one of the (four) oblique images relative to Iu, whose exterior orientation is Gv = [Rv|tv]; • let Gk = [Rk|tk] be the orientation of the oblique image relative to the nadiral image; in the case of the Maltese cross there are four such relative orientations: k = {N, S, E, W } which are fixed but unknown; • the exterior orientation of the oblique image Gv can be written as a function of those of the nadiral image and the relative orientation: (1) • let us parametrize the nadiral orientation Gu with the 6 parameters g > u = [!, , , tu], where the first three are the Euler angles; • let us parametrize the relative orientation Gk with the 6 parameters g > k = [↵, , , tk] where the first three are the Euler angles; Exploiting the rigidity of the system has two benefits: • first it reduces the number of unknowns; let m be the number of nadiral images (the total number of images is 5m).
If the oblique images are adjusted as if their orientations were independent the system has 30m + 3n unknowns, whereas, by enforcing the rigidity, the unknowns reduces to 6(m +4)+3n (4 is the number of relative orientations);

INTRODUCTION
) criticise enforcing relative orientation era systems, reporting worse re-projection here the constraints are ignored in Bun-BBA). This is in contrast with common a physical constraint that actually hold ded in an estimation problem, for this degrees of freedom of the system and ., fitting the noise). We will show that if error is measured correctly. As a matprojection error without considering the evaluating the error on the same set of ead to biased conclusions. In short, we tation constraints must always be used .
f this paper is to work out in full details obian matrix of BBA with constrained ssing in the literature so far. We will ntial calculus" formalism (Magnus and h allows a compact derivation and leads essive formulation. Compare also the on matrix to those reported in (Lucas, s a geometric error corresponding to the a projected point and a measured one.

BLEM STATEMENT
amera system composed by n cameras, e reference and the remaining n 1 have ative orientation with respect to the first creteness, one can think of the customngement, that comprises a single nadirr oblique-pointing cameras. Two of the n opposite directions cross-track, while blique cameras point in opposite direcfive cameras are housed in a suitable ty and their shutters are synchronized to • let Iu be one nadiral camera, whose exterior orientation is Gu = [Ru|tu]; • let Iv be one of the (four) oblique images relative to Iu, whose exterior orientation is Gv = [Rv|tv]; • let Gk = [Rk|tk] be the orientation of the oblique image relative to the nadiral image; in the case of the Maltese cross there are four such relative orientations: k = {N, S, E, W } which are fixed but unknown; • the exterior orientation of the oblique image Gv can be written as a function of those of the nadiral image and the relative orientation: (1) • let us parametrize the nadiral orientation Gu with the 6 parameters g > u = [!, , , tu], where the first three are the Euler angles; • let us parametrize the relative orientation Gk with the 6 parameters g > k = [↵, , , tk] where the first three are the Euler angles; Exploiting the rigidity of the system has two benefits: • first it reduces the number of unknowns; let m be the number of nadiral images (the total number of images is 5m).
If the oblique images are adjusted as if their orientations were independent the system has 30m + 3n unknowns, whereas, by enforcing the rigidity, the unknowns reduces to 6(m + 4) + 3n (4 is the number of relative orientations); • second, it improves the accuracy, as it enforces a physical constraint that is actually in place. We will show experimental evidence of this in Sec. 4

JACOBIAN MATRIX
The primary structure of the Jacobian of the classical BBA is composed of blocks of two types, which we will call A and B that contain respectively the derivatives of f wrt image orientation and wrt 3D point. In the case that we are examining in this paper, only the nadiral images give rise to this type of blocks, whereas oblique images (whose orientation is linked to the nadiral image) lead to three new blocks: d this is not the case, and provide experimental evidence for this. Moreover, we report, for the first plete derivation of the Jacobian for bundle adjustment with constrained cameras, where the orientation d relatively to others, and these relative orientations are part of the unknowns.
ODUCTION cise enforcing relative orientation stems, reporting worse re-projection the constraints are ignored in Bun-. This is in contrast with common sical constraint that actually hold an estimation problem, for this es of freedom of the system and ng the noise). We will show that r is measured correctly. As a mattion error without considering the ating the error on the same set of biased conclusions. In short, we constraints must always be used paper is to work out in full details matrix of BBA with constrained in the literature so far. We will alculus" formalism (Magnus and ws a compact derivation and leads formulation. Compare also the trix to those reported in (Lucas, ometric error corresponding to the jected point and a measured one.

M STATEMENT
a system composed by n cameras, rence and the remaining n 1 have orientation with respect to the first ness, one can think of the customent, that comprises a single nadirique-pointing cameras. Two of the osite directions cross-track, while cameras point in opposite direccameras are housed in a suitable their shutters are synchronized to • let Iu be one nadiral camera, whose exterior orientation is Gu = [Ru|tu]; • let Iv be one of the (four) oblique images relative to Iu, whose exterior orientation is Gv = [Rv|tv]; • let Gk = [Rk|tk] be the orientation of the oblique image relative to the nadiral image; in the case of the Maltese cross there are four such relative orientations: k = {N, S, E, W } which are fixed but unknown; • the exterior orientation of the oblique image Gv can be written as a function of those of the nadiral image and the relative orientation: (1) • let us parametrize the nadiral orientation Gu with the 6 parameters g > u = [!, , , tu], where the first three are the Euler angles; • let us parametrize the relative orientation Gk with the 6 parameters g > k = [↵, , , tk] where the first three are the Euler angles; Exploiting the rigidity of the system has two benefits: • first it reduces the number of unknowns; let m be the number of nadiral images (the total number of images is 5m).
If the oblique images are adjusted as if their orientations were independent the system has 30m + 3n unknowns, whereas, by enforcing the rigidity, the unknowns reduces to 6(m + 4) + 3n (4 is the number of relative orientations); • second, it improves the accuracy, as it enforces a physical constraint that is actually in place. We will show experimental evidence of this in Sec. 4

JACOBIAN MATRIX
The primary structure of the Jacobian of the classical BBA is composed of blocks of two types, which we will call A and B that contain respectively the derivatives of f wrt image orientation and wrt 3D point. In the case that we are examining in this paper, only the nadiral images give rise to this type of blocks, whereas oblique images (whose orientation is linked to the nadiral image) lead to three new blocks: the remaining pair of oblique cameras point in opposite direc-tions along-track. The five cameras are housed in a suitable frame that ensure rigidity and their shutters are synchronized to operate simultaneously.
• let I ⌫ be one nadiral camera, whose exterior orientation is ⇤ Corresponding author erivative of the rotation matrix to those reported in (L 963).
he reprojection error is a geometric error corresponding mage distance between a projected point and a measured 2. PROBLEM STATEMENT et us consider a multicamera system composed by n cam here one is taken as the reference and the remaining n 1 fixed but unknown relative orientation with respect to th ne. For the sake of concreteness, one can think of the cu ry "Maltese cross" arrangement, that comprises a single ointing camera and four oblique-pointing cameras. Two blique cameras point in opposite directions cross-track, he remaining pair of oblique cameras point in opposite ions along-track. The five cameras are housed in a su rame that ensure rigidity and their shutters are synchroni perate simultaneously.
⇤ Corresponding author is a geometric error corresponding to the n a projected point and a measured one.

OBLEM STATEMENT
icamera system composed by n cameras, he reference and the remaining n 1 have lative orientation with respect to the first ncreteness, one can think of the customrangement, that comprises a single nadirur oblique-pointing cameras. Two of the in opposite directions cross-track, while oblique cameras point in opposite direce five cameras are housed in a suitable ity and their shutters are synchronized to . r • first it ber of If the were where to 6(m • second constr menta The primar composed B that con entation an in this pap blocks, wh the nadiral Corresponding author J B enta in t bloc the • tive orientation nvolving multirors than in the in contrast with a physical conthe degrees of (i.e., fitting the y of the system unknowns and measured. As or without conthe error on the ed conclusions. n Sec. 4 t in full details ith constrained ar. We will use which allows a d by n cameras, ining n 1 have pect to the first of the customs a single nadirras. Two of the ss-track, while opposite direcd in a suitable ynchronized to r orientation is ing worse re-projection ts are ignored in Bunontrast with common int that actually hold ion problem, for this m of the system and ). We will show that d correctly. As a matithout considering the or on the same set of lusions. In short, we must always be used ork out in full details BA with constrained ture so far. We will malism (Magnus and t derivation and leads . Compare also the e reported in (Lucas, r corresponding to the and a measured one.
• let I v be one of the (fou whose exterior orientatio Exploiting the rigidity of the s • first it reduces the numbe ber of nadiral images (th If the oblique images are were independent the sy whereas, by enforcing th be the orientation of the o age relative to the nadiral image; in the case o tese cross there are four such relative orienta {N, S, E, W } which are fixed but unknown; • the exterior orientation of the oblique image written as a function of those of the nadiral im relative orientation: where the first th Euler angles; • let us parametrize the relative orientation G k parameters g > k = [↵, , , t k ] where the first th Euler angles; Exploiting the rigidity of the system has two benefit • first it reduces the number of unknowns; let m b ber of nadiral images (the total number of ima If the oblique images are adjusted as if their o were independent the system has 30m + 3n whereas, by enforcing the rigidity, the unknow to 6(m + 4) + 3n (4 is the number of relative or • second, it improves the accuracy, as it enforces sing the re-projection error without considering the reedom, or evaluating the error on the same set of rol points lead to biased conclusions. In short, we elative orientation constraints must always be used era systems.
tribution of this paper is to work out in full details for the Jacobian matrix of BBA with constrained ich was missing in the literature so far. We will trix differential calculus" formalism (Magnus and 1999), which allows a compact derivation and leads le and expressive formulation. Compare also the f the rotation matrix to those reported in (Lucas, ction error is a geometric error corresponding to the ce between a projected point and a measured one.

PROBLEM STATEMENT
der a multicamera system composed by n cameras, taken as the reference and the remaining n 1 have nknown relative orientation with respect to the first sake of concreteness, one can think of the custome cross" arrangement, that comprises a single nadirera and four oblique-pointing cameras. Two of the eras point in opposite directions cross-track, while g pair of oblique cameras point in opposite directrack. The five cameras are housed in a suitable nsure rigidity and their shutters are synchronized to ultaneously.
nding author relative orientation: where the firs Euler angles; • let us parametrize the relative orientation G parameters g > k = [↵, , , t k ] where the first Euler angles; Exploiting the rigidity of the system has two bene • first it reduces the number of unknowns; let m ber of nadiral images (the total number of im If the oblique images are adjusted as if thei were independent the system has 30m + 3 whereas, by enforcing the rigidity, the unkn to 6(m + 4) + 3n (4 is the number of relative • second, it improves the accuracy, as it enforc constraint that is actually in place. We will mental evidence of this in Sec. 4

JACOBIAN MATRIX
The primary structure of the Jacobian of the clas composed of blocks of two types, which we wi B that contain respectively the derivatives of f w entation and wrt 3D point. In the case that we a in this paper, only the nadiral images give rise to blocks, whereas oblique images (whose orientatio the nadiral image) lead to three new blocks: om, or evaluating the error on the same set of points lead to biased conclusions. In short, we ve orientation constraints must always be used systems.
ution of this paper is to work out in full details the Jacobian matrix of BBA with constrained was missing in the literature so far. We will differential calculus" formalism (Magnus and 9), which allows a compact derivation and leads nd expressive formulation. Compare also the e rotation matrix to those reported in (Lucas, error is a geometric error corresponding to the between a projected point and a measured one.
. PROBLEM STATEMENT a multicamera system composed by n cameras, en as the reference and the remaining n 1 have own relative orientation with respect to the first e of concreteness, one can think of the customss" arrangement, that comprises a single nadirand four oblique-pointing cameras. Two of the point in opposite directions cross-track, while air of oblique cameras point in opposite direck. The five cameras are housed in a suitable e rigidity and their shutters are synchronized to eously. g author (1) • let us parametrize the nadiral orientation G u with the 6 parameters g > u = [!, , , t u ], where the first three are the Euler angles; • let us parametrize the relative orientation G k with the 6 parameters g > k = [↵, , , t k ] where the first three are the Euler angles; Exploiting the rigidity of the system has two benefits: • first it reduces the number of unknowns; let m be the number of nadiral images (the total number of images is 5m).
If the oblique images are adjusted as if their orientations were independent the system has 30m + 3n unknowns, whereas, by enforcing the rigidity, the unknowns reduces to 6(m + 4) + 3n (4 is the number of relative orientations); • second, it improves the accuracy, as it enforces a physical constraint that is actually in place. We will show experimental evidence of this in Sec. 4

JACOBIAN MATRIX
The primary structure of the Jacobian of the classical BBA is composed of blocks of two types, which we will call A and B that contain respectively the derivatives of f wrt image orientation and wrt 3D point. In the case that we are examining in this paper, only the nadiral images give rise to this type of blocks, whereas oblique images (whose orientation is linked to the nadiral image) lead to three new blocks: tive orientation constraints must always be used a systems.
ibution of this paper is to work out in full details r the Jacobian matrix of BBA with constrained was missing in the literature so far. We will x differential calculus" formalism (Magnus and 99), which allows a compact derivation and leads and expressive formulation. Compare also the the rotation matrix to those reported in (Lucas, on error is a geometric error corresponding to the e between a projected point and a measured one.
2. PROBLEM STATEMENT r a multicamera system composed by n cameras, ken as the reference and the remaining n 1 have nown relative orientation with respect to the first ke of concreteness, one can think of the customross" arrangement, that comprises a single nadirra and four oblique-pointing cameras. Two of the as point in opposite directions cross-track, while pair of oblique cameras point in opposite direcck. The five cameras are housed in a suitable ure rigidity and their shutters are synchronized to aneously.
ing author • let us parametrize the nadiral orientation G u parameters g > u = [!, , , t u ], where the first Euler angles; • let us parametrize the relative orientation G k parameters g > k = [↵, , , t k ] where the first t Euler angles; Exploiting the rigidity of the system has two benefi • first it reduces the number of unknowns; let m ber of nadiral images (the total number of ima If the oblique images are adjusted as if their were independent the system has 30m + 3n whereas, by enforcing the rigidity, the unknow to 6(m + 4) + 3n (4 is the number of relative o • second, it improves the accuracy, as it enforce constraint that is actually in place. We will s mental evidence of this in Sec. 4

JACOBIAN MATRIX
The primary structure of the Jacobian of the class composed of blocks of two types, which we will B that contain respectively the derivatives of f wr entation and wrt 3D point. In the case that we are in this paper, only the nadiral images give rise to blocks, whereas oblique images (whose orientation the nadiral image) lead to three new blocks: (i.e., fitting the noise). We will show that se, if error is measured correctly. As a matre-projection error without considering the or evaluating the error on the same set of ts lead to biased conclusions. In short, we rientation constraints must always be used ms.
of this paper is to work out in full details Jacobian matrix of BBA with constrained missing in the literature so far. We will erential calculus" formalism (Magnus and hich allows a compact derivation and leads xpressive formulation. Compare also the ation matrix to those reported in (Lucas, r is a geometric error corresponding to the een a projected point and a measured one.

ROBLEM STATEMENT
lticamera system composed by n cameras, the reference and the remaining n 1 have relative orientation with respect to the first concreteness, one can think of the customarrangement, that comprises a single nadirfour oblique-pointing cameras. Two of the nt in opposite directions cross-track, while f oblique cameras point in opposite direche five cameras are housed in a suitable idity and their shutters are synchronized to sly. or • the exterior orientation of the oblique image written as a function of those of the nadiral im relative orientation: • let us parametrize the nadiral orientation G u parameters g > u = [!, , , t u ], where the first t Euler angles; • let us parametrize the relative orientation G k parameters g > k = [↵, , , t k ] where the first th Euler angles; Exploiting the rigidity of the system has two benefit • first it reduces the number of unknowns; let m b ber of nadiral images (the total number of ima If the oblique images are adjusted as if their o were independent the system has 30m + 3n whereas, by enforcing the rigidity, the unknow to 6(m + 4) + 3n (4 is the number of relative or • second, it improves the accuracy, as it enforces constraint that is actually in place. We will sh mental evidence of this in Sec. 4

JACOBIAN MATRIX
The primary structure of the Jacobian of the classi composed of blocks of two types, which we will B that contain respectively the derivatives of f wrt entation and wrt 3D point. In the case that we are in this paper, only the nadiral images give rise to t blocks, whereas oblique images (whose orientation the nadiral image) lead to three new blocks: Figure 1. Illustration of the Maltese cross example. In this case ⌫ = E and so G µ = G E G ⌫ • let I µ be one of the (four) oblique images relative to I ⌫ , whose exterior orientation is G µ = [R µ |t µ ]; • let G ⌧ = [R ⌧ |t ⌧ ] be the orientation of the oblique image relative to the nadiral image; in the case of the Maltese cross there are four such relative orientations: ⌧ = {N,S,E,W } which are fixed but unknown; • the exterior orientation of the oblique image G µ can be written as a function of those of the nadiral image and the relative orientation: • let us parametrize the nadiral orientation G ⌫ with the 6 parameters g > ⌫ = [!, ', , t ⌫ ], where the first three are the Euler angles that represent the rotation (see App. B); • let us parametrize the relative orientation G ⌧ with the 6 parameters g > ⌧ = [↵, , , t ⌫ ] where the first three are the Euler angles.

JACOBIAN MATRIX
The primary structure of the Jacobian of the classical BBA is composed of blocks of two types, which we will call J A and J B that contain respectively the derivatives of f wrt image orientation and wrt 3D point. In the case that we are examining in this paper, only the nadiral images give rise to this type of blocks, whereas oblique images (whose orientation is linked to the nadiral image) lead to three new blocks: • J C collects the derivative of the residual in the oblique images wrt the nadiral orientation their shutters are synchronized to operate simultaneously 1 .
• Let Iν be one nadir image, whose exterior orientation is Gν = [Rν |tν ] (Gν transforms 3D point coordinates from an external (object) reference frame to the camera reference, whose origin is at the centre of projection); • let Iµ be one of the (four) oblique images relative to Iν , whose exterior orientation is Gµ = [Rµ|tµ]; • let Gτ = [Rτ |tτ ] be the orientation of the oblique image relative to the nadir image (Gτ transform points from the nadir camera coordinate system to the oblique one); in the case of the Maltese cross, there are four of such relative orientations: τ = {N,S,E,W } which are fixed but unknown; • the exterior orientation of the oblique image Gµ can be written as a function of those of the nadir image and the relative orientation: • let us parametrize the nadiral orientation Gν with the 6 parameters g ν = [ω, ϕ, κ, tν ], where the first three are the Euler angles that represent the rotation (see App. B); • let us parametrize the relative orientation Gτ with the 6 parameters g τ = [α, β, γ, tτ ] where the first three are the Euler angles; • let us write the collinearity equations in compact form as: where X are the coordinates of an object point, x are the two coordinates of the corresponding image point, g = [ω, ϕ, κ, t] represent the exterior orientation of the image, and the central projection is accomplished by: • Interior orientation is assumed known and fixed (however, the estimate of the interior parameters, possibly including distortion parameters, could be added in the process).

JACOBIAN MATRIX OF CRO-BBA
Performing BBA using any variant of the Gauss-Newton method entails computing the Jacobian matrix that collects the partial derivatives of f .
As for the blocks related to oblique images: Let X 0 = R⌧ R⌫ X + R⌧ t⌫ + t⌧ , then and also @f (X, gµ) Now: and @f (X, gµ) well.
X 1 X 2 X 3 I1 I2 I3 I4 I5 I6 I7 I8 I9 I10 @g1 @g1 @gN @gS @gE @gW Blocks JA and JB correspond to nadiral images, and they are easily derived: As for the blocks related to oblique images: and also @f (X, gµ) Now: and @f (X, gµ) It is worth noting that these formulae are modular with respect to the parametrization chosen for R: changing it will cause only DR to change. Interior orientation can be easily catered for as well.
X 1 X 2 X 3   I1 I2 I3 I4 I5 I6 I7 I8 I9 I10 @g1 @g1 @gN @gS @gE @gW for the blocks related to oblique images: also @f (X, gµ) w: It is worth noting that these formulae are modular with respect to the parametrization chosen for R: changing it will cause only DR to change. Interior orientation can be easily catered for as well.
Let us write the collinearity equations in compact form as where X are the coordinates of an object point, x are the two coordinates of the corresponding image point, g = [!, ', , t] > represent the exterior orientation of the image, and the central projection is accomplished by Performing BBA using any variant of the Gauss-Newton method entails computing the Jacobian matrix that collects the partial derivatives of f (henceforth referred to as the BBA Jacobian). Instead of reverting to subscript notation for computing such derivatives, we perform the entire operation using the matrix differential calculus introduced by (?), which we will briefly review in Appendix A. After some rewriting (reported in App. C) one ends up with the following formulae for the non-zero blocks of the BBA Jacobian

EX
Let m be the number of n images is 5m), and n be t sidual is the 10mn ⇥ 1 ve xij f (Xi, gj) for i = 1 RMS error is appropriate w the same degrees of freedo look at the root of the refer RRV = where k is the number of u adjusted as if their orienta has 30m + 3n unknowns, the unknowns reduces to 6( of relative orientations).
We performed some tests o tion used 3D points and tra XXX (courtesy of XXX), w entations are copied from a and shooting frequency hav eral overlap (cross-track btw forward overlap (along trac to 60%. Real data comes f ation considered the RRV in we also computed the RM alignment of the tie-points truth ones (only available is a free-network one and t ude (isometry + scale), any produced by BBA is reveal

CO
In short, we argue that rela ways be used in multicame We assume that the oblique images relative to the same nadiral image are consecutive and ordered consistently.
Let us write the collinearity equations in compact form as where X are the coordinates of an object point, x are the two coordinates of the corresponding image point, g = [!, ', , t] > represent the exterior orientation of the image, and the central projection is accomplished by Performing BBA using any variant of the Gauss-Newton method entails computing the Jacobian matrix that collects the partial derivatives of f (henceforth referred to as the BBA Jacobian). Instead of reverting to subscript notation for computing such derivatives, we perform the entire operation using the matrix differential calculus introduced by (?), which we will briefly review in Appendix A. After some rewriting (reported in App. C) one ends up with the following formulae for the non-zero blocks of the BBA Jacobia

EX
Let m be the number of n images is 5m), and n be t sidual is the 10mn ⇥ 1 v xij f (Xi, gj) for i = RMS error is appropriate the same degrees of freed look at the root of the refer RRV = where k is the number of u adjusted as if their orienta has 30m + 3n unknowns, the unknowns reduces to 6 of relative orientations).
We performed some tests o tion used 3D points and tra XXX (courtesy of XXX), w entations are copied from a and shooting frequency hav eral overlap (cross-track btw forward overlap (along trac to 60%. Real data comes f ation considered the RRV in we also computed the RM alignment of the tie-points truth ones (only available is a free-network one and t ude (isometry + scale), any produced by BBA is reveal

CO
In short, we argue that rela ways be used in multicame We assume that the oblique images relative to the same nadiral image are consecutive and ordered consistently.
Let us write the collinearity equations in compact form as where X are the coordinates of an object point, x are the two coordinates of the corresponding image point, g = [!, ', , t] > represent the exterior orientation of the image, and the central projection is accomplished by Performing BBA using any variant of the Gauss-Newton method entails computing the Jacobian matrix that collects the partial derivatives of f (henceforth referred to as the BBA Jacobian). Instead of reverting to subscript notation for computing such derivatives, we perform the entire operation using the matrix differential calculus introduced by (?), which we will briefly review in Appendix A. After some rewriting (reported in App. C) one ends up with the following formulae for the non-zero blocks of the BBA Jacobian

EX
Let m be the number of n images is 5m), and n be th sidual is the 10mn ⇥ 1 ve xij f (Xi, gj) for i = 1 RMS error is appropriate w the same degrees of freedo look at the root of the refere RRV = where k is the number of u adjusted as if their orienta has 30m + 3n unknowns, the unknowns reduces to 6( of relative orientations).
We performed some tests o tion used 3D points and tra XXX (courtesy of XXX), w entations are copied from a and shooting frequency hav eral overlap (cross-track btw forward overlap (along track to 60%. Real data comes fr ation considered the RRV in we also computed the RMS alignment of the tie-points truth ones (only available i is a free-network one and t ude (isometry + scale), any produced by BBA is reveale

CO
In short, we argue that rela ways be used in multicamer We assume that the oblique images relative to the same nadiral image are consecutive and ordered consistently.
Let us write the collinearity equations in compact form as where X are the coordinates of an object point, x are the two coordinates of the corresponding image point, g = [!, ', , t] > represent the exterior orientation of the image, and the central projection is accomplished by Performing BBA using any variant of the Gauss-Newton method entails computing the Jacobian matrix that collects the partial derivatives of f (henceforth referred to as the BBA Jacobian). Instead of reverting to subscript notation for computing such derivatives, we perform the entire operation using the matrix differential calculus introduced by (?), which we will briefly review in Appendix A. After some rewriting (reported in App. C) one ends up with the following formulae for the non-zero blocks of the BBA Jacobian

EX
Let m be the number of n images is 5m), and n be th sidual is the 10mn ⇥ 1 ve xij f (Xi, gj) for i = 1 RMS error is appropriate w the same degrees of freedo look at the root of the refere RRV = where k is the number of u adjusted as if their orienta has 30m + 3n unknowns, the unknowns reduces to 6( of relative orientations).
We performed some tests o tion used 3D points and tra XXX (courtesy of XXX), w entations are copied from a and shooting frequency hav eral overlap (cross-track btw forward overlap (along track to 60%. Real data comes fr ation considered the RRV in we also computed the RMS alignment of the tie-points truth ones (only available i is a free-network one and t ude (isometry + scale), any produced by BBA is reveale

CO
In short, we argue that rela ways be used in multicamer We assume that the oblique images relative to the same nadiral image are consecutive and ordered consistently.
Let us write the collinearity equations in compact form as where X are the coordinates of an object point, x are the two coordinates of the corresponding image point, g = [!, ', , t] > represent the exterior orientation of the image, and the central projection is accomplished by Performing BBA using any variant of the Gauss-Newton method entails computing the Jacobian matrix that collects the partial derivatives of f (henceforth referred to as the BBA Jacobian). Instead of reverting to subscript notation for computing such derivatives, we perform the entire operation using the matrix differential calculus introduced by (?), which we will briefly review in Appendix A. After some rewriting (reported in App. C) one ends up with the following formulae for the non-zero blocks of the BBA Jacobian

EX
Let m be the number of n images is 5m), and n be t sidual is the 10mn ⇥ 1 ve xij f (Xi, gj) for i = RMS error is appropriate w the same degrees of freedo look at the root of the refer

RRV =
where k is the number of u adjusted as if their orienta has 30m + 3n unknowns, the unknowns reduces to 6 of relative orientations).
We performed some tests o tion used 3D points and tra XXX (courtesy of XXX), w entations are copied from a and shooting frequency hav eral overlap (cross-track btw forward overlap (along trac to 60%. Real data comes f ation considered the RRV in we also computed the RM alignment of the tie-points truth ones (only available is a free-network one and t ude (isometry + scale), any produced by BBA is reveal

CO
In short, we argue that rela ways be used in multicame The primary structure of the Jacobian of the classical BBA is composed of blocks of two types, which we will call JA and JB, that contain respectively the derivatives of the collinearity equations with respect to (wrt) image orientation and wrt 3D point. In the case that we are examining in this paper, which will be referred in the following as "Constrained Relative Orientations BBA" or CRO-BBA for short, only the nadir images give rise to this type of blocks, whereas oblique images (whose orientation is linked to the nadir image) lead to three new blocks ( Fig. 2): • JC collects the derivative of the residual in the oblique images wrt the nadir orientation; • JD collects the derivative of the residual in the oblique images wrt the relative orientation; • JE collects the derivative of the residual in the oblique images wrt 3D point coordinates.
We assume that the oblique images relative to the same nadir image are consecutive and ordered consistently.
Instead of reverting to subscript notation for computing such derivatives, we perform the entire operation using the matrix differential calculus introduced by (Magnus, Neudecker, 1999), which we will briefly review in App. A. After some rewriting (reported in App. C) one ends up with the following formulae for the non-zero blocks of the CRO-BBA Jacobian: ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume V-2-2020, 2020 XXIV ISPRS Congress (2020 edition) where X = Rτ Rν X+Rτ tν +tτ . The blocks JA, JC and JD have dimension 2×6, whereas JB and JE have dimension 2×3.
Using these formulae, we implemented CRO-BBA in MAT-LAB adopting camera reduction (Brown, 1958) and the Levemberg-Marquardt optimization strategy for free-network adjustment. Levemberg-Marquardt is a variation of Gauss-Newton, where a diagonal term is added to the Hessian matrix. By changing the weight of this term the algorithm can dynamically move between pure Gauss-Newton and gradient descent, improving convergence basin and speed (Börlin, Grussenmeyer, 2013). Moreover, this diagonal matrix (a.k.a. damping term) has a regularization effect that restores the full rank of the Jacobian matrix, thereby implicitly removing the datum defect (Triggs et al., 2000). Classical BBA is obtained by switching off relative orientation constraints in our MATLAB implementation.

EXPERIMENTS
Let k be the number of cameras that compose the multi-head system, let m be the number of nadir images (the total number of images is km), and n be the number of tie-points. The BBA residual is the 2kmn × 1 vector v whose entries are given by xij − f (Xi, gj) for i = 1 . . . n, j = 1 . . . km. While the RMS (root mean square error) is appropriate when comparing the results of regression to models with the same degrees of freedom, in our case one should instead look at the root of the reference variance -RRV (Kraus, 2007), computed as: where is the number of equations (and the length of v) and p is the number of unknowns.
If all tie-points were seen in all images = 2kmn, but in practice it is much smaller than that, as it depends on the visibility. As for p, if the oblique images are adjusted as if their orientations were independent, the system has p = 6km + 3n unknowns, whereas by enforcing the rigidity, the unknowns reduce to p = 6(m + k − 1) + 3n. This means that CRO-BBA solves a significantly smaller system of equations, with benefits in computing time and memory footprint.
We performed tests on simulated and real data. Simulation used 3D points and trajectory from a real LiDAR (Light Detection And Ranging) survey of the city of Udine, Italy (data courtesy by Helica s.r.l.). We assumed a multi-camera system composed of five Phase One iXA 180 (10,328 × 7,760 pixels, 50 mm focal length) in a "Maltese cross" arrangement. Angles between the optical axis of the nadir camera and those of the oblique images were set to 30 • , whereas the distances between the center of the nadir and the oblique cameras were 0.20 m. Altitude and shooting frequency were adapted so as to obtain a lateral overlap (cross-track between nadir images) of 30%, while the forward overlap (along track between nadir images) was set to 60%. As a result, this block consists of 400 images (80 nadir and 320 oblique) and 700 tie-points, with an average of 30 points visible in each image.
Real data were collected by the Leica CityMapper hybrid sensor over the city of Heilbronn, Germany (data courtesy by Leica Geosystems). Leica CityMapper combines a Hyperion LiDAR unit (1064 nm wavelength, theoretical ranging accuracy <2 cm) and a multi-camera system, featuring one nadir-looking camera head (RGB CCD size 10,320 x 7,752 pixels, NIR CCD size 3,336 x 4,500 pixels, 83 mm focal length) and four 45 • -tilted camera heads (RGB CCD size 10,320 x 7,752 pixels, 156 mm focal length). The flight plan was designed using an average nadir GSD (ground sample distance) of 12 cm, and along-across overlaps of 80% and 60%, respectively. The selected subset includes 460 images (92 nadir images and 368 oblique images) and covers an area of ca. 3.5 km x 3.5 km over the city centre (Fig. 3). As ground truth data we employed 49 control points, surveyed with RTK GNSS with a mean 3D accuracy of 5 cm.
Besides image-space errors, the evaluation considered the RMS of the residual distances between corresponding 3D points after least-squares (Procrustean) alignment of i) object points to control points; ii) COPs (Centre Of Projection) to their known positions. Please note that all tie-points in the simulated experiments are also control points and that COPs are not known in real experiments 2 .
Since our BBA/CRO-BBA produce free-network solutions and the alignment transformation is a similitude, any non-Euclidean deformation of the model generated by BBA/CRO-BBA is revealed (in object-space) by the alignment residuals.

Simulated data
To carry out the simulation, we defined the ground truth exterior orientations on the basis of the flight plan described in the previous section. Then we projected the object points on the image planes adding different values of random Gaussian noise to the image coordinates, choosing as standard deviations 10 logarithmically spaced values from 0.5 to 5.0 pixels. The initial values of the exterior orientation of nadir images were obtained by perturbing the ground truth with random Gaiussian noise with σ = 0.20 m for the position and σ = 0.2 • for the Euler angles. Moreover, we added a random noise with σ = 0.05 m and σ = 0.05 • to the relative positions and angles of the oblique cameras, respectively. For each setting the trial was run 100 times. Figure 4(a) shows that the RMS of the reprojection errors (averaged on the 100 trials) is slightly larger for CRO-BBA, while the RRV is practically the same and it is also equal to the standard deviation of the noise added to the image points, as one should expect. Concluding that CRO-BBA is worse than unconstrained BBA on the basis of the reprojection errors (Sun et al., 2016), however, is wrong, because the latter has more degrees of freedom (d.o.f.) than the former, being thus more capable of reducing the residuals by over-fitting (i.e., fitting the noise). Indeed, when the d.o.f. of the model are taken into account with RRV ( Fig. 4(b)), the two are perfectly equivalent in terms of image-space errors. The advantages of the CRO-BBA can be appreciated from the object-space errors, i.e. the alignment residuals wrt control points ( Fig.s 5 and 6). These plots clearly show (with a confidence close to 100% according with the t-test) that the unconstrained BBA produces a less accurate estimate of object points and COPs. The error can be shifted from COPs to object points by dealing differently with the datum defect, but the essence of this result does not change.
For CRO-BBA, errors affecting control points and COPs are comparable (please note the different scales of y-axes for Fig.  5a and 5b). BBA on the other hand does not impose constraints among the COPs of the same multi-camera, that are free to move: this results in an increased error of the COPs.
In a practical scenario, these residual non-Euclidean deformations would have been compensated for by ground control points, and this is why this effect is seldom detected: using a model with a number of d.o.f. in excess is balanced by additional constraints.
In Figure 6(b) one might notice that with BBA the first 80 COPs have larger errors. As a matter of fact, the first 80 indices correspond to nadir images, and the average error over them is 0.123 m, while the average error on oblique images is 0.084 m. The two distributions are different with a confidence very close to 100% according to the t-test. So, ignoring relative orientation constraints affects the localization of the nadir images more than the one of oblique images.
Please note that in Fig. 6(a) the points are sorted by decreasing ray-multiplicity, and the errors follow consistently: the lowest error is achieved by the points with highest ray-multiplicity.

Real data
In the experiments with real data we used 3DF Zephyr to extract and match SIFT-like features and fed these image correspondences to our BBA/CRO-BBA. For the interior parameters, we adopted the values reported in the calibration certificate of the system. Then we computed the errors as in the previous cases, with the difference that the COPs are unknown and that the (49) control points are only a subset of the (8049) tie-points. The RRV is 1.02 pixel for BBA and 1.04 pixel CRO-BBA, while the object-space errors wrt control points are shown in Fig. 7(a), where it can be appreciated that the CRO-BBA produces more accurate results than BBA. The RMS of the object-space errors is 0.20 m for BBA and 0.14 m for CRO-BBA, and the t-test confirm that the two distributions are indeed different with 99.85% ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume V-2-2020, 2020XXIV ISPRS Congress (2020 confidence. This experiment is in agreement with the previous simulations: enforcing relative constraints is beneficial in terms of errors in object-space, notwithstanding the image-space error (RRV) is approximately the same. Figure 7(b) is a box-plot of the statistics of the offsets from the COP of one oblique image to the COP of the corresponding nadir image, after BBA and after CRO-BBA (in the latter case the values are constant). As we do not know the calibrated reference values, this can be taken as an indicator of the precision of the COP localization achieved by BBA. Please note that the scale of y-axis is in meters.

CONCLUSIONS
We investigated bundle adjustment with constrained cameras, i.e., where the orientation of certain cameras is expressed relatively to others, and these relative orientations are part of the unknowns.
Despite some authors suggest that enforcing relative orientation constraints degrades the accuracy of the results, we argued that indeed this is not the case. On the contrary, exploiting the rigidity of the system reduces the number of unknowns and improves the accuracy, when correctly measured. Experiments with simulated and real data confirm that image-space errors is not significant to discriminate between the two solutions (with and without relative orientation constraints), while the objectspace errors demonstrate that the unconstrained solution is affected by severe deformations that impact on the accuracy of the result.
The bottom line is that there is no reason to drop relative orientation constraints in the BBA of image blocks with (synchronized) oblique multi-camera system.
It is worth noting that these formulae are modular with respect to the parametrization chosen for R: changing it will cause only DR to change. Interior orientation can be easily catered for as well.
ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume V-2-2020, 2020 XXIV ISPRS Congress (2020 edition) This contribution has been peer-reviewed. The double-blind peer-review was conducted on the basis of the full paper.