Jacobian of non-square matrices
Sophia Terry 
Let $m>n$ and $L\colon \mathbb{R}^m \to \mathbb{R}^n$ be a linear map(=matrix). The "$n$-dimensional Jacobian"$$ J^n(L) = \sqrt{\det(LL^t)} $$is of geometric significance! Let's say it makes the co-area formula possible.
Suppose $C\colon \mathbb{R}^m \to \mathbb{R}^m$ is an invertible linear map (think of it as a change of coordinates). I am interested in the precise formula relating $J^n(L)$ and $J^n(L \circ C)$.
For simplicity assume $L$ is surjective, i.e. it has rank $n$. Denote by $C^{ker}$ the restriction of $C$ to the $(m-n)$-dimensional $(\ker L\circ C)$. So, $C^{ker}\colon (\ker L\circ C) \to \ker L$ can be seen as a map $\mathbb{R}^{m-n} \to \mathbb{R}^{m-n}$. I think the answer should be:$$ J^n(L) = \frac{|\det C^{ker}|}{|\det C|}\, \cdot J^n(L \circ C)\, , $$Because for certain $C$ I do have a proof, albeit quite a tricky one.
Questions:
Is the claimed identity true?
If not, what is the right formula?
How does one prove it?
2 Answers
$\begingroup$Since $C^{\ker}$ is not an endomorphism (because its domain $\ker(L\circ C)$ and its codomain $\ker(L)$ are different), you need to be precise about the meaning of $|\det C^{\ker}|$. One way to interpret it is as follows. Let $\theta:\mathbb R^{m-n}\to\mathbb R^m$ be any linear isometry (with respect to the usual inner products on $\mathbb R^{m-n}$ and $\mathbb R^m$) whose image is $\ker(L\circ C)$. Let $\pi:\mathbb R^m\to\mathbb R^{m-n}$ be an orthogonal projection in the sense that its restriction on $\ker(L)$ is an isometry and its restriction on $(\ker L)^\perp$ is zero. Then we interpret $|\det C^{\ker}|$ as the absolute value of the determinant of the automorphism $\pi\circ C\circ\theta$. Since all other linear maps $\theta'$ and $\pi'$ with the same properties as $\theta$ and $\pi$ can be written as $\theta'=\theta\circ q_1$ and $\pi'=q_2\circ\pi$ for some isometries $q_1,q_2$ on $\mathbb R^{m-n}$, $|\det(\pi\circ C\circ\theta)|$ is well-defined. (Actually, $J^n(L)$ can also be interpreted in a similar way. Let $\widetilde\theta:\mathbb R^n\to\mathbb R^m$ be an isometry whose range is $(\ker L)^\perp$. Then $J^n(L)$ coincides with $|\det(L\circ\widetilde\theta)|$.)
With this interpretation, the identity you mentioned is true. By picking two appropriate orthonormal bases of $\mathbb R^n$ and $\mathbb R^m$, we may assume that the matrix representation of $L$ is$$ L=\begin{bmatrix}L'&0_{n\times(m-n)}\end{bmatrix} $$for some nonsingular matrix $L'$. Let $Q\in O(m,\mathbb R)$ be any orthogonal matrix whose last $m-n$ rows are orthogonal to the first $n$ rows of the matrix representation of $C$ with respect to the previously chosen basis of $\mathbb R^m$. Then $C$ is in the form of$$ C=\begin{bmatrix}X&0_{n\times(m-n)}\\ Y&Z\end{bmatrix}Q $$where $Z\in M_{m-n}(\mathbb R)$. It follows that\begin{aligned} \ker(L\circ C) &=\ker\left(\begin{bmatrix}L'X&0_{n\times(m-n)}\end{bmatrix}Q\right) =\left\{Q^T\begin{bmatrix}0_{n\times1}\\ u\end{bmatrix}\colon u\in\mathbb R^{m-n}\right\},\\ \ker(L) &=\ker\left(\begin{bmatrix}L'&0_{n\times(m-n)}\end{bmatrix}\right) =\left\{\begin{bmatrix}0_{n\times1}\\ v\end{bmatrix}\colon v\in\mathbb R^{m-n}\right\}.\\ \end{aligned}For any $x=Q^T\begin{bmatrix}0_{n\times1}\\ u\end{bmatrix}\in\ker(L\circ C)$, we have $Cx=\begin{bmatrix}0_{n\times1}\\ Zu\end{bmatrix}$. Therefore, if we define\begin{align*} &\theta:\mathbb R^{m-n}\to\mathbb R^m,\quad u\mapsto Q^T\begin{bmatrix}0_{n\times1}\\ u\end{bmatrix},\\ &\pi:\mathbb R^m\to\mathbb R^{m-n},\quad\begin{bmatrix}w\\ v\end{bmatrix}\mapsto v \end{align*}for all $u,v\in\mathbb R^{m-n}$ and $w\in\mathbb R^n$, then $\theta$ is an isometry whose image is $\ker(L\circ C)$, the restriction of $\pi$ on $\ker(L)$ is an isometry, $\pi=0$ on $(\ker L)^\perp$, and $\left(\pi\circ C\circ\theta\right)u=Zu$. Therefore $|\det C^{\ker}|=|\det Z|$ and the identity$$ J^n(L) = \frac{|\det C^{\ker}|}{|\det C|}\, J^n(L \circ C) $$holds because it simply says that$$ |\det(L')| = \frac{|\det(Z)|}{|\det(X)\det(Z)|}\, |\det(L')\det(X)|. $$
$\endgroup$ 2 $\begingroup$This turned out a bit longer than expected and I'm not sure it answers your questions but it was helpful for me to think about this; so here we go:
Suppose $u\colon (M,g) \to (N,h)$ is either an immersion or submersion (so that its differential has full rank) where $g$ and $h$ are Riemannian metrics. We will denote coordinates on $M$ and $N$ by $(x^1,\ldots,x^m)$ and $(y^1,\ldots,y^n)$, respectively. In these local coordinates we interpret $\mathrm{d}u_x$ as an $(n\times m)$-matrix. In particular, its columns can be interpreted as (the coordinate representation of) the push-forward by $u$ of the coordinate vector fields $\tfrac{\partial}{\partial x^1},\ldots,\tfrac{\partial}{\partial x^m}$, i.e. the $j$-th column of $\mathrm{d}u_x$ is (the coordinate representation of)\begin{align*} u_{\#}\big(\tfrac{\partial}{\partial x^j}\big|_x\big) = \mathrm{d}u_x\big(\tfrac{\partial}{\partial x^j}\big|_x\big) = \tfrac{\partial u}{\partial x^j}\big|_x. \end{align*}Its rows can be interpreted as (the coordinate representation of) the pullback by $u$ of the coordinate covector fields $\mathrm{d}y^1,\ldots, \mathrm{d}y^n$, i.e. the $i$-th row of $\mathrm{d}u_x$ is (the coordinate representation of) the differential of the $i$-th coordinate component of $u$, or in other words the linear form which maps $v\in T_xM$ to\begin{align*} u^{\#}(\mathrm{d}y^i)|_x(v) = \mathrm{d}y^i|_{u(x)}(\mathrm{d}u_x(v)) = \mathrm{d}(y^i\circ u)_x(v) = \mathrm{d}u^i_x(v). \end{align*}Note that if $m<n$ then everything is independent of $g$ and we only need $g$ here to connect it to your formalism! Likewise, if $m>n$ then everything is independent of h!
The Jacobian that appears in the area formula is then\begin{align*} J_m(u)(x) = \sqrt{\operatorname{det}(\mathrm{d}u_x^\ast \mathrm{d}u_x)} \end{align*}where $\mathrm{d}u_x^\ast\colon T_{u(x)}N \to T_xM$ is the adjoint of $\mathrm{d}u_x$ with respect to the inner products $g_x$ and $h_{u(x)}$. The Jacobian in the coarea formula is\begin{align*} J^n(u)(x) = \sqrt{\operatorname{det}(\mathrm{d}u_x \mathrm{d}u_x^\ast)}. \end{align*}In the case $m<n$ what this formula really says is that we need to take the determinant of the Gramian matrix of $(u^{\#}h)_x$ -- the pullback of $h$ by $u$ -- with respect to the coordinate basis above, i.e.\begin{align*} J_m(u)(x) = \sqrt{\operatorname{det}\Big( (u^{\#}h)_x\Big(\tfrac{\partial}{\partial x^i}\big|_x, \tfrac{\partial}{\partial x^j}\big|_x\Big) \Big)} = \sqrt{\operatorname{det}\Big( h_{u(x)}\Big(\mathrm{d}u_x\big(\tfrac{\partial}{\partial x^i}\big|_x\big), \mathrm{d}u_x\big(\tfrac{\partial}{\partial x^j}\big|_x\big)\Big) \Big)} \end{align*}-- here we see that everything is independent of $g$.
In the case $m>n$ it says that we need to calculate the determinant of the Gramian matrix of the induced inner product $\bar{g}_x$ (given by the inverse matrix of $g_x$) on the cotangent space with respect to the basis of pullback covector fields $u^{\#}(\mathrm{d}y^i)$:\begin{align*} J^n(u)(x) = \sqrt{\operatorname{det}\big( \bar{g}_x\big(u^{\#}(\mathrm{d}y^i)\big|_x, u^{\#}(\mathrm{d}y^j)\big|_x\big) \big)} = \sqrt{\operatorname{det}\big( \bar{g}_x\big(\mathrm{d}u^i_x, \mathrm{d}u^j_x\big) \big)} \end{align*}-- here we see that everything is independent of $h$.
Reformulating this again in terms of the exterior product, we can also write for $m<n$\begin{align*} J_m(u)(x) = \Big| \mathrm{d}u_x\big(\tfrac{\partial}{\partial x^1}\big|_x\big) \wedge \cdots \wedge \mathrm{d}u_x\big(\tfrac{\partial}{\partial x^m}\big|_x\big) \Big|_{h_{u(x)},\bigwedge^m T_{u(x)}N } \end{align*}where the norm on the exterior product is the one induced by the inner product $h_{u(x)}$ (defined exactly as the determinant of its Gramian matrix above). In this case, we can use the Plücker embedding to interpret the $m$-vector $\mathrm{d}u_x\big(\tfrac{\partial}{\partial x^1}\big|_x\big) \wedge \cdots \wedge \mathrm{d}u_x\big(\tfrac{\partial}{\partial x^m}\big|_x\big)$ as representing the tangent space $T_{u(x)}u(M)$ of the immersed manifold $u(M)$ as an element of the Grassmannian $\operatorname{Gr}_m(T_{u(x)} N)$. Its norm is then the infinitesimal area element at $u(x)$.
And similarly for the case $m>n$ (if we write $\mathrm{d} u^i_x = (\mathrm{d}u_x)^{\#}(\mathrm{d} y^i|_{u(x)})$)\begin{align*} J^n(u)(x) = \Big| (\mathrm{d}u_x)^{\#}\mathrm{d} y^1|_{u(x)} \wedge \cdots \wedge (\mathrm{d}u_x)^{\#}\mathrm{d} y^n|_{u(x)} \Big|_{\bar{g}_{x},\bigwedge^n T^\prime_{x}M }. \end{align*}Again we can interpret the $n$-covector as an element in the (co-?)Grassmannian $\operatorname{Gr}_n(T^\prime_{x} M)$. Note the role reversal of $M$ and $N$ here. I'm a little less sure of its geometric interpretation in this case.
As one last step, note that if $m<n$ any linear map $A\colon V\to W$ induces a linear map $\bigwedge^m A\colon \bigwedge^m V \to\bigwedge^m W$ by\begin{align*} (\textstyle\bigwedge^mA)(v_1\wedge\cdots\wedge v_m) = Av_1\wedge\cdots\wedge Av_m.
\end{align*}
This operation respects composition, i.e. $\bigwedge^m(A\circ B)=\bigwedge^mA\circ \bigwedge^mB$ (it is a covariant functor).
Furthermore, if $\operatorname{dim}(V)=m$ and $V=W$ then this gives a coordinate-free definition of the determinant of a matrix, i.e. if $A\colon V\to V$ then $\bigwedge^mA=\operatorname{det}(A)$. (See for example [Flanders, Differential forms with applications to the physical sciences]).
Similarly for $m>n$, we can formulate such operations on the dual spaces to get a similar operation for pullbacks by linear maps which reverses composition (it is a contravariant functor): if $A\colon V\to W$ then $(\bigwedge^n A)^{\#}\colon \bigwedge^n W^\prime \to \bigwedge^n V^\prime$ is defined by\begin{align*} (\textstyle\bigwedge^n A)^{\#}(\omega_1\wedge\cdots\wedge \omega_n) = A^{\#}\omega^1 \wedge \cdots \wedge A^{\#}\omega^n
\end{align*}and $(A^{\#}\omega)(v) = \omega(Av)$ for all $v\in V$.
Again, we can reformulate for $m<n$\begin{align*} J_m(u)(x) = \Big| \big(\textstyle\bigwedge^m\mathrm{d}u_x\big)\big(\tfrac{\partial}{\partial x^1}\big|_x \wedge \cdots \wedge \tfrac{\partial}{\partial x^m}\big|_x\big) \Big|_{h_{u(x)},\textstyle\bigwedge^m T_{u(x)}N }
\end{align*}and for $m>n$\begin{align*} J^n(u)(x) = \Big| \big(\textstyle\bigwedge^m\mathrm{d}u_x\big)^{\#} \big(\mathrm{d} y^1|_{u(x)} \wedge \cdots \wedge \mathrm{d} y^n|_{u(x)} \big) \Big|_{\bar{g}_{x},\textstyle\bigwedge^n T^\prime_{x}M }
\end{align*}Thus, finally we can compute how the Jacobian behaves under change of coordinates. Let $\tau\colon M\to M$ be a diffeomorphism. Then $\mathrm{d}\tau_x\colon T_xM \to T_{\tau(x)}M$ is a linear map between $m$-dimensional spaces but -- as user1551 has pointed out -- not an endomorphism. Suppose $(x^1,\ldots, x^m)$ and $(\tilde{x}^1, \ldots, \tilde{x}^m)$ are local coordinates around $x$ and $\tau(x)$. Then we can define a determinant of $\mathrm{d}\tau_x$ as the determinant of the endomorphism $\mathrm{d}(\tilde{x} \circ\tau\circ x^{-1})_x\colon \mathbb{R}^m\to\mathbb{R}^m$ which is independent of the chosen local coordinates (take, for example, orthonormal frames). With this interpretation we have$\bigwedge^m\mathrm{d}\tau_x = \operatorname{det}(\mathrm{d}\tau_x)$.
Then we have in the case $m<n$\begin{align*} J_m(u\circ \tau)(x) &= \Big| \big(\textstyle\bigwedge^m\mathrm{d}u_{\tau(x)} \circ \textstyle\bigwedge^m\mathrm{d}\tau_x\big)\big(\tfrac{\partial}{\partial x^1}\big|_x \wedge \cdots \wedge \tfrac{\partial}{\partial x^m}\big|_x\big) \Big|_{h_{u(\tau(x))},\textstyle\bigwedge^m T_{u(\tau(x))}N } \\ &= |\operatorname{det}(\mathrm{d}\tau_x)| \cdot \Big| \big(\textstyle\bigwedge^m\mathrm{d}u_{\tau(x)}\big)\big(\tfrac{\partial}{\partial x^1}\big|_{\tau(x)} \wedge \cdots \wedge \tfrac{\partial}{\partial x^m}\big|_{\tau(x)}\big) \Big|_{h_{u(\tau(x))},\textstyle\bigwedge^m T_{u(\tau(x))}N } \\ &= |\operatorname{det}(\mathrm{d}\tau_x)| \cdot J_m(u)(\tau(x)).
\end{align*}For the case $m>n$ we make the same calculation\begin{align*} J^n(u\circ \tau)(x) &= \Big| \big(\textstyle\bigwedge^m\mathrm{d}(u \circ \tau)_x\big)^{\#} \big(\mathrm{d} y^1|_{u(\tau(x))} \wedge \cdots \wedge \mathrm{d} y^n|_{u(\tau(x))} \big) \Big|_{\bar{g}_{x},\textstyle\bigwedge^n T^\prime_{x}M } \\ &= \Big| \big(\textstyle\bigwedge^m\mathrm{d}\tau_x\big)^{\#} \circ \big(\textstyle\bigwedge^m\mathrm{d}u_{\tau(x)}\big)^{\#} \big(\mathrm{d} y^1|_{u(\tau(x))} \wedge \cdots \wedge \mathrm{d} y^n|_{u(\tau(x))} \big) \Big|_{\bar{g}_{x},\textstyle\bigwedge^n T^\prime_{x}M } \\ &= |\operatorname{det}(\mathrm{d}\tau_x)| \cdot J^n(u)(\tau(x)).
\end{align*}
