Inverse and Implicit functions

Inverse function theorem

Theorem 1 (Inverse function theorem). Let $U \subseteq \R^d$ be open, $a \in U$, $f\colon U \to \R^d$ be $C^1$. If $Df_a$ is invertible, then there exists open sets $U’, V’$ such that $a \in U’ \subseteq U$, $f(a) \in V’$ and $f: U’ \to V’$ is bijective. Further, the inverse function $f^{-1}:V’ \to U’$ is $C^1$.

Remark 2. The assumption $Df_a$ is invertible necessary. You might be able to find a local inverse even if $Df_a$ is not invertible (e.g. $f(x) = x^3$ with $f^{-1}(x) = x^{1/3}$). However, if $Df_a$ is not invertible, $f^{-1}$ can’t be invertible.

Remark 3. Theorem 1 is easy to prove if $d = 1$ as $f$ must be monotone near $a$.

Proof sketch. The intuition is that $f(a + h) = f(a) + Df_a h + e(h)$. The function $h \mapsto Df_a h$ is invertible by assumption. If the error is small, then it should preserve this property. Proving this rigorously takes some work though.

  1. Replace $f$ with $(Df_a)^{-1} (f(x + a) - f(a))$ and without loss assume $a = 0$, $f(a) = 0$, $Df_a = I$.
  2. Let $F(x) = f(x) - x$, and find $r > 0$ such that $\max_{\abs{x} \leq r} \abs{\partial_j F_i(x)} < 1/(2d)$.
  3. Make $r$ smaller if necessary to ensure $Df$ is invertible in $B(0, r)$.
  4. Local injectivity: Show $\abs{F(x) - F(y)} < \frac{1}{2} \abs{x - y}$, and hence $\frac{1}{2}\abs{x - y} < \abs{f(x) - f(y)} < \frac{3}{2} \abs{x - y}$.
  5. Local surjectivity:
    1. Pick $u \in B(0, r/4)$, and set $\gamma (x) = \abs{f(x) - u}^2$. By the extreme vaulue theorem, there exists $x_* \in \bar B(0, r)$ such that $\gamma (x_*) \leq \gamma (x)$ for all $x \in \bar B(0, r)$.
    2. For $\abs{x} = r$, $\abs{f(x)} \geq r/2$ and so $\abs{\gamma (x)} \geq r^2 / 4 > \gamma (0)$. Thus $\abs{x_*} < r$, and so $\grad \gamma(x_*) = 0$.
    3. $\grad \gamma(x_*) = 0$ implies that for every $i\in \set{1, \dots, d}$, $\partial_i f(x_*) \cdot (f(x_*) - u) = 0$
    4. Since $Df_{x_*}$ is invertible, this forces $f(x_*) = u$ as desired.
  6. Set $V’ = B(0, r/4)$, and $U’ = f^{-1}(V’) \cap B(0, r)$. Now $f \colon U’ \to V’$ is bijective, and so has an inverse $g \colon V’ \to U’$.
  7. To show $g$ is differentiable write $f(y) = f(x) + Df_x (y-x) + e(y - x)$. Set $u = f(x)$, $v= f(y)$, multiply by $(Df_x)^{-1}$ rearrange and get \begin{equation} g(v) = g(y) + (Df_{g(u})^{-1} (v -u) + Df_{g(u)}^{-1} e( g(v) - g(u) ) \end{equation} Show the last term vanishes faster than $\abs{u - v}$ by $\epsilon$-$\delta$.
  8. Finally $g$ is $C^1$ because $Dg_u = (Df_{g(u)})^{-1}$.

Proposition 4. If in Theorem 1 the function $f$ is $C^k$ for some $k \geq 1$, then the local inverse function $f^{-1}$ is also $C^k$.

Contractions, and an alternate proof of surjectivity

The tricky part in the above proof was proving surjectivity. Here’s an alternate way to prove that.

Definition 5 (contraction). Let $f \colon X \to X$ be a function. We say $f$ is a contraction if there exists $\lambda < 1$ such that for every $x, y \in X$ we have $d(f(x), f(y)) \leq \lambda d(x, y)$.

Theorem 6 (Contraction mapping theorem). Suppose $X$ is a complete metric space, and $f \colon X \to X$ is a contraction. Then $f$ has a fixed point (i.e.  there exists $x \in X$ such that $f(x) = x$).

Proof sketch. Let $x_0 \in X$, and define $x_{n+1} = f(x_n)$. Show that the sequence $(x_n)$ is convergent.

Problem 7. Let $f, F, r$ be as in the proof of the inverse function theorem. Show that for every $u \in B(0, r/2)$ there exists $x \in B(0, r)$ such that $f(x) = u$.

Hint. Let $G(x) = u - F(x)$, and find a fixed point of $G$.

Contractions are useful in many situations, and here are a few problems to think about.

Problem 8. Let $f \colon B(0, 1) \to B(0, 1)$ be a contraction.

  1. Show that $f$ need not have a fixed point.
  2. If additionally there exists $\epsilon > 0$ such that $f( B(0, 1) ) \subseteq B(0, 1 - \epsilon)$, then show that $f$ must have a fixed point in $B(0, 1)$.

Problem 9. Let $C \subseteq \R^d$ be a non-empty closed set, and $f \colon C \to C$ be such that $\abs{f(x) - f(y)} < \abs{x - y}$ for all $x, y \in C$ with $x \neq y$.

  1. Show by example that $f$ need not have a fixed point in $C$.
  2. If additionally $C$ is bounded, then show that $f$ must have a fixed point in $C$.

Coordinate changes

Definition 10. A function $\varphi:U \to V$ is called a (differentiable) coordinate change if $\varphi$ is differentiable and bijective and $D\varphi$ is invertible at every point.

Practically, let $\varphi \colon \R^2 \to \R^2$ be a coordinate change function, and set $(u, v) = \varphi(x, y)$. Let $\psi = \varphi^{-1}$, and we write $(x, y) = \psi( u, v)$. Given a function $f:U \to \R$, we treat it as a function of $x$ and $y$. Now using $\psi$ to express $(x, y)$ in terms of $(u, v)$ can treat $f$ as functions of $(u, v)$. The standard example is polar coordinates.

Example 11. Let $u(x, y) = x^2 - y^2$ and $v(x, y) = 2xy$. Let $\varphi(x, y) = (u, v)$. For any $a \neq 0 \in \R^2$, there exists a small neighbourhood of $a$ in which $\varphi$ has a differentiable inverse.

Remark 12. The inverse function theorem guarantees locally the coordinate change can be inverted, and $x, y$ can be expressed as functions of $u, v$. This might not be true globally. In the above example we can explicitly solve and find $x, y$: \begin{equation}\label{eqnUV} x = \paren[\Big]{ \frac{\sqrt{u^2 + v^2} + u}{2} }^{1/2} \quad\text{and}\quad y = \paren[\Big]{ \frac{\sqrt{u^2 + v^2} - u}{2} }^{1/2} \end{equation} is one solution. (Negating both, gives another solution.)

Problem 13 (Spherical coordinates). Let $V = \set{ (r, \theta, \phi) \st r > 0, \theta \in (-\pi, \pi), \phi \in (0, \pi)}$ and define $\varphi(r, \theta, \phi) = (x, y, z)$ where $x = r \sin \phi \cos \theta$, $y = r \sin \phi \sin \theta$ and $z = r \cos \phi$. (Geometrically, $\phi$ is the angle between $(x, y, z)$ and the positive $z$-axis, and $\theta$ is the angle between the projection $(x, y)$ and the positive $x$-axis.)

  1. Let $U = \set{ (x, y, z) \in \R^3 \st x > 0 \text{ or } y \neq 0 }$. Verify that the inverse function theorem applies at all points in $U$, and $r, \theta, \varphi$ can be (locally) expressed as $C^1$ functions of $x, y, z$.

  2. Explicitly express $r, \theta, \phi$ as functions of $x, y, z$ and show that $\varphi:V \to U$ is bijective. Let $(r, \theta, \phi) = \psi( x, y, z)$ denote the inverse function. Compute $\det D\psi$ explicitly.

  3. If $f$ is a differentiable function compute $\partial_x f$, $\partial_y f$ and $\partial_z f$ in terms of $\partial_r f$, $\partial_\theta f$, $\partial_\phi f$, $r$, $\theta$ and $\phi$.

  4. If $g$ is a differentiable function compute $\partial_r g$, $\partial_\theta g$ and $\partial_\phi g$ in terms of $\partial_x g$, $\partial_y g$ and $\partial_z g$, $x$, $y$ and $z$.

Implicit function theorem

Definition 14. Let $U \subseteq \R^d$ be open, $f \colon U \to \R^m$. The graph of $f$ is defined to be the set $\Gamma = \set{ (x, f(x) \st x \in U } \subseteq \R^{d + m}$.

Remark 15. If $d = 1$, then $\Gamma$ is a curve. If $d = 2$, then $\Gamma$ is a surface. In general, $\Gamma$ is a $d$-dimensional manifold.

Remark 16. Not all curves are graphs (e.g. the circle in $\R^2$). However all curves (and surfaces, and manifolds) are locally graphs if you reorder coordinates.

Theorem 17 (Implicit function theorem). Let $U \subseteq \R^n$, $f \colon U \to \R^m$ be $C^1$, $d = n - m > 0$. Let $a \in \R^d$, $b \in \R^m$ such that $(a, b) \in U$, and let $c = f(a, b)$. Suppose the $m \times m$ matrix obtained by taking the last $m$ columns of $Df_{a,b}$ is invertible. Then, there exists an open set $U’ \subseteq \R^n$ such that $U’ \ni (a, b)$, and $U’ \cap \set{ f= c}$ is the graph of a $C^1$ function (of $d$ variables). Explicitly, this means there exists an open set $U_1 \subseteq \R^d$ such that $U_1 \ni a$ and a $C^1$ function $g \colon U_1 \to \R^m$ such that $g(a) = b$ and \begin{equation} U’ \cap \set{f = c} = \set{(x, g(x)) \st x \in U_1} \,. \end{equation}

Remark 18. Recall the notation $\set{f = c}$ is shorthand for $\set{z \in \operatorname{Dom}(f) \st f(z) = c}$.

Before doing the proof, a few remarks and examples are in order.

Remark 19. One can view $f(x, y) = c$ as a system of $m$ equations in $n$ unknowns $x_1$, …, $x_d$ and $y_1$, …, $y_m$. By assumption in Theorem 17, $x = a$ and $y = b$ is one solution. If $Df_{(a,b)}$ satisfies the stated conditions, then Theorem 17 says that for all $x$ close to $a$ there is a unique solution of $f(x, y) = c$, this unique solution can be found by expressing $y$ as a $C^1$ function of $x$. That is there exists a $C^1$ function $g$ such that for all $(x, y) \in U’$, the unique solution of $f(x, y)= c$ is $y = g(x)$.

Example 20. Let $f(x, y) = x^2 + y^2$, and $c = 1$, and consider $(a, b) \in \set{f = 1}$. If $b \neq 0$, one can (explicitly) find a small neighborhood of $(a, b)$ where the unique solution of $f(x, y) = 1$ is given by a $C^1$ function $y = g(x)$. (In this case $g(x) = \pm \sqrt{1 - x^2}$.)

If, however, $b = 0$, then $\set{f = 1}$ is not a graph (where $y$ is a function of $x$) near the point $(a, b)$. You can see this pictorially easily. But algebraically it’s worth noting that if $y$ is a function of $x$, then \begin{equation} x^2 + y^2 = 1 \iff 2x + 2y \frac{dy}{dx} = 0 \,, \end{equation} and one can only solve this equation for $y$ if $dy/dx \neq 0$. This was exactly our assumption in Theorem 17.

Remark 21. Suppose now $f \colon \R^n \to \R^m$ was the linear transformation with matrix $T$. Let $A$ be the $m \times d$ matrix with the first $d$ columns of $T$, and $B$ be the $d \times d$ matrix with the last $m$ columns of $T$, so $f(x, y) = A x + B y$. Now the equation $Ax + By = 0$ can be uniquely solved for $y$ if and only if $B$ is invertible.

In the nonlinear case, set $x’ = a + x$, $y’ = b + y$ and note \begin{equation} f(x’, y’) = c \iff f(a+x, b+y) - f(a, b) = 0 \,. \end{equation} When $x, y$ are small this is roughly the same as solving $Df_{a, b}(x, y) = 0$, which is a linear equation. As we saw above, this can only be uniquely solved if the $m \times m$ matrix obtained by taking the last $m$ columns of $Df_{a, b}$ is invertible.

Proof sketch of Theorem 17. Let $F(x, y) = (x, f(x, y))$, and check $DF_{(a, b)}$ is invertible. By the inverse function theorem there exists open sets $U’ \ni (a, b)$, $V’ \ni (a, c)$ such that $F \colon U’ \to V’$ is invertible, and $F^{-1}$ is $C^1$. Let $G = F^{-1}$, and note that $G(x, y) = (x, g(x, y))$ for some $C^1$ function $g$. From this it quickly follows that for any $(x, y) \in U’$, $f(x, y) = c$ if and only if $y = g(x, c)$. So $U’ \cap \set{f = c}$ is the graph of the function $x \mapsto g(x, c)$.

Proposition 22. Using the same notation and assumptions as Theorem 17, the derivative of the function $g$ is given by \begin{equation} D g_a = - (\mathcal D_y f_{(a, b)})^{-1} (\mathcal D_x f_{(a, b)}) \,. \end{equation} Here $\mathcal D_x f$ is the sub-matrix obtained by taking first $d$ columns of $Df$, and $\mathcal D_y f$ is the sub-matrix obtained by taking the last $m$ columns of $Df$.

Reordering coordinates

Remark 23. Suppose in Remark 21 we knew $\rank(T) = m$, but did not know that the matrix $B$ was invertible. In this case we can reorder coordinates to obtain the same conclusion. Let $P$ be a permutation matrix and let $(x’, y’) = P^{-1} z$ be the reordered coordinates. If $\rank(T) = m$ we can always choose $P$ so that $T P(x’, y’) = A’x’ + B’y’$, where $B’$ is invertible. Observe $T z = 0 \iff T P (x’, y’) = 0 \iff y’ = (B’)^{-1} A’ x’$. That is $Tz = 0$ if and only if in the reordered coordinates $y’$ is a (linear) function of $x’$.

Theorem 24 (Implicit function theorem). Let $U \subseteq \R^n$, $f \colon U \to \R^m$ be $C^1$, $d = n - m > 0$. Let $a \in U$, and $f(a) = c \in \R^m$. Consider a reordering of coordinates given by a permutation matrix $P$ such that the last $m$ columns of $Df_a P$ form an invertible matrix. Then, there exists an open set $U’ \subseteq \R^n$ such that $U’ \ni a$, and in the reordered coordinates, $U’ \cap \set{ f= c}$ is the graph of a $C^1$ function.

Remark 25. Unravelling what it means for $U’ \cap \set{f = c}$ to be a graph in reordered coordinates is a little notationally clunky. As before, given $z \in U$ set $(x’, y’) = P^{-1} z$ to be the reordered coordinates, and write $P^{-1} a = (a’, b’)$. Now $f(z) = 0$ if and only if $f \circ P(x’, y’) = 0$, and the last $m$ columns of $D(f \circ P)_{(x’, y’)}$ are invertible. By Theorem 17 this means there exists an open set $U’_1 \subseteq \R^d$ such that $U’_1 \ni a’$ and a $C^1$ function $g’ \colon U_1 \to \R^m$ such that $g’(a’) = b’$ and \begin{equation} U’ \cap \set{f \circ P = c} = \set{(x’, g’(x’)) \st x’ \in U_1} \,. \end{equation} Un-reordering the coordinates implies \begin{equation} P(U’) \cap \set{f = c} = \set{P(x’, g’(x’)) \st x’ \in U_1} \,. \end{equation}

Problem 26. Consider the equations \begin{equation} (x-1)^2 + y^2 + z^2 = 5 \quad\text{and}\quad (x+1)^2 + y^2 + z^2 = 5 \end{equation} for which $x = 0$, $y = 0$, $z = 2$ is one solution. For all solutions in a small neighborhood of this point, determine which of variables $x$, $y$, $z$ can be expressed as differentiable functions of the others.

Problem 27. For each of the equations below near the given point, which variables can be solved for and expressed as differentiable functions of the remaining variables (according to the implicit function theorem). For each of these variables, compute all the partials at the given point. (That is, if you say $w, x$ can be expressed as differentiable functions of $y, z$, compute $\partial_y w$, $\partial_z w$, $\partial_y x$ and $\partial_z x$ at the given point.)

  1. $x^2 + y^2 - \cos (xy) = 0$ near $(1, 0)$.
  2. $e^{xz} + y \sin(yz) + z = 0$ near $(0, 0, -1)$.
  3. $\sin(xy) + \sin(yz) + \sin(xz) = 0$ and $e^{xyz} + x + y + z = 2$, near $(0, 0, 1)$.

Problem 28. Suppose $f \colon \R^n \to \R^m$ is $C^1$, $a \in \R^n$, and $\rank Df_a = n$. True or false: There must exist $\epsilon > 0$ such that $B( f(a), \epsilon ) \subseteq f( \R^n)$. Prove it, or find a counter example.