\section{Set theory}

In this section we construct the quotient of a type by an equivalence relation. By an equivalence relation we understand a binary relation $R:A\to(A\to\prop)$ which is reflexive, symmetric, and transitive. In particular, we note that equivalence relations take values in $\prop$. The quotient $A/R$ is constructed as the type of equivalence classes, which is just the image of the map $R:A\to (A\to\prop)$. Thus, our construction of the quotient by an equivalence relation is very much like the classical construction of a quotient set. Examples of set quotients are abundant. We cover two of them: the type of rational numbers and the set truncation of a type.

There is, however, a subtle issue with our construction of the set quotient as the image of the map $R:A\to(A\to\prop)$. What is the universe level of the quotient $A/R$? Let us suppose that $\UU$ is a universe that contains $A$ and each $R(x,y)$. Then $\prop$, the type of propositions in $\UU$, is a type in the universe $\UU^+$, constructed in \cref{rmk:universe-constructions}. Therefore the type $\prop^A$ as well as the quotient $A/R$ are also types in $\UU^+$. That seems unfortunate, because in Zermelo-Fraenkel set theory the quotient of a set by an equivalence relation is an ordinary set, and not a more general class.

In Zermelo-Fraenkel set theory quotients are are sets because of the axiom schema of replacement. The replacement axioms assert that the image of any function is again a set. This leads us to wonder about a type theoretical variant of the replacement axioms. Indeed, there is such a variant. The type theoretic replacement property asserts that for any map $f:A\to B$ from a type $A$ in $\UU$ to a type $B$ of which the \emph{identity types} are equivalent to types in $\UU$, the image of $f$ is also equivalent to a type in $\UU$, and in fact this property is a theorem. We prove it in \cref{thm:replacement}, using the univalence axiom and a new construction of the image of a map.

\subsection{Equivalence relations}

\begin{defn}\label{defn:eq_rel}
Let $R:A\to (A\to\prop)$ be a binary relation valued in the propositions. We say that $R$ is an \define{equivalence relation}\index{equivalence relation} if $R$ comes equipped with
\begin{align*}
\rho & : \prd{x:A}R(x,x) \\
\sigma & : \prd{x,y:A} R(x,y)\to R(y,x) \\
\tau & : \prd{x,y,z:A} R(x,y)\to (R(y,z)\to R(x,z)),
\end{align*}
witnessing that $R$ is reflexive, symmetric, and transitive.
\end{defn}

\begin{defn}
  Let $R:A\to (A\to\prop)$ be an equivalence relation. The \define{equivalence class} of $x:A$ is defined to be
  \begin{equation*}
    [x]_R\defeq R(x).
  \end{equation*}
  More generally, a subtype $P:A\to \prop$ is said to be an \define{equivalence class} if it satisfies
  \begin{equation*}
    \mathsf{is\usc{}equivalence\usc{}class}(P)\defeq\exists_{(x:A)}P=R(x).
  \end{equation*}
  Furthermore, we define $A/R$ to be the type of equivalence classes, i.e., we define
  \begin{equation*}
    A/R\defeq \sm{P:A\to\prop}\mathsf{is\usc{}equivalence\usc{}class}(P).
  \end{equation*}
\end{defn}

In other words, $A/R$ is the image of the map $[{-}]_R:A\to (A\to\prop)$. In the following proposition we characterize the identity type of $A/R$. As a corollary, we obtain equivalences
\begin{equation*}
  ([x]_R=[y]_R)\simeq R(x,y),
\end{equation*}
justifying that the quotient $A/R$ is defined to be the type of equivalence classes. Note that in our characterization of the identity type of $A/R$ we make use of the univalence axiom.

\begin{prp}\label{prp:eq-quotient}
  Let $R:A\to (A\to\prop)$ be an equivalence relation. Furthermore, consider $x:A$ and an equivalence class $P$. Then the canonical map
  \begin{equation*}
    ([x]_R=P)\to P(x)
  \end{equation*}
  is an equivalence.
\end{prp}

\begin{proof}
  By \cref{thm:id_fundamental} it suffices to show that the total space
  \begin{equation*}
    \sm{P:A/R}P(x)
  \end{equation*}
  is contractible. The center of contraction is of course $[x]_R$, which satisfies $[x]_R(x)$ by reflexivity of $R$. It remains to construct a contraction. Since $\sm{P:A/R}P(x)$ is a subtype of $A/R$, we construct a contraction by showing that
  \begin{equation*}
    [x]_R=P
  \end{equation*}
  whenever $P(x)$ holds. Recall that $P$ is an equivalence relation, i.e., that there exists a $y:A$ such that $P=[y]_R$. Note that our goal is a proposition, so we may assume that we have such a $y$. Then we obtain that $R(x,y)$ holds from the assumption that $P(x)$ holds. Thus, we have to show that
  \begin{equation*}
    [x]_R=[y]_R
  \end{equation*}
  given that $R(x,y)$ holds. By function extensionality and the univalence axiom, it is equivalent to show that
  \begin{equation*}
    \prd{z:A}R(x,z)\simeq R(y,z)
  \end{equation*}
  We get a function $R(x,z)\to R(y,z)$ by transitivity, since $R(y,x)$ holds by symmetry. Conversely, we get a function $R(y,z)\to R(x,z)$ directly by transitivity. Thus, we obtain that
  \begin{equation*}
    R(x,z)\leftrightarrow R(y,z)
  \end{equation*}
  for any $z:A$, which is sufficient to prove that they are equivalent because $R$ is valued in $\prop$.
\end{proof}

\begin{cor}\label{cor:eq-quotient}
  Consider an equivalence relation $R$ on a type $A$, and let $x,y:A$. Then there is an equivalence
  \begin{equation*}
    ([x]_R=[y]_R)\simeq R(x,y).
  \end{equation*}
\end{cor}

\begin{proof}
  By \cref{prp:eq-quotient} we have an equivalence
  \begin{equation*}
    ([x]_R=[y]_R)\simeq R(y,x).
  \end{equation*}
  Moreover, $R(y,x)$ is equivalent to $R(x,y)$ by symmetry of $R$.
\end{proof}

\begin{comment}
The notion of $0$-equivalence relation which we defined in \cref{defn:eq_rel} fits in a hierarchy of `$n$-equivalence relations'\index{n-equivalence relation@{$n$-equivalence relation}}, the study of which is a research topic on its own. However, we already know an example of a relation that should count as an `$\infty$-equivalence relation'\index{infinity-equivalence relation@{$\infty$-equivalence relation}}: the identity type. Analogous to \cref{thm:equivalence_classes}, the following theorem shows that the canonical map
\begin{equation*}
(x=y)\to (\idtypevar{A}(x)=\idtypevar{A}(y))
\end{equation*}
is an equivalence, for any $x,y:A$. In other words, $\idtypevar{A}(x)$ can be thought of as the equivalence class of $x$ with respect to the relation $\idtypevar{A}$.

\begin{thm}
Assuming the univalence axiom on $\UU$, the map
\begin{equation*}
\idtypevar{A}:A\to (A\to\UU)
\end{equation*}
is an embedding, for any type $A:\UU$.\index{identity type!is an embedding}
\end{thm}

\begin{proof}
Let $a:A$. By function extensionality it suffices to show that the canonical map
\begin{equation*}
(a=b)\to \idtypevar{A}(a)\htpy\idtypevar{A}(b)
\end{equation*}
that sends $\refl{a}$ to $\lam{x}\refl{(a=x)}$ is an equivalence for every $b:A$, and by univalence it therefore suffices to show that the canonical map
\begin{equation*}
(a=b)\to \prd{x:A}\eqv{(a=x)}{(b=x)}
\end{equation*}
that sends $\refl{a}$ to $\lam{x}\idfunc[(a=x)]$ is an equivalence for every $b:B$. To do this we employ the type theoretic Yoneda lemma, \cref{thm:yoneda}.

By the type theoretic Yoneda lemma\index{Yoneda lemma} we have an equivalence
\begin{equation*}
\Big(\prd{x:A} (b=x)\to (a=x)\Big)\to (a=b)
\end{equation*}
given by $\lam{f} f(b,\refl{b})$, for every $b:A$. Note that any family of maps $\prd{x:A}(b=x)\to (a=x)$ induces an equivalence of total spaces by \cref{ex:contr_equiv}, since their total spaces are are both contractible by \cref{cor:contr_path}. It follows that we have an equivalence
\begin{equation*}
\varphi_b:\Big(\prd{x:A} \eqv{(b=x)}{(a=x)}\Big)\to (a=b)
\end{equation*}
given by $\lam{f} f(b,\refl{b})$, for every $b:A$.

Note that $\varphi_a(\lam{x}\idfunc[(a=x)])\jdeq\refl{a}$. Therefore it follows by another application of \cref{thm:yoneda} that the unique family of maps
\begin{equation*}
\alpha_b:(a=b)\to \Big(\prd{x:A} \eqv{(b=x)}{(a=x)}\Big)
\end{equation*}
that satisfies $\alpha_a(\refl{a})=\lam{x}\idfunc[(a=x)]$ is a family of sections of $\varphi$.
It follows that $\alpha$ is a family of equivalences. Now the proof is completed by reverting the direction of the family of equivalences in the codomain.
\end{proof}
\end{comment}

\subsection{The universal property of set quotients}

The quotient $A/R$ is constructed as the image of $R$, so we obtain a commuting triangle
\begin{equation*}
  \begin{tikzcd}[column sep=-1em]
    A \arrow[rr,"q_R"] \arrow[dr,swap,"R"] & & A/R \arrow[dl,hook,"i_R"] \\
    \phantom{A/R} & \prop^A,
  \end{tikzcd}
\end{equation*}
and the embedding $i_R:A/R\to\prop^A$ satisfies the universal property of the image of $R$. This universal property is, however, not the usual universal property of the quotient.

\begin{defn}
  Consider a map $q:A\to B$ into a set $B$ satisfying the property that $f(x)=f(y)$ whenever $R(x,y)$ holds. We say that $q$ satisfies the \define{universal property of the set quotient by $R$} if for every map $f:A\to X$ into a set $X$ such that $f(x)=f(y)$ whenever $R(x,y)$ holds, there is a unique extension
  \begin{equation*}
    \begin{tikzcd}
      A \arrow[d,swap,"q"] \arrow[dr,"f"] \\
      B \arrow[r,densely dotted] & X.
    \end{tikzcd}
  \end{equation*}
\end{defn}

\begin{rmk}
  Formally, we express the universal property of the quotient by $R$ as follows. Consider a map $q:A\to B$ that satisfies the property that
  \begin{equation*}
    H:\prd{x,y:A}R(x,y)\to (f(x)=f(y)).
  \end{equation*}
  Then there is for any set $X$ a map
  \begin{equation*}
    q^\ast:(B\to X) \to \Big(\sm{f:A\to X}\prd{x,y:A}R(x,y)\to (f(x)=f(y))\Big).
  \end{equation*}
  This map takes a function $h:B\to X$ to the pair
  \begin{equation*}
    q^\ast(h)\defeq(h\circ q,\lam{x}{y}{r}\ap{h}{H_{x,y}(r)}).
  \end{equation*}
  The universal property of the set quotient of $R$ asserts that the map $q^\ast$ is an equivalence for every set $X$. It is important to note that the universal property of set quotients is formulated with respect to sets.
\end{rmk}

\begin{thm}\label{thm:quotient_up}
  Let $R:A\to (A\to \prop)$ be an equivalence relation\index{equivalence relation}, and consider a map $q:A\to B$ into a set $B$. Then the following are equivalent.
  \begin{enumerate}
  \item The map $q$ satisfies the property that
    \begin{equation*}
      q(x)=q(y)
    \end{equation*}
    for every $x,y:A$ for which $R(x,y)$ holds, and moreover $q$ satisfies the universal property of the set quotient of $R$.
  \item The map $q$ is surjective and \define{effective}, which means that for each $x,y:A$ we have an equivalence
    \begin{equation*}
      (q(x)=q(y))\simeq R(x,y).
    \end{equation*}
  \item The map $R:A\to (A\to \prop)$ extends along $q$ to an embedding
    \begin{equation*}
      \begin{tikzcd}[column sep=tiny]
        A \arrow[rr,"q"] \arrow[dr,swap,"R"] & & B \arrow[dl,densely dotted,"i"] \\
        & \prop^A
      \end{tikzcd}
    \end{equation*}
    and the embedding $i$ satisfies the universal property of the image inclusion of $R$.
  \end{enumerate}
\end{thm}

\begin{proof}
  We first show that (ii) is equivalent to (iii), since this is the easiest part. After that, we will show that (i) is equivalent to (ii).

  Assume that (ii) holds. Then $q$ is surjective by \cref{thm:surjective}. Moreover, we have
  \begin{align*}
    R(x,y) & \simeq R(x)=R(y) \\
           & \simeq i(q(x))=i(q(y)) \\
           & \simeq q(x)=q(y)
  \end{align*}
  In this calculation, the first equivalence holds by \cref{cor:eq-quotient}; the second equivalence holds since we have a homotopy $R\htpy i\circ q$; and the third equivalence holds since $i$ is an embedding. This completes the proof that (ii) implies (iii).

  Next, we show that (iii) implies (ii). Assuming (iii), we define a map
  \begin{equation*}
    i:B\to\prop^A
  \end{equation*}
  by $i(b,a)\defeq b=q(a)$. Then the triangle
  \begin{equation*}
    \begin{tikzcd}[column sep=tiny]
      A \arrow[rr,"q"] \arrow[dr,swap,"R"] & & B \arrow[dl,"i"] \\
      & \prop^A
    \end{tikzcd}
  \end{equation*}
  commutes, since we have an equivalence
  \begin{align*}
    i(q(a),a') & \simeq R(a,a')
  \end{align*}
  for each $a,a':A$. To show that $i$ is an embedding, it suffices to show that $i$ is injective, i.e., that
  \begin{equation*}
    \prd{b,b':B}(i(b)=i(b'))\to (b=b')
  \end{equation*}
  Note that this is a property, and that $q$ is assumed to be surjective. Hence by \cref{prp:surjective} it is equivalent to show that
  \begin{equation*}
    \prd{a,a':A}(i(q(a))=i(q(a')))\to (q(a)=q(a')).
  \end{equation*}
  Since $R\htpy i\circ q$, and $q(a)=q(a')$ is assumed to be equivalent to $R(a,a')$, it suffices to show that
  \begin{equation*}
    \prd{a,a':A}(R(a)=R(a'))\to R(a,a'),
  \end{equation*}
  which follows directly from \cref{cor:eq-quotient}. Thus we have shown that the factorization $R\htpy i\circ q$ factors $R$ as a surjective map followed by an injective map. We conclude by \cref{thm:surjective} that the embedding $i$ satisfies the universal property of the image factorization of $R$, which finishes the proof that (iii) implies (ii).

  Now we show that (i) implies (ii). To see that $q$ is surjective if it satisfies the assumptions in (i), consider the image factorization
  \begin{equation*}
    \begin{tikzcd}[column sep=tiny]
      A \arrow[dr,swap,"q"] \arrow[rr,"q_q"] & & \im(q) \arrow[dl,"i_q"] \\
      \phantom{\im(q)} & B.
    \end{tikzcd}
  \end{equation*}
  We claim that the map $i_q$ has a section. To see this, we first note that we have
  \begin{equation*}
    q_q(x)=q_q(y)
  \end{equation*}
  for any $x,y:A$ satisfying $R(x,y)$, because if $R(x,y)$ holds, then $q(x)=q(y)$ and hence $i_q(q_q(x))=i_q(q_q(y))$ holds and $i_q$ is an embedding. Since $\im(q)$ is a set, we may apply the universal property of $q$ and we obtain a unique extension of $q_q$ along $q$
  \begin{equation*}
    \begin{tikzcd}
      A \arrow[d,swap,"q"] \arrow[dr,"q_q"] \\
      B \arrow[r,densely dotted,swap,"h"] & \im(q).
    \end{tikzcd}
  \end{equation*}
  Now we observe that the composite $i_q\circ h$ is an extension of $q$ along $q$, so it must be the identity function by uniqueness. Thus we have established that $h$ is a section of $i_q$. Now it follows from the fact that $i_q$ is an embedding with a section, that $i_q$ is an equivalence. We conclude that $q$ is surjective, because $q$ is the composite $i_q\circ q_q$ of a surjective map followed by an equivalence.

  Now we have to show that $q(x)=q(y)$ is equivalent to $R(x,y)$. We first apply the universal property of $q$ to obtain for each $x:A$ an extension of $R(x)$ along $q$
  \begin{equation*}
    \begin{tikzcd}
      A \arrow[d,swap,"q"] \arrow[dr,"R(x)"] \\
      B \arrow[r,densely dotted,swap,"\tilde{R}(x)"] & \prop.
    \end{tikzcd}
  \end{equation*}
  Since the triangle commutes, we have an equivalence $\tilde{R}(x,q(x'))\simeq R(x,x')$ for each $x':A$. Now we apply \cref{thm:id_fundamental} to see that the canonical family of maps
  \begin{equation*}
    \prd{y:B}(q(x)=y)\to \tilde{R}(x,y)
  \end{equation*}
  is a family of equivalences. Thus, we need to show that the type $\sm{y:B}\tilde{R}(x,y)$ is contractible. For the center of contraction, note that we have $q(x):B$, and the type $\tilde{R}(x,q(x))$ is equivalent to the type $R(x,x)$, which is inhabited by reflexivity of $R$. To construct the contraction, it suffices to show that
  \begin{equation*}
    \prd{y:B}\tilde{R}(x,y)\to (q(x)=y).
  \end{equation*}
  Since this is a property, and since we have already shown that $q$ is a surjective map, we may apply \cref{prp:surjective}, by which it suffices to show that
  \begin{equation*}
    \prd{x':A}\tilde{R}(x,q(x'))\to (q(x)=q(x')).
  \end{equation*}
  Since $\tilde{R}(x,q(x'))\simeq R(x,x')$, this is immediate from our assumption on $q$. Thus we obtain the contraction, and we conclude that we have an equivalence $\tilde{R}(x,y)\simeq (q(x)=y)$ for each $y:B$. It follows that we have an equivalence
  \begin{equation*}
    R(x,y)\simeq (q(x)=q(y))
  \end{equation*}
  for each $x,y:A$, which completes the proof that (i) implies (ii).

  It remains to show that (iii) implies (i). Assume (iii), and let $f:A\to X$ be a map into a set $X$, satisfying the property that
  \begin{equation*}
    \prd{a,a':A}R(a,a')\to (f(a)=f(a')).
  \end{equation*}
  Our goal is to show that the type of extensions of $f$ along $q$ is contractible. By \cref{ex:surjective-precomp} it follows that there is at most one such an extension, so it suffices to construct one.

  In order to construct an extension, we will construct for every $b:B$ a term $x:X$ satisfying the property
  \begin{equation*}
    P(x)\defeq \exists_{(a:A)}(f(a)=x)\land (q(a)=b).
  \end{equation*}
  Before we make this construction, we first observe that there is at most one such $x$, i.e., that the type of $x:X$ satisfying $P(x)$ is in fact a proposition. To see this, we need to show that $x=x'$ for any $x,x':X$ satisfying $P(x)$ and $P(x')$. Since $X$ is assumed to be a set, our goal of showing that $x=x'$ is a property. Therefore we may assume that we have $a,a':A$ satisfying
  \begin{align*}
    f(a) & = x & q(a) & = b \\
    f(a') & = x' & q(a') & = b.
  \end{align*}
  It follows from these assumptions that $q(a)=q(a')$, and hence that $R(a,a')$ holds. This in turn implies that $f(a)=f(a')$, and hence that $x=x'$.

  Now let $b:B$. Our goal is to construct an $x:X$ that satisfies the property
  \begin{equation*}
    \exists_{(a:A)}(f(a)=x)\land (q(a)=b).
  \end{equation*}
  Since $q$ is assumed to be surjective, we have $\brck{\fib{q}{b}}$. Moreover, since we have shown that at most one $x:X$ exists with the asserted property, we get to assume that we have $a:A$ satisfying $q(a)=b$. Now we see that $x\defeq f(a)$ satisfies the desired property.

  Thus, we obtain a function $h:B\to X$ satisfying the property that for all $b:B$ there exists an $a:A$ such that
  \begin{equation*}
    f(a)=h(b)\qquad\text{and}\qquad q(a)=b.
  \end{equation*}
  In particular, it follows that $h(q(a))=f(a)$ for all $a:A$, which completes the proof that (ii) implies (i).
\end{proof}

\subsection{The rational numbers}

\subsection{Set truncation}\label{sec:set-truncation}

\begin{lem}
For each type $A$, the relation $I_{(-1)}:A\to (A\to\prop)$ given by
\begin{equation*}
I_{(-1)}(x,y)\defeq\brck{x=y}
\end{equation*}
is an equivalence relation.
\end{lem}

\begin{proof}
For every $x:A$ we have $\bproj{\refl{x}}:\brck{x=x}$, so the relation is reflexive. To see that the relation is symmetric note that by the universal property of propositional truncation there is a unique map $\brck{\invfunc}:\brck{x=y}\to\brck{y=x}$ for which the square
\begin{equation*}
\begin{tikzcd}
(x=y) \arrow[r,"\invfunc"] \arrow[d,swap,"\bproj{\blank}"] & (y=x) \arrow[d,"\bproj{\blank}"] \\
\brck{x=y} \arrow[r,densely dotted,swap,"\brck{\invfunc}"] & \brck{y=x}
\end{tikzcd}
\end{equation*}
commutes. This shows that the relation is symmetric. Similarly, we show by the universal property of propositional truncation that the relation is transitive.
\end{proof}

\begin{defn}
For each type $A$ we define the \define{set truncation}
\begin{equation*}
\trunc{0}{A}\defeq A/I_{(-1)},
\end{equation*}
and the unit of the set truncation is defined to be the quotient map.
\end{defn}

\begin{thm}
For each type $A$, the set truncation satisfies the universal property of the set truncation.
\end{thm}

\begin{exercises}
  \exercise Consider a map $f:A\to B$ into a set $B$, and let $R:A\to(A\to\prop)$ be the equivalence relation given by
  \begin{equation*}
    R(x,y)\defeq f(x)=f(y).
  \end{equation*}
  Show that the map $q_f:A\to\im(f)$ satisfies the universal property of the set quotient of $R$.
  \exercise Show that the set truncation of a loop space is a group.
  \exercise Recall that a normal subgroup $H$ of a group $G$ is a subgroup of $G$ such that $ghg^{-1}$ is in $H$ for every $h:H$ and $g:G$. Given a normal subgroup $H$ of $G$, we write $G/H$ for the quotient of $G$ by the equivalence relation where $g\sim g'$ if and only if there is a $h:H$ such that $gh=g'$. Show that $G/H$ is again a group.
\exercise
\begin{subexenum}
\item Show that any proposition is locally small.\index{proposition!is locally small}
\item Show that any essentially small type is locally small.\index{essentially small!type!is locally small}
\item Show that the function type $A\to X$ is locally small whenever $A$ is essentially small and $X$ is locally small.
\end{subexenum}
\exercise Let $f:A\to B$ be a map. Show that the following are equivalent:
\begin{enumerate}
\item The map $f$ is \define{locally small}\index{locally small!map} in the sense that for every $x,y:A$, the action on paths of $f$
\begin{equation*}
\apfunc{f}:(x=y)\to (f(x)=f(y))
\end{equation*}
is an essentially small map.
\item The diagonal $\delta_f$ of $f$ as defined in \cref{ex:trunc_diagonal_map} is classified by the universal fibration.
\end{enumerate}
\exercise \label{ex:span_rel}Use \cref{thm:choice,thm:fam_proj,cor:times_up_out} to show that the type
\begin{equation*}
\mathsf{span}(A,B)\defeq \sm{S:\UU} (S\to A)\times (S\to B)
\end{equation*}
of small spans from $A$ to $B$ is equivalent to the type $A\to (B\to\UU)$ of small relations from $A$ to $B$.
\end{exercises}

\endinput
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{comment}
\begin{thm}\label{thm:fam_proj}
For any small type $A:\UU$ there is an equivalence
\begin{equation*}
\mathsf{map\usc{}fam}_A:\eqv{(A\to \UU)}{\Big(\sm{X:\UU} X\to A\Big)}.
\end{equation*}
\end{thm}

\begin{proof}
Note that we have the function
\begin{equation*}
\varphi :\lam{B} \Big(\sm{x:A}B(x),\proj 1\Big) : (A\to \UU)\to \Big(\sm{X:\UU}X\to A\Big).
\end{equation*}
The fiber of this map at $(X,f)$ is by univalence and function extensionality equivalent to the type
\begin{equation*}
\sm{B:A\to \UU}{e:\eqv{(\sm{x:A}B(x))}{X}} \proj 1\htpy f\circ e.
\end{equation*}
By \cref{ex:triangle_fib} this type is equivalent to the type
\begin{equation*}
\sm{B:A\to \UU}\prd{a:A} \eqv{B(a)}{\fib{f}{a}},
\end{equation*}
and by `type theoretic choice', which was established in \cref{thm:choice}, this type is equivalent to
\begin{equation*}
\prd{a:A}\sm{X:\UU}\eqv{X}{\fib{f}{a}}.
\end{equation*}
We conclude that the fiber of $\varphi$ at $(X,f)$ is equivalent to the type $\mathsf{ess\usc{}small}(f)$. However, since $f:X\to A$ is a map between small types it is essentially small. Moreover, since being essentially small is a proposition by \cref{lem:isprop_ess_small}, it follows that $\fib{\varphi}{(X,f)}$ is contractible for every $f:X\to A$. In other words, $\varphi$ is a contractible map, and therefore it is an equivalence.
\end{proof}

\begin{rmk}
The inverse of the map
\begin{equation*}
\varphi : (A\to \UU)\to \Big(\sm{X:\UU}X\to A\Big).
\end{equation*}
constructed in \cref{thm:fam_proj} is the map $(X,f)\mapsto \fibf{f}$.
\end{rmk}

\begin{thm}\label{thm:classifier}
Let $f:A\to B$ be a map. Then there is an equivalence
\begin{equation*}
\eqv{\mathsf{ess\usc{}small}(f)}{\mathsf{is\usc{}classified}(f)},
\end{equation*}
where $\mathsf{is\usc{}classified}(f)$\index{is_classified(f)@{$\mathsf{is\usc{}classified}(f)$}} is the type of quadruples $(F,\tilde{F},H,p)$ consisting of maps
$F:B\to \UU$ and $\tilde{F}:A\to \sm{X:\UU}X$, a homotopy $H:F\circ f\htpy \proj 1\circ \tilde{F}$,  such that the commuting square
\begin{equation*}
\begin{tikzcd}
A \arrow[r,"\tilde{F}"] \arrow[d,swap,"f"] & \sm{X:\UU}X \arrow[d,"\proj 1"] \\
B \arrow[r,swap,"F"] & \UU
\end{tikzcd}
\end{equation*}
is a pullback square, as witnessed by $p$\footnote{The universal property of the pullback is not expressible by a type. However, we may take the type of $p:\isequiv(h)$, where $h:A\to B\times_\UU\big(\sm{X:\UU}X\big)$ is the map obtained by the universal property of the canonical pullback.}. If $f$ comes equipped with a term of type $\mathsf{is\usc{}classified}(f)$, we also say that $f$ is \define{classified}\index{classified by the universal family} by the universal family.
\end{thm}

\begin{proof}
From \cref{ex:sq_fib} we obtain that the type of pairs $(\tilde{F},H)$ is equivalent to the type of fiberwise transformations
\begin{equation*}
\prd{b:B}\fib{f}{b}\to F(b).
\end{equation*}
By \cref{cor:pb_fibequiv} the square is a pullback square if and only if the induced map
\begin{equation*}
\prd{b:B}\fib{f}{b}\to F(b)
\end{equation*}
is a fiberwise equivalence. Thus the data $(F,\tilde{F},H,pb)$ is equivalent to the type of pairs $(F,e)$ where $e$ is a fiberwise equivalence from $\fibf{f}$ to $F$. By \cref{thm:choice} the type of pairs $(F,e)$ is equivalent to the type $\mathsf{ess\usc{}small}(f)$.
\end{proof}

\begin{rmk}
For any type $A$ (not necessarily small), and any $B:A\to \UU$, the square\index{Sigma-type@{$\Sigma$-type}!as pullback of universal family}
\begin{equation*}
\begin{tikzcd}[column sep=6em]
\sm{x:A}B(x) \arrow[d,swap,"\proj 1"] \arrow[r,"{\lam{(x,y)}(B(x),y)}"] & \sm{X:\UU}X \arrow[d,"\proj 1"] \\
A \arrow[r,swap,"B"] & \UU
\end{tikzcd}
\end{equation*}
is a pullback square. Therefore it follows that for any family $B:A\to\UU$ of small types, the projection map $\proj 1:\sm{x:A}B(x)\to A$ is an essentially small map.
To see that the claim is a direct consequence of \cref{lem:pb_subst} we write the asserted square in its rudimentary form:
\begin{equation*}
%\begin{gathered}[b]
\begin{tikzcd}[column sep=6em]
\sm{x:A}\mathrm{El}(B(x)) \arrow[d,swap,"\proj 1"] \arrow[r,"{\lam{(x,y)}(B(x),y)}"] & \sm{X:\UU}\mathrm{El}(X) \arrow[d,"\proj 1"] \\
A \arrow[r,swap,"B"] & \UU.
\end{tikzcd}%\\[-\dp\strutbox]\end{gathered}\qedhere
\end{equation*}
\end{rmk}

In the following theorem we show that a type is locally small if and only if its diagonal is classified by $\UU$.

\begin{thm}
Let $A$ be a type. The following are equivalent:
\begin{enumerate}
\item $A$ is locally small.\index{locally small}
\item There are maps $I:A\times A\to\UU$ and $\tilde{I}:A\to\sm{X:\UU}X$, and a homotopy $H:I\circ \delta_A\htpy \proj 1\circ\tilde{I}$
such that the commuting square
\begin{equation*}
\begin{tikzcd}
A \arrow[r,"\tilde{I}"] \arrow[d,swap,"\delta_A"] & \sm{X:\UU}X \arrow[d,"\proj 1"] \\
A\times A \arrow[r,swap,"{I}"] & \UU
\end{tikzcd}
\end{equation*}
is a pullback square.\index{diagonal!of a type}
\end{enumerate}
\end{thm}

\begin{proof}
In \cref{ex:diagonal} we have established that the identity type $x=y$ is the fiber of $\delta_A$ at $(x,y):A\times A$. Therefore it follows that $A$ is locally small if and only if the diagonal $\delta_A$ is essentially small.
Now the result follows from \cref{thm:classifier}.
\end{proof}

\subsubsection{Univalent universes are object classifiers}

\begin{defn}
  Consider a map $p:E\to B$ and a map $f:X\to Y$. The type $\cart(f,p)$ of \define{cartesian morphisms} from $f$ to $p$ is the type of quadruples $(g,h,H,t)$ consisting of maps
  \begin{align*}
    g & : Y\to B \\
    h & : X\to E,
  \end{align*}
  a homotopy $H:g\circ f\htpy p\circ h$, and a term $t$ witnessing that the commuting square
  \begin{equation*}
    \begin{tikzcd}
      X \arrow[r,"h"] \arrow[d,swap,"f"] & E \arrow[d,"p"] \\
      Y \arrow[r,swap,"g"] & B
    \end{tikzcd}
  \end{equation*}
  is a pullback square.
\end{defn}

\begin{defn}
  A map $p:E\to B$ is called an \define{object classifier} if for every map $f:X\to Y$, the type $\cart(f,p)$ of cartesian morphisms
  \begin{equation*}
    \begin{tikzcd}
      X \arrow[r] \arrow[d,swap,"f"] & E \arrow[d,"p"] \\
      Y \arrow[r] & B
    \end{tikzcd}
  \end{equation*}
  from $f$ to $p$ is a proposition.
\end{defn}

\begin{prp}
  Consider a map $p:E\to B$. The following are equivalent:
  \begin{enumerate}
  \item The map $p$ is an object classifier.
  \item The function
    \begin{equation*}
      \tr_{\fibf{p}}:(x=y)\to (\fib{p}{x}\simeq\fib{p}{y})
    \end{equation*}
    is an equivalence.
  \end{enumerate}
\end{prp}

\begin{cor}
  A universe is an object classifier if and only if it is univalent.
\end{cor}
\end{comment}


\section{Derivation of \Cref{heuristic:error_quad_high_order} \label{app:proof_of_error_quad_high_order}}
We are interested in computing the error incurred when approximating a \twod surface integral with an interpolatory quadrature rule.
In \oned on the interval $[-1,1]$, we're interested in the quantity
%\begin{equation}
%    R_q[f] = I[f] - Q_q[f] \label{eq:rem1d},\quad\mathrm{where}\quad
%  I[f] = \int_{-1}^1 f(x) dx, \quad
%  Q_q[f] = \sum_{i=0}^q f(x_i)w_i,
%\end{equation}
\begin{align}
  R_q[f] &= I[f] - Q_q[f] \label{eq:rem1d}\\
  \shortintertext{where}
  I[f] &= \int_{-1}^1 f(x) dx\\
  Q_q[f] &= \sum_{i=0}^q f(x_i)w_i,\\
\end{align}
for quadrature weights $w_i$ for a $q$-point quadrature rule.
For a \twod double integral, we define a similar relationship between the remainder, the exact integral and the $q$th order quadrature rule:
%\begin{equation}
%    R^{(2)}_q[f] = I^{(2)}[f] - Q^{(2)}_q[f]\label{eq:rem2d}, \quad\mathrm{where}\quad
%    I^{(2)}[f] = \int_{-1}^1\int_{-1}^1 f(s,t) dsdt,\quad
%    Q^{(2)}_q[f] = \sum_{j=0}^q\sum_{i=0}^q f(s_i,t_j)w_iw_j.
%\end{equation}
\begin{align}
  R^{(2)}_q[f] &= I^{(2)}[f] - Q^{(2)}_q[f]\label{eq:rem2d}\\
  \shortintertext{where}
    I^{(2)}[f] &= \int_{-1}^1\int_{-1}^1 f(s,t) dsdt\\
  Q^{(2)}_q[f] &= \sum_{j=0}^q\sum_{i=0}^q f(s_i,t_j)w_iw_j,
\end{align}

For a function of two variables $f(s,t)$, we will denote $I_s[f] = \int_{-1}^1f(s,\cdot)ds$ as integration with respect to the $s$ variable only, which produces a function of $t$.
The same subscript notation applies to $R_{q,s}[f]$ and $Q_{q,s}[f]$ and use similar notation  for $t$: we apply the \oned functional to the variable in the subscript, producing a \oned function in the remaining variable.
We observe that
\begin{equation}
  I^{(2)}[f] = \int_{-1}^1\left(\int_{-1}^1 f(s,t) ds\right)dt = \int_{-1}^1 I_s[f] dt = I_t[I_s[f]]
  \label{eq:rem2d_to_1d}
\end{equation}
Following the discussion in \cite{aT2}, we substitute into \cref{eq:rem2d_to_1d} and have
\begin{align}
  %R^{(2)}_n[f] &= I^{(2)}[f] - Q^{(2)}_n[f]\label{eq:rem2d}\\
  I^{(2)}[f] &= I_t[R_{q,s}[f] + Q_{q,s}[f]]\\
         &= R_{q,t}[R_{q,s}[f] + Q_{q,s}[f]] + Q_{q,t}[R_{q,s}[f] + Q_{q,s}[f]] \\
         &= R_{q,t}[R_{q,s}[f]] + Q_{q,s}[R_{q,t}[f]] + Q_{q,t}[R_{q,s}[f]] +Q_{q,t}[Q_{q,s}[f]]
\end{align}
We assume that the higher-order ``remainder of remainder'' term contributes negligibly to the error.
Although it has been shown that this term has a non-trivial contribution to a tight error estimate \cite{elliott2015complete}, %here and in \cref{sec:quad_error_heuristic},
we are able to provide a sufficiently tight upper bound.
For large $q$, the quadrature rule approaches the value of the integral, i.e., $Q_{q,\beta} \approx I_\beta$ for $\beta =s,t$, we're left with:
\begin{align}
  I^{(2)}[f] &\approx  I_s[R_{q,t}[f]] + I_t[R_{q,s}[f]] +Q^{(2)}_q[f],\\
  \shortintertext{and hence:}
  R^{(2)}_q[f] &\lesssim  I_s[R_{q,t}[f]] + I_t[R_{q,s}[f]] \label{eq:rem2d_int},
\end{align}
where $\lesssim$ means "approximately less than or equal to."
From \cite[Theorem 5.1]{trefethen2008gauss}, we recall that for a \oned function $\theta$ defined on $[-1,1]$, if $Q_q[\theta]$ is computed with Clenshaw-Curtis quadrature, $\theta$ is $C^k$ and $\|\theta^{(k)}\|_T < V$ on $[-1,1]$ for real finite $V$, then for sufficiently large $q$, the following inequality holds
\begin{equation}
  R_q[\theta] \leq \frac{32V}{15\pi k(2q+1-k)^k},
  \label{eq:tref_ccquad2d}
\end{equation}
where $\|\alpha(x)\|_T = \|\alpha^\prime/\sqrt{1-x^2}\|_1$.
We're interested in integrating a function $\tilde{\theta}$ over an interval $[-h,h]$ for various $h$.
If $\tilde{\theta}$ is $C^k$ and $\|\tilde{\theta}\|_T < V^\prime$ on $[-h,h]$ for a real constant $V^\prime$ independent of $h$, then we can define $\theta(x) = \tilde{\theta}(hx)$ on $[-1,1]$ and apply
\cref{eq:tref_ccquad2d}:
\begin{equation}
  R_q[\tilde{\theta}] \leq \frac{32h^{k+1}V^\prime}{15\pi k(2q+1-k)^k}.
  \label{eq:tref_ccquad2d_variable_interval}
\end{equation}
This follows directly from the proof of \cite[Theorem 4.2]{trefethen2008gauss} applied to $\theta$ by replacing $\theta$ with $\tilde{\theta}(hx)$ and noting that $\theta^{(k)}(x) = h^k \tilde{\theta}^{(k)}(hx)$.
The change of variables produces the first power of $h$, while each of the $k$ integration by parts produces an additional power of $h$.
In the context of \qbkix, the size of $h$ is proportional to the edge length of the subdomain $D_i$ outlined in \cref{sec:geom-def}.

Applying \cref{eq:tref_ccquad2d_variable_interval} to \cref{eq:rem2d_int}, and again letting $f(s,t) = \Theta(hs,ht)$, gives us
\begin{equation}
  R^{(2)}_q[f] \lesssim \frac{32h^{k+1}}{15\pi k (2q+1-k)^k}\left[I_s[V^\prime_t(s)] + I_t[V^\prime_s(t)]\right] \label{eq:quad_err_1d}
\end{equation}
where $V^\prime_t(s)=\max_t\|\Theta^{(k)}(hs, ht)\|_T$ and $V^\prime_s(t)=\max_s\|\Theta^{(k)}(hs, ht)\|_T$ for fixed values of $s,t$.
If we can choose a $\tilde{V}$ that is strictly greater than $V^\prime_s(t)$ and $V^\prime_t(s)$ for any $s,t$ in $\mathcal{I}^{(2)}$, we are left with
\begin{equation}
  R^{(2)}_q[f] \lesssim \frac{128h^{k+1}\tilde{V}}{15\pi k (2q+1-k)^k}.\label{eq:quad_err_final}
\end{equation}

Applying this to the integration of double layer potentials, we can simply let $\tilde{V}$ be the largest variation of the $k$th partial derivatives of the integrand of any single patch in \cref{eq:double_layer_patches}.
In fact, we know that this value is achieved at the projection of $\vx$ on the patch $P_i$ closest to $\vx$, i.e., $(s^*,t^*) = \mathrm{argmin}_{\mathcal{I}^{(2)}} \|\vx - P_i(s,t)\|_2$.
 We can also choose $h = \max_i h_i$ to observe standard high-order convergence as a function of patch domain size, which we summarize in the following theorem.
  The smoothness and bounded variation assumptions required to apply \cref{eq:tref_ccquad2d} to our layer potential follow directly from the smoothness of $u(\vx)$ in $\Omega$.
  Our heuristic directly follows.


\section{Complex: Complex Numbers}
This module has had an interesting history.  It started as a reimplementation
of NR92's Appendix C.  I [HGG] didn't know about the section 5.4, which
explains the algorithms, so I had to reverse engineering the code itself.
Once I understood the issues, I reimplemented.  I also looked for
and used alternative references for several of the functions.

Later, I added the transcendentals from Krey88, ch 12, pp765-
766.

In March of 1996, Warren Smith made contact and submitted a wealth of
new code, some dependent on his own Complex module.  I folded that into
the existing Complex, assuring both HGG and WDS conventions could be used.
I removed some of my old code and replaced it with WDS implementations.
Some of the old code I left as is.  Notice that the results depend on
structural equivalence in type checking --- doing this in a language
which required name equivalence would have been grim.

WDS generally documents his work meticulously in extended comments.
For other modules I have translated those to \TeX\ format.
However, for this
module there is relatively little documentation provided, so none is
presented here.

\subsection*{add, sub}
Trivial

\subsection*{mul}
See also Hopk88, pp35-36
\begin{verbatim}
     c3.re:=c1.re*c2.re-c1.im*c2.im;
     c3.im:=c1.im*c2.re+c1.re*c2.im
\end{verbatim}

\subsection*{div}
This is retained as hgg\_div, but WDS's implementation is exported.

The basic formula (CRC91, pg 337) is [x is re and y is im]:
\begin{equation}
  \frac{(x_1x_2+y_1y_2) + i(x_2y_1-x_1y_2)}
       {x_2^2 + y_2^2}
\end{equation}
However, this takes 8 multiplies.  Also, if x2 and y2 are
different in magnitude, there can be truncation errors.  So,
if x2 is the biggest, we divide top and bottom by 1/x2:
\begin{equation}
  \frac{(x_1+y_1(y_2/x_2)) + i(y_1 - x_1(y_2/x_2))}
       {x_2 + y_2*(y_2/x_2)}
\end{equation}
Notice how the ratio $y_2/x_2$ repeats.  Also, we use the
denominator for both the re and im portions.  We now have 6
multiplies, and the opportunity to prevent at least some
truncations.

\subsection*{abs}
See WDS's Magnitude comments.

The basic formula (CRC91, pg 337) is
\begin{equation}
  |z| = \sqrt{(x^2 + y^2)}
\end{equation}

The simple cases of $x=0$ or $y=0$ can be resolved right away.

Otherwise, if x and y are different in magnitude, we can
have truncation error.  In 64-bits, we have $EPS=1e-17$, so
$x/y$ (or $y/x$) would have to be that extreme to cause a
problem.  I'll take that chance.

If the problem does arise, or if one needs a 32-bit
representation (where the trouble arises much sooner), do a
normalization.  See also NR92:
\begin{verbatim}
     x:=|x|; y:=|y|;
     if x > y then r=y/x and:

     |z| = (x^2+y^2)^1/2
         = [x^2*(1+y^2/x^2)]^1/2
         = x*(1+(y^2/x^2)^1/2
         = x*(1+(y/x)^2)^1/2
         = x*(1+r*r)^1/2, x > 0
\end{verbatim}

Now we have $1 + something\_less\_than\_one$, which might still
give a truncation error but has a better chance than just
adding the squares.  It costs an extra multiply.  Of course,
if $y >x$, then the analysis is reversed.
\begin{verbatim}
     if y>x then r=x/y and:

       |z| = (x^2+y^2)^1/2
           = [y^2*(x^2/y^2+1)]^1/2
           = y*(x^2/y^2 +1)^1/2
           = y*((x/y)+1)^1/2
           = y*(r^2+1)^1/2, y > 0
\end{verbatim}

\subsection*{sqrt}
See WDS's Sqrt comments.

The basic formula (Krey88, pg 730, eqn 17) is
\begin{equation}
  \sqrt{z} = \pm \left[ \sqrt{\frac{1}{2}(|z| + x)} +
     \mbox{sign}(y)*i*\sqrt{\frac{1}{2}(|z| - x)}\right]
\end{equation}

This seems to do ok.  NR92, pg177 provides a different set
of formulas.  When time permits, we should do accuracy and
timing comparisons.

\subsection*{exp, ln}
From Krey88.

\subsection*{powN, powXY}
From Krey88.

\subsection*{cos, sin, tan}
For cos, the formula is:
\begin{equation}
  \cos(z)=\cos(x)\cosh(y) - i \sin(x)\sinh(y)
\end{equation}

Where:
\begin{equation}
  \cosh(x)=\frac{1}{2}(e^{x} + e^{-x})
\end{equation}

Or:
\begin{verbatim}
     ex:=exp(x); ey:=exp(y);
     tmp.re:=+0.5*cos(x)*(ex-1.0/ex)
     tmp.im:=-0.5*sin(x)*(ey-1.0/ey)
\end{verbatim}

Clearly there is opportunity for truncation in the ex and ey
terms.  Is it a problem? Taking the ex term to be specific:
There is truncation if the exponents of $e^x$ and $e^{-x}$ differ
by more than 17.  This the exponent of $e^x$ can be up to $1/2$
of this or $17/2 = 8$.  Thus x can be up to $\ln(1E8)=~18$.  By
using MathCAD, we find we can get to $x=~20$ before blowing
up.

Alternatively, factor out $e^{-x}$:
\begin{verbatim}
     tmp.re:=+0.5*cos(x)*(ex*ex+1.0)/ex;
\end{verbatim}

This may or may not be more stable, but doesn't fundamental
shift x's range.  We are still in the $|x|<=18$ range.

Is that enough?  CRC91 provides table values to $x=10$.  Let's
accept 18 as the top end, document this, and report an
exception if we go over that.  This will be true of both the
x and y inputs.
\begin{tt} \begin{verbatim}
     IF ABS(c.re) > 18.0D0 OR ABS(c.im)> 18.0D0 THEN
       RAISE Error{Err.out_of_range};
     END;
\end{verbatim} \end{tt}

sin is of course similar.  Notice that for sin and cos there
are no opportunities for ex and ey reuse.  We may as well
use the Math cosh, sinh functions (hoping they are more
robust than our own analysis here).

tan is $sin/cos$:  In this case we can probably reuse ex, ey
from the various cosh and sinh calculations. For now we
however, we will do it naively.

\subsection*{cosh, sinh, tanh}
The formula is:
\begin{equation}
  \cosh(z)=\frac{1}{2}(e^{z}+e^{-z})=\cos(i z)
\end{equation}

Where $e^z$ is found by
\begin{verbatim}
       ex:=exp(z.re);
       tmp.re:= ex*cos(z.im);
       tmp.im:= ex*sin(z.im);
\end{verbatim}

Once again, we can do a full analysis, or just take the easy
way out.  If we ever get into a problem space requiring
hyperbolics in an inner loop, we should do timing analysis.
For now we will take the easy way out:
\begin{verbatim}
     cosh(z) = cos(i*z) = cos(i*z.re + i*i*z.im) = cos(-z.im + i*z.re)

     tmp.re:=-z.im;
     tmp.im:=+z.re;
     tmp:=C.cos(tmp);
     return tmp;
\end{verbatim}

sin is similar:
\begin{verbatim}
     sinh(z) = -i*sin(i*z) = -i*(-z.im + i*z.re)

     tmp.re:=-z.im;
     tmp.im:=+z.re;
     tmp:=C.sin(tmp);
     (*tmp.re = -i*i*tmp.im = tmp.im*)
     (*tmp.im = -i*tmp.re* = -tmp.re*)
     t:=tmp.im;
     tmp.im:=-tmp.re;
     tmp.re:=t;
     return tmp;
\end{verbatim}

\subsection*{pmul, pdiv}
If we happen to be in polar form, then multiplication and
division are simple.  For pmul:
\begin{verbatim}
     tmp.radius:=p1.radius*p2.radius;
     tmp.angle:=p1.angle+p2.angle
\end{verbatim}

For pdiv:
\begin{verbatim}
     tmp.radius:=p1.radius/p2.radius;
     tmp.angle:=p1.angle-p2.angle
\end{verbatim}

In addition, we need to normalize the angles to the $-\pi \dots +\pi$
range:
\begin{tt} \begin{verbatim}
     WHILE tmp.angle < -Pi DO
       tmp.angle:=tmp.angle + TwoPi;
     END;
     WHILE tmp.angle > Pi DO
       tmp.angle:=tmp.angle - TwoPi;
     END;
\end{verbatim} \end{tt}

NOTE:  {\tt arg} and {\tt toPolar} use {\tt atan2}, which does
the normalization itself, so we don't need to handcraft
normalization code for them.


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{environ}
\usepackage{physics}

\NewEnviron{NORMAL}{%
    \scalebox{5}{$\BODY$}
}

\title{Using the Jacobian to map directional derivatives between coordinate systems}
\author{Max Sills }
\date{August 2020}

% Defining bank of important matrices for easy reuuse
\newcommand{\showMatrixOne}{\ensuremath{
\begin{bmatrix}
\frac{x}{\sqrt{x^2 + y^2}} & \frac{y}{\sqrt{x^2 + y^2}} \\
\frac{-y}{x^2 + y^2}       & \frac{x}{x^2+y^2}
\end{bmatrix}
}}

\newcommand{\showMatrixTwo}{\ensuremath{
\begin{bmatrix}
\cos\theta & sin\theta  \\
\frac{-\sin\theta}{r} &  \frac{\cos\theta}{r}
\end{bmatrix}
}}

\newcommand{\showMatrixThree}{\ensuremath{
\begin{bmatrix}
\cos\theta & -r\,\sin\theta  \\
\sin\theta &  r\,\cos\theta
\end{bmatrix}
}}

\newcommand{\showMatrixFour}{\ensuremath{
\begin{bmatrix}
\frac{x}{\sqrt{x^2 + y^2}} & -y \\
\frac{y}{\sqrt{x^2 + y^2}} & x
\end{bmatrix}
}}
% End matrix bank

\begin{document}
\maketitle

\section{Mapping gradients between coordinate systems}

Start with a function U in cartesian coordinates, and U' for that function expressed in polar coordinates.

We can use the change of coordinates formula to express U' given U

\[ x = r * cos(\theta) \]
\[ y = r * sin (\theta)\]

Now say we were able to express the gradient of U in cartesian coordinates
\[
\nabla U = \frac{\partial U} {\partial x}+
\frac{\partial U} {\partial y}
\]

How could we express the gradient of U' in polar coordinates? Well, we've already been given U' in polar coordinates, so we could compute the gradient directly. However, we can also use the chain rule, the change of coordinates formula, and our knowledge of the gradient in cartesian coordinates to directly derive the gradient in polar coordinates.

$$
\nabla U' = \frac{\partial U'} {\partial r}+
\frac{\partial U'} {\partial \theta}
$$

By the chain rule,

$$
\frac{\partial U'} {\partial r} = \frac{\partial U'} {\partial x}\frac{\partial x} {\partial r}
+
\frac{\partial U'} {\partial y}\frac{\partial y} {\partial r}
$$

$$
\frac{\partial U'} {\partial \theta} = \frac{\partial U'} {\partial x}\frac{\partial x} {\partial \theta}
+
\frac{\partial U'} {\partial y}\frac{\partial y} {\partial \theta}
$$

We can rewrite these relations as a covector acting on a Jacobian
\begingroup
\LARGE
\begin{equation}
\left\langle
\begin{matrix}\frac{\partial U}{\partial r} & \frac{\partial U} {\partial \theta}
\end{matrix}
\right\rangle=\left\langle
\begin{matrix}\frac{\partial U}{\partial x} & \frac{\partial U} {\partial y}
\end{matrix}
\right\rangle
*
\begin{bmatrix}
\frac{\partial x}{\partial r} & \frac{\partial y}{\partial r}\\\frac{\partial x}{\partial \theta} & \frac{\partial y}{\partial \theta}
\end{bmatrix}
\end{equation}
\endgroup

We want to find the gradient of U in polar coordinates, which can be represented as the partials of U with respect to r and theta. This is equal to the matrix product of the covector of the gradient in cartesian coordinates which was found above, and the Jacobian, which can be entirely derived from the change of coordinates functions from cartesian to polar.

\section{Examples}

Before we work examples, let's identify the transition functions to and from polar coordinates and the corresponding Jacobians to make things easier.

\subsection{Cartesian to Polar}

\begin{align}
r &= \sqrt{x^2 + y^2} \\
\theta &= \arctan{\left(\frac{y}{x}\right)}
\end{align}
\begin{equation}
\showMatrixOne
=
\showMatrixTwo
\end{equation}

\subsection{Polar to Cartesian}
\[ x = r * cos(\theta) \]
\[ y = r * sin (\theta)\]

\begin{equation}
\showMatrixThree
=
\showMatrixFour
\end{equation}

\subsection{Example 1: U = x + y}
\[ U = x + y\]
\[\nabla U = \pdv{U}{x} + \pdv{U}{y} = <1,1>  \]

Now to express U in polar coordinates (r, theta) we find
\[ U = r*cos\theta + r*sin\theta  \]
\[\nabla U = cos\theta + sin \theta \pdv{U}{r}
+ r (cos\theta - sin \theta \pdv{U}{\theta} \]

\begin{equation}
\begin{bmatrix}
cos + sin & r(cos-sin)
\end{bmatrix}
*
\begin{bmatrix}
\cos\theta & sin\theta  \\
\frac{-\sin\theta}{r} &  \frac{\cos\theta}{r}
\end{bmatrix}
=
\begin{bmatrix}
1 & 1
\end{bmatrix}
\end{equation}

Now, how might we take an arbitrary directional derivative in the first coordinate system and map it to a directional derivative in the second?

Our gradient in the first coordinate system is
\[\begin{bmatrix}
1 & 1
\end{bmatrix}
\]

When we apply an arbitrary directional derivative (a, b), we produce a scalar:
\begin{displaymath}
\begin{bmatrix}
1 & 1
\end{bmatrix}
*
\begin{bmatrix}
a \\
b
\end{bmatrix}
=
a + b
\end{displaymath}

Applying the Jacobian to the vector this time, instead of the gradient (covector) to the Jacobian, we find:
\begin{displaymath}
\showMatrixTwo
*
\begin{bmatrix}
a \\
b
\end{bmatrix}
=
\begin{bmatrix}
a cos(\theta) + b sin(\theta) \\
\frac{-a sin(\theta)}{r} +
\frac{b cos (\theta)}{r}
\end{bmatrix}
\end{displaymath}

Finally, applying this transformed vector to the gradient we found in polar coordinates we see:

\begin{displaymath}
\begin{bmatrix}
cos + sin & r(cos-sin)
\end{bmatrix}
*
\begin{bmatrix}
a cos(\theta) + b sin (\theta) \\
\frac{-a sin(\theta)}{r} +
\frac{b cos (\theta)}{r}
\end{bmatrix}
=
\end{displaymath}
\begin{displaymath}
\begin{aligned}
(acos + bsin)(cos + sin) + (cos- sin)(bcos -asin) \\
= acos^2 + asin cos + bsin cos + bsin^2 \\
+ bcos^2 - asin cos - bsin cos + a sin^2 \\
= acos^2 + b sin^2 + bcos^2 + a sin^2 \\
= a(cos^2 + sin^2) + b(cos^2 + sin^2) \\
= a + b
\end{aligned}
\end{displaymath}

\subsection{Example 2: U(r, theta) = r *theta}

\[U = r * \theta\]
\[\nabla U = \theta \pdv{U}{r}+ r \pdv{U}{\theta} \]

U(r, theta) in cartesian coordinates:

\[ U(x, y) =\sqrt{x^2 +y^2}
*
\arctan\frac{y}{x} \]

\begin{equation}
\nabla U =
\frac{x \atan(y/x) - y}{\sqrt{x^2 + y^2}}
\pdv{U}{x}
+
\frac{y \atan(y/x) + x}{\sqrt{x^2 + y^2}}
\pdv{U}{y}
\end{equation}
\end{document}


\part{Writing Programs}
\section{Control Flow Operations}
\subsection{Conditionals}

We often want to check a conditional statement and then do something
in response to that conditional holding, or not holding. Try (writing
in RWinEdt):
\begin{verbatim}
d <- runif(1)
if (d > 0.5) {
  cat("\n\n today is a good day \n\n")
}
\end{verbatim}

Here is what is happening:

\begin{enumerate}
\item \texttt{d} is a random uniform number drawn from (0,1)%
\item we tell \R: if \texttt{d} is greater than 0.5, then do the
  operation in the curly \texttt{} braces.%
\item the operation in the curly braces is a \texttt{cat} statement
  (short for `concatenate and print') which will print the contents of
  the quotation marks.%
\item the \verb=\n= are simply line breaks to impose a bit of
  space between the prompt and our output.  Here that means two breaks
  either side of the text.
\end{enumerate}

Perhaps we need \R{} to do something if the condition isn't met.  No
problem (note the curly braces!):
\begin{verbatim}
if(runif(1)>0.5){
  cat("\n\n the beatles are on itunes\n\n")
} else {
  cat("\n the beatles are of questionable skill \n")
}
\end{verbatim}
A simpler, `hard wired' alternative to this is \texttt{ifelse()} and
we've already encountered this.

Suppose we want to check more than one condition. Then \verb!&! will
come in handy:
\begin{verbatim}

x <- "3"
y <- cos(3)

if (runif(1) > 0.5 & rnorm(1) < 0) {
  print(x); print(y)
} else{
  (plot(density(rnorm(100))))
}

if (!(rnorm(1) > 0) | !(runif(1) < 0.5)) {
  print(x)
  print(y)
} else{
  (plot(density(rnorm(100))))
}
\end{verbatim}
Notice the use of \texttt{;} to have \R{} do a couple of things.
Sometimes you'll see the use of \verb!&&!.  This means that the second
conditional is only checked if the first one is true.  This
would make no difference to the example above. See the help file for more information.

Usefully, we can nest \texttt{if()} loops.  Try the following:
\begin{verbatim}

if (r<-runif(1) > 0.5){
  cat("r is",r,"\n")
  ##
  if(r < .6){
    print("0.5 to 0.6")
  } else {
    if(.6 < r & r < .7) {
      print("0.6 to 0.7")
    } else {
      print("bigger than 0.7")
    }
  }
}  else {
  print("smaller than 0.5")
}

\end{verbatim}

Lots of things to notice here:
\begin{enumerate}
\item \texttt{(r <- runif(1)) > 0.5} checks the conditional \textit{and}
assigns the number to \texttt{r} in one go.

\item \verb!{cat("r is",r,"\n")}! reports back the value actual
value  of \texttt{r} that has been assigned (notice the use of the
commas) \textit{outside} the quotation marks.

\item \texttt{if(r < .6){print("0.5 to 0.6")}} this occurs conditional
on \texttt{r} being greater than .5, but less than .6

\item \verb!else if (.6 < r & r < .7) {print("0.6 to 0.7")}! is checked if
\texttt{r} is greater than 0.5, and it is not less than 0.6.

\item \verb!else {print("bigger than 0.7")}! prints something if
the previous statements are false (but \texttt{r>}0.5)

\item \verb!else {(print("smaller than 0.5"))}! is the last line of
the program and matches the first conditional (i.e. we are now in
the case where \texttt{r} is less than 0.5)
\end{enumerate}

\subsection{Loops}
Loops, like crystal methamphetamine, are easy to use, and easy to
abuse. They are the workhorses of much of the \R that gets written,
partly because they are so straightforward to write.  This is
unfortunate is some ways, because they are often inefficient.
Throwing caution to the wind, try the following in:

\begin{verbatim}
for (i in 1 : 1000) {
  hist(rnorm(100), col = i, main = paste("Picture", i, sep = " "))
}
\end{verbatim}

\begin{enumerate}
\item this is a \texttt{for} loop: the give away is in the first
line.  We are saying, `for' \texttt{i} between 1 and 1000, do the
thing in the curly braces.

\item here that is: draw a histogram of 100 random normal points,
color it with color number \texttt{i} and then call it
\texttt{Picture i} where \texttt{i}, of course, is just a number.

\item actually there are not 1000 colors in \R{}'s palette, so it
recycles some.

\item see \texttt{?paste} to read the way it is used
\end{enumerate}

Of course, from a programming perspective, having this loop run
every time you run the program maybe annoying.  One option is simply
to wrap it into a function.  So:
\begin{verbatim}
homework<-function(){
  for(i in 1:1000){
    hist(rnorm(100), col = i,
         main = paste("Picture", i, sep = " ")
         )
  }
}
\end{verbatim}
Which means that the \texttt{for} loop won't run until we call it
via \texttt{homework()}

Most of the time, we want to loop through a matrix (or data set)
take something from that matrix and put it somewhere else.

First off, create a matrix---initially filled with missing
values---to take the fruit of our labors:
\begin{verbatim}
mOutput <- matrix(NA, nrow = 30 , ncol = 1)
\end{verbatim}
Now, suppose we have a matrix like this

\begin{verbatim}
mData <- matrix(runif(900), nrow = 30)
\end{verbatim}

That is, $30\times30$ with 900 random numbers.  And we want to go
row by row taking the mean of the row and outputting it.  This would
work:
\begin{verbatim}
for (i in 1 : nrow(mData)) {
  mOutput[i] <- mean(mData[i, ])
}
\end{verbatim}

Notice:
\begin{enumerate}
\item we can use pretty much anything for our index: here it is
  \texttt{i}; if we replaces every reference to \texttt{i} in this
  expression with \texttt{monkey}, that would work too

\item the end of the index is the number of rows in \texttt{mData}
  (which is 30)

\item now, we are assigning the mean of the $i^{th}$ row of
\texttt{mData} to the $i^{th}$ row of \texttt{mOutput}
\end{enumerate}

So what's the problem?  The loop is perfectly accurate, but it is
slow, and laborious to code.

We may sometimes have cause to place \texttt{for} loops within other
\texttt{for} loops (but generally try to avoid). Here is an example:
\begin{verbatim}

mLetters <- matrix(data = NA, nrow = 10, ncol = 5)
for (m in 1:10) {
  for(n in 1:5){
    mLetters[m,n] <- (letters[n + m])
  }
}
\end{verbatim}

Look at \texttt{letters}. What is it? What does this code do?

Notice that the way we are indexing the loops
matters here.  If we put \texttt{mLetters[n, m]} instead of
\texttt{mLetters[m, n]}in our code, we'll get the dreaded
\begin{verbatim}
Error: subscript out of bounds
\end{verbatim}
There are alternatives to \texttt{for} which are used in different
circumstances.  Examples are \texttt{while} and \texttt{repeat}
which are often used together.

\subsection*{Other Types of Loops}
The syntax for \texttt{while} is \texttt{while(condition)
expression} which means while a particular condition holds, the
\texttt{expression} will be executed. \texttt{repeat(expression)}
simply repeats the \texttt{expression} operation again and again and
again.  This type of thing turns up a lot in monte-carlos. Consider
the following:
\begin{verbatim}
n <- 1
mF <- matrix(NA, nrow = 10, ncol = n)

while (n < 36) {
  repeat{
    vF <- rbinom(10, 1, 0.5)
    if (sum(vF) %% 2 == 0) {
      break()
    }
  }
  mF <- matrix(cbind(mF[, 1 : (n-1)], vF), nrow = 10, ncol = n)
  n<-n + 1
}
\end{verbatim}
Here, while \texttt{n} is less than 36, I need to repeatedly sample
from a binomial (with a sample size of 10) until an even number of
the sample are ones.  So, for example, $[1,0,0,0,0,0,0,1,1,0]$ won't
do, but $[1,1,0,0,1,1,0,1,1,0]$ is fine. As soon as I get a sample
fulfilling my requirements, it should be stored (in \texttt{mF}).
Notice that I have to increment the loop with \texttt{n + 1} or else
the condition \texttt{n < 36} will be true forever, and the program
will never stop.

Just to make this point, consider
\begin{verbatim}
repeat(cat("\n lake effect snow today\n"))
\end{verbatim}
which won't end until you hit \texttt{esc} or \texttt{STOP}.

\section{Sampling}
Quite often we have to sample from a (posterior) distribution we
created. In general, we want to sample \texttt{n} values, but we want
the sample we produce to be `weighted'---in the sense that it is
proportional to the mass---for each value of our discrete support
(say, the thousand values of $\theta$ we created for our homework.

We could put together the numerical cdf, but it easier to use
\texttt{sample} directly.
\begin{verbatim}
vCand <- seq(0, 1, 0.001)
vP <- dchisq(vCand, df = 15)
vS1 <- sample(vCand, 1000, replace = TRUE, prob = vP)

vDraws <- rnorm(length(vCand))
vS2 <- sample(vDraws, length(vDraws), replace = TRUE)
\end{verbatim}

Here, I'm assuming that we first took 1000 values between zero and
one, and then we worked out that the posterior,$\Pr(\theta|y)$, was
$\chi^2_{15}$ (which is unlikely, but anyway).  Then we sampled:
\begin{enumerate}
\item the object of our sampling was our candidate $\theta$ values.%
\item we wanted a sample of size 1000
\item we \texttt{replace}d the candidates each time we sampled
\item we set \texttt{prob = vP} which means we are sampling in
proportion to the posterior we calculated.
\item What could \texttt{vS2} look like if we used \texttt{replace = FALSE}

\end{enumerate}

\section{Functional Programming}
\subsection{Basics}
Creating a function is very easy and any time you use code over and
over again (e.g.\ typed manually, in a loop, next semester) you can
and should wrap that code up in a function. Functions take in
arguments (objects) and return a single object.

Consider:
\begin{verbatim}
SayHello <- function()
  {
    cat("\n \n \n Hello World \n \n \n")
  }
\end{verbatim}
Try both \texttt{SayHello} and \texttt{SayHello()}. One displays the
function itself and one displays the output. That is, loosely
speaking, the two are the difference between $f$ and $f(x)$ for some
fixed value $x$.

In this function, there are no arguments and we don't create any
objects for latter use, but this need not be the case.

\begin{verbatim}
GetCos <- function (x = 0)
  {
    vC <- cos(x)

    return(vC)
  }
\end{verbatim}

\texttt{GetCos()} takes a single argument which has a default value of
0. Then, we calculate the cosine of $x$ and assign it to
\texttt{vC}. Lastly, before the function ends, we tell \R{} that the
output of the function is \texttt{vC}.

Try \texttt{GetCos(1)} and \texttt{GetCos()}. Have we created any new
objects in our workspace, namely \texttt{vC}? No. To save the output
for latter use we would do
\begin{verbatim}
vOut <- GetCos()
\end{verbatim}

If we have more than one object to return at the end of the function,
we can wrap these objects up together in another single object.

Consider
\begin{verbatim}
GetTan <- function (x)
  {
    vT <- tan(x)

    vTT <- "your face is a tangent!"

    return(list(vT, vTT
                )
           )
  }

GetTan(1)
\end{verbatim}

We return the output as a list (because the objects had different
modes) and we use something like \verb=lOutput2 <- GetTan(4)= to save
it for future use.

If you pass a function an argument that it was not expecting based on the
function definition, you will get an error.

If you don't pass a function an argument that it was expecting based
on the function definition, you will get an error.

\subsection{Scope}
There are some very important rules to follow when using functions to
keep things clear. First, realize that a function is a temporary
workspace (or environment in \R{}-speak). The arguments are one-way
traffic onto this island. The \texttt{return} statement is one-way
traffic off the island.
\begin{enumerate}
\item any object not created by and contained within your function
  should be passed to it as an argument

\item any object(s) to be used later should be returned by your
  function

\item functions (with the exceptions of \texttt{save}, \texttt{plot},
  etc.) should not have side-effects
\end{enumerate}

Consider this code:
\begin{verbatim}
vThing <- 4

GetSquare <- function()
  {
    vThing ^ 2
  }

GetSquare()

rm(vThing)

GetSquare()
\end{verbatim}

We violated our rule of relying on other objects only through the
arguments. And, we can see how easily our function becomes worthless
if we don't listen to that rule.

Using the \verb!<<-! operator is a quick way to violate our other
rules.

Try:
\begin{verbatim}
BadFunc <- function()
  {
    cat("i woke up at 8am and all i got was this lousy R function")

    mLetters <<- "gone! mwahhahhah"
  }
\end{verbatim}

Look at \texttt{mLetters} from before. We changed it even though we
didn't pass the function any arguments, we didn't return any object as
output, and we didn't assign output to an identifier. Had we used
\verb=<-= instead of \verb=<<-=, we would only have altered the object
\texttt{mLetters} within the function environment so nothing would
have been overwritten.

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "../full_course"
%%% End:


\subsection{Turbulence closure}\label{sec-tur}

We use the \citet{Umlauf2003} generic length-scale model:
\beqa
  \frac{\D k}{\D t}&=&\frac{\pd }{\pd z}\left( \nu_k^\Psi \frac{\pd k}{\pd z} \right)
  +\nu_t M^2+\nu_t^\theta N^2 -\epsilon \\
  \frac{\D \Psi}{\D t}&=& \frac{\pd }{\pd z}\left( \nu_\Psi \frac{\pd \Psi}{\pd z} \right)
    +\frac{\Psi}{k}(c_{\Psi 1}\nu_tM^2+c_{\Psi 3}\nu_t^\theta N^2-c_{\Psi 2}\epsilon F_{wall})
\eeqa
with natural b.c.:
\beq
   \left\{ \begin{array}{ll}
       \nu_k^\Psi \frac{\pd k}{\pd z} &=0, \mbox{ at } z=-h, \mbox{ or } \eta \\
       \nu_\Psi\frac{\pd \Psi}{\pd z} &= \kappa_0 n\nu_\Psi\frac{\Psi}{l} , \mbox{ at } z=-h \\
       \nu_\Psi\frac{\pd \Psi}{\pd z} &= -\kappa_0 n\nu_\Psi\frac{\Psi}{l} , \mbox{ at } z=\eta \\
           \end{array}
   \right.  \label{tur1}
\eeq
and essential b.c.:
\beq
   \left\{ \begin{array}{ll}
       k&=(c_\mu^0)^{-2} \nu|\frac{\pd \bs{u}}{\pd z}|, \mbox{ at } z=-h, \mbox{ or } \eta\\
       l&=\kappa_0 \D \\
       \Psi &= (c_\mu^0)^pk^m(\kappa_0 \D)^n
           \end{array}
   \right.  \label{tur2}
\eeq
where $k$ is the TKE, $l$ is the mixing length, $c_{\Psi *}$ are some constants and $\Psi=(c_\mu^0)^pk^m l^n$ is a generic length-scale variable.
The turbulence production and dissipation terms are:
\beqa
  M^2&=&\left( \frac{\pd u}{\pd z}\right)^2+\left( \frac{\pd v}{\pd z}\right)^2 \\
  N^2 &=&\frac{g}{\rho_0}\frac{\pd \rho}{\pd z} \\
  \epsilon &=& (c_\mu^0)^3k^{1.5} l^{-1}
\eeqa
In the code, the natural b.c. is applied first (see the FEM formulation below), and the essential b.c.
 is then used to overwrite the boundary values of the unknown, as suggested by the \gls{gotm} implementation.

GOTM has also been coupled to SELFE, but the native (finite element) implementation is the one used for the Bay-Delta project.


\section{Soft-thresholding and Proximal gradient method}
\label{sec:soft-thresholding}
Recall the unconstrained optimization problem
$$
\min _{x} f(x)
$$
We have seen the following results depending on the assumption for $f(x)$
\begin{enumerate}
\item If $f(x)$ is not smooth (for example: $L^1$ norm) which means $\nabla f(x)$ is not continuous, subgradient descent method gives the $\frac{1}{\sqrt{n}}$ convergence.
\item If $f(x)$ is smooth (e.g. $\nabla f(x)$ is L-Lipschitz continuous), gradient descent method gives the $\frac{1}{n}$ convergence.
\end{enumerate}
There may be other methods (may be for particular functions) which can give better performance. Consider the following example:
$$
f(x)=\frac{1}{2}\|A x-b\|_{2}^{2}+\|x\|_{1}
$$
This is a type 1 problem described above, as the second term is not smooth. However, the objective has a special form. It can be represented as sum of smooth and non -smooth function. Also note that second term $\|x\|_{1}$ is "simple" in a sense that we describe precisely below. In this case, it is conceivable that some tailored method might do better than the generic guarantees offered by sub gradient descent.

In this section, we introduce what is known as the Proximal method, and show that it provides improved guarantees.

\subsection{Proximal mapping}\label{sec:proximal}
\begin{definition}
The proximal mapping or prox-operator of a function $f(x)$ is defined as
\begin{equation}\label{pro:def}
\operatorname{Prox}_{f}(x)=\arg \min _{z}f(z)+\frac{1}{2}\|z-x\|_{2}^{2}.
\end{equation}
\end{definition}
Note that $f(x)$ is a convex function so the above problem \eqref{pro:def} a strongly convex problem because of the added Euclidean norm term; therefore the solution is unique. Also for the proximal method to be efficient, we need that $f(x)$ should be simple in the sense that solving the problem \eqref{pro:def}  should be easier and fast or may be even analytical solutions are possible. This condition is required to ensure that each step is not much costly.

\begin{properties}
From the optimality conditions of minimization in the definition, we can see that $z^*$ is optimal point of $f(x)+\frac12 \|z-x\|_2^2$ only if $0\in \partial f(x) + z^*-x$, namely
\begin{equation}
\begin{aligned}
z^{*}=\operatorname{Prox}_{f}(x) & \Longleftrightarrow 0 \in \partial f(x)+\left(z^{*}-x\right) \\
& \Longleftrightarrow x-z^{*} \in \partial f(x) \\
& \Longleftrightarrow f(w)=f(z)+(x-z)^{T}(w-z)\quad \forall w
\end{aligned}
\end{equation}
\end{properties}
This property shows the relation between proximal mapping  and subgradient of  a function.

\begin{example}
If $f(x)=0$,
$$
\operatorname{Prox}_{f}(x)=x.
$$
\end{example}
\begin{example}
If $f(x)=|x|$
$$
\begin{array}{l}
\operatorname{Prox}_{f}(x)=\left\{\begin{array}{cl}
x-1 & \text { if } x\geqslant 1 \\
0 & \text { if } -1 \leqslant x \leqslant 1    \\
x+1 & \text { if } x \leqslant-1
\end{array}\right.
\end{array}
$$
	\end{example}
\begin{example}
If
	$$
	f(x)=\left\{\begin{array}{l}
	0 \text { if } x \in[-1,1] \\
	\infty \text { otherwise }
	\end{array}\right.
	$$
$$ 	\begin{array}{l}
	\operatorname{Prox}_{f}(x)=\left\{\begin{array}{cl}
			x & \text { if } x \in [-1, 1] \\
			1 & \text { if } x>1   \\
			-1 & \text { if } x < -1
		\end{array}\right.
	\end{array}  $$
which is the projection onto the set [-1,1].
if $$
f(x)=\left\{\begin{array}{ll}
0 & \text { if } x \in C \\
\infty & x \notin C
\end{array}\right.
$$  these $ \operatorname{prox}_{f}(x) $ is the projection onto C.
	\end{example}

%\begin{example}
%	If $ X=\partial f $ where f is convex, what is $ prox_{f} $?
%
%	For $ x \subset\mathbb{R}^{d} $, solve for $ (x^{\prime}, y^{\prime}) \in \partial f $ s.t.
%	$$
%	 x^{\prime}+y^{\prime}=x.
%	$$ Notice that
%	$$
%	x \in \operatorname{argmin} f \Leftrightarrow 0 \in \partial f(x)
%	$$
%	Also, for $ x \in dom(f_{1}) \cap dom(f_{2}) $,
%	$$
%	 \partial (f_{1}+f_{2}) (x)=\partial f_{1}(x)+\partial f_{2}(x)  :=\left\{y_{1}+y_{2} ; y_{1} \in \partial f_{1}(x), y_{2} \in \partial f_{2}(x)\right\}
%	 $$
%	$$
%	0\in \partial \left(f(z)+\frac{1}{2} \|(x-z)\|^{2}\right)=\partial f(z)-x+z\Rightarrow x \in z+ \partial f(z).
%	$$
%	$$
%	\operatorname{prox}_{f}(x)=\operatorname{argmin}\limits_{z}\left(f(z)+\frac{1}{2}\|x-z\|_2^{2}\right)
%	$$
%	\end{example}

\begin{theorem}[soft-thresholding]
Let $f(x)=\lambda\|x\|_1$. Soft-thresholding  is defined as
\begin{equation}
prox_f(x) = \arg\min_z\{\frac{1}{2}\|x-z\|^2_2 + \lambda\|z\|_1 \},
\end{equation}
which is shown in the following figure
\begin{figure}
\centering
\includegraphics[width=3in]{./6DL/figures/softthreshold}
%\caption{Two linearly separable sets}
\end{figure}
The $l_1$-norm is is separable in indices. Each $i^{\text {th }}$ coordinate can be optimized separately and the $i^{t h}$ coordinate is given as
$$
prox_f(x)_{i}=\operatorname{sgn}\left(x_{i}\right)\left[\left|x_{i}\right|-\min \left\{\left|x_{i}\right|, \lambda\right\}\right]
=\begin{cases}
0&|x_i|\le \lambda
\\
x_i-\lambda&x_i>\lambda
\\
x_i+\lambda&x_i<-\lambda
\end{cases}.
$$
\end{theorem}

\begin{proof}
Note that
\begin{equation}
0 \in \nabla(\frac{1}{2}\|x-z\|^2_2) + \partial(\lambda\|z\|_1) \Leftrightarrow 0 \in z-x + \lambda\partial\|z\|_1
\end{equation}
The $l_1$-norm is separable and thus we can consider each of its components separately. Let's examine first the case where $z_i \neq 0$. Then, $\partial \|z_i\|=sign(z_i)$ and the optimum $z_i^*$ is obtained as
\begin{equation}
0 = z_i-x_i + \lambda sign(z_i) \Leftrightarrow z_i^* = x_i - \lambda sign(z_i^*)
\end{equation}
Note also that if $z_i^* < 0$, then $x_i < -\lambda$ and equivalently if $z_i^* > 0 \Rightarrow x_i > \lambda$. Thus, $|x_i| > \lambda$ and $sign(z_i^*) = sign(x_i)$. Substituting in the previous equation we get
\begin{equation}
z_i^* = x_i - \lambda sign(x_i)
\end{equation}
In the case where $z_i = 0$, the subdifferential of the $l_1$-norm is the interval $[-1,1]$ and the optimality condition is
\begin{equation}
0 \in -x_i + \lambda[-1,1] \Leftrightarrow x_i \in [-\lambda,\lambda] \Leftrightarrow |x_i| \leq \lambda
\end{equation}
Putting all together we get
\begin{equation}
[prox_f(x)]_i = z_i^* =
\left\{ \begin{array}{lr} 0 & \text{if } |x_i| \leq \lambda \\
x_i - \lambda sign(x_i) &\text{if } |x_i| > \lambda \end{array}\right.
\end{equation}
The previous equation can also be written as
\begin{align*}
[prox_f(x)]_i &= sign(x_i)\max(|x_i|-\lambda, 0) \\
               &= sign(x_i)(|x_i|-\lambda)_+
\end{align*}
where $(\cdot)_+$ denotes the positive part and it leads to sparse solution.
\end{proof}

\subsection{Properties of the proximal map}
\begin{definition}
: A map g:$ \mathbb{R}^{d} \rightarrow \mathbb{R}^{d} $ is called
\begin{itemize}
	\item Non-expansive if $ \|g(x)-g(y)\| \leqslant\|x-y\| $
	\item Firauly Non-expansive if $ \|g(x)-g(y)\|^{2} \leqslant\langle g(x)-g(y), x-y\rangle $
	\item A contraction if $ \|g(x)-g(y)\| \leqslant \lambda\|x-y\| \quad \text { for } 0 \leqslant \lambda<1 $
\end{itemize}
\end{definition}

\begin{lemma}
	If g is non-expansive, the $ \frac{1}{2}(g+I) $ is 	firmly non-expansive.
	$$
	\frac{1}{2}(g+I)(x)=\frac{1}{2}(g(x)+x)
	$$
	If g is firmly non-expansive, then $ 2g-I $ is non-expansive.
\end{lemma}

\begin{proof}
	$$
	\begin{aligned}
	\|h(x)-h(y)\|^{2}&=\left\langle\frac{1}{2}[g(x)+x]-\frac{1}{2}[g(y)+y], \frac{1}{2}[g(x)+x]-\frac{1}{2}[g(J)+y]\right\rangle\\
	&=\left\langle\frac{1}{2}(g(x)-g(y))+\frac{1}{2}(x-y), \frac{1}{2}(g(x)-g(y))+\frac{1}{2}(x-y)\right\rangle \\
	&=\frac{1}{4} \|g(x)-g(y)\|^{2}\|x-y\|^{2}+\frac{1}{2}(g(x)-g(y), x-y)+\frac{1}{4}\|x-y\|^{2} \\
	& \leqslant \frac{1}{2}\|x-y\|^{2}+\frac{1}{2}\langle g(x)-g(y), x-y\rangle \\
	&=\left\langle\frac{1}{2}(g(x)-g(y))+\frac{1}{2}(x-y), x-y\right\rangle\\
	&=\langle h(x)-h(y), x-y\rangle .
	\end{aligned}
	$$
	\end{proof}

\begin{example}
	Non-expansive $ \mathbb{R}^{2}  $, rotate by $ \theta $, has a fixed point, but we don't converge to it.
	Nonexpansive maps may fail to converge to fixed points. Firmly nonexpanisve maps always converge to their fixed points.
	Fixed points may not be unique though.
	\end{example}

\begin{example}
	$ g: \mathbb{R} \rightarrow \mathbb{R}$,
	$$  g(x)=\left\{\begin{array}{cl}
	x & \text { if } x\in [-1,1] \\
	1 & \text { if }  x > 1    \\
	-1 & \text { if } x <-1
	\end{array}\right.$$
	\end{example}
g is firmly non-expansive. Set of fixed points [-1,1]
\begin{theorem}
The proximal map is always firmly non-expansive.
	\end{theorem}
\begin{theorem}
If g is firmly nonexpansive and $ g(y)=y $	for some y, there if $ x_{n+1}=g(x_{n}) $,we have $ x_{n} \rightarrow y^{\prime} $ with $ g(y^{\prime})=y^{\prime} $
\end{theorem}

\begin{example}
$ g: \mathbb{R} \rightarrow \mathbb{R}, g(x)=x+1$ is firmly non-expansive in fact $ g(x)=prox_{f}(x), f(x)=-x$
	\end{example}

\subsection{Proximal gradient method}
Proximal point algorithm:
$$
x_{n+1}=prox_{f}(x_{n}).
$$
By the preceding if $ prox_{f}$ has a fixed point, then $ x_{n} $ converge to a fixed point.
A fixed point of $ prox_{f}$ is a point x with $0 \in \partial f(x) $, which is a minimizer.
Usually calculating $ prox_{f}$ is just as difficult as minimizing $f$ itself,
$$
prox_{f}=\mathop{argmin}\limits_{z} \frac{1}{2}\|x-z\|_2^{2}+f(z)
$$
%Later we will use 	$ prox_{f}$ and $ prox_{g}$ to construct a firmly non-expansive map whose fixed points mininize $g+h $ Douglas-Rachfound splitting.
%$$
%x_{n+1}=x_{n}-s h_{n}, \quad h_{n}=\mathop{argmax}\limits_{\|h\|_{1}}\left\langle h,\nabla f\left(x_{n}\right)\right\rangle
%$$
%Boosting: $$ \mathop{argmax}\limits_{h \in D}\left\langle h,\nabla f\left(x_{n}\right)\right\rangle $$
%If $
%f(x)=\frac{1}{2}\left\|x-x_{0}\right\|^{2}
%$get greedy approx.

Consider unconstrained problem with cost function split in two components
$$
\min f(x)=g(x)+h(x)
$$
where $h$ is convex and differentiable,
$g$ is convex and possibly nondifferentiable, with inexpensive pro-operator.
The proximal gradient algorithm is
$$
x^{(k)}=\operatorname{prox}_{t_{k} g}\left(x^{(k-1)}-t_{k} \nabla h\left(x^{(k-1)}\right)\right)
$$
where $t_{k}>0$ is step size, constant or determined by line search. Note that if $g(x)=0$, then this method reduces to a gradient descent method.

If we apply the proximal mapping to update rule,
$$
\begin{aligned}
x^{+} &=\underset{z}{\operatorname{argmin}}\left(g(z)+\frac{1}{2 t}\|z-x+t \nabla h(x)\|_{2}^{2}\right) \\
&=\underset{z}{\operatorname{argmin}}\left(g(z)+h(x)+\nabla h(x)^{T}(z-x)+\frac{1}{2 t}\|z-x\|_{2}^{2}\right).
\end{aligned}
$$
This means that
$x^{+}$ minimizes $g(z)$ plus a simple quadratic local model of $h(z)$ around $x$.

\begin{properties}
If
$$
u=\operatorname{prox}_{f}(x), v= \operatorname{prox}_{f}(y),
$$
then
$$
(u-v)^{T}(x-y) \geq\|u-v\|_{2}^{2}.
$$
This implies that
$$
x-u \in \partial f(u), \quad y-v \in \partial f(v) \quad \Longrightarrow \quad(x-u-y+v)^{T}(u-v) \geq 0,
$$
and
$$
\left\|\operatorname{prox}_{f}(x)-\operatorname{prox}_{f}(y)\right\|_{2} \leq\|x-y\|_{2}
$$
$prox_{f}$ is nonexpansive or Lipschitz continuous with constant 1.
\end{properties}

\begin{definition}
	 The proximal map of X, $ prox_{X}(x)=x^{1} $(" Backward Euler")
\end{definition}

%\begin{remark}
%\begin{enumerate}
%\item If the full gradient descent method is used, $\eta$ is a given value.
%\item If the stochastic  gradient descent method is used, $\eta$ is approaching 0 as t goes to infinity. We don't expect sparse 'minimization sequence'
%\item Can we find a different training algorithm that would give 'sparsity'?
%\end{enumerate}
%\end{remark}

%\section{Forward-Backward Splitting Algorithm(not included)}
%\begin{lemma}
%$\Theta=\{\theta:f(\theta,x)\ \text{ is a classifier}\}$ is a non-empty open set.
%\end{lemma}
%
%
%\begin{remark}
%\begin{enumerate}
%\item Question 1: Can we find some sparse $\theta\in \Theta$ by certain training algorithm?
%\begin{itemize}
%\item The desirable training algorithm should converge.
%\item The desirable training algorithm generates 'sparse' minimization sequence.
%\end{itemize}
%
%\item Question 2: Can we find a model $L(\theta)+R(\theta)$ with small $R(\theta)$ s.t. $\arg\min(L(\theta)+R(\theta)) \approx \arg\min L(\theta)$.
%\end{enumerate}
%\end{remark}
%A general approach is to use $L(\theta)+R(\theta)$, where $R(\theta)=\lambda \|\theta\|_1$.
%
%Assume f is strongly convex and smooth function and R is convex. Thus $x^*= \arg\min (f(x)+R(x)) \iff 0\in \nabla f(x^*)+\partial R(x^*)$.
%We have the following forward-backward splitting algorithm.

%\section{A summary of algorithms and convergence results}
%\includepdf[pages=-,angle=270]{6DL/SGD-summary.jpeg}


\section{Isomorphisms}

\begin{outcome}
  \begin{enumerate}
  \item Determine if a linear transformation is an isomorphism.
  \item Determine if two subspaces of $\R^n$ are isomorphic.
  \end{enumerate}
\end{outcome}

Recall the definition of a linear transformation. Let $V$ and $W$ be two subspaces of $\R^{n}$ and $\R^{m}$
respectively. A mapping $T:V\rightarrow W$ is called a\textbf{\ linear
transformation} or \textbf{linear map} if it preserves the algebraic
operations of addition and
\index{linear map}
\index{linear transformation} scalar multiplication. Specifically, if $a,b$
are scalars and $
\vect{x},\vect{y}$ are vectors,
\begin{equation*}
T(a\vect{x}+b\vect{y}) =aT(\vect{x})+bT(\vect{y})
\end{equation*}

Consider the following important definition.

\begin{definition}{Isomorphism}{isomorphism}
A linear map $T$ is called an \textbf{isomorphism}
\index{isomorphism}if the following two conditions are satisfied.

\begin{itemize}
\item $T$ is one to one. That is, if $T(\vect{x})=T(\vect{y})$, then $\vect{x}=\vect{y}$.

\item $T$ is onto. That is, if $\vect{w}\in W$, there exists $
\vect{v}\in V$ such that $T(\vect{v})=\vect{w}$.
\end{itemize}

Two such subspaces which have an isomorphism as described above are said to
be \textbf{isomorphic.}
\index{isomorphic}
\end{definition}

Consider the following example of an isomorphism.

\begin{example}{Isomorphism}{isomorphism}
Let $T: \R^2 \to \R^2$ be defined by
\[
T \begin{mymatrix}{c}
x \\
y
\end{mymatrix} = \begin{mymatrix}{c}
x + y \\
x - y
\end{mymatrix}
\]
Show that $T$ is an isomorphism.
\end{example}

\begin{solution}
To prove that $T$ is an isomorphism we must show
\begin{enumerate}
\item $T$ is a linear transformation;
\item $T$ is one to one;
\item $T$ is onto.
\end{enumerate}

We proceed as follows.

\begin{enumerate}
\item $T$ is a linear transformation:

Let $k, p$ be scalars.
\begin{eqnarray*}
T \paren{k \begin{mymatrix}{c}
x_1 \\
y_1
\end{mymatrix} + p \begin{mymatrix}{c}
x_2 \\
y_2
\end{mymatrix}} &=&
T \paren{\begin{mymatrix}{c}
kx_1 \\
ky_1
\end{mymatrix} + \begin{mymatrix}{c}
px_2 \\
py_2
\end{mymatrix}} \\
&=&
T \paren{\begin{mymatrix}{c}
kx_1 + px_2 \\
ky_1 + py_2
\end{mymatrix}} \\
&=&
\begin{mymatrix}{c}
(kx_1 + px_2) + (ky_1 + py_2) \\
(kx_1 + px_2) - (ky_1 + py_2)
\end{mymatrix} \\
&=&
\begin{mymatrix}{c}
(kx_1 + ky_1) + (px_2 + py_2) \\
(kx_1  - ky_1) + (px_2 - py_2)
\end{mymatrix} \\
&=&
\begin{mymatrix}{c}
kx_1 + ky_1  \\
kx_1  - ky_1
\end{mymatrix} +
\begin{mymatrix}{c}
px_2 + py_2 \\
px_2 - py_2
\end{mymatrix} \\
&=& k \begin{mymatrix}{c}
x_1 + y_1  \\
x_1  - y_1
\end{mymatrix} +
p \begin{mymatrix}{c}
x_2 + y_2 \\
x_2 - y_2
\end{mymatrix} \\
&=&
k T \paren{\begin{mymatrix}{c}
x_1 \\
y_1
\end{mymatrix}} + p T \paren{\begin{mymatrix}{c}
x_2 \\
y_2
\end{mymatrix}}
\end{eqnarray*}

Therefore $T$ is linear.

\item $T$ is one to one:

We need to show that if $T (\vect{x}) = \vect{0}$ for a vector $\vect{x} \in \R^2$, then it follows that $\vect{x} = \vect{0}$.  Let $\vect{x} = \begin{mymatrix}{c}
x \\
y
\end{mymatrix}$.

\[
T  \paren{\begin{mymatrix}{c}
x \\
y
\end{mymatrix}} = \begin{mymatrix}{c}
x + y\\
x - y
\end{mymatrix} = \begin{mymatrix}{c}
0 \\
0
\end{mymatrix}
\]
This provides a system of equations given by
\begin{eqnarray*}
x + y &=& 0\\
x - y &=& 0
\end{eqnarray*}
You can verify that the solution to this system if $x = y =0$. Therefore
\[
\vect{x} = \begin{mymatrix}{c}
x \\
y
\end{mymatrix}
 = \begin{mymatrix}{c}
0 \\
0
\end{mymatrix}
\]
and $T$ is one to one.

\item $T$ is onto:

Let $a,b$ be scalars. We want to check if there is always a solution to
\[
T  \paren{\begin{mymatrix}{c}
x \\
y
\end{mymatrix}} = \begin{mymatrix}{c}
x + y\\
x - y
\end{mymatrix} = \begin{mymatrix}{c}
a \\
b
\end{mymatrix}
\]

This can be represented as the system of equations
\begin{eqnarray*}
x + y &=& a\\
x - y &=& b
\end{eqnarray*}

Setting up the augmented matrix and row reducing gives
\[
\begin{mymatrix}{cc|c}
1 & 1 & a \\
1 & -1 & b
\end{mymatrix} \roweq\ldots\roweq
\begin{mymatrix}{cc|c}
1 & 0 & \vspace{0.05in}\frac{a+b}{2} \\
0 & 1 & \vspace{0.05in}\frac{a-b}{2}
\end{mymatrix}
\]
This has a solution for all $a,b$ and therefore $T$ is onto.
\end{enumerate}

Therefore $T$ is an isomorphism.
\end{solution}

An important property of isomorphisms is that its inverse is also an isomorphism.

\begin{proposition}{Inverse of an isomorphism}{inverse-isomorphism}
Let $T:V\rightarrow W$ be an isomorphism and $V,W$ be subspaces of $\R^n$. Then $T^{-1}:W\rightarrow V$ is
also an isomorphism.
\index{isomorphism!inverse}
\end{proposition}

\begin{proof} Let $T$ be an isomorphism.  Since $T$ is onto, a typical
vector in $W$ is of the form $T(\vect{v})$ where $\vect{v} \in V$. Consider then for $a,b$
scalars,
\begin{equation*}
T^{-1}(aT(\vect{v}_{1})+bT(\vect{v}_{2}))
\end{equation*}
where $\vect{v}_{1}, \vect{v}_2 \in V$. Is this equal to
\begin{equation*}
aT^{-1}(T (\vect{v}_{1})) +bT^{-1}(T(\vect{v}_{2})) =a\vect{v}_{1}+b\vect{v}_{2}?
\end{equation*}
Since $T$ is one to one, this will be so if
\begin{equation*}
T(a\vect{v}_{1}+b\vect{v}_{2}) =T(T^{-1}(aT(\vect{v}_{1})+bT(\vect{v}_{2}))
) =aT(\vect{v}_{1})+bT(\vect{v}_{2}).
\end{equation*}
However, the above statement is just the condition that $T$ is a linear map.
Thus $T^{-1}$ is indeed a linear map. If $\vect{v} \in V$ is given, then $\vect{v}=T^{-1}(T(\vect{v})) $ and so $T^{-1}$ is onto. If $T^{-1} (\vect{v})=0$, then
\begin{equation*}
\vect{v}=T(T^{-1}(\vect{v})) =T(\vect{0})=\vect{0}
\end{equation*}
and so $T^{-1}$ is one to one.
\end{proof}

Another important result is that the composition of multiple isomorphisms is also an isomorphism.

\begin{proposition}{Composition of isomorphisms}{composition-isomorphisms}
Let $T:V\rightarrow W$ and  $S:W\rightarrow Z$ be isomorphisms where $V,W,Z$ are subspaces of $\R^n$. Then $S\circ
T $ defined by $(S\circ T) (\vect{v}) = S(
T(\vect{v})) $ is also an isomorphism.
\index{isomorphism!composition}
\end{proposition}

\begin{proof}
Suppose $T:V\rightarrow W$ and  $S:W\rightarrow Z$ are isomorphisms. Why is $S\circ T$ a linear map?
For $a,b$ scalars,
\begin{eqnarray*}
S\circ T(a\vect{v}_{1}+b(\vect{v}_{2}))
&=& S(T(a\vect{v}_{1}+b\vect{v}_{2})) =S(aT\vect{v}_{1}+bT\vect{v}_{2}) \\
&=&aS(T\vect{v}_{1}) +bS(T\vect{v}_{2}) = a(S\circ
T) (\vect{v}_{1}) +b(S\circ T) (\vect{v}_{2})
\end{eqnarray*}
Hence $S\circ T$ is a linear map. If $(S\circ T) (\vect{v})
=0$, then $S(T(\vect{v})) =0$ and it follows that $T(\vect{v})=\vect{0}$ and hence by this lemma again, $\vect{v}=\vect{0}$. Thus $S\circ
T $ is one to one. It remains to verify that it is onto. Let $\vect{z} \in Z$. Then
since $S$ is onto, there exists $\vect{w} \in W$ such that $S(\vect{w})=\vect{z}$. Also, since $T$
is onto, there exists $\vect{v}\in V$ such that $T(\vect{v})=\vect{w}$. It follows that $S(
T(\vect{v})) =\vect{z}$ and so $S\circ T$ is also onto.
\end{proof}

Consider two subspaces $V$ and $W$, and suppose there exists an isomorphism mapping one to the other. In this way the two subspaces are related, which we can write as $V\iso W$. Then the previous two propositions together claim that $\iso$ is an equivalence relation. That is: $\iso$
satisfies the following conditions:
\index{isomorphic!equivalence relation}

\begin{itemize}
\item $V\iso V$

\item If $V\iso W$, it follows that $W\iso V$

\item If $V\iso W$ and $W\iso Z$, then $V\iso Z$
\end{itemize}

We leave the verification of these conditions as an exercise.

Consider the following example.

\begin{example}{Matrix isomorphism}{matrix-isomorphism}
Let $T:\R^{n}\rightarrow \R^{n}$ be defined by $T(\vect{x}) = A(\vect{x})$ where $A$ is an invertible $n\times n$-matrix. Then $T$ is
an isomorphism.
\end{example}

\begin{solution}
The reason for this is that, since $A$ is invertible, the only vector it
sends to $\vect{0}$ is the zero vector. Hence if $A(\vect{x})=A(\vect{y})$, then $A(\vect{x}-\vect{y}) =\vect{0}$ and so $\vect{x}=\vect{y}$. It is onto
because if $\vect{y}\in \R^{n},A(A^{-1} (\vect{y})) =(
AA^{-1}) (\vect{y})$ $=\vect{y}$.
\end{solution}

In fact, all isomorphisms from $\R^{n}$ to $\R^{n}$ can be expressed as $T(\vect{x}) = A(\vect{x})$ where $A$ is an invertible $n \times n$-matrix. One
simply considers the matrix whose $i\th$ column is $T\vect{e}_{i}$.

Recall that a basis of a subspace $V$ is a set of linearly independent vectors which span $V$. The following fundamental lemma describes the relation between bases and
isomorphisms.

\begin{lemma}{Mapping bases}{mapping-bases}
Let $T:V\rightarrow W$ be a linear transformation where $V,W$ are
subspaces of $\R^n$. If $T$ is one to one, then it has the property that if $\set{\vect{u}_{1},\ldots,\vect{u}_{k}} $ is linearly independent, so is $\set{T(\vect{u}_{1}),\ldots,T(\vect{u}_{k})}$.

More generally, $T$ is an isomorphism if and only if whenever $\set{
\vect{v}_{1},\ldots,\vect{v}_{n}} $ is a basis for $V$, it follows that $\set{T
(\vect{v}_{1}),\ldots,T(\vect{v}_{n})} $ is a basis for $W$.
\index{isomorphism!bases}
\index{one to one!linear independence}
\end{lemma}

\begin{proof}First suppose that $T$ is a linear transformation and is one to one
and $\set{\vect{u}_{1},\ldots,\vect{u}_{k}} $ is linearly
independent. It is required to show that $\set{T(\vect{u}_{1}),\ldots,T(\vect{
u}_{k})} $ is also linearly independent. Suppose then that
\begin{equation*}
\sum_{i=1}^{k}c_{i}T(\vect{u}_{i})=\vect{0}
\end{equation*}
Then, since $T$ is linear,
\begin{equation*}
T\paren{\sum_{i=1}^{n}c_{i}\vect{u}_{i}} =\vect{0}
\end{equation*}
Since $T$ is one to one, it follows that
\begin{equation*}
\sum_{i=1}^{n}c_{i}\vect{u}_{i}=0
\end{equation*}
Now the fact that $\set{\vect{u}_{1},\ldots,\vect{u}_{n}} $ is
linearly independent implies that each $c_{i}=0$. Hence $\set{T(\vect{u}
_{1}),\ldots,T(\vect{u}_{n})} $ is linearly independent.

Now suppose that $T$ is an isomorphism and $\set{\vect{v}_{1},\ldots,\vect{
v}_{n}} $ is a basis for $V$. It was just shown that $\set{T(\vect{v}
_{1}),\ldots,T(\vect{v}_{n})} $ is linearly independent. It remains to
verify that span$\set{T(\vect{v}_{1}),\ldots,T(\vect{v}_{n})}=W$. If $\vect{w}\in W$, then since $T$ is onto there
exists $\vect{v}\in V$ such that $T(\vect{v})=\vect{w}$. Since $\set{\vect{v}
_{1},\ldots,\vect{v}_{n}} $ is a basis, it follows that there exists
scalars $\set{c_{i}} _{i=1}^{n}$ such that
\begin{equation*}
\sum_{i=1}^{n}c_{i}\vect{v}_{i}=\vect{v}.
\end{equation*}
Hence,
\begin{equation*}
\vect{w}=T(\vect{v})=T\paren{\sum_{i=1}^{n}c_{i}\vect{v}_{i}}
=\sum_{i=1}^{n}c_{i}T(\vect{v}_{i})
\end{equation*}
It follows that span$\set{T(\vect{v}_{1}),\ldots, T(\vect{v}_{n})} =W$ showing that this set of vectors is a
basis for $W$.

Next suppose that $T$ is a linear transformation which takes a basis to a basis. This means that if $\set{\vect{v}_{1},\ldots,\vect{v}_{n}} $ is a basis for $V$, it
follows $\set{T(\vect{v}_{1}),\ldots,T(\vect{v}_{n})} $ is a basis for $
W$. Then if $w\in W$, there exist scalars $c_{i}$ such that $
w=\sum_{i=1}^{n}c_{i}T(\vect{v}_{i})=T\paren{\sum_{i=1}^{n}c_{i}\vect{v}_{i}} $
showing that $T$ is onto. If $T\paren{\sum_{i=1}^{n}c_{i}\vect{v}_{i}} =\vect{0}$
then $\sum_{i=1}^{n}c_{i}T(\vect{v}_{i})=\vect{0}$ and since the vectors $\set{T(\vect{v}
_{1}),\ldots,T(\vect{v}_{n})} $ are linearly independent, it follows
that each $c_{i}=0$. Since $\sum_{i=1}^{n}c_{i}\vect{v}_{i}$ is a typical vector in
$V$, this has shown that if $T(\vect{v})=\vect{0}$ then $\vect{v}=\vect{0}$ and so $T$ is also one to one.
Thus $T$ is an isomorphism.
\end{proof}

The following theorem illustrates a very useful idea for defining an
isomorphism. Basically, if you know what it does to a basis, then you can
construct the isomorphism.

\begin{theorem}{Isomorphic subspaces}{isomorphic-subspaces}
Suppose $V$ and $W$ are two subspaces of $\R^n$. Then the two
subspaces are isomorphic if and only if they have the same dimension. In the
case that the two subspaces have the same dimension, then for
\index{isomorphism!equivalence} a linear map $T:V\rightarrow W$, the
following are equivalent.

\begin{enumerate}
\item $T$ is one to one.

\item $T$ is onto.

\item $T$ is an isomorphism.
\end{enumerate}
\end{theorem}

\begin{proof} Suppose first that these two subspaces have the same
dimension. Let a basis for $V$ be $\set{
\vect{v}_{1},\ldots,\vect{v}_{n}} $ and let a basis for $W$ be $
\set{\vect{w}_{1},\ldots,\vect{w}_{n}}$. Now define $T$ as
follows.
\begin{equation*}
T(\vect{v}_{i})=\vect{w}_{i}
\end{equation*}
for $\sum_{i=1}^{n}c_{i}\vect{v}_{i}$ an arbitrary vector of $V$,
\begin{equation*}
T\paren{\sum_{i=1}^{n}c_{i}\vect{v}_{i}} = \sum_{i=1}^{n}c_{i}T
\vect{v}_{i}=\sum_{i=1}^{n}c_{i}\vect{w}_{i}.
\end{equation*}
It is necessary to verify that this is well-defined. Suppose then that
\begin{equation*}
\sum_{i=1}^{n}c_{i}\vect{v}_{i}=\sum_{i=1}^{n}\hat{c}_{i}\vect{v}_{i}
\end{equation*}
Then
\begin{equation*}
\sum_{i=1}^{n}(c_{i}-\hat{c}_{i}) \vect{v}_{i}=\vect{0}
\end{equation*}
and since $\set{\vect{v}_{1},\ldots,\vect{v}_{n}} $ is a basis, $
c_{i}=\hat{c}_{i}$ for each $i$. Hence
\begin{equation*}
\sum_{i=1}^{n}c_{i}\vect{w}_{i}=\sum_{i=1}^{n}\hat{c}_{i}\vect{w}_{i}
\end{equation*}
and so the mapping is well-defined. Also if $a,b$ are scalars,
\begin{eqnarray*}
T\paren{a\sum_{i=1}^{n}c_{i}\vect{v}_{i}+b\sum_{i=1}^{n}\hat{c}_{i}\vect{v}%
_{i}} &=&T(\sum_{i=1}^{n}(ac_{i}+b\hat{c}_{i}) \vect{v%
}_{i}) =\sum_{i=1}^{n}(ac_{i}+b\hat{c}_{i}) \vect{w}_{i} \\
&=&a\sum_{i=1}^{n}c_{i}\vect{w}_{i}+b\sum_{i=1}^{n}\hat{c}_{i}\vect{w}_{i} \\
&=&aT\paren{\sum_{i=1}^{n}c_{i}\vect{v}_{i}} +bT\paren{\sum_{i=1}^{n}%
\hat{c}_{i}\vect{v}_{i}}
\end{eqnarray*}
Thus $T$ is a linear transformation.

Now if
\begin{equation*}
T\paren{\sum_{i=1}^{n}c_{i}\vect{v}_{i}} =\sum_{i=1}^{n}c_{i}\vect{w}%
_{i}=\vect{0},
\end{equation*}
then since the $\set{\vect{w}_{1},\ldots,\vect{w}_{n}} $ are
independent, each $c_{i}=0$ and so $\sum_{i=1}^{n}c_{i}\vect{v}_{i}=\vect{0}$
also. Hence $T$ is one to one. If $\sum_{i=1}^{n}c_{i}\vect{w}_{i}$ is a
vector in $W$, then it equals
\begin{equation*}
\sum_{i=1}^{n}c_{i}T(\vect{v}_{i})=T\paren{\sum_{i=1}^{n}c_{i}\vect{v}_{i}}
\end{equation*}
showing that $T$ is also onto. Hence $T$ is an isomorphism and so $V$ and $W$
are isomorphic.

Next suppose $T:V \to W$ is an isomorphism, so these two subspaces are isomorphic. Then for $\set{\vect{v}_{1},\ldots,\vect{v}_{n}} $ a
basis for $V$, it follows that a basis for $W$
is $\set{T(\vect{v}_{1}),\ldots,T(\vect{v}_{n})} $ showing that the two
subspaces have the same dimension.

Now suppose the two subspaces have the same dimension. Consider the three
claimed equivalences.

First consider the claim that $1.)\Rightarrow 2.)$. If $T$ is one to one and if $\set{\vect{v}_{1},\ldots,\vect{v}
_{n}} $ is a basis for $V$, then $\set{T(\vect{v}_{1}),\ldots,T(\vect{v
}_{n})} $ is linearly independent. If it is not a basis, then it must
fail to span $W$. But then there would exist $\vect{w}\notin \sspan
\set{T(\vect{v}_{1}),\ldots,T(\vect{v}_{n})} $ and it follows that $\set{T(\vect{v}_{1}),\ldots,T(\vect{v}_{n}),\vect{w}
} $ would be linearly independent which is impossible because there exists a basis for $W$ of $n$ vectors.

Hence $\sspan\set{T(\vect{v}_{1}),\ldots,T(\vect{v}_{n})} =W$ and
so $\set{T(\vect{v}_{1}),\ldots,T(\vect{v}_{n})} $ is a basis. If $\vect{w}\in W$, there exist scalars $c_{i}$ such that
\begin{equation*}
\vect{w}=\sum_{i=1}^{n}c_{i}T(\vect{v}_{i})=T\paren{\sum_{i=1}^{n}c_{i}\vect{v}
_{i}}
\end{equation*}
showing that $T$ is onto. This shows that $1.)\Rightarrow 2.)$.

Next consider the claim that $2.)\Rightarrow 3.)$. Since $2.)$ holds, it
follows that $T$ is onto. It remains to verify that $T$ is one to one. Since
$T$ is onto, there exists a basis of the form $\set{T(\vect{v}_{i}),\ldots,T(\vect{v}_{n})}$. Then it follows that $\set{\vect{v}_{1},\ldots,
\vect{v}_{n}} $ is linearly independent. Suppose
\begin{equation*}
\sum_{i=1}^{n}c_{i}\vect{v}_{i}=\vect{0}
\end{equation*}
Then
\begin{equation*}
\sum_{i=1}^{n}c_{i}T(\vect{v}_{i})=\vect{0}
\end{equation*}
Hence each $c_{i}=0$ and so, $\set{\vect{v}_{1},\ldots,\vect{v}
_{n}} $ is a basis for $V$. Now it follows that a typical vector in $
V $ is of the form $\sum_{i=1}^{n}c_{i}\vect{v}_{i}$. If $T\paren{
\sum_{i=1}^{n}c_{i}\vect{v}_{i}} =\vect{0}$, it follows that
\begin{equation*}
\sum_{i=1}^{n}c_{i}T(\vect{v}_{i})=\vect{0}
\end{equation*}
and so, since $\set{T(\vect{v}_{i}),\ldots,T(\vect{v}_{n})} $ is
independent, it follows each $c_{i}=0$ and hence $\sum_{i=1}^{n}c_{i}\vect{v}
_{i}=\vect{0}$. Thus $T$ is one to one as well as onto and so it is an
isomorphism.

If $T$ is an isomorphism, it is both one to one and onto by definition so $
3.)$ implies both $1.)$ and $2.)$.
\end{proof}

Note the interesting way of defining a linear transformation in the first
part of the argument by describing what it does to a basis and then
``extending it linearly'' to the entire subspace.
\index{linear map!defining on a basis}

\begin{example}{Isomorphic subspaces}{}
Let $V=\R^{3}$ and let $W$ denote
\begin{equation*}
\sspan\set{\begin{mymatrix}{r}
1 \\
2 \\
1 \\
1
\end{mymatrix} ,\begin{mymatrix}{r}
0 \\
1 \\
0 \\
1
\end{mymatrix} ,\begin{mymatrix}{r}
1 \\
1 \\
2 \\
0
\end{mymatrix} }
\end{equation*}
Show that $V$ and $W$ are isomorphic.
\end{example}

\begin{solution}
First observe that these subspaces are both of dimension 3 and so they are isomorphic by Theorem~\ref{thm:isomorphic-subspaces}. The
three vectors which span $W$ are easily seen to be linearly independent by
making them the columns of a matrix and row reducing to the {\rref}.

You can exhibit an isomorphism of these two spaces as follows.
\begin{equation*}
T(\vect{e}_{1})=\begin{mymatrix}{c}
1 \\
2 \\
1 \\
1
\end{mymatrix}, T(\vect{e}_{2})=\begin{mymatrix}{c}
0 \\
1 \\
0 \\
1
\end{mymatrix}, T(\vect{e}_{3})=\begin{mymatrix}{c}
1 \\
1 \\
2 \\
0
\end{mymatrix}
\end{equation*}
and extend linearly. Recall that the matrix of this linear transformation is
just the matrix having these vectors as columns. Thus the matrix of this
isomorphism is
\begin{equation*}
\begin{mymatrix}{rrr}
1 & 0 & 1 \\
2 & 1 & 1 \\
1 & 0 & 2 \\
1 & 1 & 0
\end{mymatrix}
\end{equation*}
You should check that multiplication on the left by this matrix does
reproduce the claimed effect resulting from an application by $T$.
\end{solution}

Consider the following example.

\begin{example}{Finding the matrix of an isomorphism}{matrix-of-isomorphism}
Let $V=\R^{3}$ and let $W$ denote
\begin{equation*}
\sspan\set{\begin{mymatrix}{c}
1 \\
2 \\
1 \\
1
\end{mymatrix} ,\begin{mymatrix}{c}
0 \\
1 \\
0 \\
1
\end{mymatrix} ,\begin{mymatrix}{c}
1 \\
1 \\
2 \\
0
\end{mymatrix} }
\end{equation*}

Let $T: V \to W$ be defined as follows.
\begin{equation*}
T\begin{mymatrix}{c}
1 \\
1 \\
0
\end{mymatrix} =\begin{mymatrix}{c}
1 \\
2 \\
1 \\
1
\end{mymatrix} ,T\begin{mymatrix}{c}
0 \\
1 \\
1
\end{mymatrix} =\begin{mymatrix}{c}
0 \\
1 \\
0 \\
1
\end{mymatrix} ,T\begin{mymatrix}{c}
1 \\
1 \\
1
\end{mymatrix} =\begin{mymatrix}{c}
1 \\
1 \\
2 \\
0
\end{mymatrix}
\end{equation*}
Find the matrix of this isomorphism $T$.
\end{example}

\begin{solution}
 First note that the vectors
\begin{equation*}
\begin{mymatrix}{c}
1 \\
1 \\
0
\end{mymatrix} ,\begin{mymatrix}{c}
0 \\
1 \\
1
\end{mymatrix} ,\begin{mymatrix}{c}
1 \\
1 \\
1
\end{mymatrix}
\end{equation*}
are indeed a basis for $\R^{3}$ as can be seen by making them the
columns of a matrix and using the {\rref}.

Now recall the matrix of $T$ is a $4\times 3$-matrix $A$ which gives the same
effect as $T$. Thus, from the way we multiply matrices,
\begin{equation*}
A\begin{mymatrix}{rrr}
1 & 0 & 1 \\
1 & 1 & 1 \\
0 & 1 & 1
\end{mymatrix} =\begin{mymatrix}{rrr}
1 & 0 & 1 \\
2 & 1 & 1 \\
1 & 0 & 2 \\
1 & 1 & 0
\end{mymatrix}
\end{equation*}
Hence,
\begin{equation*}
A=\begin{mymatrix}{rrr}
1 & 0 & 1 \\
2 & 1 & 1 \\
1 & 0 & 2 \\
1 & 1 & 0
\end{mymatrix} \begin{mymatrix}{rrr}
1 & 0 & 1 \\
1 & 1 & 1 \\
0 & 1 & 1
\end{mymatrix} ^{-1}=\begin{mymatrix}{rrr}
1 & 0 & 0 \\
0 & 2 & -1 \\
2 & -1 & 1 \\
-1 & 2 & -1
\end{mymatrix}
\end{equation*}
Note how the span of the columns of this new matrix must be the same as the
span of the vectors defining $W$.
\end{solution}

This idea of defining a linear transformation by what it does on a basis
works for linear maps which are not necessarily isomorphisms.

\begin{example}{Finding the matrix of an isomorphism}{}
Let $V=\R^{3}$ and let $W$ denote
\begin{equation*}
\sspan\set{\begin{mymatrix}{c}
1 \\
0 \\
1 \\
1
\end{mymatrix} ,\begin{mymatrix}{c}
0 \\
1 \\
0 \\
1
\end{mymatrix} ,\begin{mymatrix}{c}
1 \\
1 \\
1 \\
2
\end{mymatrix} }
\end{equation*}
Let $T: V \to W$ be defined as follows.
\begin{equation*}
T\begin{mymatrix}{c}
1 \\
1 \\
0
\end{mymatrix} = \begin{mymatrix}{c}
1 \\
0 \\
1 \\
1
\end{mymatrix} ,T\begin{mymatrix}{c}
0 \\
1 \\
1
\end{mymatrix} =\begin{mymatrix}{c}
0 \\
1 \\
0 \\
1
\end{mymatrix} ,T\begin{mymatrix}{c}
1 \\
1 \\
1
\end{mymatrix} =\begin{mymatrix}{c}
1 \\
1 \\
1 \\
2
\end{mymatrix}
\end{equation*}
 Find the matrix of this linear transformation.
\end{example}

\begin{solution}
Note that in this case, the three vectors which span $W$ are not linearly independent. Nevertheless the above procedure will still work.
The reasoning is the same as before. If $A$ is this matrix, then
\begin{equation*}
A\begin{mymatrix}{rrr}
1 & 0 & 1 \\
1 & 1 & 1 \\
0 & 1 & 1
\end{mymatrix} =\begin{mymatrix}{rrr}
1 & 0 & 1 \\
0 & 1 & 1 \\
1 & 0 & 1 \\
1 & 1 & 2
\end{mymatrix}
\end{equation*}
and so
\begin{equation*}
A=\begin{mymatrix}{rrr}
1 & 0 & 1 \\
0 & 1 & 1 \\
1 & 0 & 1 \\
1 & 1 & 2
\end{mymatrix} \begin{mymatrix}{rrr}
1 & 0 & 1 \\
1 & 1 & 1 \\
0 & 1 & 1
\end{mymatrix} ^{-1}=\begin{mymatrix}{rrr}
1 & 0 & 0 \\
0 & 0 & 1 \\
1 & 0 & 0 \\
1 & 0 & 1
\end{mymatrix}
\end{equation*}
The columns of this last matrix are obviously not linearly independent.
\end{solution}