\section{Logarithms}

\begin{frame}{Base of Logarithms}
  \begin{block}{Logarithm to different bases:}
    \begin{displaymath}
      \log_a n = \dfrac{\log_b n}{\log_b a}
      = \log_b n \cdot \dfrac{1}{\log_b a}
    \end{displaymath}\\[0.5em]
    The only difference is a constant coefficient $\frac{1}{\log_b a}$
  \end{block}
  \textbf{Examples:}
  \begin{itemize}
    \item
      \begin{math}
        \log_2 4
        = \log_{10} 4 \cdot \frac{1}{\log_2 10}
        = 0.602 \ldots \cdot 3.322 \ldots
        = 2 ~{\color{Mittel-Blau}\checkmark}
      \end{math}
    \item
      \begin{math}
        \log_{10} 1000
        = \log_{\mathrm{e}} 1000 \cdot \frac{1}{\log_{\mathrm{e}} 10}
        = \ln 1000 \cdot \frac{1}{\ln 10}
        = 3 ~{\color{Mittel-Blau}\checkmark}
      \end{math}
  \end{itemize}
\end{frame}

%-------------------------------------------------------------------------------

\begin{frame}{Runtime Example}
  \textbf{Runtime of $n \; \log_2 n$:}
  \begin{itemize}
    \item<1- |handout:1>
      Assume we have constants $\color{MainA}C_1$ and $\color{MainA}C_2$
      with
      \begin{displaymath}
        \textcolor{MainA}{
          C_1 \cdot n \cdot \log_2 n \leq T(n) \leq C_2 \cdot n \cdot log_2 n
          \hspace{1.5em}\text{for}~n \geq 2
        }
      \end{displaymath}
    \item<2- |handout:1>
      $2\, \times$ elements
      $\Rightarrow$ only slightly larger than $2\, \times$ runtime\\
      \begin{itemize}
        \item<3- |handout:1>
          $C = \SI{1}{\nano\second}$
          (1 simple instruction $\approx \SI{1}{\nano\second}$)
        \item<4- |handout:1>
          $n = 2^{20}$ (1 million numbers = $\SI{4}{\mega\byte}$
          with $\SI{4}{\byte\per number}$)
          \begin{itemize}
            \item
              $C \cdot n \cdot log_2 n = \SI{e-9}{\second} \cdot 2^{20} \cdot 20
              = \SI{21.0}{\milli\second}$
          \end{itemize}
        \item<5- |handout:1>
          $n = 2^{30}$ (1 billion numbers = $\SI{4}{\giga\byte}$)
          \begin{itemize}
            \item
              $C \cdot n \cdot log_2 n = \SI{e-9}{\second} \cdot 2^{30} \cdot 30
              = \SI{32}{\second}$
          \end{itemize}
      \end{itemize}
    \item<6- |handout:1>
      \textbf{Runtime $n \; \log_2 n$ is nearly as good as linear!}
  \end{itemize}
\end{frame}


\chapter{Newtonian Gravity}
In \name{Newton}ian physics we assume that we have absolute space and time
that can be described by the a set of numbers $x^1,x^2,x^3,t$.
We express the coordinates as functions of time.
\section{Forces}
The force $\vec{F}_{AB}$, which a massive body $A$ with mass $m_A$ exerts on another massive body $B$ with mass $m_B$, is given by
\begin{equation}
    \vec{F}_{AB}=-m_B\frac{G\textsubscript{N}m_A}{r^2}\vec{e}_r\, ,
\end{equation}
where $G\textsubscript{N}$ denotes \emph{\name{Newton}'s constant},
numerically equal to $G\textsubscript{N}\approx \unitfrac[6.673\cdot 10^{-11}]{m^3}{kg\,s}$.
Although there is no need for $G\textsubscript{N}$ to be constant over time,
there is evidence that the relative variation is less than $10^{-12}$ per year.
The force can be expressed in terms of \emph{gravitational potential} $\Phi$:
\begin{equation}
    \vec{F}_{AB}=-m_B\nabla\left(-\frac{G\textsubscript{N}m_A}{r}\right)=:
    -m_B\nabla\Phi(\vec{r}_B)\, .
\end{equation}
Given $N$ particles labeled by $n$, the total force $B$ experiences is
\begin{equation}
    \vec{F}_{B}=-\sum_n \vec{F}_{nB}\, .
\end{equation}
The potential at $\vec{r}$ is then easily found to be
\begin{equation}
    \Phi(\vec{r})=-G\textsubscript{N}\sum_n\frac{m_n}{|\vec{r}-\vec{r}_n|}\, .
\end{equation}
In general, we assume a mass distribution $\varrho(\vec{r})$ and the sum is
replaced by an integral:
\begin{equation}
    \Phi(\vec{r}) = -G\textsubscript{N}\int_{\Reals^3}\dif{\vec{r}^{\prime}}
    \frac{\varrho(\vec{r}^{\prime})}{|\vec{r}-\vec{r}^{\prime}|}\,.
\end{equation}
\section{Comparison with electrostatics}
The classical theory of gravity bears a striking similarity to electrostatics.
To make this clearer, we introduce the gravitational field
$\vec{g}(\vec{r}):= -\nabla\Phi(\vec{r})$.
\begin{table}[htb]
    \caption{Comparison of electrostatics and \name{Newton}ian gravity.}
    \begin{center}
        \begin{tabulars}{lll}
            \toprule
            &\name{Newton}ian Gravity&Electrostatics\\
            \midrule
            Force&$\displaystyle\vec{F}=q\frac{kQ}{r^2}\vec{e}_r$&$\vec{F}=m\frac{G\textsubscript{N}M}{r^2}\vec{e}_r$\\
            Potential
            &$\Phi\textsubscript{el}(\vec{r})=q\frac{kQ}{r}$
            &$\Phi\textsubscript{g}(\vec{r})=m\frac{G\textsubscript{N}M}{r}$\\
            Field
            &$\vec{E}(\vec{r})=-\nabla\Phi\textsubscript{el}(\vec{r})$
            &$\vec{g}(\vec{r})=-\nabla\Phi\textsubscript{g}(\vec{r})$\\
            \name{Laplace} equation
            &$\Delta\Phi\textsubscript{el}=-4\pi k\varrho\textsubscript{el}(\vec{r})$&
            $\Delta\Phi\textsubscript{g}=4\pi
            G\textsubscript{N}\varrho\textsubscript{g}(\vec{r})$
            \\
            \bottomrule
        \end{tabulars}
    \end{center}
\end{table}
\begin{example}[Field of a spherical mass distribution]
Assume we have a spherical mass distribution, i.e.\ $\varrho(\vec{r})=\varrho(r)$.
By symmetry considerations it follows, that the gravitational field can be
expressed by
\begin{equation}
    \vec{g}(\vec{r})=g(r)\vec{e}_r\, .
\end{equation}
We integrate the divergence of the field over a ball $B$ of radius $r$
\begin{equation}
    \int_B\dif{\vec{r}}\,\nabla\vec{g}=-\int_B\dif{\vec{r}}\, \Delta\Phi
    = -4\pi G\textsubscript{N}\int_B\dif{\vec{r}}\,\varrho(r)= -4\pi
    G\textsubscript{N} M\, ,
\end{equation}
where $M$ is the mass enclosed in $B$. On the other hand we can use Gauss's theorem to deduce
\begin{equation}
    \int_B\dif{\vec{r}}\,\nabla\vec{g}=\oint_{\partial B}\dif{\vec{A}}\cdot
    \vec{g} = \oint_{\Omega}\dif{\Omega}\, g(r)r^2=4\pi r^2g(r)\, .
\end{equation}
Together the gravitational field is given by
\begin{equation}
    \vec{g}(r)=-\frac{G\textsubscript{N}M}{r^2}\vec{e}_{r}\, .
\end{equation}
\end{example}
% \subsection{Inertial systems}
% \begin{definition}
% An inertial system is a system in which force-free particles move with constant uniform velocity on straight lines.
% \end{definition}
\subsection*{Weak Equivalence Principle (WEP)}
\name{Newton}'s first law reads
\begin{equation}
    \vec{F}=m\textsubscript{I} \vec{\ddot{x}} \, ,
\end{equation}
where $m\textsubscript{I}$ is the inertial mass that works against the acceleration of the body.
The force which a body with ``active'' mass $m\textsubscript{g,a}$ exerts on
another body with mass $m\textsubscript{g,p}$ is given by
\begin{equation}
    \vec{F}=m\textsubscript{g,p}\frac{G\textsubscript{N}m\textsubscript{g,a}}{r^2}\vec{e}_r \, .
\end{equation}
\begin{figure}[hbtp!]
\centering
 \includegraphics{NewtonForce.pdf}
\caption{}
%TODO Caption
\end{figure}

A priori, there is no reason to assume any relation between this masses.
The first question one might ask is whether the active and the passive mass are equal.
Suppose we have two masses $A$ and $B$. Using \name{Newton}'s first law, we
can explicitly write
\begin{align}
    m^{B}_{\text{I}}\vec{\ddot{x}}&=\vec{F}_{AB}=-
    m^{B}_{\text{g,p}}\frac{G\textsubscript{N}m^{A}_{\text{g,a}}}{r^2}\vec{e}_r
    \, ,\\
    m^{A}_{\text{I}}\vec{\ddot{x}}&=\vec{F}_{BA}=-
    m^{A}_{\text{g,p}}\frac{G\textsubscript{N}m^{B}_{\text{g,a}}}{r^2}\vec{e}_r
    \, .
\end{align}
By the third law $\vec{F}_{AB}=-\vec{F}_{BA}$ we have
\begin{equation}
\frac{m^{B}_{\text{g,p}}}{m^{B}_{\text{g,a}}}=\frac{m^{B}_{\text{g,p}}}{m^{B}_{\text{g,a}}}\,.
\end{equation}
By proper choice of mass units we can set this quotient to one so that
\begin{equation}
m\textsubscript{g,a}=m\textsubscript{g,p}=:m\textsubscript{g}
\end{equation}
The next question is weather the inertial mass equivalent to the gravitational
mass.
By \name{Newton}'s first law we have
\begin{equation}
m\textsubscript{I}\vec{\ddot{x}}
=-m_{\text{g}}\frac{G\textsubscript{N}M_{\text{g}}}{r^2}\vec{e}_r
=-m_{\text{g}}\vec{g}\,.
\end{equation}
As a experimental result that has been measured up to a high accuracy (compare
tabular~\ref{tab:WEPExp}) all bodys recive the same acceleration due to gravity
$\ddot{\vec{x}}\sim \vec{g}$. By a 'proper' choice of units of the flight-time $t=\sqrt{\frac{m\textsubscript{I}}{m\textsubscript{g}}}\sqrt{\frac{2h}{g}}$,
we get
\begin{equation}
m\textsubscript{I}=m\textsubscript{g}=:m\,.
\end{equation}
\begin{table}
\centering
\begin{tabulars}{rllr}
\toprule
Year&Experimenter&Experiment&Accuracy\\
\midrule
1636&\name{Galilei}&inclined planes&$10^{-2}$\\
1689&\name{Newton}&pendulum&$10^{-3}$\\
1832&\name{Bessel}&pendulum&$10^{-5}$\\
1922&\name{Eötvös}&pendulum&$10^{-9}$\\
1922&\name{Shapro} et al.&pendulum&$10^{-12}$\\
1999&\name{Baesler}&torsion balance&$10^{-14}$\\
\bottomrule
\end{tabulars}
\caption{Experiments measuring the ratio
$\frac{m\textsubscript{I}}{m\textsubscript{g}}$.\label{tab:WEPExp}}
\end{table}
\subsection{Tidal Forces}
Assume we have a body of finite extension in a gravitational
potential $\Phi$, an example being the earth in the potential of the moon.
On the center of the body we have
\begin{equation}
m \dod[2]{\tensor{x}{^i}}{t}=-\dpd{\Phi}{\tensor{x}{_i}}\,.
\end{equation}
If we consider a point shifted by $\tensor{\chi}{^i}$ from the center then the
acceleration is given as
\begin{equation}
\begin{split}
m \dod[2]{}{t}\left(\tensor{x}{^i}+\tensor{\chi}{^i}\right)
&=-\dpd{\Phi\left(\tensor{x}{^i}+\tensor{\chi}{^i}\right)}{\tensor{x}{_i}}\\
&\simeq-\dpd{\Phi\left(\tensor{x}{^i}\right)}{\tensor{x}{_i}}
-\dmd{\Phi\left(\tensor{x}{^i}\right)}{2}{\tensor{x}{_i}}{}{\tensor{x}{_j}}{}\tensor{\chi}{^j}\,.
\end{split}
\end{equation}
Subtracting the previous equations yields the tidal force
\begin{equation}
m \dod[2]{\tensor{\chi}{^i}}{t}=-\dmd{\Phi\left(\tensor{x}{^i}\right)}{2}{\tensor{x}{_i}}{}{\tensor{x}{_j}}{}\tensor{\chi}{^j}\,.
\end{equation}
The tidal force tensor $\dmd{\Phi}{2}{\tensor{x}{_i}}{}{\tensor{x}{_j}}{}$ is of
the form
\begin{equation}
\dmd{\Phi}{2}{\tensor{x}{_i}}{}{\tensor{x}{_j}}{}
=\frac{G\textsubscript{N}M}{r^3}\left(\tensor{\delta}{_i_j}-3\frac{\tensor{x}{_i}\tensor{x}{_j}}{r^2}\right)\,.
\end{equation}
%TODO newtons laws?
%TODO picture


\section*{Problem 2: kNN [30 pts] (Hemank)}

In this problem, you will implement K-Nearest Neighbor approach on a synthetic dataset and on the provided dataset\footnote{\url{http://archive.ics.uci.edu/ml/datasets/Iris}}. We will figure out how does various parameters affect the performance of K-NN.

\subsection*{$k$-Nearest Neighbor}
Implement the $k$-nn algorithm and answer the following questions. We will use Euclidean distance for our implementation. For validation, we use $N$-fold cross validation($N=10$).

\textbf{Input Data}: We will generate synthetic input data. The input data will be generated using the code provided in the folder. The code is provided in R, Matlab and Python syntax. The code takes as input the following:

$n$ is the number of samples, $p$ is the dimensionality of the data and $sigma$ determines the amount of noise.

\begin{enumerate}
\item{Why is it suitable to choose odd-$k$?}

\item{Generate $n=1000$ points with $p=10$ and $sigma=0.001$. For different values of $k$, ranging from 1 to 50 (in increments of 5), plot the cross-validation train-error and test-error. What can you say about the effect of $k$? What is the optimal value of $k$, which you can infer from the plot? How does performance changes when you move from smaller values of $k$ to larger values?}

\item{Generate $n=400$ points with $sigma=0.001$. Vary $p$ in range of $2$ to $50$. Run $k$-NN with $k=9$. Plot the cross-validation test-error and comment about effect of dimensionality on the algorithm.}

\item{Generate $n=1000$ points with $sigma=0.001$ and $p=10$. Run $k$-NN with $k=9$. Vary the $N$ for $N$-fold validation from $2$ to $10$ and plot the cross-validation test error. What do you observe from the plot?}

\item{On the given iris dataset, plot with varying $K=1,3,5,7$, the $N$-fold cross-validation test-error.}

\end{enumerate}
\newpage


\section{Relative Hurewicz and J.~H.~C.~Whitehead}
Here is an ``alternative definition'' of connectedness:
\begin{definition}
    Let $n\geq 0$.
    The space $X$ is said to be \emph{$(n-1)$-connected} if, for all $0\leq k\leq n$, any map $f:S^{k-1}\to X$ extends:
    \begin{equation*}
	\xymatrix{
	S^{k-1}\ar[d]\ar[r] & X\\
	D^k\ar@{-->}[ur]_\exists &
	}
    \end{equation*}
\end{definition}
When $n=0$, we know that $S^{-1} = \emptyset$, and $D^0 = \ast$.
Thus being $(-1)$-connected is equivalent to being nonempty.
When $n=1$, this is equivalent to path connectedness. You can check that this is exactly the same as what we said before, using homotopy groups.

As is usual in homotopy theory, there is a relative version of this definition.
\begin{definition}
    Let $n\geq 0$. Say that a pair $(X,A)$ is \emph{$n$-connected} if, for all $0\leq k\leq n$, any map $f:(D^k,S^{k-1}) \to (X,A)$ extends:
    \begin{equation*}
	\xymatrix{
	    (D^k,S^{k-1})\ar[r]^f\ar@{-->}[d] & (X,A)\\
	    (A,A)\ar[ur] &
	    }
    \end{equation*}
    up to homotopy.
    In other words, there is a homotopy between $f$ and a map with image in $A$, such that $f|_{S^{k-1}}$ remains unchanged.
\end{definition}
$0$-connectedness implies that $A$ meets every path component of $X$.
Equivalently:
    \begin{definition}
	$(X,A)$ is $n$-connected if:
	\begin{itemize}
	    \item when $n=0$, the map $\pi_0(A)\to \pi_0(X)$ surjects.
	    \item when $n>0$, the canonical map $\pi_0(A)\xrightarrow{\simeq}\pi_0(X)$ is an isomorphism,
		and for all $a\in A$, the group $\pi_k(X,A,a)$ vanishes for $1\leq k\leq n$.
		(Equivalently, $\pi_0(A)\xrightarrow{\simeq}\pi_0(X)$ and $\pi_k(A,a)\to\pi_k(X,A)$ is an isomorphism for $1\leq k<n$ and is onto for $k=n$.)
	\end{itemize}
    \end{definition}
\subsection{The relative Hurewicz theorem}
    Assume that $\pi_0(A) = \ast = \pi_0(X)$, and pick $a\in A$.
    Then, we have a comparison of long exact sequences, arising from the classical (i.e., non-relative) Hurewicz map:
    \begin{equation*}
	\xymatrix{
	    \cdots\ar[r] & \pi_1(A)\ar[r]\ar[d]^h & \pi_1(X)\ar[r]\ar[d]^h & \pi_1(X,A)\ar[r]\ar[d]^h & \pi_0(A)\ar[r]\ar[d]^h & \pi_0(X)\ar[d]^h & \\
	    \cdots\ar[r] & H_1(A)\ar[r] & H_1(X)\ar[r] & H_1(X,A)\ar[r] & H_0(A)\ar[r] & H_0(X)\ar[r] & H_0(X,A)
	    }
    \end{equation*}
To define the relative Hurewicz map, let $\alpha\in \pi_n(X,A)$, so that $\alpha:(D^n,S^{n-1})\to (X,A)$;
pick a generator of $H_n(D^n,S^{n-1})$, and send it to an element of $H_n(X,A)$ via the induced map
$\alpha_\ast:H_n(D^n,S^{n-1})\to H_n(X,A)$.

Because $H_n(X,A)$ is abelian, the group $\pi_1(A)$ acts trivially on $H_n(X,A)$; in other words,
$h(\omega(\alpha)) = h(\alpha)$.
Consequently, the relative Hurewicz map factors through the group $\pi_n^\dagger(X,A)$, defined to be
the quotient of $\pi_n(X,A)$ by the normal subgroup generated by $(\omega\alpha)\alpha^{-1}$,
where $\omega\in\pi_1(A)$ and $\alpha\in \pi_n(X,A)$.
This begets a map $\pi_n^\dagger(X,A)\to H_n(X,A)$.
\begin{theorem}[Relative Hurewicz]
    Let $n\geq 1$, and assume $(X,A)$ is $n$-connected.
    Then $H_k(X,A) = 0$ for $0\leq k\leq n$, and the map $\pi_{n+1}^\dagger(X,A)\to H_{n+1}(X,A)$ constructed above
    is an isomorphism.
\end{theorem}
We will prove this later using the Serre spectral sequence.
\subsection{The Whitehead theorems}
J.~H.~C.~Whitehead was a rather interesting character. He raised pigs.

Whitehead was interested in determining when a continuous map $f:X\to Y$ that is an isomorphism in homology or homotopy
is a homotopy equivalence.
\begin{definition}
    Let $f:X\to Y$ and $n\geq 0$. Say that $f$ is a \emph{$n$-equivalence}\footnote{Some sources sometimes use ``$n$-connected''.}
    if, for every $\ast\in Y$, the homotopy fiber $F(f,\ast)$ is $(n-1)$-connected.
\end{definition}
For instance, $f$ being a $0$-equivalence simply means that $\pi_0(X)$ surjects onto $\pi_0(Y)$ via $f$.
For $n>0$, this says that $f:\pi_0(X)\to \pi_0(Y)$ is a bijection, and that for every $\ast\in X$:
\begin{equation*}
    \pi_k(X,\ast)\to\pi_k(Y,f(\ast)) \text{ is }\begin{cases}
	\text{an isomorphism } & 1\leq k<n\\
	\text{onto } & k = n.
    \end{cases}
\end{equation*}
Using the ``mapping cylinder'' construction (see Exercise \ref{cofibrep}), we can always assume $f:X\to Y$ is a cofibration;
in particular, that $X\hookrightarrow Y$ is a closed inclusion.
Then, $f:X\to Y$ is an $n$-equivalence if and only if $(Y,X)$ is $n$-connected.
\begin{theorem}[Whitehead]
    Suppose $n\geq 0$, and $f:X\to Y$ is $n$-connected. Then:
    \begin{equation*}
	H_k(X)\xar{f} H_k(Y) \text{ is }\begin{cases}
	    \text{an isomorphism } & 1\leq k<n\\
	    \text{onto } & k = n.
	\end{cases}
    \end{equation*}
\end{theorem}
\begin{proof}
    When $n=0$, because $\pi_0(X)\to \pi_0(Y)$ is surjective, we learn that
    $H_0(X)\simeq \Z[\pi_0(X)]\to \Z[\pi_0(Y)]\simeq H_0(Y)$ is surjective.
    To conclude, use the relative Hurewicz theorem.
    (Note that the relative Hurewicz dealt with $\pi_n^\dagger(X,A)$, but the map $\pi_n(X,A)\to\pi_n^\dagger(X,A)$ is surjective.)
\end{proof}
The case $n=\infty$ is special.
\begin{definition}
    $f$ is a \emph{weak equivalence} (or an $\infty$-equivalence, to make it sound more impressive) if it's an $n$-equivalence for all $n$, i.e., it's a $\pi_\ast$-isomorphism.
\end{definition}
Putting everything together, we obtain:
\begin{corollary}
    A weak equivalence induces an isomorphism in integral homology.
\end{corollary}
How about the converse?

If $H_0(X)\to H_0(Y)$ surjects, then the map $\pi_0(X)\to \pi_0(Y)$ also surjects.
Now, assume $X$ and $Y$ path connected,  and that $H_1(X)$ surjects onto $H_1(Y)$.
We would like to conclude that $\pi_1(X)\to\pi_1(Y)$ surjects.
Unfortunately, this is hard, because $H_1(X)$ is the abelianization of $\pi_1(X)$.
To forge onward, we will simply give up, and assume that $\pi_1(X)\to \pi_1(Y)$ is surjective.

Suppose $H_2(X)\to H_2(Y)$ surjects, and that $f_\ast:H_1(X)\xrightarrow{\simeq}H_1(Y)$.
We know that $H_2(Y,X) = 0$.
On the level of the Hurewicz maps, we are still stuck, because we only obtain information about $\pi_2^\dagger$.
Let us assume that $\pi_1(X)$ is trivial\footnote{This is a pretty radical assumption; for the following argument to work,
it would technically be enough to ask that $\pi_1(X)$ acts trivially on $\pi_2(Y,X)$: but this is basically impossible to check.}.
Under this assumption, we find that $\pi_1(Y) = 0$.
This implies $\pi_2(Y,X)$ is trivial.
Arguing similarly, we can go up the ladder.
\begin{theorem}[Whitehead]
    Let $n\geq 2$, and assume that $\pi_1(X) = 0 = \pi_1(Y)$.
    Suppose $f:X\to Y$ such that:
    \begin{equation*}
	H_k(X)\to H_k(Y) \text{ is }\begin{cases}
	    \text{an isomorphism } & 1\leq k<n\\
	    \text{onto } & k = n;
	\end{cases}
    \end{equation*}
    then $f$ is an $n$-equivalence.
\end{theorem}
Setting $n=\infty$, we obtain:
\begin{corollary}
    Let $X$ and $Y$ be simply-connected.
    If $f$ induces an isomorphism in homology, then $f$ is a weak equivalence.
\end{corollary}
This is incredibly useful, since homology is actually computable!
To wrap up the story, we will state the following result, which we will prove in a later section.
\begin{theorem}\label{weakhtpyequiv}
    Let $Y$ be a CW-complex.
    Then a weak equivalence $f:X\to Y$ is in fact a homotopy equivalence.
\end{theorem}


%!TEX root = Calculus_I.tex
\chapter{A Library of Functions}
\section{Functions and Change}

\textbf{\textit{function}} -- An item used to represent the dependence of one quantity upon another. $f(x)$ means $f$ is a function of $x$.

\textbf{\textit{domain}} -- The input values of a function ($x$ part of $f(x)$). Also referred to as the \textbf{\textit{independent variable}}.

\textbf{\textit{range}} -- The output values of a function ($y$ part of $y = f(x)$). Also referred to as the \textit{dependent variable} because it depends on $x$.
\vspace{0.1in}
Some variables assume discrete values, while others are continuous. Some examples of discrete variables are listed below.
\begin{enumerate}
\item Date\\
\vspace{-0.25in}
\item Cost\\
\vspace{-0.25in}
\item Number of ...
\end{enumerate}

Note that some quantities, such as ``Date" in the above list, are discretized values from a continuous variable, time. Measurement of a continuous variable results in a set of discrete values.

\vspace{0.1in}
Domain and Range values are often written using \textbf{\textit{Interval Notation}}. This notation is used to describe the extrema of a set of numbers. The following cases are used to describe numeric sets using this notation:

\begin{equation}
a \leq x \leq b = [a, b]
\end{equation}

\begin{equation}
a \leq x < b = [a, b)
\end{equation}

\begin{equation}
a < x \leq b = (a, b]
\end{equation}

\begin{equation}
a < x < b = (a, b)
\end{equation}

\vspace{0.1in}
If the domain is not specified, it is usually assumed to be the the set of real numbers, that is $x \in \mathbb{R}$.

\vspace{0.25in}
\textbf{\textit{Linear Function}} -- A function is linear if the \textbf{\textit{slope}}, or rate of change of the function, is constant.
\textbf{\textit{slope}} -- The rate that the dependent variable changes with respect to the independent variable.

\vspace{0.1in}
With this, certain assumptions about the function hold:
\begin{enumerate}
\item The Principle of Superposition applies\\
\vspace{-0.25in}
\item Highest degree of function is 1\\
\end{enumerate}

\vspace{0.1in}
The Greek letter $\bm{\Delta}$ is used to indicate ``change in" a particular variable; thus, $\bm{\Delta} x$ means ``Change in $x$." This is commonly used to express the \textbf{\textit{slope}} of a function, $m$:
\begin{equation}
m = \frac{Rise}{Run} = \frac{\Delta y}{\Delta x} = \frac{f\left(x_2\right) - f\left(x_1\right)}{x_2 - x_1}
\end{equation}

\vspace{0.1in}
If the magnitude of $f(x)$ increases as $x$ increases, then $f(x)$ is classified as an \textbf{\textit{Increasing Function}}. Contrarily, if the magnitude of $f(x)$ decreases as $x$ increases, then $f(x)$ is classified as a \textbf{\textit{Decreasing Function}}. If this is true $\forall x$ (for all values of $x$), then the function can also be classified as \textbf{\textit{monotonic}}.

\begin{center}
\section*{\small Examples}
Coming soon$!^{\text{TM}}$
\end{center}

\section{Exponential Functions}
\textbf{\textit{Exponential Functions}} are a class of functions that can be described by:
\begin{equation}
P = P_0 a^t
\end{equation}
where $P_0$ is some initial quantity (value at $t=0$), and $a$ is the factor by which $P$ changes when $t$ is increased by 1. If $a > 1$, then exponential growth occurs. If $0 < a < 1$, then exponential decay is present. These functions can also be described by their concavity. If the ``opening" of the function points toward the positive $Y$-axis, then the function is \textbf{\textit{concave up}}. Similarly, if it points toward the negative $Y$-axis, then the function is \textbf{\textit{concave down}}.

\vspace{0.1in}
\textbf{\textit{Half-Life}} -- The time required for an exponentially decaying quantity to reach 50\% of the initial value, $P_0$.

\textbf{\textit{Doubling Time}} -- The time required for an exponentially growing quantity to reach 200\% of the initial value, $P_0$.

\vspace{0.1in}
Some exponential functions use the natural number, $e \approx 2.71828$ as the base of the exponential quantity. Because this ensures the base is positive, then exponential growth may also occur when:
\begin{equation}
P = P_0 e^{kt}
\end{equation}
and exponential decay may occur when:
\begin{equation}
P = P_0 e^{-kt}
\end{equation}
This is true if $k>0$ and $t > 0$.

\begin{center}
\section*{\small Examples}
Coming soon$!^{\text{TM}}$
\end{center}

\section{New Functions from Old}
\textbf{\textit{Translations}} of functions occur though additive operations. \textbf{\textit{Stretches}} occur when a function is multiplied by a factor, $k$, when $k$ is not 0 or 1. \textbf{\textit{Reflections}} occur when a function is multiplied by a factor, $k$, when $k < 0$.

\vspace{0.1in}
\textbf{\textit{Composite Functions}} are functions that depend on quantities that can be described by other functions. The example given in the text is the following:
\begin{align}
\nonumber
A = f(&r) = \pi r^2\\
\nonumber
r = g(t&) = 1+t\\
\nonumber
 A = \pi r^2 &= \pi \left(1 + t\right)^2\\
\nonumber
A = f\left(g\left(t\right)\right) = \pi &\left(g\left(t\right)\right)^2 = \pi \left(1 + t\right)^2
\end{align}
Here, the area of a circle, which is mathematically defined by $\pi r^2$, is used as the \textit{outside function}. $r\left(t\right)$ is the \textit{inside function} describes the radius as a \textbf{\textit{monotonically increasing}} quantity with time. Therefore, the area can also be described in terms of time using the definition of the \textbf{\textit{composite function}}.

\vspace{0.1in}
A function can be classified as \textbf{\textit{even}} or \textbf{\textit{odd}} if it is symmetrical about the $Y$-axis. \textbf{\textit{Even}} functions have the property of:
\begin{equation}
f(-x) = f(x)\hphantom{-} \forall x
\end{equation}
An example of this type of function is $\cos\left(x\right)$. On the other hand, \textbf{\textit{odd}} functions follow:
\begin{equation}
f(-x) =- f(x)\hphantom{-} \forall x
\end{equation}
An example of this type of function is  $\sin\left(x\right)$. Many functions are neither \textbf{\textit{even}} nor \textbf{\textit{odd}}.

\vspace{0.1in}
For a given function $f(x)$, the inverse of the function, $f^{-1}(x)$, is given by:
\begin{equation}
f^{-1}(y) = x\hphantom{-} ==\hphantom{-} y = f(x)
\end{equation}
Not all functions are invertible. For this to be true, $f(x)$ must be \textit{single-valued}, or each $y$ value uniquely corresponds to one $x$ value. More specifically, a function has an inverse if, and only if, it intersects any horizontal line at most once. Thus, lines of constant $y$ values cannot correspond to multiple $x$ values for the inverse to exist.

\begin{center}
\section*{\small Examples}
Coming soon$!^{\text{TM}}$
\end{center}

\section{Logarithmic Functions}
\textbf{\textit{Logarithmic Functions}} are the inverse of an exponential function, provided both functions share a common base. These functions are written out as:
\begin{equation}
\log_{10}x = k \hphantom{-} == \hphantom{-} 10^k = x
\end{equation}
where $k$ is a real number. Here, 10 is the common base of the logarithmic and exponential functions. The \textbf{\textit{Natural Logarithm}} uses the natural number, $e$, as its base. It is written as:
\begin{equation}
\ln x = k \hphantom{-} == \hphantom{-} e^c = x
\end{equation}
In logarithmic functions, $x > 0$ because no power of a real number results in zero, and negative values of $x$ are infeasible for positive bases with real exponents.

\vspace{0.1in}
The following table outlines the properties of \textbf{\textit{logarithmic functions}}:
\begin{table*}[h]
\begin{center}
\begin{tabular}{lll}
& \textbf{Base} $A$ & \textbf{Natural Logarithm}\\
\hline
1. & $\log\left(AB\right) = \log A + \log B$ & $\ln\left(AB\right) = \ln A + \ln B$\\
2. &  $\log\left(\frac{A}{B}\right) = \log A - \log B$ & $\ln\left(\frac{A}{B}\right) = \ln A - \ln B$\\
3. & $\log\left(A^p\right) = p \log A$ & $\ln \left(A^p\right) = p \ln A$\\
4. & $\log_A\left(A^x\right) = x$ & $\ln e^x = x$\\
5. & $10^{\log x} = x$ & $e^{\ln x} = x$
\end{tabular}
\end{center}
\end{table*}

These types of equations are useful when solving for unknown exponents.

\begin{center}
\section*{\small Examples}
Coming soon$!^{\text{TM}}$
\end{center}

\section{Trigonometric Functions}
The input of the basic trigonometric functions, $\sin$, $\cos$, and $\tan$ are angles, which are measured in \textbf{\textit{radians}} or \textbf{\textit{degrees}}. To convert between the two, this relationship is used:
\begin{equation}
1 \hphantom{-} \text{radian} = \frac{\pi}{180} \hphantom{-} \text{degrees}
\end{equation}
The angle of 1 \textbf{\textit{radian}} on a unit circle has an arc length of 1. Often, if no units are prescribed for an angular measurement, it is understood to be in \textit{\textbf{radians}}. The arclength equation is given by:
\begin{equation}
s = r\theta
\end{equation}
where $s$ is the arc length, $r$ is the radius of the circle, and $\theta$ is the angular measurement in radians. If a point $P$ on a circle has coordinates in an $(x, y)$ coordinate frame, then we can use the trigonometric functions to relate it's position with the angle $\theta$ by:
\begin{align}
\cos\theta &= x\\
\sin\theta &= y
\end{align}
Because the equation of a circle is given by:
\begin{equation}
x^2 + y^2 = r^2
\end{equation}
a substitution using the trigonometric functions can be made:
\begin{equation}
\cos^2 \theta + \sin^2 \theta = r^2
\end{equation}
For a unit circle, which as a radius $r = 1$, this reduces to the trigonometric identity:
\begin{equation}
\cos^2 \theta + \sin^2 \theta = 1
\end{equation}
As $\theta$ increases, the values of $\sin\theta$ and $\cos\theta$ oscillate between $\left[-1, 1\right]$, and repeats every $2\pi$ \textbf{\textit{radians}} or $360^\circ$. Thus, $\sin \theta$ and $\cos \theta$ are \textit{periodic functions}, or functions that repeat their values after a regular interval.

\vspace{0.1in}
The \textbf{\textit{period}} of a \textit{periodic function} is the length of that regular interval. The \textbf{\textit{amplitude}} of the function is $\frac{1}{2}$ the distance between the function's maximum and minimum values. It should be noted that the $\sin$ and $\cos$ functions can be related through a \textbf{\textit{phase shift}}, or angular \textbf{\textit{translation}}:
\begin{equation}
\cos \theta = \sin\left(\theta + \frac{\pi}{2}\right)
\end{equation}
Functions whose shape can be described using $\sin\theta$ and $\cos\theta$ are given the name \textbf{\textit{sinusoidal functions}}. To summarize their properties:
\begin{equation}
f(\theta) = A\sin\left(B\theta\right) \hphantom{-} g(\theta) = A \cos\left(B\theta\right)
\end{equation}
where $|A|$ is the \textbf{\textit{amplitude}}, $\frac{2\pi}{|B|}$ is the period. Horizontal \textbf{\textit{translations}} occur when the argument $B\theta$ is replaced by $B\theta \pm h$. Vertical \textbf{\textit{translations}} occur when a constant $C$ is added to the functions:
\begin{equation}
f(\theta) = A\sin\left(B\theta\right) + C \hphantom{-} g(\theta) = A \cos\left(B\theta\right) + C
\end{equation}
The \textit{tangent function} is used as a relationship between \textit{sine} and \textit{cosine} functions. It is defined as:
\begin{equation}
\tan\theta=\frac{\sin\theta}{\cos\theta}
\end{equation}
This function has vertical \textit{asymptotes} at points where $\cos\theta = 0$, or $\forall \theta$ defined as $\pm \frac{(2n-1)\pi}{2}$, where $n \in \mathbb{N}$, or the set of \textit{Natural Numbers}. The $\tan$ function has a period of $\pi$ \textbf{\textit{radians}}.

\vspace{0.1in}
Trigonometric functions also have \textbf{\textit{inverse functions}}. These are used to find an angular value given the $(x,y)$ coordinates of a point:
\begin{equation}
\sin x = 0.45
\end{equation}
For the inverse $\sin$ function, which is commonly written as $\arcsin\theta$, $\text{a}\sin\theta$ or $\sin^{-1}\theta$, a domain of $\left[-\frac{\pi}{2}, \frac{\pi}{2}\right]$ is used. Thus, for $y \in \left[-1, 1\right]$:
\begin{equation}
\arcsin y = x \hphantom{-} == \hphantom{-} \sin x = y, \hphantom{-} x\in \left[-\frac{\pi}{2}, \frac{\pi}{2}\right]
\end{equation}
For the inverse $\cos$ function, which is commonly written as $\arccos\theta$, $\text{a}\cos\theta$ or $\cos^{-1}\theta$, the domain is also $\left[-\frac{\pi}{2}, \frac{\pi}{2}\right]$, but the range is $x \in \left[0, \pi\right]$. Lastly, for the inverse $\tan$ function, commonly written as $\arctan\theta$, $\text{a}\tan\theta$ or $\tan^{-1}\theta$, a domain of $\left[-\frac{\pi}{2}, \frac{\pi}{2}\right]$ is used, but has a range $y \in \mathbb{R}$.

\begin{center}
\section*{\small Examples}
Coming soon$!^{\text{TM}}$
\end{center}

\section{Powers, Polynomials, and Rational\\Functions}
A \textbf{\textit{power function}} is a function $f(x)$ where the \textbf{\textit{dependent variable}}, $y$ is proportional to a power of the \textbf{\textit{independent variable}}, $x$:
\begin{equation}
f(x) = kx^p
\end{equation}
where $k$ and $p$ are constant. Examples of these functions are the volume of a sphere:
\begin{equation}
V = \frac{4}{3}\pi r^3
\end{equation}
or Newton's Law of Gravitation:
\begin{equation}
F = kr^{-2} = \frac{k}{r^2}
\end{equation}
For functions of the form $x^n$, where $n$ is a positive integer, odd values of $n>1$ pass through the origin and can assume negative values these are \textbf{\textit{monotonically increasing functions}}. Even values of $n > 1$ also pass through the origin, but $\forall x < 0$, $f(x)$ is a \textbf{\textit{decreasing}} function, and $\forall x > 0$, $f(x)$ is \textbf{\textit{increasing}}. For large exponents $n$, the function value grows faster. Though \textbf{\textit{power functions}} may equal values greater than some arbitrary \textbf{\textit{exponential funtion}}, \textit{every} \textbf{\textit{exponential function}} will eventually dominate \textit{every} \textbf{\textit{power function}} at some value of $x$ if the base of the \textbf{\textit{exponential function}}, $a$, is greater than 1.

\vspace{0.1in}
\textbf{\textit{Polynomials}} are the sum of \textbf{\textit{power functions}} with non-negative integer exponents:
\begin{equation}
y = p(x) = a_nx^n + a_{n-1} x^{n-1} + ... + a_1 x + a_0
\end{equation}
The highest exponent $n$ in a \textbf{\textit{polynomial}} is referred to as the\textbf{\textit{degree}} of the polynomial.

\vspace{0.1in}
\textbf{\textit{Rational Functions}} are ratios of \textbf{\textit{polynomials}}, $p(x)$ and $q(x)$:
\begin{equation}
f(x) = \frac{p(x)}{q(x)}
\end{equation}
These functions may have \textbf{\textit{horizontal or vertical asymptotes}} which occur when:
\begin{equation}
f(x) \rightarrow L \hphantom{-} \text{as} \hphantom{-} x \rightarrow \pm \infty
\end{equation}
for \textbf{\textit{horizontal asymptotes}} or
\begin{equation}
y \rightarrow \pm \infty \hphantom{-} \text{as} \hphantom{-} x \rightarrow K
\end{equation}
for \textbf{\textit{vertical asymptotes}}. Here, the horizontal asymptote is $y = L$, and the \textbf{\textit{vertical asymptote}} is $x = K$.
The function behavior as $x \rightarrow \pm \infty$ is referred to as \textbf{\textit{end behavior}}.

\begin{center}
\section*{\small Examples}
Coming soon$!^{\text{TM}}$
\end{center}

\section{Introduction to Continuity}
This section focuses on the idea of \textbf{\textit{continuity}} along an interval, $[a, b]$, and at a point, $p$. \textbf{\textit{Continuous Functions}} have many desirable properties. For functions along an interval, a general rule of continuity is that a function is continuous along an interval $[a, b]$ if it has no breaks, jumps, or holes within that interval. Many functions are not continuous $\forall x$, such as $\frac{1}{x}$ , which is undefined at $x = 0$, but is continuous for any interval $[a, b]$ that does not contain 0. \textbf{\textit{Exponential}}, \textbf{\textit{power}}, and the \textit{sine} and \textit{cosine} functions are continuous along any interval $[a, b]$. \textbf{\textit{Rational functions}} are continuous on any interval that the denominator is non-zero. Functions derived via addition or multiplication of other continuous functions, and \textbf{\textit{composite functions}} are continuous if the functions used to create them are continuous.

\vspace{0.1in}
\textbf{\textit{Intermediate Value Theorem}} -- Assume $f(x)$ is continuous on a closed interval $[a, b]$. If $k$ is any number between $f(a)$ and $f(b)$, then $\exists$ at least one number $c \in [a, b]$ such that $f(c) = k$.

\vspace{0.1in}
A function is continuous if nearby values of the independent variable, $x$, give nearby values of the dependent variable, $y$. Continuity is important because it implies that small perturbations in $x$ do not result in large changes in $y$. To check continuity at a point, for example, $x = 2$, check nearby values to the left and right of that point, $x = 1.99, 1.98$ and $x = 2.01, 2.02$. If $f(x)$ changes significantly, then the function has a discontinuity at $x =2$.

\begin{center}
\section*{\small Examples}
Coming soon$!^{\text{TM}}$
\end{center}

\section{Limits}
The idea of the \textbf{\textit{limit}} is fundamental to the study of Calculus. The \textbf{\textit{limit}} makes sense of a function ``approaching" a value. Limit notation is defined as:
\begin{equation}
\lim_{x \rightarrow c} f(x) = L
\end{equation}
which means the function $f(x)$ approaches the value $L$ as the independent variable, $x$ approaches the value $c$. $x$ is never equals $c$, but is infinitesimally close to it.

\vspace{0.1in}
A function $f$ is defined on an interval around $c$, except at the point $x = c$. The \textbf{\textit{limit}} of the function $f(x)$ as $x$ approaches $c$ is equal to a number $L$, should such a limit exist, such that $f(x)$ is as close to $L$ as we want whenever $x$ is sufficiently close to $c$. The distance between $f(x)$ and $L$ is given by:
\begin{equation}
\text{Distance} = |f(x) - L|
\end{equation}
which we want to be sufficiently close. The Greek letter $\epsilon$ is used to refer to small numbers; thus, we want:
\begin{equation}
|f(x) - L| < \epsilon
\end{equation}
to show limit convergence. Similarly, we want the following to be true:
\begin{equation}
|x - c| < \delta
\end{equation}
for a chosen value of $\delta$. The definition of the limit can then be rewritten as:

\vspace{0.1in}
\textbf{\textit{Limit}} -- The limit $\lim_{x\rightarrow c} f(x)$ is equal to the number $L$, if one exists, such that $\forall \epsilon > 0$ (as small as we want), there is a $\delta > 0$ (sufficiently small) such that if $|x - c| < \delta$ and $x \neq c$, then $|f(x) - L| < \epsilon$.

\vspace{0.1in}
\textbf{\textit{Limits}} have the following properties, assuming all the limits on the right-hand side exist:
\begin{enumerate}
\item If $b$ is constant, then $\lim_{x\rightarrow c}\left(bf(x)\right) = b\left(\lim_{x \rightarrow c}f(x)\right)$\\
\vspace{-0.25in}
\item $\lim_{x\rightarrow c}\left(f(x) + g(x)\right) = \lim_{x \rightarrow c} f(x) + \lim_{x \rightarrow c} g(x)$\\
\vspace{-0.25in}
\item  $\lim_{x\rightarrow c}\left(f(x) g(x)\right) =  \left(\lim_{x\rightarrow c}f(x)\right)\left(\lim_{x\rightarrow c}g(x)\right)$\\
\vspace{-0.25in}
\item  $\lim_{x\rightarrow c} \frac{f(x)}{g(x)} = \frac{\lim_{x\rightarrow c}f(x)}{\lim_{x\rightarrow c}g(x)}$, provided $\lim_{x\rightarrow c}g(x) \neq 0$\\
\vspace{-0.25in}
\item For any constant $k$, $\lim_{x\rightarrow c} k = k$\\
\vspace{-0.25in}
\item $\lim_{x\rightarrow c} x = c$
\end{enumerate}

\vspace{0.1in}
\textbf{\textit{Limits}} can be taken from both the right and the left. The general form:
\begin{equation}
\lim_{x\rightarrow c} f(x) = L
\end{equation}
means that $f(x) \rightarrow L$ as $x \rightarrow c$ \textit{from both directions}. Piecewise-defined functions may have different values from the right or the left if the limit is taken at a jump discontinuity. To indicate a limit from the left, the following notation is used:
\begin{equation}
\lim_{x\rightarrow c^-} f(x) = L^-
\end{equation}
and from the right:
\begin{equation}
\lim_{x\rightarrow c^+} f(x) = L^+
\end{equation}
Again, the limit from the right or left is not guaranteed to converge to the same value, so $L^-$ and $L^+$ are used, but  $L^-$ may equal $L^+$.

\vspace{0.1in}
\textbf{\textit{Limits}} do not exist when there is no finite number $L$ that the function value assumes at the point the limit is taken. Sometimes, limits are taken at $\pm\infty$ to understand the \textbf{\textit{end behavior}} of a function $f(x)$. Here $\infty$ does not represent a number, just sufficiently large values of $x$. These limits are written as:
\begin{equation}
\lim_{x\rightarrow \pm\infty} f(x) = L
\end{equation}
if the limit, $L$, exists.

\vspace{0.1in}
Using this definition of the limit, \textbf{\textit{continuity}} of a function can be formally defined as:
\textbf{\textit{Continuity}} -- A function $f(x)$ is continuous at $x = c$ if $f$ is defined at $x = c$ and if:
\begin{equation}
\lim_{x \rightarrow c} f(x) = f(c)
\end{equation}
If $c$ is an endpoint of an interval $[a, b]$, then a one-sided limit is used.

\vspace{0.1in}
Using the continuity of sums of products, we can show that any \textbf{\textit{polynomial}} is a continuous function. The following rules can be used to determine of more complicated functions, such as $\sin\theta$, $\cos\theta$, or $e^x$ are continuous, provided $f$ and $g$ are continuous on an interval $[a, b]$ and $b$ is constant:
\begin{enumerate}
\item $bf(x)$ is continuous\\
\vspace{-0.25in}
\item $f(x) + g(x)$ is continuous\\
\vspace{-0.25in}
\item $f(x)g(x)$ is continuous\\
\vspace{-0.25in}
\item $\frac{f(x)}{g(x)}$ is continuous, provided $g(x) \neq 0$ on the interval $[a,b]$.
\end{enumerate}

For \textbf{\textit{composite functions}}, if $f$ and $g$ are continuous and if the \textbf{\textit{composite function}} $f(g(x))$ is defined on an interval $[a, b]$, then $f(g(x))$ is continuous on $[a, b]$.

\begin{center}
\section*{\small Examples}
Coming soon$!^{\text{TM}}$
\end{center}


\subsection{Generalized Linear Models (GLM)}

\noindent{\bf Description}
\smallskip

Generalized Linear Models~\cite{Gill2000:GLM,McCullagh1989:GLM,Nelder1972:GLM}
extend the methodology of linear and logistic regression to a variety of
distributions commonly assumed as noise effects in the response variable.
As before, we are given a collection
of records $(x_1, y_1)$, \ldots, $(x_n, y_n)$ where $x_i$ is a numerical vector of
explanatory (feature) variables of size~\mbox{$\dim x_i = m$}, and $y_i$ is the
response (dependent) variable observed for this vector.  GLMs assume that some
linear combination of the features in~$x_i$ determines the \emph{mean}~$\mu_i$
of~$y_i$, while the observed $y_i$ is a random outcome of a noise distribution
$\Prob[y\mid \mu_i]\,$\footnote{$\Prob[y\mid \mu_i]$ is given by a density function
if $y$ is continuous.}
with that mean~$\mu_i$:
\begin{equation*}
x_i \,\,\,\,\mapsto\,\,\,\, \eta_i = \beta_0 + \sum\nolimits_{j=1}^m \beta_j x_{i,j}
\,\,\,\,\mapsto\,\,\,\, \mu_i \,\,\,\,\mapsto \,\,\,\, y_i \sim \Prob[y\mid \mu_i]
\end{equation*}

In linear regression the response mean $\mu_i$ \emph{equals} some linear combination
over~$x_i$, denoted above by~$\eta_i$.
In logistic regression with $y\in\{0, 1\}$ (Bernoulli) the mean of~$y$ is the same
as $\Prob[y=1]$ and equals $1/(1+e^{-\eta_i})$, the logistic function of~$\eta_i$.
In GLM, $\mu_i$ and $\eta_i$ can be related via any given smooth monotone function
called the \emph{link function}: $\eta_i = g(\mu_i)$.  The unknown linear combination
parameters $\beta_j$ are assumed to be the same for all records.

The goal of the regression is to estimate the parameters~$\beta_j$ from the observed
data.  Once the~$\beta_j$'s are accurately estimated, we can make predictions
about~$y$ for a new feature vector~$x$.  To do so, compute $\eta$ from~$x$ and use
the inverted link function $\mu = g^{-1}(\eta)$ to compute the mean $\mu$ of~$y$;
then use the distribution $\Prob[y\mid \mu]$ to make predictions about~$y$.
Both $g(\mu)$ and $\Prob[y\mid \mu]$ are user-provided.  Our GLM script supports
a standard set of distributions and link functions, see below for details.

\smallskip
\noindent{\bf Usage}
\smallskip

{\hangindent=\parindent\noindent\it%
{\tt{}-f }path/\/{\tt{}GLM.dml}
{\tt{} -nvargs}
{\tt{} X=}path/file
{\tt{} Y=}path/file
{\tt{} B=}path/file
{\tt{} fmt=}format
{\tt{} O=}path/file
{\tt{} Log=}path/file
{\tt{} dfam=}int
{\tt{} vpow=}double
{\tt{} link=}int
{\tt{} lpow=}double
{\tt{} yneg=}double
{\tt{} icpt=}int
{\tt{} reg=}double
{\tt{} tol=}double
{\tt{} disp=}double
{\tt{} moi=}int
{\tt{} mii=}int

}

\smallskip
\noindent{\bf Arguments}
\begin{Description}
\item[{\tt X}:]
Location (on HDFS) to read the matrix of feature vectors; each row constitutes
an example.
\item[{\tt Y}:]
Location to read the response matrix, which may have 1 or 2 columns
\item[{\tt B}:]
Location to store the estimated regression parameters (the $\beta_j$'s), with the
intercept parameter~$\beta_0$ at position {\tt B[}$m\,{+}\,1$, {\tt 1]} if available
\item[{\tt fmt}:] (default:\mbox{ }{\tt "text"})
Matrix file output format, such as {\tt text}, {\tt mm}, or {\tt csv};
see read/write functions in SystemML Language Reference for details.
\item[{\tt O}:] (default:\mbox{ }{\tt " "})
Location to write certain summary statistics described in Table~\ref{table:GLM:stats},
by default it is standard output.
\item[{\tt Log}:] (default:\mbox{ }{\tt " "})
Location to store iteration-specific variables for monitoring and debugging purposes,
see Table~\ref{table:GLM:log} for details.
\item[{\tt dfam}:] (default:\mbox{ }{\tt 1})
Distribution family code to specify $\Prob[y\mid \mu]$, see Table~\ref{table:commonGLMs}:\\
{\tt 1} = power distributions with $\Var(y) = \mu^{\alpha}$;
{\tt 2} = binomial or Bernoulli
\item[{\tt vpow}:] (default:\mbox{ }{\tt 0.0})
When {\tt dfam=1}, this provides the~$q$ in $\Var(y) = a\mu^q$, the power
dependence of the variance of~$y$ on its mean.  In particular, use:\\
{\tt 0.0} = Gaussian,
{\tt 1.0} = Poisson,
{\tt 2.0} = Gamma,
{\tt 3.0} = inverse Gaussian
\item[{\tt link}:] (default:\mbox{ }{\tt 0})
Link function code to determine the link function~$\eta = g(\mu)$:\\
{\tt 0} = canonical link (depends on the distribution family), see Table~\ref{table:commonGLMs};\\
{\tt 1} = power functions,
{\tt 2} = logit,
{\tt 3} = probit,
{\tt 4} = cloglog,
{\tt 5} = cauchit
\item[{\tt lpow}:] (default:\mbox{ }{\tt 1.0})
When {\tt link=1}, this provides the~$s$ in $\eta = \mu^s$, the power link
function; {\tt lpow=0.0} gives the log link $\eta = \log\mu$.  Common power links:\\
{\tt -2.0} = $1/\mu^2$,
{\tt -1.0} = reciprocal,
{\tt 0.0} = log,
{\tt 0.5} = sqrt,
{\tt 1.0} = identity
\item[{\tt yneg}:] (default:\mbox{ }{\tt 0.0})
When {\tt dfam=2} and the response matrix $Y$ has 1~column,
this specifies the $y$-value used for Bernoulli ``No'' label.
All other $y$-values are treated as the ``Yes'' label.
For example, {\tt yneg=-1.0} may be used when $y\in\{-1, 1\}$;
either {\tt yneg=1.0} or {\tt yneg=2.0} may be used when $y\in\{1, 2\}$.
\item[{\tt icpt}:] (default:\mbox{ }{\tt 0})
Intercept and shifting/rescaling of the features in~$X$:\\
{\tt 0} = no intercept (hence no~$\beta_0$), no shifting/rescaling of the features;\\
{\tt 1} = add intercept, but do not shift/rescale the features in~$X$;\\
{\tt 2} = add intercept, shift/rescale the features in~$X$ to mean~0, variance~1
\item[{\tt reg}:] (default:\mbox{ }{\tt 0.0})
L2-regularization parameter (lambda)
\item[{\tt tol}:] (default:\mbox{ }{\tt 0.000001})
Tolerance (epsilon) used in the convergence criterion: we terminate the outer iterations
when the deviance changes by less than this factor; see below for details
\item[{\tt disp}:] (default:\mbox{ }{\tt 0.0})
Dispersion parameter, or {\tt 0.0} to estimate it from data
\item[{\tt moi}:] (default:\mbox{ }{\tt 200})
Maximum number of outer (Fisher scoring) iterations
\item[{\tt mii}:] (default:\mbox{ }{\tt 0})
Maximum number of inner (conjugate gradient) iterations, or~0 if no maximum
limit provided
\end{Description}

\begin{table}[t]\small\centerline{%
\begin{tabular}{|ll|}
\hline
Name & Meaning \\
\hline
{\tt TERMINATION\_CODE}  & A positive integer indicating success/failure as follows: \\
                         & $1 = {}$Converged successfully;
                           $2 = {}$Maximum \# of iterations reached; \\
                         & $3 = {}$Input ({\tt X}, {\tt Y}) out of range;
                           $4 = {}$Distribution/link not supported \\
{\tt BETA\_MIN}          & Smallest beta value (regression coefficient), excluding the intercept \\
{\tt BETA\_MIN\_INDEX}   & Column index for the smallest beta value \\
{\tt BETA\_MAX}          & Largest beta value (regression coefficient), excluding the intercept \\
{\tt BETA\_MAX\_INDEX}   & Column index for the largest beta value \\
{\tt INTERCEPT}          & Intercept value, or NaN if there is no intercept (if {\tt icpt=0}) \\
{\tt DISPERSION}         & Dispersion used to scale deviance, provided in {\tt disp} input argument \\
                         & or estimated (same as {\tt DISPERSION\_EST}) if {\tt disp} argument is${} \leq 0$ \\
{\tt DISPERSION\_EST}    & Dispersion estimated from the dataset \\
{\tt DEVIANCE\_UNSCALED} & Deviance from the saturated model, assuming dispersion${} = 1.0$ \\
{\tt DEVIANCE\_SCALED}   & Deviance from the saturated model, scaled by {\tt DISPERSION} value \\
\hline
\end{tabular}}
\caption{Besides~$\beta$, GLM regression script computes a few summary statistics listed above.
They are provided in CSV format, one comma-separated name-value pair per each line.}
\label{table:GLM:stats}
\end{table}

\begin{table}[t]\small\centerline{%
\begin{tabular}{|ll|}
\hline
Name & Meaning \\
\hline
{\tt NUM\_CG\_ITERS}     & Number of inner (Conj.\ Gradient) iterations in this outer iteration \\
{\tt IS\_TRUST\_REACHED} & $1 = {}$trust region boundary was reached, $0 = {}$otherwise \\
{\tt POINT\_STEP\_NORM}  & L2-norm of iteration step from old point ($\beta$-vector) to new point \\
{\tt OBJECTIVE}          & The loss function we minimize (negative partial log-likelihood) \\
{\tt OBJ\_DROP\_REAL}    & Reduction in the objective during this iteration, actual value \\
{\tt OBJ\_DROP\_PRED}    & Reduction in the objective predicted by a quadratic approximation \\
{\tt OBJ\_DROP\_RATIO}   & Actual-to-predicted reduction ratio, used to update the trust region \\
{\tt GRADIENT\_NORM}     & L2-norm of the loss function gradient (omitted if point is rejected) \\
{\tt LINEAR\_TERM\_MIN}  & The minimum value of $X \pxp \beta$, used to check for overflows \\
{\tt LINEAR\_TERM\_MAX}  & The maximum value of $X \pxp \beta$, used to check for overflows \\
{\tt IS\_POINT\_UPDATED} & $1 = {}$new point accepted; $0 = {}$new point rejected, old point restored \\
{\tt TRUST\_DELTA}       & Updated trust region size, the ``delta'' \\
\hline
\end{tabular}}
\caption{
The {\tt Log} file for GLM regression contains the above \mbox{per-}iteration
variables in CSV format, each line containing triple (Name, Iteration\#, Value) with Iteration\#
being~0 for initial values.}
\label{table:GLM:log}
\end{table}

\begin{table}[t]\hfil
\begin{tabular}{|ccccccc|}
\hline
\multicolumn{4}{|c}{INPUT PARAMETERS}              & Distribution  & Link      & Cano- \\
{\tt dfam} & {\tt vpow} & {\tt link} & {\tt\ lpow} & family        & function  & nical?\\
\hline
{\tt 1}    & {\tt 0.0}  & {\tt 1}    & {\tt -1.0}  & Gaussian      & inverse   &       \\
{\tt 1}    & {\tt 0.0}  & {\tt 1}    & {\tt\ 0.0}  & Gaussian      & log       &       \\
{\tt 1}    & {\tt 0.0}  & {\tt 1}    & {\tt\ 1.0}  & Gaussian      & identity  & Yes   \\
{\tt 1}    & {\tt 1.0}  & {\tt 1}    & {\tt\ 0.0}  & Poisson       & log       & Yes   \\
{\tt 1}    & {\tt 1.0}  & {\tt 1}    & {\tt\ 0.5}  & Poisson       & sq.root   &       \\
{\tt 1}    & {\tt 1.0}  & {\tt 1}    & {\tt\ 1.0}  & Poisson       & identity  &       \\
{\tt 1}    & {\tt 2.0}  & {\tt 1}    & {\tt -1.0}  & Gamma         & inverse   & Yes   \\
{\tt 1}    & {\tt 2.0}  & {\tt 1}    & {\tt\ 0.0}  & Gamma         & log       &       \\
{\tt 1}    & {\tt 2.0}  & {\tt 1}    & {\tt\ 1.0}  & Gamma         & identity  &       \\
{\tt 1}    & {\tt 3.0}  & {\tt 1}    & {\tt -2.0}  & Inverse Gauss & $1/\mu^2$ & Yes   \\
{\tt 1}    & {\tt 3.0}  & {\tt 1}    & {\tt -1.0}  & Inverse Gauss & inverse   &       \\
{\tt 1}    & {\tt 3.0}  & {\tt 1}    & {\tt\ 0.0}  & Inverse Gauss & log       &       \\
{\tt 1}    & {\tt 3.0}  & {\tt 1}    & {\tt\ 1.0}  & Inverse Gauss & identity  &       \\
\hline
{\tt 2}    & {\tt  *}   & {\tt 1}    & {\tt\ 0.0}  & Binomial      & log       &       \\
{\tt 2}    & {\tt  *}   & {\tt 1}    & {\tt\ 0.5}  & Binomial      & sq.root   &       \\
{\tt 2}    & {\tt  *}   & {\tt 2}    & {\tt\  *}   & Binomial      & logit     & Yes   \\
{\tt 2}    & {\tt  *}   & {\tt 3}    & {\tt\  *}   & Binomial      & probit    &       \\
{\tt 2}    & {\tt  *}   & {\tt 4}    & {\tt\  *}   & Binomial      & cloglog   &       \\
{\tt 2}    & {\tt  *}   & {\tt 5}    & {\tt\  *}   & Binomial      & cauchit   &       \\
\hline
\end{tabular}\hfil
\caption{Common GLM distribution families and link functions.
(Here ``{\tt *}'' stands for ``any value.'')}
\label{table:commonGLMs}
\end{table}

\noindent{\bf Details}
\smallskip

In GLM, the noise distribution $\Prob[y\mid \mu]$ of the response variable~$y$
given its mean~$\mu$ is restricted to have the \emph{exponential family} form
\begin{equation}
Y \sim\, \Prob[y\mid \mu] \,=\, \exp\left(\frac{y\theta - b(\theta)}{a}
+ c(y, a)\right),\,\,\textrm{where}\,\,\,\mu = \E(Y) = b'(\theta).
\label{eqn:GLM}
\end{equation}
Changing the mean in such a distribution simply multiplies all \mbox{$\Prob[y\mid \mu]$}
by~$e^{\,y\hspace{0.2pt}\theta/a}$ and rescales them so that they again integrate to~1.
Parameter $\theta$ is called \emph{canonical}, and the function $\theta = b'^{\,-1}(\mu)$
that relates it to the mean is called the~\emph{canonical link}; constant~$a$ is called
\emph{dispersion} and rescales the variance of~$y$.  Many common distributions can be put
into this form, see Table~\ref{table:commonGLMs}.  The canonical parameter~$\theta$
is often chosen to coincide with~$\eta$, the linear combination of the regression features;
other choices for~$\eta$ are possible too.

Rather than specifying the canonical link, GLM distributions are commonly defined
by their variance $\Var(y)$ as the function of the mean~$\mu$.  It can be shown
from Eq.~(\ref{eqn:GLM}) that $\Var(y) = a\,b''(\theta) = a\,b''(b'^{\,-1}(\mu))$.
For example, for the Bernoulli distribution $\Var(y) = \mu(1-\mu)$, for the Poisson
distribution \mbox{$\Var(y) = \mu$}, and for the Gaussian distribution
$\Var(y) = a\cdot 1 = \sigma^2$.
It turns out that for many common distributions $\Var(y) = a\mu^q$, a power function.
We support all distributions where $\Var(y) = a\mu^q$, as well as the Bernoulli and
the binomial distributions.

For distributions with $\Var(y) = a\mu^q$ the canonical link is also a power function,
namely $\theta = \mu^{1-q}/(1-q)$, except for the Poisson ($q = 1$) whose canonical link is
$\theta = \log\mu$.  We support all power link functions in the form $\eta = \mu^s$,
dropping any constant factor, with $\eta = \log\mu$ for $s=0$.  The binomial distribution
has its own family of link functions, which includes logit (the canonical link),
probit, cloglog, and cauchit (see Table~\ref{table:binomial_links}); we support these
only for the binomial and Bernoulli distributions.  Links and distributions are specified
via four input parameters: {\tt dfam}, {\tt vpow}, {\tt link}, and {\tt lpow} (see
Table~\ref{table:commonGLMs}).

\begin{table}[t]\hfil
\begin{tabular}{|cc|cc|}
\hline
Name & Link function & Name & Link function \\
\hline
Logit   & $\displaystyle \eta = 1 / \big(1 + e^{-\mu}\big)^{\mathstrut}$ &
Cloglog & $\displaystyle \eta = \log \big(\!- \log(1 - \mu)\big)^{\mathstrut}$ \\
Probit  & $\displaystyle \mu  = \frac{1}{\sqrt{2\pi}}\int\nolimits_{-\infty_{\mathstrut}}^{\,\eta\mathstrut}
          \!\!\!\!\! e^{-\frac{t^2}{2}} dt$ &
Cauchit & $\displaystyle \eta = \tan\pi(\mu - 1/2)$ \\
\hline
\end{tabular}\hfil
\caption{The supported non-power link functions for the Bernoulli and the binomial
distributions.  (Here $\mu$~is the Bernoulli mean.)}
\label{table:binomial_links}
\end{table}

The observed response values are provided to the regression script as matrix~$Y$
having 1 or 2 columns.  If a power distribution family is selected ({\tt dfam=1}),
matrix $Y$ must have 1~column that provides $y_i$ for each~$x_i$ in the corresponding
row of matrix~$X$.  When {\tt dfam=2} and $Y$ has 1~column, we assume the Bernoulli
distribution for $y_i\in\{y_{\mathrm{neg}}, y_{\mathrm{pos}}\}$ with $y_{\mathrm{neg}}$
from the input parameter {\tt yneg} and with $y_{\mathrm{pos}} \neq y_{\mathrm{neg}}$.
When {\tt dfam=2} and $Y$ has 2~columns, we assume the
binomial distribution; for each row~$i$ in~$X$, cells $Y[i, 1]$ and $Y[i, 2]$ provide
the positive and the negative binomial counts respectively.  Internally we convert
the 1-column Bernoulli into the 2-column binomial with 0-versus-1 counts.

We estimate the regression parameters via L2-regularized negative log-likelihood
minimization:
\begin{equation*}
f(\beta; X, Y) \,\,=\,\, -\sum\nolimits_{i=1}^n \big(y_i\theta_i - b(\theta_i)\big)
\,+\,(\lambda/2) \sum\nolimits_{j=1}^m \beta_j^2\,\,\to\,\,\min
\end{equation*}
where $\theta_i$ and $b(\theta_i)$ are from~(\ref{eqn:GLM}); note that $a$
and $c(y, a)$ are constant w.r.t.~$\beta$ and can be ignored here.
The canonical parameter $\theta_i$ depends on both $\beta$ and~$x_i$:
\begin{equation*}
\theta_i \,\,=\,\, b'^{\,-1}(\mu_i) \,\,=\,\, b'^{\,-1}\big(g^{-1}(\eta_i)\big) \,\,=\,\,
\big(b'^{\,-1}\circ g^{-1}\big)\left(\beta_0 + \sum\nolimits_{j=1}^m \beta_j x_{i,j}\right)
\end{equation*}
The user-provided (via {\tt reg}) regularization coefficient $\lambda\geq 0$ can be used
to mitigate overfitting and degeneracy in the data.  Note that the intercept is never
regularized.

Our iterative minimizer for $f(\beta; X, Y)$ uses the Fisher scoring approximation
to the difference $\varDelta f(z; \beta) = f(\beta + z; X, Y) \,-\, f(\beta; X, Y)$,
recomputed at each iteration:
\begin{gather*}
\varDelta f(z; \beta) \,\,\,\approx\,\,\, 1/2 \cdot z^T A z \,+\, G^T z,
\,\,\,\,\textrm{where}\,\,\,\, A \,=\, X^T\!\diag(w) X \,+\, \lambda I\\
\textrm{and}\,\,\,\,G \,=\, - X^T u \,+\, \lambda\beta,
\,\,\,\textrm{with $n\,{\times}\,1$ vectors $w$ and $u$ given by}\\
\forall\,i = 1\ldots n: \,\,\,\,
w_i = \big[v(\mu_i)\,g'(\mu_i)^2\big]^{-1}
\!\!\!\!\!\!,\,\,\,\,\,\,\,\,\,
u_i = (y_i - \mu_i)\big[v(\mu_i)\,g'(\mu_i)\big]^{-1}
\!\!\!\!\!\!.\,\,\,\,
\end{gather*}
Here $v(\mu_i)=\Var(y_i)/a$, the variance of $y_i$ as the function of the mean, and
$g'(\mu_i) = d \eta_i/d \mu_i$ is the link function derivative.  The Fisher scoring
approximation is minimized by trust-region conjugate gradient iterations (called the
\emph{inner} iterations, with the Fisher scoring iterations as the \emph{outer}
iterations), which approximately solve the following problem:
\begin{equation*}
1/2 \cdot z^T A z \,+\, G^T z \,\,\to\,\,\min\,\,\,\,\textrm{subject to}\,\,\,\,
\|z\|_2 \leq \delta
\end{equation*}
The conjugate gradient algorithm closely follows Algorithm~7.2 on page~171
of~\cite{Nocedal2006:Optimization}.
The trust region size $\delta$ is initialized as $0.5\sqrt{m}\,/ \max\nolimits_i \|x_i\|_2$
and updated as described in~\cite{Nocedal2006:Optimization}.
The user can specify the maximum number of the outer and the inner iterations with
input parameters {\tt moi} and {\tt mii}, respectively.  The Fisher scoring algorithm
terminates successfully if $2|\varDelta f(z; \beta)| < (D_1(\beta) + 0.1)\hspace{0.5pt}\eps$
where $\eps > 0$ is a tolerance supplied by the user via {\tt tol}, and $D_1(\beta)$ is
the unit-dispersion deviance estimated as
\begin{equation*}
D_1(\beta) \,\,=\,\, 2 \cdot \big(\Prob[Y \mid \!
\begin{smallmatrix}\textrm{saturated}\\\textrm{model}\end{smallmatrix}, a\,{=}\,1]
\,\,-\,\,\Prob[Y \mid X, \beta, a\,{=}\,1]\,\big)
\end{equation*}
The deviance estimate is also produced as part of the output.  Once the Fisher scoring
algorithm terminates, if requested by the user, we estimate the dispersion~$a$ from
Eq.~\ref{eqn:GLM} using Pearson residuals
\begin{equation}
\hat{a} \,\,=\,\, \frac{1}{n-m}\cdot \sum_{i=1}^n \frac{(y_i - \mu_i)^2}{v(\mu_i)}
\label{eqn:dispersion}
\end{equation}
and use it to adjust our deviance estimate: $D_{\hat{a}}(\beta) = D_1(\beta)/\hat{a}$.
If input argument {\tt disp} is {\tt 0.0} we estimate $\hat{a}$, otherwise we use its
value as~$a$.  Note that in~(\ref{eqn:dispersion}) $m$~counts the intercept
($m \leftarrow m+1$) if it is present.

\smallskip
\noindent{\bf Returns}
\smallskip

The estimated regression parameters (the $\hat{\beta}_j$'s) are populated into
a matrix and written to an HDFS file whose path/name was provided as the ``{\tt B}''
input argument.  What this matrix contains, and its size, depends on the input
argument {\tt icpt}, which specifies the user's intercept and rescaling choice:
\begin{Description}
\item[{\tt icpt=0}:] No intercept, matrix~$B$ has size $m\,{\times}\,1$, with
$B[j, 1] = \hat{\beta}_j$ for each $j$ from 1 to~$m = {}$ncol$(X)$.
\item[{\tt icpt=1}:] There is intercept, but no shifting/rescaling of~$X$; matrix~$B$
has size $(m\,{+}\,1) \times 1$, with $B[j, 1] = \hat{\beta}_j$ for $j$ from 1 to~$m$,
and $B[m\,{+}\,1, 1] = \hat{\beta}_0$, the estimated intercept coefficient.
\item[{\tt icpt=2}:] There is intercept, and the features in~$X$ are shifted to
mean${} = 0$ and rescaled to variance${} = 1$; then there are two versions of
the~$\hat{\beta}_j$'s, one for the original features and another for the
shifted/rescaled features.  Now matrix~$B$ has size $(m\,{+}\,1) \times 2$, with
$B[\cdot, 1]$ for the original features and $B[\cdot, 2]$ for the shifted/rescaled
features, in the above format.  Note that $B[\cdot, 2]$ are iteratively estimated
and $B[\cdot, 1]$ are obtained from $B[\cdot, 2]$ by complementary shifting and
rescaling.
\end{Description}
Our script also estimates the dispersion $\hat{a}$ (or takes it from the user's input)
and the deviances $D_1(\hat{\beta})$ and $D_{\hat{a}}(\hat{\beta})$, see
Table~\ref{table:GLM:stats} for details.  A log file with variables monitoring
progress through the iterations can also be made available, see Table~\ref{table:GLM:log}.

\smallskip
\noindent{\bf Examples}
\smallskip

{\hangindent=\parindent\noindent\tt
\hml -f GLM.dml -nvargs X=/user/biadmin/X.mtx Y=/user/biadmin/Y.mtx
  B=/user/biadmin/B.mtx fmt=csv dfam=2 link=2 yneg=-1.0 icpt=2 reg=0.01 tol=0.00000001
  disp=1.0 moi=100 mii=10 O=/user/biadmin/stats.csv Log=/user/biadmin/log.csv

}

\smallskip
\noindent{\bf See Also}
\smallskip

In case of binary classification problems, consider using L2-SVM or binary logistic
regression; for multiclass classification, use multiclass~SVM or multinomial logistic
regression.  For the special cases of linear regression and logistic regression, it
may be more efficient to use the corresponding specialized scripts instead of~GLM.


In this problem we are going t approximate the function
$$ f(x)=e^{\sin(5x)}$$
using Chebyshev's interpolation of degree ten (using eleven points) and using the orthonormal basis provided by the \textit{Legendre polynomials}. It will be shown that the Legendre polynomials, which we show in the figure 1, give the best approximation of that degree as we proved in Problem 1.

\begin{figure}[H]
\centering     %%% not \center
{\includegraphics[scale=0.75]{LegendrePols.eps}}
\caption{Legendre Polynomials.}
\end{figure}

In the following figure we show the function $f$ and the two approximations made. To the naked eye both of them seem very accurate, however we can see how the Legendre polynomials give us a better approximation by looking at the error in figure 3.
\begin{figure}[H]
\centering     %%% not \center
{\includegraphics[scale=0.6]{Approximations_p2.eps}}
\caption{Approximations using Chebyshev and Legendre polynomials.}
\end{figure}
The sudden drops in the error are in fact the points used to do the approximation, so the error is indeed zero. Since we are using logarithmic scale we see thos drops. We can see how the error of Chebyshev's method is higher. We can compute the $L_2$ error of the two methods,
\begin{align*}
e_{Chebyshev}=0.07800109011,~~~~~~e_{Legendre}=0.05183155462,
\end{align*}
and see how the Legendre polynomials, as proved in problem 1, give us the best approximation.
\begin{figure}[H]
\centering     %%% not \center
{\includegraphics[scale=0.6]{Error_p2.eps}}
\caption{Error of the approximations.}
\end{figure}
\subsection*{Matlab code for this problem}
\begin{verbatim}
%% Problem 2
clear all
close all
clc
format long
legendfontsize=12;
labelfontsize=14;
f = @(x) exp(sin(5*x));
f= chebfun(f);
f_cheb = chebfun(f,11);
xx=linspace(-1,1,1000);
P=legpoly(0:10,'norm');

figure
plot(P)
grid on
xlabel('$x$','fontsize',labelfontsize,'interpreter','latex')
saveas(gcf,'Latex/FIGURES/LegendrePols','epsc')
saveas(gcf,'Latex/FIGURES/LegendrePols','fig')

f_N=0;
for k=1:11
    f_N=f_N+(f'*P(:,k))*P(:,k);
end
%%
figure
plot(f,'linewidth',2)
hold on
plot(f_N,'linewidth',2)
plot(f_cheb,'linewidth',2)
grid on
legend({'$f(x)=e^{\sin(5x)}$','Legendre','Chebishev'}...
    ,'fontsize',legendfontsize,'interpreter','latex','location','north')
xlabel('$x$','fontsize',labelfontsize,'interpreter','latex')
saveas(gcf,'Latex/FIGURES/Approximations_p2','epsc')
saveas(gcf,'Latex/FIGURES/Approximations_p2','fig')

Error_cheb=norm(f-f_cheb,2)
Error_app=norm(f-f_N,2)
%%
figure
semilogy(abs(f-f_cheb))
hold on
semilogy(abs(f-f_N))
grid on
axis([-1 1 1e-4 0.35])
legend({'$e_{Cheb}$','$e_{Leg}$'},'fontsize',legendfontsize,'interpreter','latex')
xlabel('$x$','fontsize',labelfontsize,'interpreter','latex')
saveas(gcf,'Latex/FIGURES/Error_p2','epsc')
saveas(gcf,'Latex/FIGURES/Error_p2','fig')

\end{verbatim}


% 18ConnectionsandAssociatedBundles.tex

\subsection{ 18.1. Forms with Values in a Lie Algebra}

\begin{quote}
  What do we mean by $g^{-1} dg$?
\end{quote}

\subsubsection{ 18.1.a. The Maurer-Cartan Form }

If we think of $\omega$ as being a form that takes its values in the fixed vector space $\mathfrak{g}$, rather than as a matrix of 1-forms, we shall have an equivalent picture that is in many ways more closely related to the terminology used in physics.

exterior form is differential form

\textbf{ Maurer-Cartan 1-form on $G$}

Let $\lbrace E_R \rbrace$ basis for $\mathfrak{g}$  \\
\phantom{Let} $\lbrace X_R \rbrace$ left invariant fields on $G$ obtained by left translating $E$'s  \\
\phantom{Let} $\lbrace \sigma^R \rbrace$ left invariant 1-forms on $G$ forming, $\forall \, g \in G$, basis dual to $\lbrace X_R \rbrace$  \\

$\sigma^R(X_S) = \delta^R_{ \, \, S}$

Then
\begin{equation}
  \Omega \equiv E_R \otimes \sigma^R \quad \quad \quad \, (18.1)
\end{equation}
\[
\Omega(Y_g) = E_R \sigma^R(Y_g) = E_R Y^R
\]
$Y = X_R Y^R$ at $g\in G$, left translates back to 1

$\Omega : T_gG \to T_e G$

cf. Nakahara
\[
\Omega : Y \mapsto (L_{g^{-1}})_* Y = (L_g)^{-1}_* Y, \, Y \in T_g G
\]

Classically, Cartan wrote $\forall \, p \in M$, vector valued 1 form taking each $Y$ vector at $p$ into itself
\[
dp = \partial_i \otimes dx^i = \partial_i \otimes \delta^i_{ \, \, j }dx^j
\]
\begin{equation}
  \Omega = g^{-1} dg \quad \quad \quad \, (18.2)
\end{equation}

$dg$ takes $Y$ at $g$ into $Y$, $g^{-1}$ left transltates $Y$ back to $e$

LaTeX_and_pdfs/the geometry of physics problems/18ConnectionsandAssociatedBundles.tex


\documentclass[]{article}

\usepackage{amsmath}
\usepackage{bm}

\usepackage{siunitx}

\newcommand{\cp}{c_{p}}
\newcommand{\xd}[4]{%
	\def\temp{#1}\ifx\temp\empty
		\frac{#4 #2}{#4 #3}
	\else
		\frac{#4^{#1} #2}{#4 #3^{#1}}
	\fi
	}
\newcommand{\pd}[3][]{\xd{#1}{#2}{#3}{\partial}}
\newcommand{\Dd}[3][]{\xd{#1}{#2}{#3}{\Delta}}
\newcommand{\D}[2][]{%
	\def\temp{#1}\ifx\temp\empty
		\Delta #2
	\else
		\Delta^{#1} #2
	\fi
	}
\newcommand{\id}[1]{\:d#1}
\newcommand{\dq}{\dot{q}}
\newcommand{\vdq}{\bm{\dq}}
\newcommand{\cnm}{\bm{M}}
\newcommand{\cnb}{\bm{b}}
\newcommand{\cnd}{\bm{d}}

%opening
\title{A 3D Finite Volume Method Solver for Arbitrary Solidification Boundary-Value Problems}
\author{William E Warriner}

\begin{document}

\maketitle

\section{The Heat Equation}

Consider a point in Euclidean space and let the temperature of that point be \(u\), its density \(\rho\) and constant-pressure specific heat capacity \(\cp\). It is assumed that all properties are temperature dependent. Then the heat equation may be formulated as
\begin{align}
\rho \cp \pd{u}{t} &= \nabla \cdot \vdq
\end{align}
where \(\vdq\) is the heat flux vector across the point. The equation states that the temperature of a point changes proportionally to the net heat flux through it. Both sides of the equation have units of \(\si{W.m^{-3}}\).

There are three types of heat flux: conductive, convective, and radiative. It is assumed that a boundary cannot experience both conductive and convective heat transfer. Conductive heat transfer is reserved for boundaries between points with the same material, while convective is for different materials. Radiative heat transfer occurs in both cases and is added to both. The heat flux equations are
\begin{align}
\vdq_{k} &= -k \nabla u \\
\vdq_{h} &= \bm{h} \circ \left( \bm{u}_{\infty} - u \right) \\
\vdq_{\sigma} &= \sigma \varepsilon \left( \bm{u}^{4}_{\infty} - u^{4} \right)
\end{align}
where \(k\) is thermal conductivity, \(\bm{h}\) is the vector convective heat transfer coefficient at the boundary, \(u_{\infty}\) is the ambient temperature, \(\sigma\) is the Stefan-Boltzmann constant and \(\varepsilon\) is a constant dependent on emissivity. The symbol ``\(\circ\)'' is the Hadamard product, and is performed element-wise. The scalar temperature \(u\) at a point is implicitly expanded to a vector with an appropriate number of elements whose values are identical. All of these have units of \(\si{W.m^{-2}}\).

The heat equation in the case of pure conductivity becomes the more familiar
\begin{align}
\rho \cp \pd{u}{t} &= -\nabla \cdot k \nabla u \\
&= -\nabla k \cdot \nabla u - k \nabla^2 u
\end{align}
whereas in the case of pure convection it becomes
\begin{align}
\rho \cp \pd{u}{t} = \nabla \cdot \bm{h} \circ \left( \bm{u}_{\infty} - u \right)
\end{align}
with the assumption that heat transfer modes can be freely mixed provided the point is replaced by a differential element. Such a replacement naturally leads to a finite volume method formulation for numerical solutions to the heat equation.

\section{Finite Volume Method}

Suppose that the problem domain is subdivided into axially-aligned rectangular prisms---or elements---whose side lengths are \(\Delta x\), \(\Delta y\) and \(\Delta z\) along the \(X\)-, \(Y\)- and \(Z\)-axes, respectively. Then the volume of the cell is \(V = \Delta x \Delta y \Delta z\), and the surface area components are \(S_{yz} = \Delta y \Delta z\), \(S_{zx} = \Delta z \Delta x\), \(S_{xy} = \Delta x \Delta y\) for boundaries normal to the \(X\)-, \(Y\)- and \(Z\)-axes respectively. Note the side lengths may vary along each axis without changing the following. Let \(n\) denote the sequence of axes \(\left\{ x,y,z \right\}\). Assume each cell has uniform temperature and properties. Then the convective heat equation for a cell then becomes
\begin{align}
\rho \cp \Dd{u}{t} &= -\sum_{n} \Dd{k_{n}}{n} \Dd{u}{n} - \sum_{n} k_{n} \Dd[2]{u}{n} \\
\Dd{u}{t} &= -\sum_{n} \frac{1}{\rho_{n} c_{p,n}} \Dd{k_{n}}{n} \Dd{u}{n} - \sum_{n} \frac{k_{n}}{\rho_{n} c_{p,n}} \Dd[2]{u}{n} \\
\D{u} &= -\sum_{n} \frac{\D{t}}{\rho_{n} c_{p,n} \D{n}^{2}}\D{k}_{n}\D{u} - \sum_{n} \frac{k_{n} \D{t}}{\rho_{n} c_{p,n} \D{n^{2}}}\D[2]{u} \\
\D{u} &= -\sum_{n} \frac{\D{t}}{\rho_{n} c_{p,n}\D{n}^{2}} \left( \D{k}_{n}\D{u} + k_{n}\D[2]{u} \right)
\end{align}
where the thermal conductivity of the face \(i+1/2\) between cells \(i\) and \(i+1\) is computed as
\begin{align}
k_{n,i+1/2} &= \left( \frac{\Delta n_{i}}{\Delta n_{i} + \Delta n_{i+1}} \cdot \frac{1}{k_{i}} + \frac{\Delta n_{i+1}}{\Delta n_{i} + \Delta n_{i+1}} \cdot \frac{1}{k_{i+1}} \right)^{-1} \\
&= \left(\Delta n_{i} + \Delta n_{i+1}\right) \left( \frac{\Delta n_{i}}{k_{i}} + \frac{\Delta n_{i+1}}{k_{i+1}} \right)^{-1}
\end{align}
so that the conductivity between cells is the half-length-weighted harmonic mean of their respective conductivities. If the cells have uniform thickness, then the weighting becomes half for each contribution, and the result is simply double the harmonic mean.

Similarly, the value of \(\rho \cp\) for boundary \(i+1/2\) may be determined as the volume-weighted atrithmetic mean of cells \(i\) and \(i+1\) as
\begin{align}
\rho_{i+1/2} c_{p,i+1/2} &= \frac{\left( \Delta n_{i} \rho_{i} + \Delta n_{i+1} \rho_{i+1} \right) \left( \Delta n_{i} c_{p,i} + \Delta n_{i+1} c_{p,i+1} \right)}{\Delta n_{i} + \Delta n_{i+1}}
\end{align}
where the areas cancel and the volume-weighting becomes length-weighting. If the cells have uniform thickness then the weighted average becomes a simple arithmetic mean.

The convective heat equation becomes
\begin{align}
\rho \cp \Dd{u}{t} &= \sum_{n} h_{n} \Dd{\left( u_{\infty,n} - u \right )}{n} \\
\D{u} &= \sum_{n} \frac{h_{n}\D{t}}{2 \rho_{n} c_{p,n}\D{n}} \Delta u
\end{align}
because the ambient temperature at boundary \(i+1/2\) may be expressed as
\begin{align}
u_{\infty,n,i+1/2} &= \frac{u_{i} + u_{i+1}}{2}
\end{align}
which is the arithmetic mean of the temperature in cells \(i\) and \(i+1\). The unweighted arithmetic mean is reasonable because it is symmetric and because it uses the average temperature at exactly the boundary. Note the convection coefficient need not be averaged because it is empirically determined for the boundary in question. Every pair of materials which have an interface must have a well-defined convection coefficient.

\section{Discretization Approach}

Because the conductive heat equation is second order and the governing partial differential equation is parabolic we choose to use a generalized Crank-Nicolson approach. The approach can be thought of as a linear interpolation of a fully explicit and fully implicit second-order central difference. Because the next time step is implicitly dependent on the previous time step, a system of equations must be solved. The overall system may be expressed as
\begin{align}
\cnm^{j+1}_{m} u^{j+1} + \cnb^{j+1}_{m} &= \cnm^{j}_{e} u^{j} + \cnb^{j}_{e} + \cnd^{j}
\end{align}
where \(\cnm\) are the coefficient matrices, \(u^{j}\) is the temperature vector of the \(j\)-th time step, \(\cnb\) are the global boundary conditions and subscripts \(m\) and \(e\) represent implicit and explicit parts of the Crank-Nicolson equations. The term \(\cnd^{j}\) is the \(-\nabla k \cdot \nabla u\) and will be discussed in more detail later. The solution to the system of equations is given by
\begin{align}
u^{j+1} &= \left(\cnm^{j+1}_{m}\right)^{-1} \left( \cnm^{j}_{e} u^{j} + \cnb^{j}_{e} - \cnb^{j+1}_{m} + \cnd^{j} \right)
\end{align}
provided \(\cnm_{m}^{j+1}\) is invertible, and is the next time-step of the temperature field.

The matrix \(\cnm_{e}\) has one row per cell in the domain. The matrix is zero everywhere except on the primary diagonal and six off-diagonals. The primary diagonal entries correspond to the cells of the domain, while the off-diagonals correspond to the contributions from each cell's six neighbors. The off-diagonal entries in the upper-triangular portion are equal to \(\left( 1 - \theta \right) \alpha_{n,i}\) where \(n \in \left\{x,y,z\right\}\) and \(i\) is the \(i\)-th row of the matrix. The coefficient \(\theta\) is implicitness parameter, and may take any value in \(\left[0,1\right]\). The value \(0\) implies the system is fully explicit, \(1\) fully implicit, and \(0.5\) the Crank-Nicolson method proper. The values of the \(\alpha\) may be computed as
\begin{align}
\alpha_{k,n,i} &= \frac{k_{n,i+1/2}\D{t}}{\rho_{n,i+1/2} c_{p,n,i+1/2}\D{n}^2} = \frac{k \D{t}}{\rho \cp \D{n}^2} \\
\alpha_{h,n,i} &= \frac{h_{n,i+1/2}\D{t}}{\rho_{n,i+1/2} c_{p,n,i+1/2}\D{n}} = \frac{h \D{t}}{\rho \cp \D{n}}
\end{align}
where the subscript \(k\) denotes a conductivity heat flux and \(h\) denotes convection. The sign of the half-index is flipped for the lower-triangular portion. All of the alphas are symmetric with respect to the sign of the half-indiex, so the matrix will be symmetric.

Matrix symmetry implies that diffusivity between elements \(i\) and \(i'\) is the same as between \(i'\) and \(i\), which must hold by physical symmetry. Because the matrix is symmetric, the lower-triangular portion may be constructed directly from the upper portion. The diagonal indices of the upper-triangular portion containing non-zero entries are the strides of the underlying discretization of the domain. The stride along the \(X\)-axis is \(1\), along the \(Y\)-axis is the number of elements along the \(X\)-axis, and along the \(Z\)-axis is the product of the number of elements along the \(X\)- and \(Y\)-axes.

Off-diagonal entries which imply the neighboring cell lies outside the domain are set to zero in \(\cnm_{e}\). These entries require application of the global boundary condition, which is the origin of the \(\cnb\) terms. For some elements these values are implicit in the matrix. For other elements they lie in rows or columns outside the choice of matrix. Rather than expanding the matrix, which is unwieldy and destroys the regularity of the band structure, instead all such entries are set to zero. As a result the boundary conditions must be exported to a vector \(\cnb_{e}\). The vector is the product of \(u_{\infty}\) with \(\left( 1 - \theta \right) \alpha_{\infty,n,i}\) for each such boundary entry. The boundary entries for a given cell are summed together in the vector. For an element at the vertex of the domain there are thus three \(\alpha_{\infty,n,i}\) terms, at an edge two, on a face one, and zero everywhere else. For the vector \(\cnb_{m}\) the elements are sums of \(-\theta \alpha_{\infty,n,i}\).

The values on the primary diagonal of \(\cnm_{e}\) are equal to \(p_{i} = 2 - 2 \left( 1 - \theta \right) \sum_{n} \alpha_{n,i} - 2 \left( 1 - \theta \right) \alpha_{\infty,n,i}\) where \(\alpha_{n,i}\) is the upper diagonal element associated with axis \(n\) in the \(i\)-th row and \(\alpha_{\infty,n,i}\) is the \(i\)-th ambient diffusivity as used in \(\cnb_{e}\). The doubling of the boundary and vector contributions is due to matrix symmetry. The matrix \(\cnm_{m}\) is identical to \(\cnm_{e}\) except off-diagonal entries are \(-\alpha_{n,i}\) and primary diagonal entries are \(p_{i} = 2 + 2 \theta \sum_{n} \alpha_{n,i} + 2 \theta \alpha_{\infty,n,i}\), where the entries of the last term come from \(\cnb_{m}\).

Up to this point only the convective heat flux and second term of the conductive heat flux have been considered. It is believed the value of \(\cnd\) will be largest between liquid and solid because there is often a sharp change in conductivity with solidification. A product of central difference schemes will be used to compute those values where possible. If a convective boundary exists on one face of an element along a direction, then that element will use the forward or backward difference for that direction instead, as appropriate. If the element has two convective boundaries on the same axis, then the value of the term is assumed to be zero along that axis. While it is possible to incorporate the term into the general Crank-Nicolson solver with additional mathematical effort, it is simpler to assume the term acts like an additional boundary condition and add the term to the right-hand side.

\section{Adaptive Time-stepping}

The previous section dealt with the solution to a single time-step. Determining how large that time-step must be is another matter. Adaptive time-stepping is used based on a target heat extraction rate. An initial guess is made at a time step, and a candidate temperature field computed using the method described in the previous section. The enthalpy of every point in the previous temperature field and candidate field are computed. The enthalpy change of every point is computed. If the largest enthalpy change over the domain is within some tolerance of a target value, then the candidate is accepted as the next time step. If not, then a modified bisection method is used to find the next candidate time step. The initial range of the bisection method is zero to infinity. If the enthalpy change is smaller than the target and the upper limit is still infinity, then the next guess is double the current guess. Otherwise the bisection proceeds as usual.

\section{Numerical Methods}

To solve the system of equations it is possible to use any linear system solver. The choice made here is a preconditioned conjugate gradient method using an initial guess. The preconditioner is an incomplete Cholesky factorization of \(\cnm_{m}\). The initial guess is whichever previous candidate temperature field is closest to the target while determining the next time step.

It is common for the system to be stiff when real solidification properties are used due to the larger latent heat evolution over short time frames. A regularization procedure is used when calculating \(\cp\). The values of enthalpy are precomputed from the values of \(\cp\) for each temperature. The values of enthalpy are looked up using the current and previous time steps. The difference in enthalpy divided by difference in temperature is a linear approximation to \(\cp\). Note that the properties are lagged in time by one step. Other methods are available but require considerably more effort to implement. For the first time step, there is not yet a previous time step. In that case \(\cp\) is used directly.

\end{document}