\subsection{Separation axioms}\label{subsec:separation_axioms}

\begin{definition}\label{def:topological_space_separation}
  Two subsets \( A, B \subseteq X \) of a topological space \( (X, \mscrT) \) are called \term{separated} or \term{separated using neighborhoods} if there exist disjoint open sets \( U \supseteq A \) and \( V \supseteq B \). In particular, two points are separated if their respective singleton sets are separated.

  We say that \( A \) and \( B \) are \term{functionally separated} if there exists a continuous function \( f: X \to [0, 1] \) such that \( f(A) = 0 \) and \( f(B) = 1 \).
\end{definition}

\begin{definition}\label{def:separation_axioms}
  We can classify topological spaces using the following separation axioms. Fix a topological space \( (X, \mathcal{T}) \).

  \begin{thmenum}
    \thmitem[def:separation_axioms/T0]{T0} (Kolmogorov) \( X \) is \( T_0 \) if for every two different points \( x, y \in X \), there exists an open set \( U \in \mathcal{T} \) such that either \( x \in U \) or \( y \in U \).
    \thmitem[def:separation_axioms/T0.5]{T0.5} \( X \) is \( T_{0.5} \) if every singleton set \( \{ x \} \) is either open or closed.
    \thmitem[def:separation_axioms/T1]{T1} (Frechet) \( X \) is \( T_1 \) if every singleton set \( \{ x \} \) is closed.
    \thmitem[def:separation_axioms/T2]{T2} (Hausdorff) \( X \) is \( T_2 \) if every two different points \( x, y \in X \) can be separated using neighborhoods, i.e. there exist disjoint open sets \( U \ni x \) and \( V \ni y \).

    \thmitem[def:separation_axioms/T3]{T3} \( X \) is \term{regular} if every point and every closed set can be separated using \hyperref[def:topological_space_separation]{neighborhoods}.

    If in addition to being regular \( X \) is \ref{def:separation_axioms/T0}, we say that \( X \) is a \( T_3 \) space.

    \thmitem[def:separation_axioms/T3.5]{T3.5} (Tychonoff) \( X \) is \term{completely regular} if every point and every closed set can be functionally \hyperref[def:topological_space_separation]{separated}.

    If in addition to being completely regular \( X \) is \ref{def:separation_axioms/T0}, we say that \( X \) is a \( T_{3.5} \) space.

    \thmitem[def:separation_axioms/T4]{T4}(Urysohn) \( X \) is \term{normal} every two closed sets \( F, G \in \mathcal{F}_{\mathcal{T}} \) can be separated using neighborhoods, i.e. there exist disjoint open sets \( U \supseteq F \) and \( V \supseteq G \).

    If in addition to being normal \( X \) is \ref{def:separation_axioms/T1}, we say that \( X \) is a \( T_4 \) space.

    \thmitem[def:separation_axioms/T5]{T5} If every subspace of a \( T_4 \) space \( X \) is \ref{def:separation_axioms/T4}, we say that \( X \) is a \( T_5 \) space or a \term{completely normal space}.

    \thmitem[def:separation_axioms/T6]{T6} If every closed set in a \( T_4 \) space \( X \) is \( G_\delta \) (see \fullref{def:borel_algebra}), we say that \( X \) is a \( T_6 \) space or a \term{perfectly normal space}.
  \end{thmenum}
\end{definition}

\begin{proposition}\label{thm:separation_axioms_cascade}
  Each numbered axiom in \fullref{def:separation_axioms} implies the previous one.
\end{proposition}

\begin{proposition}\label{thm:t2_iff_singleton_limits}
  A topological space is \hyperref[def:separation_axioms/T2]{Hausdorff} if and only if every \hyperref[def:topological_net]{net} has at most one \hyperref[def:net_convergence/limit]{limit}.
\end{proposition}
\begin{proof}
  \SufficiencySubProof Let \( X \) be Hausdorff and assume that there exists a net \( \{ x_k \}_{k \in \mscrK} \) such that \( y \) and \( z \) are not necessarily distinct limit points.

  Fix neighborhoods \( U \) of \( y \) and \( V \) of \( z \). Since both are limit points, there exist indices \( k_U \) and \( k_V \) such that \( k \geq k_U \) implies \( x_k \in U \) and \( i \geq i_k \) implies \( x_k \in V \).

  Since \( \mscrK \) is a directed set, there exists an upper bound \( k_0 \) of \( k_U \) and \( k_V \). Thus,
  \begin{equation*}
    x_k \in U \cap V \quad\forall k \geq k_0.
  \end{equation*}

  In particular, the intersection \( U \cap V \) is nonempty and is a neighborhood of both \( y \) and \( z \).

  If \( y \neq z \), then we have two distinct points such that no two neighborhoods of \( y \) and \( z \), respectively, are disjoint. This contradicts the assumption that \( X \) is Hausdorff. Thus, \( y = z \).

  \NecessitySubProof Conversely, if \( X \) is not Hausdorff, then for every two distinct points \( y \) and \( z \) and every two neighborhoods \( U \ni y \) and \( V \ni z \), their intersection \( U \cap V \) is nonempty.

  Let \( \mathcal{U} \) and \( \mathcal{V} \) be the sets of all neighborhoods of \( y \) and \( z \), respectively. Since they are both partially ordered by set inclusion \( \subseteq \), define the directed set \( (\mathcal{U} \times \mathcal{V}, \leq) \) with order
  \begin{equation*}
    (U, V) \leq (U', V') \iff U \supset V \T{and} U' \supset V'.
  \end{equation*}

  For each \( (U, V) \in \mathcal{U} \times \mathcal{V} \), choose a point \( x_{(U, V)} \) from \( U \cap V \).

  Thus, the net \( \{ x_{(U, V)} \}_{(U, V) \in \mathcal{U} \cap \mathcal{V}} \) has both \( y \) and \( z \) as its limit points, which contradicts our initial assumption.
\end{proof}

\begin{lemma}[Urysohn's lemma]\label{thm:urysohns_lemma}\mcite[1.5.11]{Engelking1989}
  In a \hyperref[def:separation_axioms/T4]{normal space}, every pair \( A, B \) of disjoint closed sets can be functionally \hyperref[def:topological_space_separation]{separated}.
\end{lemma}

\begin{theorem}\label{thm:separation_axioms_of_product}
  Fix is an indexed family \( \{ X_k \}_{k \in \mscrK} \) of topological spaces. Denote their \hyperref[def:topological_product]{product} by \( X \).

  \begin{thmenum}
    \thmitem{thm:separation_axioms_of_product/direct}\cite[theorem 2.3.11]{Engelking1989} If each one of \( X_k \) is a \( T_i \) space for \ref{def:separation_axioms/T0}-\ref{def:separation_axioms/T3.5}, then \( X \) is also a \( T_i \) space.

    \thmitem{thm:separation_axioms_of_product/inverse}\cite[theorem 2.3.11]{Engelking1989} If \( X \) is a \( T_i \) space for \ref{def:separation_axioms/T0}-\ref{def:separation_axioms/T6}, then each component \( X_k \) is also a \( T_i \) space.
  \end{thmenum}
\end{theorem}


\subsection{Linear Approximations with the Gradient}
\noindent
We can rewrite our linear approximation of $f$ at $x_0$ using the gradient.
\begin{equation*}
	f(x) \approx f(x_0) + (\nabla f)(x_0) \cdot (x - x_0)
\end{equation*}


\subsection{Convergence in probability and o-notation}

\subsubsection{Introduction}

Converges in probability

\(P(distance(X_n, X)>\epsilon )\rightarrow 0\)

For all \(\epsilon \).

\(X_n \rightarrow^P X\)

\subsubsection{Little o notation}

Little o notation is used to describe convergence in probability.

\(X_n=o_p(a_n)\)

mean that

\(\dfrac{X_n}{a_n}\)

Converges to \(0\) and \(n\) approaches something

Can be wrtiten:

\(\dfrac{X_n}{a_n}=o_p(1)\)

\subsubsection{Big O notation}

Big O notation is used to describe boundedness.

\(X_n=O_p(a_n)\)

means that:

If something is little o, it is big O.


\section{Classes of Distributions}

\subsection{Stable Distributions}

Let $X_1,X_2,...X_n$ be i.i.d. random variables. Let $S_n = \sum_i X_i$. $X_s$ has stable distibution if, for sufficiently large $n$, the (properly normalized) sum $S_n$ converges in distribution to the distribution of $X_s$.

\begin{equation}
\frac{S_n-\alpha_n}{\beta_n} \xrightarrow{dist} X_s
\end{equation}

\subsection{Sub-exponential}

\subsection{Exponential}

The exponential family of distributions includes many of the most commonly encountered (normal, exponential, gamma, chi-squared, beta, Dirichlet, Bernoulli, categorical, Poisson, Wishart, inverse Wishart). They have a number of very useful properties, such as conjugate priors and the "summarization" of arbitrary amounts of data in terms of sufficient statistics (for example, to fit a Gaussian, you only need the mean and the variance of an arbitrarily large dataset).

\begin{equation}
f_X(x|\mathbf{\theta}) = h(x) \exp{\mathbf{\eta(\theta)} \cdot \mathbf{T(x)} - A(\mathbf{\theta})}
\end{equation}

Where $\mathbf{\theta}$ is a vector of distribution parameters, $\mathbf{\eta}$ is the natural parameter (the set of values for which $f(x|\mathbf{\theta})$ is finite is the natural parameters space and is always convex), $T(x)$ is the sufficient statistic (it encapsulates all the information that the data $x$ holds with respect to the parameters of the distribution) and $A(\eta)$ is the log partition function.

\subsubsection{Example: Normal Distribution}

For a normal distribution with unknown mean and unknown variance:

\begin{equation}
f(x|\mu,\sigma) = \frac{1}{\sqrt{2\pi}\sigma}\exp{-\frac{(x-\mu)^2}{2\pi \sigma^2}	}
\end{equation}

\begin{itemize}
\item $\mathbf{\theta} = \left[\mu, \theta \right]$
\item $\mathbf{\eta(\theta)} = \left[ \frac{\mu}{\sigma^2}, -\frac{1}{2\sigma^2}\right]$
\item $\mathbf{T(x)} = \left[x, x^2 \right]$
\item $A(\mathbf{\theta}) = \frac{\mu^2}{2\sigma^2} + \log|\sigma|$
\item $h(x) = \frac{1}{\sqrt{2\pi} \sigma} \exp{-\frac{x^2}{2\pi\sigma^2}}$
\end{itemize}

\subsection{Elliptical}


\documentclass[10pt]{article}
\author{Alex Peyrard}
\title{Information and coding theory assignment 2}

\begin{document}
\maketitle

\section{$H(X,Y|Z)$}

\[H(X,Y|Z)=-\sum\limits_{x,y,z}\mathcal{P}(x,y,z)\log(\mathcal{P}(x,y|z))\]
\[H(X,Y|Z)=-\sum\limits_{x,y,z}\mathcal{P}(x,y,z)\log(\frac{\mathcal{P}(x,y,z)}{\mathcal{P}(z)})\]
\[H(X,Y|Z)=-(1/4\log(\frac{1/4}{1/2})+1/4\log(\frac{1/4}{1/2})+1/4\log(\frac{1/4}{1/2})+1/4\log(\frac{1/4}{1/2})+0+0+0+0)\]
\[H(X,Y|Z)=-\log(\frac{1/4}{1/2})\]
\[H(X,Y|Z)=\log(2)\]

\section{$H(X,Y|Z=0)$}

\[H(X,Y|Z=0)=-\sum\limits_{x,y,z=0}\mathcal{P}(x,y,z=0)\log(\mathcal{P}(x,y|z=0))\]
\[H(X,Y|Z=0)=-\sum\limits_{x,y,z=0}\mathcal{P}(x,y,z=0)\log(\frac{\mathcal{P}(x,y,z=0)}{\mathcal{P}(z=0)})\]
\[H(X,Y|Z=0)=-(1/4\log(\frac{1/4}{1/2})+1/4\log(\frac{1/4}{1/2})+0+0)\]
\[H(X,Y|Z=0)=-1/2\log(\frac{1/4}{1/2})\]
\[H(X,Y|Z=0)=1/2\log(2)\]

\section{$I(X,Y|Z)$}

\[I(X,Y|Z)=\sum\limits_{x,y,z}\mathcal{P}(x,y,z)\log(\frac{\mathcal{P}(x,y|z)}{\mathcal{P}(x|z)\mathcal{P}(y|z)})\]
\[I(X,Y|Z)=4*1/4\log(\frac{1/2}{1/4*1/4})\]
\[I(X,Y|Z)=3\]

\section{$I(X,Y|Z=0)$}

\[I(X,Y|Z=0)=\sum\limits_{x,y,z=0}\mathcal{P}(x,y,z=0)\log(\frac{\mathcal{P}(x,y|z=0)}{\mathcal{P}(x|z=0)\mathcal{P}(y|z=0)})\]
\[I(X,Y|Z=0)=2*1/4\log(\frac{1/2}{1/4*1/4})\]
\[I(X,Y|Z)=3/2\]

\section{Proof of $I(X,Y)=0$}

\[I(X,Y)=\sum\limits_{x,y}\mathcal{P}(x,y)\log(\frac{\mathcal{P}(x,y)}{\mathcal{P}(x)\mathcal{P}(y)})\]
\[I(X,Y)=4*1/4\log(\frac{1/4}{1/2*1/2})\]
\[I(X,Y)=\log(1)\]
\[I(X,Y)=0\]
QED

\end{document}


\subsection{Matrix representation}

\subsubsection{Representing linear maps as matrices}

We previously discsussed morphisms on vector spaces. We can write these as matrices.

Matrices represents transformations of vector spaces

\subsubsection{Representing vectors as matrices}

We can represent vectors as row or column matrices.

\(v=\begin{bmatrix}a_{1} & a_{2}&...&a_{n}\end{bmatrix}\)

\(v=\begin{bmatrix}a_{1}\\a_{2}\\...\\a_{m}\end{bmatrix}\)


\chapter{Calculations}\todo{This is not meant to be here, just calculations for the project}

\section{12V 5A MOLEX Pin}

\hspace{3.5mm} $I = \dfrac{P}{U}$

\vspace{2mm}

$\Updownarrow P = U \cdot I$

\vspace{4mm}

We have a 12V MOLEX pin which can deliver 5A

\vspace{2mm}

$\mathbf{12V \cdot 5A = 60W}$

\vspace{2mm}

This means that we have 60W in total, on that MOLEX pin

\vspace{2mm}

\paragraph{Raspberry PI} - Consumes 5V 3A

\vspace{2mm}

$\mathbf{5V \cdot 3A = 15W}$

\vspace{2mm}

This means that the Raspberry PI consumes 15W at peak.

\vspace{2mm}

$\mathbf{60W - 15W = 45W}$

\vspace{4mm}

Leaving 45W left on the MOLEX pin which is equal to

\vspace{2mm}

$\mathbf{\dfrac{45W}{12V} = 3.75A}$

\section{19V 2A MOLEX Pin}

$I = \dfrac{P}{U}$, $P = U \cdot I$

\vspace{4mm}

We have a 19V MOLEX pin which can deliver 2A

\vspace{2mm}

$\mathbf{19V \cdot 2A = 38W}$


\documentclass[9pt, oneside]{amsart}   	% use "amsart" instead of "article" for AMSLaTeX format
\usepackage[top=1cm,left=1.2cm, right=1.2cm, bottom=1cm]{geometry}                		% See geometry.pdf to learn the layout options. There are lots.
\geometry{letterpaper}                   		% ... or a4paper or a5paper or ...
%\geometry{landscape}                		% Activate for rotated page geometry
%\usepackage[parfill]{parskip}    		% Activate to begin paragraphs with an empty line rather than an indent
\usepackage{graphicx}				% Use pdf, png, jpg, or eps§ with pdflatex; use eps in DVI mode
\usepackage{bm}							% TeX will automatically convert eps --> pdf in pdflatex
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{subcaption}
\usepackage{float}
\pagenumbering{gobble}
%SetFonts

%SetFonts

\title{}
\author{}
\date{}							% Activate to display a given date or no date

\begin{document}

\section{Learning in discrete graphical models}
Let $(\mathbf{x}_i ,\mathbf{z}_i)_{1, ..., n}$ be an i.i.d sample of observations. $p(z)$ can be written as $p(\mathbf{z}_i|\bm{\pi}) = \prod_{m=1}^{M}\pi_m^{z_{im}}$ and $p(\mathbf{x}_i|\mathbf{z}_i, \bm{\Theta}) = \prod_{m,k}(\theta_{mk})^{z_{im}x_{ik}}$
The log-likelihood of the observations as a function of $\bm{\pi}$ and $\bm{\Theta} = (\theta_{mk})$:

\begin{align*}
	\ell(\bm{\pi}, \bm{\Theta}) &= \sum_{i=1}^n \log p(\mathbf{x}_i, \mathbf{z}_i|\bm{\pi}, \bm{\Theta}) = \sum_{i=1}^n\log p(\mathbf{x}_i|\mathbf{z}_i, \bm{\Theta}) + \sum_{i=1}^n \log p(\mathbf{z}_i|\bm{\pi})
\end{align*}
The maximum likelihood estimator of the right term is given by $\bm{\hat{\pi}} = (\frac{n_1}{n}, \frac{n_0}{n}, ..., \frac{n_M}{n})$ (multinomial distribution and the two terms depend on different parameters). For the left term, with $\mathcal{A}_m = \{i | z_{im}=1\}$ the indexes of the pairs with $z=m$, $n_m = |\mathcal{A}_m|$ and $n_{mk}$ the number of pairs $(x=k, z=m)$:

\begin{align*}
	\sum\log p(\mathbf{x}_i|\mathbf{z}_i, \bm{\Theta}) = \sum_i\sum_{k,m} z_{im}x_{ik}\log\theta_{mk} = \sum_{m=1}^M \left(\sum_{i\in\mathcal{A}_m}\sum_{k=1}^K x_{ik}\log\theta_{mk}\right) = \sum_{m=1}^M  \sum_{k=1}^Kn_{mk}\log\theta_{mk}
\end{align*}
By analogy with the multinomial distribution maximum likelihood estimator, each term in the sum over $m$ is maximized by $\bm{\hat{\theta}}_m = \left(\frac{n_{mk}}{|\mathcal{A}_m|}\right)_k$, hence $\boxed{\bm{\hat{\Theta}}= \left(\frac{n_{mk}}{n_m}\right)_{km}}$ and $\boxed{\bm{\hat{\pi}} = \left(\frac{n_m}{n}\right)_m}$
\newline

\section{Linear classification - Generative model (LDA) (2.1 (a))}
Let $(\mathbf{x}_i , y_i)_{1, ..., n}$ be an i.i.d sample of observations.
$$p(\mathbf{x}_i, y_i | \pi, \bm{\bm{\mu}}, \bm{\Sigma}) = p(\mathbf{x}_i|y_i)\cdot p(y_i) = \mathcal{N}(\mathbf{x}_i | \bm{\mu}_1, \bm{\Sigma})^{y_i} \mathcal{N}(\mathbf{x}_i | \bm{\mu}_0, \bm{\Sigma})^{1-y_i}\cdot \pi^{y_i}(1-\pi)^{1-y_i}$$
The log-likelihood as a function of all parameters is
\begin{align*}
	\ell(\pi, \bm{\mu}_1, \bm{\mu}_0, \bm{\Sigma}) = \sum_{i}\left( y_i\log\pi+(1-y_i)\log(1-\pi) \right)+ \sum_i y_i \log\mathcal{N}(\mathbf{x}_i| \bm{\mu}_1, \bm{\Sigma}) + \sum_i (1-y_i) \log\mathcal{N}(\mathbf{x}_i| \bm{\mu}_0, \bm{\Sigma})
\end{align*}
The first sum depends only on $\pi$ and is maximized by $\boxed{\hat{\pi} = n_1/n}$. Since we have for the terms depending on $\bm{\mu}_1$
\begin{align*}
	\sum_i y_i \log\mathcal{N}(\mathbf{x}_i|\bm{\mu}_1, \bm{\Sigma}) = -1/2 \sum_i y_i (\mathbf{x}_i- \bm{\mu}_1)^T\bm{\Sigma}^{-1}(\mathbf{x}_i- \bm{\mu}_1) + const.
\end{align*}
By setting the derivatives with respect to $\bm{\mu}_1$ to 0 and by symmetry for $\bm{\mu}_0$: $\boxed{\hat{\bm{\mu}}_1= (1/n_1)\sum y_i\mathbf{x}_i}$ and $\boxed{\hat{\bm{\mu}}_0 = (1/n_0)\sum (1-y_i)\mathbf{x}_i}$. The terms depending on $\bm{\Sigma}$ are

\begin{align*}
	-\dfrac{1}{2}\sum_i y_i\left(\log|\bm{\Sigma}| + (\mathbf{x}_i - \bm{\mu}_1)^T\bm{\Sigma}^{-1} (\mathbf{x}_i - \bm{\mu}_1)\right) - \dfrac{1}{2}\sum_i (1-y_i)\left(\log|\bm{\Sigma}| + (\mathbf{x}_i - \bm{\mu}_0)^T\bm{\Sigma}^{-1} (\mathbf{x}_i - \bm{\mu}_0)\right) =\\ -\dfrac{n}{2}\log|\bm{\Sigma}| -\dfrac{n}{2}\text{tr}(\bm{\Sigma}^{-1}\mathbf{S})
\end{align*}
Where $\mathbf{S} = \frac{n_1}{n}\left(\frac{1}{n_1}\sum_{y_i=1} (\mathbf{x}_i-\bm{\mu}_1)(\mathbf{x}_i - \bm{\mu}_1)^T\right) + \frac{n_0}{n}\left(\frac{1}{n_0}\sum_{y_i=0} (\mathbf{x}_i-\bm{\mu}_0)(\mathbf{x}_i - \bm{\mu}_0)^T\right)$ convex sum of the covariances in each class. We can then find that maximum likelihood estimator for $\bm{\Sigma}$ is $\boxed{ \hat{\bm{\Sigma}} = \mathbf{S}}$ by setting the derivatives to 0 in the above expression.
We now study $p(y|\mathbf{x}) \propto p(\mathbf{x}|y)p(y)$
\begin{align*}
	p(y|\mathbf{x}) \propto \pi^y(1-\pi)^{1-y}\mathcal{N}(\mathbf{x}|\mu_1, \bm{\Sigma})^y \mathcal{N}(\mathbf{x}|\mu_0, \bm{\Sigma})^{1-y} \propto \exp\left( y \left(\log\frac{\pi}{1-\pi} + \frac{1}{2}(\mu_0^T\bm{\Sigma}^{-1}\mu_0 -\mu_1^T\bm{\Sigma}^{-1}\mu_1 ) \right) + y(\mu_1-\mu_0)^T\bm{\Sigma}^{-1}X \right)
\end{align*}
Thus, $p(y=k|\mathbf{x}) = \dfrac{\exp(ya+y\mathbf{b}^T\mathbf{x})}{1 + \exp(a+\mathbf{b}^T\mathbf{x} )}$ and $\boxed{p(y=1|\mathbf{x}) = \sigma(a+\mathbf{b}^T\mathbf{x})}$ with $a$ and $\mathbf{b}$ defined in the above formula. It is therefore equivalent to a logistic regression with a closed formula for the coefficients based on the empirical means, covariances and class repartition of the data.
\newline

\section{Linear classification - QDA model (2.5 (a))}
Results are the same as above for $\hat{\pi}$, $\hat{\bm{\mu}}_1$, $\hat{\bm{\mu}}_0$ but $\hat{\bm{\Sigma}}_1$ and $\hat{\bm{\Sigma}}_0$ must be computed separately. The term of the log-likelihood that depends on $\bm{\Sigma}_1$ is
\begin{align*}
	-\dfrac{1}{2}\sum_i y_i\left(\log|\bm{\Sigma}_1| + (\mathbf{x}_i - \bm{\mu}_1)^T\bm{\Sigma}_1^{-1} (\mathbf{x}_i - \bm{\mu}_1)\right) = -\dfrac{n_1}{2}\log|\bm{\Sigma}_1| -\dfrac{1}{2}\text{tr}\left(\bm{\Sigma}_1^{-1}\sum_{i|y_i=1}(\mathbf{x}_i - \bm{\mu}_1)(\mathbf{x}_i - \bm{\mu}_1)^T\right)
\end{align*}
Hence $\boxed{\hat{\bm{\Sigma}}_1 = \frac{1}{n_1}\sum_{i|y_i=1}(\mathbf{x}_i - \bm{\mu}_1)(\mathbf{x}_i - \bm{\mu}_1)^T}$ and $\boxed{\hat{\bm{\Sigma}}_0 = \frac{1}{n_0}\sum_{i|y_i=0}(\mathbf{x}_i - \bm{\mu}_0)(\mathbf{x}_i - \bm{\mu}_0)^T}$.

We also have $\boxed{p(y=1|\mathbf{x}) = \sigma\left(a + \mathbf{b}^T\mathbf{x} + \mathbf{x}^T\mathbf{C}\mathbf{x}\right)}$ with $a= \log\frac{\pi}{1-\pi} + \frac{1}{2}(\mu_0^T\bm{\Sigma_0}^{-1}\mu_0 -\mu_1^T\bm{\Sigma_1}^{-1}\mu_1)$, $\mathbf{b}=\mu_1^T\bm{\Sigma_1}^{-1} - \mu_0^T\bm{\Sigma_0}^{-1}$ and $\mathbf{C} = \bm{\Sigma_0}^{-1} - \bm{\Sigma_1}^{-1}$. The model can be interpreted as a logistic regression in the space of the features $(\mathbf{x}_1, \mathbf{x}_2, \mathbf{x}_1\mathbf{x}_2, \mathbf{x}_1^2, \mathbf{x}_2^2)$ with closed formula for the coefficients.

\clearpage
\begin{figure}[t!]
\centering
\begin{subfigure}{.45\textwidth}
  \centering
  \includegraphics[width=\linewidth]{classificationA_lda.pdf}
 % \caption{Linear Fourier Approximation}
\end{subfigure} \hspace{5pt}
\begin{subfigure}{.45\textwidth}
  \centering
  \includegraphics[width=\linewidth]{classificationA_logistic.pdf}
  %\caption{Non-linear Fourier Approximation}
\end{subfigure}
\\[+5pt]
\begin{subfigure}{.45\textwidth}
  \centering
  \includegraphics[width=\linewidth]{classificationA_linear.pdf}
  %\caption{Linear Wavelets Approximation}
\end{subfigure} \hspace{5pt}
\begin{subfigure}{.45\textwidth}
  \centering
  \includegraphics[width=\linewidth]{classificationA_qda.pdf}
  %\caption{Linear Wavelets Approximation}
\end{subfigure}
\end{figure}

\hfill
 \begin{minipage}{.45\textwidth}
  \begin{table}[H]
   \centering
    \begin{tabular}{c|cccc}
    Method & LDA & Logistic & Linear & QDA \\
    \hline
    Train Error & 0.0134 & 0.0000 & 0.0134 & 0.0067 \\
    \hline
    Test Error & 0.0207 & 0.0354 & 0.0207 & 0.0193 \\
    \end{tabular}
\end{table}

 \end{minipage}\hfill
 \begin{minipage}{.45\textwidth}
	Linear Regression and LDA give very similar results, with very close decision boundaries.\\
	The QDA outperforms the other three methods in terms of classification error on the test set.\\
	Since the data is linearly separable, the logistic regression is able to find a separating hyperplane, achieving a 100\% classification score on the train set at the expense of the test error that is higher than for all other methods.
 \end{minipage}\hspace{1cm}

\clearpage
\begin{figure}[t!]
\centering
\begin{subfigure}{.45\textwidth}
  \centering
  \includegraphics[width=\linewidth]{classificationB_lda.pdf}
 % \caption{Linear Fourier Approximation}
\end{subfigure} \hspace{5pt}
\begin{subfigure}{.45\textwidth}
  \centering
  \includegraphics[width=\linewidth]{classificationB_logistic.pdf}
  %\caption{Non-linear Fourier Approximation}
\end{subfigure}
\\[+5pt]
\begin{subfigure}{.45\textwidth}
  \centering
  \includegraphics[width=\linewidth]{classificationB_linear.pdf}
  %\caption{Linear Wavelets Approximation}
\end{subfigure} \hspace{5pt}
\begin{subfigure}{.45\textwidth}
  \centering
  \includegraphics[width=\linewidth]{classificationB_qda.pdf}
  %\caption{Linear Wavelets Approximation}
\end{subfigure}
\end{figure}

\hfill
 \begin{minipage}{.45\textwidth}
  \begin{table}[H]
   \centering
    \begin{tabular}{c|cccc}
    Method & LDA & Logistic & Linear & QDA \\
    \hline
    Train Error & 0.0301 & 0.0201 & 0.0301 & 0.0234 \\
    \hline
    Test Error & 0.0415 & 0.0430 & 0.0415 & 0.0235 \\
    \end{tabular}
\end{table}

 \end{minipage}\hfill
 \begin{minipage}{.45\textwidth}
 The results for linear regression and LDA are again very close. Their decision boundaries don't seem very good compared to the others. For the LDA this is in part due to the fact that the two sets have very different covariances, and assuming they are equal harms the performance of the classification. For the linear regression, the points farther to the left of the yellow set have a much higher least square penalty (linear regression penalizes values far from the objective either above or below) than the points of the blue set, leading to a shift of the decision boundary to the left.

 Logistic regression performs better than LDA and linear regression but still is limited by the linear nature of the boundary it assumes.

 The QDA model performs very well because it can cope with the geometric differences between the two sets by letting the covariances being different in the model.
 \end{minipage}\hspace{1cm}

\clearpage
\begin{figure}[t!]
\centering
\begin{subfigure}{.45\textwidth}
  \centering
  \includegraphics[width=\linewidth]{classificationC_lda.pdf}
 % \caption{Linear Fourier Approximation}
\end{subfigure} \hspace{5pt}
\begin{subfigure}{.45\textwidth}
  \centering
  \includegraphics[width=\linewidth]{classificationC_logistic.pdf}
  %\caption{Non-linear Fourier Approximation}
\end{subfigure}
\\[+5pt]
\begin{subfigure}{.45\textwidth}
  \centering
  \includegraphics[width=\linewidth]{classificationC_linear.pdf}
  %\caption{Linear Wavelets Approximation}
\end{subfigure} \hspace{5pt}
\begin{subfigure}{.45\textwidth}
  \centering
  \includegraphics[width=\linewidth]{classificationC_qda.pdf}
  %\caption{Linear Wavelets Approximation}
\end{subfigure}
\end{figure}

\hfill
\begin{minipage}{.45\textwidth}
  \begin{table}[H]
   \centering
    \begin{tabular}{c|cccc}
    Method & LDA & Logistic & Linear & QDA \\
    \hline
    Train Error & 0.0551 & 0.0401 & 0.0551 & 0.0526 \\
    \hline
    Test Error & 0.0420 & 0.0227 & 0.0423 & 0.0403 \\
    \end{tabular}
\end{table}

 \end{minipage}\hfill
 \begin{minipage}{.45\textwidth}
 LDA and logistic regression are affected by the variance and mean shift that the blob of data introduces. The least square penalty moves the frontier such that the blob would be classified closer to 1.  The LDA assumes the data is centered in between the two yellow blobs because it cannot deal with separate sets, which affects its performance negatively.

 The logistic regression give better results and is able to handle the blob of data because it is robust to a large number of data points localized in a set and tries to optimize an error function which depends on the quality of classification.

The QDA isn't the most effective model for this data distribution. It is able to handle non linear boundaries that are useful for this kind of complex dataset, but still assumes that data within a class has a Gaussian distribution, which is not the case here.
 \end{minipage}\hspace{1cm}

 \end{document}


\chapter{Adding and Subtracting Polynomials}

Watch Khan Academy's \textbf{Adding polynomias} video at \url{https://youtu.be/ahdKdxsTj8E}

When adding two monomials of the same degree, you sum their coefficients:
\begin{equation*}
  7x^3 + 4x^3 = 11x^3
\end{equation*}\index{adding!monomials}

Using this idea, when adding two polynomials, you convert it into one long
polynomial and then simplify by combining terms with the same degree. For example:
\begin{multline*}
  (10x^3 - 2x + 13) + (-5x^2 + 7x -12) \\
  = 10x^3 + (-2)x + 13 + (-5)x^2 + 7x + (-12) \\
  = 10x^3 + (-5)x^2 + (-2 + 7)x + (13 - 12) \\
  = 10x^3 - 5x^2 + 5x + 1
\end{multline*}\index{adding!polynomials}

\begin{Exercise}[title=Adding Polynomials Practice, label=addpns]
  Add the following polynomials:
  \Question{$2x^3 - 5x^2 + 3x - 9$ and $x^3 - 2x^2 - 2x - 9$}
  \vspace{20mm}
  \Question{$3x^5 - 5x^3 + 3x^2 - x - 3$ and $2x^4 - 2x^3 - 2x^2 + x - 9$}
  \vspace{20mm}
\end{Exercise}
\begin{Answer}[ref=addpns]$3x^3 - 7x^2 + x - 18$ and $3x^5 - 7x^3 + x^2 - 12$\end{Answer}

Notice that in the second question, the degree 1 term disappears completely: $(-x) + x = 0$

One more tricky thing that can happen: Sometimes the coefficients don't add nicely.  For example:
\begin{equation*}
  \pi x^2 - 3 x^2 = (\pi - 3) x^2
\end{equation*}
That is as far as you can simplify it.

\section{Subtraction}

Now watch Khan Academy's \textbf{Subtracting polynomials} at \url{https://youtu.be/5ZdxnFspyP8}.

When subtracting one polynomial from the other, it is a lot like
adding two polynomials. The difference: when make the two polynomials
into one long polynomial, we multiply each monomial that is being
subtracted by -1. For example:
\begin{multline*}
  (2x^2 - 3x + 9) - (5x^2 - 7x + 4) \\
  = 2x^2 + (-3)x + 9 + (-5)x^2 + 7x + (-4) \\
  = (2 - 5)x^2 + (-3 + 7)x + (9-4) \\
  = -3x^2 + 4x + 5
\end{multline*}

\begin{Exercise}[title=Subtracting Polynomials Practice, label=subtractpns]
  Add the following polynomials:
  \Question{$(2x^3 - 5x^2 + 3x - 9) - (x^3 - 2x^2 - 2x - 9)$}
  \vspace{20mm}
  \Question{$(3x^5 - 5x^3 + 3x^2 - x - 3) - (2x^4 - 2x^3 - 2x^2 + x - 9)$}
  \vspace{20mm}
\end{Exercise}
\begin{Answer}[ref=subtractpns]$x^3 - 3x^2 + 5x$ and $x^5 - 3x^3 + 5x^2 - 2x + 6$\end{Answer}

\section{Adding Polynomials in Python}

As a reminder, in our Python code, we are representing a polynomial
with a list of coefficients.  The first coefficient is the constant
term. The last coefficient is the leading coefficient. So, we can
imagine $-5x^3 + 3x^2 - 4x + 9$ and $2x^3 +4x^2 - 9$ would look
like this: \textit{FIXME: Diagram here}

To add the two polynomials then, we sum the coefficients for each degree.
\textit{FIXME: Diagram here}

Create a file called \filename{add\_polynomials.py}, and type in the following:
\begin{Verbatim}
def add_polynomials(a, b):
    degree_of_result = len(a)
    result = []
    for i in range(degree_of_result):
        coefficient_a = a[i]
        coefficient_b = b[i]
        result.append(coefficient_a + coefficient_b)
    return result

polynomial1 = [9.0, -4.0, 3.0, -5.0]
polynomial2 = [-9.0, 0.0, 4.0, 2.0]
polynomial3 = add_polynomials(polynomial1, polynomial2)

print('Sum =', polynomial3)
\end{Verbatim}

Run the program.

Unfortunately, this code only works if the polynomails are the same length. For
example, try making \pyvar{polynomial1} have a larger degree than
\pyvar{polynomial2}:
\begin{Verbatim}
# x**4 - 5x**3 + 3x**2 - 4x + 9
polynomial1 = [9.0, -4.0, 3.0, -5.0, 1.0]

# 2x**3 + 4x**2 - 9
polynomial2 = [-9.0, 0.0, 4.0, 2.0]
polynomial3 = add_polynomials(polynomial1, polynomial2)
print('Sum =', polynomial3)
\end{Verbatim}

See the problem?

\begin{Exercise}[title=Dealing with polynomials of different degrees, label=pyaddpolys]

Can you fix the function \pyfunction{add\_polynomials} to handle polynomials of different degrees?

Here is a hint: In Python, there is a \pyfunction{max} function that returns the largest of the numbers it is passed.
\begin{Verbatim}
biggest = max(5,7)
\end{Verbatim}
Here \pyvar{biggest} would be set to 7.

Here is another hint: If you have an array \pyvar{mylist}, \pyvar{i},
a non-negative integer, is only a legit index if \texttt{i <
  len(mylist)}.
\end{Exercise}
\begin{Answer}[ref=pyaddpolys]
\begin{Verbatim}
def add_polynomials(a, b):
    degree_of_result = max(len(a), len(b))
    result = []
    for i in range(degree_of_result):
        if i < len(a):
            coefficient_a = a[i]
        else:
            coefficient_a = 0.0

        if i < len(b):
            coefficient_b = b[i]
        else:
            coefficient_b = 0.0

        result.append(coefficient_a + coefficient_b)
    return result
\end{Verbatim}
\end{Answer}

\section{Scalar multiplication of  polynomials}

If you multiply a polynomial with a number, the distributive property applies:
\begin{equation*}
  (3.1)(2x^2 + 3x + 1) = (6.2)x^2 + (9.3)x + 3.1
\end{equation*}
(When we are talking about things that are more complicated than a number, we use the word \emph{scalar} to mean ``Just a number''. So this is the product of a scalar and a polynomial.)

In \filename{add\_polynomials.py}, add a function to that multiplies a scalar and a polynomial:
\begin{Verbatim}
def scalar_polynomial_multiply(s, pn):
    result = []
    for coefficient in pn:
        result.append(s * coefficient)
    return result
\end{Verbatim}

Somewhere near the end of the program, test this function:
\begin{Verbatim}
polynomial4 = scalar_polynomial_multiply(5.0, polynomial1)
print('Scalar product =', polynomial_to_string(polynomial4))
\end{Verbatim}

\begin{Exercise}[title=Subtract polynomials in Python, label=pysubpoly]
Now implement a function that does subtraction using
\pyfunction{scalar\_polynomial\_multiply} and
\pyfunction{add\_polynomials}.

It should look like this:
\begin{Verbatim}
def subtract_polynomial(a, b):
    ...Your code here...

polynomial5 = [9.0, -4.0, 3.0, -5.0]
polynomial6 = [-9.0, 0.0, 4.0, 2.0, 1.0]
polynomial7 = subtract_polynomial(polynomial5, polynomial6)
print('Difference =', polynomial_to_string(polynomial7))
\end{Verbatim}
\end{Exercise}
\begin{Answer}[ref=pysubpoly]
\begin{Verbatim}
def subtract_polynomial(a, b):
    neg_b = scalar_polynomial_multiply(-1.0, b)
    return add_polynomials(a, neg_b)
\end{Verbatim}
\end{Answer}


\subsection{The Euler equation}

If we have exponential discounting we have:

\(U_T=E[ \sum_{t=T}^\infty (1+\delta )^t U(C_t) ]\)

The first-order conditions give us:

\(u'(x_t)=(1+\delta)(1+r_t)u'(x_{t+1})\)


\section{Linear transformations}
In the previous section we introduced real vectors and their most important properties. In this section we explore a special set of operations that can act on vectors, namely \emph{linear transformations}.

Linear transformations are a special set of transformations that are relatively easy to analyze. Their use is extremely widespread all throughout different fields of mathematics and its application, e.g. (just to name a few): in computer graphics, machine learning, quantum physics and engineering.

\subsection{Definition}
As mentioned in \autoref{chapter:intro}, a ``transformation`` is simply another name for a function. Thus in our context, linear transformations are some functions that act on vectors: a linear transformation $T$ takes a vector as an input, and outputs another vector, possibly of a different dimension, i.e.
\begin{equation}
	T:\Rs{n}\to\Rs{m}.
	\label{eq:linear transformation signature}
\end{equation}

What makes linear transformations more ``special'' than other functions is their property of \emph{linearity}, which entails the following two properties:
\begin{listitemize}
	\item[Scalability] for any scalar $\alpha$ and vector $\vec{v}$,
		\[
			T\left( \alpha\vec{v} \right) = \alpha T\left( \vec{v} \right).
		\]
	\item[Additivity] for any two vectors $\vec{u},\vec{v}$
		\[
			T\left( \vec{u}+\vec{v} \right) = T\left( \vec{u} \right) + T\left( \vec{v} \right).
		\]
\end{listitemize}

\begin{example}{A linear transformation}{linear transformation}
	\textbf{Claim}: the following $\Rs{3}\to\mathbb{R}$ transformation is linear:
	\[
		T\left(\colvec{x;y;z}\right) = 2x+3y-z.
	\]

	\textbf{Proof}: We can show this using the properties of linear transformations.
	\begin{listitemize}
	\item[Scalability] given a scalar $\alpha\in\mathbb{R}$,
		\[
			T\left(\colvec{\alpha x;\alpha y; \alpha z}\right)
			= 2(\alpha x)+3(\alpha y)-(\alpha z) = \alpha\left( 2x+3y-z \right)
			= \alpha T\left(\colvec{x;y;z}\right).
		\]
	\item[Additivity] given two vectors $\vec{u}=\colvec{u_{x};u_{y};u_{z}}$ and $\vec{v}=\colvec{v_{x};v_{y};v_{z}}$,
		\begin{align*}
			T\left( \colvec{u_{x};u_{y};u_{z}} + \colvec{v_{x};v_{y};v_{z}} \right) &= T\left( \colvec{u_{x}+v_{x};u_{y}+v_{y};u_{z}+v_{z}} \right)\\
			&= 2\left( u_{x}+v_{x} \right) +3\left( u_{y}+v_{y} \right) - \left( u_{z}+v_{z} \right)\\
			&= T\left( \colvec{u_{x};u_{y};u_{z}} \right) + T\left( \colvec{v_{x};v_{y};v_{z}} \right).
		\end{align*}
	\end{listitemize}
\end{example}

\begin{example}{A non-linear transformation}{non-linear transformation}
	\textbf{Claim}: the following $\Rs{3}\to\mathbb{R}$ transformation is \textbf{not} a linear transformation:
	\[
		T\left(\colvec{x;y;z}\right) = 2x^{2}+3y-z.
	\]

	\textbf{Proof}: this time we only need to show a single case where linearity breaks - let's choose \textit{scalability}. Given the vector $\vec{v}=\colvec{v_{x};v_{y};v_{z}}$, on one hand
	\[
		T\left( \alpha\colvec{u_{x};u_{y};u_{z}} \right) = 2\left(\alpha u_{x}\right)^{2}+3\alpha u_{y}-\alpha u_{z} = 2\alpha^{2}u_{x}^{2}+3\alpha u_{y}-\alpha u_{z}.
	\]
	On the other hand
	\[
		\alpha T\left( \colvec{u_{x};u_{y};u_{z}} \right) = \alpha\left( 2u_{x}^{2}+3u_{y}-u_{z} \right) = 2\alpha u_{x}^{2}+3\alpha u_{y}-\alpha u_{z}.
	\]
	For any $a\notin\left\{ 0,1 \right\}$ we get that $T\left( \alpha\vec{v} \right)\neq\alpha T\left( \vec{v} \right)$. Therefore, $T$ is not linear.
\end{example}

\subsection{Developing intuition}
Before moving on to explore the algebraic properties of linear transformations, we first shift our focus to gain some intuition about them. Much like in the last section, we do this using graphical representations of linear transformations in $\Rs{2}$ and $\Rs{3}$. We start with a single vector under transformation: let $\vec{u}=\colvec{2;4}$ and $T:\Rs{2}\to\Rs{2}$ defined by
\begin{equation}
	T\left( \colvec{x;y} \right) = \colvec{2x;-y}.
\end{equation}
(to the reader: verify that this transformation is indeed linear)

Applying $T$ to $\vec{u}$ yields the vector $\vec{v}=\colvec{4;-4}$ (see \autoref{fig:single vector LT}), i.e. it scales the $x$-component of $\vec{u}$ by $2$ and flips over its $y$-component.

\begin{figure}
	\centering
	\begin{subfigure}[t]{0.45\textwidth}
		\begin{tikzpicture}[]
			\begin{axis}[
				vector plane,
				width=7cm, height=7cm,
				xmin=-7, xmax=7,
				ymin=-7, ymax=7,
				xtick={-6,-4,...,6},
				ytick={-6,-4,...,6},
			]
			\tikzstyle{every node}=[font=\Large]
			\draw[vector, xred] (0,0) -- (2,4) node[pos=1.15] (u) {$\vec{u}$};
			\draw[vector, xblue] (0,0) -- (4,-4) node[pos=1.1] (v) {$\vec{v}$};
			\draw[vector, dashed, xpurple] (u) to [out=-20, in=70] node[pos=0.35, xshift=9pt] {$T$} (v);
			\end{axis}
		\end{tikzpicture}
		\caption{The vector $\vec{u}=\colvec{2;4}$ is transformed by $T$ yielding the vector $\vec{v}=\colvec{4;-4}$.}
		\label{fig:single vector LT}
	\end{subfigure}
	\hfill
	\begin{subfigure}[t]{0.45\textwidth}
		\centering
		\begin{tikzpicture}[]
			\begin{axis}[
				vector plane,
				width=7cm, height=7cm,
				xmin=-7, xmax=7,
				ymin=-7, ymax=7,
				xtick={-6,-4,...,6},
				ytick={-6,-4,...,6},
			]
			\tikzstyle{every node}=[font=\Large]
			\draw[vector, xred] (0,0) -- (-2,2) node[pos=1.3] (a) {$\vec{a}$};
			\draw[vector, xblue] (0,0) -- (1,-6) node[pos=0.95, right] (b) {$\vec{b}$};
			\draw[vector, xred, dashed] (0,0) -- (-4,-4) node[pos=1.3] (Ta) {$T\left(\vec{a}\right)$};
			\draw[vector, xblue, dashed] (0,0) -- (2,6) node[pos=0.95, right] (Tb) {$T\left(\vec{b}\right)$};
			\end{axis}
		\end{tikzpicture}
		\caption{The vectors $\vec{a}=\colvec{-2;2}$ and $\vec{b}$ are transformed by the same $T$.}
		\label{fig:two vectors LT}
	\end{subfigure}
\end{figure}

If we take other vectors, e.g. $\vec{a}=\colvec{-2;-2}$ and $\vec{b}=\colvec{1;-6}$ we see that $T$ transforms them in the exact same manner: it scales their $x$-components by $2$ and flipps over their $y$-components (\autoref{fig:two vectors LT}). This is a fundamental aspect of linear transformations: they always transform all vectors in the exact same manner. We can use this fact to help visualize transformations, by looking at how they transform the entire space. For example, we can draw all grid lines and observe how they are transformed.

In \autoref{fig:linear transformation grid} a schematic of $\Rs{2}$ is shown before and after the application of a linear transformation $T$, by placing a transformed grid (blue) ontop of an untouched grid (gray). In this view, one can see how each point in space is transformed: assuming for example that each two adjacent grid points are 1 unit apart, the gray point at $(-2,2)$ is transformed to where the blue point is, i.e. $(-1,1)$ when measured using the original axes.

\colorlet{xbefore}{black!40}
\colorlet{xafter}{xblue}

\begin{figure}
	\centering
	\begin{tikzpicture}
		\begin{axis}[
			vector plane,
			width=8cm, height=8cm,
			anchor=center,
			at={(current page.center)},
			xtick={-4,-2,...,4},
			ytick={-4,-2,...,4},
			xticklabels={,,},
			yticklabels={,,},
			grid style={line width=.1pt, draw=xbefore!35},
			major grid style={line width=.2pt, draw=xbefore!50},
			every tick/.style={xbefore},
			axis line style={xbefore, very thick},
		]
		\addplot[mark=*, mark size=2.5pt] coordinates {(-4,4)};
		% \fill[gray!50] (2,-2) rectangle (4,-4);
		\coordinate (p) at (-2,2);
		\coordinate (xlbl) at (5.5,2.05);
		\coordinate (ylbl) at (2.6,4.95);
		% \addplot[mark=*, xblue] coordinates {(-2,2)};
		\end{axis}
        \pgftransformcm{1}{0.4}{0.5}{0.9}{\pgfpoint{0cm}{0cm}}
		\begin{axis}[
			vector plane,
			width=8cm, height=8cm,
			anchor=center,
			at={(current page.center)},
			xtick={-4,-2,...,4},
			ytick={-4,-2,...,4},
			xticklabels={,,},
			yticklabels={,,},
			xlabel={},
			ylabel={},
			grid style={line width=.1pt, draw=xafter!35},
			major grid style={line width=.2pt, draw=xafter!50},
			every tick/.style={xafter},
			axis line style={xafter, ultra thick},
		]
		% \fill[xblue!50] (2,-2) rectangle (4,-4);
		\end{axis}
        \pgftransformreset
		\fill[xblue] (p) circle (3pt);
		\large
		\node[xblue] at (xlbl) {$x$};
		\node[xblue] at (ylbl) {$y$};
	\end{tikzpicture}
	\caption{$\Rs{2}$ after application of a linear transformation (blue), placed ontop of $\Rs{2}$ before the transformation (gray). Note the black point at the top left at $(-2,2)$ transforming into the blue point at $(-1,1)$.}
	\label{fig:linear transformation grid}
\end{figure}

For comparison, \autoref{fig:nonlinear transformation grid} shows a non linear transformation applied to $\Rs{2}$.

\begin{figure}
	\centering
	\begin{tikzpicture}
		\large
		\draw[black!25, step=0.5] (-3,-3) grid (3,3);
		\draw[stealth-stealth, very thick] (-3,0) -- (3,0) node[pos=1.1] {$x$};
		\draw[stealth-stealth, very thick] (0,-3) -- (0,3) node[pos=1.1] {$y$};
		\fill[black!50] (-3,-3) rectangle (-2.5,-2.5);
		\fill[black!50] (3,3) rectangle (2.5,2.5);
		\begin{scope}
			\pgftransformnonlinear{\nltransA}
			\draw[xblue!25, step=0.5] (-3,-3) grid (3,3);
			\draw[xblue, stealth-stealth, very thick] (-3,0) -- (3,0) node[pos=1.1] {$x$};
			\draw[xblue, stealth-stealth, very thick] (0,-3) -- (0,3) node[pos=1.1] {$y$};
			\fill[xblue!50] (-3,-3) rectangle (-2.5,-2.5);
			\fill[xblue!50] (3,3) rectangle (2.5,2.5);
		\end{scope}
	\end{tikzpicture}
	\caption{A non linear transformation applied to $\Rs{2}$ for comparison.}
	\label{fig:nonlinear transformation grid}
\end{figure}

\autoref{fig:linear transformation grid} shows some important properties of linear transformation (cf. \autoref{fig:nonlinear transformation grid}):
\begin{enumerate}
	\item The origin stays at the same place after application of the transformation, i.e. $T\left(\vec{0}\right) = \vec{0}$.
	\item Parallel lines remain parallel after application of the transformation.
	\item All areas are scaled by the same amount.
\end{enumerate}

It is rather easy to prove the first two properties.
\begin{proof}{Two properties of linear transformations}{}
	\begin{enumerate}
		\item Let $T$ be a transformation that does not perserve the origin, i.e.
	\[
		T\left(\vec{0}\right) = \vec{v} \neq \vec{0}.
	\]
	We can scale $\vec{0}$ by a scalar $\alpha\neq0$, which would yield
	\[
		T\left(\alpha\vec{0}\right) = T\left(\vec{0}\right) = \vec{v}.
	\]
	However, for $T$ to be linear we expect (due to scalability)
	\[
		T\left(\alpha\vec{0}\right) = \alpha\vec{v},
	\]
	but since $\alpha\neq0$ and $\vec{v}\neq\vec{0}$ this does not happen. Therefore, $T$ can not be linear - and in turn linear transformations must preserve the origin.

	\item A line is defined using a point $\vec{a}$, and a direction $\hat{v}$ as the set of all the points $\left\{x=\vec{a}+s\hat{v}, s\in\mathbb{R}\right\}$:

	\begin{center}
		\begin{tikzpicture}
			\begin{axis}[
				vector plane,
				width=8cm, height=8cm,
				xticklabels={,,},
				yticklabels={,,},
			]
			\coordinate (a) at (axis cs:2,2);
			\coordinate (v) at (axis direction cs:1,-0.5);
			\draw[black!50, dashed] ($(a)-6*(v)$) -- ($(a)+3*(v)$);
			\draw[vector, xred] (0,0) -- (a) node[midway, above left] {$\vec{a}$};
			\draw[vector, xblue] (a) -- ($(a)+(v)$) node[midway, below, xshift=-5pt, yshift=1pt] {$\hat{v}$};
			\end{axis}
			\foreach \s in {-6,-3.5,-1,0,1,2}{
				\fill ($(a)+\s*(v)$) circle (0.05) node[above, anchor=west, yshift=2pt] {\tiny$s=\s$};
			}
		\end{tikzpicture}
	\end{center}
	Parallel lines have the same direction $\hat{s}$, i.e. $x_{1}=\vec{a}_{1}+s_{1}\hat{v}$ and $x_{2}=\vec{a}_{2}+s_{2}\hat{v}$ are parallel lines. Applying a linear transformation $T$ to these lines yields (using the two defining properties of linear transformations)
	\begin{align*}
		T\left(x_{1}\right) &= T\left(\vec{a}_{1}\right) + s_{1}T\left(\hat{v}\right),\\
		T\left(x_{2}\right) &= T\left(\vec{a}_{2}\right) + s_{2}T\left(\hat{v}\right).
	\end{align*}
	We can see that the two right-hand side equations represent two new lines with the same direction, i.e. $T\left(\hat{v}\right)$. Therefore parallel lines remain parallel under a linear transformation.
	\end{enumerate}
\end{proof}

We will prove the the third property (all areas are scaled by the same amount) later in the chapter.

All linear transformations in $\Rs{2}$ can be created by composing transformations from a set of linear transformation which we will refer to as the \emph{basic linear transformations}\footnote{not an official name.}. To visualize these basic transformations we apply them on a figure of a tapir\footnote{They are here, they are a trans tapir. Get used to it.}:

\begin{figure}[H]
	\centering
	\tapirTrans{1}{0}{0}{1}{12cm}
\end{figure}

\autoref{fig:basicLinearTrans} shows the basic linear transformations applied to our happy tapir.

\begin{figure}[H]
	\centering
	\begin{subfigure}[c]{0.29\textwidth}
		\centering
		\tapirTrans{1}{0}{0}{1}{5cm}
		\caption{Identity - no change.}
		\label{fig:}
	\end{subfigure}
	\begin{subfigure}[c]{0.37\textwidth}
		\centering
		\tapirTrans{1.4}{0}{0}{1}{5cm}
		\caption{Scaling in the $x$-axis.}
		\label{fig:}
	\end{subfigure}
	\begin{subfigure}[c]{0.29\textwidth}
		\centering
		\tapirTrans{1}{0}{0}{1.3}{5cm}
		\caption{Scaling in the $y$-axis.}
		\label{fig:}
	\end{subfigure}
	\hfill
	\begin{subfigure}[c]{0.3\textwidth}
		\centering
		\tapirTrans{1}{0}{0.25}{1}{5cm}
		\caption{Skew in the $x$-axis.}
		\label{fig:}
	\end{subfigure}
	\begin{subfigure}[c]{0.3\textwidth}
		\centering
		\tapirTrans{1}{0.5}{0}{1}{5cm}
		\caption{Skew in the $y$-axis.}
		\label{fig:}
	\end{subfigure}
	\begin{subfigure}[c]{0.35\textwidth}
		\centering
		\tapirTrans{0.707}{-0.707}{0.707}{0.707}{5cm}
		\caption{Rotation around origin.}
		\label{fig:}
	\end{subfigure}
	\hfill
	\begin{subfigure}[c]{0.3\textwidth}
		\centering
		\tapirTrans{-1}{0}{0}{1}{5cm}
		\caption{Reflection about line.}
		\label{fig:}
	\end{subfigure}
	\begin{subfigure}[c]{0.3\textwidth}
		\centering
		\tapirTrans{-1}{0}{0}{-1}{5cm}
		\caption{Reflection through origin.}
		\label{fig:}
	\end{subfigure}
	\caption{The basic linear transformations, exemplified using a very happy tapir.}
	\label{fig:basicLinearTrans}
\end{figure}

\begin{example}{Composing basic linear transformations}{ltcomposing}
	Given the following two linear transformations:
	\begin{enumerate}
		\item Scale by $1.5$ in the $x$-direction,
		\item Rotate by $\frac{\pi}{4}$ anti-clockwise around the origin,
	\end{enumerate}
	two composite linear transformations can be created: first scale then rotate, and first rotate then scale. As can be seen in the figure bellow, changing the order of composition results in a different linear transformations all together:

	\begin{center}
		\begin{tikzpicture}
			\tapirTransComp{1}{0}{0}{1}{0}{0}{I}

			\tapirTransComp{0.707}{0.707}{-0.707}{0.707}{1cm}{5cm}{A}
			\tapirTransComp{0.707}{0.707}{-1.414}{1.414}{7cm}{5cm}{B}
			\draw[-stealth, thick] (0,1cm) -- ++(0.5cm,2cm) node[midway, above, rotate=75, text width=1.3cm] {first rotate};
			\draw[-stealth, thick] (2.2cm,4.5cm) -- ++(2.5cm,0cm) node[midway, above, text width=1.5cm] {then scale in $x$};

			\tapirTransComp{2}{0}{0}{1}{1cm}{-5cm}{C}
			\tapirTransComp{0.707}{1.414}{-0.707}{1.414}{8cm}{-5cm}{D}
			\draw[-stealth, thick] (0,-1.5cm) -- ++(0.5cm,-2.5cm) node[midway, above, rotate=-80, text width=1.5cm] {first scale in $x$};
			\draw[-stealth, thick] (3cm,-5.5cm) -- ++(3cm,0cm) node[midway, above, text width=2cm] {then rotate};
		\end{tikzpicture}
	\end{center}

	This is not a suprising result: in \autoref{chapter:intro} we learned that function composition is not a commutative operation.
\end{example}

Some of the basic linear transformations can be created as compositions of other basic linear transformations. For example, the composition of reflection across the $y$-axis followed by reflection across the $x$-axis results in a $\ang{180}$ rotation around the origin (see \autoref{fig:two_reflections_rotation}).

\begin{figure}
	\centering
	\begin{tikzpicture}
		\begin{axis}[
			vector plane,
			width=10cm, height=10cm,
			xticklabels={,},
			yticklabels={,},
		]
		\node[anchor=east] (A) at (-0.5,2) {\includesvg[scale=0.75]{figures/linear_algebra/tapir_transform2}};
		\node[anchor=east, xscale=-1] (B) at (0.5,2) {\includesvg[scale=0.75]{figures/linear_algebra/tapir_transform2}};
		\node[anchor=east, xscale=-1, yscale=-1] (C) at (0.5,-2) {\includesvg[scale=0.75]{figures/linear_algebra/tapir_transform2}};

		\draw[vector, xred] (A) -- (B);
		\draw[vector, xred] (B) -- (C);
		\draw[decoration={
            text along path,
			text color=xblue,
            text={Rotation},
            text align={center},
            raise=2mm}, decorate] (A.south) to [out=-90, in=180, looseness=0.75] (C.east);
		\draw[vector, xblue] (A.south) to [out=-90, in=180, looseness=0.75] (C.east);
		\end{axis}
	\end{tikzpicture}
	\caption{Reflection across the $y$-axis (red arrow) followed by a reflection across the $x$-axis (red arrow) results in a rotation by $\ang{180}$ around the origin (blue arrow).}
	\label{fig:two_reflections_rotation}
\end{figure}

\subsection{3D Linear transformations}
The basic linear transformations in $\Rs{3}$ are very much similar to those in $\Rs{2}$ with some small differences worth mentioning. For a start, scaling and skewing can be done in three different directions instead of just two directions (namely, $x,y$ and $z$ instead of just $x$ and $y$). In addition, there are infinetly many axes of rotation: in $\Rs{2}$ there is just a single axis (actually a point) of rotation - the origin. In $\Rs{3}$ any line that goes through the origin can be an axis of rotation (see \autoref{fig:rotation_around_axis}). Lastly, there are three types of reflections: about the origin (a point), across a line going through the origin, and across a plane (see fig?).

\begin{figure}
	\centering
	\begin{subfigure}[c]{0.47\textwidth}
		\begin{center}
			\def\angThe{75}
			\def\angPhi{65}
			\tdplotsetmaincoords{\angThe}{\angPhi}
			\begin{tikzpicture}[tdplot_main_coords]
				\draw[stealth-stealth, very thick] (-3,0,0) -- (3,0,0) node[pos=1.05] {$x$};
				\draw[stealth-stealth, very thick] (0,-3,0) -- (0,3,0) node[pos=1.05] {$y$};
				\draw[stealth-stealth, very thick] (0,0,-3) -- (0,0,3) node[pos=1.05] {$z$};
				\draw[ultra thick, xpurple, dotted] (-0.6675,-1.335,-2.005) -- (0,0,0);
				\draw[ultra thick, xpurple] (0,0,0) -- (0.267,0.534,0.802);
				\begin{scope}[canvas is plane={O(0.267,0.535,0.802)x(1.231,0.386,0.549)y(0.267,-0.298,1.356)}]
					\fill[xpurple!20, opacity=0.2] (-1,-1) rectangle (1,1);
					\draw[step=0.5, xdarkpurple!30] (-1,-1) grid (1,1);
					\draw[vector, xpurple!80, dotted] (0.5,0) arc (0:177:0.5);
					\draw[vector, xpurple!80, dotted] (-0.5,0) arc (180:362:0.5);
				\end{scope}
				\draw[vector, xpurple] (0.267,0.534,0.802) -- (0.534,1.069,1.604) node[pos=1.2] {$\hat{r}$};
			\end{tikzpicture}
		\end{center}
	\end{subfigure}%
	\begin{subfigure}[c]{0.47\textwidth}
		\begin{center}
			\def\angThe{75}
			\def\angPhi{225}
			\tdplotsetmaincoords{\angThe}{\angPhi}
			\begin{tikzpicture}[tdplot_main_coords]
				\draw[stealth-stealth, very thick] (-3,0,0) -- (3,0,0) node[pos=1.05] {$x$};
				\draw[stealth-stealth, very thick] (0,-3,0) -- (0,3,0) node[pos=1.05] {$y$};
				\draw[stealth-stealth, very thick] (0,0,-3) -- (0,0,3) node[pos=1.05] {$z$};
				\draw[ultra thick, xpurple, dotted] (-0.6675,-1.335,-2.005) -- (0,0,0);
				\draw[ultra thick, xpurple] (0,0,0) -- (0.267,0.534,0.802);
				\begin{scope}[canvas is plane={O(0.267,0.535,0.802)x(1.231,0.386,0.549)y(0.267,-0.298,1.356)}]
					\fill[xpurple!20, opacity=0.2] (-1,-1) rectangle (1,1);
					\draw[step=0.5, xdarkpurple!30] (-1,-1) grid (1,1);
					\draw[vector, xpurple!80, dotted] (0.5,0) arc (0:177:0.5);
					\draw[vector, xpurple!80, dotted] (-0.5,0) arc (180:362:0.5);
				\end{scope}
				\draw[vector, xpurple] (0.267,0.534,0.802) -- (0.534,1.069,1.604) node[pos=1.15] {$\hat{r}$};
			\end{tikzpicture}
		\end{center}
	\end{subfigure}
	\caption{Rotation around a direction $\hat{r}$ from two different viewpoints, showing an example of a plane orthogonal to $\hat{r}$ on which the rotation happens.}

	\label{fig:rotation_around_axis}
\end{figure}


\section{Forces}
\subsection{Force and Potential Energy in One (Spacial) Dimension}
Consider a point mass $m$ moving on a straight line with position given by $x(t)$.
We assume that the force $F=F(x)$ depends entirely on position, not velocity and time.
\begin{definition}
    The potential energy $V(x)$ is any function that satisfies $F(x)=-\mathrm dV/dx$.
\end{definition}
Equivalently,
$$V(x)=V(0)+\int_0^xF(x)\,\mathrm dx$$
where $V(0)$ can be taken arbitrarily.
The equation of motion is simply $m\ddot{x}=-\mathrm dV/dx$ by Newton's Second Law.
\begin{definition}
    The KInetic energy $T$ is defined by $T=m|\dot{x}|^2/2$
\end{definition}
\begin{theorem}
    Under the assumptions and definitions above, we have $\mathrm d(T+V)/\mathrm dt=0$.
\end{theorem}
\begin{proof}
    $$\frac{\mathrm d(T+V)}{\mathrm dt}=\frac{2m\dot{x}\ddot{x}}{2}+\frac{\mathrm dV}{\mathrm dx}\frac{\mathrm dx}{\mathrm dt}=\dot{x}(m\ddot{x}+\frac{\mathrm dV}{\mathrm dx})=0$$
    By Newton's Second Law.
\end{proof}
Note that if we lost that restriction on the time and velocity independence of the force, we lose the conservation of energy in general.
\begin{example}
    Consider a harmonic oscillator, so $F(x)=-kx$ where $k$ is a positive constant.
    So $V(x)=kx^2/2$ by choosing the arbitrary constant as $0$.
    We want to calculate all the stuff to verify the conservation of energy.
    \footnote{Hey, you literally just proved it.}
    We can solve the motion by solving $m\ddot{x}=-kx$ which solves to $x=A\sin(\sqrt{k/m}t)+B\cos(\sqrt{k/m}t)$.
    And plugging in gives $\mathrm dE/\mathrm dt=0$.
\end{example}
As the first instance of Newton's Second Law, the conservation of energy is a useful rule to determine a one dimensional motion.
Using conservation of energy, we have
$$\dot{x}=\pm\sqrt{\frac{2}{m}(E-V(x))}$$
which is a first order ODE.
So
$$\int_{x_0}^x\frac{\mathrm du}{\sqrt{2(E-V(u))/m}}=t-t_0$$
where $x(t_0)=x_0$.
In principle we can solve it to obtain the motion.\\
We can also have some qualitative insight from conservation of energy.
Consider $V(x)=\lambda(x^3-3\beta^2x)$ where $\lambda,\beta>0$ are constants.
We can sketch the potential energy to find that $V$ has a local maximum at $-\beta$, which value happens again at $x=2\beta$.
And it has a local minimum at $\beta$, where it again obtain the same value at $-2\beta$.
So we can find certain properties of the motion from the graph if the motion start at rest, then we must have $V(x)\le V(x_0)$\\
Case 1: $x_0<-\beta$, it will moves to left so as to reduce the potential and gain speed.\\
Case 2: $-\beta<x_0<2\beta$, then the particle will be restricted in the region $-\beta<x_0<2\beta$ and will oscillate.\\
Case 3: $x_0>2\beta$, then it will move to the right.\\
The case becomes special if we turn to the stationary (or equilibrium) points.
Obviously $x_0=-\beta$ is an unstable fixed point and $x_0=\beta$ is a stable fixed points.
So at $x_0=2\beta$, it will end its notion at the fixed point $x=-\beta$.
In this case, we can analyse the behaviour by writing down the integral.
This can show that the time to reach $x=-\beta$ is infinite when we approach to $2\beta$.
\subsection{Equilibriums}
The points $x=\pm\beta$ in this case are called equilibrium points, at which the particle can always stay at rest.
The condition for this to happen is $V^\prime(x_0)=0$.
We are going to analyze the motion near the equilibrium at $x_0$ by expanding its Taylor series
$$V(x)\approx V(x_0)+(x-x_0)V^\prime(x_0)+\frac{(x-x_0)^2}{2}V^{\prime\prime}(x_0)=V(x_0)+\frac{(x-x_0)^2}{2}V^{\prime\prime}(x_0)$$
We assume for a moment that $V^{\prime\prime}(x_0)$ does not vanish.
\footnote{If it does vanish, we will have to look at higher order terms.}
So we have $m\ddot{x}=-(x-x_0)V^{\prime\prime}(x_0)$.\\
If $V^{\prime\prime}(x_0)>0$, it's a minimum of $V$ which produces the equation of a harmonic oscillator with period $\sqrt{V^{\prime\prime}(x_0)/m}$.
In this case, we say it is a stable equilibrium.\\
If $V^{\prime\prime}(x_0)<0$, it's a maximum of $V$ which produces the equation of an exponentially growing solution.
Hence it is an unstable equilibrium with growth rate $\sqrt{-V^{\prime\prime}(x_0)/m}$.
\begin{example}
    We look back to a pendulum with mass $m$, length $l$ and angle $\theta$.
    If we think of Newton's Second Law, one can obtain
    $$F=ml\ddot{\theta}=-mg\sin\theta=-\frac{\mathrm d}{\mathrm d\theta}(-mg\cos\theta)$$
    So we have $E=T+V=ml^2\dot{\theta}^2/2-mgl\cos\theta$.
    One can check that $\dot{E}=0$.
    Also the potential $V(\theta)=-mg\cos\theta$ has stable equilibrium at $\theta=2\pi k,k\in\mathbb Z$ and unstable at $\theta=\pi+2\pi k,k\in\mathbb Z$.
    So if the initial value of $\theta$ is in $(-\pi,\pi)$ (or $|V|<mgl$), the pendulum will oscillate.
    If $|V|>mgl$, then it will go round and round.\\
    Now we want to analyze the period of oscillations.
    Suppose the original angle is at $\theta_0\in (0,\pi)$, then the oscillation is going to be $\theta_0\to 0\to-\theta_0\to0\to\theta_0$, so the period is $4$ times the time taken for $\theta_0$ to $0$, hence
    $$P=4\int_0^{\theta_0}\frac{\mathrm d\theta}{\sqrt{2gl(\cos\theta-\cos\theta_0)/l^2}}=4\sqrt{\frac{l}{g}}\int_0^{\theta_0}\frac{\mathrm d\theta}{\sqrt{2\cos\theta-2\cos\theta_0}}=\sqrt{\frac{l}{g}}F(\theta_0)$$
    For small $\theta_0$ we have
    $$F(\theta_0)\approx 4\int_0^{\theta_0}\frac{\mathrm d\theta}{\sqrt{\theta_0^2-\theta^2}}=2\pi$$
    Hence $P\approx 2\pi\sqrt{l/g}$.
\end{example}
\subsection{Force and Potential in Three Dimensions}
Consider a particle $\underline{r}$ in motion in three dimensional space.
Then $m\underline{\ddot{r}}$ and $T=m|\underline{\dot{r}}|^2/2$.
And the rate of change of $T$ is then
$$\frac{\mathrm dT}{\mathrm dt}=m\underline{\dot{r}}\cdot\underline{\ddot{r}}=\underline{\dot{r}}\cdot\underline{F}$$
Suppose the particle tranverse a path $C$ from $t_0$ to $t_1$, then
\begin{definition}
    The work done is
    $$\int_{t_0}^{t_1}\underline{F}\cdot\underline{\dot{r}}\,\mathrm dt=\int_C\underline{F}\cdot\mathrm d\underline{r}$$
\end{definition}
We can also write that the total work equals
$$\int_C\underline{F}\cdot\mathrm d\underline{r}=\int_C F_x\,\mathrm dx+F_y\,\mathrm dy+F_z\,\mathrm dz$$
Suppose that the force is a function of the position $\underline{F}(\underline{r})$ (also called a force field).
\begin{definition}
    A force field $\underline{F}(\underline{r})$ is called conservative if $\underline{F}=\nabla V$ for some $V:\mathbb R^3\to\mathbb R$.
\end{definition}
If a force field $\underline{F}(\underline{r})$ is conservative, we say $V$ is the potential function.
Also in this case $E=T+V(\underline{r})$ conserved.
Indeed,
$$\frac{\mathrm dE}{\mathrm dt}=\frac{\mathrm dT}{\mathrm dt}+\frac{\mathrm dV}{\mathrm dt}=m\underline{\dot{r}}\cdot\underline{\ddot{r}}+\nabla V\cdot\underline{\dot{r}}=\underline{\dot{r}}\cdot(m\underline{\ddot{r}}-\underline{F})=0$$
The total work done by a conservative force $\underline{F}$ is
$$\int_C\underline{F}\cdot\mathrm d\underline{r}=\int_C-\nabla V\cdot\mathrm d\underline{r}=V(\underline{r}(t_0))-V(\underline{r}(t_1))$$
So the work done is independent of the path taken.\\
In particular, if the curve is closed, no work is done.\\
$\underline{F}$ is conservative if $\nabla\times\underline{F}=\underline{0}$ (given that the domain is simply connected).
\subsection{Angular Momentum}
\begin{definition}
    The angular momentum for a particle with mass $m$ and velocity $\underline{\dot{r}}$ is defined as
    $$\underline{L}=\underline{r}\times\underline{p}=m\underline{r}\times\underline{\dot{r}}$$
    And
    $$\underline{G}=\frac{\mathrm d\underline{L}}{\mathrm dt}=m\underline{\dot{r}}\times\underline{\dot{r}}+m\underline{r}\times\underline{\ddot{r}}=\underline{r}\times\underline{F}$$
    is defined as the torque, or moment of force.
\end{definition}
Note that $\underline{L},\underline{G}$ both depend on the choice of origin, so we must specify them when talking about angular stuff.
\begin{remark}
    If $\underline{r}\times\underline{F}=\underline{0}$ then $\underline{G}=0$, thus $\underline{L}$ is constant.
    In this case, we say the angular momentum is conserved.
\end{remark}
\subsection{Central Forces}
A special type of conservative force occurs when the potential $V$ depends entirely on $|\underline{r}|$, so $V(\underline{r})=V(|\underline{r}|)=V(r)$,
\footnote{It's just a tiny abuse of notation. vErY tInY.}
$$\underline{F}(\underline{r})=-\nabla V(|\underline{r}|)=-\frac{\mathrm dV}{\mathrm dr}\underline{\hat{r}},\underline{\hat{r}}=\frac{\underline{r}}{|\underline{r}|}$$
So $\underline{F}$ and $\underline{r}$ are parallel, therefore $\mathrm d\underline{L}/\mathrm dt=\underline{G}=\underline{F}\times\underline{r}=0$.
\subsection{Gravity}
Recall that Newton's Gravitational Law states
$$V=-\frac{GMm}{|\underline{r}|},\underline{F}=-\nabla V=-\frac{GMm}{|\underline{r}|^2}\underline{\hat{r}}=-\frac{GMm}{|\underline{r}|^3}\underline{r}$$
Note that the $m$ here can be ignored (in the way shown below) if we are only interested in the motion due to Newton's Second law:
\begin{definition}
    The gravitational potential is defined by $\Phi_g(\underline{r})=V/M=-GM/|\underline{r}|$, and the gravitational field by $\underline{g}=-\nabla\Phi_g(\underline{r})=-GM\underline{\hat{r}}/r^2$.
\end{definition}
The gravitational field and its potential, as functions, are dependent of $M$ alone.
We also have $m\Phi_g=V,m\underline{g}=\underline{F}$.
For a set of more than one masses, we can simple generalize by adding the corresponding fields and potential together by the superposition principle.
Hence for continuous bodies, we can replace the sum by an integral.
In particular, if the body is spherical with radius $R$ and we have $|\underline{r}|>R$, then we do have $\Phi_g(\underline{r})=-GM/|\underline{r}|$, thus spherical bodies do behave like a point when measuring from above its surface.
\begin{note}
    The mass $m$ in Newton's Second Law $m\underline{\ddot{r}}=\underline{F}$ is called the inertial mass, whilist the mass in Newton's Gravitational Law $\underline{F}=-GMm\underline{r}/|\underline{r}|^2$ is the gravitational mass.
    These two definitions of mass are different in relativity but are very closely related (about a difference of $10^{-12}$).
    The precise difference will be discussed in General Relativity.
\end{note}
There are a few results about the effect of gravity.
\begin{example}[Potential Energy near the Surface]
    For a mass $m$ at height $z$ above a spherical mass $M$ with radius $R$, if $z<<R$, then the potential energy is given by
    \begin{align*}
        V(R+z)&=-\frac{GMm}{R+z}\\
        &=-\frac{GMm}{R}+\frac{GMm}{R^2}z+o(R^{-2})\\
        &\approx -\frac{GMm}{R}+mgz\\
        &=\text{const}+mgz
    \end{align*}
    For earth, we have the approximation $g\approx 9.8{\rm ms^{-2}}$
\end{example}
\begin{example}[Escape Velocity]
    We want to find the critical velocity $\underline{v}$, perpendicular to $\underline{r}$, to leave a planet.
    Due to the conservation of energy $E=T+V=m|\underline{v}|^2/2-GMm/R$, the particle can escape (i.e. $\underline{v}$ is nonnegative at infinity) iff the initial energy has $E_0\ge 0$, which happens iff
    $$m|\underline{v}|^2/2\ge GMm/R\implies |\underline{v}|\ge \sqrt{\frac{2GM}{R}}=v_{\rm esc}$$
\end{example}
\subsection{Electromagnetic Forces}
We have seen previously that for a point charge $q$, the force has the expression $\underline{F}=q(\underline{E}+\underline{\dot{r}}\times\underline{B})$
In general $\underline{E},\underline{B}$ are functions of $\underline{r}$ and $\underline{t}$.
These are known as the Lorentz Force Law.
For convenience or something, we are going to restrict ourselves to time-independent fields.
So we want the electric field to be conservative, i.e. $\underline{E}=-\nabla\Phi_e$ where $\Phi_e$ is called the electrostatic potential.
\begin{claim}
    In a time-independent electromagnetic field, the energy
    $$E=T+V=\frac{m|\underline{\dot{r}}|^2}{2}+q\Phi_e(\underline{r})$$
    is conserved.
\end{claim}
\begin{proof}
    \begin{align*}
        \frac{\mathrm dE}{\mathrm dt}&=\frac{\mathrm d}{\mathrm dt}\left(\frac{m|\underline{\dot{r}}|^2}{2}+q\Phi_e(\underline{r})\right)\\
        &=m\underline{\dot{r}}\cdot\underline{\ddot{r}}-q\underline{\dot{r}}\cdot\underline{E}\\
        &=\underline{\dot{r}}(m\underline{\ddot{r}}-q\underline{E})\\
        &=0
    \end{align*}
    So $E$ is constant.
\end{proof}
\begin{law}
    Now consider a point charge $Q$ located at the origin.
    It generates an electrostatic field
    $$\Phi_e(\underline{r})=\frac{Q}{4\pi\epsilon_0|\underline{r}|},\underline{E}=\frac{Q}{4\pi\epsilon_0|\underline{r}|^2}\underline{\hat{r}}$$
    where $\epsilon_0$ is called the electric constant.
\end{law}
So the force exerted on our point charge $q$ is
$$\underline{F}=q\underline{E}=\frac{Qq}{4\pi\epsilon_0|\underline{r}|^2}\underline{\hat{r}}$$
which is called the Coulomb force.
One observe the similarity of this with the gravitational law (inverse-square law).
Also, by considering the signs, we find that same signed charges repel, opposite charges attract.
\subsection{Friction}
The friction is a contact force, which occurs when two body touches each other (they may not be of the same form though).
It is a convenient description of complicated molecular-scale physics.
So friction is not a kind of fundamental forces (gravity, EM, strong force, weak force).\\
We first consider a special kind of friction that is \textit{dry friction}.
Solids stand on each other exerts an action and reaction pair of normal forces (normal to the surface of contact), which prevents an object from merging with the other.
There is also a tangential force, which is a tangent to the trajectory on the surface where the solid is moving.\\
Now imagine we place a block on a slope.
If it remains at rest, the tangential force is called the static force, which exists even without relative motion.
In this case we have
\begin{law}
    The static force $\underline{F}$ has $|\underline{F}|\le\mu_s|\underline{N}|$ where $\mu_s$ is a constant (depending on the materials) called the coefficient of static friction, and $\underline{N}$ is the normal force.
\end{law}
So the block can rest on the plane provided that $\alpha\le\tan^{-1}(\mu_s)$ where $\alpha$ is the inclination.\\
There is a kinetic frictional force as well, which depends on the kinetic motion of the object.
\begin{law}
    The kinetic frictional force has $\underline{F}=\mu_k|\underline{N}|$ where $\mu_k$ is also a constant depending on the materials.
\end{law}
Normally $\mu_s>\mu_k>0$.\\
The most complicated type of friction is the \textit{fluid drag}, which is the friction exerted by a solid moving in fluid medium.
The model of linear drag says that
$$\underline{F}=-k_1\underline{\underline{u}}$$
where $\underline{u}$ is the velocity along the direction of motion and $k$ is a constant.
This model is relevent if we are considering a small object moving through a viscous fluid.
\begin{law}[Stokes' Law]
    Consider a sphere with radius $R$ moving in a viscous fluid with viscosity of $\eta$, then we have $k_1=6\pi\eta R$.
\end{law}
Another drag regime is called the quadratic model, which is for large bodies moving in less viscous fluid.
$$\underline{F}=-k_2|\underline{u}|\underline{u}$$
Typically we have $\rho R^2C_0$ where $\rho$ is the density of the fluid, $R$ is the radius and $C_0$ is the drag coefficient.\\
In the case of a linear drag, the rate of work done is $\underline{F}\cdot\underline{u}=-k_1|\underline{u}|^2$ and for quadratic law $\underline{F}\cdot\underline{u}=-k_2|\underline{u}|^3$.
The fluid gains energy due to this energy loss by the solid, obviously.\\
Recall from differential equations that the damp oscillator $m\ddot{x}=-kx-\lambda\dot{x}$ where the last term is a drag.
We know how to solve this.
\begin{example}
    Projectiles moving under uniform gravity and experiencing linear drag force.
    The equation of motion is hence
    $$m\ddot{x}=m\underline{g}-k\underline{\dot{x}}$$
    Consider the particle start at the origin with some velocity $\underline{U}$, so the initial conditions are $\underline{x}(0)=\underline{0},\underline{\dot{x}}(0)=\underline{U}$.
    We can solve the equation in $\underline{\dot{x}}$ and substituting the initial condition gives
    $$\underline{\dot{x}}=\frac{m\underline{g}}{k}+(\underline{U}-m\underline{g}/k)e^{-kt/m}$$
    Integrate it again and plug in the other initial condition,
    $$\underline{x}=\frac{m\underline{g}}{k}t+\frac{m}{k}(\underline{U}-m\underline{g}/k)(1-e^{-kt/m})$$
    Set $\underline{x}=(x,y,z),\underline{U}=U(\cos\theta,0,\sin\theta),\underline{g}=(0,0,-g)$.
    Hence we have, by simply plugging things in, that $y$ direction is irrelevant at all, while
    $$\dot{x}=U\cos\theta e^{-kt/m},\dot{y}=0,\dot{z}=(U\sin\theta+mg/k)e^{-kt/m}-mg/k$$
    so the $x$-velocity will eventually go to $0$ and $z$ velocity to a terminal value $mg/k$ (roughly after $t=m/k$).\\
    As for displacement, we have
    $$x=mU\cos\theta/k(1-e^{-kt/m}),z=-mgt/k+m/k(U\sin\theta+mg/k)(1-e^{-kt/m})$$
    so $x$ is bounded but $z$ is eventually moving (linearly as $t$ large).\\
    Now we want to turn to analyze the range $R(U,\theta,m,k,g)$ of the projectile for it to reach its original position (assuming we project it upwards), so by dimensional analysis we get the dimensionless quantity to be $f(\theta,kU/mg)=f(\theta,(U/g)/(m/k))$.
    Note that $U/g$ is proportional to the time taken to reduce velocity such that it vanishes; and $m/k$ is the approximate time to achieve the terminal velocity.
    So weak friction means $kU/mg<<1$ and strong firction means $kU/mg>>1$.\\
    We have $R=U^2/gf(\theta,kU/mg)$.
    If $kU/mg<<1$, then $R\approx U^2/g(2\sin\theta\cos\theta)$.
    If $kU/mg>>1$, then $R\approx U^2/g(\cos\theta(mg/kU))$.
\end{example}