\section{Projection mapping}\label{sec:projection-mapping}

Projection mapping is a spatial augmented reality \cite{Bimber2005} technique in which video projectors are used to overlay virtual geometry on top of real objects or surfaces. This allows to create an immersive environment that together with 3D perception systems can be used to develop interactive interfaces that show contextual information for helping or teaching human operators performing complex tasks faster. The next sections describe the mathematical projector modeling and the associated calibration that is necessary for performing proper 3D rendering of the virtual world in order to achieve high accuracy projection.

\subsection{Projector modeling}

Over the years, several projection technologies were developed according to the requirements of color fidelity / saturation, image sharpness, brightness, contrast, refresh rate and price. Currently, the video projection market is split between reflective \gls{dlp} and transmissive \gls{lcd} projection technology, with a small percentage of projectors consisting of a hybrid between the two technologies (\gls{lcos}).

For video projection mapping purposes \cite{Raskar1998,Bimber2005,Tan2013,Fujimoto2014}, reflective projectors are better suited than the remaining technologies given their ability to create images with smaller gaps between the projected pixels (smoother images) and they also have higher contrast, better color accuracy / uniformity, much fewer dead pixels and the image quality does not degrade over time. The main stages of the image creation in a \gls{dlp} projector are show in \cref{fig:dlp-projector-diagram-dmd}. The first phase is the generation of light from either a lamp or a \gls{led} / laser array, which is later on condensed on a lens in order to pass through a moving color wheel to become one of the 3 primary additive colors (red, green, blue). The colored light then passes through a shaping lens and hits a \gls{dmd} which has an electronic controllable mirror for each projection pixel that either reflects the light into the projection lens or into a heat sink. Color shading is achieved by controlling how long and how often the micro mirrors in the \gls{dmd} are reflecting each light color into the lens (or into the heat sink).

The mathematical modeling of a \gls{dlp} projector can be seen as an inverse pinhole camera \cite{Hartley2003} (shown in \cref{fig:camera-intrinsics}), given the grid disposition of the mirrors in the \gls{dmd} and the very low distortion that modern \gls{dlp} projectors have. As such, scene rendering can be performed efficiently using the \gls{opengl} projection matrix (shown in \cref{eq:projection-matrix,eq:ndc-matrix,eq:perspective-matrix}), which incorporates the focal lengths (Fx, Fy), principal point (Cx, Cy) and axis skew (S) intrinsic camera parameters (in pixel units). The correction of lens distortion for \gls{dlp} projectors typically uses 3 coefficients for removing radial distortions and 2 coefficients for accounting for the tangential distortions.

\begin{figure}[H]
	\begin{floatrow}[2]
		\ffigbox[\FBwidth]
		{\includegraphics[height=.19\textheight]{dlp-projector-diagram-dmd}}
		{\caption[Single chip \glsentrytext{dlp} diagram]{Single chip \glsentrytext{dlp} diagram\protect\footnotemark}\label{fig:dlp-projector-diagram-dmd}}
		\ffigbox[\FBwidth]
		{\includegraphics[height=.19\textheight]{camera-intrinsics}}
		{\caption[Pinhole camera model]{Pinhole camera model\protect\footnotemark}\label{fig:camera-intrinsics}}
	\end{floatrow}
\end{figure}
\footnotetext[\the\numexpr\value{footnote}-1\relax]{\url{https://vimeo.com/blog/post/display-tech-home-projectors}}
\footnotetext[\value{footnote}]{\url{http://perso.ensta-paristech.fr/~filliat/Courses/2011_projets_C10-2/BRUNEAU_DUBRAY_MURGUET/monoSLAM_bruneau_dubray_murguet_en.html}}

{
	\scriptsize
	\begin{equation}\label{eq:projection-matrix}
	ProjectionMatrix = \glsentrytext{ndc}Matrix \times PerspectiveMatrix
	\end{equation}

	\begin{equation}\label{eq:ndc-matrix}
	NDCMatrix =
	\begin{bmatrix}
	\frac{2}{ImageWidth} & 0 & 0 & -1 \\
	0 & \frac{2}{ImageHeight} & 0 & -1 \\
	0 & 0 & \frac{-2}{ClipFar - ClipNear} & \frac{-(ClipFar + ClipNear)}{ClipFar - ClipNear} \\
	0 & 0 & 0 & 1
	\end{bmatrix}
	\end{equation}

	\begin{equation}\label{eq:perspective-matrix}
	PerspectiveMatrix =
	\begin{bmatrix}
	Fx & S & -Cx & 0 \\
	0 & Fy & -Cy & 0 \\
	0 & 0 & ClipNear + ClipFar & ClipNear \times ClipFar \\
	0 & 0 & -1 & 0
	\end{bmatrix}
	\end{equation}
}

\subsection{Projector calibration}

High accuracy projection mapping requires proper hardware / software calibration of the camera / projector and also appropriate positioning within the intended workspace in order to avoid occlusions caused by the objects 3D shape or the human operators. This calibration aims to compute the intrinsic parameters of the projector (that do not change when the projector is moved within the workspace) along with the extrinsic parameters that are needed to know where is the projector in the global reference frame in order to be able to do proper 3D rendering of the scene that will be projected.

The intrinsic parameters of a \gls{dlp} projector can be computed using image analysis of complementary gray code patterns (example in \cref{fig:dlp-calibration-pattern-wall}) projected into a chessboard. The calibration system proposed in \cite{Moreno2012} was used to retrieve the 5 intrinsic parameters (Fx, Fy, Cx, Cy, S) of the projector along with the 3D position and rotation of the projector in relation to the camera (that remains firmly attached to the projector support for fast recalibration of the extrinsic parameters). It was used 5 sets of 42 gray code image patterns captured with the chessboard in different positions and orientations in relation to the projector, that was pointing to the table workspace at a distance of 0.81 meters. After calibration, it was projected a validation pattern to evaluate the accuracy of the projection, and as can be seen in \cref{fig:dlp-projected-chessboard}, the white squares were projected into the chessboard with sub-millimeter accuracy.

\begin{figure}[H]
	\begin{floatrow}[2]
		\ffigbox[\FBwidth]
		{\includegraphics[height=.205\textheight]{dlp-calibration-pattern-wall}}
		{\caption{One of the \glsentrytext{dlp} projector calibration patterns}\label{fig:dlp-calibration-pattern-wall}}
		\ffigbox[\FBwidth]
		{\includegraphics[height=.205\textheight]{chessboard}}
		{\caption{\glsentrytext{dlp} projector validation pattern}\label{fig:dlp-projected-chessboard}}
	\end{floatrow}
\end{figure}

After having the intrinsic parameters of the camera and projector along with the relative position of the projector in relation to the camera, computing the global position of the projector in relation to the chessboard reference frame can be done by multiplying the $4 \times 4$ homogeneous matrix that gives the transformation from the chessboard origin (shown in \cref{fig:chess-board-detection}) to the camera frame, with the $4 \times 4$ homogeneous matrix that gives the transformation from the camera frame to the projector frame.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.47\linewidth]{chess-board-detection}
	\caption{Camera pose estimation in relation to the chessboard origin (using the Kinect 2 RGB camera)}
	\label{fig:chess-board-detection}
\end{figure}

\subsection{Scene rendering}

For achieving accurate projection mapping, the Gazebo simulator\footnote{\url{http://gazebosim.org}} camera implementation was improved to allow the setting of a custom projection matrix in order to perform 3D rendering with a camera model that takes into account the projector intrinsic parameters. Moreover, it was added the possibility to dynamically change image, video and text during runtime for allowing the display of the relevant information for each assembly step.
For efficient 3D scene rendering, the Gazebo simulator relies on the cross platform open source Ogre3D graphics engine\footnote{\url{http://www.ogre3d.org}}, that in turn uses the \gls{opengl} \gls{gpu} \gls{api} to take advantage of the massively parallel graphics cards currently available to generate raster images for the \gls{dlp} projector (example of a rendered scene for the last assembly step in \cref{fig:scene-rendering}).

For user interface, the Gazebo simulator has a Qt\footnote{\url{https://www.qt.io/}} \gls{gui} that allows visual inspection of the scene while also giving the option to add new objects or move and rotate existing models. Moreover, for lightweight rendering, it can also start in server mode without a \gls{gui}.

\begin{figure}
	\centering
	\includegraphics[width=0.8\linewidth]{scene-rendering}
	\caption{3D scene rendering using the Gazebo simulator}
	\label{fig:scene-rendering}
\end{figure}

\section{Human machine interaction}\label{sec:human-machine-interaction}

The immersive \gls{hmi} system developed (shown in \cref{fig:scene-rendering,fig:human-machine-interface}) projects into the workspace detailed textual information of the current assembly task along with a video showing the operation being performed by an expert operator. Given the high variability of assembly / maintenance operations, the system was designed to decompose the assembly process into a set of small and concise operations. This allows to keep the operator focused on the current task and reduces the required projection area. Moreover, the operator can pause and move the video forwards and backwards, allowing to inspect a given complex operation with more time.

The user interaction with the projected \gls{hmi} is done by analyzing the 3D point cloud sensor data that falls within a set of \glspl{roi} (shown in \cref{fig:interaction-rois}). When a minimum number of points falls within a given \gls{roi}, the cluster of points centroid is extracted (shown as spheres in \cref{fig:interaction-rois}) and if this \gls{roi} is associated with a button, the user needs to hold the finger for at least 0.25 seconds to trigger the action. Moreover, to avoid unintentional action triggering, the user needs to remove and insert the finger into the \gls{roi} to request the action again. On the other hand, if the \gls{roi} is associated with the video seek bar \gls{roi} (the vertical yellow box in \cref{fig:interaction-rois}), the video seek time is computed considering the relative position of the finger within the \gls{roi} (the bottom of the \gls{roi} is associated with the start of the current video while the top corresponds to the end of the current video).

Besides video play / pause functionality (provided by the blue middle \gls{roi} shown in \cref{fig:interaction-rois}), the \gls{hmi} also allows the operator to navigate within the assembly operations (using the green \glspl{roi} visible in \cref{fig:interaction-rois}). As can be seen in \cref{fig:human-machine-interface,fig:interaction-rois}, there are buttons for moving to the first, previous, next and last assembly step. Moreover, it is also shown what is the number of the current assembly step and how many steps are required to complete the assembly.

\begin{figure}[H]
	\begin{floatrow}[2]
		\ffigbox[\FBwidth]
		{\includegraphics[height=.35\textheight]{human-machine-interface}}
		{\caption{Rendering of the human machine interface}\label{fig:human-machine-interface}}
		\ffigbox[\FBwidth]
		{\includegraphics[height=.35\textheight]{interaction-rois}}
		{\caption{\glspl{roi} for the \gls{hmi} (overlaid on top of the Kinect 2 point cloud sensor data)}\label{fig:interaction-rois}}
	\end{floatrow}
\end{figure}

\section{Object 3D reconstruction}\label{sec:object-reconstruction}

The last step of the assembly process is a visual inspection by the operator of the final product in order to ensure that every part was mounted correctly. To speedup this analysis the projection mapping system projects into the workspace the expected product outline (generated by colorizing the mesh using the surface curvature information). To be able to perform proper 3D rendering and also to detect and track the object within the workspace, it is necessary the 3D \gls{cad} model of the final product. Given the lack of public available \gls{cad} models for the Mitsubishi M000T20873 starter motor (shown in \cref{fig:starter-motor}), it was necessary to perform object 3D reconstruction. The 3D mesh model shown in \cref{fig:object-reconstruction} was generated using the David Laser 3D structured light system\footnote{\url{http://www.david-3d.com}}, and was built by surface matching algorithms using sensor data retrieved from 38 scans in which the starter motor was moved and rotated several times in order to capture enough sensor data to be able to reconstruct the entire surface. This particular object created some challenges for the structured light scanner, given that it has polished metallic sections and also black coated regions. As such, it was necessary to capture the same object sections several times with different projector brightness and camera exposure times (the dark regions required the maximum projector brightness and very high exposure times while the polished sections required dimmer projector brightness and very short exposure times in order for the surface to be fully reconstructed).

\begin{figure}[H]
	\begin{floatrow}[2]
		\ffigbox[\FBwidth]
		{\includegraphics[height=.136\textheight]{mitsubishi-m000t20873-front}\includegraphics[height=.136\textheight]{mitsubishi-m000t20873-back}}
		{\caption{Mitsubishi M000T20873 starter motor}\label{fig:starter-motor}}
		\ffigbox[\FBwidth]
		{\includegraphics[height=.136\textheight]{object-reconstruction-front}\includegraphics[height=.136\textheight]{object-reconstruction-back}}
		{\caption{3D model of the starter motor reconstructed using a structured light scanner}\label{fig:object-reconstruction}}
	\end{floatrow}
\end{figure}

\section{Object recognition}\label{sec:object-recognition}

Robust and accurate object recognition and pose estimation is a requirement when developing projection mapping systems with dynamic objects. This is one of the most challenging perception tasks, which has received a lot of research over the years, both at the hardware and software level. The next sections describe the main processing stages of the robot localization system (described in \cite{Costa2016}) that was reconfigured and improved to perform 3D object recognition.

\subsection{Reference models}

The first step in any perception system is the definition of the geometry of the object that we intend to recognize. As such, given a \gls{cad} / mesh model, the object recognition system generates the respective point cloud and the associated keypoints and feature descriptors.

\subsection{Point cloud assembly}

The object recognition system can use any sensor that provides point clouds, namely RGB-D / \gls{tof} cameras, \glspl{lidar}, stereo vision systems among many others. Each of these types of sensors have very different operation rates and measurement accuracy. As such, the recognition system allows the assembly of several sensor scans using a circular buffer in order to reduce the impact of sensor noise and increase the observed surface area of the objects.

\subsection{Filtering and down sampling}

The time it takes to perform cloud registration increases considerable as the amount of points in the sensor point cloud and in the reference model becomes larger. As such, adjusting the level of detail of the point clouds by using voxel grids and random sampling gives some control over the desired pose estimation accuracy and the computational resources that will be required. Moreover, when we know the expected workspace, we can crop the sensor data to a given \gls{roi} (such as a given volume above the working table), which removes any unnecessary environment clutter, resulting in a preliminary segmentation of the sensor data that later will be analyzed for object recognition. This stage is also useful to mitigate the measurement errors of the depth sensors, since the centroid of a voxel that contains points from several scans will be closer to the real surface (if the voxels have dimensions slightly larger than the expected measurement errors).

\subsection{Normal estimation}

Most of feature detection, description and matching algorithms along with some registration methods rely on the point's surface normal and curvature. These algorithms analyze the neighborhood of a given point in order to compute the surface normal, and as such, the correct specification of what points should be included in the estimation is crucial to achieve accurate results. This depends on the environment geometry and the level of detail that is required, and is usually done by specifying a radius distance or by limiting the number of neighboring points to use.

\subsection{Keypoint detection}

Aligning two point clouds with overlapping views of the environment requires the establishment of point correspondences. If both point clouds have similar sensor origins, these can be determined with nearest neighbor's searches and filtered with correspondence rejectors (using other point properties such as reflectance and color). But if they were acquired in two very different positions, then more advanced techniques must be employed.

One of those techniques uses histograms to describe the geometric properties of the environment around a given point. This allows points to be matched even if they have completely different Euclidean coordinates. Also, by using histograms and sampling techniques, these descriptors are much more robust against sensor noise and varying level of point density. However, these advantages come with a heavy computational cost, and as such, point descriptors should only be computed on the most descriptive areas of the environment.

Identifying these environment points is known as feature / keypoint detection \cite{Filipe2014}, and usually involves finding interesting points, such as corners and edges. Besides uniqueness, these points must also be repeatable. This means that the detection algorithms should be able to find the same points even if they are present in different point clouds with sensor noise and varying point density. This is of the utmost importance, because if the same keypoints are not identified on both clouds, then matching the point descriptors will likely fail.

The object recognition systems used the \gls{sift} \cite{Lowe2004} algorithm on the point's curvature, but it also supports the \gls{iss3d} \cite{Zhong2009} keypoint detector on the point's normals.

\subsection{Keypoint description}

Describing a keypoint usually involves analyzing its neighboring points and computing a given metric or histogram that quantifies the neighbor's relative spatial distribution, their normals angular relation, associated geometry or other metrics that are deemed useful. Several approaches were suggested over the years according to different recognition needs and they are the basis of feature matching algorithms used in the initial pose estimation.

The object recognition system used the \gls{fpfh} \cite{Rusu2009} keypoint descriptor, but it also supports the \gls{pfh} \cite{Rusu2008a}, the \gls{shot} \cite{Tombari2011}, the \gls{sc3d} \cite{Frome2004}, the \gls{usc} \cite{Tombari2010} and the \gls{esf} \cite{Wohlkinger2011}.

\subsection{Cloud registration}

Point cloud registration is the process of finding the transformation matrix (usually translation and rotation only) that when applied to a given ambient cloud will minimize an error metric (such as the mean square error of the ambient point cloud in relation to a given reference point cloud). Several approaches were suggested over the years and they can be categorized as point or feature cloud registration.

\subsubsection{Initial alignment with keypoints descriptor matching}\label{subsec:localization-system_feature-registration}

Feature registration is the process of matching keypoint descriptors in order to find an initial alignment between two point clouds. The object recognition system uses a feature registration method similar to the \gls{sacia} algorithm presented in \cite{Rusu2009}. It uses a \gls{ransac} approach to select the best registration transformation after a given number of iterations. In each iteration a subsample of randomly selected descriptors from the ambient cloud is retrieved. Then for each of these descriptors, $k$ best matching descriptors in the reference point cloud are searched (using a kd-tree) and one of them is chosen randomly. Later after having filtered these correspondences between ambient and reference descriptors, the registration matrix is computed. If this registration matrix results in a point cloud overlap that has a minimum of inliers percentage (a point in the ambient cloud is an inlier if it has a point in the reference cloud closer than a given distance), then it is considered an acceptable initial pose. In the end of all iterations, the best initial pose (if found) is refined with a point cloud registration algorithm.

\subsubsection{Final alignment with point cloud error minimization}

Point cloud registration algorithms such as \gls{icp} \cite{Besl1992} (with its several known variations \cite{Rusinkiewicz2001,Pomerleau2013} like \gls{icp} point-to-point, \gls{icp} point-to-point non-linear, \gls{icp} point-to-plane and generalized \gls{icp} \cite{Segal2009}) and the \gls{ndt} \cite{Magnusson2009} are among the most popular algorithms to register point clouds. They can achieve very accurate cloud registration but they require an approximate initial pose for the registration to successfully converge to a correct solution (otherwise they may achieve only partial cloud overlap or even fail to converge to a valid solution).

In \cref{fig:initial-pose-estimation} is shown the estimation of the pose of the starter motor. When the system started (left image), it subsampled the reference pointcloud (small green circles) and computed the keypoints (large yellow circles) and associated feature descriptors. Then, when sensor data arrived, it found the keypoints (blue circles) and their descriptors. Finally it applied the feature matching and registration refinement and achieved the pose estimation showed on the right image of \cref{fig:initial-pose-estimation} (observe the overlap between the sensor data and the reference point cloud). After the successful initial pose estimation phase, the recognition system enters in tracking mode and relies only on point cloud matching algorithms (given their better accuracy and lower computational cost). If the tracking is lost, the feature matching algorithms are used again to find a plausible estimation of the object position that is then refined with the point cloud registration refinement algorithms.

%\begin{figure}[!ht]
%	\centering
%	\includegraphics[height=.27\textheight]{initial-pose-estimation-1-before}
%	\hspace{0.5em}
%	\includegraphics[height=.27\textheight]{initial-pose-estimation-1-after}
%	\caption{Initial pose estimation of assembled object}
%\end{figure}

\begin{figure}[!ht]
	\centering
	\includegraphics[height=.182\textheight]{initial-pose-estimation-2-before}
	\hspace{0.5em}
	\includegraphics[height=.182\textheight]{initial-pose-estimation-2-after}
	\caption{Initial pose estimation of assembled starter motor}
	\label{fig:initial-pose-estimation}
\end{figure}

\subsection{Outlier detection}

Detecting which points from the sensor data are not part of the reference point cloud can be very useful to evaluate the quality of point cloud registration as well as to analyze if the object recognition was successful (if a high number of points were considered outliers then the object recognition is likely to have failed). It's computation splits the ambient cloud into two point sets. One containing inliers (points correctly registered and present in the reference point cloud) and the other having the outliers (points that are either incorrectly registered or not present in the reference cloud). A given ambient point can be classified as outlier if the corresponding closest point in the reference cloud is farther way than a given distance threshold. These operations can be done efficiently by using the reference point cloud kd-tree.


\documentclass[12pt]{article}
\usepackage[fleqn]{amsmath}
\usepackage{textcomp}
\usepackage{gensymb}
\usepackage{enumitem}
%\usepackage{tikz}  % Include for figures.
%\usepackage{subfiles}  % Include for subfiles.

\newcommand{\HOMEWORKNUM}{3}
\newcommand{\NAME}{D. Choi}
\newcommand{\DATE}{2020-05-27}

\title{\vspace{-4\baselineskip}MATH 225 - Homework \#\HOMEWORKNUM}
\author{\NAME}
\date{\DATE}

%\pagenumbering{gobble}  % Include for single-page document.

\begin{document}
\maketitle

\section*{1.}
\textit{Find the $2 \times 2$ matrix $M$ that is associated with the
orthographic projection onto ${y = -\sqrt{3}x}$ in 2 ways.}
% Hey wow, you can put {braces} around a math expression to make it an atom,
% preventing line breaks from sneaking in! (Shouldn't matter here, though.)
\begin{enumerate}[label=(\alph*)]
	\item \textit{``Hard geometry, easy algebra.''}
	\\[\baselineskip]
	Note that $y = -\sqrt{3}x = \tan(120 \degree)x$. \\
	Let $\hat u(\theta) =
	\begin{pmatrix} \cos \theta \\ \sin \theta \end{pmatrix}$.
	\begin{gather*}
		\begin{pmatrix} 1 \\ 0 \end{pmatrix}
		\overset{M}{\longrightarrow}
		\text{proj}_{\hat u(120 \degree)} \hat u(0 \degree)
		=
		\cos(120 \degree) \cdot \hat u(120 \degree)
		=
		\begin{pmatrix} \frac{1}{4} \\ -\frac{\sqrt{3}}{4} \end{pmatrix}
		\\
		\begin{pmatrix} 0 \\ 1 \end{pmatrix}
		\overset{M}{\longrightarrow}
		\text{proj}_{\hat u(120 \degree)} \hat u(90 \degree)
		=
		\cos(30 \degree) \cdot \hat u(120 \degree)
		=
		\begin{pmatrix} -\frac{\sqrt{3}}{4} \\ \frac{3}{4} \end{pmatrix}
	\end{gather*}
	Thus,
	\begin{equation*}
		M =
		\boxed{
			\begin{pmatrix}
				\frac{1}{4} & -\frac{\sqrt{3}}{4} \\
				-\frac{\sqrt{3}}{4} & \frac{3}{4}
			\end{pmatrix}
		}
	\end{equation*}
	\item \textit{``Easy geometry, hard algebra.''}
	\begin{gather*}
		\begin{pmatrix} \sqrt{3} \\ 1 \end{pmatrix}
		=
		2 \cdot \hat u(30 \degree)
		\overset{M}{\longrightarrow}
		\begin{pmatrix} 0 \\ 0 \end{pmatrix}
		\\
		\begin{pmatrix} 1 \\ -\sqrt{3} \end{pmatrix}
		=
		-2 \cdot \hat u(120 \degree)
		\overset{M}{\longrightarrow}
		\begin{pmatrix} 1 \\ -\sqrt{3} \end{pmatrix}
	\end{gather*}
	\newpage
	Let $M = \begin{pmatrix} a & b \\ c & d \end{pmatrix}$.
	\begin{gather*}
		\sqrt{3}a + b = 0 \\
		a - \sqrt{3}b = 1 \\
		\sqrt{3}c + d = 0 \\
		c - \sqrt{3}d = -\sqrt{3}
	\end{gather*}
	$a = \frac{1}{4}, b = -\frac{3}{4}, c = -\frac{3}{4}, d = \frac{3}{4}$
	\\[\baselineskip]
	Thus,
	\begin{equation*}
		M =
		\boxed{
			\begin{pmatrix}
				\frac{1}{4} & -\frac{\sqrt{3}}{4} \\
				-\frac{\sqrt{3}}{4} & \frac{3}{4}
			\end{pmatrix}
		}
	\end{equation*}
\end{enumerate}

\section*{2.}
\textit{Find the matrix $B$ that reflects across $y = -\sqrt{3}x$.}
\\[\baselineskip]
Let $\text{Ref}(\theta)$ be the reflection matrix where
\begin{equation*}
	\text{Ref}(\theta) =
	\begin{pmatrix}
		\cos(2 \theta) & \sin(2 \theta) \\
		\sin(2 \theta) & -\cos(2 \theta)
	\end{pmatrix}
	.
\end{equation*}
Note that $y = -\sqrt{3}x = \tan(120 \degree)x$. \\
Thus,
\begin{equation*}
	B = \text{Ref}(120 \degree) =
	\begin{pmatrix}
		\cos(2 \cdot 120 \degree) & \sin(2 \cdot 120 \degree) \\
		\sin(2 \cdot 120 \degree) & -\cos(2 \cdot 120 \degree)
	\end{pmatrix}
	=
	\boxed{
		\begin{pmatrix}
			-\frac{1}{2} & -\frac{\sqrt{3}}{2} \\
			-\frac{\sqrt{3}}{2} & \frac{1}{2}
		\end{pmatrix}
	}
\end{equation*}
\newpage

\section*{3.}
\textit{Let $A$ be a $30 \degree$ rotation, let $B$ be the same matrix
from \textbf{(2)}, and let $M$ be the same matrix from \textbf{(1)}. \\
Find the following.}
\\[\baselineskip]
First, let $\text{Rot}(\theta)$ be the rotation matrix where
\begin{equation*}
	\text{Rot}(\theta) =
	\begin{pmatrix}
		\cos \theta & -\sin \theta \\
		\sin \theta & \cos \theta
	\end{pmatrix}
	.
\end{equation*}
\begin{enumerate}[label=(\alph*)]
	\item $A^{1001} = \text{Rot}(1001 \cdot 30 \degree) =
	\text{Rot}(1001 \cdot 30 \degree \text{mod} 360) =
	\boxed{\text{Rot}(150 \degree)}$
	\item $B^{1001} = (B^2)^{500} \cdot B =
	([\text{Ref}(120 \degree)]^2)^{500} \cdot \text{Ref}(120 \degree) =
	\boxed{\text{Ref}(120 \degree)}$
	\item $AB = \text{Rot}(30 \degree) \text{Ref}(120 \degree) =
	\text{Ref}(120 \degree + \frac{1}{2} \cdot 30 \degree) =
	\boxed{\text{Ref}(135 \degree)}$
	\item $BA = \text{Ref}(120 \degree) \text{Rot}(30 \degree) =
	\text{Ref}(120 \degree - \frac{1}{2} \cdot 30 \degree) =
	\boxed{\text{Ref}(105 \degree)}$
	\item $A^{-1} B^5 = A^{-1} (B^2)^2 \cdot B = A^{-1} B =
	\text{Rot}(-30 \degree) \text{Ref}(120 \degree) = \\
	\text{Ref}(120 \degree + \frac{1}{2} \cdot -30 \degree) =
	\boxed{\text{Ref}(105 \degree)}$
	\item $M^{1001} = \boxed{M}$ \quad ($M$ is a projection, so $M^2 = M$.)
	\item
	$AB \begin{pmatrix} \cos(40 \degree) \\ \sin(40 \degree) \end{pmatrix}$
	\begin{align*}
		&=
		\text{Rot}(30 \degree)
		\text{Ref}(120 \degree)
		\begin{pmatrix} \cos(40 \degree) \\ \sin(40 \degree) \end{pmatrix} \\
		&=
		\text{Rot}(30 \degree)
		\begin{pmatrix} \cos(200 \degree) \\ \sin(200 \degree) \end{pmatrix}
		\\
		&=
		\boxed{
			\begin{pmatrix}
				\cos(230 \degree) \\ \sin(230 \degree)
			\end{pmatrix}
		}
	\end{align*}
\end{enumerate}

\section*{4.}
\textit{Find a $3 \times 3$ matrix $C$ that rotates 3-space $30 \degree$ about
the z-axis.}
\begin{gather*}
	\begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix}
	\overset{C}{\longrightarrow}
	\begin{pmatrix} \cos(30 \degree) \\ \sin(30 \degree) \\ 0 \end{pmatrix}
	\\
	\begin{pmatrix} 0 \\ 1 \\ 0 \end{pmatrix}
	\overset{C}{\longrightarrow}
	\begin{pmatrix} \cos(120 \degree) \\ \sin(120 \degree) \\ 0 \end{pmatrix}
	\\
	\begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix}
	\overset{C}{\longrightarrow}
	\begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix}
\end{gather*}
\begin{equation*}
	C =
	\boxed{
		\begin{pmatrix}
			\cos(30 \degree) & cos(120 \degree) & 0 \\
			\sin(30 \degree) & sin(120 \degree) & 0 \\
			0 & 0 & 1
		\end{pmatrix}
	}
\end{equation*}

\end{document}


\section{Cutting}
\label{sec:cutting}
\index{stack!cutting}

\index{cut@\fun{cut/2}|(} Let us consider the problem of cutting a
stack~\(s\) at the~\(k\)th place. Obviously, the result is a pair of
stacks. More precisely, let \(\pair{t}{u}\) be the value of
\(\fun{cut}(s,k)\), such that \(\fun{cat}(t,u) \twoheadrightarrow
s\)\index{cat@\fun{cat/2}} and \(t\)~contains \(k\)~items, that is to
say, \(\fun{len}(t) \twoheadrightarrow k\). In particular, if~\(k =
0\), then \(t = \el\); invalid inputs lead to unspecified
behaviours. For instance, \(\fun{cut}([4,2],0) \twoheadrightarrow
\pair{\el}{[4,2]}\) and \(\fun{cut}([5,3,6,0,2],3) \twoheadrightarrow
\pair{[5,3,6]}{[0,2]}\), but, for the sake of simplicity, nothing is
said about \(\fun{cut}([0],7)\) and \(\fun{cut}([0],-1)\). We derive
two cases: \(k = 0\) or else the stack is not empty. The former is
easy to guess:
\begin{equation*}
\fun{cut}(s,0)           \rightarrow \pair{\el}{s};\qquad
\fun{cut}(\cons{x}{s},k) \rightarrow \fbcode{CCCCCC}\,.
\end{equation*}
A big\hyp{}step design\index{design!big-step $\sim$} uses some
sub\-structural recursive calls to set the structure of the value in
the right\hyp{}hand side. Because \fun{cut/2} takes two arguments, we
expect a lexicographic order\index{induction!lexicographic order}
(definition~\eqref{def:lexico}, page~\pageref{def:lexico}):
\begin{equation*}
\fun{cut}(s_0,k_0) \succ \fun{cut}(s_1,k_1)
:\Leftrightarrow
\text{\(s_0 \succ s_1\) or (\(s_0 = s_1\) and \(k_0 > k_1\))}.
\end{equation*}
Using the proper subterm order\index{induction!proper subterm order}
(section~\ref{par:well-founded}, page~\pageref{par:well-founded}) on
stacks for (\(\succ\)),
\begin{equation*}
\fun{cut}(\cons{x}{s},k) \succ \fun{cut}(s,j);\quad
\fun{cut}(\cons{x}{s},k) \succ \fun{cut}(\cons{x}{s},j),\;
\text{if \(k > j\)}.
\end{equation*}
In the latter case, we can set \(j=k-1\), but the value of
\(\fun{cut}(s,j)\) needs to be projected into \(\pair{t}{u}\) so
\(x\)~is injected in and yields \(\pair{\cons{x}{t}}{u}\). This can be
achieved with an auxiliary
function~\fun{push/2}\index{push@\fun{push/2}}:
\begin{equation*}
\begin{array}{@{}r@{\;}l@{\;}lr@{\;}l@{\;}l@{}}
\fun{cut}(s,0) & \rightarrow & \pair{\el}{s};
& \fun{push}(x,\pair{t}{u}) & \rightarrow & \pair{\cons{x}{t}}{u}.\\
\fun{cut}(\cons{x}{s},k) & \rightarrow
& \fun{push}(x,\fun{cut}(s,k-1)).
\end{array}
\end{equation*}

\paragraph{Inference systems}
\label{par:infsys}
\index{inference system}

When the value of a recursive call needs to be destructured, it is
convenient to use an extension of our language to avoid creating
auxiliary functions like \fun{push/2}:
\begin{mathpar}
\inferrule*{}{\fun{cut}(s,0) \rightarrow \pair{\el}{s}}
\;\TirName{Nil}
\qquad
\inferrule
  {\fun{cut}(s,k-1)         \twoheadrightarrow \pair{t}{u}}
  {\fun{cut}(\cons{x}{s},k) \twoheadrightarrow \pair{\cons{x}{t}}{u}}
\,\TirName{Pref}
\end{mathpar}
The new construct is called an \emph{inference rule}\index{inference
  system!rule} because it means: `For the value of
\(\fun{cut}(\cons{x}{s},k)\) to be \(\pair{\cons{x}{t}}{u}\), we infer
that the value of \(\fun{cut}(s,k-1)\) must be \(\pair{t}{u}\).' This
interpretation corresponds to an upwards reading of the rule
\TirName{Pref} (\emph{prefix}). Just as we compose horizontally
rewrite rules, we compose inference rules vertically, stacking them,
as in
\begin{mathpar}
\inferrule
  {\inferrule{
     \inferrule{\fun{cut}([0,2],0) \rightarrow \pair{\el}{[0,2]}}
               {\fun{cut}([6,0,2],1) \twoheadrightarrow \pair{[6]}{[0,2]}}}
      {\fun{cut}([3,6,0,2],2) \twoheadrightarrow \pair{[3,6]}{[0,2]}}}
  {\fun{cut}([5,3,6,0,2],3) \twoheadrightarrow \pair{[5,3,6]}{[0,2]}}
\end{mathpar}
When determining the cost of \(\fun{cut}(s,k)\), we take into account
the hidden function \fun{push/2}, so \(\Call{\fun{cut}([5,3,6,0,2],3)}
= 7\). In general, \index{cut@$\C{\fun{cut}}{n}$} \(\C{\fun{cut}}{k} =
2k + 1\).

Beyond simplifying programs, what makes this formalism interesting is
that it enables two kinds of interpretation: logical and
computational. The computational reading, called
\emph{inductive}\index{inference system!inductive reading} in some
contexts, has just been illustrated. The logical understanding
considers inference rules as logical implications of the form \(P_1
\wedge P_2 \wedge \ldots \wedge P_n \Rightarrow C\), written
\begin{equation*}
\inferrule
  {P_1 \\ P_2 \\ \ldots \\ P_n}
  {C}
\end{equation*}
The propositions~\(P_i\) are called \emph{premises}\index{inference
  system!rule!premise} and \(C\)~is the \emph{conclusion}. In the case
of \TirName{Pref}, there is only one premise. When premises are
lacking, as in \TirName{Nil}, then \(C\)~is called an \emph{axiom} and
no horizontal line is drawn. The composition of inference rules is a
\emph{derivation}. In the case of \fun{cut/2}, all derivations are
isomorphic to a stack, whose top is the conclusion.

The logical reading of rule \TirName{Pref} is: `If \(\fun{cut}(s,k-1)
\twoheadrightarrow \pair{t}{u}\), then we have
\(\fun{cut}(\cons{x}{s},k) \twoheadrightarrow
\pair{\cons{x}{t}}{u}\).'  This top\hyp{}down reading qualifies as
\emph{deductive}\index{inference system!deductive reading}. The
previous derivation then can be regarded as the proof of the theorem
\(\fun{cut}([5,3,6,0,2],3) \twoheadrightarrow \pair{[5,3,6]}{[0,2]}\).

\paragraph{Induction on proofs}
\index{induction!$\sim$ on proofs}

A single formalism with such a dual interpretation is powerful because
a definition by means of inference rules enables the proof of theorems
about a function by \emph{induction on the structure of the proof}. As
we have done previously, structural induction can be applied to stacks
considered as a data type (objects). Since, in the case of
\fun{cut/2}, derivations are stacks in themselves, so can induction be
applied to their structure (as meta\hyp{}objects).

Let us illustrate this elegant inductive technique with a proof of the
\emph{soundness}\index{soundness} of
\fun{cut/2}\index{cut@\fun{cut/2}}.

\mypar{Soundness}
\label{par:cut_sound}

The concept of soundness or \emph{correctness} \citep{McCarthy_1962,
  Floyd_1967, Hoare_1971,
  Dijkstra_1976}\index{correctness|see{soundness}} is a binary
relationship, so we always ought to speak of the soundness of a
program with respect to its
\emph{specification}\index{specification}. A specification is a
logical description of the expected properties of the output of a
program, given some assumptions on its input. In the case of
\(\fun{cut}(s,k)\), we already mentioned what to expect: the value
must be a pair \(\pair{t}{u}\) such that the catenation of
\(t\)~and~\(u\) is~\(s\) and the length of~\(t\) is~\(k\).

Formally, let \(\pred{CorCut}{s,k}\)\index{CorCut@\predName{CorCut}|(}
be the proposition
\begin{quote}
  \textsl{If \(\fun{cut}(s,k) \twoheadrightarrow \pair{t}{u}\), then
    \(\fun{cat}(t,u) \twoheadrightarrow s\) and \(\fun{len}(t)
    \twoheadrightarrow k\),}
\end{quote}
where the function \fun{len/1}\index{len@\fun{len/1}|(} is defined in
equation~\eqref{eq:len} \vpageref{eq:len}.

Let us suppose the antecedent of the implication to be true, otherwise
the theorem is vacuously true, so there exists a derivation~\(\Delta\)
whose conclusion is \(\fun{cut}(s,k) \twoheadrightarrow
\pair{t}{u}\). This derivation is a (meta) stack whose top is the
conclusion in question, which makes it possible to reckon by induction
on its structure, that is, we assume that \predName{CorCut} holds for
the immediate sub\-derivation of~\(\Delta\) (the induction hypothesis)
and then proceed to prove that \predName{CorCut} holds for the entire
derivation. This is the immediate subterm induction we use when
reasoning on a stack as an object: we assume the theorem to hold
for~\(s\) and then move to prove it for~\(\cons{x}{s}\).

A case by case analysis on the kind of rule that can end~\(\Delta\)
guides the proof. To avoid clashes between variables in the theorem
and in the inference system, we will overline the latter ones, like
\(\overline{s}\), \(\overline{t}\) etc. which may differ from
\(s\)~and~\(t\) in \predName{CorCut}.
\begin{itemize}

\item \emph{Case where \(\Delta\) ends with \TirName{Nil}.} There are
  no premises, as~\TirName{Nil} is an axiom. In this case, we have to
  establish \predName{CorCut} without induction. The matching of
  \(\fun{cut}(s,k) \twoheadrightarrow \pair{t}{u}\) against
  \(\fun{cut}(\overline{s},0) \rightarrow \pair{\el}{\overline{s}}\)
  yields \(\overline{s} = s\), \(0=k\), \(\el=t\) and
  \(\overline{s}=u\). Therefore, \index{cat@\fun{cat/2}}
  \(\fun{cat}(t,u) = \fun{cat}(\el,s) \xrightarrow{\smash{\alpha}}
  s\), which proves half of the conjunction. Moreover
  \(\fun{len}(t) = \fun{len}(\el) \xrightarrow{\smash{a}} 0\). This is
  consistent with \(k=0\), so \(\pred{CorCut}{s,0}\) is
  true.\index{CorCut@\predName{CorCut}|)}

  \item \emph{Case where \(\Delta\) ends with \TirName{Pref}.} The
    shape of~\(\Delta\) is thus as follows:
    \begin{mathpar}
      \inferrule*[right=Pref]
        {\inferrule*
           {\inferrule*[vdots=1.5em]{}{ }}
           {\fun{cut}(\overline{s},\overline{k}-1)
            \twoheadrightarrow \pair{\overline{t}}{\overline{u}}}
        }
        {\fun{cut}(\cons{\overline{x}}{\overline{s}},\overline{k})
         \twoheadrightarrow
         \pair{\cons{\overline{x}}{\overline{t}}}{\overline{u}}}
    \end{mathpar}
    The matching of \(\fun{cut}(s,k) \twoheadrightarrow \pair{t}{u}\)
    against the conclusion yields \(\cons{\overline{x}}{\overline{s}}
    = s\), \(\overline{k} = k\), \(\cons{\overline{x}}{\overline{t}} =
    t\) and \(\overline{u} = u\). The induction hypothesis in this
    case is that the theorem holds for the sub\-derivation,
    therefore\index{cat@\fun{cat/2}|(} \(\fun{cat}(\overline{t},
    \overline{u}) \twoheadrightarrow \overline{s}\) and
    \(\fun{len}(\overline{t}) \twoheadrightarrow \overline{k} -
    1\).\index{len@\fun{len/1}|)} The induction principle requires
    that we establish now
    \(\fun{cat}(\cons{\overline{x}}{\overline{t}}, \overline{u})
    \twoheadrightarrow \cons{\overline{x}}{\overline{s}}\) and
    \(\fun{len}(\cons{\overline{x}}{\overline{t}}) \twoheadrightarrow
    \overline{k}\). From the definition of \fun{cat/2} and part of the
    hypothesis, we easily deduce
    \(\fun{cat}(\cons{\overline{x}}{\overline{t}}, \overline{u})
    \xrightarrow{\smash{\beta}}
    \cons{\overline{x}}{\fun{cat}(\overline{t}, \overline{u})}
    \twoheadrightarrow \cons{\overline{x}}{\overline{s}}\). Now the
    other part: \(\fun{len}(\cons{\overline{x}}{\overline{t}})
    \xrightarrow{\smash{b}} 1 + \fun{len}(\overline{t})
    \twoheadrightarrow 1 + (\overline{k} - 1) =
    \overline{k}\).\index{cat@\fun{cat/2}|)}\hfill\(\Box\)

\end{itemize}

\paragraph{Exercise}

Write an equivalent definition of \fun{cut/2} in tail
form.\index{functional language!tail form}
\index{cut@\fun{cut/2}|)}


\section{Fixed notation}

The following notation will be fixed.

\begin{description}
	\item[$\dayc(t,T)$] Day count convention fraction between the dates $t < T$.
	\item[$r(t)$] Short-rate at the time $t$.
	\item[$\Bank(t)$] The value of an idealized bank account at the time $t$.
	\item[$\DF (t,T)$] Stochastic discount factor between the dates $t,T$.
	\item[$\Bond(t,T)$] The price of $T$-bond (zero coupon bond with the maturity $T$) at the time $t$.
	\item[$\Rflt(t,T)$] Simple spot rate between dates $t$ and $T$
	\item[$\Rflt^k(t,T)$] $k$-times compounded simple spot rates between dates $t$ and $T$
	\item[$\FRA(t,T,S,K)$] The price of a forward rate agreement at the time $t$, where $K$ is the fixed price and the interest is paid between the dates $T < S$.
	\item[$\Rflt(t,T,S)$] Simple forward rate at the time $t$ between dates $T$ and $S$
	\item[$\Forwardrate(t,T)$] Instantaneous forward rate at the time $t$ for date $T$.
	\item[$\FUT(t,T,S)$] Futures rate at the time $t$ between dates $T$ and $S$.
	\item[$\DBond(t,T)$] The price of defaultable $T$-bond (zero coupon bond with the maturity $T$) at the time $t$.
	\item[$\ZBC(t,S,T,K)$] The price of call option at the time $t$ maturing at $S$ written on $T$-bond.
	\item[$\default$] The time of a default.
	\item[$\LGD$] The loss given the default (LGD) of a contract.
	\item[$\Rec$] The recovery value of a contract.
	\item[$\Protection(t)$] The value of a protection leg of a credit default swap.
	\item[$\Premium(t,C)$] The value of a premium leg of a credit default swap with a coupon rate $C$.
\end{description}


\documentclass[12pt]{article}
\usepackage[utf8]{inputenc}
\usepackage[english]{babel}

\usepackage{geometry}
 \geometry{
 a4paper,
 total={170mm,250mm},
 left=31mm,
 right=31mm,
 top=34mm,
 bottom=29mm
 }

\usepackage{amsfonts,amsmath}
\usepackage{mathtools}
\usepackage{graphicx}
\usepackage{amssymb}
\usepackage{amsthm}
\usepackage{tikz-cd}
\usepackage{mathrsfs}
\usepackage{subcaption}
\usepackage{multicol}
\usepackage[colorinlistoftodos]{todonotes}
\usepackage{enumitem}
\usepackage{hyperref}
\usepackage{url}
\hypersetup{
   colorlinks=true,
    linkcolor=blue,
    citecolor=red,
    urlcolor=cyan,
}
%\usepackage{yfonts}
\usepackage{mathdots}

\usepackage{fancyhdr}
\pagestyle{fancy}
\fancyhf{}
\rhead{\textit{Your name}}
\lhead{\small Your title}
\cfoot{\thepage}

\setcounter{MaxMatrixCols}{20} % Enable us to create matrices with more than 10 columns

\title{Your title}

\author{\textit{Your name}\footnote{Your institution}}

\date{\today}

\setlength{\footnotesep}{0.5cm}
\setlength{\skip\footins}{1cm}

\newtheorem{thm}{Theorem}[section]
\newtheorem{lem}[thm]{Lemma}
\newtheorem{cor}[thm]{Corollary}
\newtheorem{prop}[thm]{Proposition}
\newtheorem{defn}[thm]{Definition}
\newtheorem{obs}[thm]{Observation}
\theoremstyle{definition}
\newtheorem{eg}[thm]{Example}
\newtheorem{ex}[thm]{Exercise}

\begin{document}
\maketitle

\begin{abstract}
A concise summary of the whole reading report and main findings
\end{abstract}

\section{Introduction}

A brief overview of the subject or research field that you are going to discuss.

\section{Background}

\begin{defn}
A list of vectors $x_1,...,x_k\in \mathbf{C}^n$ is \emph{orthogonal} if $x_i^*x_j=0$ for all $i\neq j, i,j\in\{1,...,k\}$. If, in addition, $x_i^*x_i=1$ for all $i=1,...,k$ (that is, the vectors are \emph{normalized}), then the list is \emph{orthonormal}.
\end{defn}

\begin{eg}[normalization]
\label{Normalization}
If $y_1,...,y_k\in \mathbf{C}^n$ are orthogonal and nonzero, the vectors $x_1,...,x_k$ defined by $x_i=(y_i^*y_i)^{-\frac{1}{2}}y_i, i=1,...,k$ are orthonormal.
\end{eg}

\begin{thm}
Every orthogonal list of vectors in $\mathbf{C}^n$ is linearly independent.
\end{thm}
\begin{proof}
Suppose that $\{y_1,...,y_k\}$ is an orthogonal set. Normalize them as Example \ref{Normalization} did and obtain an orthonormal list of vectors $\{x_1,...,x_k\}$. Assume that $0=\alpha_1x_1+\cdots+\alpha_k x_k$. Then $0=(\alpha_1x_1+\cdots+\alpha_k x_k)^*(\alpha_1x_1+\cdots+\alpha_k x_k)=\sum\limits_{i,j} \bar{\alpha}_i \alpha_j x_i^*x_j= \sum\limits_{i=1}^k |\alpha_i|^2 x_i^*x_i=\sum\limits_{i=1}^k |\alpha_i|^2$ because the vectors $x_i$ are orthonormal. Thus, all $\alpha_i=0$ and hence $\{x_1,...,x_k\}$ is a linearly independent set, which in turn means that $\{y_1,...,y_k\}$ is linearly independent.
\end{proof}

\section{Method and Evaluation}

Main section of the reading report. It contains your solutions to the exercises, your methodology to address the problem, and your experimental results.

\section{Conclusions}

Summarize what you have written in previous sections and discuss your understandings of possible future research directions.

\vspace{1cm}
\begin{thebibliography}{00}

\bibitem{Horn} Roger A. Horn, Charles R. Johnson (2012) {\em Matrix Analysis, Second Edition.} Cambridge University Press.

% \bibitem{PowerIterWiki} Wikipedia: The Free Encyclopedia \textbf{\em Power iteration.} URL: \url{https://en.wikipedia.org/wiki/Power_iteration} Retrieved 12 April, 2018.

\end{thebibliography}

\end{document}


\documentclass[10pt, preprint]{aastex}

\usepackage{natbib}
\bibliographystyle{apj}

\usepackage{minted}
\usepackage{float}
\usepackage{graphicx}
\usepackage{subfig}
\usepackage{amsmath}
\usepackage[toc,page]{appendix}
\usepackage[utf8]{inputenc}
\usepackage{hyperref}
\hypersetup{
    colorlinks=true,
    linkcolor=blue,
    filecolor=magenta,
    urlcolor=blue,
    citecolor=blue,
}
\usepackage{booktabs}
\renewcommand{\arraystretch}{1}

\title{Assignment 2 - CTA200H}
\author{Rebecca Ceppas de Castro}

\begin{document}

\maketitle

\section*{Question 1}

For this question, two methods of numerical approximations for the derivative of $sin(x)$ at $x_0 = 0.1$ are compared to its analytical derivative. For reference, Equation \ref{method1} is called method 1 in the python script and it assumes $h$ is infinitesimally small. Equation \ref{method2} is referred to as method 2 and it more realistically assumes $h$ is a small but finite step size.

\begin{equation}\label{method1}
    d_xf|_{x_0} \approx \frac{f_{x_0 + h} - f_{x_0}}{h}
\end{equation}

\begin{equation}\label{method2}
    d_xf|_{x_0} \approx \frac{f_{x_0 + h} - f_{x_0 - h}}{2h}
\end{equation}

\begin{figure}[h]
    \centering
    \includegraphics[width=0.7\textwidth]{derivatives.pdf}
    \caption{Comparison of the Fractional error resulting from using method 1 and method 2 with the analytical derivative of sin(x) at 0.1}
    \label{derivatives}
\end{figure}

Figure \ref{derivatives} shows that the fractional errors from both methods of numerical approximations are relatively small for small values of h and increase at different rates for higher values of h, as expected. The first to notice is that method 2 always has a smaller fractional error, which agrees with the fact that that method assumes h is finite and we, in fact, used finite values of h in the calculations. The slope shows that the error in the approximation when using method 2 is more susceptible to changes in h, while method 1 shows less of this variation. In other words, while both methods have high errors for large step sizes, using method 2 produces more accurate results at a faster rate than if you were using method 1 as you decrease the step size.

\section*{Question 2}

In this question, we plot the Mandelbrot set, obtained through the recurring relation $z_{i+1} = z_{i}^2 + c$, where $c$ is a complex number given by $c = x + iy$ for $-2 < x < 2$ and $-2 < y < 2$. For different complex numbers $c$, the set diverges after a certain number of iterations, while for others it is bounded. In my function \textit{get\_z}, I use try and except to keep track of the bounded sets, as well as the number of iterations ran through before the divergent sets diverged. For the bounded sets, I designated a value of 10000 iterations, as if I used the real value - infinite number of iterations before divergence - we would lose some details in Figure \ref{mandelbrot}.

\begin{figure}[h!]
    \centering
    \subfloat[Divergent and Bounded points c]{{\includegraphics[width=0.5\textwidth]{duo.png}}}
    \hfill
    \subfloat[Mandelbrot Set with colour map based on number of iterations before divergence]{{\includegraphics[width=0.5\textwidth]{iterations.png}}}
    \caption{Mandelbrot Set}
    \label{mandelbrot}
\end{figure}

\section*{Question 3}

In the SIR model, we can use a system of differential equations to model the spread of a disease in a population. We have three parameters - $S(t)$: those susceptible but not infected, $I(t)$: those infected, $R(t)$: those recovered and immune. These 3 groups add up to the total population number $N$ and vary with time according to system of Equations \ref{SIR}.

\begin{equation*}
    \frac{dS}{dt} = -\frac{\beta S I}{N}
\end{equation*}

\begin{equation}\label{SIR}
    \frac{dI}{dt} = -\frac{\beta S I}{N} - \gamma I
\end{equation}

\begin{equation*}
    \frac{dR}{dt} = \gamma I
\end{equation*}

Where $\gamma$ represents the recovery rate and $\beta$ represents the transmission rate.

I used the scipy \textit{odeint} solver to find these parameters at different times, using different values of $\gamma$ and $\beta$ to see how the different groups evolved with time for each case. For all of the plots shown in Figure \ref{SIR}, I have taken $N = 1000$, and started with 1 infected person, none recovered and 999 susceptible.

I chose 4 combinations of $\gamma$ and $\beta$ to represent different scenarios. In the first case, I have the same transmission and recovery rate, so we see that the one infected person transmits the disease to more people, but the recover rate is at the same rate, so the disease doesn't spread as much and at around 60 (days, months, minutes - whatever time scale is relevant for the disease), it flattens out and the disease is pretty much eradicated. In the second case, we have a much higher transmission rate than recovery rate, so there is a steep peak in people infected and a much slower recovery curve. In this case, as well as the third one with $\gamma = 0.01$ and $\beta = 0.1$, the people susceptible reach 0, which means the whole population got infected at some point. However, we see that when the transmission and recovery rates have more similar values, the infection curve is less steep, as expected. Finally, for some combinations of $\gamma$ and $\beta$, the infection curve rises but is balance by an almost equal in magnitude recovery rate and it falls again. In this last scenario, the disease is eradicated before everyone is infected, but with much more damage that that shown in the first case.

\begin{figure}[h]
    \centering
    \includegraphics[width=0.8\textwidth]{SIR.pdf}
    \caption{Evolution of the three groups of the population using the SIR model with different $\beta$ and $\gamma$ values.}
    \label{SIR}
\end{figure}

\end{document}


\documentclass{article}
\usepackage{bibentry}
\usepackage{etoolbox}
\usepackage[]{geometry}
\usepackage[english]{babel}
\usepackage{mathpazo}
\usepackage{listings}
\usepackage{amsmath}
\usepackage{mathtools}
\makeatletter
\makeatother
\title{Supervised Learning}
\bibliographystyle{plain}
\author{}
\date{}
\pagestyle{empty}
\thispagestyle{empty}
\pagenumbering{gobble}
\setlength{\parindent}{0em}
\setlength{\parskip}{1em}
\begin{document}
\nocite{*}
\maketitle
\section{Introduction}

Supervised learning implies that the data is prelabeled ie already
distinguished. The model is responsible for correctly predicting this outcome.

\section{Introduction to Regression}

Supervised learning is the process whereby given examples of inputs and outputs
an algorithm is then given a new input and asked to predict its output.

Regression is this process specifically done on continuous inputs and outputs.

Linear regression is the attempt to model a linear relationship between a
dependent variable and independent variables. Such that given the dependent
variable $y$ and the independent variables $x_1, x_2, \ldots x_n$ that we then
generate a function $y = \theta_0 + \theta_1x_1 + \ldots + \theta_nx_n$

The $\theta$ values are referred to as weights and determine how important the
corresponding independent variable is.

Gradient descent is the method by which we handling linear regression, using the
sum of squared errors rather than absolute errors since sum-of-squared is
differentiable.

The aim of linear regression in this case is to minimize the amount of total
error across all of the provided input/output values.

There are various different error functions that can be used in order to find
the model with the least error. For gradient descent sum of squared error is the
best since its differentiable.

Generally we try to fit the function
$f(x) = c_0 + c_1x + c_2x^2 + \ldots + c_nx^n$ and by using a higher order
function we then have more degrees of freedom. We are limited so that the max
order $n$ is only $\mid X \mid$ where $X$ is the dataset of all training points.

Picking too high an order of freedom results in over fitting and the resultant
model fits the data but overcommits to the data.

Given the equation $Xw = Y$ where $X$ is the set of independent variables, $w$
is the vector of constants $c_0, c_1, \ldots c_n$ and $Y$ is the set of
dependent variables. We are then able to solve for the constant vector $w$ as
follows:
$$X^TXw  = X^TY$$
$$(X^TX)^{-1}X^TXw = (X^TX)^{-1}X^TY$$
$$w = (X^TX)^{-1}X^TY$$

This is using projections in linear algebra.

Training data has errors that is we model not $f$ but $f + \epsilon$. Errors can
result from a number of sources.

Assume that data is independent and identically distributed. This implies that
all the data comes from the same source and there are no implicit biases in
the data. This is not a fundamental assumption of supervised learning however,
it is an assumption of certain algorithms within supervised learning.

We can split the training set into a cross validation set (a stand-in for the
actual testing data). Cross validation splits data into $n$ folds. Then test on
each fold, training on the rest of the data. You then choose the degree model
with the lowest error and average the coefficients together.

Without enough complexity you underfit the data. Too much and you overfit the
data.

Vector inputs are possible and generalize to hyperplanes (that is we aren't
limited to a small number of axes).

We can use discrete, vector, or scalar data can be input into regression models
by using ordinal conversion function. \textbf{Note}: A conversion function can
be approximated by using another machine learning algorithm.

Boolean vectors can be used to encode complex data.

\section{More Regressions}

Parametric regression represents a model with a number of parameters (ie
regression).

K nearest neighbor finds the k neighbors closest to the input value and then
averages their output values together using a couple of different metrics.

Kernel regression is similar to KNN and each data point is weighted unlike KNN
where each point is simply left as they are.

Instance based methods keep the data and use it to generate a
prediction when provided with an unknown label.

The parametric model is biased (as in we already have an idea of the solution)
vs an instance method which is unbiased.

Parametric models are space efficient however cannot be updated easily. Training
is slow but querying is fast.

Instance based methods are not space efficient, training is fast but querying is
slow. Also have the added benefit of fitting patterns that are too complex for
parametric models.

\section{Regressions in sklearn}

Continuous supervised learning is regression.

Continuous learning works on data sets with some type of ordering.

Target variable is the dependent variable and input variables are the
independent variables.

Linear regression minimizes the sum of the squared errors. The best regression
is the model that minimizes the SSE metric.

Ordinary least squares (OLS) (used in sklearn) and gradient descent are the
two algorithms used to minimize SSE.

There is a fundamental ambiguity. There can be multiple lines that minimize the
$\sum \mid \textsf{error} \mid$ but only one line minimizes
$\sum \textsf{error}^2$.

Furthermore, using SSE makes the implementation easier as the function is
differentiable.

SSE does not account for the number of data points and thus as the data set
size grows the SSE grows as well. The SSE metric can not compare fits between
two different data sets.

$R^2$ explains how much of my change in the output is explained by my change
in the input and is bounded $0 < r^2 < 1.0$. $R^2$ is not generally affected
by the number of data points.

Regression requires data that fits its structures well. Data can be transformed
before hand as a fix.

Reinforcing the previous notes concerning regression vs classification note that
discrete labels and decision boundary indicate supervised classification method.
In the case of regression the object is have a continuous output and a line of
best fit. Classification uses accuracy as its measure where regression uses
sum of squared error.

Multivariate regression is regression on several independent variables reducing
to a single output variable.

\section{Decision Trees}

Classification maps a set of independent variables to a set of discrete labels.

Instances are inputs, vectors of independent values of input.

Concepts are represented by functions. A relationship between some input and
output. A concept is formally defined in this case as a set of things.

The target concept is the concept from the possible concept space that most
effectively describes the inputs in terms of the outputs by some metric.

The hypothesis class is the set of concepts that are possibly useful to use.

A sample is a training set of inputs (independent variables) and labels
(outputs).

Candidate is a concept that we evaluate to be an effective target concept.

The testing set is a set of inputs and outputs that is specifically used to
evaluate the competency of the candidate concept.

The representation of a decision tree is a n-ary tree. Where each set of
children the node represents a decision and the node represents a specific
selection against that decision. Decision trees only use data that is relevant,
it will ignore any data that is not required to model the data properly.

\textbf{Note}: Decision trees start from the root node.

Decision trees attempt to create the most equal splits possible when splitting
a decision tree. The deeper questions are predicated on the earlier
possibilities.

The algorithm for decision trees can be defined as follows:
\begin{enumerate}
        \item Pick the best attribute (gives the most even split).
        \item Asked the question
        \item Follow the answer path
        \item Until you get an answer go to 1
\end{enumerate}

A decision tree building algorithm follows all possible path until all possible
questions are answered. Whereas the above algorithm only defines how to use
a prebuilt decision tree.

A decision that keeps the data as it is, is less useful than any possible
decision split that splits the data in any capacity. A decision that doesn't
actually split the data is essentially a mapping of the current state of the
data to itself and simply wastes cpu time.

Decision trees can be considered a more general class of the behavior of a
transistor. It can model most logical operators. A decision tree can be
considered a form of logical primitive in this case.

Not all decision trees are commutative but they do exist.

The number of nodes in a decision tree represents how complex it is. The
trees for n-or and n-and are linear that is there are $n$ nodes that need
to be traversed.

The n-xor tree on the other hand, $2^n - 1$. The n-xor tree is exponential, and
is a NP hard problem.

Splitting on a choice we construct a decision table to represent all the
possibilities. The resulting space in which a decision tree can exist, of the
many possible choices we must consider is
$m_{output}^{\prod^{n_{decisions}}\mid k_n\mid}$ where $k$ the cardinality
of the decision outputs for a given decision.

An algorithm like that is very expressive, meaning that the hypothesis space is
very large. However, this means that the algorithm we use to search this space
needs to be extremely efficient in order to prevent nearly infinite runtimes.

\subsection{ID3}

The following algorithm is used to efficiently build decision trees.

We begin by looping forever until a solution is found
\begin{itemize}
    \item $A \leftarrow$ best attribute
    \item Assign $A$ as decision attribute for $\text{node}$
    \item For each value of $A$, create a decedent of $\text{node}$
    \item Sort training examples to leaves
    \item If examples are perfectly classified stop. Else iterate over the
        leaves
\end{itemize}

Information gain is a mathematical representation of the reduction in randomness
and is defined
$\text{Gain} (S,A) = \text{entropy}(S) - \sum_v \frac{S_v}{S}\text{entropy}(S)$.

Entropy is defined as a measure of impurity in a bunch of examples.

The formula for entropy is defined $-\sum_v p(v)\text{log}_2(p(v))$ where $p(v)$
is the probability of seeing a value. Entropy in machine learning refers to
the homogeneity of a dataset

The best attribute is the one that maximizes information gain.

ID3 terminates under two conditions: When there is no relevant data left to
split on (ie its perfectly classified) or when there are no more attributes
available to divide the data with.

The ID3 algorithm is recursively called against itself unless a terminating
condition is met.

Information gain is calculated and then sorted for each remaining attribute
during the process of building the decision tree.

ID3 is a greedy algorithm as it has no complex look ahead behavior and only
focuses on the immediately next set of nodes.

As the number of examples (the size of the training data) and/or the number of
attributes to analyze increases, the likelihood the tree returned by ID3 is
suboptimal.

With continuous data bucketing can be used to enable ID3 to use it.

If entries are missing from data there are several ways to fill the gaps.

\begin{itemize}
	\item You can replace the gap with the most common (in the data) element
		for that specific attribute.
	\item You can also assign probabilities to each possibility for the
		given attribute and then randomly assigning gaps using a
		pseudo random generator until no gaps remain.
\end{itemize}

Since ID3 fits until there is no fitting left to do, it can drastically overfit
the data. There are two major ways to deal with this:

\begin{enumerate}
	\item Stop growing the tree before it becomes too large
	\item Pruning the tree once it is done
\end{enumerate}

A limit to the growth can be established through a limit on the layers/depth.
Cross validation can also be used to help establish the best tree structure.

Pruning however requires testing the fully built tree against the pruned trees
and seeing if it the pruned tree performs better on the test data.

Finally we are able to manipulate the formula used for the information gain
analysis. If there are many buckets and not enough data we can wind up with
sparse separation which is difficult for the information gain formula to
evaluate.

The formula called Gain Ratio overcomes the aforementioned difficulty by
measuring how evenly and how broadly the data is distributed. It is expressed:
$\text{GainRatio}(S,A) = \frac{\text{Gain}(S,A)}{\text{SplitInformation}(S,A)}$
where $\text{SplitInformation}$ is defined
$\text{SplitInformation}(S,A) = - \sum_{i = 1}^c\frac{\mid S_i\mid}{\mid S\mid}
\text{log}_2\frac{\mid S_i\mid}{\mid S\mid}$

There are two major types of bias:

\begin{enumerate}
	\item Restriction bias: Refers to the hypothesis set (anything outside
	is not considered)
	\item Preference bias: What hypotheses in the hypothesis set are
	preferred
\end{enumerate}

ID3 prefers splits at the top, correct trees and trees that are shorter.

\subsection{Decision Trees Continued}

Use range buckets in order to represent bounded continuous attributes.

Using separate bucketing methods to extract discrete value attributes allows
continuous values to be reused in the same child-to-root path for a given
decision trees.

Fitting the training data perfectly is a non-starter since there is the
possibility of noise in the data.

A complex decision tree conflicts with occam's razor. Using cross validation it
is possible to get a more efficient tree by comparing different trees against
one another.

Besides hard limits to prevent a tree from growing to large, the information
gain function can add weights to cause it to prefer breadth first growth.

Information gain is not easily used on regression. So we need to change the
spitting criteria. It is possible to specify a local output function for each
leaf.

Another way to measure the decision when to stop is a voting function applied
over the leaf nodes.

\section {More Decision Trees}

Linearly separable data is data that can be partitioned by a $y = m_0x_0 +
\ldots + m_nx_n + b)$.

Decision trees allow multiple yes/no questions (binary decision tree).

Analogue: Decision tree is a 8bit coast building algorithm.

In sklearn, the default split criterion is the Gini index.

Decision trees are easy to use and easy to grow. Easy to display.
Bigger classifiers can be built using decision trees using ensemble methods.

Decision trees are prone to overfitting with lots of data and be careful
with parameter tunes. Outliers can drastically affect the growth of the tree
if the tune does not modify the minimum split criteria.

\section{Neural Networks}
Via approximation, artificial neural networks are cartoonish in comparison to
actual neurons.

An artificial neuron has a set of inputs $X$ with a set of weights $W$ to the
threshold of the neuron $\theta$ and an output $y$.

The activation is defined as $\sum_{i = 1}^k x_i \times w_i$. If the activation
is equal to or exceeds the threshold than the neuron outputs.

The artificial neuron is referred to as a perceptron and is considered the most
basic form on an artificial neuron.

Perceptrons compute a half plane, since the activation function is linear, the
decision boundary that they compute is always linear.
 can express basic logical constructs (or, and, not) etc with a
single perceptron unit.

When using a bounded evaluation structure and an unbounded (continuous) internal
representation there is often an infinite number of viable expressions that
can represent the bounded evaluation metric. That is AND/OR/NOT/XOR are easy
to express with a variety of different perceptron formulations. Note this does
not imply that all values are possible to represent all solutions, there are
just an infinite number of bounded solutions.

There are two different rules are
\begin{itemize}
	\item Perceptron (thresholded output)
	\item Gradient descent/delta (not thresholded)
\end{itemize} Defining the perceptron rule we begin with the input labels $X$, the output $y$, the evaluated output $\hat{y}$, the learning rate $N$ and the weights $W$.

When doing calculations using the perceptron we have the following expression:
$\hat{y} = \sum_i w_ix_i \geq \theta$ we can modify this function to
$\hat{y} = \sum_i w_ix_i \geq \theta > 0$. This enables us to calculate the
threshold as a weight, improving calculation times. In order to take advantage
of this we add a single column of '1's to the input labels $X$ in order to
account for this.

We perform several iteration steps of the perceptron rule function, by which
we modify the weights. The weight changing function is defined $w_i = w_i + N
(y - (\sum_i w_ix_i \geq 0))X_i$. The inequality in the middle assumes a binary
yes/no output structure.

If there is a half-plane that separates the data, then the data is linearly
separable and then the perceptron rule can find the answer in a finite number
of iterations.

Only run the perceptron rule while there is some error.

An algorithm that is not solely focused on linearly separable data would be
gradient descent. Gradient works by defining an error metric over the data set.
The error function is defined as $E(x) = \frac{1}{2}\sum_{(x,y) \in D} (y -
(\sum_i x_iw_i))^2$. We then simply minimize the error by finding the derivative
and then solving for the minimum. We finally wind up the equation:
$\sum_{(x,y)\in D}(y - (\sum_i x_iw_i)(-x_i))$ The weight varying function for
gradient descent is defined as $\delta w_i = N(y - a)x_i$.

Perceptron rule has a guarantee of finite convergence in the case of linearly
separability.

Gradient descent is more robust (data doesn't need to be linearly separable) but
as a trade-off only converges to a local optimum.

We use the perceptron rule for cases where the data thresholding function is
non-differentiable.

The sigmoid function acts like the perceptron function however it is
differentiable. The sigmoid function is defined as $\sigma(a) =
\frac{1}{1 + e^{-a}}$. The derivative of the $\sigma$ function is written
$D\sigma(a) = \sigma(a)(1 - \sigma(a))$.

The sigmoid function is useful in allowing gradient descent to be used
universally, where previously the perceptron function itself could only be used.

Neural networks are layers of nodes that compute the sigmoid weights of the
previous layers. There are hidden layers (not available to IO) that are used
for larger representations.  The mapping from input to output using a sigmoid
ann is entirely differentiable.

Back propagation arises from the computationally beneficial organization of the
chain rule. That is data flows forwards while error flows backwards. Also known
as error back propagation.

Back propagation occurs so long as the error function is differentiable.

Sigmoid units are analogous to a perceptron but have many local optima.

There are other advanced methods of optimization (learning) that can be used:
\begin{itemize}
\item Momentum: pushes the ball so it doesn't necessarily settle in local
	optimum
\item Higher order derivatives: combinations of weights, hamiltonians etc
\item Randomized optimization
\item Penalties for complexity, prevents overfitting (less nodes, less layers,
	large numbers)
\end{itemize}

Large numbers affect network complexity (increase the complexity and chance of
overfitting).

\begin{itemize}
	\item Boolean functions can be represented by a network of
		threshold-like units.
	\item Continuous functions can be represented by a single hidden layer.
	\item Arbitrary functions can be represented by a double hidden layer,
		stitched together.
\end{itemize}

There is a danger of overfitting due to the capability of networks to represent
complex data. Cross validation can be used to determine the number of hidden
layers, number of nodes in each layer and when to cap the size of weights.

The initial weights for a neural network, typically small random values are
chosen (different random values for different runs).

Neural networks prefer simple explanations, occam's razor as well as correct
values.  Occam's razor: entities should not be multiplied unnecessarily.

\section{Support Vector Machines}

Fits that are too close to the data overfit the data. An svm finds a line
that fits the data but is least committed to it.

We consider the generalized linear function $y = w^tx + b$. Where $y$ is the
labels, $x$ is the input, $w$ and $b$ are the parameters of the splitting
hyperplane.

We want the plane that correctly splits the data while being as far away from it
as possible.

The equation for the splitting hyperplane is specifically formulated as
$w^tx + b = 0$.

The output labels $y \in {-1, 1}$, and we can then say the lines tangential to
the positive and negative sets can be defined as $w^tx + b = 1$ and
$w^tx + b = -1$. When then maximize the distance between these two lines. This
distance is defined as $\frac{2}{\vert\vert w\vert\vert}$

Goal of support vector machines is to maximize the margin such that the margin
$m$ is defined as $\frac{w^T}{\vert\vert w\vert\vert}(x_1 - x_2)$. However we
need to correctly classify everything which is represented by the margin $m$
being constrained such that $m = \frac{2}{\vert\vert w\vert\vert}$.

In order to classify everything correctly we have consider the inequality
$y_i(w^Tx_i + b) \geq 1$ considering $ \forall_i$. That inequality is used
because correctly predicted non-members multiple double negatives and invert the
sign of the predicted value. However maximizing $\frac{2}{\vert\vert w\vert\vert
}$ is difficult so instead we minimize $\frac{1}{2}\vert\vert w\vert\vert^2$.
The form is easier to solve as it is a quadratic programming problem. We then
formalize the problem in the quadratic normal form $W(\alpha) = \sum_i \alpha_i
 - \frac{1}{2}\sum_{ij}\alpha_i \alpha_j y_i y_j x_i^T x_j$. This is done under
 the constraints that $\alpha_i > 0$ and $\sum_i \alpha_i y_i = 0$ and try to
 maximize the $W(\alpha)$.

 Retrieving the initial values we have $w = \sum_i \alpha_i y_i x_i$. Most
 $\alpha$ are zero. Most data doesn't matter when considering how to split the
 data. The data that is used to actually define the support vector machine are
 considered the support vectors.

The points that are close to the decision boundary are important while the ones
that are farther from the decision boundary are not considered important. Its an
example of instance based learning that gets rid of the unnecessary points. The
dot product can be considered a measure of similarity.

Data sets that are linearly married imply that a support vector machine cannot
separate the data properly as any line chosen still results in errors.  We
use a kernel function $\Phi$. The formula $\Phi$ is used because when applying
it in support vector machines and apply the dot product we can transform
the data so that we can still apply the SVM.

There is some higher dimensional space that is equivalent to the current one,
via the kernel. However that kernel function may need to transform an
unbounded number of equations.

The general form of the SVM is $W(\alpha) = \sum \alpha_i - \frac{1}{2}
\sum_{i,j} \alpha_i \alpha_j y_i y_j K(x_i, x_j)$ where $K(x_i, x_j)$ is the
kernel where we inject domain knowledge into the SVM. A common kernel form is
$(x^Ty + c)^p$. A good kernel function captures domain knowledge properly. The
kernel function measures the similarity of two items in a specific domain.

Kernel functions should explain the Mercer condition: it acts as a well behaved
distance function.

Margins are useful in determining how generalization and overfitting happen.

\section{SVMs in practice}
SVMs split data of two classes.

Margin is the distance between the line and the points that is maximized by the
SVM.

SVM lines that are too close to the data are more susceptible to noise

An svm prioritizes a correct separation of the data over the maximization of the
margin.

SVMs are capable of tolerating small/individual outliers.

The $C$ parameter controls the tradeoff between a smooth decision boundary and
classifying the training data points correctly. A higher $C$ implies more
training points will be correct. That is we fit tightly to the data.

The $\gamma$ defines how far the influence of a single training example reaches.
The higher $\gamma$ the closer the decision boundary fits to the data.

SVM work well on complicated domains.

Doesn't work well on large data sets ($O(n^3)$)

Does not handle lots of noise well, overfits to the noise.

\section{Instance based learning}
Instance based learning remembers the data, its very fast to insert new data and
its generally simpler.

Distance between training examples is important. There are multiple metrics
to measure distance in a traditional sense, let alone for more abstract
classifications. Where distance and similarity are analogues for one another.

\subsection{K-NN}
Given the training data $D = {x_i, y_i}$, a distance metric (domain knowledge)
$d(q,x)$ and the number of neighbors $k$ (also considered domain knowledge).
Finally we are also given the query point $q$.

We define the nearest neighbor set $NN$ as the set ${i: d(q, x_i) \text{min} k}$
. Where we then return the mode/plurality of the $NN$ in the case of
classification and in the case of regression we take the mean.

\subsection{Use of KNN}
Ties in classification are handled by a designer metric.

Most of the features of the KNN algorithm are defined by the designer which
allows the algorithm to be apparently simple in its structure.

Instance based learning is slower to query then learn while other methods are
slow to train and quick to query. Lazy vs eager learners.

The choice of distance metric and k drastically affect the performance of the
KNN.

KNN assumes near points are similar to one another, the distance function
encodes that assumption. There is always one best distance function. KNN by
averaging things together, expects functions to behave smoothly. Dependent on
the distance function, most KNN metrics assume most features matter equally,
there is no explicit feature preferencing.

KNN like most other algorithms suffers from the curse of dimensionality:
as feature/dimensions grow the data required to generalize increases
exponentially.

The choice of distance function is integral to the performance of KNN.

Weighting  in the distance function can be used to help with the curse of
dimensionality.

Setting $k=n$ and using a weighted average we wind up using a specific
technique called locally weighted regression (which regressed by averaging the
local data points more heavily than others).

Similarity is just another way of capturing domain knowledge.

Instance based method of using linear regressions which allow us to compose
several learning methods together. (The KNN averaging function and distance
functions can be used to encode other algorithms).

\section{Naive Bayes}

The decision boundary is a separator of two or more classifiers.

Save about $10\%$ of data for testing.

$p(E_2 \vert E_1)$ is the sensitivity and $p(\neg E_2 \vert \neg E_1)$ having it is the specificity.

Prior probability and test evidence results in the posterior probability.
$P(E_1\vert E_2) = P(E_1) P(E_2 \vert E_1)$ and $P(\neg E_1 \vert E_2) =
P(\neg E_1) P(E_2 \vert \neg E_1)$.

We then normalize each item by $P(E_1\vert E_2) + P(\neg E_1 \vert E_2)$

Naive Bayes is naive because it ignores greater context.

\section{Bayesian Learning}

Basics of machine learning are to learn the most probable hypothesis given some
data and some domain knowledge. In this case we equate best to likelihood. This
can be represented as $\text{argmax}_{h \in H} P(h\vert D)$ where $h$ is a
particular hypothesis and $D$ refers to a set of data.

Formally speaking Bayes Rule is written $P(h\vert D) = \frac{P(D\vert h) P(h)}
{P(D)}$

We can derive Bayes' Rule from the standard chain rule in probability:
$\Pr(a,b) = \Pr(a \vert b)\Pr(b) = \Pr(b \vert a)\Pr(a)$

In Bayes' rule $\Pr(D)$ refers to your prior belief about the data.
$\Pr(D \vert h)$ refers to the data given the hypothesis and refers to the
actual running the hypothesis. $\Pr(h)$ refers to out prior belief refers to our
belief in a certain hypothesis over any others, this generally refers to domain
knowledge. This implies better domain knowledge or higher accuracy $\Pr(h)$ and
$\Pr(D\vert h)$ improve our overall probability.

If a test is not useful, then via Bayesian reasoning it makes sense to alter
the population you perform the test on.
The algorithm for Bayesian Learning we for each $h \in H$, calculate $\Pr(h\vert D) = \frac{\Pr(D \vert h)\Pr(h)}{\Pr(D)}$ and then find the maximal
probability in the set of outputs and choose that as out $h$. That is $h =
\text{argmax}_h \Pr(h \vert D)$.  Its not necessary to find $P(D)$ since we only
need to find the $\text{argmax}$.

The Maximum a posteriori or MAP is defined $\Pr(h \vert D) \approx
\Pr(D \vert h) \Pr(h)$.

It is often difficult to derive any useful domain knowledge $\Pr(h)$ we then
reduce the computed hypothesis to the maximum likelihood hypothesis which is
defined as $h_{ml} = \text{argmax}_h \Pr(D \vert h))$ since we assume all
hypotheses are equally likely.

Difficult to use the most likely hypothesis model directly since hypothesis
space may be infinite. Maximum likelihood can be used as a gold standard.

Given a set of data ${<x_i, d_i>}$ as a set of noise free
examples of a hypothesis $c$, that $c \in H$ and there is a uniform prior
aka an uninformed prior. We then define $\Pr(h) = \frac{1}{\vert H \vert}$.
We also have $\Pr(D \vert h) = 1 \text{ s.t. } \forall d_i \in D d_i = c_i)$. And
finally we have $\Pr(D) = \sum_{h_i \in H} P(D \vert h_i)\Pr(h_i)$ assuming that
all hypotheses are mutually exclusive. Deriving further we only select the
hypotheses that are consistent with the data (that is all points agree)
since otherwise their probability is 0. So we get $\Pr(D) = \sum_{h_i \in
VS_{H,D}} 1 * \frac{1}{\vert H \vert} = \frac{\vert VS \vert}{\vert H \vert}$
Finally we have $\Pr(h\vert H) = \frac{1}{\vert VS \vert}$
This states that all hypotheses in the version space are all equally as usable,
however this depends on the correct hypothesis being in the hypothesis space,
noise free data and a uniform prior.

A derivation for Bayesian learning with noise follows: Given a set of inputs
and labels $<x_i, d_i>$. The function that we are trying to analyze $f$ is
defined such that $d_i = f(x_i) + \epsilon_i$. The error $\epsilon$ is drawn
from a normal distribution $N$ with a $0$ mean and a variance $\sigma^2$ whose
relevance is minimal. Each epsilon is independent and identically distributed.

We begin with $h_{ML} = \text{argmax} \Pr(D \vert h)$ and then via the chain
rule have $\text{argmax} \prod_i \Pr(d_i \vert h)$. Then since each
$\epsilon_i$  is distributed over a normal distribution and is iid, then we can
express the function as a product of the likelihood of any specific errors
occurring in the form of a Gaussian $$\text{argmax} \prod_i \frac{1}
{\sqrt{2\pi\sigma^2}} e^{-\frac{(d_i - f(x_i))^2}{2\sigma^2}}$$ Since we are
using argmax we can remove any constant expressions so using $\text{log}$ is
monotonic then we can drop the exponential and the constant term $\frac{1}
{\sqrt{2\pi\sigma^2}}$ can be dropped. We then have $\text{argmax} \sum_i
\frac{-(d_i - f(x_i))^2}{2\sigma^2}$. Then minimizing instead of maximizing
and throwing out constants we get $\text{argmin} sum_i (d_i - h(x_i))^2$ and
this function is represented by the sum of squared errors.

Several machine learning algorithms 'pop' out of the Bayesian derivation, and
our assumptions. We actually as a result assume Gaussian noise corrupting all
label data but not corrupting all input.

A derivation of the minimum description length follows: Given $h_{MAP}$ such
that $h_{MAP} = \text{argmax} \Pr(D\vert h) \Pr(h)$ we then apply $\text{log}_2$
to get $\text{argmax} \text{ lg}\Pr(D \vert h) + \text{lg}\Pr(h)$ and then
multiply by $-1$ to get $\text{argmin }[- \text{lg}\Pr(D\vert h) - \text{lg}\Pr
(h)]$. Then finally from information theory we have the optimal code for event
$w$ with probability $p$ has a length $-\text{lg }p$. We then have $
\text{length}(D \vert h) + \text{length}(h)$. Reduced we want to minimize
$\text{error} + \text{size}$. This shows that we desire shorter
and correct descriptions. In essence it is a Bayesian representation of Occam's
razor.

Sadly the tradeoff between misclassification error and the size of the
hypothesis is built in and can't be entirely overcome.

However, Bayesian learning is focused on finding the best $h$ not the best label
$d$ as a result we need to actually calculate $d$ which is done by a weighted
vote where for $h \in H$ the weight of the vote is $\Pr(h \vert D)$. This
is represented by the equation $$d_{MAP} = \text{argmax}_d \sum_h \Pr(d
\vert h)\Pr(h \vert D)$$ This is known as the Bayes Optimal Classifier. There
is no way to do any better on average than the BOC.

\section{Bayesian Inference}
Bayesian networks are good at representing joint probabilities over complex
spaces.

A joint distribution is a distribution of combined factors with a total
probability of one.

We use conditional probability and independence  in order to prevent the
exponentiation of the factors every time we add parameters.

Conditional independence is defined as: $X$ is conditionally independent of
$Y$ given $Z$ if the probability distribution governing $X$ is independent of
the value of $Y$ given the value of $Z$ or formally $\forall X,Y,Z
\Pr(X = x \vert Y = y, Z = z) = \Pr(X = x\vert Z = z)$ or more compactly
$\Pr(X \vert Y, Z) = \Pr(X \vert Z)$. The independence of two independent
variables means their joint probabilities is the product of their margins. It
is not possible to necessarily possible to be as strong as independent
probability. However, conditional independence can be used more generally.

Belief networks are also know as Bayes networks or graphical models. The nodes
correspond to the variables and edges correspond to dependencies.

The more influences on a node the number of nodes in the probability table to
calculate the resultant probability table grows exponentially. That is the
cardinality of the probability table $\vert P(E) \vert = 2^k$ where $k$ is the
number of influential nodes on event $E$. This deals with the indegree of the
belief network structure.

Belief network shows relationship between probabilities. Are not able to infer
causality between. Merely talking about statistical dependence.

Sampling from a joint distribution is done in topological order, however this
depends on the graph must be acyclic.

We depend on Bayes belief networks being acyclic so that we can properly
compute joint probability distributions over them.

Cyclic graphs make everything more complicated and can result in complicated
predicating and constraints on the conditional probability.

We are able to use the joint distribution to compute any joint subgraph. Given
a joint distribution where there exist nodes for conditional evens $a \in A$ we
we compute the subgraph via the fact that each event $a$ and its probability
table $P(a)$ is written such that if that event $a_i$ is conditionally
independent predicated on events $a_j, a_k, \ldots a_n$ these can be considered
a subset $A_i$ then we write the probability table for $a_i$ as $\Pr(a_i \vert
A_i)$ and then the joint probability distribution of any subgraph $A_s$ can
be expressed as $\prod_i \Pr(a_i\vert A_i)$ where $\forall a_i, A_i$ that
$a_i \in A_s$ and $A_i \subset A_s$

The number of boolean variables required to represent the subgraph is normally
$2^{\vert A_s\vert}$ but using a joint probability distribution where the
elements are conditionally independent we gain the benefit of expressing it with
fewer values and the minimal expression can be done via $\sum_i 1 + \vert A_i
\vert$. If all events are independent than the minimal expression possible is
$\vert A_s \vert$.

Distributions are useful for determining the probability of a certain value
and also generating values. If a distribution represents a process, it allows
us to approximately inference and to model that process and to simulate it. We
are able to visualize the distribution and to get a feeling for the behavior
of the distribution. We use approximation since exact is hard and/or
approximation is faster. Solving arbitrarily complicated belief networks is
NP hard. (Due to the exponential explosion of complicated conditional
independent events).

There are some basic probabilistic inference rules such as marginalization
where the formula $\Pr(x \vert = \sum_y \Pr(x,y))$ and the chain rule which is
$\Pr(X,Y) = P(X)P(Y \vert X)$ and then we have Bayes Rule.

If you know your in a specific context there are features that are common and
their are probabilities of those features occurring given the context.

The probability of finding the probability of a root node given a number of
attributes. Then we have the formula $\Pr(V \vert a_1, a_2, \ldots a_n) =
\prod_i \frac{\Pr(a_i \vert V) P(V)}{Z}$ where $Z$ is the normalization factor
across all of the attributes. This type of formalization allows for the use
of attributes to determine the general class of something. This classification
formula is $\text{MAP class} = \text{argmax } \Pr(V)\prod \Pr(a_i \vert V)$

Inference is cheap with naive Bayes, there is a linear number of parameters.
We can estimate parameters with labeled data (count the number of occurrences in
that class over the number of items in that class). Naive Bayes connects
inference and classification and Naive Bayes is empirically successful.

Naive Bayes assumes conditionally independent features if you know the class. No
inner relationships among the attributes are not considered.

Its OK if the probabilities are wrong so long as the answer is in the right
direction ('OK in practice validation').

Ordering is preserved, Naive Bayes believes its answers too much. One unseen
attribute spoils the entire product.  A way to avoid the full veto is to
initialize the values to small probabilities (this is considered overfitting).
By smoothing the data we have an inductive bias that everything is possible.

Naive Bayes allows us to link Bayesian Probability and classification. It
handles missing attributes well, it allows inference in any direction.

\section{Ensemble B\&B}
Using a number of simple rules can help to determine but collectively cannot
represent eh whole corpus of rules required to fully represent a concept or
classification.

Ensemble learning combines simple rules into a complex rule. Subsets of the data
are used to generate these simple rules. These rules are then combined.

Simplest method for subsets is uniformly and randomly picking data. Simplest
method is combining is mean/voting.

Ensemble learning gets the benefits of cross validation, that is we don't fit
the training data as closely and generalize better, getting rid of some
overfitting.

Choosing random subsets to apply learners to is known as bagging or bootstrap
aggregation.

Subsets should be chosen iteratively by creating subsets based on those with the
highest error for the current/previous learners and then they are weighted for
the final computation.

We define error in boosting as $\Pr_D[h(x) \neq c(x)]$.

A learning algorithm is weak when no matter the distribution, it will do better
than chance. Formally speaking this implies $\forall D \Pr_D[\cdot] \leq
\frac{1}{2} - \epsilon$. This implies the learner always learns something from
the data.

However if the distribution space has a potential weighting situation that
prevents some hypothesis from satisfying the definition, then we have no weak
learner. Small hypothesis spaces with bad hypotheses make having a weak learner
difficult.

\subsection{Boosting Algorithm}
Assume a classification circumstance

Given training data $\{(x_i, y_i)\}$ where $y \in \{-1, 1\}$

For $t = 1 \text{ to } T$ we:
\begin{itemize}
	\item Construct $D_T$
	\item Find a weak classifier with $h_t(x)$ with a small error
		$\epsilon_t = \Pr_{D_t} [h_t(x) = y_i]$
\end{itemize}

Then output $H_{final}$
The $\epsilon$ is bounded such that $\epsilon < \frac{1}{2}$.

To find the distribution the first distribution $D_1(i) = \frac{1}{n}$
At every time step we have distribution $D_{t + 1}(i) = \frac{D_{t}(i) \cdot
e^{-\alpha_ty_uh_t(x_i)}}{Z_t}$ where $\alpha_t = \frac{1}{2}\text{ln}\frac
{1 -\epsilon_t}{\epsilon_t}$. This utilizes the fact that the algorithm is
working for a classifier (if the predicted and actual label have the same sign
the probability decreases depending on the normalization constant.

$Z_t$ stands for whatever normalization constant is required to make the
distribution work.

Then we get the weighted average $H_{final} = \text{sign}(\sum_t \alpha_th_t
(x))$.

\subsection{Boosting Continued}
A feature of ensemble methods is that $H \leq \{\sum h \in H\}$ whereby we are
referring to the complexity of the hypothesis classes and therefore can be
more expressive.

Confidence refers to how strongly you believe in a specific hypothesis.

Variance can be used as a stand-in for confidence (low variance = high
confidence).

Using a modification to the weighting formula normalizing it, we can obtain a
measure of confidence. That is $H_{final}(x) = \text{sign}(\frac{\sum_t
\alpha_t h_t(x)}{\sum_t \alpha_t}$. The closer the value is to the maximal $-1$
or $1$ the more confident the ensemble method is in that value.

Essentially as iterative steps are done, error stays the same but confidence
increases. Boosting increases the margin between the confidence and larger
margins counter overfitting.

Weak learners tend to overfit in the circumstances where the underlying learners
overfit. Pink noise also causes boosting to overfit. Pink noise is uniform
noise.

White noise is Gaussian noise.

There is no direct definition of 'strong learner'.
\nobibliography{../../citations}
\end{document}