\lab{Nearest Neighbor Search}{Nearest Neighbor Search}
\label{lab:NNS}

\objective{Teach about branch and bound and the curse of dimensionality using the nearest neighbor search problem.}

\section*{The Nearest Neighbor Search Problem}

You move into a city that has several post offices.
You want to know which one is the closest.
This problem is  known as the nearest neighbor search problem or the post-office problem.
The general problem is to find the closest of a set of points to any new point.

This has many applications which include computer vision, pattern recognition, internet marketing and data compression.

The naive way to solve this problem is to check the distance of all the data against the point.

\begin{problem}
Write a function that solves the nearest neighbor search problem by exhaustively checking all the distances.
The function should take in the set of points that is the data and a single point.
The output should be the distance to the closest data point and the index of that point.
Your function should be able to take in data in an arbitrary dimension.
\end{problem}

The complexity of this algorithm is $O(kn)$.
Where  is the $k$ number of dimensions and $n$ is the number of data points.

\section*{K-D Trees}

A faster way to solve this problem is to build a k-d tree and search the k-d tree for the nearest neighbor.

A k-d tree is a binary tree where the nodes to the left of parent node have a lower value in the i-th dimension and the nodes to the right of the parent node have a greater value in the i-th dimension.
Which dimension you split the nodes alternates at different levels.
In the $3$ dimensional case the root node is divided in the $x$ dimension, children in the $y$ dimension, grandchildren in the $z$ dimension, and the great-grandchildren in the $x$ dimension and so on.
Each node stores its location, left child and right child.
This requires sorting at each level so the complexity is $O(n log^2(n))$, but we only need to build the k-d tree once and after that we can query it as many times as we want.

Included is a function that takes in a set of data and builds k-d tree.
The leaf nodes' children are  python's ``None" object.

The search of the tree is done recursively.
Our search function will accept a parent node on which to search, a search point to which we want to find the nearest neighbor, the current best point, the current best distance, and a dimension $i$.
The search function should return the best point and best distance currently held in the algorithm.
We start the algorithm on the root node of the tree using it's point and distance to the search point as the best point and best distance, and start with $0$ as the first dimension.

For each recursive step, we first check if the euclidean distance between the parent node's point to the search point is less than the current best.
If so, we update it the best distance and best point to match the parent node.

We then compare the values in the $i$-th dimension of the search point and the parent's point.
If the search point's value is less than that of the parent node's point we recursively call the search function again on the left child of the parent node on the dimension $i+1$, flipping the dimension to $0$ if necessary, and set best point and best distance to it's output.
Then we have to check if the hypersphere around the point with radius being the current best distance crosses the dividing hyperplane created by the parent.
In code we accomplish this by adding the best distance to the point's value in the $i$-th dimension.
If this sum is greater than the parent's point value in the i-th dimension, we then search additionally the right child of the parent (using the dimension $i+1$) and update the best point and best distance to the output.

If the point's value was less than the parent's in th i-th dimension, we apply the last two steps mirrored to the right and left children.
We first search the right child with the $i+1$-th dimension and set the best point and best distance to the output.
We then check if the search point's value in the $i$-th dimension minus the best distance is less than the parent node's value in the $i$-th dimension.
If it is, we will also search the left child one the $i+1$-th dimension and set the best point and best distance to the output.

Below the algorithm summarized in pseudo code

\begin{algorithm}
\begin{algorithmic}[1]
\Procedure{KDSearch}{search\_point,parent\_node,b\_point,b\_distance,i}
    \If { Distance(search\_point,parent\_node.point) $<$ b\_distance }
        \State b\_point = parent\_node.point
        \State b\_distance = Distance from search\_point to parent\_node.point
    \EndIf

    \If{search\_point[i] $<$ parent\_node.point[i]}
        \State b\_point, b\_distance =
            \State KDSearch(search\_point,parent\_node.left\_child,b\_point,b\_distance,i+1)
        \If { search\_point[i] + b\_distance $geq$ parent\_node.point}
            \State b\_point, b\_distance =
                \State KDSearch(search\_point,parent\_node.right\_child,b\_point,b\_distance,i+1)
        \EndIf
    \Else
        \State b\_point, b\_distance =
            \State KDSearch(search\_point,parent\_node.right\_child,b\_point,b\_distance,i+1)
        \If {search\_point[i] + b\_distance $leq$ parent\_node.point}
            \State b\_point, b\_distance =
                \State KDSearch(search\_point,parent\_node.left\_child,b\_point,b\_distance,i+1)
        \EndIf
    \EndIf
\EndProcedure
\end{algorithmic}
\caption{Nearest Neighbor}
\label{alg:nearestneighbor}
\end{algorithm}

\begin{problem}
Write a function that solves the nearest neighbor search problem by searching through a k-d tree.
The function should take in k-d tree and a single point.
The output should be the distance to the closest data point and the coordinates of that point.
\end{problem}

\begin{problem}
Time both the functions you have created with the number of data points being $10,000-100,000$ every multiple of $10,000$ with $4$ dimensions.
Time only the searching of the k-d tree not the building of it.
Plot both times on the same plot.
How do the two algorithms compare?

\begin{figure}[H]
\includegraphics[width=\textwidth]{fourDTime.pdf}
\caption{Your graph should look like this.
The green line is the naive version and blue line is using a kd-tree.
As you can see the kd-tree is much faster.}
\label{fig:fourDTime}
\end{figure}
\end{problem}

The complexity of this algorithm is $O(klog(n))$ in optimal time.
Its worst case is $O(k*n^{1-\frac{1}{k}})$ where $k$ is the number of dimensions and $n$ is the number of points in the tree.
The reasons for this are discussed in the next section.

\section*{Curse of Dimensionality}

As you increase the number of dimensions the number of times that you have to go down both branches increases.
You get to the point where you eliminate very few points by using a k-d tree.

\begin{problem}
Time both algorithms for the number of data points being $10,000-100,000$ every multiple of $10,000$ with $20$ dimensions.
Plot both times on the same plot. Now how do the two algorithms compare?
\end{problem}

\begin{figure}[H]
\includegraphics[width=\textwidth]{twentyDTime.pdf}
\caption{
Your graph should look like this.
The green line is the naive version and blue line is using a kd-tree.
As you can see the kd-tree is slightly slower than the naive version.}
\label{fig:twentyDTime}
\end{figure}

\begin{problem}
Time the SciPy built in function for searching a k-d tree (do not time the building of the kd-tree) for the number of dimensions points being $2-50$ with $20,000$ data points.
Plot the time.
What do you notice?
\li{from scipy.spatial import KDTree} will import the built in k-d tree.
Create the tree by \li{tree = KDTree(data)} and search it by \li{tree.query(point)}.

\li{scipy.spatial} includes a \li{cKDTree} object which is implemented in C.
Is it any better?

\begin{figure}[H]
\includegraphics[width=\textwidth]{curseD.pdf}
\caption{
Your graph should look similar to this.
Around 15 dimensions the time jumps.
At that point using a kd-tree is not more effective than checking all the combinations.}
\end{figure}
\end{problem}

\section*{Classification}
A common problem is correctly classifying data.
Suppose that you have a ten marbles, five of which are blue with a green stripe and the other five red with a purple stripe.
If your friend gave you an eleventh marble that was blue with a brown stripe, which group would you put it in?
Probably you would include it with the other blue marbles.
What if the marble that your friend gave you was blue with a purple stripe?
This marble shares characteristics with both groups.
Where you end up grouping it will depend on which characteristics are most important to you.

This is the intuitive classfication problem.
If we have data that is already grouped into distinct sets, into which set do we put new data?
Classification has myriad and sundry applications.
In this lab we will use it in the context of optical character recognition.

\section*{Nearest Neighbor Classification}
We will now more formally describe the classification problem and explain the nearest neighbor classification algorithm.
Suppose we have a collection of vectors $\{x_1, ..., x_m\}$ in $\R^n$ with corresponding labels $\{l_1, ..., l_k\}$ describing to which group each datum belongs.
This collection of vectors and labels is called our training set.
Each entry of a vector is called a feature and $n$ is the size of our feature set.
For example, consider the following vectors in $\R^3$

\begin{center}
\begin{tabular}{cc}
$(2,0,0)$ & $1$ \\
$(3,0,0)$ & $1$ \\
$(0,3,0)$ & $2$ \\
$(0,2,0)$ & $2$ \\
$(1/10,2,0)$ & $2$ \\
$(0,0,4)$ & $3$ \\
$(0,0,7)$ & $3$ \\
\end{tabular}
\end{center}

If we also have a metric on our space, we may determine the distance between all of these points.
If we are given a new datum and we wish to decide which of the three groups to include it in, one option is to choose the group to which its closest neighbor belongs.
Let us use the euclidean metric to classify $(0,0,5)$ against our training set.
We see that the distance from $(0,0,5)$ and $(0,0,4)$ is only $1$, while the distance to the remaining points is at least $2$.
The label of $(0,0,4)$ is $3$, and so we assign $(0,0,5)$ the same label.

Now, what if we wished to classify $(5/2,5/2,0)$?
This presents a problem since this point is equidistant from the points in label $1$ and label $2$.
In such a case it is up to the programmer to decide how to break the tie.

One case we need to consider is when the data that we are using in our search are on different scales.
For example, suppose we wished to classify people applying for a loan at a bank as `risky' or `safe.'
Further suppose that we know their age, education level, current debt, and how many credit cards they have.
We could encode their education level as integers between $0$ and $4$.

Suppose that Bob is 30 years old, well educated, has a debt of \$5000, and 3 credit cards.
Let's say that James is 18 years old, barely out of high school, \$5000 dollars in debt, and owns 4 credit cards.
If we were to use the Euclidean metric $d$ to measure how close Bob is to James, we would get a distance of
\[
d((30,4,5000,3),(18,0,5000,4)) = 12^2 + 4^2 + 1^2 = 161.
\]

However, if Alice is 29 years old, well educated, has a debt of \$4900, and 3 credit cards, then her distance from Bob is
\[
d((30,4,5000,3),(29,4,4900,3)) = 1^2 + 100^2 = 100001.
\]

Is James closer to Bob than Alice?  Most Banks would say no.
In order to classify these individuals better, we need to measure distance differently.
There are two different ways to handle this.
The first solution is to scale the data before inputting it into our algorithm.
In our previous example we could divide age by 100, education by 4, debt by 1000, and credit cards by 10 and then preform a nearest neighbor search on Bob.
Then the distance from Bob to James is
\[
d\left( \left( \frac{30}{100},\frac{4}{4},\frac{5000}{1000},\frac{3}{10} \right),\left(\frac{18}{100},\frac{0}{4},\frac{5000}{1000},\frac{4}{10}\right)\right) = .12^2 + 1^2 + .1^2 = 1.0244.
\]
while the distace from Alice to Bob is
\[
d\left(\left(\frac{30}{100},\frac{4}{4},\frac{5000}{1000},\frac{3}{10}\right),\left(\frac{29}{100},\frac{4}{4},\frac{4900}{1000},\frac{3}{10}\right)\right) = .1^2 + .01^2 = 0.0101.
\]
So now Bob is closer to Alice.

\begin{problem}
Write a function that takes in data points and the a vector that is used to be the scale.
The function outputs the scaled data points.
So if we were using it for our baking example it would take in the array representing the data Alice, Bob and James (where each person is a row) and the vector $[100,4,1000,10]$ and output the scaled data of Alice, Bob and James.
\end{problem}

Another alternative is to change the metric we use to measure distance with.
In both the examples above we used the Euclidean metric to measure distance.
Suppose we wished to apply the Euclidean metric to data set of colors.
We could map each color to an integer, say red to 1, violet to 2, blue to 3, green to 4, yellow to 5, and orange to 6.
If we use Euclidean metric green is closer to red than orange is when we'd probably consider orange to be the same distance from red as violet is.
In this case the Euclidean metric is not satisfactory and so we might create a new metric to measure the distance between colors.

\section*{K-Nearest Neighbor Classification}
Often we can improve the accuracy of a classifier by looking for other points besides the nearest neighbor.
Instead, we may choose an arbitrary number $k$ and give the point to be classified the majority label in from it's $k$ nearest neighbors.

There are pitfalls to this approach.
Consider a point that's closest neighbor has the label $0$.
If we only considered the nearest neighbor, then we would be finished.
However, what if the next $10$ nearest neighbors all had the label $1$?
Do you think that we should still classify the point as $0$?
What if the nearest point has a distance of $0.1$ and the next $10$ points have a distance of at least $100$?
The answer to these questions depend on the kind of data we are working with and the metric that we choose.
One should ensure to consider how to treat situations like this when working on classification algorithms.

The sklearn library has a neighbors module with a class KNeighborsClassifier for solving the $k$ nearest neighbors problem
\begin{lstlisting}
from sklearn import neighbors
nbrs = neighbors.KNeighborsClassifier(n_neighbors=8, weights='distance', p=2)
\end{lstlisting}

The \li{neighbors.KNeighborsClassifier} sets up the knearestnieghbor algorithm.
\li{n_neighbors} is how many neighbors you would like to find and \li{weights} specifies how you would like to weight the neighbors you have to make the classification.
Weights can be \li{'uniform'}, where the majority classification of the $k$ nearest neighbors determines the new data point's classification, or \li{'distance'}, where the neighbors nearer to the point have more weight than those farther away.
The argument \li{p} correpsods to the distance metric.
For this lab we will use \li{p=2} which is the Euclidean distance.

\begin{lstlisting}
nbrs.fit(points, labels)
\end{lstlisting}
Points and labels are your training data.
The \li{fit} function makes \li{nrbs} create a data structure containing those points and labels ready to be queried.

\begin{lstlisting}
nbrs.predict(testpoints)
\end{lstlisting}

The function \li{predict} takes in points to classify and outputs their respective labels.
%More information about this package can be found at http://scikit-learn.org/stable/modules/neighbors.html.
\begin{problem}
Get the post office handwritten digit data set. Load them with
\begin{lstlisting}
labels, points, testlabels, testpoints = np.load('PostalData.npz').items()
\end{lstlisting}
This contains a training set and a test set.
When you load the data the first entry of each array will be a name.
So \li{points[1]} and \li{labels[1]} point to the actual points and labels you want to use.
Each point is a  image that is $28 \times 28$ matrix of pixels that has been flattened.
The corresponding label indicates which number was written.
Try classifying the testpoints with \li{n_neighbors} as 4 and then as 10 and with \li{weights} \li{'uniform'} and then \li{'distance'}. Then do the classfication with \li{n_neighbors} being 1.
For each one return a report indicating how your classifier performs in terms of misclassifications as a percentange (testlabels are the true labels that correspond to the testpoints).
Which combination gives the most correct classifications?
(You may wish to streamline this process by writting a function that takes in \li{n_neighbors} and \li{weights} as arguments calls the neighbors functions appropriately)

A similar classification process is used by the United States Postal Service to automatically determine the zip code to send a letter to.

\begin{figure}[H]
\includegraphics[width=.25\textwidth]{Example.png}
\caption{An example of the number 6 taken from the data set}
\end{figure}
\end{problem}


\section{Application: Systems of linear differential equations}

\begin{outcome}
  \begin{enumerate}
  \item Solve a system of second order linear differential equations.
  \end{enumerate}
\end{outcome}

% ----------------------------------------------------------------------
\subsection{Differential equations}

Recall from calculus that if $y=f(x)$ is a function, then $y' = f'(x)$
is its derivative%
\index{derivative} and $y'' = f''(x)$ is its second derivative.  For
example,
\begin{equation*}
  \begin{array}{c@{~}c@{~}c}
    y &=& \sin(x), \\
    y' &=& \cos(x), \\
    y''&=& -\sin(x). \\
  \end{array}
\end{equation*}
Note that if $y=\sin(x)$, the second derivative $y''$ is exactly the
negative of $y$, i.e.,
\begin{equation*}
  y'' = -y.
\end{equation*}
This last equation is called a \textbf{differential equation}%
\index{equation!differential|see{differential equation}}%
\index{differential equation}. Unlike an ordinary equation, which is
about an unknown {\em number}, a differential equation is about an
unknown {\em function}. Typically, a differential equation mentions
the function and one or more of its derivatives. A differential
equation that only mentions the first derivative $y'$ is called a
\textbf{first-order}%
\index{differential equation!first order} differential equation. A
differential equation that also mentions the second derivative $y''$
is called a \textbf{second-order}%
\index{differential equation!second order} differential equation.
If the equation is a linear function of $y$ and its derivatives, it is
called a \textbf{linear differential equation}%
\index{linear differential equation}%
\index{differential equation!linear}.

We say that the function $y=\sin(x)$ is a \textbf{solution}%
\index{differential equation!solution} of the differential equation
$y'' = -y$. It is not the only solution. Another solution is
$y=\cos(x)$, because in that case, $y''=-\cos(x)$, and therefore
$y''=-y$. From calculus, we know that the \textbf{general solution} to
the differential equation $y'' = -y$ is given by
\begin{equation*}
  y = a\sin(x) + b\cos(x),
\end{equation*}
where $a$ and $b$ are any real numbers, i.e., parameters. Using the
terminology of linear algebra, we can say that the general solution of
the equation $y'' = -y$ is a \textbf{linear combination}%
\index{linear combination!of basic solutions!of differential equation}
of the \textbf{basic solutions}%
\index{basic solution!of differential equation}%
\index{differential equation!basic solution} $y=\sin(x)$ and
$y=\cos(x)$. More generally, we have the following theorem from
calculus:

\begin{theorem}{Solutions of $y''=qy$}{differential-equation}
  Let $q$ be a positive real number.  The differential equation
  \begin{equation*}
    \begin{array}{l@{~~}l@{~}clllll}
    y'' &=& -qy &\quad\mbox{has basic solutions}\quad\quad &
    y = \sin(\sqrt{q}\,x) & \mbox{and} & y = \cos(\sqrt{q}\,x), \\
    y'' &=& 0 &\quad\mbox{has basic solutions}\quad\quad &
    y = 1 & \mbox{and} & y = x, \\
    y'' &=& qy &\quad\mbox{has basic solutions}\quad\quad &
    y = e^{\sqrt{q}\,x} & \mbox{and} & y = e^{-\sqrt{q}\,x}. \\
    \end{array}
  \end{equation*}
\end{theorem}

\begin{proof}
  By taking derivatives, it is easy to check that each of the six
  functions is a solution of the corresponding differential
  equation. For example, for $y=\sin(\sqrt{q}\,x)$, we have
  $y'=\sqrt{q}\cos(\sqrt{q}\,x)$ and
  $y''=-q\sin(\sqrt{q}\,x)$. Therefore, $y''=-qy$.

  Note that we can obtain the general solution of each of the
  differential equations as a linear combination of the basic
  solutions. Thus, the general solution of $y''=-qy$ is
  \begin{equation*}
      y = a\sin(\sqrt{q}x) + b\cos(\sqrt{q}x),
  \end{equation*}
  the general solution of $y''=0$ is
  \begin{equation*}
      y = a + bx,
  \end{equation*}
  and the general solution of $y''=qy$ is
  \begin{equation*}
      y = ae^{\sqrt{q}\,x} + be^{-\sqrt{q}\,x},
  \end{equation*}
  where $a$ and $b$ are parameters. The fact that each of these
  solutions is indeed the most general one is proved in a calculus
  course.
\end{proof}

% ----------------------------------------------------------------------
\subsection{Systems of linear differential equations}

In the same way that a system of linear equations consists of several
linear equations about several variables, a \textbf{system of
  differential equations}%
\index{system of differential equations}%
\index{differential equation!system of} consists of several
differential equations about several unknown functions and their
derivatives. For example, the following is a system of second order
linear differential equations:
\begin{equation*}
  \begin{array}{c@{~}c@{~}r@{~}r@{~}r}
    y'' &=& 4y &-& 3z, \\
    z'' &=& 6y &-& 5z. \\
  \end{array}
\end{equation*}
In reading these equations, it is important to understand that we are
looking for two unknown {\em functions} $y=f(x)$ and $z=g(x)$ such
that their second derivatives satisfy both of the equations
$y''=4y-3z$ and $z''=6y-5z$. The reason that this is in principle a
difficult problem is that the equation for $y''$ mentions not only
$y$, but also $z$, and the equation for $z''$ mentions not only $z$,
but also $y$. Therefore, it is not possible to solve this system one
function at a time. We say that the variables $y$ and $z$ are
\textbf{coupled}%
\index{coupled variables}%
\index{differential equation!coupled variables}.

The following example shows how we can use diagonalization to decouple
the variables in a system of differential equations. This makes it
possible to solve the equations.

\begin{example}{A system of linear differential equations}{system-differential2}
  Solve the following system of second order linear differential equations:
  \begin{equation*}
    \begin{array}{c@{~}c@{~}r@{~}r@{~}r}
      y'' &=& 4y &-& 3z, \\
      z'' &=& 6y &-& 5z. \\
    \end{array}
  \end{equation*}
\end{example}

\begin{solution}
  We start by writing the system in matrix form:
  \begin{equation*}
    \begin{mymatrix}{c} y'' \\ z'' \end{mymatrix}
    =
    \begin{mymatrix}{rr} 4 & -3 \\ 6 & -5 \end{mymatrix}
    \begin{mymatrix}{c} y \\ z \end{mymatrix}.
  \end{equation*}
  Let us write
  \begin{equation*}
    \vect{v} = \begin{mymatrix}{c} y \\ z \end{mymatrix}
    \quad\mbox{and}\quad
    A = \begin{mymatrix}{rr} 4 & -3 \\ 6 & -5 \end{mymatrix}.
  \end{equation*}
  With these notations, the system of differential equations take the
  form
  \begin{equation}\label{eqn:system-differential2-1}
    \vect{v}'' = A\vect{v}.
  \end{equation}
  Our next step is to diagonalize the matrix $A$. Following the usual
  diagonalization procedure, we find that $A=PDP^{-1}$, where
  \begin{equation*}
    P = \begin{mymatrix}{rr} 1 & 1 \\ 1 & 2 \end{mymatrix}
    \quad\mbox{and}\quad
    D = \begin{mymatrix}{rr} 1 & 0 \\ 0 & -2 \end{mymatrix}.
  \end{equation*}
  With this, our equation takes the form
  $\vect{v}'' = PDP^{-1}\vect{v}$, which we can also write as
  $P^{-1}\vect{v}'' = DP^{-1}\vect{v}$. We now introduce a
  \textbf{change of variables}%
  \index{change of variables!for a differential equation}%
  \index{differential equation!change of variables}. Let
  $\vect{w} = P^{-1}\vect{v}$. Then our system of differential
  equations can be written as
  \begin{equation}\label{eqn:system-differential2-2}
    \vect{w}'' = D\vect{w}.
  \end{equation}
  Note that the equation {\eqref{eqn:system-differential2-2}} is of
  exactly the same form as the equation
  {\eqref{eqn:system-differential2-1}}, but with the crucial
  difference that the matrix in {\eqref{eqn:system-differential2-2}}
  is diagonal. Let us give a name to the components of $\vect{w}$:
  \begin{equation*}
    \vect{w} = \begin{mymatrix}{c} u \\ v \end{mymatrix}.
  \end{equation*}
  Then the equation {\eqref{eqn:system-differential2-2}} can be
  written as
  \begin{equation*}
    \begin{mymatrix}{c} u'' \\ v'' \end{mymatrix}
    =
    \begin{mymatrix}{rr} 1 & 0 \\ 0 & -2 \end{mymatrix}
    \begin{mymatrix}{c} u \\ v \end{mymatrix},
  \end{equation*}
  or equivalently,
  \begin{equation*}
    \begin{array}{c@{~}c@{~}c}
      u'' &=& u, \\
      v'' &=& -2v. \\
    \end{array}
  \end{equation*}
  Note that the variables $u$ and $v$ are not coupled! This happened
  because the matrix $D$ is diagonal. We can therefore use
  Theorem~\ref{thm:differential-equation} to solve the equations for
  $u$ and for $v$ separately. By
  Theorem~\ref{thm:differential-equation}, the general solution for
  the equation $u'' = u$ is
  \begin{equation*}
    u = a e^{x} + b e^{-x},
  \end{equation*}
  and the general solution for the equation $v''=-2v$ is
  \begin{equation*}
    v = c\sin(\sqrt{2}\,x) + d\cos(\sqrt{2}\,x).
  \end{equation*}
  Here, $a$, $b$, $c$, and $d$ are parameters. Therefore, the general
  solution for {\eqref{eqn:system-differential2-2}} is
  \begin{equation*}
    \vect{w} = \begin{mymatrix}{c} u \\ v \end{mymatrix}
    = \begin{mymatrix}{c}
      a e^{x} + b e^{-x} \\
      c\sin(\sqrt{2}\,x) + d\cos(\sqrt{2}\,x) \\
    \end{mymatrix}
    = a\,\begin{mymatrix}{c} e^{x} \\ 0 \end{mymatrix}
    + b\,\begin{mymatrix}{c} e^{-x} \\ 0 \end{mymatrix}
    + c\,\begin{mymatrix}{c} 0 \\ \sin(\sqrt{2}\,x) \end{mymatrix}
    + d\,\begin{mymatrix}{c} 0 \\ \cos(\sqrt{2}\,x) \end{mymatrix}.
  \end{equation*}
  We have therefore found the four basic solution of
  {\eqref{eqn:system-differential2-2}}.  But what about our original
  equation {\eqref{eqn:system-differential2-1}}? We can undo our
  change of variables. Since $\vect{w} = P^{-1}\vect{v}$, we have
  $\vect{v}=P\vect{w}$. Therefore, the general solution to our
  original system of differential equations is
  \begin{eqnarray*}
    \vect{v} ~=~ P\vect{w}
    &=& a\,P\begin{mymatrix}{c} e^{x} \\ 0 \end{mymatrix}
    + b\,P\begin{mymatrix}{c} e^{-x} \\ 0 \end{mymatrix}
    + c\,P\begin{mymatrix}{c} 0 \\ \sin(\sqrt{2}\,x) \end{mymatrix}
    + d\,P\begin{mymatrix}{c} 0 \\ \cos(\sqrt{2}\,x) \end{mymatrix} \\
    &=& a\,e^{x}\begin{mymatrix}{c} 1 \\ 1 \end{mymatrix}
    + b\,e^{-x}\begin{mymatrix}{c} 1 \\ 1 \end{mymatrix}
    + c\,\sin(\sqrt{2}\,x)\begin{mymatrix}{c} 1 \\ 2 \end{mymatrix}
    + d\,\cos(\sqrt{2}\,x)\begin{mymatrix}{c} 1 \\ 2 \end{mymatrix}.
  \end{eqnarray*}
\end{solution}

% ----------------------------------------------------------------------
\subsection{Example: coupled train cars}

One of the reasons that differential equations are important is that
the \textbf{laws of nature}%
\index{laws of nature} often take the form of differential equations.
For example, \textbf{Newton's second law of motion} asserts that the
acceleration of an object is equal to the total force on the object
divided by the mass of the object. In physics, it is common to use $t$
instead of $x$ for the independent variable, and $x$ instead of $y$
for the dependent variable, so that we write $x=f(t)$ instead of
$y=f(x)$. If $x$ is the position of the object at time $t$, then the
object's acceleration is $x''$, and Newton's second law takes the form
\begin{equation*}
  x'' = \frac{F}{m}.
\end{equation*}
This is a differential equation. In the following example, we will
need another law of physics, namely \textbf{Hooke's law}%
\index{spring (mechanics)!Hooke's law}%
\index{Hooke's law} about the force exerted by a spring. A
\textbf{spring}%
\index{spring (mechanics)} is an object made from an elastic material
(often in the shape of a coil), which returns to its original shape
after being stretched or compressed. Hooke's law states that the force
exerted by a spring to both of its ends is equal to
\begin{equation*}
  F = k x.
\end{equation*}
Here $x$ is the \textbf{extension}%
\index{spring (mechanics)!extension} of the spring, i.e., the change
in length of the spring, relative to its relaxed (natural)
length. Also, $k$ is a constant called the \textbf{spring constant}%
\index{spring (mechanics)!spring constant}, measured in units of
$\frac{\SI{N}}{\m}$ (Newtons per meter). Of course the direction of
the force on one end of the spring is the opposite of the direction on
the other end. Hooke's law is not a differential equation, because it
does not mention any derivatives. It is just an ordinary
equation. Nevertheless, both the force $F$ and the extension $x$ can
vary with time, i.e., they can both be functions of $t$.

\begin{example}{Coupled train cars}{coupled-train-cars}
  Consider a train made up of three cars of mass $1\kg$, $2\kg$, and
  $1\kg$, which are aligned on a linear track and connected by
  springs. Assume that each spring has the same spring constant
  $k=2\frac{\SI{N}}{\m}$.
  \begin{equation*}
    \begin{tikzpicture}
      \draw[thick] (0,0) -- (16.4,0);
      \begin{scope}
        \draw[fill=white] (2,0.5) rectangle node {$1\kg$} (4,1.5);
        \draw[fill=white] (2.5,0.25) circle (0.25);
        \draw[fill=white] (3.5,0.25) circle (0.25);
      \end{scope}
      \begin{scope}[xshift=5.2cm]
        \draw[fill=white] (1.8,0.5) rectangle node {$2\kg$} (4.2,1.9);
        \draw[fill=white] (2.5,0.25) circle (0.25);
        \draw[fill=white] (3.5,0.25) circle (0.25);
      \end{scope}
      \begin{scope}[xshift=10.4cm]
        \draw[fill=white] (2,0.5) rectangle node {$1\kg$} (4,1.5);
        \draw[fill=white] (2.5,0.25) circle (0.25);
        \draw[fill=white] (3.5,0.25) circle (0.25);
      \end{scope}
      \begin{scope}[xshift=4cm]
        \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
        -- ++(0.26,-0.6) -- ++(0.26,0.6)
        -- ++(0.26,-0.6) -- ++(0.26,0.6)
        -- ++(0.26,-0.6) -- ++(0.26,0.6)
        -- ++(0.26,-0.6) -- ++(0.26,0.6)
        -- ++(0.26,-0.6) -- ++(0.13,0.3)
        -- ++(0.2,0);
        \draw (1.5,1.3) node[above] {$k$};
      \end{scope}
      \begin{scope}[xshift=9.4cm]
        \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
        -- ++(0.26,-0.6) -- ++(0.26,0.6)
        -- ++(0.26,-0.6) -- ++(0.26,0.6)
        -- ++(0.26,-0.6) -- ++(0.26,0.6)
        -- ++(0.26,-0.6) -- ++(0.26,0.6)
        -- ++(0.26,-0.6) -- ++(0.13,0.3)
        -- ++(0.2,0);
        \draw (1.5,1.3) node[above] {$k$};
      \end{scope}
    \end{tikzpicture}
  \end{equation*}
  Find and solve the equations of motion of this system.
\end{example}

\begin{solution}
  Let us start by defining appropriate coordinates.  Let $x$ be the
  position of the first car, $y$ the position of the second car, and
  $z$ the position of the third car, measured in meters from left to
  right, relative to each car's natural resting position. The $x$-,
  $y$-, and $z$-axes are shown in the following picture.  All three
  axes are parallel to the train tracks, but they have their origins
  in different places. The coordinates are chosen so that when $x=0$,
  $y=0$, and $z=0$, then all three cars are at rest and both springs
  are in their natural relaxed state.
  \begin{equation*}
    \begin{tikzpicture}[scale=0.8, color=blue!50!black]
      \draw[thick] (0,0) -- (16.4,0);
      \begin{scope}
        \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
        \draw[fill=blue!10] (2.5,0.25) circle (0.25);
        \draw[fill=blue!10] (3.5,0.25) circle (0.25);
      \end{scope}
      \begin{scope}[xshift=5.2cm]
        \draw[fill=blue!10] (1.8,0.5) rectangle (4.2,1.9);
        \draw[fill=blue!10] (2.5,0.25) circle (0.25);
        \draw[fill=blue!10] (3.5,0.25) circle (0.25);
      \end{scope}
      \begin{scope}[xshift=10.4cm]
        \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
        \draw[fill=blue!10] (2.5,0.25) circle (0.25);
        \draw[fill=blue!10] (3.5,0.25) circle (0.25);
      \end{scope}
      \begin{scope}[xshift=4cm]
        \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
        -- ++(0.26,-0.6) -- ++(0.26,0.6)
        -- ++(0.26,-0.6) -- ++(0.26,0.6)
        -- ++(0.26,-0.6) -- ++(0.26,0.6)
        -- ++(0.26,-0.6) -- ++(0.26,0.6)
        -- ++(0.26,-0.6) -- ++(0.13,0.3)
        -- ++(0.2,0);
      \end{scope}
      \begin{scope}[xshift=9.4cm]
        \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
        -- ++(0.26,-0.6) -- ++(0.26,0.6)
        -- ++(0.26,-0.6) -- ++(0.26,0.6)
        -- ++(0.26,-0.6) -- ++(0.26,0.6)
        -- ++(0.26,-0.6) -- ++(0.26,0.6)
        -- ++(0.26,-0.6) -- ++(0.13,0.3)
        -- ++(0.2,0);
      \end{scope}
      \begin{scope}[yshift=-1cm, xshift=0cm, color=black]
        \draw[->] (1,0) -- node[above] {$x$-axis} +(4,0);
        \draw (3,0) -- +(0,-0.2) node[below=-4pt] {$x=0$};
      \end{scope}
      \begin{scope}[yshift=-1cm, xshift=5.2cm, color=black]
        \draw[->] (1,0) -- node[above] {$y$-axis} +(4,0);
        \draw (3,0) -- +(0,-0.2) node[below=-4pt] {$y=0$};
      \end{scope}
      \begin{scope}[yshift=-1cm, xshift=10.4cm, color=black]
        \draw[->] (1,0) -- node[above] {$z$-axis} +(4,0);
        \draw (3,0) -- +(0,-0.2) node[below=-4pt] {$z=0$};
      \end{scope}
    \end{tikzpicture}
  \end{equation*}
  Then the extension of the left spring is $y-x$, and therefore the
  left spring's contracting force is
  \begin{equation*}
    F_1 = k(y-x).
  \end{equation*}
  Similarly, the extension of the right spring is $z-y$, and therefore
  its contracting force is
  \begin{equation*}
    F_2 = k(z-y).
  \end{equation*}
  The total force acting on the left car is $F_1$, the total force
  acting on the middle car is $F_2-F_1$, and the total force acting on
  the right car is $-F_2$. By Newton's second law, the acceleration
  of each car is given by $x'' = \frac{F_1}{m_1}$,
  $y'' = \frac{F_2-F_1}{m_2}$, and $z''=\frac{-F_2}{m_3}$. We
  therefore have the following equations of motion:
  \begin{eqnarray*}
    x'' &=& \frac{k}{m_1}(y-x), \\
    y'' &=& \frac{k}{m_2}(x-2y+z), \\
    z'' &=& \frac{k}{m_3}(z-y).
  \end{eqnarray*}
  Let us ignore the physical units and plug in the masses $m_1=1$,
  $m_2=2$, and $m_3=1$ and the spring constant $k=2$. Then the equations
  of motion are:
  \begin{eqnarray*}
    x'' &=& 2(y-x), \\
    y'' &=& x-2y+z, \\
    z'' &=& 2(z-y),
  \end{eqnarray*}
  or equivalently in matrix form:
  \begin{equation*}
    \begin{mymatrix}{c} x'' \\ y'' \\ z'' \end{mymatrix}
    =
    \begin{mymatrix}{rrr}
      -2 & 2 & 0 \\
      1 & -2 & 1 \\
      0 & 2 & -2 \\
    \end{mymatrix}
    \begin{mymatrix}{c} x \\ y \\ z \end{mymatrix}.
  \end{equation*}
  We can also write this as $\vect{v}''=A\vect{v}$, where
  \begin{equation*}
    \vect{v} = \begin{mymatrix}{c} x \\ y \\ z \end{mymatrix}
    \quad\mbox{and}\quad
    A = \begin{mymatrix}{rrr}
      -2 & 2 & 0 \\
      1 & -2 & 1 \\
      0 & 2 & -2 \\
    \end{mymatrix}.
  \end{equation*}
  To solve the equation, we diagonalize the matrix $A$. Using the usual
  method for diagonalization, we find that $A=PDP^{-1}$, where
  \begin{equation*}
    P = \begin{mymatrix}{rrr}
      -1 &  1 & 1 \\
      0  & -1 & 1 \\
      1  &  1 & 1 \\
    \end{mymatrix}
    \quad\mbox{and}\quad
    D = \begin{mymatrix}{rrr}
      -2 &  0 & 0 \\
      0  & -4 & 0 \\
      0  &  0 & 0 \\
    \end{mymatrix}.
  \end{equation*}
  The equation $\vect{v}''=A\vect{v}$ then becomes
  $\vect{v}''=PDP^{-1}\vect{v}$, or equivalently
  $P^{-1}\vect{v}'' = DP^{-1}\vect{v}$. We then diagonalize the
  equation by performing the change of variables
  $\vect{w} = P^{-1}\vect{v}$. The equation becomes
  \begin{equation*}
    \vect{w}'' = D\vect{w}.
  \end{equation*}
  If the components of $\vect{w}$ are called $u$, $v$, and $w$, we can
  write this as
  \begin{equation*}
    \begin{mymatrix}{c} u'' \\ v'' \\ w'' \end{mymatrix}
    =
    \begin{mymatrix}{rrr}
      -2 &  0 & 0 \\
      0  & -4 & 0 \\
      0  &  0 & 0 \\
    \end{mymatrix}
    \begin{mymatrix}{c} u \\ v \\ w \end{mymatrix},
  \end{equation*}
  or equivalently,
  \begin{equation*}
    \begin{array}{c@{~}c@{~}c}
      u'' &=& -2u, \\
      v'' &=& -4v, \\
      w'' &=& 0.   \\
    \end{array}
  \end{equation*}
  Since the variables are now decoupled, we can solve each
  differential equation individually.
  \begin{itemize}
  \item \textbf{Solutions for $\eigenvar=-2$:}
    By Theorem~\ref{thm:differential-equation}, the basic solutions
    for $u'' = -2u$ are
    \begin{equation*}
      u=\sin(\sqrt{2}\,t)
      \quad\mbox{and}\quad
      u=\cos(\sqrt{2}\,t).
    \end{equation*}
    This translates into the following basic solutions for $\vect{w}$:
    \begin{equation*}
      \vect{w}
      = \sin(\sqrt{2}\,t)\begin{mymatrix}{r} 1 \\ 0 \\ 0 \end{mymatrix}
      \quad\mbox{and}\quad
      \vect{w}
      = \cos(\sqrt{2}\,t)\begin{mymatrix}{r} 1 \\ 0 \\ 0 \end{mymatrix}.
    \end{equation*}
    Using $\vect{v} = P\vect{w}$ to change to the original variables,
    we get the basic solutions
    \begin{equation*}
      \vect{v}
      = \sin(\sqrt{2}\,t)\begin{mymatrix}{r} -1 \\ 0 \\ 1 \end{mymatrix}
      \quad\mbox{and}\quad
      \vect{v}
      = \cos(\sqrt{2}\,t)\begin{mymatrix}{r} -1 \\ 0 \\ 1 \end{mymatrix}.
    \end{equation*}
    For example, the first of these basic solutions, written in the
    coordinates $x$, $y$, and $z$, gives
    \begin{equation*}
      \begin{array}{c@{~~}c@{~}l}
        x &=& -\sin(\sqrt{2}\,t), \\
        y &=& ~~~\,0, \\
        z &=& ~~~\sin(\sqrt{2}\,t). \\
      \end{array}
    \end{equation*}
    This corresponds to a periodic oscillation of the train where the
    middle car is stationary, and the left car moves left when the
    right car moves right. Each oscillation takes
    $2\pi/\sqrt{2}\approx 4.4$ seconds.  Here is a ``movie'' showing
    one oscillation:
    \begin{equation*}
      \begin{tikzpicture}[scale=0.8, color=blue!50!black]
        \begin{scope}[yshift=0cm]
          \draw (-3,1) node[right] {$t=0\s$};
          \draw[thick] (0,0) -- (16.4,0);
          \begin{scope}[xshift=0cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=5.2cm]
            \draw[fill=blue!10] (1.8,0.5) rectangle (4.2,1.9);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=10.4cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.13,0.3)
            -- ++(0.2,0);
          \end{scope}
          \begin{scope}[xshift=9.4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.13,0.3)
            -- ++(0.2,0);
          \end{scope}
        \end{scope}
        \begin{scope}[yshift=-2cm]
          \draw (-3,1) node[right] {$t=1.1\s$};
          \draw[thick] (0,0) -- (16.4,0);
          \begin{scope}[xshift=-1cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=5.2cm]
            \draw[fill=blue!10] (1.8,0.5) rectangle (4.2,1.9);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=11.4cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=3cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.18,0.3)
            -- ++(0.36,-0.6) -- ++(0.36,0.6)
            -- ++(0.36,-0.6) -- ++(0.36,0.6)
            -- ++(0.36,-0.6) -- ++(0.36,0.6)
            -- ++(0.36,-0.6) -- ++(0.36,0.6)
            -- ++(0.36,-0.6) -- ++(0.18,0.3)
            -- ++(0.2,0);
          \end{scope}
          \begin{scope}[xshift=9.4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.18,0.3)
            -- ++(0.36,-0.6) -- ++(0.36,0.6)
            -- ++(0.36,-0.6) -- ++(0.36,0.6)
            -- ++(0.36,-0.6) -- ++(0.36,0.6)
            -- ++(0.36,-0.6) -- ++(0.36,0.6)
            -- ++(0.36,-0.6) -- ++(0.18,0.3)
            -- ++(0.2,0);
          \end{scope}
        \end{scope}
        \begin{scope}[yshift=-4cm]
          \draw (-3,1) node[right] {$t=2.2\s$};
          \draw[thick] (0,0) -- (16.4,0);
          \begin{scope}[xshift=0cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=5.2cm]
            \draw[fill=blue!10] (1.8,0.5) rectangle (4.2,1.9);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=10.4cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.13,0.3)
            -- ++(0.2,0);
          \end{scope}
          \begin{scope}[xshift=9.4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.13,0.3)
            -- ++(0.2,0);
          \end{scope}
        \end{scope}
        \begin{scope}[yshift=-6cm]
          \draw (-3,1) node[right] {$t=3.3\s$};
          \draw[thick] (0,0) -- (16.4,0);
          \begin{scope}[xshift=1cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=5.2cm]
            \draw[fill=blue!10] (1.8,0.5) rectangle (4.2,1.9);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=9.4cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=5cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.08,0.3)
            -- ++(0.16,-0.6) -- ++(0.16,0.6)
            -- ++(0.16,-0.6) -- ++(0.16,0.6)
            -- ++(0.16,-0.6) -- ++(0.16,0.6)
            -- ++(0.16,-0.6) -- ++(0.16,0.6)
            -- ++(0.16,-0.6) -- ++(0.08,0.3)
            -- ++(0.2,0);
          \end{scope}
          \begin{scope}[xshift=9.4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.08,0.3)
            -- ++(0.16,-0.6) -- ++(0.16,0.6)
            -- ++(0.16,-0.6) -- ++(0.16,0.6)
            -- ++(0.16,-0.6) -- ++(0.16,0.6)
            -- ++(0.16,-0.6) -- ++(0.16,0.6)
            -- ++(0.16,-0.6) -- ++(0.08,0.3)
            -- ++(0.2,0);
          \end{scope}
        \end{scope}
        \begin{scope}[yshift=-8cm]
          \draw (-3,1) node[right] {$t=4.4\s$};
          \draw[thick] (0,0) -- (16.4,0);
          \begin{scope}[xshift=0cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=5.2cm]
            \draw[fill=blue!10] (1.8,0.5) rectangle (4.2,1.9);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=10.4cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.13,0.3)
            -- ++(0.2,0);
          \end{scope}
          \begin{scope}[xshift=9.4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.13,0.3)
            -- ++(0.2,0);
          \end{scope}
        \end{scope}
        \begin{scope}[xshift=3cm,yshift=1cm,rotate=-90]
          \draw[ultra thick,red] (0,0) sin (2,-1) cos (4,0) sin (6,1) cos (8,0);
        \end{scope}
        \begin{scope}[xshift=8.2cm,yshift=1cm,rotate=-90]
          \draw[ultra thick,red] (0,0) sin (2,0) cos (4,0) sin (6,0) cos (8,0);
        \end{scope}
        \begin{scope}[xshift=13.4cm,yshift=1cm,rotate=-90]
          \draw[ultra thick,red] (0,0) sin (2,1) cos (4,0) sin (6,-1) cos (8,0);
        \end{scope}
      \end{tikzpicture}
    \end{equation*}
    The other basic solution, with $\cos$ instead of $\sin$, is the
    same motion, just starting at a different offset in time. Note how
    the eigenvector of $A$,
    \begin{equation*}
      \begin{mymatrix}{r} -1 \\ 0 \\ 1 \end{mymatrix},
    \end{equation*}
    describes the relative motion of the three cars, i.e., the first
    and last cars are moving in opposite directions, whereas the
    middle car is stationary. The corresponding eigenvalue
    $\eigenvar=-2$ determines the frequency. The frequency, which is
    $\sqrt{2}/2\pi$ oscillations per second, is also called an
    \textbf{eigenfrequency}%
    \index{eigenfrequency}%
    \index{differential equation!eigenfrequency} or \textbf{resonance
      frequency}%
    \index{resonance!frequency}%
    \index{differential equation!resonance frequency}%
    \index{frequency!resonance} of the system.
  \item \textbf{Solutions for $\eigenvar=-4$:}
    By Theorem~\ref{thm:differential-equation}, the basic solutions
    for $v'' = -4v$ are
    \begin{equation*}
      v=\sin(2t)
      \quad\mbox{and}\quad
      v=\cos(2t).
    \end{equation*}
    This translates into the following basic solutions for $\vect{w}$:
    \begin{equation*}
      \vect{w}
      = \sin(2t)\begin{mymatrix}{r} 0 \\ 1 \\ 0 \end{mymatrix}
      \quad\mbox{and}\quad
      \vect{w}
      = \cos(2t)\begin{mymatrix}{r} 0 \\ 1 \\ 0 \end{mymatrix}.
    \end{equation*}
    We change this to the original variables using $\vect{v} =
    P\vect{w}$, and get the basic solutions
    \begin{equation*}
      \vect{v}
      = \sin(2t)\begin{mymatrix}{r} 1 \\ -1 \\ 1 \end{mymatrix}
      \quad\mbox{and}\quad
      \vect{v}
      = \cos(2t)\begin{mymatrix}{r} 1 \\ -1 \\ 1 \end{mymatrix}.
    \end{equation*}
    Writing the first of these basic solutions in the coordinates $x$,
    $y$, and $z$, we get
    \begin{equation*}
      \begin{array}{c@{~~}c@{~}l}
        x &=& ~~~\sin(2t), \\
        y &=& -\sin(2t), \\
        z &=& ~~~\sin(2t). \\
      \end{array}
    \end{equation*}
    This corresponds to a periodic oscillation of the train where the
    outer cars move right at the same time that the middle car moves
    left. Each oscillation takes
    $2\pi/2\approx 3.14$ seconds. Here is a ``movie'' of the motion:
    \begin{equation*}
      \begin{tikzpicture}[scale=0.8, color=blue!50!black]
        \begin{scope}[yshift=0cm]
          \draw (-3,1) node[right] {$t=0\s$};
          \draw[thick] (0,0) -- (16.4,0);
          \begin{scope}[xshift=0cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=5.2cm]
            \draw[fill=blue!10] (1.8,0.5) rectangle (4.2,1.9);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=10.4cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.13,0.3)
            -- ++(0.2,0);
          \end{scope}
          \begin{scope}[xshift=9.4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.13,0.3)
            -- ++(0.2,0);
          \end{scope}
        \end{scope}
        \begin{scope}[yshift=-2cm]
          \draw (-3,1) node[right] {$t=0.79\s$};
          \draw[thick] (0,0) -- (16.4,0);
          \begin{scope}[xshift=-1cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=6.2cm]
            \draw[fill=blue!10] (1.8,0.5) rectangle (4.2,1.9);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=9.4cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=3cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.23,0.3)
            -- ++(0.46,-0.6) -- ++(0.46,0.6)
            -- ++(0.46,-0.6) -- ++(0.46,0.6)
            -- ++(0.46,-0.6) -- ++(0.46,0.6)
            -- ++(0.46,-0.6) -- ++(0.46,0.6)
            -- ++(0.46,-0.6) -- ++(0.23,0.3)
            -- ++(0.2,0);
          \end{scope}
          \begin{scope}[xshift=10.4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.03,0.3)
            -- ++(0.06,-0.6) -- ++(0.06,0.6)
            -- ++(0.06,-0.6) -- ++(0.06,0.6)
            -- ++(0.06,-0.6) -- ++(0.06,0.6)
            -- ++(0.06,-0.6) -- ++(0.06,0.6)
            -- ++(0.06,-0.6) -- ++(0.03,0.3)
            -- ++(0.2,0);
          \end{scope}
        \end{scope}
        \begin{scope}[yshift=-4cm]
          \draw (-3,1) node[right] {$t=1.57\s$};
          \draw[thick] (0,0) -- (16.4,0);
          \begin{scope}[xshift=0cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=5.2cm]
            \draw[fill=blue!10] (1.8,0.5) rectangle (4.2,1.9);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=10.4cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.13,0.3)
            -- ++(0.2,0);
          \end{scope}
          \begin{scope}[xshift=9.4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.13,0.3)
            -- ++(0.2,0);
          \end{scope}
        \end{scope}
        \begin{scope}[yshift=-6cm]
          \draw (-3,1) node[right] {$t=2.36\s$};
          \draw[thick] (0,0) -- (16.4,0);
          \begin{scope}[xshift=1cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=4.2cm]
            \draw[fill=blue!10] (1.8,0.5) rectangle (4.2,1.9);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=11.4cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=5cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.03,0.3)
            -- ++(0.06,-0.6) -- ++(0.06,0.6)
            -- ++(0.06,-0.6) -- ++(0.06,0.6)
            -- ++(0.06,-0.6) -- ++(0.06,0.6)
            -- ++(0.06,-0.6) -- ++(0.06,0.6)
            -- ++(0.06,-0.6) -- ++(0.03,0.3)
            -- ++(0.2,0);
          \end{scope}
          \begin{scope}[xshift=8.4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.23,0.3)
            -- ++(0.46,-0.6) -- ++(0.46,0.6)
            -- ++(0.46,-0.6) -- ++(0.46,0.6)
            -- ++(0.46,-0.6) -- ++(0.46,0.6)
            -- ++(0.46,-0.6) -- ++(0.46,0.6)
            -- ++(0.46,-0.6) -- ++(0.23,0.3)
            -- ++(0.2,0);
          \end{scope}
        \end{scope}
        \begin{scope}[yshift=-8cm]
          \draw (-3,1) node[right] {$t=3.14\s$};
          \draw[thick] (0,0) -- (16.4,0);
          \begin{scope}[xshift=0cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=5.2cm]
            \draw[fill=blue!10] (1.8,0.5) rectangle (4.2,1.9);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=10.4cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.13,0.3)
            -- ++(0.2,0);
          \end{scope}
          \begin{scope}[xshift=9.4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.13,0.3)
            -- ++(0.2,0);
          \end{scope}
        \end{scope}
        \begin{scope}[xshift=3cm,yshift=1cm,rotate=-90]
          \draw[ultra thick,red] (0,0) sin (2,-1) cos (4,0) sin (6,1) cos (8,0);
        \end{scope}
        \begin{scope}[xshift=8.2cm,yshift=1cm,rotate=-90]
          \draw[ultra thick,red] (0,0) sin (2,1) cos (4,0) sin (6,-1) cos (8,0);
        \end{scope}
        \begin{scope}[xshift=13.4cm,yshift=1cm,rotate=-90]
          \draw[ultra thick,red] (0,0) sin (2,-1) cos (4,0) sin (6,1) cos (8,0);
        \end{scope}
      \end{tikzpicture}
    \end{equation*}
    As before, the other basic solution, using $\cos$ instead of
    $\sin$, is the same motion, but shifted in time. Also, the
    eigenvector
    \begin{equation*}
      \begin{mymatrix}{r} 1 \\ -1 \\ 1 \end{mymatrix}
    \end{equation*}
    describes the relative motion of the three cars; here the first
    and last car move in the same direction while the middle car moves
    in the opposite direction. The eigenfrequency of this oscillation,
    at $2/2\pi$ oscillations per second, is slightly higher than the
    first one, due to the larger magnitude of the eigenvalue
    $\eigenvar=-4$.
  \item \textbf{Solutions for $\eigenvar=0$:} The last eigenvalue is
    zero. By Theorem~\ref{thm:differential-equation}, the general
    solution of $w'' = 0$ is
    \begin{equation*}
      w = a+bt,
    \end{equation*}
    where $a$ and $b$ are arbitrary constants. This translates into
    the following solution for $\vect{w}$:
    \begin{equation*}
      \vect{w}
      = (a+bt)\begin{mymatrix}{r} 0 \\ 0 \\ 1 \end{mymatrix},
    \end{equation*}
    and by the change of variables $\vect{v} = P\vect{w}$, we get
    \begin{equation*}
      \vect{v}
      = (a+bt)\begin{mymatrix}{r} 1 \\ 1 \\ 1 \end{mymatrix}.
    \end{equation*}
    Writing this in terms of the coordinates $x$, $y$, and $z$, we get
    \begin{equation*}
      \begin{array}{c@{~~}c@{~}l}
        x &=& a+bt, \\
        y &=& a+bt, \\
        z &=& a+bt. \\
      \end{array}
    \end{equation*}
    This simply describes a linear motion: the three cars are moving
    down the track at constant speed $b$ from some initial starting
    position $a$.
    \begin{equation*}
      \begin{tikzpicture}[scale=0.8, color=blue!50!black]
        \begin{scope}[yshift=0cm]
          \draw (-3,1) node[right] {$t=0\s$};
          \draw[thick] (0,0) -- (16.4,0);
          \begin{scope}[xshift=-1cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=4.2cm]
            \draw[fill=blue!10] (1.8,0.5) rectangle (4.2,1.9);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=9.4cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=3cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.13,0.3)
            -- ++(0.2,0);
          \end{scope}
          \begin{scope}[xshift=8.4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.13,0.3)
            -- ++(0.2,0);
          \end{scope}
        \end{scope}
        \begin{scope}[yshift=-2cm]
          \draw (-3,1) node[right] {$t=1\s$};
          \draw[thick] (0,0) -- (16.4,0);
          \begin{scope}[xshift=0cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=5.2cm]
            \draw[fill=blue!10] (1.8,0.5) rectangle (4.2,1.9);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=10.4cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.13,0.3)
            -- ++(0.2,0);
          \end{scope}
          \begin{scope}[xshift=9.4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.13,0.3)
            -- ++(0.2,0);
          \end{scope}
        \end{scope}
        \begin{scope}[yshift=-4cm]
          \draw (-3,1) node[right] {$t=2\s$};
          \draw[thick] (0,0) -- (16.4,0);
          \begin{scope}[xshift=1cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=6.2cm]
            \draw[fill=blue!10] (1.8,0.5) rectangle (4.2,1.9);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=11.4cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=5cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.13,0.3)
            -- ++(0.2,0);
          \end{scope}
          \begin{scope}[xshift=10.4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.13,0.3)
            -- ++(0.2,0);
          \end{scope}
        \end{scope}
        \begin{scope}[yshift=-6cm]
          \draw (-3,1) node[right] {$t=3\s$};
          \draw[thick] (0,0) -- (16.4,0);
          \begin{scope}[xshift=2cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=7.2cm]
            \draw[fill=blue!10] (1.8,0.5) rectangle (4.2,1.9);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=12.4cm]
            \draw[fill=blue!10] (2,0.5) rectangle (4,1.5);
            \draw[fill=blue!10] (2.5,0.25) circle (0.25);
            \draw[fill=blue!10] (3.5,0.25) circle (0.25);
          \end{scope}
          \begin{scope}[xshift=6cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.13,0.3)
            -- ++(0.2,0);
          \end{scope}
          \begin{scope}[xshift=11.4cm]
            \draw (0,1) -- ++(0.2,0) -- ++(0.13,0.3)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.26,0.6)
            -- ++(0.26,-0.6) -- ++(0.13,0.3)
            -- ++(0.2,0);
          \end{scope}
        \end{scope}
        \begin{scope}[xshift=3cm,yshift=1cm,rotate=-90]
          \draw[ultra thick, red] (0,-1) -- (6,2);
        \end{scope}
        \begin{scope}[xshift=8.2cm,yshift=1cm,rotate=-90]
          \draw[ultra thick, red] (0,-1) -- (6,2);
        \end{scope}
        \begin{scope}[xshift=13.4cm,yshift=1cm,rotate=-90]
          \draw[ultra thick, red] (0,-1) -- (6,2);
        \end{scope}
      \end{tikzpicture}
    \end{equation*}
    As before, the eigenvector
    \begin{equation*}
      \begin{mymatrix}{r} 1 \\ 1 \\ 1 \end{mymatrix}
    \end{equation*}
    describes the relative motion of the three cars (in this case all
    moving in the same direction). The eigenvalue $\eigenvar=0$
    indicates linear motion.
  \end{itemize}
  What we have described here are the {\em basic} solutions of the
  system. The solutions corresponding to each of the eigenvalues are
  also called \textbf{modes}%
  \index{mode of resonance}%
  \index{resonance!mode}%
  \index{differential equation!mode} of the system. Thus, the train
  has three modes, each corresponding to a particular eigenvalue of
  the matrix $A$. In the first mode, the middle car is stationary and
  the other two cars oscillate in opposite directions.  In the second
  mode, the middle car oscillates in the opposite direction of the two
  outer cars. The third mode is a linear movement along the track.

  As always, the general solution is a linear combination of basic
  solutions; for example, the cars might be oscillating in both the
  first and second modes at their respective frequencies, while also
  moving down the tracks.
\end{solution}


% !TEX root = Main.tex
\section{Neural Networks}
\textbf{Neurons}: $F_\sigma(\mathbf{x};\mathbf{w}) = \sigma(w_0 + \sum_{i=1}^M{x_iw_i})$.\\
\textbf{Activation}: $s(x)=\frac{1}{1+e^{-x}}$, $s'(x)=s(x)(1-s(x))$; $tanh(x) = \tfrac{sinh(x)}{cosh(x)}$, $tanh'(x) = 1 - tanh^2(x)$;  $ReLU(x) = max(0, x)$, $ReLU'(x) = 0\  or\  1$\\
\textbf{Output}: Linear Regression: $\hat{\mathbf{y}} = \mathbf{W}^L\mathbf{x}^{L-1}$\\
Binary Classification (Logistic):\\
$\hat{y_1} = \text{P}[Y=1|\mathbf{x}] = \frac{1}{1 + \exp[-\langle \mathbf{w}_1^L,\mathbf{x}^{L-1}\rangle]}$\\
Multiclass (Softmax):\\
$\hat{y_k} = \text{P}[Y=k|\mathbf{x}]= \frac{\exp[\langle \mathbf{w}_k^L,\mathbf{x}^{L-1}\rangle]}{\sum_{m=1}^{K}{\exp[\langle \mathbf{w}_m^L, \mathbf{x}^{L-1}\rangle]}}$.\\
(softmax: $\sigma(z)_j = e^{z_j}/\sum_{k=1}^K e^{z_k}$ for $j=1,\dots,K$)\\
\textbf{Loss Function:} Squared Loss $\frac{1}{2}(y - \hat{y})^2$\\
Cross-Entropy Loss: $-y \log \hat{y} - (1-y)\log(1-\hat{y})$.\\
\textbf{Units \& Layers}: FW Prop: $\mathbf{x}^{l} = \sigma^{l}\left(\mathbb{W}^{\left(l\right)}\mathbf{x}^{\left(l-1\right)}\right)$. L-layer network: $\mathbf{y}=\sigma^{\left(L\right)}\left(\mathbf{W}^{(L)}\sigma^{(L-1)}\left(\cdots\left(\sigma^{(1)}\left(\mathbf{W}^{(1)}\mathbf{x}\right)\cdots\right)\right)\right)$

\subsection*{Backpropagation (L2L Jacobian)}
$\mathbf{x}$ = Prev Layer Act, $\mathbf{x^+}$ = Curr Layer Act\\$\mathbf{J}$ = $J_{ij}$ (Jacobian of mapping $\mathbf{x}\rightarrow\mathbf{x^+}$)\\ $\mathbf{x_i^+} = \sigma(\mathbf{w}_i^\top\mathbf{x})$, $J_{ij} = \frac{\partial \mathbf{x_i^+}}{\partial \mathbf{x}_j} = w_{ij}\cdot\sigma'(\mathbf{w}_i^\top\mathbf{x})$.\\
Multi-Layer (First FW, then BW):\\
$\frac{\partial\mathbf{x}^{(l)}}{\partial\mathbf{x}^{(l-n)}} = \mathbf{J}^{(l)}\cdot\frac{\partial\mathbf{x}^{(l-1)}}{\partial\mathbf{x}^{(l-n)}}=\mathbf{J}^{(l)}\cdot\mathbf{J}^{(l-1)}\cdots\mathbf{J}^{(l-n+1)}$, then BP $ \nabla_{x^{(l)}}^\top\ell=\nabla_{y}^\top\ell\cdot\mathbf{J}^{(L)}\cdots\mathbf{J}^{(l+1)}$

\subsection*{Convolutional Neural Networks}
Translation invariance of images $\rightarrow$ neurons compute same func, shift invariant filters; weights defined as filter masks, e.g. convolution: $F_{n,m}(\mathbf{x};\mathbf{w}) = \sigma(b + \sum_{k=-2}^2\sum_{l=-2}^{2}{w_{k,l}x_{n+k,m+l}})$.\\
$i',j'$ win. pos, $k$: win. size, $K$: weigths, $c$: chan.:\\
 $(I\star K)_{i', j'} = \sum_c \sum_{-k\leq i,j \leq k} pad(X_c)_{i'+i,j'+j} \cdot(K_c)_{i,j}$\\
Max-Pooling (size 3): $\max_{-1\leq i', j' \leq 1}(\cdot)_{i+i', j+j'}$ % TODO: Make prettier, remove $\cdot$

\subsection*{ELBO}
$\lambda^* = argmax_\lambda ELBO(\lambda)$ where $ELBO=$\\
$log p(y,\theta) \geq E_q[log p(x^{1:L},y; \theta)-log q(x^{1:L}, y)]$

\subsection*{Autoregressive models}
$p(x)=\pi_{i=1}^{n^2} p(x_i|x_1,\dots ,x_{i-1})$


%&LaTeX

\section{Discrete Fourier Transform}

\subsection{Lab Background}
By the end of this lab you should have a firm understanding of how the
Discrete Fourier Transform (DFT) can be implemented exactly using the
Fast Fourier Transform (FFT). In addition you should be able to
identify common problems using the DFT to analyze signals. You will
also be familiar with a new tool, the spectrogram, that uses the DFT
as a function of time.

\subsection{Implementing the DFT}

% TODO: have them work with this block for a good implementation of the J-DSP
Recall that the DFT can be implemented directly from the analysis
equation. For a length $N$ signal $x[n]$,
\begin{align}
X[k]=\sum_{n=0}^{N-1}x[n]e^{-j \frac{2\pi}{N} nk} && \text{for $k = 0, 1, 2, \cdots N-1$}
\label{eq:dft}
\end{align}
The order of the implementation is $O(N)=N^2$. The following java code
can be used to implement the DFT in J-DSP. It is implemented without
any algorithmic speedup (i.e., it exactly mirrors equation
\ref{eq:dft}).  Example of raw DFT implementation:
	\begin{lstlisting}
public class MyFunction1
{
 public void myCode(double[]x1,double[]x2,double[]y1,double[]y2,
 	double[]b1,double[]a1,double[]b2,double[]a2,
 	double para1, double para2, double para3)
  {
  // x1, x2 - input at pin 0 and pin1
  // y1, y2 - output at pin 4 and pin5
  // raw DFT implementation
  double[] yImag;
  double[] yReal;
  yImag = new double[256];
  yReal = new double[256];
  double twoPiOverN = 2*Math.PI/256;
  for( int k = 0 ; k < 256 ; k++)
  {
    yReal[k] = 0;
    yImag[k] = 0;
    for( int n = 0 ; n < 256 ; n++)
    {
      yReal[k] += x1[n]*Math.cos(n*k*twoPiOverN);
      yImag[k] += -x1[n]*Math.sin(n*k*twoPiOverN);
    }
    y1[k] = Math.sqrt(yReal[k]*yReal[k]+yImag[k]*yImag[k]);
  }
 }
}
	\end{lstlisting}
Note that there is not a native support in java for complex numbers so
this arithmetic is written out explicitly in the code above. For
example, the equation $y=x\times e^{a}$ (where x is a real number)
must be explicitly written out using Euler's formula, and the real and
imaginary portions saved in separate variables, $y_{real}=x\times
\cos(a)$ and $y_{imag}=x\times \sin(a)$.

The FFT algorithm, on the other hand, can be used to reduce the
computation time of the DFT to $O(N)=N\log_2 N$ - a significant
speedup for even modest length signals.

\paragraph{Step 1.1}
Write some example code to multiply \emph{two} complex numbers,
$c=a\times b$. Remember to represent the variables by their real and
imaginary parts and save the real and imaginary parts in separate
variables.

\paragraph{Step 1.2}
Use the \block{UserDefinedFun} block to implement the FFT using loops
(i.e., do not use recursion). The first part of this code performs a
bit reversal on the input array. Use the following code to perform the
bit reversal:
% make more explict, bit rev vrbl
\begin{lstlisting}
// bit reversal
int  index;
int  N = 256;
double[] reversedX1 = new double[256];
// need to shift 32 bit integer to make it an 8 bit reversal
int shift = (32-(int)Math.round(Math.log(N)/Math.log(2)));
for( int n = 0 ; n < N ; n++){
  index = (Integer.reverse(n))>>>shift; //reverse and then shift down to 8 bit
  reversedX1[n] = x1[index];
}
\end{lstlisting}
Then iterate over the array $\log_2 N$ times! Remember to explicitly
carry out any complex arithmetic and take the magnitude of the output
array once the FFT is computed. Include a copy of the java code in
your report.

\paragraph{Step 1.3}
Check your results in J-DSP using the \block{FFT} block. Your results
should be identical. Take the FFT of a sinusoid with a frequency of
$\pi/4$ radians per second using your FFT implementation and the
\block{FFT} block provided in J-DSP. Include a screenshot of the
output plots.

\subsection{Using the DFT}
% switch over to spectrogram example with DTMF - organize introduction
% for ease of use over multiple steps
\paragraph{Step 2.1}
Create a sum of two sinusoids in J-DSP. Use the \block{FFT} block to
compute the FFT of the sum and then plot the FFT magnitude. Use a
frequency of $0.13\pi$ and $0.19\pi$ for the two sinusoids. Make sure
that the \option{pulsewidth} for each is set to 256. What does the
result look like? Does this make sense?

\paragraph{Step 2.2} At what index (or indices) does the FFT magnitude
reach its peak value(s)? What frequency (or frequencies) does this
correspond to?

\paragraph{Step 2.3} Change the frequencies of the sinusoids to
$0.13\pi$ and $0.14\pi$. Repeat steps 2.1 and 2.2. Do the results
still make sense?

\paragraph{Step 2.4} Replace the sum of two sinusoids in J-DSP with
the \block{DTMF} block under the \menu{Audio Effects} function
list. Press some of the keys in the \block{DTMF} block. Does the FFT
block show separate frequencies for each button?

\subsection{Spectrograms}
Comparing FFT graphs (as in the step 2.4) can be difficult to do. But
what if we could analyze the frequency content of a signal as a
function of time? That would make it easier to see differences in
frequency if a signal started changing (like a string of DTMF keys
pressed in turn). To do this we will need a new tool called the
\emph{Spectrogram}. The spectrogram is simply an algorithm for
computing the FFT of a signal at different times and plotting them as
a function of time. The spectrogram is computed in the following way:

\begin{enumerate}
\item A given signal is ``windowed.'' This means that we only take a
  certain number of points from the signal (for this example assume we
  are using a window of length 128 points). To start out, we take the
  first 128 points of the signal (points 0 through 127 of the input
  array).
\item Take the FFT of the window and save the it in a separate array.
\item Advance the window in time by a certain number of points. For
  instance we can advance the window by 64 points so that we now have
  a window of indices 64 through 191 from the input signal array.
\item Repeat steps 1-3, saving the FFT of each window, until there are
  no longer any points in the input array.
\item Form a 2-D matrix whose columns are the FFT magnitudes of each
  window (placed in chronological order). In this way, each row
  represents a certain frequency, each column represents a given
  instant in time, and the value of the matrix represents the
  magnitude of the FFT.
\end{enumerate}

The result is called a Spectrogram and is usually displayed as an RGB
image where blue represents small FFT magnitudes and red represents
larger FFT magnitudes (the \emph{Jet} colormap if you are familiar
with color visualizations). There is an art to choosing the correct
parameters of the spectrogram (i.e., window size, FFT size, how many
points to advance the FFT, etc.). Each parameter has tradeoffs for the
time and frequency resolution of the resulting spectrogram. For our
purposes here, we will not be concerned with these tradeoffs. Instead
we will be more interested in getting familiar with analysis using
spectrograms.

\paragraph{Step 3.1} Use the DTMF setup from step 2.4. Instead of
using the \block{FFT} block, use the \block{Spectrogram} block under
the \menu{Statistical DSP} function stack. Keep all parameters in the
\block{Spectrogram} set to the default. Set the \block{DTMF} block to
play five frames per key press. Press the number ``1.'' What does the
spectrogram look like? Are both frequencies present?

\paragraph{Step 3.2} Now set the \block{DTMF} block to
\option{Record}. Also set the \option{Resolution} parameter inside the
\block{Spectrogram} block to 4 or 8 (otherwise you may notice a lag
for updating the spectrogram - it can be considerably CPU intensive
for large input signals). The \option{Resolution} parameter controls
how many points the sliding window advances at each time step (larger
steps mean we take fewer FFTs). Press each key in the \block{DTMF}
block in turn. Does the spectrogram make it easier to judge the
frequency content of the keys? Include a screenshot for your report.

% LocalWords:  WebQ MATLAB DSP


\section*{Exercises}

\begin{ex}
  Which of the following sets are subspaces of $\R^3$? Explain.
  \begin{enumerate}
  \item $V_1=\set{\left.\vect{u}=\begin{mymatrix}{c}
          u_1 \\ u_2 \\ u_3
        \end{mymatrix}~\right\vert~ \abs{u_1} \leq 4}$.
  \item $V_2=\set{\left.\vect{u}=\begin{mymatrix}{c}
          u_1 \\ u_2 \\ u_3
        \end{mymatrix}~\right\vert~\text{$u_i\geq 0$ for each $i=1,2,3$}}$.
  \item $V_3=\set{\left.\vect{u}=\begin{mymatrix}{c}
          u_1 \\ u_2 \\ u_3
        \end{mymatrix}~\right\vert~ u_3+u_1=2u_2}$.
  \item $V_4=\set{\left.\vect{u}=\begin{mymatrix}{c}
          u_1 \\ u_2 \\ u_3
        \end{mymatrix}~\right\vert~ u_3\geq u_1}$.
  \item $V_5=\set{\left.\vect{u}=\begin{mymatrix}{c}
          u_1 \\ u_2 \\ u_3
        \end{mymatrix}~\right\vert~ u_3=u_1=0}$.
  \end{enumerate}

  \begin{sol}
    \begin{enumerate}
    \item
      No. We have $\begin{mymatrix}{r}
        1 \\ 0 \\ 0 \\ 0
      \end{mymatrix} \in V_1$ but $10\begin{mymatrix}{r}
        1 \\ 0 \\ 0 \\ 0
      \end{mymatrix} \notin V_1$.
    \item
      This is not a subspace. The vector $\begin{mymatrix}{r}
        1 \\ 1 \\ 1 \\ 1
      \end{mymatrix}$
      is in $V_2$. However, $(-1) \begin{mymatrix}{r}
        1 \\ 1 \\ 1 \\ 1
      \end{mymatrix}$ is not.
    \item This is a subspace. It contains the zero vector and is
      closed with respect to vector addition and scalar
      multiplication.
    \item
      This
      is not a subspace. The vector $\begin{mymatrix}{r}
        0 \\ 0 \\ 1 \\ 0
      \end{mymatrix}$ is in $V_4$. However $(-1) \begin{mymatrix}{r}
        0 \\ 0 \\ 1 \\ 0
      \end{mymatrix}  = \begin{mymatrix}{r}
        0 \\ 0 \\ -1 \\ 0
      \end{mymatrix}$ is not.
    \item This is a subspace. It contains the zero vector and is
      closed with respect to vector addition and scalar
      multiplication.
    \end{enumerate}
  \end{sol}
\end{ex}

\begin{ex}
  Let $\vect{w}\in \R^4$ be a given fixed vector. Let
  \begin{equation*}
    M=\set{\left.\vect{u}
        =\begin{mymatrix}{c}
          u_1 \\ u_2 \\ u_3 \\ u_4
        \end{mymatrix} \in \R^4~\right\vert~ \vect{w}\dotprod \vect{u}
      =0}.
  \end{equation*}
  Is $M$ a subspace of $\R^4$? Explain.
  \begin{sol}
    Yes, this is a subspace because it contains the zero vector and is
    closed with respect to vector addition and scalar
    multiplication. For example, if $\vect{u},\vect{v}\in M$, then
    $\vect{w}\dotprod\vect{u}=0$ and $\vect{w}\dotprod\vect{v}=0$,
    therefore $\vect{w}\dotprod(\vect{u}+\vect{v})=0$, therefore
    $\vect{u}+\vect{v}\in M$.
  \end{sol}
\end{ex}

\begin{ex}
  Let $\vect{w},\vect{v}$ be given vectors in $\R^4$ and define
  \begin{equation*}
    M=\set{\left.\vect{u}=\begin{mymatrix}{c}
        u_1 \\ u_2 \\ u_3 \\ u_4
      \end{mymatrix} \in \R^4
      ~\right\vert~
    \text{$\vect{w}\dotprod \vect{u}=0$ and $\vect{v}\dotprod \vect{u}=0$}}.
  \end{equation*}
  Is $M$ a subspace of $\R^4$? Explain.
  \begin{sol}
    Yes, this is a subspace.
  \end{sol}
\end{ex}

\begin{ex}
  In this exercise, we use scalars from the field $\Z_2$ of integers
  modulo $2$ instead of real numbers (see Section~\ref{sec:fields},
  ``Fields''). Which of the following sets are subspaces of
  $(\Z_2)^3$?
  \begin{enumerate}
  \item $V_1 = \set{
      \begin{mymatrix}{r} 1 \\ 1 \\ 0 \end{mymatrix},
      \begin{mymatrix}{r} 0 \\ 1 \\ 1 \end{mymatrix},
      \begin{mymatrix}{r} 1 \\ 0 \\ 1 \end{mymatrix},
      \begin{mymatrix}{r} 0 \\ 0 \\ 0 \end{mymatrix}
    }$.
  \item $V_2 = \set{
      \begin{mymatrix}{r} 1 \\ 1 \\ 1 \end{mymatrix},
      \begin{mymatrix}{r} 1 \\ 1 \\ 0 \end{mymatrix},
      \begin{mymatrix}{r} 1 \\ 0 \\ 0 \end{mymatrix},
      \begin{mymatrix}{r} 0 \\ 0 \\ 0 \end{mymatrix}
    }$.
  \item $V_3 = \set{
      \begin{mymatrix}{r} 0 \\ 0 \\ 0 \end{mymatrix}
    }$.
  \item $V_3 = \set{
      \begin{mymatrix}{r} 1 \\ 1 \\ 1 \end{mymatrix},
      \begin{mymatrix}{r} 1 \\ 0 \\ 0 \end{mymatrix},
      \begin{mymatrix}{r} 0 \\ 1 \\ 0 \end{mymatrix},
      \begin{mymatrix}{r} 0 \\ 0 \\ 1 \end{mymatrix}
    }$.
  \end{enumerate}
  \begin{sol}
    \begin{enumerate}
    \item $V_1$ is a subspace.
    \item $V_2$ is not a subspace: not closed under addition. For example,
      \begin{equation*}
        \begin{mymatrix}{r} 1 \\ 1 \\ 1 \end{mymatrix} \in V_2,\quad
        \begin{mymatrix}{r} 1 \\ 1 \\ 0 \end{mymatrix} \in V_2,\quad
        \mbox{but}\quad
        \begin{mymatrix}{r} 1 \\ 1 \\ 1 \end{mymatrix}
        + \begin{mymatrix}{r} 1 \\ 1 \\ 0 \end{mymatrix}
        = \begin{mymatrix}{r} 0 \\ 0 \\ 1 \end{mymatrix} \not\in V_2.
      \end{equation*}
    \item $V_3$ is a subspace.
    \item $V_4$ is not a subspace. For example, it does not contain
      the zero vector.
    \end{enumerate}
  \end{sol}
\end{ex}

\begin{ex}
  Suppose $V, W$ are subspaces of $\R^n$. Let $V\cap W$ be the set
  of all vectors that are in both $V$ and $W$. Show that $V\cap W$
  is also a subspace.
  \begin{sol}
    Because $\vect{0}\in V$ and $\vect{0}\in W$, we have
    $\vect{0}\in V\cap W$. To show that $V\cap W$ is closed under
    addition, assume $\vect{u},\vect{v}\in V\cap W$. Then
    $\vect{u},\vect{v}\in V$, and since $V$ is a subspace we have
    $\vect{u}+\vect{v}\in V$. Similarly $\vect{u},\vect{v}\in W$, and
    since $W$ is a subspace we have $\vect{u}+\vect{v}\in W$. It
    follows that $\vect{u}+\vect{v}$ is in both $V$ and $W$, and
    therefore in $V\cap W$. To show that $V\cap W$ is closed under
    scalar multiplication, assume $\vect{u}\in V\cap W$ and
    $k\in\R$. Then $\vect{u}\in V$, and therefore $k\vect{u}\in
    V$. Similarly $k\vect{u}\in W$, and therefore also
    $k\vect{u}\in V\cap W$.
  \end{sol}
\end{ex}

\begin{ex}
  Let $V$ be a subset of $\R^n$. Show that $V$ is a subspace if and
  only if it is non-empty and the following condition holds: for all
  $\vect{u},\vect{v}\in V$ and all scalars $a,b\in\R$,
  \begin{equation*}
    a\vect{u} + b\vect{v} \in V.
  \end{equation*}
\end{ex}

\begin{ex}
  Let $\vect{u}_1,\ldots,\vect{u}_k$ be vectors in $\R^n$, and let
  $S=\sspan\set{\vect{u}_1,\ldots,\vect{u}_k}$. Show that $S$ is the
  {\em smallest} subspace of $\R^n$ that contains
  $\vect{u}_1,\ldots,\vect{u}_k$.  Specifically, this means you have
  to show: if $V$ is any other subspace of $\R^n$ such that
  $\vect{u}_1,\ldots,\vect{u}_k\in V$, then $S\subseteq V$.
\end{ex}


\chapter{Short-Time Speech Processing}
\newpage

\section{Segmentations/Frames}
An underlying assumption of speech signal processing is that the speech signal is slowly-time-varying, i.e. it changes slowly with time. Based on this assuption, we can perform short-time signal processing by splitting speech signals into isolated segmentations or frames.

\subsection{Mathematical Framework of Short-Time Processing}
Short-time analysis is represented in a general form of
\[ Q_{\hat{n}} = \sum_{m=-\infty}^{+\infty}T(x[m]w[\hat{n}-m]) = \sum_{m=-\infty}^{+\infty}T(x[m])\tilde{w}[\hat{n}-m] \]
where $\tilde{w}[\hat{n}-m]$ is a sliding analysis window and $T()$ is the operation on input signal. $Q_{\hat{n}}$ represents the short-time representation of signal $\tilde{x}$ at time $\hat{n}$.
\begin{remark}
    It's actually a discrete-time convolution of $T(x[m])$ with $\tilde{w}[n]$
\end{remark}

\subsection{Length of Segments}
The shorter the segement, the less likely a signal will vary significantly over the segment duration (due to its slowly-changing nature), and thus tracking abrupt waveform changes is best for shorter segements.
However, parameters estimated from short may be highly variable because the data available for processing is small.

\subsection{Stepsize}
Typically, the window is moved in jumps of $R>1$ samples, which corresponds to downsampling the output of the signal by a factor of $R$. If the window is of length $L$, then we should choose $R<L$ so that each sample is included in at least one segement.

Typically, the analysis windows overlap by more than $50\%$ of the window length.

\subsection{Commonly Used Windows}
\begin{itemize}
    \item Rectangular Window
    \[ w_R[n] =
    \begin{cases}
        1 &(0 \le n \le L-1)\\
        0 &(\text{o.w.})
    \end{cases}
    \]
    \item Hamming Window
    \[ w_H[n] =
    \begin{cases}
        0.54-0.46\cos(2\pi n/(L-1)) &(0 \le n \le L-1)\\
        0 &(\text{o.w.})
    \end{cases}
    \]
\end{itemize}

\section{Short-Time Energy and Short-Time Magnitude}
\subsection{Energy of Signal}
\begin{definition}[Energy]
    The energy of a discrete-time signal is
    \[ E = \sum_{m=-\infty}^{+\infty}(x[n])^2 \]
\end{definition}
Useless because it does not give any time-dependent properties of a speech signal.

\subsection{Short-Time Energy}
\begin{definition}[Short-Time Energy]
    \[ E_{\hat{n}} = \sum_{m=-\infty}^{+\infty} (x[m]w[\hat{n}-m])^2 = \sum_{m=-\infty}^{+\infty} (x[m])^2\tilde{w}[\hat{n}-m]\]
\end{definition}
where $w[n]$ is the window applied to $x[n]$ before squaring, and $\tilde{w}[n]$ is the corresponding window that can be applied equivalently after squaring.

For an $L$-point window,
\[ E_{\hat{n}} = \sum_{m=\hat{n}-L+1}^{\hat{n}} (x[m]w[\hat{n}-m])^2 \]

$E_{\hat{n}}$ provides a basis for distinguishing voiced speech segments from unvoiced speech segments. Unvoiced segments have significantly smaller short-time energy compared with Voiced segments.

For high-quality speech signal (high signal-to-noise ratio), $E_{\hat{n}}$ can also be used to distinguish speech from silence.

\subsection{Short-Time Magnitude}
Short-time energy can be very sensitive to large signal levels. This can be addressed by taking square roots, or using the short-time magnitude.
\begin{definition}[Short-Time Magnitude]
    \[ M_{\hat{n}} = \sum_{m=-\infty}^{+\infty} |x[m]w[\hat{n}-m]| = \sum_{m=-\infty}^{+\infty} |x[m]|\tilde{w}[\hat{n}-m]\]
\end{definition}

\section{Short-Time Zero-Crossing Rate}
A \textbf{zero-crossing} is said to occur if successive waveform samples have different algebraic signs.
\begin{definition}[Short-Time Zero-Crossing Rate]
    \[ Z_{\hat{n}} = \frac{1}{2L_{eff}}\sum_{m=\hat{n}-L+1}^{\hat{n}} |\sgn(x[m])| - \sgn(x[m-1])| \tilde{w}[\hat{n}-m] \]
\end{definition}
Typically, the window used here is a rectangular window, so
\[ Z_{\hat{n}} = \frac{1}{2L}\sum_{m=\hat{n}-L+1}^{\hat{n}} |\sgn(x[m])| - \sgn(x[m-1])| \]

Voiced speech will have relatively low zero-crossing rate, and Unvoiced speech will have relatively high zero-crossing rate.

\section{Short-Time AutoCorrelation}
\subsection{AutoCorrelation}
\begin{definition}[AutoCorrelation]
    For determinisitc or aperiodic signals
    \[ \phi[k] = \sum_{m=-\infty}^{+\infty}x[m]x[m+k] \]
    For stationary random or periodic signals
    \[ \phi[k] = \lim_{N\to\infty}\frac{1}{2N+1}\sum_{m=-N}^{N}x[m]x[m+k] \]
\end{definition}
\begin{remark}
    AutoCorrelation highlights the period of signals because a local maximum is achieved at samples $0, \pm N, \pm 2N, \dots$, regardless of the time origin of the periodic signal.
\end{remark}
\begin{proposition}
    Properties of autocorrelation:
    \begin{itemize}
        \item For periodic signals:
        \[ \phi[k] = \phi[k+N] \]
        \item $\phi[k]$ = $\phi[-k]$
        \item $\phi[0] \le |\phi[k]|$ for all $k$
        \item $\phi[0]$ equals to the total energy for determinisitc signals and average power of random signals.
    \end{itemize}
\end{proposition}

\subsection{Short-Time AutoCorrelation}
\begin{definition}[Short-Time AutoCorrelation]
    \[ R_{\hat{n}}[k] = \sum_{m=-\infty}^{+\infty}(x[m]w[\hat{n}-m])(x[m+k]w[\hat{n}-k-m]) \]
\end{definition}
\begin{remark}
    Short-Time AutoCorrelation preserves the properties of autocorrelation in the previous section, and $R_{\hat{n}}[0]$ is equivalent to the short-time energy.
\end{remark}

\section{Short-Time Average Magnitude Difference Function}
The computation of autocorrelation involves considerable arithmetic.

Note that for a truly periodic signal,
\[ d[k] = x[n] -x[n-k] \]
would be $0$ whenever $k=0,\pm N, \pm 2N, \dots$. So it is reasonable to expect that $d[n]$ will be small at multiples of period for short segements of voiced speech.
\begin{definition}[Short-Time AMDF]
    \[ \gamma_{\hat{n}}[k] = \sum_{m=-\infty}^{+\infty} |x[\hat{n}+m]w_1[m] - x[\hat{n}+m-k]w_2[m-k] | \]
\end{definition}
\begin{remark}
    $\gamma_{\hat{n}}[k]$ would drop sharply near multiples of period.
\end{remark}

AutoCorrelation and AMDF are used to find the \textbf{pitch} (aka \textbf{foundamental frequency}) of the speech.

\section{Short-Time Fourier Transform}
\begin{definition}[Continuous Time STFT]
    \[ X(\hat{t}, \Omega) = \int_{-\infty}^{+\infty} w(\hat{t}- t)x(t)e^{-j\Omega t}\mathrm{d}t \]
\end{definition}
\begin{definition}[Discrete Time STFT]
    \[ X_{\hat{n}}(e^{j\omega}) = \sum_{m=-\infty}^{+\infty}w[\hat{n}-m]x[m]e^{-j\omega m} \]
\end{definition}
\begin{definition}[Spectrogram]
    A spectrogram is a gray-scale image whose $x$-axis is time and $y$-axis is frequency, and its colormap indicates the log amplitude.
\end{definition}
\begin{remark}
    If the band pass filter has wide bandwidth(300-900Hz), then the spectrogram has good temporal resolution but poor frequency resolution. On the other hand, if the band pass filter has narrow bandwidth(30-90Hz), then the spectrogram has poor temporal resolution but good frequency resolution.
\end{remark}