\subsection{Kendall rank correlation}

Concordant and discordant pairs

\(\tau = \dfrac{n_{concordant}-n_{discordant}}{\begin{pmatrix}n\\2\end{pmatrix}}\)


\subsection{Introduction}

A single-variable polynomial is an equation of the form:

\(\sum^n_{i=0} a_i x^i=0\)

For example:
\begin{itemize}
\item \(x=1\)
\item \(x^2=4\)
\item \(x^2-3x+2=0\)
\end{itemize}

\subsection{Degrees}

The degree of a polynomial is the highest-order term.

For example \(x^3+x=0\) has degree \(3\).

\subsection{Roots of single-variable polynomials}

A solution to a polynomial is a root.

For example \(1\) and \(2\) are roots of \(x^2-3x+2=0\)


\subsection{Finding an $LU$ factorization by inspection}

Which matrices have an $LU$ factorization? It turns out it is those whose
{\ef} can be achieved without switching rows. In other words matrices which
only involve using row operations of type 2 or 3 to obtain the {\ef}.

\begin{example}{An $LU$ factorization}{}
Find an $LU$ factorization of $A=\begin{mymatrix}{cccc}
1 & 2 & 0 & 2 \\
1 & 3 & 2 & 1 \\
2 & 3 & 4 & 0%
\end{mymatrix}$.
\end{example}

One way to find the $LU$ factorization%
\index{LU factorization!by inspection} is to simply look for it directly.
You need
\begin{equation*}
\begin{mymatrix}{cccc}
1 & 2 & 0 & 2 \\
1 & 3 & 2 & 1 \\
2 & 3 & 4 & 0%
\end{mymatrix} =\begin{mymatrix}{ccc}
1 & 0 & 0 \\
x & 1 & 0 \\
y & z & 1%
\end{mymatrix} \begin{mymatrix}{cccc}
a & d & h & j \\
0 & b & e & i \\
0 & 0 & c & f%
\end{mymatrix} .
\end{equation*}
Then multiplying these you get
\begin{equation*}
\allowbreak \begin{mymatrix}{cccc}
a & d & h & j \\
xa & xd+b & xh+e & xj+i \\
ya & yd+zb & yh+ze+c & yj+iz+f%
\end{mymatrix}
\end{equation*}
and so you can now tell what the various quantities equal. From the first
column, you need $a=1,x=1,y=2$. Now go to the second column. You need $
d=2,xd+b=3$ so $b=1,yd+zb=3$ so $z=-1$. From the third column, $h=0,e=2,c=6$.
Now from the fourth column, $j=2,i=-1,f=-5$. Therefore, an $LU$
factorization is
\begin{equation*}
\begin{mymatrix}{rrr}
1 & 0 & 0 \\
1 & 1 & 0 \\
2 & -1 & 1
\end{mymatrix} \begin{mymatrix}{rrrr}
1 & 2 & 0 & 2 \\
0 & 1 & 2 & -1 \\
0 & 0 & 6 & -5
\end{mymatrix} .
\end{equation*}
You can check whether you got it right by simply multiplying these two.


\chapter{Bayesian Inference}

\begin{ex}
  Let $f$ and $g$ be PDFs of normal distributions with means $\mu_f$ and $\mu_g$
  and variances $\sigma_f^2$ and $\sigma_g^2$ respectively. Note that since
  \begin{align*}
    \frac{(x-\mu_f)^2}{\sigma_f^2}
    +\frac{(x-\mu_g)^2}{\sigma_g^2}
     & =\frac{\sigma_g^2(x^2-2x\mu_f+\mu_f^2)+\sigma_f^2(x^2-2x\mu_g+\mu_g^2)}{\sigma_f^2\sigma_g^2}                                                                                \\
     & =\frac{(\sigma_f^2+\sigma_g^2)x^2-2(\sigma_g^2\mu_f+\sigma_f^2\mu_g) x+\sigma_g^2\mu_f^2+\sigma_f^2\mu_g^2}{\sigma_f^2\sigma_g^2}                                                                  \\
     & =\frac{x^2-2\frac{\sigma_g^2\mu_f+\sigma_f^2\mu_g}{\sigma_f^2+\sigma_g^2} x+\frac{\sigma_g^2\mu_f^2+\sigma_f^2\mu_g^2}{\sigma_f^2+\sigma_g^2}}{\frac{\sigma_f^2\sigma_g^2}{\sigma_f^2+\sigma_g^2}} \\
     & =\frac{\left(x-\frac{\sigma_g^2\mu_f+\sigma_f^2\mu_g}{\sigma_f^2+\sigma_g^2}\right)^2}{\frac{\sigma_f^2\sigma_g^2}{\sigma_f^2+\sigma_g^2}} +C,
  \end{align*}
  for some $C$ not depending on $x$, we have
  \begin{align*}
    f(x)g(x)
     & \propto \exp\left\{-\frac{(x-\mu_f)^2}{2\sigma_f^2}\right\}
    \exp\left\{-\frac{(x-\mu_g)^2}{2\sigma_g^2}\right\}                                                                                \\
     & \propto \exp\left\{-\frac{1}{2}\frac{\left(x-\frac{\sigma_g^2\mu_f+\sigma_f^2\mu_g}{\sigma_f^2+\sigma_g^2}\right)^2}{\frac{\sigma_f^2\sigma_g^2}{\sigma_f^2+\sigma_g^2}}\right\},
  \end{align*}
  which, by inspection, is proportional to the PDF of a normal distribution with
  mean
  \[
    \frac{\sigma_g^2\mu_f+\sigma_f^2\mu_g}{\sigma_f^2+\sigma_g^2}
  \]
  and variance
  \[
    \frac{\sigma_f^2\sigma_g^2}{\sigma_f^2+\sigma_g^2}.
  \]

  Consider the product of $n$ normal PDFs each with mean $X_i$ and variance
  $\sigma^2$. Since we can group it as the product of the first $n-1$ PDFs and
  the last PDF, and since by the previous argument, the product of two normal
  PDFs is proportional to a normal PDF, it follows by induction that so is the
  product of $n$ PDFs. It only remains to figure out the mean and variance.
  Note that for $n=2$ we have
  \[
    \sigma_2^2
    =\frac{\sigma^2\sigma^2}{\sigma^2+\sigma^2}=\frac{\sigma^2}{2}
    \text{ and }
    \mu_2
    =\frac{X_1\sigma^2+X_2\sigma^2}{\sigma^2+\sigma^2}
    =\frac{X_1+X_2}{2}
    =\Xbar_2.
  \]
  Suppose that for $n-1$ we have
  \[
    \sigma_{n-1}^2=\frac{\sigma^2}{n-1}
    \text{ and }
    \mu_{n-1}=\Xbar_{n-1},
  \]
  and note that then
  \[
    \sigma_n^2
    =\frac{\sigma_{n-1}^2\sigma^2}{\sigma_{n-1}^2+\sigma^2}
    =\frac{\frac{\sigma^2}{n-1}\sigma^2}{\frac{\sigma^2}{n-1}+\sigma^2}
    =\frac{\sigma^4}{n-1} / \frac{n\sigma^2}{n-1}
    =\frac{\sigma^2}{n}
    =\se^2,\text{ and }
  \]
  \[
    \mu_n
    =\frac{\Xbar_{n-1}\sigma^2+X_n\sigma_{n-1}^2}{\sigma_{n-1}^2+\sigma^2}
    =\frac{\Xbar_{n-1}\sigma^2+X_n\frac{\sigma^2}{n-1}}{\frac{\sigma^2}{n-1}+\sigma^2}
    =\frac{(\sum_{i=1}^n X_i)\sigma^2}{n-1}/\frac{n\sigma^2}{n-1}
    =\Xbar_n.
  \]
  The result follows by induction.

  Next, consider the product of $n$ normal PDFs with mean $X_i$ and
  variance $\sigma^2$ respectively, and a normal PDF with mean $a$ and variance
  $b^2$. The product of the first $n$ PDFs is proportional to an
  $N(\Xbar, \se^2)$ PDF, and therefore the full product is
  proportional to an $N(\overline{\theta}, \tau^2)$ PDF, where
  \[
    \tau^2=\frac{\se^2b^2}{\se^2+b^2},\text{ and}
  \]
  \begin{align*}
    \overline{\theta}
     & =\frac{\Xbar b^2+a \se^2}{\se^2+b^2}                        \\
     & =\frac{\Xbar b^2}{\se^2+b^2}+\frac{a \se^2}{\se^2+b^2}      \\
     & =\Xbar\frac{\frac{1}{\se^2}}{\frac{1}{\se^2}+\frac{1}{b^2}}
    +a\frac{\frac{1}{b^2}}{\frac{1}{\se^2}+\frac{1}{b^2}}          \\
     & =\Xbar\frac{\frac{1}{\se^2}}{\frac{1}{\se^2}+\frac{1}{b^2}}
    +a\left[1-\frac{\frac{1}{\se^2}}{\frac{1}{\se^2}+\frac{1}{b^2}}\right].
  \end{align*}

  Finally, let $X_1,\ldots,X_n\sim N(\theta,\sigma^2)$, where $\sigma^2$ is
  known. We take a prior $\theta\sim N(a, b^2)$, and compute the
  posterior for $\theta$. Note that
  \[
    f(\theta\,|\, X^n)
    \propto \L(\theta\,|\, X^n)f(\theta)
    = f(\theta)\prod_{i=1}^n\L(\theta\,|\, X_i)
    = f(\theta)\prod_{i=1}^n f_i(\theta),
  \]
  where $f_i$ is the PDF for an $N(X_i,\sigma^2)$ distribution. However, this is
  precisely the setting of our last proposition and therefore the result
  follows.
\end{ex}

\begin{ex}~
  \inputminted{python}{../code/11-02.py}
  \inputminted{text}{../output/11-02.txt}

  \begin{enumerate}[(a)]
    \item[(b)] Let $f(\mu)=1$. Recall that the posterior is proportional to the
          product of the likelihood and the prior and that we therefore have
          \begin{align*}
            f(\mu\,|\,x^n)\propto \prod_{i=1}^n\frac{1}{\sqrt{2\pi}}
            \exp\left\{-\frac{(x_i-\mu)^2}{2}\right\},
          \end{align*}
          and therefore, by comparison to the probability density function of a
          multivariate normal distribution,
          \begin{align*}
            f(\mu\,|\,x^n)=
            (2\pi)^{-n/2}
            \exp\left\{-\frac{\sum_{i=1}^n(x_i-\mu)^2}{2}\right\}.
          \end{align*}

          \begin{figure}[H]
            \centering
            \includegraphics[scale=0.8]{../images/11-02b}
            \caption{Plot of the posterior density of $\mu$.}
          \end{figure}
    \item[(c)]~
          \begin{figure}[H]
            \centering
            \includegraphics[scale=0.8]{../images/11-02c}
            \caption{Histogram of $1000$ simulated values drawn from the
              posterior density of $\mu$.}
          \end{figure}
    \item[(d)] Let $\theta=e^\mu$. We then have
          \begin{align*}
            \P{\Theta\leq \theta\,|\,x^n}
             & =\P{e^\mu\leq \theta\,|\,x^n}                            \\
             & =\P{\mu\leq \log{\theta}\,|\,x^n}                        \\
             & =\int_{-\infty}^{\log{\theta}}\!f(\mu\,|\,x^n)\,\d{\mu},
          \end{align*}
          and therefore, by differentiating under the integral sign,
          \[
            f(\theta\,|\,x^n)
            =\frac{\d}{\d\theta}\P{\Theta\leq \theta\,|\,x^n}
            =\frac{1}{\theta}(2\pi)^{-n/2}
            \exp\left\{-\frac{\sum_{i=1}^n(x_i-\log{\theta})^2}{2}\right\}.
          \]
          \begin{figure}[H]
            \centering
            \includegraphics[scale=0.54]{../images/11-02d}
            \caption{A histogram of $10,000$ simulated values of $\theta$ drawn
              using the posterior density of $\mu$ (left) and a plot of the
              exact analytically computed posterior distribution of $\theta$
              (right).}
          \end{figure}
    \item[(e)] As per the output of the code giving in the listing at the start
          of this problem, a 95\% posterior interval for $\mu$ is given by
          $[4.754, 5.146]$.
    \item[(f)] From the output of the code listing at the top, a 95\%
          posterior interval for $\theta$ is given by $[116.004, 171.681]$. Note
          that we are asked for a confidence interval, but per Theorem
          11.5 we may approximate it with a Bayesian posterior interval instead.
  \end{enumerate}
\end{ex}

\begin{ex}
  Note that
  \[
    \L(\theta, x_i)=\begin{cases}
      0          & \theta < x_i,     \\
      x_i/\theta & \text{otherwise},
    \end{cases}
  \]
  and that therefore
  \[
    \L(\theta, x^n)=\begin{cases}
      0                                  & \theta < x_{(n)}, \\
      \frac{\prod_{i=1}^n x_i}{\theta^n} & \text{otherwise}.
    \end{cases}
  \]
  Thus, if $f(\theta)\propto 1/\theta$,
  \[
    f(\theta\,|\, x^n)\propto \begin{cases}
      0                                      & \theta < x_{(n)}, \\
      \frac{\prod_{i=1}^n x_i}{\theta^{n+1}} & \text{otherwise},
    \end{cases}
  \]
  or, since
  \[
    \int_{x_{(n)}}^\infty \frac{\prod_{i=1}^n x_i}{\theta^{n+1}}\,\d\theta
    =\frac{1}{n}\prod_{i=1}^n\frac{x_i}{x_{(n)}},
  \]
  \[
    f(\theta\,|\, x^n)=\frac{n(x_{(n)})^n}{\theta^{n+1}}I_{[x_{(n)},\infty)}(\theta).
  \]
\end{ex}

\begin{ex}~
  \begin{enumerate}[(a)]
    \inputminted{python}{../code/11-04.py}
    \inputminted{text}{../output/11-04.txt}
    \item By Exercise 9.7, the MLE for $\tau$ is given by
          \[
            \tauhat = \phat_1 -\phat_2=X_1/n_1-X_2/n_2,
          \]
          with
          \[
            \sehat=\sqrt{
              \frac{\phat_1(1-\phat_1)}{n_1}
              +\frac{\phat_2(1-\phat_2)}{n_2}.
            }
          \]
          In particular, we have that the MLE for $\tau$ is $0.2$. The estimated
          standard error is $0.08944$, and a 90\% confidence interval
          is given by $[0.05288, 0.3471]$.
    \item From the code listing given at the start of the problem, we can see
          that a 90\% confidence interval for $\tau$ using the parametric
          bootstrap method is given by $[0.06, 0.34]$.
    \item Using the prior $f(p_1,p_2)=1$, it follows that the posterior mean of
          $\tau$ is $0.1953$, and a 90\% posterior confidence interval is given
          by $[0.04632, 0.3393]$.
    \item Let
          \[
            \psi=g(p_1,p_2)=\log\left(
            \left(\frac{p_1}{1-p_1}\right)\div
            \left(\frac{p_2}{1-p_2}\right)
            \right),
          \]
          and note that then
          \begin{align*}
            \nabla g=\begin{pmatrix}
              \frac{1}{p_1-p_1^2} \\
              \frac{1}{p_2^2-p_2}
            \end{pmatrix}.
          \end{align*}
          By the equivariance of the MLE, $\psihat=g(\phat_1,\phat_2)$, and by
          the multiparameter delta method
          \begin{align*}
            \sehat(\psihat)
             & =\sqrt{
              \begin{pmatrix}
                \frac{1}{\phat_1-\phat_1^2} &
                \frac{1}{\phat_2^2-\phat_2}
              \end{pmatrix}
              \begin{pmatrix}
                \frac{\phat_1(1-\phat_1)}{n_1} & 0                              \\
                0                              & \frac{\phat_2(1-\phat_2)}{n_2}
              \end{pmatrix}
              \begin{pmatrix}
                \frac{1}{\phat_1-\phat_1^2} \\
                \frac{1}{\phat_2^2-\phat_2}
              \end{pmatrix}
            }
            =\sqrt{
              \frac{1}{n_1(\phat_1-\phat_1^2)}+
              \frac{1}{n_2(\phat_2^2-\phat_2)}
            }.
          \end{align*}
          In particular, we have that the MLE of $\psi$ is $0.9808$. The
          estimated standard error is $0.2041$, and a 90\% confidence interval
          for $\psi$ is given by $[0.6451, 1.317]$.
    \item Under the prior $f(p_1,p_2)=1$, the posterior mean of $\psi$ is
          $0.9693$, a 90\% posterior confidence interval is given by
          $[0.2244, 1.737]$.
  \end{enumerate}
\end{ex}

% 5
\begin{ex}~
  \inputminted{python}{../code/11-05.py}
  \begin{figure}[H]
    \centering
    \includegraphics[scale=0.57]{../images/11-05a}
    \caption{
      Plots of the prior distributions of $p$ given by
      $\text{Beta}(\alpha, \alpha)$ distributions for different values of
      $\alpha$. Note that in all cases, the distribution is centered at $1/2$,
      but becomes more sharply peaked as $\alpha$ increases.
    }
  \end{figure}
  \begin{figure}[H]
    \centering
    \includegraphics[scale=0.57]{../images/11-05b}
    \caption{
      Plots of the different posterior distributions for $p$ given the prior
      distributions from the previous figure. Note that the larger $\alpha$ is,
      the less sensitive the posterior is to the data.
    }
  \end{figure}
\end{ex}

\begin{ex}
  Let $X_1,\ldots,X_n\sim\text{Poisson}(\lambda)$.
  \begin{enumerate}[(a)]
    \item Recall that if $\lambda\sim\text{Gamma}(\alpha,\beta)$,
          \[
            f(\lambda)=\frac{1}{\beta^\alpha\Gamma(\alpha)}\lambda^{\alpha-1}e^{-\lambda/\beta},
          \]
          and that therefore
          \begin{align*}
            f(\lambda\,|\,x^n)
             & \propto \left(\prod_{i=1}^n e^{-\lambda}\frac{\lambda^{x_i}}{x_i!}\right)
            \frac{1}{\beta^\alpha\Gamma(\alpha)}\lambda^{\alpha-1}e^{-\lambda/\beta}     \\
             & \propto \lambda^{\alpha-1+\sum_{i=1}^nx_i}e^{-\lambda(\beta n+1)/\beta},
          \end{align*}
          and hence
          \[
            f(\lambda\,|\,x^n)
            =\frac{1}{\beta'^{\alpha'}\Gamma(\alpha')}\lambda^{\alpha'-1}e^{-\lambda/\beta'},
          \]
          where
          \[
            \alpha'=\alpha+\sum_{i=1}^nx_i,\text{ and }
            \beta'=\frac{\beta}{n\beta +1}.
          \]
          Note that the mean of a $\text{Gamma}(\alpha', \beta')$ distribution
          is at $\alpha'\beta$, and that therefore the posterior mean is
          \[
            \left(\alpha+\sum_{i=1}^nx_i\right)
            \left(\frac{\beta}{n\beta +1}\right).
          \]
    \item Recall that for a $\text{Poisson}(\lambda)$ distribution,
          \[
            \ell_n(\lambda)=\sum_{i=1}^n-\lambda+x_i\log(\lambda)+\log(x_i!),
          \]
          and therefore
          \[
            \frac{\d\ell_n(\lambda)}{\d\lambda}
            =\sum_{i=1}^n-1+x_i/\lambda,
            \text{ and  }
            \frac{\d^2\ell_n(\lambda)}{\d\lambda^2}
            =-\sum_{i=1}^n x_i/\lambda^2.
          \]
          Therefore, $I(\lambda)=n/\lambda$, and the Jeffreys' prior
          $f(\lambda)\propto 1/\sqrt{\lambda}$. Hence,
          \begin{align*}
            f(\lambda\,|\,x^n)
             & \propto \left(\prod_{i=1}^n e^{-\lambda}\frac{\lambda^{x_i}}{x_i!}\right)\frac{1}{\sqrt{\lambda}} \\
             & \propto \lambda^{-1/2+\sum_{i=1}^n x_i}e^{-\lambda/n^{-1}},
          \end{align*}
          and therefore the posterior is given by
          \[
            f(\lambda\,|\,x^n)
            =\frac{1}{n^{-(1/2+\sum_{i=1}^nx_i)}\Gamma(1/2+\sum_{i=1}^nx_i)}\lambda^{\sum_{i=1}^nx_i+1/2-1}e^{-\lambda/n^{-1}}.
          \]
  \end{enumerate}
\end{ex}

\begin{ex}
  Let
  \[
    \psihat=\frac{1}{n}\sum_{i=1}^n\frac{Y_iR_i}{\xi_{X_i}}.
  \]
  Then
  \begin{align*}
    \E{\psihat}
     & =\E{\frac{1}{n}\sum_{i=1}^n\frac{Y_iR_i}{\xi_{X_i}}}               \\
     & =\frac{1}{n}\sum_{i=1}^n\E{\frac{Y_iR_i}{\xi_{X_i}}}               \\
     & =\frac{1}{n}\sum_{i=1}^n\E{\cE{\frac{Y_iR_i}{\xi_{X_i}}}{Y_i,X_i}} \\
     & =\frac{1}{n}\sum_{i=1}^n\E{\frac{Y_i}{\xi_{X_i}}\cE{R_i}{Y_i,X_i}} \\
     & =\frac{1}{n}\sum_{i=1}^n\E{Y_i}                                    \\
     & =\E{Y_i}                                                           \\
     & =\psi,
  \end{align*}
  and since $\delta\leq \xi_{X_i}\leq 1-\delta$,
  \begin{align*}
    \var{\frac{Y_iR_i}{\xi_{X_i}}}
     & =\E{\frac{Y_i^2R_i^2}{\xi^2_{X_i}}}
    -\left[\E{\frac{Y_iR_i}{\xi^2_{X_i}}}\right]^2 \\
     & =\E{\frac{Y_iR_i}{\xi^2_{X_i}}}
    -\left[\E{\frac{Y_iR_i}{\xi_{X_i}}}\right]^2   \\
     & \leq\frac{1}{\delta^2}\E{Y_iR_i}
    -\frac{1}{(1-\delta)^2}\E{Y_iR_i}^2            \\
     & \leq\frac{1}{\delta^2},
  \end{align*}
  and therefore
  \[
    \var{\psihat}
    =\frac{1}{n}\sum_{i=1}^n\var{\frac{Y_iR_i}{\xi_{X_i}}}
    \leq\frac{1}{n\delta^2}.
  \]
\end{ex}

\begin{ex}
  We are testing the hypothesis $H_0:\mu=0$ versus $H_1:\mu\neq 0$. We take the
  priors $\P{H_0}=\P{H_1}=1/2$, and under $H_1$ we take the prior
  $\mu\sim N(0, b^2)$. Recall from Section 11.8 that then
  \begin{align*}
    \cP{H_0}{X=x}
    =\frac{\L(0)}{\L(0)+\int_{-\infty}^\infty \L(\mu)f(\mu)\,\d{\mu}}.
  \end{align*}
  We have
  \begin{align*}
    \int_{-\infty}^\infty \L(\mu)f(\mu)\,\d\mu
     & =\int_{-\infty}^\infty \frac{1}{\sqrt{2\pi}}\exp\left\{-\frac{(x-\mu)^2}{2}\right\}
    \frac{1}{\sqrt{2\pi b^2}}\exp\left\{-\frac{\mu^2}{2b^2}\right\}\,\d\mu                 \\
     & =\frac{1}{\sqrt{2\pi}}
    \int_{-\infty}^\infty\frac{1}{\sqrt{2\pi b^2}}
    \exp\left\{-\frac{(x-\mu)^2}{2}-\frac{\mu^2}{2b^2}\right\}\,\d\mu                      \\
     & =\frac{1}{\sqrt{2\pi(b^2+1)}}
    \exp\left\{-\frac{x^2}{2(b^2+1)}\right\}
    \int_{-\infty}^\infty\frac{\sqrt{b^2+1}}{\sqrt{2\pi b^2}}
    \exp\left\{
    -\frac{\left(\mu-\frac{xb^2}{b^2+1}\right)^2}{2b^2/(b^2+1)}
    \right\}\,\d\mu                                                                        \\
     & =\frac{1}{\sqrt{2\pi(b^2+1)}}\exp\left\{-\frac{x^2}{2(b^2+1)}\right\},
  \end{align*}
  since
  \begin{align*}
    b^2(x-\mu)^2+\mu^2
     & =(b^2+1)\mu^2-2xb^2\mu+b^2x^2                                              \\
     & =(b^2+1)\left(\mu^2-\frac{2xb^2}{b^2+1}\mu+\frac{x^2b^4}{(b^2+1)^2}\right)
    -\frac{x^2b^4}{b^2+1}+b^2x^2                                                  \\
     & =(b^2+1)\left(\mu-\frac{xb^2}{b^2+1}\right)^2
    +\frac{-x^2b^4+b^4x^2+b^2x^2}{b^2+1}                                          \\
     & =(b^2+1)\left(\mu-\frac{xb^2}{b^2+1}\right)^2
    +\frac{b^2x^2}{b^2+1}.
  \end{align*}

  Therefore,
  \begin{align*}
    \cP{H_0}{X=x}
     & =\frac{\L(0)}{\L(0)+\int_{-\infty}^\infty \L(\mu)f(\mu)\,\d{\mu}}              \\
     & =\frac{\frac{1}{\sqrt{2\pi}}\exp\left\{-\frac{x^2}{2}\right\}}{
      \frac{1}{\sqrt{2\pi}}\exp\left\{-\frac{x^2}{2}\right\}
      +\frac{1}{\sqrt{2\pi(b^2+1)}}\exp\left\{-\frac{x^2}{2(b^2+1)}\right\}
    }                                                                                \\
     & =\frac{\exp\left\{-\frac{x^2}{2}\right\}}{
      \exp\left\{-\frac{x^2}{2}\right\}
      +\frac{1}{\sqrt{b^2+1}}\exp\left\{-\frac{x^2}{2(b^2+1)}\right\}}                \\
     & =\frac{\sqrt{b^2+1}}{\sqrt{b^2+1}+\exp\left\{\frac{x^2b^2}{2(b^2+1)}\right\}}.
  \end{align*}
  The $p$-value of the Wald test is given by
  \[
    \P{|Z|>|x|}=2\Phi(-|x|).
  \]

  \inputminted{python}{../code/11-08a.py}
  \begin{figure}[H]
    \centering
    \includegraphics[scale=0.54]{../images/11-08a}
    \caption{Comparison of the posterior probability of $H_0$ for different
      values of $b$ and the Wald test $p$-value as a function of $x$.}
  \end{figure}

  If instead we have a sample of size $n$,
  \begin{align*}
     & \int_{-\infty}^\infty \L(\mu)f(\mu)\,\d\mu                                                                       \\
     & \,=\int_{-\infty}^\infty\prod_{i=1}^n\left[\frac{1}{\sqrt{2\pi}}\exp\left\{-\frac{(x_i-\mu)^2}{2}\right\}\right]
    \frac{1}{\sqrt{2\pi b^2}}\exp\left\{-\frac{\mu^2}{2b^2}\right\}\,\d\mu                                              \\
     & \, =(2\pi)^{-n/2}(b^2n+1)^{-1/2}\exp\left\{-\frac{\beta-\alpha^2/(b^2n+1)}{2}\right\}
    \int_{-\infty}^\infty \frac{\sqrt{b^2n+1}}{\sqrt{2\pi b^2}}\exp\left\{
    -\frac{\left(\mu-\frac{b^2\alpha}{b^2n+1}\right)^2}{2b^2/(b^2n+1)}
    \right\}\,\d{\mu}                                                                                \\
     & \, =(2\pi)^{-n/2}(b^2n+1)^{-1/2}\exp\left\{-\frac{\beta(b^2n+1)-\alpha^2}{2(b^2n+1)}\right\},
  \end{align*}
  since
  \begin{align*}
    b^2\sum_{i=1}^n(x_i-\mu)^2+\mu^2
     & =(b^2n+1)\mu^2-2b^2\left(\sum_{i=1}^nx_i\right)\mu+b^2\left(\sum_{i=1}^n x_i^2\right)                                        \\
     & =(b^2n+1)\left(\mu^2-2\frac{b^2\alpha}{b^2n+1}\mu+\frac{b^4\alpha^2}{(bn^2+1)^2} \right)-\frac{b^4\alpha^2}{b^2n+1}+b^2\beta \\
     & =(b^2n+1)\left(\mu-\frac{b^2\alpha}{b^2n+1}\right)^2+b^2\left(\beta-\frac{b^2\alpha^2}{b^2n+1}\right)                        \\
  \end{align*}
  where $\alpha=\sum_{i=1}^n x_i$ and $\beta=\sum_{i=1}^n x_i^2$.

  Therefore,
  \begin{align*}
    \cP{H_0}{X^n=x^n}
     & =\frac{\L(0)}{\L(0)+\int_{-\infty}^\infty \L(\mu)f(\mu)\,\d{\mu}}                        \\
     & =\frac{(2\pi)^{-n/2}\exp\left\{-\frac{1}{2}\beta\right\}}{
      (2\pi)^{-n/2}\exp\left\{-\frac{1}{2}\beta\right\}
      +(2\pi)^{-n/2}(b^2n+1)^{-1/2}\exp\left\{-\frac{\beta (b^2n+1)-b^2\alpha^2}{2(b^2n+1)}\right\}
    }                                                                                \\
     & =\frac{\sqrt{b^2n+1}}{\sqrt{b^2n+1}+\exp\left\{\frac{b^2\alpha^2}{2(b^2n+1)}\right\}}    \\
     & =\frac{\sqrt{b^2n+1}}{\sqrt{b^2n+1}+\exp\left\{\frac{b^2n^2\Xbar^2}{2(b^2n+1)}\right\}},
  \end{align*}
  while the Wald test $p$-value for a sample of size $n$ is
  \[
    \P{|Z|>\sqrt{n}|\Xbar|}
    =2\Phi\left(-\sqrt{n}|\Xbar|\right).
  \]

  \inputminted{python}{../code/11-08b.py}
  \begin{figure}[H]
    \centering
    \includegraphics[scale=0.545]{../images/11-08b}
    \caption{Comparison of the posterior probability of $H_0$ for different
      values of $b$ and the Wald test $p$-value as a function of the sample
      average and for different sample sizes. Note that as $n$ increases, a
      region forms in which the $p$-value is near $0$, but the conditional
      probability of $H_0$ is near $1$.}
  \end{figure}

\end{ex}


\chapter{Self-organizing Feature Maps}
Neurons in layers excite its closest neighbours via \emph{lateral connections} and inhibit the distance neighbours. \emph{Lateral} interactions create \emph{competition and cooperation} among them.

\section{Self-Organizing Neural Networks}
\begin{figure}[!h]
\centering
\includegraphics[width=8cm]{chapter9_1}
\end{figure}

\section{Similarity Measures}
Euclidean distance:
\begin{equation*}
\begin{split}
distance &= \|\boldsymbol{x}_i - \boldsymbol{x}_j \| \\
&= \sqrt{(\boldsymbol{x}_i - \boldsymbol{x}_j)^{T} (\boldsymbol{x}_i - \boldsymbol{x}_j)} \\
&= \sqrt{\sum_k (x_{ik} - x_{jk})^{2}}
\end{split}
\end{equation*}
Cosine similarity:
$$\cos \theta = \frac{\boldsymbol{x}_i^{T} \boldsymbol{x}_j}{\|\boldsymbol{x_i}\|\|\boldsymbol{x_j}\|}$$

\section{Kohonen's Neural Network}
Synaptic input to the output layer:
$$\mathbf{u = Wx}$$
The ouput of the network:
$$\mathbf{y = f(u)}$$

\subsection{Winner-Takes-All Rule}
If winner is the $m$ th neuron, then:
$$m = \arg\!\max_{i=1...M} u_i$$
The output of the neuron:
$$y_m = 1$$
The output of other neurons:
$$y_i = 0$$

\subsection{Kohonen Learning Rule}
Prior to learning, all the weight vectors are randomly initialized and then normalized:
$$\mathbf{\hat{w}} = \frac{\mathbf{w}_j}{\|\mathbf{w}_j\|}$$
Next, \emph{find the winner} and update its weight:
\begin{equation*}
\begin{split}
\mathbf{w}_m^{t+1} &= \mathbf{\hat{w}}_m^{t} + \Delta \mathbf{\hat{w}}_m^{t} \\
&= \mathbf{\hat{w}}_m^{t} + \alpha^{t}(\mathbf{x} - \mathbf{\hat{w}}_m^{t})
\end{split}
\end{equation*}
The new weight is then normalized.

\subsubsection{Neurons with the highest synaptic inputs are winner}
We want to find the closest weight vector $w$ to current input $x$:
$$\| \mathbf{x} - \mathbf{\hat{w}}_m \| = \!\min_{j=1...M} \| \mathbf{x} - \mathbf{\hat{w}}_j \|$$
\begin{equation*}
\begin{split}
\| \mathbf{x} - \mathbf{\hat{w}}_m \| &= \Big[(\mathbf{x} - \mathbf{\hat{w}}_m)^{T}(\mathbf{x} - \mathbf{\hat{w}}_m) \Big]^{\frac{1}{2}} \\
&= \Big(\mathbf{x^{T}x - 2\hat{w}_{m}x + 1} \Big)^{\frac{1}{2}}
\end{split}
\end{equation*}
Searching for the closest weight vector is equivalent to:
\begin{equation*}
\begin{split}
\mathbf{\hat{w}}_m^{T} \mathbf{x} &= \!\max_{j=1...M} \mathbf{\hat{w}}_j^{T} \mathbf{x} \\
u_m &= \!\max_{j=1...M} u_j
\end{split}
\end{equation*}

\subsubsection{Impact of the Learning Rule}
Kohonen learning rule should increase the chances of winning of neuron $m$ after each iteration. It means that:
$$u_m^{new} > u_m^{old}$$
Proof:
$$u_m^{new} - u_m^{old} = \alpha (\| x \|^{2} - \| \hat{w}_m^{T} \| \|x\|\cos \theta)$$
But $\cos \theta \le 1$, $1 > \alpha > 0$ and $\|\hat{w}\| = 1$, assume $\|x\| = 1$, so:
$$u_m^{new} > u_m^{old}$$
That's why we need to normalize weight after each iterations.

\subsubsection{Geometrical Interpretation of the Learning Rule}
$\Delta \mathbf{\hat{w}}_m$ is a rotation of $\mathbf{\hat{w}}_m$ toward input vector $\mathbf{\hat{x}}$ without significant length change.
\begin{figure}[!h]
\centering
\includegraphics[width=5cm]{chapter9_2}
\end{figure}

\subsubsection{Initialization of the Weights}
Convex combination:
$$\mathbf{w}_j^{0} = \frac{1}{\sqrt{n}} (1 1 ... 1)^{T}$$

\subsubsection{Limitation of Basic Kohonen Network}
\begin{enumerate}
\item Linearly non-separable pattern cannot be efficiently handled
\item Not always successful for linearly separable patterns
\item Neurons with initial weights far from any input vector may never win
\end{enumerate}

\section{Kohonen Self-Organizing Feature Maps}
In the training phase, we update weight vector of winning neuron $m$ \emph{and its neighbours} according to the radius $N_m^{t}$. The weight of winning neuron is updated using the method described in previous section. The weight of neighbours is updated by:
$$\Delta \mathbf{w}_j^{new} = \alpha(j,t) (\mathbf{x} - \mathbf{w}_j^{old})$$
$$\forall j \in N_m(t)$$

\section{Vector Quantization}
Vector quantizer $Q$ is a mapping:
$$Q:\mathbf{R}^{n} \rightarrow C$$
\begin{center}where $C={c_1 c_2 \ldots c_M}$ is a set of vectors or codewords\end{center}
For an input vector $\mathbf{x} \in R^{n}$ the output of quantizer ca be implemented in competitive neural network
$$Q(\mathbf{x}) = c_m$$
$$if\ \|\mathbf{x-c_m} \| = \!\min_{i} \| \mathbf{x-c_i} \|$$

\subsection{LVQ1}
LVQ is a supervised form of algorithm for Vector Quantization
\begin{enumerate}
\item The weight vector $\mathbf{w_j}$ are defined using SOFM
\item For each input pattern, find winning neuron $m$
\item If the desired winner for $\mathbf{x}$ is $q$
$$
\mathbf{w}_m^{new} =
\begin{cases}
\mathbf{w}_m + \alpha (\mathbf{x} - \mathbf{w}_m) & m = q \\
\mathbf{w}_m - \alpha (\mathbf{x} - \mathbf{w}_m) & m \ne q
\end{cases}
$$
$$w_j^{new} = w_j, \forall j \ne m$$
\end{enumerate}

\subsection{LVQ2}
Same as LVQ1, but learning is applied only if:
\begin{enumerate}
\item The desired winner is not the winner, $m \ne q$
\item The second closest prototype vector is $w_{m*}$ the correct class
\item The input vector is close to the boundary between $\mathbf{w}_m$ and $w_{m*}$
\end{enumerate}
\begin{equation*}
\begin{split}
\mathbf{w}_{m*}^{new} &= \mathbf{w}_{m*} + \alpha (\mathbf{x} - \mathbf{w}_{m*}) \\
\mathbf{w}_{m}^{new} &= \mathbf{w}_{m} + \alpha (\mathbf{x} - \mathbf{w}_{m}) \\
w_j^{new} &= w_j, \forall j \ne m, m*\end{split}
\end{equation*}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% CS624: Analysis of Algorithms
% Copyright 2015 Pejman Ghorbanzade <pejman@ghorbanzade.com>
% Creative Commons Attribution-ShareAlike 4.0 International License
% More info: https://github.com/ghorbanzade/beacon
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section*{Question 7}

Prove that the following are equivalent.
\begin{enumerate}
  \item
  \begin{enumerate}
    \item $V_1$ is a vertex cover of $G$.
    \item $V - V_1$ is an independent set in $G$.
  \end{enumerate}
  \item
  \begin{enumerate}
    \item $V_2$ is an independent set in $G$.
    \item $V_2$ is a clique in $G^c$.
  \end{enumerate}
\end{enumerate}

\subsection*{Solution}

\begin{enumerate}
  \item To prove 1(a) and 1(b) are equivalent, we need to show vertex cover and independent set problems are polynomially equivalent.
  \begin{enumerate}
    \item Objective is to prove any independent set of vertices $V_2$ for an undirected graph $G = (V, E)$ is reducible to a vertex cover on $G' = (V_1, E')$ where $V_1 = V - V_2$.
    Since $V_1$ is an independent set, for any edge $e = (u,v) \in E$ either $u\in V_2$ or $v\in V_2$ which means either $u \in V_1$ or $v \in V_1$. Since $E$, by definition, contains all the edges in graph $G$, any $e \in E$ has at least one end in $V_1$, which means $V_1 = V - V_2$ is a vertex cover.
    \item Objective is to show any vertex cover $V_1 \in V$ in $G(V, E)$ is reducible to an independent set of vertices $V_2 = V - V_1$. Proof is given by contradiction that $V_2$ must form an independent set. Suppose $V_2$ is not an independent set which means there is an edge $e = (u,v) \in E$ such that both its endpoints are in $V_2$. Therefore $\{u, v\} \notin V_1$ which means set of vertices $V_1$ does not cover $e$ and therefore $V_1$ is not a vertex cover of $G$ which is a contradiction.
  \end{enumerate}
  Thus both problems 1(a) and 1(b) are polynomially reducible to each other which means they are equally hard.
  \item To prove 2(a) and 2(b) are equivalent, we need to show independent set and clique problems are polynomially equivalent.
  \begin{enumerate}
    \item Objective is to prove any independent set of vertices $V_2$ for an undirected graph $G = (V, E)$ is reducible to a clique on $G^c = (V, E')$. Since $V_2$ form an independent set, for any pair of vertices $\{u, v\} \in V_2$, $e = (u, v) \notin E$ which means $(u, v) \in E'$. Thus since all vertices in $V_2$ are mutually connected in $G^c = (V, E')$, they form a clique in $G'$.
    \item Objective is to prove any set of vertices $V_2$ that form a clique in $G^c=(V, E')$ form an independent set in $G = (V, E)$. Similar to previous proof, for any pair of vertices $\{u, v\}\in V_2$, $e = (u, v) \in E'$ which means $(u,v) \notin E$, i.e. no two vertices of $V_2$ are directly connected in $G=(V,E)$ which means they form an independent set.
  \end{enumerate}
  Thus both problems 2(a) and 2(b) are polynomially reducible to each other which means they are equally hard.
\end{enumerate}


\subsection{Subrings}

A subring is a subset of the ring, where the addition and multiplication operations on the subring result in elements also in the subring.

\subsubsection{Example}

The even numbers are a subring of the integers.


\section{Model}

%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Algorithm analysis}
  \begin{itemize}
	\item Given a problem $P$
	\item design an alg. $A$
	\item input space $\mathcal{X}_{n}$: inputs of size $n$
  \end{itemize}

  \pause
  \[
	W(n) = T_{\text{worst-case}}(n) = \max_{X \in \mathcal{X}_{n}} T(X)
  \]

  \pause
  \[
	B(n) = T_{\text{best-case}}(n) = \min_{X \in \mathcal{X}_{n}} T(X)
  \]

  \pause
  \[
	A(n) = T_{\text{average-case}}(n) = \sum_{X \in \mathcal{X}_{n}} T(X) \cdot Pr(X) \pause = E_{X \in \mathcal{X}_{n}} [T(X)]
  \]
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{(Problem 1.1.8)}
  \begin{align*}
	A &= \sum_{X \in \mathcal{X}} T(X) \cdot Pr(X) \\
	  &= T(1) Pr(1) + T(2) Pr(2) + \cdots + T(n) Pr(n) \\
	  &= \cdots  \\
  \end{align*}
\end{frame}
%%%%%%%%%%%%%%%%%%%%
\begin{frame}{Average-case analysis of Quicksort}
  \[
	A(n) = n-1 + \frac{1}{n} \sum_{i=0}^{i=n-1} (A(i) + A(n-i-1))
  \]

  \[
	A(n) = E_{X \in \mathcal{X}_{n}} [T(X)] = \sum_{X \in \mathcal{X}_{n}} T(X) \cdot Pr(X)
  \]

  \pause

  \begin{align*}
	A(n) &= E[T(X)] \\
	    &\textcolor{red}{= E[E[T(X)|I]]} \\
		&= \sum_{i=0}^{i=n-1} Pr(I = i)E[T(X) \mid I = i] \\
		&= \sum_{i=0}^{i=n-1} \frac{1}{n} [n-1 + A(i) + A(n-i-1)]
  \end{align*}
\end{frame}
%%%%%%%%%%%%%%%%%%%%


\documentclass{scrartcl}

\input{../../shared.tex}
\theoremstyle{definition}
\newtheorem{definition}{Definition}[section]
\newtheorem{lemma}{Lemma}[section]
\usepackage{enumerate}

\begin{document}

\section*{Problem set: Gradient descent and Fixed points}%

\paragraph{Exercise (i)} (1P)  Let $f_1, f_2, \dots, f_m$ be smooth with parameters $L_1, L_2, \dots L_m$. Show that the function $f:= \sum_{i=1}^{m}f_i$f is smooth with parameter $\sum_{i=1}^{m}L_i$.

\paragraph{Exercise (ii)} (1P)  Let $f$ be smooth with parameter $L$ and $A$ a matrix. Show that $f\circ A$ is smooth with parameter $L \Vert A \Vert^2$.

\subsection*{Computing Fixed Points}%

Gradient descent turns up in a surprising number of situations which apriori have nothing to do with optimization.
In this exercise we will see how computing the fixed point of functions can be seen as a form of gradient descent.
Suppose that we have a $1$-Lipschitz continuous function $g : \R \to \R$ such that we want to solve for
\begin{equation}
  g(x) = x .
\end{equation}
A simple strategy for finding such a fixed point is to run the following algorithm: starting from an arbitary $x_0$,
we iteratively set
\begin{equation}
  \label{fpi}
  x_{k+1} = g(x_k) .
\end{equation}

\paragraph{Exercise (iii)} (3P) Enter the missing code snippets in the jupyter notebook. Partial credit will be awarded.

We will try solve for $x$ starting from $x_0 = 1$ in the following two equations:
\begin{equation}
  \label{log}
  x = \log(1 + x), \quad \text{and} \quad x = \log(2 + x).
\end{equation}
What difference do you observe in the rate of convergence between the two problems? Let’s understand why this happens:

\paragraph{Exercise (iv)} (3P) Theoretical fixed point questions.
\begin{itemize}
  \item We want to re-write the update~\eqref{fpi} as a step of gradient descent. To do this, we need to find a function $f$
        such that the gradient descent update is identical to~\eqref{fpi}:
        \begin{equation}
          x_{k+1} = x_k - \alpha f'(x_k) = g(x_k) .
        \end{equation}
        Derive such a function $f$.
  \item Give sufficient conditions on $g$ to ensure convergence of procedure~\eqref{fpi}. What $\alpha$ would you need to pick?
        Hint: We know that gradient descent on f with fixed step-size converges if $f$ is convex and smooth. What
        does this mean in terms of $g$?
  \item What condition does $g$ need to satisfy to ensure linear convergence? Are these satisfied for the problems in~\eqref{log}
\end{itemize}

\end{document}


\problemname{Interval Product}
%\illustration{.5}{filename}{Image by \href{url}{Author}}

There is an array of $N$ positive integers, $x_1, x_2, ... x_N$.
Your friend gives you $Q$ queries, which can be:
\begin{itemize}
\item {\tt q l r} : Consider the product $P = x_l \times x_{l+1} \times ... \times x_{r-1} \times x_r$. How many trailing zeroes are there in the binary representation of $P$? (Yay! Your friend no longer cares about the product being positive, negative, or zero.)
\item {\tt s i v} : Set the value of $x_i$ to $v$.
\end{itemize}
Since your friend have a large number of queries, you have to find a way to answer them efficiently.

\section*{Input}
The first line of the input has an integer $T$, the number of test cases.\\
Each case has two integer $N, Q$ on the first line.\\
The next line has $N$ integers, $x_1, x_2, ... x_N$.\\
Each of the next $Q$ lines gives a query, which can be one of the two types introduced above.

\section*{Output}

For each query of the first type, output the number of trailing zeroes in the binary representation of the interval product.

\section*{Constraints}
\begin{itemize}
\item $1 \leq T\leq 15$
\item $1 \leq N, Q \leq 10^5$
\item $1 \leq l \leq r \leq N$ for all queries of the first type. $1 \leq i \leq N$ for all queries of the second type.
\item It is guaranteed that $1 \leq x_i \leq 10^9$ for $1 \leq i \leq N$ before the first query and after every query is processed.
\end{itemize}

\section*{Subtasks}
\begin{itemize}
\item There are only queries of the first type. That is, there will be no changes on the array.
\item Original constraints
\end{itemize}

Note that the last sample case will not appear in the {\it Small} subtask, as it contains queries of the second type.

\section*{Hint}
The number of trailing zeroes in the binary representation of a decimal integer $P$ equals how many times $P$ can be evenly divided by 2. For example, $44 = (101100)_2$ has two trailing zeroes and can be evenly divided at most twice: $44 / 2 = 22$, $22 / 2 = 11$, and $11$ is odd.


\section{Calculus with Vector Functions}\label{sec:calculus with vector functions}

A vector function $\vect{r}(t)=\langle f(t),g(t),h(t)\rangle$ \index{vector function}is a
function of one variable---that is, there is only one ``input''
value. What makes vector functions more complicated than the functions
$y=f(x)$ that we studied in the first part of this book is of course
that the ``output'' values are now three-dimensional vectors instead
of simply numbers. It is natural to wonder if there is a corresponding
notion of derivative for vector functions. In the simpler case of a
function $y=s(t)$, in which $t$ represents time and $s(t)$ is position
on a line, we have seen that the derivative $s'(t)$ represents
velocity; we might hope that in a similar way the derivative of a
vector function would tell us something about the velocity of an
object moving in three dimensions.

One way to approach the question of the derivative for vector
functions is to write down an expression that is analogous to the
derivative we already understand, and see if we can make sense of
it. If we say that what we mean by the limit of a vector is the vector of
the individual coordinate limits, this gives us
\begin{align*}
  \vect{r}^{\prime}(t)&=\lim_{\Delta t\to0}{\vect{r}(t+\Delta t)-\vect{r}(t)\over
  \Delta t}	\\
  &=\lim_{\Delta t\to0}{\langle f(t+\Delta t)-f(t),g(t+\Delta t)-g(t),
  h(t+\Delta t)-h(t)\rangle\over \Delta t}	\\
  &=\lim_{\Delta t\to0}\langle {f(t+\Delta t)-f(t)\over\Delta t},
  {g(t+\Delta t)-g(t)\over\Delta t},
  {h(t+\Delta t)-h(t)\over \Delta t}\rangle	\\
  &=\langle f'(t),g'(t),h'(t)\rangle,
\end{align*}

Starting with a familiar
expression for what appears to be a derivative, we find that we can
make good computational sense out of it---but what does it actually
mean?

We know how to interpret $\vect{r}(t+\Delta t)$ and $\vect{r}(t)$---they
are vectors that point to locations in space; if $t$ is time, we can
think of these points as positions of a moving object at times that
are $\Delta t$ apart. We also know what $\Delta \vect{r}= \vect{r}(t+\Delta t)-\vect{r}(t)$
means---it is a vector that points from the head of $\vect{r}(t)$ to
the head of $\vect{r}(t+\Delta t)$, assuming both have their tails at
the origin. So when $\Delta t$ is small,
$\Delta \vect{r}$ is a tiny vector pointing from one
point on the path of the object to a nearby point. As $\Delta t$ gets
close to 0, this vector points in a direction that is closer and
closer to the direction in which the object is moving; geometrically,
it approaches a vector tangent to the path of the object at a
particular point.

\begin{figure}[H]
\centerline{
\vbox{\beginpicture
\normalgraphs
\setcoordinatesystem units <6truemm,6truemm>
\setplotarea x from 0 to 11, y from 0 to 5.1
\arrow <5pt> [0.17, 0.5] from 0 0 to 2 3
\arrow <5pt> [0.17, 0.5] from 2 3 to 7 4
\arrow <5pt> [0.17, 0.5] from 0 0 to 7 4
\put {$\vect{r}(t)$} [l] <8pt,7pt> at 1 1.5
\put {$\Delta\vect{r}$} [br] <0pt,-10pt> at 4.5 3.5
\put {$\vect{r}(t+\Delta t)$} [tl]
  <3pt,-3pt> at 3.5 2
\setquadratic
%\plot 0.000 1.926 0.275 2.117 0.550 2.293 0.825 2.454 1.100 2.602
%1.375 2.737 1.650 2.860 1.925 2.971 2.200 3.073 2.475 3.164
%2.750 3.247 3.025 3.321 3.300 3.389 3.575 3.450 3.850 3.505
%4.125 3.556 4.400 3.602 4.675 3.646 4.950 3.687 5.225 3.727
%5.500 3.766 5.775 3.805 6.050 3.844 6.325 3.886 6.600 3.930
%6.875 3.977 7.150 4.029 7.425 4.085 7.700 4.147 7.975 4.216
%8.250 4.292 8.525 4.376 8.800 4.469 9.075 4.572 9.350 4.685
%9.625 4.810 9.900 4.947 10.175 5.097 10.450 5.261 10.725 5.439
%11.000 5.633 /
\plot 0.000 0.850 0.275 1.243 0.550 1.603 0.825 1.930 1.100 2.227
1.375 2.494 1.650 2.733 1.925 2.946 2.200 3.134 2.475 3.299
2.750 3.443 3.025 3.565 3.300 3.669 3.575 3.756 3.850 3.827
4.125 3.884 4.400 3.928 4.675 3.961 4.950 3.984 5.225 3.999
5.500 4.007 5.775 4.010 6.050 4.010 6.325 4.007 6.600 4.003
6.875 4.001 7.150 4.000 7.425 4.004 7.700 4.012 7.975 4.028
8.250 4.052 8.525 4.086 8.800 4.132 9.075 4.190 9.350 4.262
9.625 4.351 9.900 4.457 10.175 4.582 10.450 4.727 10.725 4.894
11.000 5.085 /
\endpicture}}
\caption{Approximating the derivative. \label{fig:vector derivative}}
\end{figure}

Unfortunately, the vector $\Delta\vect{r}$ approaches 0 in length; the
vector $\langle 0,0,0\rangle$ is not very informative. By dividing by
$\Delta t$, when it is small, we effectively keep magnifying the
length of $\Delta\vect{r}$ so that in the limit it doesn't disappear. Thus the
limiting vector $\langle f'(t),g'(t),h'(t)\rangle$ will (usually) be a
good, non-zero vector that is tangent to the curve.

What about the length of this vector? It's nice that we've kept it
away from zero, but what does it measure, if anything?
Consider the length of one of the vectors that approaches the tangent
vector:
%$$\eqalign{
%|\langle {f(t+\Delta t)-f(t)\over\Delta t}&,
%{g(t+\Delta t)-g(t)\over\Delta t},
%{h(t+\Delta t)-h(t)\over \Delta t}\rangle|=\cr
%&{\sqrt{(f(t+\Delta t)-f(t))^2+(g(t+\Delta t)-g(t))^2+(h(t+\Delta t)-h(t))^2}
%\over\Delta t}\cr
%}$$
$$\left|{\vect{r}(t+\Delta t)-\vect{r}(t)\over
\Delta t}\right|={|\vect{r}(t+\Delta t)-\vect{r}(t)|\over|\Delta t|}$$
The numerator is the length of the vector that points from one position
of the object to a ``nearby'' position; this length is approximately
the distance travelled by the object between times $t$ and $t+\Delta
t$. Dividing this distance by the length of time it takes to travel
that distance gives the average speed. As $\Delta t$ approaches zero,
this average speed approaches the actual, instantaneous speed of the
object at time $t$.

By performing an ``obvious'' calculation to get something that
looks like the derivative of $\vect{r}(t)$, we get precisely what we
would want from such a derivative: the vector $\vect{r}'(t)$
points in the direction of travel of the object and its length tells
us the speed of travel. In the case that $t$ is time, then, we call
$\vect{v}(t)=\vect{r}'(t)$ the velocity vector. Even if $t$ is not time,
$\vect{r}'(t)$ is useful---it is a vector tangent to the curve.\index{vector function!derivative}

\begin{example}{}{}
We have seen that $\vect{r}=\langle \cos t,\sin t,t\rangle$ is a helix. Compute $\vect{r}'$ and $|\vect{r}'|$.
\end{example}

\begin{solution}
We compute $\vect{r}'=\langle -\sin t,\cos t,1\rangle$, and
$|\vect{r}'|=\sqrt{\sin^2 t+\cos^2 t+1}=\sqrt2$. Thinking of this as
a description of a moving object, its speed is always $\sqrt2$; see
Figure~\ref{fig:helix with tangent}.
\end{solution}

\begin{figure}[H]
\centerline{
\vbox{\beginpicture
\normalgraphs
\setcoordinatesystem units <3truecm,3truecm>
\setplotarea x from 0 to 1.1, y from 0 to 1.1
\put {\hbox{\epsfxsize5cm\epsfbox{images/helix_with_tangent.eps}}} at 0 0
\endpicture}}
\caption{A tangent vector on the helix. \label{fig:helix with tangent}}
\end{figure}

\begin{example}{}{}
The velocity vector for $\langle \cos t,\sin
t,\cos t\rangle$ is $\langle -\sin t,\cos
t,-\sin t\rangle$. As before, the first two coordinates mean that from
above this curve looks like a circle. The $z$ coordinate is now also
periodic, so that as the object moves around the curve its height
oscillates up and down. In fact it turns out that the curve is a
tilted ellipse, as shown in Figure~\ref{fig:tilted ellipse}.
\end{example}

\begin{figure}[H]
\centerline{
\vbox{\beginpicture
\normalgraphs
\setcoordinatesystem units <3truecm,3truecm>
\setplotarea x from 0 to 1.1, y from 0 to 1.1
\put {\hbox{\epsfxsize5cm\epsfbox{images/tilted_ellipse.eps}}} at 0 0
\endpicture}}
\caption{The ellipse $\vect{r}=\langle \cos t,\sin
t,\cos t\rangle$. \label{fig:tilted ellipse}}
\end{figure}

\begin{example}{}{}
The velocity vector for $\langle \cos t,\sin
t,\cos 2t\rangle$ is $\langle -\sin t,\cos
t,-2\sin 2t\rangle$. The $z$ coordinate is now oscillating twice as
fast as in the previous example, so the graph is not surprising; see
Figure~\ref{fig:roller coaster}.
\end{example}

\begin{figure}[H]
\centerline{
\vbox{\beginpicture
\normalgraphs
\setcoordinatesystem units <3truecm,3truecm>
\setplotarea x from 0 to 1.1, y from 0 to 1.1
\put {\hbox{\epsfxsize5cm\epsfbox{images/coaster_with_tangent.eps}}} at 0 0
\endpicture}}
\caption{$\langle \cos t,\sin
t,\cos 2t\rangle$. \label{fig:roller coaster}}
\end{figure}

\begin{example}{}{anglebetweencurves}
Find the angle between the curves $\langle t+1,t-2,t^2-4 \rangle$ and
$\langle t+2,-t+3,t^2+1\rangle$ where they meet.
\end{example}
\begin{solution}
The angle between two curves at a point is the angle between their
tangent vectors---any tangent vectors will do, so we can use the
derivatives. We need to find the point of intersection, evaluate the
two derivatives there, and finally find the angle between them.

To find the point of intersection, we need to solve the equations
\begin{align*}
t+1&=u+2	\\
t-2&=-u+3	\\
t^2-4&=u^2+1
\end{align*}
Solving either of the first two equations for $u$ and substituting in
the third gives $t=3$. This together with
$u=2$ satisfies all three equations. Thus the two curves meet at
$(4,1,5)$, the first curve when $t=3$ and the second curve when
$t=2$.

The derivatives are $\langle 1,1,2t\rangle$ and
$\langle 1,-1,2t\rangle$; at the intersection point these are
$\langle 1,1,6\rangle$ and $\langle 1,-1,4\rangle$. The cosine of the
angle between them is then
$$\cos\theta = {1-1+24\over\sqrt{38}\sqrt{18}}={4\over\sqrt{19}},$$
so $\theta=\arccos(4/\sqrt{19})\approx0.41$.
\end{solution}

The derivatives of vector functions obey some familiar looking rules,
which we will occasionally need.

\begin{theorem}{Vector Derivative Properties}{vector derivative properties}
Suppose $\vect{r}(t)$ and $\vect{s}(t)$ are differentiable vector functions,
$f(t)$ is a differentiable function, and $a$ is a real number.\index{vector function!derivative properties}
\begin{enumerate}
\item	$\ds {d\over dt} a\vect{r}(t)= a\vect{r}'(t)$
\item	$\ds {d\over dt} (\vect{r}(t)+\vect{s}(t))=
\vect{r}'(t)+\vect{s}'(t)$
\item	$\ds {d\over dt} f(t)\vect{r}(t)= f(t)\vect{r}'(t)+f'(t)\vect{r}(t)$
\item	$\ds {d\over dt} (\vect{r}(t)\cdot\vect{s}(t))=
\vect{r}'(t)\cdot\vect{s}(t)+\vect{r}(t)\cdot\vect{s}'(t)$
\item	$\ds {d\over dt} (\vect{r}(t)\times\vect{s}(t))=
\vect{r}'(t)\times\vect{s}(t)+\vect{r}(t)\times\vect{s}'(t)$
\item	$\ds {d\over dt} \vect{r}(f(t))= \vect{r}'(f(t))f'(t)$
\end{enumerate}
\end{theorem}

Note that because the cross product is not commutative you must
remember to do the three cross products in formula (5.) in the correct
order.

When the derivative of a function $f(t)$ is zero, we know that the
function has a horizontal tangent line, and may have a local maximum
or minimum point. If $\vect{r}'(t)=\vect{0}$, the geometric
interpretation is quite different, though the interpretation in terms
of motion is similar. Certainly we know that the object has speed zero
at such a point, and it may thus be abruptly changing direction. In
three dimensions there are many ways to change direction;
geometrically this often means the curve has a cusp or a point, as in
the path of a ball that bounces off the floor or a wall.

\begin{example}{}{}
Suppose that $\vect{r}(t)=\langle 1+t^3,t^2,1\rangle$, so
$\vect{r}'(t)=\langle 3t^2,2t,0\rangle$. This is $\vect{0}$ at $t=0$, and
there is indeed a cusp at the point $(1,0,1)$, as
shown in Figure~\ref{fig:spacecurve cusp}.
\end{example}

\begin{figure}[H]
\centerline{
\vbox{\beginpicture
\normalgraphs
\setcoordinatesystem units <3truecm,3truecm>
\setplotarea x from 0 to 1.1, y from 0 to 1.1
\put {\hbox{\epsfxsize5cm\epsfbox{images/spacecurve_cusp.eps}}} at 0 0
\endpicture}}
\caption{$\langle 1+t^3,t^2,1\rangle$ has a cusp at $\langle 1,0,1\rangle$. \label{fig:spacecurve cusp}}
\end{figure}

Sometimes we will be interested in the direction of $\vect{r}'$ but not
its length. In some cases, we can still work with $\vect{r}'$, as when
we find the angle between two curves. On other occasions it will be
useful to work with a unit vector in the same
direction as $\vect{r}'$; of course, we can compute such a vector by
dividing $\vect{r}'$ by its own length. This standard unit tangent
vector is usually denoted by $\vect{T}$:
$$\vect{T}={\vect{r}'\over|\vect{r}'|}.$$\index{vector function!standard unit tangent vector}
In a sense, when we computed the angle between two tangent vectors we
have already made use of the unit tangent, since
$$\cos\theta = {\vect{r}'\cdot\vect{s}'\over|\vect{r}'||\vect{s}'|}=
{\vect{r}'\over|\vect{r}'|}\cdot{\vect{s}'\over|\vect{s}'|}$$

Now that we know how to make sense of $\vect{r}'$, we immediately know
what an antiderivative must be, namely
$$\int \vect{r}(t)\,dt = \langle \int f(t)\,dt,\int g(t)\,dt,\int h(t)\,dt
\rangle,$$ if $\vect{r}=\langle f(t),g(t),h(t)\rangle$. What about
definite integrals?  Suppose that $\vect{v}(t)$ gives the velocity of
an object at time $t$. Then $\vect{v}(t)\Delta t$ is a vector that
approximates the displacement of the object over the time $\Delta t$:
$\vect{v}(t)\Delta t$ points in the direction of travel, and $|\vect{v}(t)\Delta t|=|\vect{v}(t)||\Delta t|$ is the speed of the object
times $\Delta t$, which is approximately the distance travelled. Thus,
if we sum many such tiny vectors:
$$\sum_{i=0}^{n-1}\vect{v}(t_i)\Delta t$$
we get an approximation to the displacement vector over the time
interval $[t_0,t_n]$. If we take the limit we get the exact
value of the displacement vector:
$$\lim\sum_{i=0}^{n-1}\vect{v}(t_i)\Delta t = \int_{t_0}^{t_n}\vect{v}(t)\,dt = \vect{r}(t_n)-\vect{r}(t_0).$$
Thus, given the velocity vector we can compute the vector function
$\vect{r}$ giving the location of the object:
$$\vect{r}(t)=\vect{r}_0+\int_0^t \vect{v}(u)\,du.$$

\begin{example}{}{}
An object moves with velocity vector $\langle \cos t, \sin t,
\cos t\rangle$, starting at $(1,1,1)$. Find the function
$\vect{r}$ giving its location.
\end{example}
\begin{solution}
\begin{align*}
\vect{r}(t)&=\langle 1,1,1\rangle+\int_0^t \langle \cos u, \sin u,
\cos u\rangle\,du	\\
&=\langle 1,1,1\rangle+\left.\langle \sin u, -\cos u,\sin u\rangle
\right|_0^t	\\
&=\langle 1,1,1\rangle+\langle \sin t, -\cos t,\sin t\rangle-
\langle 0,-1,0\rangle	\\
&=\langle 1+\sin t, 2-\cos t,1+\sin t\rangle
\end{align*}
See Figure~\ref{fig:velocity cos sin cos}.
\end{solution}

\begin{figure}[H]
\centerline{
\vbox{\beginpicture
\normalgraphs
\setcoordinatesystem units <3truecm,3truecm>
\setplotarea x from 0 to 1.1, y from 0 to 1.1
\put {\hbox{\epsfxsize5cm\epsfbox{images/path_with_tangent.eps}}} at 0 0
\endpicture}}
\caption{Path of the object with its initial velocity vector. \label{fig:velocity cos sin cos}}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:calculus with vector functions}}

\begin{enumialphparenastyle}

\begin{ex}
Find $\vect{r}'$ and $\vect{T}$ for
$\vect{r} = \langle t^2,1,t\rangle$.
\begin{sol}
	$\langle 2t,0,1\rangle$, $\vect{r}'/\sqrt{1+4t^2}$
\end{sol}
\end{ex}

\begin{ex}
Find $\vect{r}'$ and $\vect{T}$ for
$\vect{r} = \langle \cos t, \sin 2t, t^2\rangle$.
\begin{sol}
	$\langle -\sin t, 2\cos 2t,2t\rangle$,
	$\vect{r}'/\sqrt{\sin^2t + 4\cos^2(2t)+4t^2}$
\end{sol}
\end{ex}

\begin{ex}
Find $\vect{r}'$ and $\vect{T}$ for
$\vect{r} = \langle \cos(e^t),\sin(e^t),\sin t\rangle$.
\begin{sol} $\langle -e^t\sin(e^t),e^t\cos(e^t),\cos t\rangle$,
$\vect{r}'/\sqrt{e^{2t}+\cos^2 t}$
\end{sol}
\end{ex}

\begin{ex}
Find a vector function for the line tangent to the helix
$\langle \cos t,\sin t, t\rangle$ when $t=\pi/4$.
\begin{sol} $\langle \sqrt2/2,\sqrt2/2,\pi/4\rangle+
t\langle -\sqrt2/2,\sqrt2/2,1\rangle$
\end{sol}
\end{ex}

\begin{ex}
Find a vector function for the line tangent to
$\langle \cos t,\sin t, \cos 4t \rangle$ when $t=\pi/3$.
\begin{sol} $\langle 1/2,\sqrt3/2,-1/2\rangle+
t\langle -\sqrt3/2,1/2,2\sqrt3\rangle$
\end{sol}
\end{ex}

\begin{ex}
Find the cosine of the angle between the curves $\langle
0,t^2,t\rangle$ and $\langle \cos(\pi t/2),\sin(\pi t/2), t\rangle$
where they intersect.
\begin{sol} $2/\sqrt5/\sqrt{4+\pi^2}$
\end{sol}
\end{ex}

\begin{ex}
Find the cosine of the angle between the curves $\langle
\cos t,-\sin(t)/4,\sin t\rangle$ and $\langle \cos t,\sin t, \sin(2t)\rangle$
where they intersect.
\begin{sol} $7\sqrt{5}\sqrt{17}/85$, $-9\sqrt{5}\sqrt{17}/85$
\end{sol}
\end{ex}

\begin{ex}\label{ex:derivative is perpendicular}
Suppose that $|\vect{r}(t)|=k$, for some constant $k$. This
means that $\vect{r}$ describes some path on the sphere of radius $k$
with center at the origin. Show that $\vect{r}$ is perpendicular to $\vect{r}'$ at every point. Hint: Use Theorem~\ref{thm:vector derivative properties}, part (d).
\end{ex}

\begin{ex}
A bug is crawling along the spoke of a wheel that lies along
a radius of the wheel. The bug is crawling at 1 unit per second and
the wheel is rotating at 1 radian per second. Suppose the wheel lies
in the $y$-$z$ plane with center at the origin, and at time $t=0$ the
spoke lies along the positive $y$ axis and the bug is at the origin.
Find a vector function $\vect{r}(t)$
for the position of the bug at time $t$, the velocity vector
$\vect{r}'(t)$, the unit tangent $\vect{T}(t)$, and the speed of the bug
$|\vect{r}'(t)|$.
\begin{sol} $\langle 0,t\cos t,t\sin t\rangle$,
$\langle 0,\cos t-t\sin t,\sin t+t\cos t\rangle$,
$\vect{r}'/\sqrt{1+t^2}$, $\sqrt{1+t^2}$
\end{sol}
\end{ex}

\begin{ex}
An object moves with velocity vector $\langle \cos t, \sin t,
t\rangle$, starting at $\langle 0,0,0\rangle$ when $t=0$. Find the function
$\vect{r}$ giving its location.
\begin{sol} $\langle \sin t,1-\cos t,t^2/2\rangle$
\end{sol}
\end{ex}

\begin{ex}
The position function of a particle is given by $\vect{r}(t) =
\langle t^2,5t,t^2-16t\rangle$, $t\geq 0$.  When is the speed of the particle
a minimum?
\begin{sol} $t=4$
\end{sol}
\end{ex}

\begin{ex}
A particle moves so that its position is given by
$\langle \cos t, \sin t, \cos(6t)\rangle$. Find the maximum and
minimum speeds of the particle.
\begin{sol} $37$, $1$
\end{sol}
\end{ex}

\begin{ex}
An object moves with velocity vector $\langle t, t^2,
\cos t\rangle$, starting at $\langle 0,0,0\rangle$ when $t=0$.
 Find the function
$\vect{r}$ giving its location.
\begin{sol} $\langle t^2/2,t^3/3,\sin t\rangle$
\end{sol}
\end{ex}

\begin{ex}
What is the physical interpretation of the dot product of two
  vector valued functions?  What is the physical interpretation of the
  cross product of two vector valued functions?
\end{ex}

\begin{ex}
Show, using the rules of cross products and differentiation,
  that
$${d\over dt} (\vect{r}(t) \times \vect{r}'(t))=
\vect{r}(t) \times \vect{r}''(t).$$
\end{ex}

\begin{ex}
Determine the point at which $\vect{f}(t)=\langle t, t^2, t^3
\rangle$ and $\vect{g}(t) =\langle \cos(t), \cos(2t), t+1 \rangle$
intersect, and find the angle between the curves at that point.  (Hint:
You'll need to set this one up like a line intersection problem,
writing one in $s$ and one in $t$.) If these two functions were the
trajectories of two airplanes on the same scale of time, would the
planes collide at their point of intersection?  Explain.
\begin{sol} $(1,1,1)$ when $t=1$ and $s=0$; $\theta=\arccos(3/\sqrt{14})$; no
\end{sol}
\end{ex}

\begin{ex}
Find the equation of the plane perpendicular to the curve $\vect{r}(t)
 = \langle 2\sin(3t),t,2\cos(3t)\rangle$ at the point $(0,\pi,-2)$.
\begin{sol} $-6x+(y-\pi)=0$
\end{sol}
\end{ex}

\begin{ex}
Find the equation of the plane perpendicular to
$\langle \cos t, \sin t, \cos(6t)\rangle$ when $t=\pi/4$.
\begin{sol} $\ds -x/\sqrt2+y/\sqrt2+6z=0$
\end{sol}
\end{ex}

\begin{ex}
At what point on the curve
$\vect{r}(t) = \langle t^3,3t,t^4\rangle$ is the
plane perpendicular to the curve also parallel to the plane $6x+6y-8z=1$?
\begin{sol} $(-1,-3,1)$
\end{sol}
\end{ex}

\begin{ex}
Find the equation of the line tangent to
$\langle \cos t, \sin t, \cos(6t)\rangle$ when $t=\pi/4$.
\begin{sol} $\langle 1/\sqrt2,1/\sqrt2,0\rangle+t\langle -1,1,6\sqrt2\rangle$
\end{sol}
\end{ex}

\end{enumialphparenastyle}


\subsection{Computing Exact Trigonometric Ratios}
The \dfont{unit circle} is often used to determine the \ifont{exact} value of a particular trigonometric function.

$$\includegraphics[width=5in]{images/unit-circle}$$

Reading from the unit circle one can see that $\cos (5\pi/6)=-\sqrt 3/2$ and $\sin (5\pi/6)=1/2$ (remember that the $x\,$-coordinate is $\cos\theta$ and the $y$-coordinate is $\sin\theta$ for the unit circle).
However, we don't always have access to the unit circle.
In this case, we can compute the exact trigonometric ratios for $\theta=5\pi/6$ by using \dfont{special triangles} and the \dfont{CAST rule} described below.

The first special triangle has angles of $45^\circ,45^\circ,90^\circ$ (i.e., $\pi/4,\pi/4,\pi/2$) with side lengths $1,1,\sqrt 2$, while the second special triangle has angles of $30^\circ,60^\circ,90^\circ$ (i.e., $\pi/6,\pi/3,\pi/2$) with side lengths $1,2,\sqrt 3$. They are classically referred to as the $1-1-\sqrt{2}$ triangle, and the $1-2-\sqrt{3}$ triangle, respectively, shown below.
$$\includegraphics[width=4.0in]{images/trig4}$$

\begin{formulabox}[Mnemonic]
The first triangle should be easy to remember.
To remember the second triangle, place the largest number ($2$) across from the largest angle ($90^\circ=\pi/2$).
Place the smallest number ($1$) across from the smallest angle ($30^\circ=\pi/6$).
Place the middle number ($\sqrt 3\approx 1.73$) across from the middle angle ($60^\circ=\pi/3$).
Double check using the Pythagorean Theorem that the sides satisfy $a^2+b^2=c^2$.
\end{formulabox}

The special triangles allow us to compute the exact value (excluding the sign) of trigonometric ratios, but to determine the sign, we can use the \ifont{CAST rule}.

\begin{formulabox}[The CAST Rule]
The CAST rule says that in quadrant I all three of $\sin\theta$, $\cos\theta$, $\tan\theta$ are positive.
In quadrant II, only $\sin\theta$ is positive, while $\cos\theta$, $\tan\theta$ are negative.
In quadrant III, only $\tan\theta$ is positive, while $\sin\theta$, $\cos\theta$ are negative.
In quadrant IV, only $\cos\theta$ is positive, while $\sin\theta$, $\tan\theta$ are negative.
To remember this, simply label the quadrants by the letters C-A-S-T starting in the bottom right and labelling counter-clockwise.
\end{formulabox}

$$\includegraphics[width=2.5in]{images/trig5}$$

\begin{example}{Determining Trigonometric Ratios Without Unit Circle}{CASTRule}
Determine $\sin (5\pi/6)$, $\cos (5\pi/6)$, $\tan (5\pi/6)$, $\sec (5\pi/6)$, $\csc (5\pi/6)$ and $\cot (5\pi/6)$ exactly by using the special triangles and CAST rule.
\end{example}

\begin{solution}
We start by drawing the $xy$-plane and indicating our angle of $5\pi/6$ in standard position (positive angles rotate \ifont{counterclockwise} while negative angles rotate \ifont{clockwise}).
Next, we drop a perpendicular to the $x\,$-axis (never drop it to the $y$-axis!).
$$\includegraphics[width=4in]{images/trig6}$$
Notice that we can now figure out the angles in the triangle.
Since $180^\circ=\pi$, we have an interior angle of $\pi-5\pi/6=\pi/6$ inside the triangle.
As the \ifont{angles of a triangle add up to $180^\circ=\pi$}, the other angle must be $\pi/3$.
This gives one of our special triangles.
We label it accordingly and add the CAST rule to our diagram.
$$\includegraphics[width=3in]{images/trig7}$$
From the above figure we see that $5\pi/6$ lies in quadrant II where $\sin\theta$ is positive and $\cos\theta$ and $\tan\theta$ are negative.
This gives us the \ifont{sign} of $\sin\theta$, $\cos\theta$ and $\tan\theta$.
To determine the \ifont{value} we use the special triangle and SOH CAH TOA.

Using $\sin\theta=opp/hyp$ we find a value of $1/2$.
But $\sin\theta$ is positive in quadrant II, therefore,
$$\sin \left(\frac{5\pi}{6}\right)=\frac{1}{2}.$$

Using $\cos\theta=adj/hyp$ we find a value of $\sqrt 3/2$.
But $\cos\theta$ is negative in quadrant II, therefore,
$$\cos \left(\frac{5\pi}{6}\right) =-\frac{\sqrt 3}{2}.$$

Using $\tan\theta=opp/adj$ we find a value of $1/\sqrt 3$.
But $\tan\theta$ is negative in quadrant II, therefore,
$$\tan \left(\frac{5\pi}{6}\right) =-\frac{1}{\sqrt 3}.$$

To determine $\sec\theta$, $\csc\theta$ and $\cot\theta$ we use the definitions:
$$\csc \left(\frac{5\pi}{6}\right)  = \ds\frac{1}{\sin \left(\frac{5\pi}{6}\right)}  = 2,
\qquad \sec \left(\frac{5\pi}{6}\right)  = \ds\frac{1}{\cos \left(\frac{5\pi}{6}\right) } = -\frac{2}{\sqrt 3},
\qquad\cot \left(\frac{5\pi}{6}\right) = \ds\frac{1}{\tan \left(\frac{5\pi}{6}\right)} = -\sqrt 3.$$
\end{solution}

\begin{example}{CAST Rule}{CASTRule2}
If $\cos\theta=3/7$ and $3\pi/2<\theta< 2\pi$, then find $\cot\theta$.
\end{example}

\begin{solution}
We first draw a right angle triangle.
Since $\cos\theta=adj/hyp=3/7$, we let the adjacent side have length $3$ and the hypotenuse have length $7$.
$$\includegraphics[height=1.1in]{images/trig8}$$
Using the Pythagorean Theorem, we have $3^2+(\mbox{opp})^2=7^2$.
Thus, the opposite side has length $\sqrt{40}$.
$$\includegraphics[height=1.1in]{images/trig9}$$
%Recall that \green{right angle triangles} can give us the \red{value} of trigonometric ratios while the \green{CAST rule} gives us the \red{sign}.**
To find $\cot\theta$ we use the definition:
$$\cot\theta=\frac{1}{\tan\theta}.$$
Since we are given $3\pi/2<\theta< 2\pi$, we are in the fourth quadrant. By the CAST rule, $\tan\theta$ is negative in this quadrant.
As $\tan\theta=opp/adj$, it has a value of $\sqrt{40}/3$, but by the CAST rule it is negative, that is,
$$\tan\theta=-\frac{\sqrt{40}}{3}.$$
Therefore,
$$\cot\theta=-\frac{3}{\sqrt{40}}.$$
\end{solution}


\chapter{Foundamentals of Speech Signal Processing}
The basic concepts of signal processing have been covered in EI015 Signals and Systems,  and therefore will not be mentioned in this note.

In other words, I am lazy.
\newpage
\section{Discrete Fourier Transform}
Similar to the continuous case, given a periodic discrete signal $\tilde{x}[n]$\footnote{The tilde sign here means that $\tilde{x}[n]$ is periodic.} with period $N$, it can be represented by \emph{a discrete sum of sinusoids}, rather than an integral (Recall the DTFT Synthesis Formula. Thank you, EI015!).

$\tilde{x}[n]$ can be represented as a sum of complex exponentials with radian frequency $(2\pi k / N)$, where $k=0,1,\dots,N-1$.
\[ \tilde{X}[k] = \sum_{n=0}^{N-1}\tilde{x}[n]e^{-j\frac{2\pi}{N}kn} \]

And the corresponding synthesis expression is
\[ \tilde{x}[n] = \frac{1}{N}\sum_{k=0}^{N-1}\tilde{X}[k]e^{j\frac{2\pi}{N}kn} \]

This representation of a periodic discrete signal is \emph{exact}. However, the DFT is generally used in another case, where $x[n]$ is a \emph{finite} sequence of signal. Since performing DFT only need $\tilde{x}[n]$ in a period $0 \le n \le N-1$, and whatever is out of this range does not matter, we may extend $x[n]$ and assume an ``implicit periodic sequence'' $\tilde{x}[n]$:
\[ \tilde{x}[n] = \sum_{r = -\infty}^{+\infty} x[n + rN] \]

And here comes our DFT.
\begin{definition}[Discrete Fourier Transform]
    \[ X[k] = \sum_{n=0}^{N-1} x[n]e^{-j\frac{2\pi}{N}kn} \quad k = 1,2,\dots,N-1 \]
    \[ x[n] = \sum_{k=0}^{N-1} X[k]e^{j\frac{2\pi}{N}kn} \quad n = 1,2,\dots,N-1 \]
\end{definition}
\begin{remark}
    Bear in mind that when using DFT representations, all signals behave as if they were implicitly \emph{periodic}, as the DFT is originally defined on periodic signals.
\end{remark}

\section{Voiced, Unvoiced and Silence}
The speech waveform can be classified into basically 3 stages.
\paragraph*{Unvoiced} Produced by creating a constriction somewhere in the vocal tract tube and forcing air through that constriction, thereby creating turbulent air flow, which acts as \emph{a broad-spectrum noise excitation of the vocal tract tube}.
\paragraph*{Voiced} Produced by forcing air through the glottis with the tension of the vocal cords adjusted so that they vibrate in a relaxation oscillation, leading to \emph{quasi-periodic} waveforms.
\paragraph*{Silence} Usually occurs at the beginning or the end of speech, lacks characteristics of either voiced sounds or unvoiced sounds.


%!TEX root = ms.tex
\section{Introduction}
\subsection{Model selection under linear restrictions}
Consider a linear regression problem with an $n\times 1$ response vector $y$ and an $n\times p$ design matrix $X$. The true model is generated from
\begin{equation}
y=X \beta_0 + \epsilon,
\label{eq:truemodel}
\end{equation}
where $\beta_0$ is a $p \times 1$ true coefficient vector, and the $n \times 1$ vector $\epsilon$ is independent of $X$, with $\{\epsilon_i\}_{i=1}^n \stackrel {iid} {\sim} N(0,\sigma_0^2)$. Note that $\beta_0$ represents the true parameters, not an intercept term.
We consider an approximating model
\begin{equation*}
y=X \beta + u,
%\label{eq:approxmodel}
\end{equation*}
where $\beta$ is $p \times 1$ and the $n \times 1$ vector $u$ is independent of $X$, with $\{u_i\}_{i=1}^n \stackrel {iid} {\sim} N(0,\sigma^2)$. For this approximating model, we further impose $m$ linear restrictions on the coefficient vectors $\beta$ that are given by
\begin{equation}
  R \beta = r,
  \label{eq:restriction}
\end{equation}
where $R$ is an $m \times p$ matrix with linearly independent rows ($\text{rank}(R)=m$) and $r$ is an $m \times 1$ vector. Both $R$ and $r$ are nonrandom. Examples of such restrictions include setting some slopes equal to 0 (which corresponds to variable selection), setting slopes equal to each other (which corresponds to using the sum of predictors in a model), and setting sums of slopes to 0 (which for pairs of predictors corresponds to using the difference of the predictors in a model).

Suppose first that $X$ is deterministic; we refer to this as the fixed-X design. Denote $f(y_i|x_i,\beta,\sigma^2)$ as the density for $y$ conditional on the $i$-th row of $x_i$. We have the log-likelihood function (multiplied by $-2$)
\begin{equation}
-2 \log f(y|X,\beta,\sigma^2) = -2 \sum_{i=1}^n \log f(y_i|x_i,\beta,\sigma^2) = n \log (2\pi \sigma^2) + \frac{1}{\sigma^2} || y-X\beta||_2^2.
\label{eq:loglike_fixedx}
\end{equation}
By minimizing \eqref{eq:loglike_fixedx} subject to \eqref{eq:restriction}, we obtain the restricted maximum likelihood estimator (MLE)
\begin{equation}
\begin{aligned}
\hat{\beta} &= \hat{\beta}^f + (X^T X)^{-1} R^T ( R(X^T X)^{-1} R^T)^{-1} (r-R \hat{\beta}^f),\\
\hat \sigma^2 &= \frac{1}{n} ||y-X \hat{\beta}||^2,
\end{aligned}
\label{eq:betahat_sigmahatsq}
\end{equation}
where $\hat{\beta}^f = (X^T X)^{-1} X^T y$ is the unrestricted least squares estimator. Since the errors are assumed to be Gaussian, $\hat\beta$ is also the restricted least squares estimator.

In practice, a sequence of estimators $\hat\beta(R_i,r_i|X,y)$, each based on a different set of restrictions, is often generated, and the goal is to choose the one with the best predictive performance. This can be done on the basis of information criteria, which are designed to estimate the predictive accuracy for each considered model. Note that the notion of predictive accuracy can be as simple as distance of a predicted value from a future value, as is the case in squared-error prediction measures, but also can encompass the more general idea that the log-likelihood is a measure of the accuracy of a fitted distribution as a prediction for the distribution of a future observation. This idea can be traced back to \citet{Akaike1973}, as noted in an interview with Akaike \citep{findley1995conversation}; see also \citet{efron1986biased}.

\subsection{Variable selection under fixed-X}
\label{sec:intro_subsetselection}
An important example of comparing models with different linear restrictions on $\beta$ is variable selection. We consider fitting the ordinary least squares (OLS) estimator on a predetermined subset of predictors with size $k$, and without loss of generality, the subset includes the first $k$ predictors of $X$, i.e. $\hat\beta^f(X_1,\cdots,X_k,y)$. By letting $R_k= \irow{0 & I_{p-k}}_{(p-k) \times p} $ and $r_k = \irow{0}_{(p-k) \times 1}$, it is easy to verify that $\hat{\beta}(R_k,r_k|X,y)=\hat\beta^f(X_1,\cdots,X_k,y)$. Therefore, comparing OLS fits on different subsets of predictors falls into the framework of comparing estimators with different linear restrictions on $\beta$.

Information criteria are designed to provide an unbiased estimate of the test error. We simplify the notation by denoting $\hat\beta(k) = \hat{\beta}(R_k,r_k|X,y)$. We also denote errF as the in-sample training error and ErrF as the out-of-sample test error. errF measures how well the estimated model fits on the training data $(X,y)$, while ErrF measures how well the estimated model predicts the new test data $(X,\tilde{y})$, where $\tilde{y}$ is an independent copy of the original response $y$, i.e. $\tilde{y}$ is drawn from the conditional distribution of $y|X$. The notations of errF and ErrF are based on those in \citet{efron2004estimation}, and the notation F here indicates that we have a fixed-$X$ design. \citet{efron1986biased} defined the optimism of a fitting procedure as the difference between the test error and the training error, i.e.
\begin{equation*}
\text{optF} = \text{ErrF} - \text{errF},
%\label{eq:optF}
\end{equation*}
and introduced the optimism theorem,
\begin{equation*}
E_y(\text{optF}) = E_y(\text{ErrF}) - E_y(\text{errF}),
\end{equation*}
where $E_y$ represents the expectation taken under the true model with respect to the random variable $y$. The optimism theorem provides an elegant framework to obtain an unbiased estimator of $E(\text{ErrF})$,
that is
\begin{equation*}
\widehat{\text{ErrF}} = \text{errF} + E_y(\text{optF}),
%\label{eq:ErrFhat}
\end{equation*}
where the notation $\widehat{\text{ErrF}}$ follows from \citet{efron2004estimation}. It turns out that many existing information criteria can be derived using the concept of optimism.

A typical measure of the discrepancy between the true model and an approximating model is the squared error (SE), i.e.
\begin{equation*}
\text{ErrF}_\text{SE} = E_{\tilde{y}}\left( \lVert \tilde{y}-X\hat{\beta} \rVert_2^2 \right).
%\label{eq:ErrF_SE}
\end{equation*}
The training error is $\text{errF}_\text{SE} = \displaystyle \lVert y-X\hat{\beta} \rVert_2^2$. \citet{ye1998measuring} and \citet{efron2004estimation} showed that for any general fitting procedure $\hat\mu$ and any model distribution (not necessarily Gaussian)
\begin{equation}
E_y(\text{optF}_\text{SE}) = 2\sum_{i=1}^n \text{Cov}_y(\hat\mu_i,y_i),
\label{eq:EoptF_SE}
\end{equation}
which is often referred to as the covariance penalty. For the OLS estimator $\hat\mu(k) = X\hat\beta(k)$ it is easy to verify that $E_y(\text{optF}_\text{SE}(k)) = 2 \sigma_0^2 k$. We denote RSS$(k)$ as the residual sum of squares for the OLS estimator, i.e. $\text{RSS}(k)=\lVert y- X\hat\beta(k) \rVert_2^2$. Hence,
\begin{equation*}
\widehat{\text{ErrF}}_\text{SE}(k) = \text{RSS}(k) + 2 \sigma_0^2 k
\end{equation*}
is an unbiased estimator of $E_y(\text{ErrF}_\text{SE})$. As suggested by \citet{mallows1973some}, typically $\sigma_0^2$ is estimated using the OLS fit on all the predictors, i.e. $\hat\sigma_0^2=\text{RSS}(p)/(n-p)$. We then obtain the Mallows' C$_p$ criterion \citep{mallows1973some}
\begin{equation}
\text{C}_p(k) = \text{RSS}(k) + \frac{\text{RSS}(p)}{n-p} 2k.
\label{eq:cp_subsetselection}
\end{equation}
An alternative is to use the OLS fit based on the $k$ predictors in the subset to estimate $\sigma_0^2$. i.e. $\hat\sigma_0^2 = \text{RSS}(k)/(n-k)$, which yields the final prediction error \citep{akaike1969fitting,akaike1970statistical}
\begin{equation}
\text{FPE}(k) = \text{RSS}(k)\frac{n+k}{n-k}.
\label{eq:cptilde_subsetselection}
\end{equation}

Another commonly-used error measure is (twice) the Kullback-Leibler (KL) divergence \citep[see, e.g.,][Section 3]{konishi2008information}
\begin{equation}
\text{KLF} = E_{\tilde{y}}\left[ 2\log f(\tilde{y} | X,\beta_0,\sigma_0^2 ) - 2\log f(\tilde{y} | X,\hat\beta,\hat\sigma^2 ) \right].
\label{eq:KLF}
\end{equation}
The right-hand side of \eqref{eq:KLF} evaluates the predictive accuracy of the fitted model, by measuring the closeness of the distribution of $\tilde{y}$ based on the fitted model and the distribution of $\tilde{y}$ based on the true model. The term $E_{\tilde{y}}\left[ 2\log f(\tilde{y} | X,\beta_0,\sigma_0^2 ) \right]$ is the same for every fitted model. Therefore, an equivalent error measure is the expected likelihood
\begin{equation*}
\text{ErrF}_\text{KL} = E_{\tilde{y}}\left[ -2\log f(\tilde{y} | X,\hat\beta,\hat\sigma^2 ) \right].
%\label{eq:ErrF_KL}
\end{equation*}
The training error is
\begin{equation*}
\text{errF}_\text{KL} = -2\log f(y|X,\hat\beta,\hat\sigma^2).
%\label{eq:errF_KL}
\end{equation*}
For the OLS estimator $\hat\beta(k)$, \citet{sugiura1978further} and \citet{hurvich1989regression} showed that under the Gaussian error \eqref{eq:truemodel}
\begin{equation*}
E_y(\text{optF}_\text{KL}(k)) = n\frac{n+k}{n-k-2}-n,
%\label{eq:EoptF_KL}
\end{equation*}
and hence
\begin{equation*}
\widehat{\text{ErrF}}_\text{KL}(k) = n\log\left(\frac{\text{RSS}(k)}{n}\right) + n\frac{n+k}{n-k-2} + n\log(2\pi)
\end{equation*}
is an unbiased estimator of $E_y(\text{ErrF}_\text{KL})$. Since the term $n\log(2\pi)$ appears in all of the models being compared, and thus is irrelevant when comparing criteria for the models, the authors dropped it and introduced the corrected AIC
\begin{equation}
\text{AICc}(k) = n \log\left( \frac{\text{RSS}(k)}{n}\right) + n\frac{n+k}{n-k-2}.
\label{eq:aicc_subsetselection}
\end{equation}
\citet{Hurvich1991} showed that AICc has superior finite-sample predictive performance compared to AIC \citep{Akaike1973}
\begin{equation*}
\text{AIC}(k) = n \log\left( \frac{\text{RSS}(k)}{n}\right) + n + 2(k+1),
\end{equation*}
which does not require a Gaussian error assumption but relies on asymptotic results. The derivations of AICc and AIC require the assumption that the true model is included in the approximating models. Neither AICc nor AIC involve $\sigma_0^2$, a clear advantage over C$_p$. Note that the second term of AICc can be rewritten as $n[1 + (2k+2)/(n-k-2)]$, which approximately equals the sum of the second and third terms of AIC when $n$ is large relative to $k$, demonstrating their asymptotic equivalence when $n\rightarrow\infty$ and $p$ is fixed.

\subsection{From fixed-X to random-X}
The assumption that $X$ is fixed holds in many applications, for example in a designed experiment where categorical predictors are represented using indicator variables or effect codings. However, in many other cases where the data are observational and the experiment is conducted in an uncontrolled manner, fixed-X is not valid and it is more appropriate to treat $(x_i,y_i)_{i=1}^n$ as $iid$ random draws from the joint distribution of $X$ and $y$. We refer to this as the random-X design. %For example, in the prostate cancer example from \citet[p.~49]{hastie2009elements}, the response variable $y$ is the level of prostate-specific antigen, and the predictors $X$ are clinical measures such as the weight of prostate, the volume of cancer, the age of patient, and etc. The goal is to predict the level of prostate-specific antigen based on the clinical measures of a new unseen patient. This fits the random-X assumption.

As noted by \citet{breiman1992submodel}, the choice between fixed-X and random-X is conceptual, and is normally determined based on the nature of the study. The extra source of randomness from $X$ results in larger test error compared to the fixed-X situation, and therefore the information criteria designed under fixed-X can be biased estimates of the random-X test error. Furthermore, when applied as selection rules, the authors showed in simulations that C$_p$ leads to significant overfitting under the random-X design. This motivates the derivation of information criteria for the random-X situation.

For the random-X design, we assume that the row vectors of $X$, $\{x_i\}_{i=1}^n$, are $iid$ multivariate normal with mean $E(x_i)=0$ and covariance matrix $E(x_i x_i^T)=\Sigma_0$. Let $f(y_i,x_i|\beta,\sigma^2,\Sigma)$ denote the joint multivariate normal density for $y_i$ and $x_i$. Let $g(x_i|\Sigma)$ denote the multivariate normal density for $x_i$. By partitioning the joint density of $(y,X)$ into the product of the conditional and marginal densities, and by separating the parameters of interest, we have the log-likelihood function (multiplied by $-2$)
\begin{equation}
\begin{aligned}
-2 \log f(y, X|\beta,\sigma^2,\Sigma) &= \sum_{i=1}^n -2 \log f(y_i, x_i|\beta,\sigma^2,\Sigma) = -2 \sum_{i=1}^n [\log f(y_i|x_i,\beta,\sigma^2) + \log g(x_i|\Sigma)] \\
&= \left [ n \log (2\pi \sigma^2) + \frac{1}{\sigma^2} || y-X\beta||_2^2 \right ] + \left [np \log(2\pi) + n \log |\Sigma| + \sum_{i=1}^n x_i^T \Sigma^{-1} x_i \right ].
\end{aligned}
\label{eq:loglike_randomx}
\end{equation}
Minimizing \eqref{eq:loglike_randomx} subject to \eqref{eq:restriction}, we find that the MLE $(\hat{\beta}, \hat{\sigma}^2)$ of $(\beta, \sigma^2)$ remains the same as in the fixed-X design, i.e. \eqref{eq:betahat_sigmahatsq}. The MLE of $\Sigma$ is given by
\begin{equation*}
\hat \Sigma = \frac{1}{n} \sum_{i=1}^n x_i \, x_i^T = \frac{1}{n} X^T X.
%\label{eq:Sigmahat}
\end{equation*}

Since $\hat\beta$ is unchanged when we move from fixed-X to random-X, variable selection as an example of linear restrictions on $\beta$ is based on the same parameter estimates as in Section \ref{sec:intro_subsetselection}. Denote errR, ErrR and optR as the training error, test error and the optimism under random-X, respectively. We generate $X^{(n)}$ as an independent copy of $X$, where the rows $\{x_i^{(n)}\}_{i=1}^n$ are $iid$ multivariate normal $\mathcal{N}(0,\Sigma_0)$. The new copy of the response $y^{(n)}$ is generated from the conditional distribution $y|X^{(n)}$. The optimism for random-X can be defined in the same way as for fixed-X, i.e. $\text{optR}=\text{ErrR}-\text{errR}$. \citet{rosset2020fixed} discussed the optimism for general fitting procedures, when the discrepancy between the true and approximating models is measured by the squared error (SE), i.e.
\begin{equation*}
\text{ErrR}_\text{SE} = E_{X^{(n)},y^{(n)}} \left( \lVert y^{(n)} - X^{(n)} \hat\beta \rVert_2^2 \right).
%\label{eq:ErrR_SE}
\end{equation*}
The training error is $\text{errR}_\text{SE} = \lVert y - X \hat\beta \rVert_2^2$. For the OLS estimator, the authors showed that
\begin{equation*}
E_{X,y}(\text{optR}_\text{SE}(k)) = \sigma_0^2 k \left(2 + \frac{k+1}{n-k-1} \right),
%\label{eq:EoptR_SE}
\end{equation*}
and hence
\begin{equation*}
\widehat{\text{ErrR}}_\text{SE}(k) = \text{RSS}(k) + \sigma_0^2 k \left(2 + \frac{k+1}{n-k-1} \right)
\end{equation*}
is an unbiased estimator of $E_{X,y}(\text{ErrR}_\text{SE})$. The result holds for arbitrary joint distributions of $(x_y,y_i)$ , and it only requires $x_i$ being marginally normal. As in the fixed-X case, if we use the unbiased estimate of $\sigma_0^2$ based on the full OLS fit, we have the analog of the C$_p$ rule for random-X,
\begin{equation}
\text{RC}_p(k) = \text{RSS}(k) + \frac{\text{RSS}(p)}{n-p} k\left(2 + \frac{k+1}{n-k-1}\right).
\label{eq:rcp_subsetselection}
\end{equation}
If we use the alternative estimate of $\sigma_0^2$ based on the OLS fit on the $k$ predictors in the subset, i.e. $\hat\sigma_0^2=\text{RSS}(k)/(n-k)$, we have the analog of the FPE rule for random-X,
\begin{equation}
\text{S}_p(k) = \text{RSS}(k)\frac{n(n-1)}{(n-k)(n-k-1)}.
\label{eq:sp_subsetselection}
\end{equation}
\citet{hocking1976biometrics} refers to \eqref{eq:sp_subsetselection} as the S$_p$ criterion of \citet{sclove1969criteria}; see also \citet{thompson1978a,thompson1978b}. Note that the notation used here is slightly different from that in \citet{rosset2020fixed}, where the authors used RC$_p$ to denote the infeasible criterion involving $\sigma_0^2$ and used $\widehat{\text{RC}}_p$ to denote the feasible criterion S$_p$. The RC$_p$ criterion in our notation was not studied in their paper.

Another class of selection rules is cross-validation (CV), which does not impose parametric assumptions on the model. A commonly used type of CV is the so-called K-fold CV. The data are randomly split into K equal folds. For each fold, the model is fitted using data in the remaining folds and is evaluated on the current fold. The process is repeated for all K folds, and an average squared error is obtained. In particular, the n-fold CV or leave-one-out (LOO) CV provides an approximately unbiased estimator of the test error under the random-X design, i.e. $E_{X,y}(\text{ErrR}_\text{SE})$. \citet{burman1989comparative} showed that for OLS, LOOCV has the smallest bias and variance in estimating the squared error-based test error, among all K-fold CV estimators. LOOCV is generally not preferred due to its large computational cost, but for OLS, the LOOCV error estimate has an analytical expression: the predicted residual sum of squares (PRESS) statistic \citep{allen1974relationship}
\begin{equation*}
\text{PRESS}(k) = \sum_{i=1}^n \left( \frac{y_i-x_i^T\hat\beta(k)}{1-H_{ii}(k)} \right)^2,
%\label{eq:press}
\end{equation*}
where $H(k) = X(k)(X(k)^T X(k))^{-1}X(k)^T$ and $X(k)$ contains the first $k$ columns of $X$.

\subsection{General linear restrictions}
Variable selection is a special case of linear restrictions on $\beta$, where certain entries of $\beta$ are restricted to be zero. In practice, we may restrict predictors to have the same coefficient (e.g. $\beta_1=\beta_2=\beta_3$), or we may restrict the sum of their effects (e.g. $\beta_1+\beta_2+\beta_3=1$). Using the structure in \eqref{eq:restriction}, we formulate a sequence of models, each of which imposes a set of general restrictions on $\beta$, where the goal is to select the model with best predictive performance. The previously defined information criteria and PRESS cannot be applied to this problem, although \citet{tarpey2000note} derived the PRESS statistic for the estimator under general restrictions as
\begin{equation*}
\text{PRESS}(R,r) = \sum_{i=1}^n \left( \frac{y_i-x_i^T\hat\beta}{1-H_{ii}+{H_Q}_{ii}} \right)^2,
%\label{eq:press}
\end{equation*}
where $H=X(X^T X)^{-1} X^T$ and $H_Q = X (X^T X)^{-1} R^T \left[ R (X^T X)^{-1} R^T \right]^{-1} R (X^T X)^{-1} X^T$.

\subsection{The contribution of this paper}
The information criteria introduced in Section \ref{sec:intro_subsetselection} have been studied primarily in the context of variable selection problems under fixed-X. In this paper we discuss how such criteria can be generalized to model comparison under general linear restrictions with either a fixed-X or a random-X (in both cases including the special case of variable selection). Note that a selection rule is preferred if it chooses the models that lead to the best predictive performance. This is related to, but not the same as, providing the best estimate of the test error. These two goals are fundamentally different \citep[see, e.g.,][Section 7]{hastie2009elements}, and we focus on the predictive performance of the selected model.

In Section \ref{sec:ic_fixedx}, we consider the fixed-X situation and derive general versions of AICc, C$_p$ and FPE for arbitrary linear restrictions on $\beta$. Random-X is assumed in Section \ref{sec:ic_randomx} and a version of RC$_p$ and S$_p$ for general linear restrictions is obtained. Furthermore, we propose and justify a novel criterion, RAICc, for general linear restrictions and discuss its connections with AICc. We further show that expressions of the information criteria for variable selection problems can be recovered as special cases of their expressions derived under general restrictions. In Section \ref{sec:simulation}, we show via simulations that AICc and RAICc provide consistently strong predictive performance for both variable selection and general restriction problems. Lastly, in Section \ref{sec:conclusion}, we provide conclusions and discussions of potential future work.