\newpage
\section{First-Order Differential Equations}

In this chapter, methods will be given for solving first- order differential equations. First-order means that the first derivative of the unknown function is the highest derivative appearing in the equation. This implied that the most general first-order differential equation has the form $F(t,x,t')=0$ for some function $F$, in this chapter, we will assume that the equation can be solved explicitly for $x'$. This means that our first-order differential equations can always be put in the form:

\[
  x'=f(t,x)
\]

where $f$ denotes an arbitrary function of two variables. To see why such an assumption makes sense, supposed the differential equation is:

\[
  (x'(x))^2+4x'(t)+3x(t)=t.
\]

It would be messy, but not impossible to use the quadratic formula to extract two differential equations of the form $x'=f(t,x)$ from this quadratic equation. However, one could also imagine equations where solving for $x'(t)$ is not even possible, and in such a case, some of our methods might not be applicable.

The material in this chapter will cover several analytic methods for solving first-order differential equations, each requiring the function $f$ to have a special form. Two different graphical methods are also described; one for the general equation depending only on $x$. Numerical methods for first-order equations are introduced and theoretical issues of existence and uniqueness of solutions are discussed.

\subsection{Separable First-Order Equations}

  The first analytic method we will consider applies to first-order equations that can be written in the form

  \[
    \frac{dx}{dt} = g(t)h(t);
  \]

  that is, when the function $f(t, x)$ can be factored into a product of a function of $t$ times a function of $x$. Such a differential equation is called separable.

  \begin{problem}
    Determine which of the following first-order differential equations are separable. Hint: try to factor the right- hand side if the equation does not initially appear to be separable.

    \[
      x' = xt + 2x\to x' = x(t + 2)\to g(t) = t + 2,h(x) = x\\
    \]

    \[
      x' = x + \cos (t)\\
    \]

    \[
      x' = xt^2 + t^2 - tx\to x' =(t^2 - t)(x + 1)\to g(t) =(t^2 - t),h(x) =(x + 1)\\
    \]

    \[
      x' = x^2 + x + 3\to x' = (1)(x^2 + x + 3)\to g(t) = 1,h(x) = x^2 + x + 3
    \]

    \begin{enumerate}
      \item
        If $h(x) = 1,$ the separable equation $x'=g(t)$ is just an integrating problem and the solution is
        \[
          x=\int g(t)dt;
        \]
        that is, $x$ is just the **indefinite integral** of the function $g(t)$. Remember that this means that $x$ can be *any* function $G(t)$ such that $G'(t)=g(t)$, and this introduces an arbitrary constant into the solution. As an example, the solution of $x'=t+1$ is

        \[
          x(t)=\int(t+1)dt=\frac{t^2}{2}+t+ c
        \]

        Even in this simple case the solution is an infinite one-parameter family of functions.

      \item If $g(t)=1$, the separable equation $x'=h(x)$ is called an **autonomous** first-order differential equation. Unless $h(x)$ is a constant, it is no longer possible to solve the equation by simple integration, and the method given below must be used. Autonomous first-order differential equations are important and will be investigated more thoroughly in section 2.7. In the above examples, only the last equation is autonomous. The other three contain functions of $t$ (other than the unknown function $x(t)$) on the right-and side.
    \end{enumerate}
  \end{problem}

  \begin{problem}
    Solve the differential Equation $\frac{dx}{dt}=-tx^2$.

    Solution. Split $dx/dt$ into two pieces, $dx$ and $dt$, and do a bit of algebra to write:

    \[
      - \frac{dx}{x^2} = tdt
    \]

    Integrate each side with respect to its own variable to obtain:

    \[
      \int \left( - \frac{1}{x^2}\right) =\int tdt\to \frac{1}{x} = \frac{t^2}{x} + C.
    \]

    where the arbitrary constants on each side have been collected on the right. Solve this equation for x to obtain the one parameter family of solutions

    \[
      x=\frac{1}{(t^2/2)+C}.
    \]

    We should check that the function $x(t)$ does satisfy the differential equation for any value of the constant C. It appears that this method works, but splitting $dx/dt$ into two pieces is not a mathematically condoned operation; therefore, a justification of the method needs to be given.

    If an equation is separable, and $x'(t)$ is written as $dx/dt$, both sides of the equation $dx/dt=g(t)h(x)$ can be divided by $h(x)$, and the equation becomes

    \[
      \frac{1}{h(x(t))}\left(\frac{dx}{dt}\right)dt=\int g(t)dt+C.
    \]

    The method of simple substitution can be applied to the integral on the left. If we substitute $u=x(t)$, then $du=(dx/dt)dt$, and the equation becomes

    \[
      \int\frac{1}{h(u)}du=\int g(t)dt+C.
    \]

    Now let $H(u)$ be any function such that $H'(u)=1/h(u)$ and $G(t)$ any function with $G'9t)=g(t)$. Then the equation above implies that

    \[
      H(u)+C_1=G(t)+C_2\to H(u)=G(t)+C,
    \]

    Where C is the constant $C_2-C_1$

    Replacing $u$ again by $x(t)$:

    \[
      H(x(t))=G(t)+ C
    \]
  \end{problem}

  Check carefully that the expression $H(x)=G(t)+C$ is exactly the same as the solution obtained above. It is an **implicit solution** of $-dx/x^2=tdt$; that is, it defines a relationship between the unknown function $x$ and its independent variable $t$. If it can be solved explicitly for $x$ as a function of $t$, the result is called an **explicit solution of the differential equation. As expected, the integration produces an infinite on-parameter family of solutions

  \begin{theorem}
    To solve a separable first-order differential equation, $x'(t)=g(t)h(x)$:
    \begin{itemize}
      \item Write the equation in the form $dx/dt=g(t)h(x)$.
      \item Multiply both sides by $dt$, divide by $h(x)$, and integrate, to put the equation in the form
        \[
          \int \frac{1}{h(x)}dx =\int g(t)dt.
        \]
      \item Find any function $H(x)$ such that $H'(x)=1/h(x)$ and any function $G(t)$ such that %G'(t)=g(t).
      \item Write the solution as $H(x)=G(t)+C$
      \item If possible, solve the equation from the previous step explicitly for x, as a function of t.
    \end{itemize}
  \end{theorem}

\subsection{Solving Linear ODEs}

 \begin{definition}
   An ODE is linear for the dependent variable $y$ if it is homogeneous when $g(x)=0$ and otherwise it is non-homogeneous.
   \[
     a_1(x)\frac{dy}{dx}+a_0(x)y=g(x)
   .\]
   The standard form of a linear ODE is
   \[
     \frac{dy}{dx}+P(x)y=f(x)
   .\]
 \end{definition}

 If the homogeneous ODE is separable: $y=e^{-\int P(x)dx}$, and $y_c=cy_1(x)$ where $y_1=e^{-\int P(x)dx}$. For a non-homogeneous ODE, we need to use the process of “variation of parameters”. First we need to find the function $u$ so that $y_p=u(x)y_1(x)=u(x)e^{-\int P(x)dx}$, and then we follow the following steps

 \begin{enumerate}
   \item Put in standard form
   \item Determine $P(x)$ and integrating factor $e^{\int P(x)dx}$.
   \item Multiply std form by integrating factor.
   \item Write \[
       \frac{d}{dx}\left[e^{\int P(x)dx}y\right]=e^{\int P(x)dx}f(x)
   .\]
  \item Integrate both sides.
 \end{enumerate}

\subsection{}

\subsection{Existence and Uniqueness of Initial Value Problems}

  Given an initial value problem, how can I know that a solution exists, and if so, the uniqueness of that solution.

  \begin{problem}
    $\frac{dx}{dt}=\sqrt{x}$, $x(0)=0$ has two solutions. These solutions are $x(t)=\frac{t^2}{4}$ and $0$. We can change the initial condition to be $x(0)=5$. We can separate our values to get:

    \[
      x(t) =\left(\frac{t}{2} + \sqrt{5}\right)^2
    \]

    If we change the initial condition to $x(5)=-5$ we get no solutions. We can also change the initial condition to $x(5)=0$ which gives us two solutions:

    \begin{align*}
      x(t) =\left(\frac{t - 5}{2}\right)^2\\
      x(t) = 0
    \end{align*}
  \end{problem}

\subsection{Exact Ordinary Differential Equations}

  This highlights an analytic technique to find the solutions to an ordinary differential equation. This technique depends on if we are able to write an ODE in the form $\frac{d}{dx}(F(x,y(x)))=0$. How can we tell if this is possible?

    We can determine this by using the multi-variable chain rule from calculus 3:

    \[
      \frac{d}{dx}(F(x,y(x))) = \frac{\partial F}{\partial x} + \frac{\partial F}{\partial y} \frac{dy}{dx} = 0
    \]

    Now in order to actually solve the ordinary differential equation, we will turn the equation above into something that we can work with:

    \[
      M(x,y) + N(x,y)\frac{dy}{dx} = 0
    \]

    Now we ask, does $f(x,y)$ exist such that $\frac{\partial F}{\partial x} =  M(x,y)$ and $\frac{\partial F}{\partial y} = N(x,y)$? We can test this by checking to see if we can do the following:

    \[
      \frac{\partial^2F}{\partial x\partial y}=\frac{\partial^2 F}{\partial y\partial x}\rightarrow\boxed{\frac{\partial M}{\partial y}=\frac{\partial N}{\partial x}}
    \]

    If it passed the test, we use the equations to find $F(x,y)$, and then we can use $F(x,y(x))=C$ to find the solution to the problem.

    \begin{problem}
      Calculus 3 example. Consider a field $\vec{v}=<2y+4x,2x-5y>$. Is this a gradient field? If so, what is its potential function. In order to find this, there must be a function $f(x,y)$ such that $\bigtriangledown f=\vec{v}$.

      \[
        {\partial f}{\partial y}\big>=<6xy,2x-5y>\\
      \]

      \begin{align*}
        \frac{\partial f}{\partial x}=6xy,&& \frac{\partial f}{\partial y}=2x-5y
      \end{align*}

      Therefore, we know that $f_{xy}=f_{yx}$ Now we just need to set the two equations equal to each other and solve:

      \[
        \frac{\partial}{\partial y}(2y+4x)\stackrel{?}{=}\frac{\partial}{\partial x}(2x-5y)
      \]

      And we can solve that equation algebraically and get $2=2$, therefore this is the solution to the ordinary differential equation

    \end{problem}

    \begin{problem}
      Solve $(4x+2y)+(2x-5y)\frac{dy}{dx}=0$. We should set $M(x,y)=4x+2y$ and set $N(x,y)=(2x-5y)$.

    \begin{enumerate}
      \item Testing to see if $\frac{\partial M}{\partial y}=\frac{\partial N}{\partial x}$
        \[
          \frac{\partial}{\partial y}(4x+2y)\stackrel{?}{=}\frac{\partial}{\partial x}(2x-5y)
        \]
      \item Now we need to find $F(x,y)$:
        \begin{align*}
          \frac{\partial F}{\partial x}=4x+2y && \frac{\partial F}{\partial y}=2x-5y\\
          F(x,y)=\int(4x+2y)dx=2x^2+2xy+c_y, && F(x,y)=\int(2x-5y)dy=2xy-\frac{5}{2}y^2+c_x
        \end{align*}
        We can determine that $c_y=\frac{5}{2}y^2$ and $c_x=2x^2$ by looking at the two equations next to each other. Therefore, we know that $F(x,y)=2x^2+2xy-\frac{5}{2}y^2$.
    \end{enumerate}
    Our resulting solution is:

    \[
      \boxed{2xy-\frac{5}{2}y^2+2x^2=C}
    \]

    This was an implicitly defined solution, so $F(x,y)=2xy-\frac{5}{2}y^2+2x$, and our ode is:

    \begin{align*}
      \frac{d}{dx}\left[2xy(y)-\frac{5}{2}(y(x))^2+2x^2\right]=\frac{d}{dx}[C]\\
      (2y+2xy')-\frac{5}{2}2(y)+4x=0\\
    \end{align*}

    \paragraph{Example 2} Solve the differential equation,

    \[
      2x^2y+e^y+(x^3+xe^y-2y)\frac{dy}{dx}=0.
    \]

    We can see that $3x^2y+e^y = M(x,y)$ and $(x^3+xe^y-2y)\frac{dy}{dx}=0$.

    We can test this by checking to see if:

    \[
      \frac{\partial M}{\partial y}\stackrel{?}{=}\frac{\partial N}{\partial x}
    \]

    and by taking the partial derivatives of each of the equations, we can see that $3x^2+e^y=3x^2+e^y$. Now that we know that this is an exact ordinary differential equation, we need to solve for $F(x,y)$. In order to find $F(x,y)$, we must do

    \begin{align*}
      \frac{\partial F}{\partial x}=M && \frac{\partial F}{\partial y}=N.
      \end{align*}

      This means that

      \begin{align*}
        F(x,y)=\int(3x^2y+e^y)dx && F(x,y)=\int(x^3+xe^y-2y)dy\\
        =x^3y+xe^y+C_y && =x^3y+xe^y-y^2+C_x\\
      \end{align*}

      So, $F(x,y)=x^3yxe^y-y^2$, and our solution is:

      \[
        \boxed{x^3y+xe^y-y^2=C}\\
      \]

      Now we need to find a solution that satisfies the initial value problem, $y(-2)=1$.

      \begin{align*}
        (-2)^3(1)+(-2)e^1-(1)^2&=C\\
        -8-2e-1&=C\\
        -9-2e&=C\\
      \end{align*}

      Which results in our particular solution being:

      \[
        \boxed{x^3y+xe^y-y^2=-9-2e}
      \]
    \end{problem}

  \subsubsection{Substitution}
    Sometimes we can make a substitution that turns one ordinary differential equation into one that we can solve.

    \begin{problem}
      Let's take the equation $\frac{dy}{dx}=\frac{1-x-y}{x+y}$. In order to do this you need to trade out $y(x)$ for $u(x)$:

      \[
        \frac{dy}{dx}=\frac{du}{dx}-1
      \]

      \[
        \frac{du}{dx}-1=\frac{1-u}{u}=\frac{1}{u}-1
      \]

      After we do those steps, we can see that this is a separable ordinary differential equation, and we can use the normal method to solve a separable equation:

      \[
        \begin{aligned}
          \int udu&=\int1dx\\
          \frac{u^2}{2}&=x+C\\
          u^2&=2x+D\leftarrow D=2C\\
          (x+y)^2&=2x+D\rightarrow x+y=\pm\sqrt{2x+D}\\
          y&=-x\pm\sqrt{2x+D}
        \end{aligned}
      \]

      Where the initial condition determines the last equation above. Now we need to solve for the initial condition of $y(6)=4$, and we can do this by just plugging in the values, and determining the sign on the square root:

      \[
        \begin{aligned}
           4&-6\pm\sqrt{12+D}\\
          10&=\pm\sqrt{12+D}\\
          D&=88\to y(x)=-x+\sqrt{2x+88}\\
        \end{aligned}
      \]
    \end{problem}

  \subsubsection{Bernoulli Ordinary Differential Equations}

    We use Bernoulli's solution to an ordinary differential equation if the equation can be written in the form $\frac{dx}{dt}+p(t)x=q(t)x^n$, where if $n=0$, the equation is linear and where if $n=1$, the equation is both linear and separable. We can do a substitution for Bernoulli ordinary differential equations where we let $v=x^{1-n}$, and we trade out $x(t)\to v(t)$ to turn the differential into a linear differential. Another formula we can use for plugging in to find the separable ordinary differential equation is $\frac{du}{dx}+(1-n)p(x)u=(1-n)q(x)$, but we do not have an example to show for that method.

    \begin{problem}
      Consider the Ordinary differential equation $t\frac{dy}{dt}+y=\frac{1}{y^2}$, and solve. Our first step for this problem is to divide both sides by $t$ to rearrange it into the Bernoulli form:

      \[
        \frac{dy}{dt}+\frac{1}{t}y=\frac{1}{t}y^-2
      \]

      Because $y$ is our dependent variable, we will be looking for our $n$ value, and we will find it on the $y$ on the right-hand side. This gives us $n=-2$. Now we need to let $v(t)=y^{1-(-2)}=y^3$ and this means that $y(t)=(v(t))^{1/3}$. Now wee need to take the differential of this equation in order to find a substitution for $\frac{dy}{dt}$:

      \begin{align*}
        \frac{dy}{dt}=\frac{1}{3y^2}\frac{dv}{dt}\\
        \frac{dv}{dt}=3y^{-2}\frac{dy}{dt}\\
      \end{align*}

      Once we have found a substitution for $\frac{dy}{dt}$, we can simply plug in our equation and solve for $v(t)$:

      \[
        \frac{1}{3y^2}\frac{dv}{dt}+y=\frac{1}{y^2}
      \]

      \[
        \frac{t}{3}\frac{dv}{dt}+y^3=1
      \]

      We solve for $v(t)$, and then we can use the integrating factors method to solve the ordinary differential equation:

      \[
        \begin{aligned}
          \frac{dv}{dt}+\frac{3}{t}v&=\frac{3}{t}\\
          \mu(t)&=e^{\int\frac{3}{t}dt}=e^{3ln(t)}=t^3\\
          t^3\frac{dv}{dt}+3t^2v&=3t^2\\
          \frac{d}{dt}(t^3v)&=3t^2\\
          \int t^3v&=\int3t^2dt=t^3+C\\
          v(t)&=1+\frac{C}{t^3}\to y(t)=(v(t))^{\frac{1}{3}}\\
        \end{aligned}
      \]\\

      Now we just need to let $v(t)=y^3$ and plug this back into the equation to get $y(t)=\left(1+\frac{C}{t^3}\right)^{\frac{1}{3}}$ as our final solution.
    \end{problem}

    Homogeneous ordinary differential equations can be written in the form $\frac{dx}{dt}=f\left(\frac{x}{t}\right)$. This means that if we just substitute $v$ for $\frac{x}{t}$, we will turn the Homogeneous equation into a separable ordinary differential equation.

    \begin{problem}
      Consider the homogeneous ODE, $(5y-2x)\frac{dy}{dx}=4x+2y$. We can divide the $5y-2x$, so we will end up with the equation

      \[
        \frac{dy}{dx}=\frac{4x+2y}{5y-2x}=\frac{4+\frac{2y}{x}}{5\frac{y}{x}-2}
      \]

      The reason we can make this change is that we are dividing everything on the left side by $x$ to get it in the form $\frac{y}{x}$. Our next step is to trade out $y(x)$ for $v(x)$:

      \[
        \frac{4+2v}{5v+2}=v+x\frac{dv}{dx}\\
      \]

      Which we can further simplify into $\frac{dv}{dx}=\frac{1}{x}\left(\frac{4+2v}{5v-2}-v\right)$. Now, we are able to solve this as a separable ordinary differential equation:

      \[
        \begin{aligned}
          \frac{dv}{dx}&=\frac{1}{x}\left(\frac{4+4v-5v^2}{5v-2}\right)\\
          \frac{5v-2}{4+4v-5v^2}dv&=\frac{1}{x}dx\\
        \end{aligned}
      \]

      Now we need to let $u-4+4v-5v^2$, and let $du=(4-10v)dv=-2(5v-2)dv$. And then we integrate with respect to $u$ to end up with

      \[
        \begin{aligned}
          -\frac{1}{2}\int\frac{1}{u}du&=\int\frac{1}{x}dx\\
          -\frac{1}{2}ln|u|&=ln|x|+C\\
          ln|4+4v-5v^2|&=-2ln|x|+D\\
          4+4v+5v^2&=e^{ln|x|^-2+D}\\
          &=e^{ln|x|^{-2}}e^D\\
          4+4\left(\frac{y}{x}\right)-5\left(\frac{y}{x}^2\right)&=e^{ln|x|^{-2}}A
        \end{aligned}
      \]

      \[
        \boxed{4x^2+4xy-5y^2=A}
      \]
    \end{problem}

\subsection{}
\subsection{Phase Lines and Equilibrium Solutions}

  This is a qualitative technique for autonomous ordinary differential equations. This is used for finding the long term behavior of solutions for various initial conditions. An equilibrium solution is a constant function satisfying the ODE:

  \[
    y(t)=k\to\frac{dy}{dt}=0
  \]

  and we need to look for the roots of $f(y)$.

  \begin{problem}
    Consider the ODE: $\frac{dy}{dx}=4-y^2$, we know that the equilibrium solutions (stationary points / fixed points / critical points) are $0=4-y^2$ and $y=\pm2$.  If we were to plot this as a slope field, we would see an image like the following:

    \begin{center}
      \includegraphics{resource/images/2.7 Example 1.jpg}
    \end{center}

    We need to classify each of the lines here as stable (a sink), unstable (a source), or semi-stable (a node). From this slope diagram we can make a phase diagram.

    The arrows in this diagram indicate whether $\frac{dy}{dx}$ is above or below $0$. We can see that the long term behavior of the solution:

    \[
      \begin{aligned}
        y_0 = y(t_0) > 2,\to y(t)\to 2\\
        - 2 < 2_0,\to y(t)\to 2\\
        y_0 <- 2, y(t)\to -\infty\\
      \end{aligned}
    \]
  \end{problem}

  Let's take a look at another example problem:

  \begin{problem}
    Compare the pivots of the two ordinary differential equations, $\frac{dy}{dt}=y^2(1-y)^2$, and $\frac{dy}{dx}=y^2(1-y^2)$.

    \begin{align*}
      \frac{dy}{dt}=y^2(1-y)^2 && \frac{dy}{dt}=y^2(1-y^2)\\
      0=y^2(1-y)^2 && 0=y^2(1-y^2)\\
      && y=0,\pm1\\
    \end{align*}
    From finding the equilibrium points of the differential equations, we can now craft a pivot diagram:\newline
    \begin{center}
    \includegraphics{resource/images/2.7 Example 2-1.jpg}
    \includegraphics{resource/images/2.7 Example 2-2.jpg}
    \end{center}

    From this we can see that in the long term, the equations are going to look like this:

    \begin{align*}
      y_0<0,y(t)\to0\\
      0<y_0<1,y(t)\to1\\
      1<y_0,y(t)\to\infty\\
    \end{align*}
  \end{problem}

  \subsubsection{Bifurcations}

  For ordinary differential equations with a parameter, there are times that a small change in the value of the parameter results in a huge change in the behavior of the solutions. It causes a change in the number or stability of the equilibrium solutions.

  \begin{problem}
    Find the bifurcation value for

    \[
      \frac{dp}{dt}=0.5p\left(1-\frac{p}{100}\right)-H
    \]

    The first step for finding a bifurcation for an equation is finding the equilibrium solutions. In order to do that, we set equation (2.69) equal to 0 and then rearrange it, so it is in the form

    \[
      p^2-100p+250H=0
    \]

    In this case, we are going to need to use the quadratic formula to find the equilibrium points:

    \begin{align*}
      p=\frac{100\pm\sqrt{10000-1000H}}{2}\\
      =50\pm5\sqrt{100-10H}
    \end{align*}
    We can determine that our bifurcation value is $H=10$, because if we take what's under the square root, and set it equal to zero we can find those values.
    Now we can plot our bifurcations:
    \begin{center}
      \includegraphics{resource/images/2.7 Example 3 1.jpg}
    \end{center}
  \end{problem}
  \subsection{Numerical Methods}\footnote{We got here from 4.1}


\subsection{Monopoly}

In the monopoly model we have:

\(\pi = pq-c\)

\(p[1+\dfrac{q}{p}\dfrac{\delta p}{\delta q}]=MC\)

The price elasticity of demand is: \(\epsilon = \dfrac{p}{q}\dfrac{\delta q}{\delta p}\)

\(p[1+\dfrac{1}{\epsilon }]=MC\)

\(\dfrac{1}{\epsilon }=\dfrac{MC}{p}-1\)

\(\dfrac{p-MC}{p}=-\dfrac{1}{\epsilon }\)

\subsection{The Lerner index}

The Lerner index is:

\(\dfrac{p-MC}{p}\)


% To be compiled with pdflatex
% This file is to be included into master file via \input command
% Note that there is no \begin{document} \end{document} brackets!

\newpage
\section{Appendix: \\ Matrix derivatives}
\label{sec:MDerivative}

Derivative of linear matrix functional $$\bm{p}^T \mathcal{F} \bm{z}$$
with respect to matrix $\mathcal{F}$
is computed as follows:
\begin{enumerate}
 \item Write the functional in element-wise form:
   $$ f = \bm{p}^T \mathcal{F} \bm{z} =
      \sum_i \sum_j p_i \mathcal{F}_{ij} z_j. $$
 \item Differentiate with respect to $\mathcal{F}_{ij}$:
    $$ \frac{\partial f}{\partial \mathcal{F}_{ij}} = p_i z_j. $$
 \item Thus $$ \frac{\partial}{\partial \mathcal{F}}
 (\bm{p}^T \mathcal{F} \bm{z}) = \bm{p} \bm{z}^T. $$
\end{enumerate}

Derivative of quadratic matrix functional
$$\bm{z}^T \mathcal{F}^T \mathcal{A} \mathcal{F} \bm{z},$$
where $\mathcal{A}$ is
symmetric real valued matrix, with respect to matrix $\mathcal{F}$ is computed
as follows:
\begin{enumerate}
 \item Write the functional in element-wise form:
		$$ f = \bm{z}^T \mathcal{F}^T \mathcal{AF} \bm{z} =
		\sum_{i=1}^m \sum_{l=1}^n \sum_{j=1}^m \sum_{k=1}^n
		\mathcal{F}_{il} \mathcal{F}_{jk} \mathcal{A}_{ij} z_l z_k. $$
 \item Most conveniently, by writing down the above equation for a case of
 small matrix size, say, 2x2, find by inspection that
		$$ \frac{\partial}{\partial \mathcal{F}}
		(\bm{z}^T \mathcal{F}^T \mathcal{AF} \bm{z}) = 2 \mathcal{AF} \bm{zz}^T. $$
\end{enumerate}

Analogously, the derivatives with respect to vector are:
$$ \frac{\partial}{\partial \bm{z}} ( \mathcal{F} \bm{z} ) = \mathcal{F}, $$
$$
	\frac{\partial}{\partial \bm{z}} \bm{z}^{T} \mathcal{F} \bm{z} =
	2 \bm{z}^{T} \mathcal{F}.
$$


\subsection{Elementary events}

We have a sample space, \(\Omega \) consisting of elementary events.

All elementary events are disjoint sets.

\subsection{Non-elementary events}

We have a \(\sigma\)-algebra over \(\Omega \) called \(F\). A \(\sigma\)-algebra takes a set a provides another set containing subsets closed under complement. The power set is an example.

All events \(E\) are subsets of \(\Omega\)

\(\forall E\in F E\subseteq \Omega\)

\subsection{Mutually exclusive events}

Events are mutually exclusive if they are disjoint sets.

\subsection{Complements}

For each event \(E\), there is a complementary event \(E^C\) such that:

$E\lor E^C=\Omega$

$E\land E^C=\varnothing$

This exists by construction in the measure space.

\subsection{Union and intersection}

As events are sets, we can define algebra on sets. For example for two events \(E_i\) and \(E_j\) we can define:

\begin{itemize}
\item \(E_i\land E_j\)
\item \(E_i\lor E_j\)
\end{itemize}


\section{Moment Generating Functions, Laplace Transforms}

The moment generating function (MGF) of a random variable $X$ is:

\begin{equation}
\psi_X(t) = \mathbb{E}(e^{tX}) = \int e^{tx}dF(x)
\end{equation}

The MGF allows for interchanging the operations of differentiation and "taking expectation".

\section{Cumulant Generating Functions}
The cumulants provide an alternative to the moments of the distribution.

\begin{equation}
K_X(t) = \log\mathbb{E}(e^{tX}) = \log\int e^{tx}dF(x)
\end{equation}

Then the $n$th cumulant $\kappa_n = K_X^{(n)}(0)$ is the $n$th derivative of the cumulant generating function.

The first cumulant is the mean, the second cumulant is the variance.

\section{Characteristic Function}

\begin{equation}
C_X(t) = \mathbb{E}(e^{itX}) = \int e^{itx}dF(x)
\end{equation}

The characteristic function has the advantage that it is well-defined for all real values of $t$ even when $\mathbb{E}e^{tX}$ is not well defined.


\subsection{Powerset function}

The power set of \(s\), \(P(s)\), contains all subsets of \(s\).

\(\forall x x\subseteq s \leftrightarrow x\in P(x)\)

Do all subsets exist?? show elsewhere.


\input{../header_class}

%---------- start document ---------- %
 \section{intresidue -- integer residue}\linkedzero{intresidue}
intresidue module provides integer residue classes or $\mathbf{Z}/m\mathbf{Z}$.

 \begin{itemize}
   \item {\bf Classes}
   \begin{itemize}
     \item \linkingone{intresidue}{IntegerResidueClass}
     \item \linkingone{intresidue}{IntegerResidueClassRing}
   \end{itemize}
   %\item {\bf Functions}
   %  \begin{itemize}
   %    \item \linkingone{rational}{innerProduct}
   %  \end{itemize}
 \end{itemize}

\C

 \subsection{IntegerResidueClass -- integer residue class}\linkedone{intresidue}{IntegerResidueClass}

 This class is a subclass of \linkingone{ring}{CommutativeRingElement}.

  \initialize
  \func{IntegerResidueClass}
       {\hiki{representative}{integer},\
       \hiki{modulus}{integer}}
       {\out{Integer}}\\
  \spacing
  % document of basic document
  \quad Create a residue class of modulus with residue representative.
  % added document
  \spacing
  % input, output document
  \param{modulus} must be positive integer.
  %\begin{at}
  %  \item[compo]\linkedtwo{vector}{Vector}{compo}:\\ It expresses component of Vector.
  %\end{at}
  \begin{op}
    \verb|a+b| & addition.\\
    \verb|a-b| & subtraction.\\
    \verb|a*b| & multiplication.\\
    \verb|a/b| & division.\\
    \verb|a**i,pow(a,i)| & power.\\
    \verb|-a| & negation.\\
    \verb|+a| & make a copy.\\
    \verb|a==b| & equality or not.\\
    \verb|a!=b| & inequality or not.\\
    \verb|repr(a)| & return representation string.\\
    \verb|str(a)| & return string.\\
  \end{op}
%\begin{ex}
%>>> A = vector.Vector([1,2])
%>>> A
%Vector([1, 2])
%>>> A.compo
%[1, 2]
%>>>
%\end{ex}%Don't indent!
  \method
  \subsubsection{getRing -- get ring object}\linkedtwo{intresidue}{IntegerResidueClassRing}{getRing}
   \func{getRing}{\param{self}}{\out{IntegerResidueClassRing}}\\
   \spacing
   % document of basic document
   \quad Return a ring to which it belongs.
   %\spacing
   % added document
   %\quad \negok Note that this function returns integer only.\\
   %\spacing
   % input, output document
   %\quad \param{a} must be int, long or rational.Integer.\\
%
  \subsubsection{getResidue -- get residue}\linkedtwo{intresidue}{IntegerResidueClassRing}{getResidue}
   \func{getResidue}{\param{self}}{\out{integer}}\\
   \spacing
   % document of basic document
   \quad Return the value of residue.
   %\spacing
   % added document
   %\quad \negok Note that this function returns integer only.\\
   %\spacing
   % input, output document
   %\quad \param{a} must be int, long or rational.Integer.\\
%
  \subsubsection{getModulus -- get modulus}\linkedtwo{intresidue}{IntegerResidueClassRing}{getModulus}
   \func{getModulus}{\param{self}}{\out{integer}}\\
   \spacing
   % document of basic document
   \quad Return the value of modulus.
   %\spacing
   % added document
   %\quad \negok Note that this function returns integer only.\\
   %\spacing
   % input, output document
   %\quad \param{a} must be int, long or rational.Integer.\\
%
  \subsubsection{inverse -- inverse element}\linkedtwo{intresidue}{IntegerResidueClassRing}{inverse}
   \func{inverse}{\param{self}}{\out{IntegerResidueClass}}\\
   \spacing
   % document of basic document
   \quad Return the inverse element if it is invertible. Otherwise raise ValueError.
   %\spacing
   % added document
   %\quad \negok Note that this function returns integer only.\\
   %\spacing
   % input, output document
   %\quad \param{a} must be int, long or rational.Integer.\\
%
  \subsubsection{minimumAbsolute -- minimum absolute representative}\linkedtwo{intresidue}{IntegerResidueClassRing}{minimumAbsolute}
   \func{minimumAbsolute}{\param{self}}{\out{\linkingone{rational}{Integer}}}\\
   \spacing
   % document of basic document
   \quad  Return the minimum absolute representative integer of the residue class.
   %\spacing
   % added document
   %\quad \negok Note that this function returns integer only.\\
   %\spacing
   % input, output document
   %\quad \param{a} must be int, long or rational.Integer.\\
%
  \subsubsection{minimumNonNegative -- smallest non-negative representative}\linkedtwo{intresidue}{IntegerResidueClassRing}{minimumNonNegative}
   \func{minimumNonNegative}{\param{self}}{\out{\linkingone{rational}{Integer}}}\\
   \spacing
   % document of basic document
   \quad Return the smallest non-negative representative element of the residue class.
   %\spacing
   % added document
   \quad \negok this method has an alias, named toInteger.\\
   %\spacing
   % input, output document
   %\quad \param{a} must be int, long or rational.Integer.\\
%
%\begin{ex}
%>>> A = module.HogeClass((1,2))
%>>> A.hogemethod1(2)
%(2, 4)
%>>>
%\end{ex}%Don't indent!
\C
 \subsection{IntegerResidueClassRing -- ring of integer residue}\linkedone{intresidue}{IntegerResidueClassRing}
 The class is for rings of integer residue classes.

 This class is a subclass of \linkingone{ring}{CommutativeRing}.

  \initialize
  \func{IntegerResidueClassRing}{\hiki{modulus}{integer}}{\out{IntegerResidueClassRing}}\\
  \spacing
  % document of basic document
  \quad Create an instance of IntegerResidueClassRing.
  % added document
  The argument \param{modulus} = $m$ specifies an ideal $m\mathbb{Z}$.
  % \spacing
  % input, output document
  %See \linkingone{module}{point} for \param{point}.
  \begin{at}
    \item[zero]\linkedtwo{integer}{IntegerRing}{zero}:\\ It expresses The additive unit 0. (read only)
    \item[one]\linkedtwo{integer}{IntegerRing}{one}:\\ It expresses The multiplicative unit 1. (read only)
  \end{at}
  \begin{op}
  %  \verb|+| & Vector sum.\\
  %  \verb|-| & Vector subtraction.\\
  %  \verb|*| & Scalar multiplication.\\
  %  \verb|//| & Scalar division.\\
  %  \verb|-(unary)| & element negation.\\
    \verb|R==A| & ring equality.\\
  %  \verb|!=| & inequality or not.\\
  %  \verb+V[i]+ & Return the coefficient of i-th element of Vector.\\
  %  \verb+V[i] = c+ & Replace the coefficient of i-th element of Vector by c.\\
    \verb|card(R)| & return cardinality. See also \linkingzero{compatibility} module.\\
    \verb|e in R| & return whether an element is in or not.\\
    \verb|repr(R)| & return representation string.\\
    \verb|str(R)| & return string.\\
  \end{op}
%\begin{ex}
%>>> A = vector.Vector([1,2])
%>>> A
%Vector([1, 2])
%>>> A.compo
%[1, 2]
%>>>
%\end{ex}%Don't indent!
  \method
  \subsubsection{createElement -- create IntegerResidueClass object}\linkedtwo{intresidue}{IntegerResidueClassRing}{createElement}
   \func{createElement}{\param{self},\ \hiki{seed}{integer}}{\out{Integer}}\\
   \spacing
   % document of basic document
   \quad Return an IntegerResidueClass instance with \param{seed}.
   %\spacing
   % added document
   %\quad \negok Note that this function returns integer only.\\
   %\spacing
   % input, output document
   %\quad \\
%
  \subsubsection{isfield -- field test}\linkedtwo{intresidue}{IntegerResidueClassRing}{isfield}
   \func{isfield}{\param{self}}{\out{bool}}\\
   \spacing
   % document of basic document
   \quad Return True if the modulus is prime, False if not. Since a finite domain is a field, other ring property tests are merely aliases of isfield; they are isdomain, iseuclidean, isnoetherian, ispid, isufd.
   % added document
   %\quad \negok Note that this function returns integer only.\\
   %\spacing
   % input, output document
   %\quad if \param{as\_column} is True, try to create column matrix.\\
%
  \subsubsection{getInstance -- get instance of IntegerResidueClassRing}\linkedtwo{intresidue}{IntegerResidueClassRing}{getInstance}
   \func{getInstance}{\param{cls},\ \hiki{modulus}{integer}}{\out{IntegerResidueClass}}\\
   \spacing
   % document of basic document
   \quad Return an instance of the class of specified modulus. Since this is a class method, use it as:

\verb|IntegerResidueClassRing.getInstance(3)|

to create a $\mathbb{Z}/3\mathbb{Z}$ object, for example.
%\begin{ex}
%>>> A = module.HogeClass((1,2))
%>>> A.hogemethod1(2)
%(2, 4)
%>>>
%\end{ex}%Don't indent!

\C

%---------- end document ---------- %

\input{../footer}


\chapter{Backtracking}
\section{Introduction}
\runinhead{Difference between backtracking and dfs.} \textit{Backtracking} is a more general purpose algorithm. \textit{Dfs} is a specific form of backtracking related to searching tree structures.

\runinhead{Prune.} Backtrack need to think about pruning using the condition \pyinline{predicate}.

\section{Sequence}
\runinhead{k sum.} Given $n$ unique integers, number $k$ and target. Find all possible $k$ integers where their sum is target.

Complexity: $O(2^n)$.

Pay attention to the pruning condition.

\begin{python}
def dfs(self, A, i, k, cur, remain, ret):
    """self.dfs(A, 0, k, [], target, ret)"""
    if len(cur) == k and remain == 0:
        ret.append(list(cur))
        return

    if (i >= len(A) or len(cur) > k
        or len(A)-i+len(cur) < k):
        return

    self.dfs(A, i+1, k, cur, remain, ret)
    cur.append(A[i])
    self.dfs(A, i+1, k, cur, remain-A[i], ret)
    cur.pop()
\end{python}

\section{String}
\subsection{Palindrome}
\subsubsection{Palindrome partition.} Given \pyinline{s = "aab"}, return: \\
\pyinline{[["aa","b"], ["a","a","b"]]}
\\
\runinhead{Core clues:}
\begin{enumerate}
\item Expand the search tree \textbf{horizontally}.
\end{enumerate}
\rih{Search process:}
\begin{python}
input: "aabbc"

"a", "abbc"
     "a", "bbc"
          "b", "bc"
               "b", "c" (o)
               "bc" (x)
          "bb", "c" (o)
          "bbc" (x)
     "ab", "bc" (x)
     "abb", "c" (x)
     "abbc" (x)
"aa", "bbc"
      "b", "bc"
           "b", "c" (o)
           "bc" (x)
      "bb", "c" (o)
      "bbc" (x)
"aab", "bc" (x)
"aabb", "c" (x)
\end{python}
Code:

\begin{python}
def partition(self, s):
    ret = []
    self.backtrack(s, [], ret)
    return ret

def backtrack(self, s, cur_lvl, ret):
    """
    Let i be the scanning ptr.
    If s[:i] passes predicate, then backtrack s[i:]
    """
    if not s:
        ret.append(list(cur_lvl))

    for i in xrange(1, len(s)+1):
        if self.predicate(s[:i]):
            cur_lvl.append(s[:i])
            self.backtrack(s[i:], cur_lvl, ret)
            cur_lvl.pop()

def predicate(self, s):
    return s == s[::-1]
\end{python}

\section{Math}
\subsection{Decomposition}
\subsubsection{Factorize a number}\label{factorization}
\runinhead{Core clues:}
\begin{enumerate}
\item Expand the search tree \textbf{horizontally}.
\end{enumerate}
\runinhead{Search tree:}
\begin{python}
Input: 16
get factors of cur[-1]
[16]
[2, 8]
[2, 2, 4]
[2, 2, 2, 2]

[4, 4]
\end{python}
Code:
\begin{python}
def dfs(self, cur, ret):
  if len(cur) > 1:
    ret.append(list(cur))

  n = cur.pop()
  start = cur[-1] if cur else 2
  for i in xrange(start, int(sqrt(n))+1):
    if self.predicate(n, i):
      cur.append(i)
      cur.append(n/i)
      self.dfs(cur, ret)
      cur.pop()

def predicate(self, n, i):
  return n%i == 0

\end{python}
\runinhead{Time complexity.} The search tree's size is $O(2^n)$ where $n$ is the number
of prime factors. Choose $i$ prime factors to combine then, and keep the rest uncombined：

$$\sum_i {n \choose i} = 2^n$$

\section{Arithmetic Expression}
\subsection{Unidirection}
\rih{Insert operators.} Given a string that contains only digits 0-9 and a target value,
return all possibilities to add binary operators (not unary) +, -, or * between the
digits so they evaluate to the target value.

Example:
\begin{align*}
``123", 6 \rightarrow [``1+2+3", ``1*2*3"] \\
``232", 8 \rightarrow [``2*3+2", ``2+3*2"] \\
\end{align*}
Clues:
\begin{enumerate}
\item Backtracking with \textit{horizontal} expanding
\item Special handling for multiplication - caching the expression \textit{predecessor}
for multiplication association.
\item Detect \textit{invalid} number with leading 0's
\end{enumerate}

\begin{python}
def addOperators(self, num, target):
  ret = []
  self.dfs(num, target, 0, "", 0, 0, ret)
  return ret

def dfs(self, num, target, pos,
        cur_str, cur_val,
        mul, ret
):
  if pos >= len(num):
    if cur_val == target:
      ret.append(cur_str)
  else:
    for i in xrange(pos, len(num)):
      if i != pos and num[pos] == '0':
        continue

      nxt_val = int(num[pos:i+1])
      if not cur_str:  # 1st number
        self.dfs(num, target, i+1,
            "%d"%nxt_val, nxt_val,
            nxt_val, ret)
      else:  # +, -, *
        self.dfs(num, target, i+1,
            cur_str+"+%d"%nxt_val, cur_val+nxt_val,
            nxt_val, ret)
        self.dfs(num, target, i+1,
            cur_str+"-%d"%nxt_val, cur_val-nxt_val,
            -nxt_val, ret)
        self.dfs(num, target, i+1,
    cur_str+"*%d"%nxt_val, cur_val-mul+mul*nxt_val,
            mul*nxt_val, ret)
\end{python}
\subsection{Bidirection}
\rih{Insert parenthesis.} Given a string of numbers and operators, return all possible
results from computing all the different possible ways to group numbers and operators.
The valid operators are +, - and *.

Examples:
\begin{align*}
(2*(3-(4*5))) &= -34 \\
((2*3)-(4*5)) &= -14 \\
((2*(3-4))*5) &= -10 \\
(2*((3-4)*5)) &= -10 \\
(((2*3)-4)*5) &= 10
\end{align*}
Clues: Iterate the operators, divide and conquer - left parts and right parts and then
combine result. \\
Code:
\begin{python}
def dfs_eval(self, nums, ops):
  ret = []
  if not ops:
    assert len(nums) == 1
    return nums

  for i, op in enumerate(ops):
    left_vals = self.dfs_eval(nums[:i+1], ops[:i])
    right_vals = self.dfs_eval(nums[i+1:], ops[i+1:])
    for l in left_vals:
      for r in right_vals:
        ret.append(self._eval(l, r, op))

  return ret
\end{python}
\section{Tree}
\subsection{BST}
\subsubsection{Generate Valid BST}
Generate all valid BST with nodes from 1 to $n$.
\runinhead{Core clues:}
\begin{enumerate}
\item Iterate pivot
\item Generate left and right
\end{enumerate}
Code:
\begin{python}
def generate(self, start, end):
  roots = []
  if start > end:
    roots.append(None)
    return roots

  for pivot in range(start, end+1):
    left_roots = self.generate_cache(start, pivot-1)
    right_roots = self.generate_cache(pivot+1, end)

    for left_root in left_roots:
      for right_root in right_roots:
        root = TreeNode(pivot)
        root.left = left_root
        root.right = right_root

        roots.append(root)

  return roots

\end{python}


\section{Formulary: Queueing Theory}
\label{sec:Formulary-Queueing-Theory}

\subsection{Foundations}

\begin{description}

	\item [Inter-arrival time]
		\begin{equation}
		\expected{\tau} = \frac{1}{\lambda}
		\end{equation}

	\item [Service time]
		\begin{equation}
		\expected{S} = \frac{1}{\mu}
		\end{equation}

	\item [Response time]
		\begin{equation}
		\expected{T} = \expected{T_{Q}} + \expected{S}
		\end{equation}

	\item [Jobs]
		\begin{equation}
		\expected{N} = \expected{N_{Q}} + \expected{N_{S}}
		\end{equation}

	\item [Server demand]
		\begin{equation}
		\expected{D_{i}} = \expected{V_{i}} \cdot \expected{S_{i}}
		\end{equation}

		\begin{equation}
		\expected{D_{i}} = \frac{B_{i}}{C}
		\end{equation}

	\item [Total demands]
		\begin{equation}
		D = \sum_{i=1}^{m}\expected{D_{i}}
		\end{equation}

	\item [Maximum demand]
		\begin{equation}
		D_{max} = \max\{\expected{D_{i}}\}
		\end{equation}

\end{description}

\subsection{Operational Laws}

\begin{description}

	\item [Stability Law]
		\begin{equation}
		\lambda < \mu
		\end{equation}

	\item [Little's Law for open systems]
		\begin{equation}
		\expected{N} = \lambda \cdot \expected{T}
		\end{equation}

	\item [Little's Law for batch systems]
		\begin{equation}
		N = X \cdot \expected{T}
		\end{equation}

	\item [Little's Law for interactive systems]
		\begin{equation}
		\expected{T_{R}} = \frac{N}{X} - \expected{Z}
		\end{equation}

	\item [Utilization Law]
		\begin{equation}
		\varrho_{i} = \frac{\lambda_{i}}{\mu_{i}}
		\end{equation}

		\begin{equation}
		\varrho_{i} = X \cdot \expected{D_{i}}
		\end{equation}

		\begin{equation}
		\varrho_{i} = \probability{server \; i \; busy} = \expected{requests \; to \; server \; i} = \expected{N_{S,i}}
		\end{equation}

	\item [Forced Flow Law]
		\begin{equation}
		X_{i} = \expected{V_{i}} \cdot X
		\end{equation}

	\item [Bottleneck Law]
		\begin{equation}
		\expected{D_{i}} = \expected{V_{i}} \cdot \expected{S_{i}}
		\end{equation}

\end{description}

\subsection{Asymptotic Analysis for closed systems}

\begin{description}

	\item [Throughput Asymptotes]
		\begin{equation}
		X \leq \min \Big\{ \frac{N}{D + \expected{Z}} , \frac{1}{D_{max}} \Big\}
		\end{equation}

	\item [Response Time Asymptotes]
		\begin{equation}
		\expected{R} \geq \max\{D , N \cdot D_{max} - \expected{Z} \}
		\end{equation}

	\item [Queue Point]
		\begin{equation}
		N^{*} = \frac{D + \expected{Z}}{D_{max}}
		\end{equation}

\end{description}

\subsection{M/M/1}

\begin{description}

	\item [Utilization]
		\begin{equation}
		\varrho = \frac{\lambda}{\mu}
		\end{equation}

	\item [State Probability]
		\begin{equation}
		\pi_{i} = \varrho^{i}(1-\varrho)
		\end{equation}

	\item [Mean System Jobs]
		\begin{equation}
		\expected{N} = \frac{\varrho}{(1-\varrho)}
		\end{equation}

	\item [Mean Response Time]
		\begin{equation}
		\expected{T} = \frac{1}{\mu-\lambda}
		\end{equation}

\end{description}

\subsection{M/M/m}

\begin{description}

	\item [Utilization]
	Both system and each server
		\begin{equation}
		\varrho = \frac{\lambda}{m\mu}
		\end{equation}

	\item [Resource Requirement]
		\begin{equation}
		R = \expected{N_{S}} = \frac{\lambda}{\mu}
		\end{equation}

	\item [State Probability]
		\begin{equation}
		\pi_{i} = \left\{\begin{matrix}
		\Big(\frac{\lambda}{\mu}\Big)^{i}\frac{1}{i!}\pi_{0} & i \leq m\\
		\Big(\frac{\lambda}{\mu}\Big)^{i}\frac{1}{m!}\Big(\frac{1}{m}\Big)^{i-m}\pi_{0} & i > m
		\end{matrix}\right.
		\end{equation}

		or equivalently

		\begin{equation}
		\pi_{i} = \frac{m^{q} \varrho^{i}}{q!} \cdot \pi_{0} \qquad with \quad q=\min\{i,m\}
		\end{equation}

		with

		\begin{equation}
		\pi_{0} = \Big[ \sum_{i=0}^{m-1} \frac{(m \varrho)^{i}}{i!} + \frac{(m \varrho)^{m}}{m! (1- \varrho)} \Big]^{-1}
		\end{equation}

		where $q$ is the number of busy servers.

	\item [Queue Probability]
		\begin{equation}
		P_{Q} = \frac{(m \varrho)^{m}}{m!(1-\varrho)} \cdot \pi_{0}
		\end{equation}

	\item [Mean Queue Jobs]
		\begin{equation}
		\expected{N_{Q}} = \frac{\varrho}{(1-\varrho)} \cdot P_{Q}
		\end{equation}

\end{description}

\subsection{M/M/m/m}

\begin{description}

	\item [Utilization]
	Both system and each server
		\begin{equation}
		\varrho = \frac{\lambda}{m\mu}
		\end{equation}

	\item [State Probability]
		\begin{equation}
		\pi_{i} = \Big( \frac{\lambda}{\mu} \Big)^{i} \frac{1}{i!} \pi_{0}
		\end{equation}

		with

		\begin{equation}
		\pi_{0} = \Big[ \sum_{i=0}^{m} \Big( \frac{\lambda}{\mu} \Big)^{i} \frac{1}{i!} \Big]^{-1}
		\end{equation}

	\item [Block Probability]
		\begin{equation}
		P_{block} = \pi_{m} = \frac{\Big(\frac{\lambda}{\mu}\Big)^{m} \frac{1}{m!}}{\sum_{i=0}^{m} \Big( \frac{\lambda}{\mu} \Big)^{i} \frac{1}{i!}}
		\end{equation}

		or equivalently

		\begin{equation}
		\begin{split}
		P_{block} & = \pi_{m} \cdot \frac{e^{-\frac{\lambda}{\mu}}}{e^{-\frac{\lambda}{\mu}}} \\
		& = \frac{e^{-\frac{\lambda}{\mu}} \cdot \Big(\frac{\lambda}{\mu}\Big)^{m} \frac{1}{m!}}{\sum_{i=0}^{m} e^{-\frac{\lambda}{\mu}} \cdot \Big( \frac{\lambda}{\mu} \Big)^{i} \frac{1}{i!}} \\
		& = \frac{\probability{X = m}}{\probability{X \leq m}}
		\end{split}
		\end{equation}

		where $X \sim Poisson(\frac{\lambda}{\mu})$.

\end{description}

\subsection{M/M/\texorpdfstring{$\infty$}{Infinity}}

\begin{description}

	\item [Utilization]
		\begin{equation}
		\varrho = \frac{\lambda}{\mu}
		\end{equation}

	\item [State Probability]
		\begin{equation}
		\pi_{i} = \Big(\frac{\lambda}{\mu})^{i} \frac{1}{i!} \pi_{0}
		\end{equation}

		with

		\begin{equation}
		\pi_{0} = e^{-\frac{\lambda}{\mu}}
		\end{equation}

		That is $N^{M/M/\infty} \sim Poisson(\frac{\lambda}{\mu})$.

	\item [Mean System Jobs]
		\begin{equation}
		\expected{N} = \frac{\lambda}{\mu}
		\end{equation}

	\item [Mean Queue Jobs]
		\begin{equation}
			\expected{N_{Q}} = 0
		\end{equation}

	\item [Response Time]
		\begin{equation}
			\expected{T} = \expected{T_{S}} = \frac{1}{\mu}
		\end{equation}

	\item [Delay]
		\begin{equation}
		\expected{T_{Q}} = 0
		\end{equation}

\end{description}

\subsection{M/G/1}

\begin{description}

	\item [Mean Queue/Excess Time]
		\begin{equation}
		\expected{T_{Q}}=\frac{\varrho}{1-\varrho}\expected{S_{e}}
		\end{equation}

	\item [Pollaczek-Khinchin Formula]
		\begin{equation}
		\expected{T_{Q}}=\frac{\varrho}{1-\varrho}\cdot\frac{\expected{S^{2}}}{2\expected{S}}
		\end{equation}

		\begin{equation}
		\expected{T_{Q}}=\frac{\varrho}{1-\varrho}\cdot\frac{\expected{S}}{2}\cdot(C_{S}^{2}+1)
		\end{equation}

		\begin{equation}
		\expected{T_{Q}}=\frac{\lambda\expected{S^{2}}}{2(1-\varrho)}
		\end{equation}

\end{description}

\subsection{Relations between queues}

\begin{description}

	\item [Block M/M/m/m vs. Queueing M/M/1]
		\begin{equation}
		P_{block} = \frac{(1-\varrho)P_{Q}}{1-\varrho P_{Q}}
		\end{equation}

\end{description}

\subsection{Capacity Provisioning}

\begin{description}

	\item [Square-Root Staffing Rule]
	For any $M/M/m$ with high resource requirement $R$, the minimum number of servers $m_{\alpha}^{*}$ to ensure $P_{Q} < \alpha$ is

	\begin{equation}
	m_{\alpha}^{*} \approx R + c \sqrt{R}
	\end{equation}

	with $c$ solution of the equation

	\begin{equation}
	\frac{c \Phi(c)}{\phi(c)} = \frac{1 - \alpha}{\alpha}
	\end{equation}

	where
	$\Phi(\cdot)$ is the c.d.f. of the Standard Normal, and
	$\phi(\cdot)$ is its p.d.f.
\end{description}

\subsection{Open Jackson Networks}

\begin{description}

	\item [State Probability]
		\begin{equation}
		\pi_{n_{1},...,n_{m}} = \prod_{i=1}^{m} \varrho_{i}^{n_{i}} (1-\varrho_{i})
		\end{equation}

	\item [Probability Jobs at Server]
		\begin{equation}
		\probability{n_{i} jobs at server i} = \varrho_{i}^{n_{i}} (1-\varrho_{i})
		\end{equation}

	\item [Mean Server Jobs]
		\begin{equation}
		\expected{N_{i}} = \frac{\varrho_{i}}{1 - \varrho_{i}}
		\end{equation}

\end{description}

\subsection{Closed Jackson Networks}

\begin{description}

	\item [State Probability]
		\begin{equation}
		\pi_{n_{1},...,n_{m}} = C \cdot \prod_{i=1}^{m} \varrho_{i}^{n_{i}}
		\end{equation}

		where $C$ is the \textit{Normalizing Constant} determined as the solution of

		$
		\sum_{(n_{1},...,n_{m}) \\ \in States} \pi_{n_{1},...,n_{m}} = 1
		$

\end{description}

\subsection{Mean Value Analysis}

\begin{description}

	\item [MVA Response Time]
		\begin{equation}
		\expected{T_{j}^{M}} =
		\frac{1}{\mu_{j}} +
		\frac{p_{j} \lambda^{(M-1)} \expected{T_{j}^{(M-1)}}}{\mu_{j}}
		\end{equation}

	\item [MVA Fraction of Arrivals]
		\begin{equation}
		p_{j} = \frac{\lambda_{j}^{M}}{\lambda^{M}} = \frac{V_{j}}{\sum_{j=1}^{m} V_{j}}
		\end{equation}

	\item [MVA Total Arrival Rate]
		\begin{equation}
		\lambda^{(M-1)} = \frac{M-1}{\sum_{j=1}^{m} p_{j} \expected{T_{j}^{(M-1)}}}
		\end{equation}

\end{description}

\subsection{Network of PS Servers}

\begin{description}

	\item [BCMP Theorem]
		\begin{equation}
		\pi_{n_{1},...,n_{m}} = \prod_{i=1}^{m} \varrho_{i}^{n_{i}} (1-\varrho_{i})
		\end{equation}
		where $\varrho_{i}=\lambda_{i}\expected{S_{i}}$

	\item [Probability Jobs at Server]
		\begin{equation}
		\probability{n_{i} jobs at server i} = \varrho_{i}^{n_{i}} (1-\varrho_{i})
		\end{equation}
		where $\varrho_{i}=\lambda_{i}\expected{S_{i}}$

\end{description}

\subsection{Renewal-Reward Theory}

\begin{description}

	\item [Renewal-Reward Theorem]
		\begin{equation}
		\lim_{t \to \infty}\frac{R(t)}{t}=\frac{\expected{R}}{\expected{X}} \quad \forall 0\leq\expected{R}<\infty,0<\expected{X}<\infty
		\end{equation}
		where $X$ is the inter-arrival time, $R is the number of rewards earned$, and $R(t)$ is the total rewards by time $t$.

\end{description}

\subsection{Task Assignment for Server Farms}

\begin{description}

	\item [Spare Server]
		\begin{equation}
		\sharp SpareServers=m-\lceil R\rceil
		\end{equation}

	\item [Mean Queue Time for M/G/m]
		\begin{equation}
		\expected{T_{Q}^{M/G/m}}\approx\Big(\frac{C^{2}+1}{2}\Big)\expected{T_{Q}^{M/M/m}}
		\end{equation}

	\item [Worst-Case Competitive Ratio]
		\begin{equation}
		CR_{\mathcal{P}}=\max_{\mathcal{A}}r_{\mathcal{P}}(\mathcal{A})
		\end{equation}

		with

		\begin{equation}
		r_{\mathcal{P}}(\mathcal{A})=\frac{\expected{T(\mathcal{A})}^{\mathcal{P}}}{\expected{T(\mathcal{A})}^{\mathcal{OPT}}}
		\end{equation}

\end{description}

\subsection{Scheduling}

\begin{description}

	\item [Slowdown]
		\begin{equation}
		Slowdown=\frac{T}{S}
		\end{equation}

		\begin{equation}
		\expected{Slowdown(x)}=\frac{\expected{T(x)}}{x}
		\end{equation}

		\begin{equation}
		\expected{Slowdown}=\int_{x}\frac{\expected{T(x)}}{x}f_{S}(x)\partial x
		\end{equation}

	\item [Fairness]
		\begin{equation}
		\expected{Slowdown^{\mathit{P}}(x)}<\expected{Slowdown^{\mathit{PS}}(x)}\quad\forall x
		\end{equation}

	\item [Fairness]
		\begin{equation}
		\expected{Slowdown^{\mathit{P}}(x)}>\expected{Slowdown^{\mathit{PS}}(x)}\quad\exists x
		\end{equation}

	\item [Starvation]
		\begin{equation}
		\exists x.\expected{Slowdown^{\mathit{P}}(x)}>\expected{Slowdown^{\mathit{PS}}(x)}
		\end{equation}

	\item [Response Time for M/G/1/FCFS]
		\begin{equation}
		\expected{T}=\expected{S}+\frac{\lambda\expected{S^{2}}}{2(1-\varrho)}
		\end{equation}

	\item [Response Time for M/G/1/PS]
		\begin{equation}
		\expected{T(x)}^{\mathit{M/G/1/PS}}=\frac{x}{1-\varrho}
		\end{equation}

	\item [Response Time for M/G/1/PLCFS]
		\begin{equation}
		\expected{T(x)}^{\mathit{M/G/1/PLCFS}}=\frac{x}{1-\varrho}
		\end{equation}

	\item [Queue Time for M/G/1/FB]
		\begin{equation}
		\expected{T(x)}^{\mathit{FB}}=
		\frac{x(1-\varrho_{\overline{x}})+\frac{1}{2}\lambda\expected{S_{\overline{x}}^{2}}}
			{\Big(1-\varrho_{\overline{x}}\Big)^{2}}
		\end{equation}

	\item [Queue Time for M/G/1/SJF]
		\begin{equation}
		\expected{T_{Q}(x)}^{\mathit{SJF}}=
		\frac{\varrho\expected{S^{2}}}{2\expected{S}}\cdot\frac{1}{\Big(1-\varrho_{x}\Big)^{2}}
		\end{equation}

		where $\varrho_{x}=\lambda F(x)\cdot\int_{t=0}^{x}t\frac{f(t)}{F(t)}\partial t$ is the arrival rate of jobs of size no more than $x$, namely $\lambda F(x)$, multiplied by the expected size of jobs of size no more than $x$, namely $\int_{t=0}^{x}t\frac{f(t)}{F(t)}\partial t$.

	\item [Response Time for M/G/1/PSJF]
		\begin{equation}
		\expected{T(x)}^{\mathit{PSJF}}=
		\expected{Res(x)}^{\mathit{PSJF}}+\expected{Wait(x)}^{\mathit{PSJF}}=
		\frac{x}{1-\varrho_{x}}+
		\frac{\frac{\lambda}{2}\int_{0}^{x}f(t)t^{2}\partial t}
		{\Big(1-\varrho_{x}\Big)^{2}}
		\end{equation}
		where $\varrho_{x}=\lambda\int_{t=0}^{x}tf(t)\partial t$ is the load made up by jobs of size less than $x$.

	\item [Response Time for M/G/1/SRPT]
		\begin{equation}
		\expected{T(x)}^{\mathit{SRPT}}=
		\expected{Wait(x)}^{\mathit{SRPT}}+\expected{Res(x)}^{\mathit{SRPT}}=
		\frac{\lambda}{2}\frac{\int_{t=0}^{x}t^{2}f(t)\partial t + x^{2}(1-F(x))}{\Big(1-\varrho_{x}\Big)^{2}}+
		\int_{t=0}^{x}\frac{1}{1-\varrho_{t}}
		\end{equation}

	\item [Relation between scheduling policies]

	\begin{equation}
	\expected{T(x)}^{\mathit{SRPT}}\leq\expected{T(x)}^{\mathit{PS}}\quad \forall x,\forall\varrho<\frac{1}{2}
	\end{equation}

	\begin{equation}
	\expected{T(x)}^{\mathit{SRPT}}\leq\expected{T(x)}^{\mathit{FB}}
	\end{equation}

	\begin{equation}
	\expected{T^{\mathit{FCFS}}}=\expected{T^{\mathit{LCFS}}}=\expected{T^{\mathit{RO}}}
	\end{equation}

\end{description}


\section{Surface traction}
\label{iso_section:surface_traction}
\paragraph{}
In structural analysis, it is common to have boundary conditions such as displacement constraints and applied loads.
Due to the property of the NURBS that the control points are not necessarily on the curve, surface tractions can not be applied by the same method used in conventional numerical method like FEM or SBFEM.
A surface traction $\Phi$ can be regarded as a Neumann boundary condition which can be expressed as
    \begin{equation}
        {F}=-\int_{\Gamma}
        [N]
        \Phi_n
        d\Gamma
    \label{iso_eq:neumann_bc}
    \end{equation}
where $[N]$ describe the shape functions and $\Phi_n$ is the surface tractions on the nodes.

\paragraph{}
As mentioned in \ref{iso_section:numerical_integration}, numerical integration would be preferred over mathematical deduction when the target function is an input.
In the flavour of numerical integration, Eq.~\ref{iso_eq:neumann_bc} can be expressed as followed.
    \begin{equation}
        {F}=-\sum_{i=1}^n
        a_i
        [N(\xi_i)]
        \Phi_n
    \label{iso_eq:neumann_bc_numerical}
    \end{equation}
where $\xi_i$ is the integration point and $a_i$ is the weight,
$n$ is the number of integration points and different quadrature rules require different number of points in order to achieve an optimal accuracy.

\paragraph{}
It can be found that the term $[N(\xi_i)] \Phi_n$ is corresponding to $f(x)$ in Eq.~\ref{iso_eq:numerical_integration}.
In conventional FEM or SBFEM, $\Phi_n$ can be determined as the real values on the nodes because, geometrically speaking, its shape function is interpolated from the given set of points.
In other words, all nodes that determine the shape function in traditional FEM or SBFEM must be on the interpolating function.
However, this is not the case in NURBS curves where it is the control points that play the same role as the nodes in existing shape function.
In NURBS curves, apart from the first and the last points, the control points are not necessarily on the curves as illustrated in Fig.~\ref{lr_fig:nurbs_knotins}.
This prevent us from adopting the physical value on the nodes as $\Phi_n$ in Eq.~\ref{iso_eq:neumann_bc_numerical}.
Instead, a set of ``control stress'' $\Phi_c$, the control points of another NURBS curve that represent the surface traction geometrically, need to be determined as
    \begin{equation}
        \Phi_c = \argmin_{\Phi_c}
            \frac{1}{2}
            \int_{-1}^1
            \|
                \Phi(\xi)-
                    \left[ N(\xi) \right]
                    \Phi_c
            \|^2
            d\xi
    \label{iso_eq:surface_traction_fitting}
    \end{equation}
%
It means that ``control stress'' $\Phi_c$ describe a minimum mean squared error between surface traction NURBS curve and the real traction $\Phi$.
One of the simplest mathematical method to determine $\Phi_c$ will be the least square method.
Given the fact that the shape functions of this NURBS curve will be the same as that describe the geometry, $\left[ N(\xi) \right]$ can be considered as known.
By selecting $n$ sample points over the domain of the $\Phi$, Eq.~\ref{iso_eq:surface_traction_fitting} can be rewrite as
    \begin{equation}
        \Phi_c = \argmin_{\Phi_c}
            \frac{1}{n}
            \sum_{i=1}^n
            \|
                \Phi(\xi_i)-
                    \left[ N(\xi_i) \right]
                    \Phi_c
            \|^2
    \label{iso_eq:surface_traction_fitting_discrete}
    \end{equation}
Then ``control stress'' $\Phi_c$ can be solved by least square as
    \begin{equation}
        \Phi_c= \left(
            \left[ N(\xi) \right] ^T
            \left[ N(\xi) \right]
        \right)^{-1}
        \left[ N(\xi) \right]^T
        \Phi(\xi)
    \end{equation}
and Eq.~\ref{iso_eq:neumann_bc_numerical} in the case where NURBS is in use can be rewrite as
    \begin{equation}
        {F}=-\sum_{i=1}^n
        a_i
        [N(\xi_i)]
        \Phi_c
    \label{iso_eq:neumann_bc_numerical_NURBS}
    \end{equation}


\subsection{Problem Formulation}
We use TVM~\cite{Chen18} to investigate the performance of matrix tiling for GEMM. TVM facilitates tiling optimization by generating Intermediate Representation (IR) of a particular configuration. Fig.~\ref{fig:gemm_ir} is a simple example IR of GEMM tiling configuration with a blocking factor of 32 on x86 CPU for GEMM with $(m=1024, k=1024, n=1024)$ (short as $(1024,1024,1024)$).
\begin{figure}[htb]
    \centering
    \includegraphics[width=3in]{3_GEMM_backgrounds/gemm_ir.png}
    \caption{IR of GEMM with a blocking factor of 32}
    \label{fig:gemm_ir}
\end{figure}

% \begin{definition}
\noindent
\textbf{Definition:}
Generally, a GEMM tiling configuration can be defined as
\begin{equation}
    \overrightarrow{\xi} = \overrightarrow{\xi_m} \times \overrightarrow{\xi_k} \times \overrightarrow{\xi_n},
\end{equation}
where
\begin{equation}
     \overrightarrow{\xi_m} =\{ \left[m_0,\ldots,m_i, \ldots m_{d_m-1} \right] |  \Pi_{i=0}^{d_m-1} m_i=m \},
\end{equation}
\begin{equation}
     \overrightarrow{\xi_k} =\{ \left[k_0,\ldots,k_l, \ldots k_{d_k-1} \right] |  \Pi_{l=0}^{d_k-1} k_l=k \},
\end{equation}
\begin{equation}
     \overrightarrow{\xi_n} =\{ \left[n_0,\ldots,n_j, \ldots n_{d_n-1} \right] |  \Pi_{j=0}^{d_n-1} n_j=n \}.
\end{equation}
% \end{definition}
Multiplication of two matrices $A(m\times k)$ and $B(k\times n)$ produces matrix $C(m\times n)$. $d_m$, $d_k$ and $d_n$ are the number of nested loops for each dimension $m$, $k$ and $n$, respectively.  $m_i, k_l, n_j$, $\forall i \in [0, d_m)$ $\forall l \in [0, d_k)$ $\forall j \in [0, d_n)$, are the number of iterations of a respective loop. The configuration in Fig.~\ref{fig:gemm_ir} is $m_0=m_1=32$, $k_0=256, k_1=4$, $n_0=n_1=32$, and $d_m=d_k=d_n=2$.

Constrained by the definition of tiling configuration, we can formulate the following optimization problem:
\begin{equation}\nonumber
    \begin{array}{l}
       \mathop {\min }\limits_{s} T_{cost}(s ; m,k,n,d_m,d_k,d_n).
    \end{array}
\end{equation}
The objective function of this problem is to find an optimal tiling configuration that has minimal running time on target hardware. $T_{cost}$ denotes the running time for the configuration $s$, given the dimension of matrices as $m,k,n$ and the number of the nested loops on each dimension as $d_m, d_k, d_n$.

%In our experiments, we run GEMM tiling configurations on Nivida CUDA GPU, more specifically $d_m=4$, $d_k=2$ and $d_n=4$.


\chapter{Efficient Backpropagation for Convolutional Neural Networks}

{\bf Structure of CNN}.
\begin{figure}[ht]
\centering
\includegraphics[scale=0.7]{CNN}
\caption{Convolutional Neural Network}
\end{figure}
A convolutional neural network (CNN) consists of three types of layers: \textit{full-connected layers, convolutional layers, and pooling layers}. Fully-connected layers are defined in the same way as a layer in the basic neural network model. Convolutional layers and pooling layers are unique to CNN. A convolution layer in CNN is a composition of two functions: $f^{conv}$ and ${\bm\sigma(x)}$ representing \textit{quasi-convolution} and \textit{activation function} separately. A pooling layer is represented by $f^{pool}$, the \textit{pooling function}. We will provide definitions to these terminologies and function in the following sections.

\section{Basic Tensor Notations}

{\bf Tensor}
\begin{itemize}
\item
Definition of tensor: an multi-dimensional array $\mathcal{F}\in\mathbb{R}^{n_1\times n_2\times\cdots\times n_d}$.
\item
Element-wise representation:
$$
\mathcal{F}(i_1,...,i_d)\in\mathbb{R}, i_j=0,...,n_j-1, j=1,...,d.
$$
\item
Fiber: mode-1 tensor (vector) obtained by fixing every index but one.
\item
Slice: mode-2 tensor (matrix) obtained by fixing every index but two.
\item
Tensor Norm:
$$
\|\mathcal{F}\| := \sqrt{\sum_{i_1=1}^{I_1}\cdots\sum_{i_N=1}^{I_N} \mathcal{F}(i_1,...,i_N)^2}
$$
\end{itemize}
\begin{figure}[ht]
\centering
\includegraphics[scale=0.5]{fiber}
\end{figure}
\begin{figure}[ht]
\centering
\includegraphics[scale=0.5]{slice}
\end{figure}

{\bf Matricization}.
The mode-$k$ matricization of a tensor $\mathcal{X}\in\mathbb{R}^{n_1\times n_2\times\cdots\times n_d}$ is denoted by $X_{(k)}$ and arranges the mode-$k$ fibers to be the columns of the resulting matrix. Under matricization, the tensor element $\mathcal{X}(i_1,...,i_d)$ is mapped to matrix element $\big(\bm{X_{(k)}}\big)(i_k,j)$, where
\begin{equation}
j = \sum_{1\le s\le d, s\neq k}\big(i_s\prod_{1\le t\le s-1, t\neq k}n_t\big)
\end{equation}
An example is given in Figure \ref{fmat}.
\begin{figure}[H]\label{fmat}
\centering
\includegraphics[scale=0.6]{metricization}
\caption{Tensor Matricization}
\end{figure}

%{\bf Tensor-Matrix Multiplication}
Tucker decomposition is best illustrated through \textit{tensor-matrix multiplication}. A product of a tensor $\mathcal{X}$ and a matrix $ M=\big(m_{ij}\big)_{i,j}$ on the $k$-th mode is a tensor defined as:
\begin{equation}
    \big(\mathcal{X}\times_kM\big)[i_1,...,i_n] =
    \sum_j \mathcal{X}[i_1,...,i_{k-1},j,i_{k+1},...,i_n]m_{ji_k}.
\end{equation}
$M\times_k\mathcal{X}$ can be defined similarly as
\begin{equation}
    \big(M\times_k\mathcal{X}\big)[i_1,...,i_n] =
    \sum_j m_{i_kj}\mathcal{X}[i_1,...,i_{k-1},j,i_{k+1},...,i_n]m_{i_kj}.
\end{equation}
The following properties are easily provable:
\begin{lemma}
[Tensor-Matrix Multiplication]
\begin{equation}
\aligned
&(\mathcal{X}\times_k M)\times_l N = (\mathcal{X}\times_l N)\times_k M =: \mathcal{X}\times_k M\times_l N\\
&(\mathcal{X}\times_k M)\times_k N = \mathcal{X}\times_k(M\cdot N)
\endaligned
\end{equation}
\end{lemma}

{\bf Tensor Multiplication}.
Given $\mathcal{F}\in\mathbb{R}^{n_1\times\cdots\times n_{d_1}}$, $\mathcal{G}\in\mathbb{R}^{n_1\times\cdots\times n_{d_2}}$, and a mode $k\in\{1,...,\min(d_1,d_2)\}$, the \textit{tensor multiplication on mode $k$} is a tensor such that
\begin{equation}
\aligned
&\big(\mathcal{F}\times_k\mathcal{G}\big)(i_1,...,i_{k-1},i_{k+1},...,i_{d_1},j_1,...,j_{k-1},j_{k+1},...,j_{d_2}) \\
=& \sum_l \mathcal{F}(i_1,...,i_{k-1},l,i_{k+1},...,i_{d_1})\mathcal{G}(j_1,...,j_{k-1},l,j_{k+1},...,j_{d_2}).
\endaligned
\end{equation}
Note that the definition of tensor multiplication can be extended to multiplication on multiple modes.

\section{Quasi-Convolution in CNN}
The mathematical convolution is a operation on two functions ($f$ and $g$); it produces a third function, that is typically viewed as a modified version of one of the original function.

\begin{itemize}
\item
Continuous convolution
$(f*g)(x) := \int_{-\infty}^{\infty}f(t)g(x-t)dt$
\item
Discrete convolution
$(f*g)[n] := \sum_{-\infty}^\infty f[m]g[n-m]$
%\item
%Two-Dimensional convolution
%$(F*G)[i,j] := \sum_{s = -\infty}^\infty\sum_{t = -\infty}^\infty F[s, t]G[i-s, j-t]$
\item
d-Dimensional convolution
$$(\mathcal{F}\ast \mathcal{G})(i_1, i_2, \cdots ,i_d) = \sum_{j_i, j_2,\cdots,j_d \in \mathbb{Z}} \mathcal{F}(j_1, j_2,\cdots,j_d)\mathcal{G}(i_1 - j_1, i_2 - j_2, \cdots, i_d - j_d)$$
\end{itemize}

\begin{lemma}[Commutativity of Tensors]\label{lconv}
$\mathcal{F}\ast \mathcal{G} = \mathcal{G}\ast \mathcal{F}$.
\end{lemma}

The convolution of two finite-dimensional tensors is defined by extending the sequences to finitely supported functions on the set of integers. In order to understand the computation of a convoltional layer, we need to define a \textit{reversed tensor} and a \textit{quasi-convolution}. A reversed tensor $rev(\mathcal{F})$ of a tensor $\mathcal{X}$ is a tensor with elements in the reversed order, namely
\begin{equation}\label{erevt}
rev(\mathcal{F})(i_1,i_2,...,i_d) := \mathcal{F}(-i_1,-i_2,...,-i_d), i_1,...,i_d\in\mathbb{Z}.
\end{equation}

Moreover, for two tensors $\mathcal{F}\in\mathcal{R}^{n_1\times\cdots\times n_d}$ and $\mathcal{G}\in\mathcal{R}^{m_1\times\cdots\times m_d}$ with $n_1\ge m_1,...,n_d\ge m_d$, we define a \textit{quasi-convolution} $\mathcal{F}\odot\mathcal{G}$ as a truncated convolution of $\mathcal{F}$ and $rev(\mathcal{G})$:
\begin{equation}
\label{eqconv}
\mathcal{F}\odot\mathcal{G} := \big(\mathcal{G}\ast rev(\mathcal{F})\big)(0:(n_1-m_1), 0:(n_2-m_2),...,0:(n_d-m_d))\in\mathbb{R}^{(n_1-m_1+1)\times\cdots\times(n_d-m_d+1)}.
\end{equation}
\begin{lemma}
\label{lqconv}
Let $\mathcal{F}\in\mathcal{R}^{n_1\times\cdots\times n_d}$ and $\mathcal{G}\in\mathcal{R}^{m_1\times\cdots\times m_d}$ with $n_1\ge m_1,...,n_d\ge m_d$. Then their quasi-convoltion $\mathcal{H}:=\mathcal{F}\odot\mathcal{G}$ can be computed as
\begin{equation}
\mathcal{H}(i_1,...,i_d) =
\sum_{0\le j_1\le m_1-1}\cdots\sum_{0\le j_d\le m_d-1}\big(\mathcal{F}(i_1+j_1,...,i_d+j_d)\mathcal{G}(j_1,...,j_d)\big).
\end{equation}
\end{lemma}
In CNN, the input of a convolutional layer is a 3-mode tensor $\mathcal{X}\in\mathbb{R}^{n_1\times n_2\times n_3}$, usually visualized as a collection of $n_3$ matrices of size $n_1\times n_2$. Each such matrix is refered to as an \textit{input channel}. The kernels of a convolutional layer forms a 4-mode tensor $\mathcal{K}\in\mathbb{R}^{d\times d\times n_3\times n_4}$. The function is defined as a quasi-convolution
\begin{equation}
\label{econv}
\aligned
f^{conv}(\mathcal{X}, \mathcal{K}) =& \mathcal{Y}\in\mathbb{R}^{(n_1-d+1)\times (n_2-d+1)\times n_4},\\
\mathcal{Y}(:,:,l) :=& \big(\mathcal{X}\odot\mathcal{K}(:,:,:,l)\big)(:,:,0).
\endaligned
\end{equation}
\begin{lemma}[Express $f^{conv}$ through 2D-convolutions]
Let $\mathcal{X}\in\mathbb{R}^{n_1\times n_2\times n_3}$, $\mathcal{K}\in\mathbb{R}^{d\times d\times n_3\times n_4}$, and $f^{conv}(\mathcal{X}, \mathcal{K}) = \mathcal{Y}\in\mathbb{R}^{(n_1-d+1)\times (n_2-d+1)\times n_4}$. Then
\begin{equation}
\label{e2dconv}
\mathcal{Y}(:,:,l) = \sum_{k=0}^{n_3-1}\big(\mathcal{X}(:,:,k)\odot\mathcal{K}(:,:,k,l)\big).
\end{equation}
A slice $\mathcal{Y}(:,:,l)$ is sometimes called a \textit{feature map}.
\end{lemma}

Two-dimensional quasi-convolution can be visualized as in Figure \ref{fconv}. Here the kernel slides though the entire input matrix.
\begin{figure}[ht]
\label{fconv}
\centering
\includegraphics[scale=0.4]{Convolution1}
\caption{Convolution in CNN}
\end{figure}

\section{Convolution via Fast Fourier Transform}

{\bf Discrete Fourier Transform (DFT)}. For a positive integer $n$, the \textit{Fourier transform matrix of degree $n$} is define as the following $n\times n$ matrix
\begin{equation}
\Omega_n:=\big(\omega_n^{-ij}\big)_{i,j}=\frac{1}{\sqrt{n}}\left[
\begin{array}{cccc}
\omega_n^0 & \omega_n^0 & \cdots & \omega_n^0\\
\omega_n^0 & \omega_n^{-1} & \cdots & \omega_n^{-(n-1)}\\
\vdots	& \vdots & & \vdots\\
\omega_n^0 & \omega_n^{-(n-1)} & \cdots & \omega_n^{-(n-1)^2}
\end{array}
\right],
\end{equation}
where $\omega_n = exp(\frac{2\pi i}{n})$. For an $n$-dimensional vector $\bm v = (v_0,...,v_{n-1})$, its \textit{discrete Fourier transform} is defined as
\begin{equation}
fft(\bm v):=\Omega_n\bm v.
\end{equation}
Similarly, for an $m\times n$ matrix $X$, its \textit{2-dimensional discrete Fourier transform} is defined as
\begin{equation}
2Dfft(X):= \Omega_m X \Omega_n.
\end{equation}
We have the following lemma for the inverse of Fourier transform.
\begin{lemma}
The inverse of 1D DFT is:
\begin{equation}
ifft(\bm v) = \Omega^{-1}\bm v = \Omega^H\bm v.
\end{equation}
The inverse of 2D DFT is:
\begin{equation}
2Difft(X) = \Omega_m^{-1} X \Omega_n^{-1} = \Omega_m^H X \Omega_n^H.\
\end{equation}
Here $X^H$ denotes the hermitian transpose.
\end{lemma}

{\bf DFT and convolution}. Discrete convolution can be achieved by DFT, element-wise multiplication, and inverse DFT.
\begin{lemma}
Let $F, G\in\mathbb{R}^{n_1\times n_2}$, then
\begin{equation}
(F\ast G) = 2Difft(2Dfft(F)\circ 2Dfft(G)).
\end{equation}
Here $\circ$ denotes matrix element-wise multiplication.
\end{lemma}

{\bf Fast Fourier transform (FFT)}.
For $n=2^h$, the matrix-vector multiplication $\Omega_n\bm v$ has complexity $O(n\log n)$ if one applies the following lemma recursively:
\begin{lemma}
Let $h\ge 1$ be an integer, $n=2^h$, $\omega_n=exp(\frac{2\pi i}{n})$, and $D = diag(1, \omega,...,\omega^{2^{h-1}-1})$. Then
\begin{equation}
\Omega_n = \left[
\begin{array}{cc}
I_{n/2} & \\
 & D
\end{array}
\right]\cdot
\left[
\begin{array}{cc}
I_{n/2} & I_{n/2}\\
I_{n/2} & -I_{n/2}
\end{array}
\right]\cdot
\left[
\begin{array}{cc}
\Omega_{n/2} &\\
 & \Omega_{n/2}
\end{array}
\right].
\end{equation}
\end{lemma}
By applying this "divide-and-conquer" method, the complexity of 1D DFT and 2D DFT can be reduced to $O(n\log n)$ and $O(mn\log m\log n)$, respectively. Later in this chapter we will show that FFT can be used to accelerate the training of CNN.

\section{Pooling Layer}

{\bf Pooling layer}. Pooling layer subsamples statistics to obtain summary statistics that are invariant to shifts and distortions in some degree. More precisely we have
\begin{equation}
f^{pool}(\mathcal{X}(:,:,l))(i,j) = f(\mathcal{X}(2i-1:2i,2j-1:2j,l)),
\end{equation}
where commonly used functions $f$ are
\begin{enumerate}
\item
Mean pooling: $f^{mean}(\mathcal{Z})=\big(\frac{\sum_k z_k}{m}\big)$
\item
Max pooling: $f^{max}(\mathcal{Z})=\big(\max_k\{z_k\}\big)$
\item
$l^p$ pooling: $f^{lp}(\mathcal{Z})=\big(\|(z_1,...,z_m)\|_p\big)$
\end{enumerate}
The partial derivative of the pooling function is usually called the \textit{upsample} function $f^{upsample}$. For example. for the max pooling function, the corresponding upscale function is $\mathbbm{1}_{z = \max_k\{z_k\}}$ (distributing the gradient to the entry with maximum value).

Figure \ref{ffea} illustrates features extracted the first three convolutional Layers  on a facial recognition task.

\begin{figure}[ht]
\label{ffea}
\centering
\includegraphics[scale=1]{Features}
\caption{Features}
\end{figure}

\section{Training CNN}
For simplicity, assume that there are no fully-connected layers and there are $D$ convolutional layers and $D$ pooling layers in the CNN model with $f^{conv}$ and $f^{pool}$ defined as in the previous sections. Choose an activation function $\sigma$. Let $S=\{(\mathcal{X}^1, \mathcal{Y}^1),...,(\mathcal{X}^N, \mathcal{Y}^N)\}$ be the training data set. For $(\mathcal{X}, \mathcal{Y})\in S$, the feedforward process $f(\mathcal{X};\mathcal{K}_1,...,\mathcal{K}_D)$ can be described progressively as
\begin{equation}
\aligned
\mathcal{X}_1 =& \mathcal{X}\\
\mathcal{Y}_{i}(\mathcal{X};\mathcal{K}_1,...,\mathcal{K}_i) :=& f^{conv}(\mathcal{X}_i(\mathcal{X};\mathcal{K}_1,...,\mathcal{K}_{i-1}), \mathcal{K}_i)\\
\mathcal{Z}_i(\mathcal{X};\mathcal{K}_1,...,\mathcal{K}_i) :=& \bm\sigma(\mathcal{Y}_{i}(\mathcal{X};\mathcal{K}_1,...,\mathcal{K}_i))\\
\mathcal{X}_{i+1}(\mathcal{X};\mathcal{K}_1,...,\mathcal{K}_i) :=& f^{pool}(\mathcal{Z}_i(\mathcal{X};\mathcal{K}_1,...,\mathcal{K}_i)), i=1,...,D\\
f(\mathcal{X};\mathcal{K}_1,...,\mathcal{K}_D):=&\mathcal{X}_{D+1}(\mathcal{X};\mathcal{K}_1,...,\mathcal{K}_D).
\endaligned
\end{equation}

Then choose a loss function $L(\mathcal{X},\mathcal{Y};\mathcal{K}_1,...,\mathcal{K}_D)$. The goal of training is to minimize the total loss
\begin{equation}
L_{total}(\mathcal{K}_1,...,\mathcal{K}_D) = \sum_{i=1}^N L(\mathcal{X}^i,\mathcal{Y}^i;\mathcal{K}_1,...,\mathcal{K}_D)
\end{equation}
In order to reduce the number of computations, the \textit{stochastic gradient descent} method samples a subset $S'$ of $S$ (called \textit{mini-batch}) with $|S'|\ll |S|$ and use the following loss function to approximate $L_{total}$:

\begin{equation}
L_{SGD}(\mathcal{K}_1,...,\mathcal{K}_D) = \sum_{(\mathcal{X},\mathcal{Y})\in S'} L(\mathcal{X},\mathcal{Y};\mathcal{K}_1,...,\mathcal{K}_D)
\end{equation}
The gradient of $L_{SGD}$ is the sum of all the corresponding gradients of $L(\mathcal{X},\mathcal{Y})$.

\begin{figure}[ht]\label{fcnnbp}
\includegraphics[scale=0.5]{CNNBP}
\caption{Convolutional Layer}
\end{figure}

The backpropagation process can be described as (see Figure \ref{fcnnbp}):
\begin{equation}
\aligned
\delta_{D+1} :=& \frac{\partial L}{\partial \mathcal{X}_{D+1}}\\
\delta_i^{pool} :=&\frac{\partial L}{\partial \mathcal{Z}_i} = \delta_{i+1}\times_{1,2,3}\frac{\partial \mathcal{X}_{i+1}}{\partial \mathcal{Z}_i}=\delta_{i+1}\circ f^{upsample}(\mathcal{X}_{i+1})\\
\delta_i^{sig}:=&\frac{\partial L}{\partial \mathcal{Y}_i} = \delta_i^{pool}\times_{1,2,3}\frac{\partial \mathcal{Z}_i}{\partial \mathcal{Y}_i}=\delta_i^{pool}\circ\bm\sigma'(\mathcal{Y}_i)\\
\delta_i :=&\frac{\partial L}{\partial \mathcal{X}_i}= \delta_i^{sig}\times_{1,2,3}\frac{\partial \mathcal{Y}_i}{\partial \mathcal{X}_i}\\
\delta_i(:,:,l) =& \sum_k \delta_i^{sig}\ast\mathcal{K}_i(:,:,k,l)\\
\frac{\partial L}{\partial\mathcal{K}_i} =& \frac{\partial L}{\partial \mathcal{Y}_i}\times_{1,2,3}\frac{\partial \mathcal{Y}_i}{\partial \mathcal{K}_i}\\
\frac{\partial L}{\partial\mathcal{K}_i}(:,:,k,l) =& \mathcal{X}_i(:,:,k)\odot \delta_i^{sig}(:,:,l)
, i=D,D-1,...,1
\endaligned
\end{equation}

%To backpropagate through one convolutional layer and one pooling layer, one need to compute the derivative of three functions:
%\begin{equation}
%\aligned
%\bm\delta^{conv}_{ikl} =& \frac{\partial L}{\partial x_{ikl}} =
%\sum_{j,k,l}\frac{\partial L}{\partial y_{jkl}}\frac{\partial y_{jkl}}{\partial x_{ikl}} = \sum_{k,l=0}^{r-1}\bm\delta^{sig}_{ikl}\\
%\bm\delta^{sig}_{jkl} =& \frac{\partial L}{\partial y_{jkl}} = \sigma'(y_{jkl})\\
%\bm\delta^{pool}_{jkl} =& \frac{\partial L}{\partial z_{jkl}} = \frac{\partial L}{\partial x^{next}_{jkl}}\frac{\partial x^{next}_{jkl}}{\partial z_{jkl}} = \bm\delta^{next, conv}_{jkl}\\
%\frac{\partial L}{\partial f_{ijkl}} =& \sum_{j,k,l}\frac{\partial L}{\partial y_{jkl}}\frac{\partial y_{jkl}}{\partial f_{ijkl}}=\sum_{s,t=0}^{d-1}\bm\delta^{sig}_{jst}x_{i, s+k, t+l}
%\endaligned
%\end{equation}
Finally, the kernels are updated by the gradient as
\begin{equation}
\mathcal{K}_i \leftarrow \mathcal{K}_i - \eta\frac{\partial L}{\partial\mathcal{K}_i}.
\end{equation}
Note that the complexity of the training process is dominated by the convolution operation in the forward and backward passes.

\section{FFT Based Backpropagation}

Recall that convolution is equivalent to element-wise product in the Fourier form. Typically, this method is used when the size of the convolution kernel is close to that of the input image. Note that a convolution of an image of size $n\times n$ with a kernel of size $d\times d$ using the direct method requires $(n - d + 1)^2d^2$
 operations. The complexity of the FFT-based method requires $(6Cn^2\log n+4n^2)$ operations. Here $C$ represents the constant overhead of FFT. Thus it is not preferable to use FFT for a single convolution in CNN training.

However, notice that each channel $\mathcal{X}(:,:,l)$ and each 2D kernel $\mathcal{K}(:,:,k,l)$ appears in $2n$ different convolutions in a forward pass and a backward pass, and one single FFT is sufficient for each matrix. Also notice that from equation (10) only the summation is needed, rather than the convolution results for each summand, suggesting that one can take the sum first and then perform the inverse FFT. Under these two considerations, the total complexity of the training process with FFT is lower than the direct convolution computation.

\begin{figure}[H]
\centering
\includegraphics[scale=0.5]{FFTconv}
\caption{Illustration of the FFT based convolution}
\end{figure}

% {\bf Other Implementation Details(future topics)}
% \begin{itemize}
% \item Random sampling
% \item Normalizing the inputs
% \item Choice of activation function
% \item Choosing target values
% \item Initialization
% \item Adaptive learning rates
%\end{itemize}


\subsection{Enumerative combinatorics}\label{subsec:enumerative_combinatorics}

This subsection lists several results of various importance that don't really belong to any more consistent theory.

\begin{theorem}[Dirichlet's pigeonhole principle]\label{def:pigeonhole_principle}
  If we are given more pigeons than pigeonholes, then at least one pigeonhole must contain multiple pigeons in it.

  More formally, if \( \card(A) > \card(B) \), then there exists no injective function from \( A \) to \( B \).
\end{theorem}
\begin{proof}
  This is a corollary of \fullref{thm:set_domination_relation_trichotomy}.
\end{proof}

\begin{definition}\label{def:binomial_coefficient}
  The \term{binomial coefficient} of the \hyperref[def:integer_signum]{nonnegative integers} \( n \) and \( k \) is
  \begin{equation*}
    \binom n k \coloneqq \frac {n!} {k!(n-k)!}
  \end{equation*}

  They are motivated by \fullref{thm:binomial_theorem}.
\end{definition}

\begin{theorem}[Pascal's identity]\label{thm:pascals_identity}
  \hyperref[def:binomial_coefficient]{Binomial coefficients} have the following property:
  \begin{equation}\label{eq:thm:pascals_identity}
    \binom n k = \binom {n - 1} k + \binom {n - 1} {k - 1}.
  \end{equation}
\end{theorem}
\begin{proof}
  \begin{balign*}
    \binom {n - 1} k + \binom {n - 1} {k - 1}
    &=
    \frac {(n - 1)!} {k! (n - 1 - k)!} + \frac {(n - 1)!} {(k - 1)! (n - k)!}
    = \\ &=
    \frac {(n - 1)!} {(k - 1)! (n - 1 - k)!} \bracks*{ \frac 1 k + \frac 1 {n - k} }
    = \\ &=
    \frac {(n - 1)!} {(k - 1)! (n - 1 - k)!} \frac n {k(n - k)}
    = \\ &=
    \frac {n!} {k! (n - k)!}
    = \\ &=
    \binom n k.
  \end{balign*}
\end{proof}

\begin{theorem}[Newton's binomial theorem]\label{thm:binomial_theorem}
  If, in some \hyperref[def:semiring]{semiring}, the members \( x \) and \( y \) commute (i.e. \( xy = yx \)), then
  \begin{equation}\label{eq:thm:binomial_theorem}
    (x + y)^n = \sum_{k=0}^n \binom n k x^k y^{n-k}.
  \end{equation}
\end{theorem}
\begin{proof}
  We use induction on \( n \). For \( n = 0 \), the theorem trivially holds. Assume that the theorem holds for \( 1, \ldots, n \). Then
  \begin{balign*}
    (x + y)^{n+1}
     & =
    x (x + y)^n + y (x + y)^n
    = \\ &=
    \sum_{k=0}^n \binom n k x^{k+1} y^{n-k} + y \sum_{k=0}^n \binom n k x^k y^{n-k}
    = \\ &=
    x^{n+1} + y \sum_{k=0}^{n-1} \binom n k x^{k+1} y^{n-(k+1)} + y \sum_{k=0}^n \binom n k x^k y^{n-k}
    = \\ &=
    x^{n+1} + y \parens*{ \sum_{k=1}^n \binom n {k-1} x^k y^{n-k} + y^n \sum_{k=1}^n \binom n k x^k y^{n-k} } + y^{n+1}
    = \\ &\reloset {\eqref{eq:thm:pascals_identity}} =
    x^{n+1} + y \sum_{k=1}^n \binom {n+1} k x^k y^{n-k} + y^{n+1}
    = \\ &=
    \sum_{k=0}^n \binom {n+1} k x^k y^{(n+1)-k}.
  \end{balign*}
\end{proof}

\begin{proposition}\label{thm:xn_minus_yn_factorization}
  For every \hyperref[def:ring]{ring} element \( x \) and every nonnegative integer \( n \), we have
  \begin{equation}\label{eq:thm:xn_minus_yn_factorization}
    x^{n + 1} - y^{n + 1} = (x - y)(x^n + x^{n-1} y + \cdots + y^n) = (x - y) \sum_{k=0}^n x^k y^{n-k}.
  \end{equation}
\end{proposition}
\begin{proof}
  \begin{align*}
    (x - y) \sum_{k=0}^n x^k y^{n-k}
    &=
    \sum_{k=0}^n x^{k+1} y^{n-k} - \sum_{k=0}^n x^k y^{n-k+1}
    = \\ &=
    \sum_{k=1}^{n+1} x^k y^{(n+1)-k} - \sum_{k=0}^n x^k y^{(n+1)-k}
    =
    x^{n+1} - y^{n+1}.
  \end{align*}
\end{proof}

\begin{definition}\label{def:factorial}
  The \term{factorial} of a \hyperref[def:integer_signum]{nonnegative integer} \( n \) is defined recursively as
  \begin{equation*}
    n! \coloneqq \begin{cases}
      1,          &n = 0 \\
      (n - 1)! n, &n > 0.
    \end{cases}
  \end{equation*}
\end{definition}

\begin{proposition}\label{thm:gamma_function_interpolates_factorial}
  For every \hyperref[def:integer_signum]{nonnegative integer} \( n \) we have
  \begin{equation*}
    \Gamma(n + 1) \coloneqq n!,
  \end{equation*}
  where \( \Gamma \) is the Gamma function defined in \fullref{def:gamma_function}.
\end{proposition}
\begin{proof}
  We use induction on \( n \).
  \begin{itemize}
    \item If \( n = 0 \), then
    \begin{equation*}
      \Gamma(1)
      =
      \int_0^\infty x^0 e^{-x} \dl x
      =
      -e^{-x}\restr_{x=0}^\infty
      =
      -\underbrace{\lim_{x \to \infty} e^{-x}}_{0} + 1
      =
      1
      =
      0!
    \end{equation*}

    \item If \( n > 0 \) and \( \Gamma(n) = (n - 1)! \), then
    \begin{balign*}
      \Gamma(n + 1)
      &=
      \int_0^\infty x^n \cdot e^{-x} \dl x
      = \\ &=
      \underbrace{(- x^n e^{-x})\restr_{x=0}^\infty}_{-(0 - 0)} + n \int_0^\infty e^{-x} x^{n-1} \dl x
      = \\ &=
      n \Gamma(n)
      = \\ &=
      n (n - 1)!
      = \\ &=
      n!
    \end{balign*}
  \end{itemize}
\end{proof}

\begin{theorem}[Stirling's factorial approximation]\label{thm:stirlings_factorial_approximation}
  For every \hyperref[def:integer_signum]{nonnegative integer} \( n \) there exists some constant \( \theta \in (0, 1) \) such that
  \begin{equation*}
    n! = \sqrt{2 \pi n} \cdot \parens*{ \frac n e }^n \cdot e^{\frac 1 {12n + \theta}}.
  \end{equation*}
\end{theorem}
\begin{proof}
  Follows from \fullref{thm:gamma_function_interpolates_factorial} and \fullref{thm:stirlings_gamma_approximation}.
\end{proof}


\subsection{Conditional Average Treatment Effect (CATE)}

\(E[y_{i1}-y_{i0}|\mathbf x_i]\)


\section{Association Rules Discovery}
\begin{itemize}
	\item Goal: discover \textbf{correlation among attributes} or other relationships in large databases.
	\item Use-case: Market Basket Analysis, cross/up-selling
	\item \textbf{Unsupervised learning}: no dependent variable defined, no labeled training data.
\end{itemize}

\subsection{Terminology}

\paragraph{Rule} if $A$ and $B$ then $C$ and $D$. denote as $R: A,B \Rightarrow C,D$. It only describes \textbf{correlation}, not causality.

\paragraph{Transaction Database} an instance/observation is a transaction. Each \textbf{attribute} in the database is converted to \textbf{binary flags 0/1}.
\paragraph{Item} single element/attribute. eg: Milk/Bread
\paragraph{Itemset} a set of items. eg: {Milk, Bread, Butter}
\paragraph{Frequent Itemset} the itemset $I$ that meets the \textbf{minimum support}. $$supp(I) \geq \min supp$$
\paragraph{Support}
\begin{itemize}
	\item support of \textbf{an item set}: \textbf{relative frequency} of the transactions that contain the item-set in \textbf{all transactions}
	\item support of \textbf{a rule}: the support of all item sets it contains.
	$$supp(A,B \Rightarrow C,D) = supp(\{A,B,C,D\})$$
	The \textbf{order, the arrow} of the rule \textbf{doesn't matter in computing support}.
	$$supp(\text{Milk} \Rightarrow \text{Bread}) = supp(\{\text{Milk, Bread}\}) = supp(\{\text{Bread} \Rightarrow \text{Milk}\})$$

	\item support \textbf{estimation}: lower bound + upper bound.
	\begin{itemize}
		\item lower bound: the support of a subset is always higher than its superset. \textbf{subset property}, every subset of a frequent set is frequent.
		$$supp(\{B,C\}) \geq supp(\{A,B,C,D\})$$
		\item upper bound: use Venn-Diagramm.
	\end{itemize}
\end{itemize}

\paragraph{Confidence of a Rule} the likeliness to apply to the dataset. $\rightarrow$ the probability that X and Y coexist given that X exists.
$$conf(R: X \Rightarrow Y) = \frac{supp(X \cup Y)}{supp(X)}$$

$$conf(\{\text{Milk,Bread}\} \Rightarrow \{\text{Butter}\}) = \frac{supp(\{\text{Milk, Bread, Butter}\})}{supp(\{\text{Milk, Bread}\})}$$

\paragraph{Strong Rule} association rules with \textbf{minimum support \& confidence}.

\paragraph{Lift of a Rule} indicates \textbf{by how much (ratio)} the \textbf{confidence of a rule} surpasses the \textbf{expected value}.
$$Lift(R: X \Rightarrow Y) = \frac{conf(R)}{expConf(R)} = \dfrac{\frac{supp(X \cup Y)}{supp(X)}}{supp(Y)} = \frac{supp(X \cup Y)}{supp(X)\cdot supp(Y)}$$

\subparagraph{Interpretation of Lifts}
\begin{itemize}
	\item lift < 1: X has \textbf{positive} effect on Y. Item-sets X and Y appears \textbf{more frequent than expected value}.
	\item lift = 1: X and Y are \textbf{independent}. X has \textbf{no effect} on Y.
	\item lift > 1: X has \textbf{negative} effect on Y. Item-sets X and Y appears \textbf{less frequent than expected value}.
\end{itemize}

\subsection{A priori Algorithm: Generation of Itemsets and Rules}

\begin{itemize}
	\item Idea: if X is a frequent k-item set, then all (k-1)-item subsets of X have to be frequent item sets as well.

	$\rightarrow$ iteratively compute frequent item sets, compute k-item sets by merging (k-1)-item sets.

	\item Process:
	\begin{enumerate}[label= \protect \circled{\arabic*} ]
		\item Generation of \textbf{Item sets}:
		\begin{itemize}
			\item start with item sets in \textbf{size 1}.
			\item only select those that \textbf{exceeds minimum support} $\rightarrow$ frequent.
			\item iteratively build item sets in larger sizes based on previous sizes.
		\end{itemize}
	\begin{figure}[H]
		\centering
		\includegraphics[width=0.5\textwidth]{itemset.png}
	\end{figure}
		\item Generation of \textbf{Rules} based on frequent item sets:
		\begin{itemize}
			\item start with rules with only \textbf{1 item on the right}.
			\item rule $X \Rightarrow Y$ is different from $Y \Rightarrow X$. Compute \textbf{both directions}.
			\item only select rules that \textbf{exceeds minimum confidence}.
			\item evaluate rules containing \textbf{multiple items on the right} by checking whether single item on the right side. \textbf{Only expand if single rules exceeds minimum confidence.}
			$$X \Rightarrow Y, Z \quad \text{bases on} \quad X \Rightarrow Y \quad \text{and} \quad X \Rightarrow Z$$
		\end{itemize}
	\begin{figure}[H]
		\centering
		\includegraphics[width=0.6\textwidth]{ruleset.png}
	\end{figure}
	\end{enumerate}
\end{itemize}

\section{Recommendation Systems}
\begin{itemize}
	\item Approaches:
	\begin{itemize}
		\item Association Rules: discover \textbf{correlations}
		\begin{itemize}
			\item product association
			\item user association
			\item combination of both
		\end{itemize}
		\item Collaborative Filtering: discover \textbf{similarity}
		\item Singular Value Decomposition
	\end{itemize}
\end{itemize}

\subsection{Collaborative Filtering}
\begin{itemize}
	\item Idea:
	\begin{itemize}
		\item maintain a database of \textbf{user's rating} on items.
		\item for a \textbf{given active user}, find other \textbf{similar users} whose \textbf{rating strongly correlates} with the active user.

		$\rightarrow$ recommend items highly rated by similar users, which is \textbf{not rated} by active user.
	\end{itemize}

\end{itemize}

\subsubsection{Process}
\begin{enumerate}[label= \protect \circled{\arabic*} ]
	\item define \textbf{active user} $a$ and \textbf{other users} $u$.
	\item calculate \textbf{weighted correlation} $w_{a,u}$ based on \textbf{number of co-rated items m} .
	\begin{itemize}
		\item calculate average of the \textbf{co-rated items} $\bar{r}_a, \bar{r}_u$
		\item calculate the variance $\sigma_{r_{a}}^2, \sigma_{r_{u}}^2$ and standard deviation.
		\item calculate the covariance. \textbf{Don't forget the minus/plus symbol!!!}
		\item calculate the weighted correlation.
	\end{itemize}

	$$w_{a,u} = s_{a,u} \cdot c_{a,u}$$
	$$c_{a,u} = \frac{Cov(r_{a}, r_{u})}{\sigma_{r_{a}} \cdot \sigma_{r_{u}}}$$
	$$Cov(r_{a}, r_{u}) = \frac{1}{m-1}\cdot \Sigma (r_a - \bar{r}_a) (r_u - \bar{r}_u)$$
	\item \textbf{rating prediction} for item i for active user.
	\begin{itemize}
		\item calculate \textbf{average rating for all rated items} $\bar{r}_a$ of active user a.
		\item calculate \textbf{average rating for all rated items} $\bar{r}_u$ of each other user u.
		\item $r_{u,i}$: other user u's rating on the i-th item.
	\end{itemize}
	$$p_{a,i} = \bar{r}_a + \Sigma_{u = 1}^k \dfrac{w_{a,u} \cdot (r_{u,i} - \bar{r}_u)}{\Sigma_{u=1}^k |w_{a,u}|}$$
\end{enumerate}

\subsubsection{Limitation in Collaborative Filtering}
\begin{itemize}
	\item \textbf{Cold Start}: \textbf{enough users and ratings} are needed to generate recommendations.
	\item \textbf{Sparsity}: the user/rating matrix can be sparse even there are many users

	$\rightarrow$ hard to find \textbf{co-rated} items.
	\item \textbf{First Rater}: with a \textbf{new product}, there must first be consumers who test and evaluate it.
	\item \textbf{Popularity Bias}: cannot recommend items to users with \textbf{unique taste}. Tend to recommend popular items.
\end{itemize}

$\rightarrow$ Alternative: \textbf{Content-Based Filtering}
\begin{itemize}
	\item idea: based on information of the content of items.
	\item solve:
	\begin{itemize}
		\item combat popularity bias
		\item combat first rater.
		\item no need of user ratings $\rightarrow$ cold start + sparsity combated.
	\end{itemize}
\end{itemize}

\subsection{Singular Value Decomposition}
\begin{itemize}
	\item Idea: produce a low-dimensional representation of the customer-product space.
	\item Model:
	$$A = U \cdot S \cdot V^T$$
	\begin{itemize}
		\item A: the rating matrix, or the rating we want to predict.
		\item U: maps \textbf{users to concepts}
		\item S: strength of concepts/categories
		\item $V^T$: maps \textbf{venues/products to concepts}
	\end{itemize}
	\item \textbf{Rating Prediction}: rating for item i from user.
	\begin{itemize}
		\item calculate/consider the average rating of user.
	\end{itemize}
	$$r_{u,i} = \bar{r}_u + U(user) \cdot S \cdot V^T(item)$$

	\item Interpretation of values:
	\begin{itemize}
		\item User matrix(U):
		\begin{itemize}
			\item positive: higher interest
			\item negative: lower interest
			\item 0: no interest
		\end{itemize}
		\item Product matrix($V^T$):
		\begin{itemize}
			\item positive: \textbf{positively represented} in the i-th latent factor. Users having preference in i-th latent factor will \textbf{prefer items with positive value over items with negative values}.
			\item negative: \textbf{negatively represented} in the i-th latent factor. Users having preference in i-th latent factor will \textbf{like item less}.
		\end{itemize}
	\end{itemize}
\end{itemize}


\section*{Exercises}

\begin{ex}
  Consider $\R^3$ with the usual dot product. Let
  \begin{equation*}
    \vect{u}_1 = \begin{mymatrix}{r} -1 \\ 1 \\ 1 \end{mymatrix},
    \quad
    \vect{u}_2 = \begin{mymatrix}{r} -1 \\ -2 \\ 1 \end{mymatrix},
    \quad\mbox{and}\quad
    \vect{v} = \begin{mymatrix}{r} -1 \\ 5 \\ 3 \end{mymatrix}.
  \end{equation*}
  Note that $\vect{u}_1$ and $\vect{u}_2$ are orthogonal.
  Find the best approximation of $\vect{v}$ in $\sspan\set{\vect{u}_1,
    \vect{u}_2}$.
  \begin{sol}
    The best approximation is $\vect{v}' = 3\vect{u}_1 - \vect{u}_2 =
    \begin{mymatrix}{r} -2 \\ 5 \\ 2 \end{mymatrix}$.
  \end{sol}
\end{ex}

\begin{ex}
  Consider $\R^4$ with the usual dot product. Let
  \begin{equation*}
    \vect{u}_1 = \begin{mymatrix}{r} 0 \\ 0 \\ 1 \\ 3 \end{mymatrix},
    \quad
    \vect{u}_2 = \begin{mymatrix}{r} 1 \\ 1 \\ 0 \\ 0 \end{mymatrix},
    \quad
    \vect{u}_3 = \begin{mymatrix}{r} 1 \\ -1 \\ 3 \\ -1 \end{mymatrix},
    \quad\mbox{and}\quad
    \vect{v} = \begin{mymatrix}{r} 6 \\  -2 \\ -5 \\  5 \end{mymatrix}.
  \end{equation*}
  Note that $\vect{u}_1$, $\vect{u}_2$, and $\vect{u}_3$ are
  orthogonal.  Find the best approximation of $\vect{v}$ in
  $\sspan\set{\vect{u}_1,\vect{u}_2, \vect{u}_3}$.
  \begin{sol}
    The best approximation is
    $\vect{v}' = \vect{u}_1 + 2\vect{u}_2 - \vect{u}_3
    = \begin{mymatrix}{r} 1 \\ 3 \\ -2 \\ 4 \end{mymatrix}$.
  \end{sol}
\end{ex}

\begin{ex}
  In the inner product space $V=C[-1,1]$, consider the function
  $f\in V$ given by
  \begin{equation*}
    f(x)
    ~=~ \begin{cases}
      1 & \text{if $x<0$,} \\
      1-x & \text{if $x\geq 0$.}
    \end{cases}
  \end{equation*}
  Find the closest approximation to $f$ by a polynomial of degree at
  most 0, 1, 2, 3, and 4. Graph both $f$ and the approximating
  polynomials.
  \begin{sol}
    Let $p_0,p_1,\ldots$ be the Legendre polynomials from Section~\ref{sec:gram-schmidt}.
    The approximating polynomials are:
    \begin{eqnarray*}
      f_0(x) &=& \frac{3}{4}p_0, \\
      f_1(x) &=& \frac{3}{4}p_0 - \frac{1}{2}p_1, \\
      f_2(x) &=& \frac{3}{4}p_0 - \frac{1}{2}p_1 - \frac{15}{32}p_2, \\
      f_3(x) &=& \frac{3}{4}p_0 - \frac{1}{2}p_1 - \frac{15}{32}p_2 + 0p_3, \\
      f_4(x) &=& \frac{3}{4}p_0 - \frac{1}{2}p_1 - \frac{15}{32}p_2 + 0p_3 + \frac{105}{256}p_4.
    \end{eqnarray*}
    \begin{center}
      \begin{tikzpicture}[domain=-1:1, scale=3, samples=25]
        \def\fA#1{1}
        \def\fB#1{(#1)}
        \def\fC#1{(abs((#1)^2)-1/3)}
        \def\fD#1{((#1)^3-3/5*(#1))}
        \def\fE#1{(abs((#1)^4)-6/7*abs((#1)^2)+3/35)}
        \def\fF#1{((#1)^5 - 10/9*(#1)^3+5/21*(#1))}
        \def\fG#1{(abs((#1)^6)-15/11*abs((#1)^4)+5/11*abs((#1)^2)-5/231)}
        \def\fH#1{((429*(#1)^7 - 693*(#1)^5 + 315*(#1)^3 - 35*(#1))/429)}
        \def\fI#1{((6435*abs((#1)^8)-12012*abs((#1)^6)+6930*abs((#1)^4)-1260*abs((#1)^2)+35)/6435)}
        \draw[thick,color=fourier0] (-1,1) -- (0,1) -- (1,0);
        \path[color=fourier0t] (1,0) node[right=2ex] {$f$};
        \draw[thick,color=fourier1]         plot (\x,{0.75*\fA{\x}}) (1,0.75) node[right=1ex,color=fourier1t] {$f_0$};
        \draw[thick,color=fourier4]          plot (\x,{0.75*\fA{\x} - 0.5*\fB{\x}}) (1,0.25) node[right=1ex,color=fourier4t] {$f_1$};
        \draw[thick,color=fourier5]      plot (\x,{0.75*\fA{\x} - 0.5*\fB{\x} - 15/32*\fC{\x}}) (1,-0.125) node[right=1ex,color=fourier5t] {$f_2=f_3$};
        \draw[thick,color=fourier6,samples=50] plot (\x,{0.75*\fA{\x} - 0.5*\fB{\x} - 15/32*\fC{\x} + 105/256*\fE{\x}}) (1,0.125) node[right=1ex,fourier6t] {$f_4$};
        \draw[->] (-1.5,0) -- (1.5,0) node[right] {$x$};
        \draw[->] (0,-0.3) -- (0,1.1);
        \draw (1,0) -- (1,-0.05) node[below] {$1$};
        \draw (-1,0) -- (-1,-0.05) node[below] {$-1$};
      \end{tikzpicture}
    \end{center}
  \end{sol}
\end{ex}

\begin{ex}
  In the inner product space $C[-\pi,\pi]$, consider the orthogonal
  set of functions from Example~\ref{exa:orthogonal-set-sin-cos}:
  $\set{1, \sin x, \cos x, \sin 2x, \cos 2x, \sin 3x, \cos 3x, \ldots}$.  Let
  $f(x) = x^2$, where $x\in[-\pi,\pi]$. Find the Fourier series of
  $f$.
  \begin{sol}
    $\frac{\pi^2}{3} - \frac{4}{1}\cos x + \frac{4}{4}\cos 2x - \frac{4}{9} \cos 3x + \frac{4}{16} \cos 4x - \frac{4}{25} \cos 5x \pm \ldots$.
  \end{sol}
\end{ex}