\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{hyperref}
\usepackage{tikz}
\usepackage{pgfplots}
\usepackage[margin=1.5in]{geometry}
\usepackage[T1]{fontenc}

\newcommand{\norm}[1]{\lvert #1 \rvert}

% see the following for various vector notations
% http://www.tapdancinggoats.com/latex-vector-notation.htm

\title{Quintic Splines for FTC}
\author{Ryan Brott}
\date{}

\begin{document}

\maketitle

\section{Introduction}
In this paper, we explore the use of quintic splines for more sophisticated robot pathing in FTC. Traditionally, FTC robot autonomous motion consists of a series of linear movements (including holonomic drive strafing) and point turns. Although these simple path primitives generally suffice, they are not time-optimal under reasonable kinematic constraints, especially for nonholonomic (e.g., tank/differential) drivetrains. That is, the fastest route between two poses in general is not two point turns and a line.\footnote{The keen reader will realize that this is not necessarily the case for holonomic drives. All pose-to-pose movements can be executed with a combination of strafing and rotating. Nevertheless, splines can still be of use. For one, traveling along the tangent can reduce dead reckoning odometrical errors accrued from translating on the lateral axis (especially for mecanum drives). Additionally, splines can help navigate around obstacles in situations where piecewise linear paths would normally be required.} To address this, we propose the use of quintic splines to achieve quick, smooth motion on the field.

In the first part of the paper, we will describe the problem in depth and give motivation for splines. In the second half, we explore some of the mathematics behind quintic splines (and parametric curves more generally) including interpolation, reparametrization, and heading computation.

\section{Problem}
In FTC, it's standard to have a sequence a poses that you want the robot to follow in autonomous (although pre-planned motions can also be utilized in TeleOp). For example, in the Relic Recovery game, a common movement task may be moving from one pose on the balancing stone to another in front of the cryptobox. This is traditionally handled with a series of straight lines and turns that are executed with 1D position PID controllers and potentially motor-based velocity PID (e.g., \texttt{RUN\_USING\_ENCODER}).

For many game tasks and teams, this is a perfectly viable approach. However, in scenarios where speed is desirable (such as the effectively unlimited scoring potential in Relic Recovery autonomous), conventional methods tend to become less effective. Higher speeds usually lead to greater wheel accelerations and slippage, hindering odometry. Additionally, the feedback controllers may have more chaotic transient behavior or decreased steady-state performance.

In this case, motion profiling can be used to achieve higher speeds without sacrificing accuracy by observing the robot's physical constraints (e.g., maximum velocity, acceleration).\footnote{For a good introduction to motion profiling, see the canonical talk by mentors from FRC teams 254 and 971: \url{https://www.youtube.com/watch?v=8319J1BEHwM}}. However, this is only part of the solution; in fact, motion profiling amplifies the discontinuities between the elements of a conventional pose-to-pose path. Each transition between rotation and translations demands a full deceleration-acceleration cycle.

To eliminate this unnecessary pause, we seek to combine the rotation and translation into a single smooth curve. Although many curves suffice, splines are typically employed for this purpose. For simplicity, this paper only considers polynomial  splines\footnote{\url{http://www.ryanjuckett.com/programming/biarc-interpolation/} is a great introduction to biarc splines.}. Quintic splines in particular were selected for their balance between continuity and curvature although the methods described can be easily extended to polynomials of arbitrary odd degree.

\begin{figure}
    \centering

    \begin{tikzpicture}
        \begin{axis}[
            width=8cm,
            height=8cm,
            axis x line=center,
            axis y line=center,
            xtick={-72,-48,...,72},
            ytick={-72,-48,...,72},
            x dir=reverse,
            xlabel={$y$},
            ylabel={$x$},
            xlabel style={below left, at={(-0.02, 0.5)}},
            ylabel style={above right, at={(0.52, 0.98)}},
            xmin=-80,
            xmax=80,
            ymin=-80,
            ymax=80,
            axis equal=true]
            \addplot graphics[xmin=-72,ymin=-72,xmax=72,ymax=72] {../image/transparent_field.png};
            \addplot [mark=none, style=very thick, black!30!green, domain=0:1] ({-108.00000*x^5+288.00000*x^4-216.00000*x^3+0.00000*x^2+0.00000*x+48.00000},{44.73506*x^5-119.29351*x^4+89.47013*x^3+0.00000*x^2-50.91169*x+48.00000});
            \addplot[mark=none, style=very thick, black!30!green] coordinates {(-48,48) (-48,30) (-12,12)};
        \end{axis}
    \end{tikzpicture}

    \caption{Sample coordinate system for describing positions on the Relic Recovery field. This right-handed frame can be extended to three dimensions with the positive z-axis protruding from the origin. The path on the blue side is a typical autonomous route seen in FTC (don't forget there's a turn between the line segments --- three profiles in total). The path on the red side is a quintic spline version of roughly the same move. The spline path is 1.6 times faster than the conventional path with reasonable drive constraints.}
    \label{fig:coordinate_system}
\end{figure}

\section{Preliminaries}

\subsection{Coordinate System}
Before constructing paths, it is imperative to define a consistent coordinate system. With this coordinate frame, points on the field can be uniquely described. This allows for the specification of robot poses which consist of the position and a heading direction. This heading is typically encoded as angle from the $x$-axis. \autoref{fig:coordinate_system} shows a sample coordinate system for the Relic Recovery field. When determining a coordinate system, it is advantageous to choose axes around the symmetries of the environment. For the standard square FTC field, this comprises putting the origin at the center and placing the planar axes perpendicular to the field walls.

\subsection{Vectors and Parametric Curves}
To fully understand the following mathematics, it helps to grasp a little vector calculus (don't be scared --- only basic knowledge of single-variable calculus is required). Recall that a Euclidean vector $\mathbf{v}$ in $n$ dimensions ($\mathbf{v} \in \mathbb{R}^n$) uniquely encodes a single point in the space (commonly represented as an arrow from the origin to the point). This vector can be decomposed into $n$ components:
$$
\mathbf{v} = (x_1, x_2, \ldots, x_n)
$$
The length of a vector is given by its norm:
$$
\norm{\mathbf{v}} = \sqrt{x_1^2 + x_2^2 + \ldots + x_n^2}
$$

Aside from vector addition and scalar multiplication, two additional operations are commonly defined between Euclidean vectors. First, the dot product is defined as the scalar sum of the element-wise products of two vectors:
$$
\mathbf{v} \cdot \mathbf{w} = v_1w_1 + v_2w_2 + ... + v_nw_n
$$
Note that the norm can be alternatively expressed as $\norm{\mathbf{v}} = \sqrt{\mathbf{v} \cdot \mathbf{v}}$. Second, the cross product is defined for three-dimensional vectors as the pseudo-determinant of this quasi-matrix:
$$
\mathbf{v} \times \mathbf{w} =
\begin{vmatrix}
    \hat{\imath} & \hat{\jmath} & \hat{k}\\[4pt]
    v_1 & v_2 & v_3\\[4pt]
    w_1 & w_2 & w_3
\end{vmatrix}
$$

Two-dimensional curves can be represented as graphs of one-dimensional functions ($y = f(x)$). However, this puts unnecessary constraints on the set of possible curves (e.g., no two points on the curve can share the same x-value, no self-intersections, no vertical tangents), especially in higher dimensions. Instead, we will represent curves as a series of vectors that trace out the shape. This is analogous to a parametric curve $\mathbf{r}(t)$ which maps a single real parameter $t$ to the corresponding path vector ($\mathbf{r}:\mathbb R\to\mathbb R^n$). Intuitively, $\mathbf{r}(t)$ can be thought of as the trajectory of a fly located at $\mathbf{r}(t_0)$ in the instant $t=t_0$.

Parametric curves can be represented as a vector of single-variable functions:
$$
\mathbf{v}(t) = (x_1(t), x_2(t), \ldots, x_n(t))
$$
As one might expect, $\mathbf{r}(t)$ can be differentiated component-by-component:
$$
\mathbf{v}'(t) = (x_1'(t), x_2'(t), \ldots, x_n'(t))
$$
Derivatives of dot products and cross products can also be computed:
\begin{equation*}
\begin{split}
    \frac{d}{dt} \Big[\ \mathbf{r}_1(t) \cdot \mathbf{r}_2(t)\ \Big] &= \mathbf{r}_1(t) \cdot \mathbf{r}_2'(t) +\mathbf{r}_1'(t) \cdot \mathbf{r}_2(t) \\
    \frac{d}{dt} \Big[\ \mathbf{r}_1(t) \times \mathbf{r}_2(t)\ \Big] &= \mathbf{r}_1(t) \times \mathbf{r}_2'(t) +\mathbf{r}_1'(t) \times \mathbf{r}_2(t)
\end{split}
\end{equation*}

A parametric curve is said to be $C^n$ if its nth-order derivatives are defined and continuous everywhere on its domain ($n$ here is distinct from the $n$ in $\mathbb R^n$). For the rest of the paper, we will restrict our attention to two-dimensional parametric curves $\mathbf{r}(t)=(x(t), y(t))$.

\section{Interpolation}
Quintic splines consist of a series of segments assembled together into a single piecewise curve. Each of the segments is a parametric curve with a quintic polynomial for each component. The ith segment of a two-dimensional quintic spline of $n$ segments can be represented as the following:
$$
    \begin{cases}
        x^{(i)}(t) = a^{(i)}_xt^5 + b^{(i)}_xt^4 + c^{(i)}_xt^3 + d^{(i)}_xt^2 + e^{(i)}_xt + f^{(i)}_x\\
        y^{(i)}(t) = a^{(i)}_yt^5 + b^{(i)}_yt^4 + c^{(i)}_yt^3 + d^{(i)}_yt^2 + e^{(i)}_yt + f^{(i)}_y
    \end{cases}
    \quad \text{where} \quad 0 \leq t \leq 1
$$
Now the goal of interpolation is to ``fit'' these polynomials between a series of $n+1$ points (commonly referred to as knots) labeled $(x_i, y_i)$. To accomplish this, we can impose the conditions $x^{(i)}(0) = x_i$ and $x^{(i)}(1) = x_{i+1}$ (and correspondingly for $y^{(i)}(t)$). Additionally, to ensure greater continuity, we also match the first and second derivatives at each knot point\footnote{This is generally the best choice for splines in the context of path planning although other schemes are occasionally employed. For instance, one can instead force the third and fourth derivatives to equal zero.}:
\begin{equation*}
\begin{split}
    \frac{dx^{(i)}}{dt}\biggr\rvert_{t=0} &= x'_i\\
    \frac{dx^{(i)}}{dt}\biggr\rvert_{t=1} &= x'_{i+1}\\
    \frac{d^2x^{(i)}}{dt^2}\biggr\rvert_{t=0} &= x^{\prime\prime}_i\\
    \frac{d^2x^{(i)}}{dt^2}\biggr\rvert_{t=1} &= x^{\prime\prime}_{i+1}\\
\end{split}
\end{equation*}
$$(\text{and analogously for }y)$$
Collectively, these constraints guarantee that the overall spline will be (by construction) $C^2$. They also fully define the polynomial coefficients for each spline segment. To actually compute the coefficients, we simply need to solve the linear system with following matrix representation:
\[
\begin{bmatrix}
    0 & 0 & 0 & 0 & 0 & 1 \\[4pt]
    0 & 0 & 0 & 0 & 1 & 0 \\[4pt]
    0 & 0 & 0 & 2 & 0 & 0 \\[4pt]
    1 & 1 & 1 & 1 & 1 & 1 \\[4pt]
    5 & 4 & 3 & 2 & 1 & 0 \\[4pt]
    20 & 12 & 6 & 2 & 0 & 0
\end{bmatrix}
\begin{bmatrix}
    a_x \\[4pt]
    b_x \\[4pt]
    c_x \\[4pt]
    d_x \\[4pt]
    e_x \\[4pt]
    f_x
\end{bmatrix}
=
\begin{bmatrix}
    x_i \\[4pt]
    x'_i \\[4pt]
    x^{\prime\prime}_i \\[4pt]
    x_{i+1} \\[4pt]
    x'_{i+1} \\[4pt]
    x^{\prime\prime}_{i+1}
\end{bmatrix}
\]
This equation can then be solved to yield the coefficients with your favorite matrix library (or put in row echelon form to yield a set of back-substitutable equations).

\section{Reparametrization}

\subsection{Arc Length}
Now that we've interpolated the spline segments, we have to join together all of the spline segments into a single piecewise function of a unified variable. To accomplish this, we're going to reparametrize $\mathbf{r}(t)$ from $t \in [0, 1]$ to the arc length parameter $s$. Although it sounds complex, $s$ is just the true displacement along the path, and it traces out the curve with unit speed ($\norm{\mathbf{r}'(s)} = \sqrt{\big(\frac{dx}{ds}\big)^2 + \big(\frac{dy}{ds}\big)^2} = 1$). This procedure is completely generic, allowing you to combine arbitrary parametric curve with (basically) the same knot derivatives.

To shift from $t$ to $s$, we first have to define $t(s)$. This is difficult to represent analytically although the inverse function $s(t)$ is relatively simple:
$$
    s(t) = \int_0^t \! \norm{\mathbf{r}'(\tau)} \, d\tau = \int_0^t \! \sqrt{\bigg(\frac{dx}{d\tau}\bigg)^2 + \bigg(\frac{dy}{d\tau}\bigg)^2} \, d\tau
$$
In practice, $t(s)$ can be obtained by numerically evaluating the above integral and stopping when the integral sum reaches $s$.

By computing $t(s)$, we can determine $\mathbf{r}(s)$ by composition: $\mathbf{r}(s) = \mathbf{r}(t(s))$. Next, differentiating yields the new parametrized derivative:
\begin{equation*}
\begin{split}
    \mathbf{r}'(s) &= \mathbf{r}'(t)\ t'(s) \\
                   &= \frac{\mathbf{r}'(t)}{s'(t)} \\
                   &= \frac{\mathbf{r}'(t)}{\norm{\mathbf{r}'(t)}} \\
\end{split}
\end{equation*}
This makes sense intuitively as $\frac{\mathbf{r}'(t)}{\norm{\mathbf{r}'(t)}}$ is just a normalized version of $\mathbf{r}'(t)$ that satisfies the condition $\norm{\mathbf{r}'(s)} = 1$. We similarly obtain the new second and third derivatives by differentiating again\footnote{This derivation depends heavily on the BAC-CAB identity: $\mathbf{a} \times (\mathbf{b} \times \mathbf{c}) = \mathbf{b} (\mathbf{a} \cdot \mathbf{c}) - \mathbf{c} (\mathbf{a} \cdot \mathbf{b})$.}$^{,}$\footnote{$\kappa(s) = \norm{\mathbf{r}''(s)}$ is the curvature of $\mathbf{r}(s)$. Note that quintic splines also guarantee continuous curvature.}:
\begin{equation*}
\begin{split}
    \mathbf{r}''(s) &= \frac{\mathbf{r}''(t)}{\norm{\mathbf{r}'(t)}^2} - \frac{\mathbf{r}'(t)\ \mathbf{r}''(t) \cdot \mathbf{r}'(t)}{\norm{\mathbf{r}'(t)}^4} \\
                    &= \frac{\mathbf{r}''(t)\ \mathbf{r}'(t) \cdot \mathbf{r}'(t) - \mathbf{r}'(t)\ \mathbf{r}''(t) \cdot \mathbf{r}'(t)}{\norm{\mathbf{r}'(t)}^4} \\
                    &= \frac{\mathbf{r}'(t) \times (\mathbf{r}''(t) \times \mathbf{r}'(t))}{\norm{\mathbf{r}'(t)}^4} \\
    \mathbf{r}'''(s) &= \frac{\mathbf{r}'(t) \times (\mathbf{r}'''(t) \times \mathbf{r}'(t)) + \mathbf{r}''(t) \times (\mathbf{r}''(t) \times \mathbf{r}'(t))}{\norm{\mathbf{r}'(t)}^9}
\end{split}
\end{equation*}

\subsection{Trajectory}
Now, all of the spline segments and various other parametric curves are combined into a single curve $\mathbf{r}(s)$. However, this still needs to be combined with the motion profile to yield the robot's kinematic state over time.

Let $s(t)$ be the generated motion profile (note: this $t$ actually refers to time; it's different from the $t$ used earlier). Now the velocity and acceleration of the robot can be determined:
\begin{equation*}
\begin{split}
    \mathbf{v}(t) &= \frac{d}{dt} \Big[\ \mathbf{r}(s(t))\ \Big] \\
                  &= \mathbf{r}'(s(t)) s'(t) \\
    \mathbf{a}(t) &= \frac{d}{dt} \Big[\ \mathbf{r}'(s(t)) s'(t)\ \Big] \\
                  &= \mathbf{r}''(s(t)) [s'(t)]^2 + \mathbf{r}'(s(t)) s''(t)
\end{split}
\end{equation*}

\section{Heading}
The discussion in the previous sections has been limited to the translational components of the path. This, of course, must be accompanied by some sort of angular motion. For holonomic drives, the heading can be controlled completely independently. In this case, heading can be treated as a third independent path component that can be specified by an arbitrary parametric curve.

However, for nonholonomic drives, the heading is constrained to the direction of travel. For parametric curves, this direction is given by the vector $\mathbf{r}'(s)$:
$$
    \theta(s) = \arctan \frac{y'(s)}{x'(s)}
$$
In practice, $\theta(s)$ is computed with the two-argument version of $\arctan$, eliminating issues arising from signs and cases when $x'(s) = 0$. Of course, the derivatives are also necessary:
\begin{equation*}
\begin{split}
    \theta'(s) &= \frac{1}{\big(\frac{y'(s)}{x'(s)}\big)^2+1} \cdot \frac{x'(s)y''(s) - y'(s)x''(s)}{[x'(s)]^2} \\
               &= \frac{x'(s)y''(s) - y'(s)x''(s)}{[x'(s)]^2 + [y'(s)]^2} \\
               &= x'(s)y''(s) - y'(s)x''(s) \\
    \theta''(s) &= \big[x''(s)y''(s) + x'(s)y'''(s)\big] - \big[y''(s)x''(s) - y'(s)x'''(s)\big] \\
                &= x'(s)y'''(s) - y'(s)x'''(s)
\end{split}
\end{equation*}
Notice that the expressions for $\theta^{(n)}(s)$ involve $\mathbf{r}(s)$ derivatives of order $n + 1$ and lower. Hence, $C^2$ $\mathbf{r}(s)$ only guarantees $C^1$ $\theta(s)$. This heading derivative continuity is one of the primary reasons for quintic over cubic splines.

\section{Conclusion}
This paper discussed the motivation for splines in FTC and some mathematics for generating basic quintic splines. The author hopes this will help further the proliferation of advanced motion planning techniques in FTC.

\end{document}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{hyperref}
\DeclareMathOperator*{\argmax}{arg\,max}
\DeclareMathOperator*{\argmin}{arg\,min}

\usepackage{graphicx}

\usepackage{array}
\newcolumntype{C}[1]{>{\centering\arraybackslash\hspace{0pt}}m{#1}}

\setlength{\parindent}{0cm}

\title{Udacity Navigation Project}
\author{Thomas Lecat}
\date{April 2021}

\begin{document}

    \maketitle

    \section{Environment}\label{sec:environment}

    The environment is described as Markov Decision Process $(S, A, R, p, \gamma)$ with unknown dynamics.
    The goal is to approximate the optimal policy $\pi^*$ that maximizes the upcoming cumulative reward in every state.

    This project uses value based methods, which seek to approximate the optimal $Q$ value $Q^*$, defined as:
    \[
    Q^*(s, a) = \max_{\pi} \mathbb{E}_{\pi} (\sum_{t} \gamma^t r_t | S_0=s, A_0=a),  \forall (a, s) \in A \times S
    \]

    Given $Q^*$, one can derive $\pi^*$ to solve the problem:
    $\pi^*(s) = \argmax_a Q^*(s, a), \forall s \in S$

    \section{Agent}\label{sec:agent}

    \subsection{DQN}\label{subsec:dqn}

    A vanilla \href{https://storage.googleapis.com/deepmind-media/dqn/DQNNaturePaper.pdf}{DQN} agent
    is implemented as the base solution.

    DQN approximates the $Q$ value by a neural network parameterized by $\theta$.
    Its loss function is defined as the mean squared TD error of the Bellman optimality equation:
    \[
    \mathcal{L}(\theta) = \mathbb{E}_{s, a, r, s'} \big[((r + \gamma \max_{a'} Q(s', a' | \theta^-)) - Q(s, a | \theta))^2\big]
    \]

    where $\theta^-$ are the parameters of a target network, slowly updated towards $\theta$ at every training step.
    This target network makes the TD target $r + \gamma \max_{a'} Q(s', a' | \theta^-)$ independent from the parameters $\theta$,
    which stabilizes the training.

    This loss is minimized by an Adam optimizer, a variant of the Stochastic Gradient Descent algorithm.
    At each step, the gradient is approximated from a minibatch of experiences ${(s, a, r, s', d)_i}$.
    To avoid a strong correlation between the steps of the minibatch, transitions are stored in a large replay buffer
    during rollouts and then sampled uniformly.

    \subsection{Double DQN}\label{subsec:double-dqn}

    The DQN agent is implemented in a modular way, to enable extension from recent papers to be added later.

    As a start, we implemented the improvements from the \href{https://arxiv.org/abs/1509.06461}{Double DQN} paper:
    Instead of using the target network to chose the next action and estimate its value in the target, use $Q_\theta$ to chose the action,
    and $Q_{\theta^-}$ to estimate its value.

    \section{Hyperparemeters}\label{sec:hyperparemeters}

    The network architecture is composed of 2 fully connected layers with 64 neurons each.
    The agent follows an $\epsilon$-greedy policy, with epsilon gradually decreasing from $1$ to $0.1$ over $30,000$ steps.

    The table below summarizes the main hyperparameters:

    \begin{tabular}{ |p{5cm} C{1cm}|p{3cm}|p{3cm}| }
        \hline
        \multicolumn{4}{|c|}{Hyperparameters} \\
        \hline
        \multicolumn{2}{|c|}{Parameter}   & Vanilla DQN & Double DQN                    \\
        \hline
        learning rate                     & $\alpha$    & \multicolumn{2}{|c|}{5e-04}   \\
        \hline
        discount factor                   & $\gamma$    & \multicolumn{2}{|c|}{0.99}    \\
        \hline
        target network update coefficient & $\tau$      & \multicolumn{2}{|c|}{0.001}   \\
        \hline
        buffer size                       &             & \multicolumn{2}{|c|}{100,000} \\
        \hline
        batch size                        &             & \multicolumn{2}{|c|}{64}      \\
        \hline
    \end{tabular}

    \section{Results}\label{sec:results}

    Both vanilla DQN and double DQN have been trained with 3 seeds.
    The reward curves are averaged over seeds and plotted below:

    \begin{figure}
        \centering
        \includegraphics[scale=0.5]{results/reward_per_episode.png}\label{fig:figure}
    \end{figure}

    The addition of the double DQN strategy doesn't improve the results much and both
    agents manage to solve the environment in less than 400 episodes.

    \section{Next steps}\label{sec:next-steps}

    The next steps for the project consist in implemented other DQN extensions such as
    \href{https://arxiv.org/abs/1511.06581}{Dueling DQN} and
    \href{https://arxiv.org/abs/1511.05952}{Prioritized Experience Replay}.
    Hyperparemeters could also be further tuned using strategies such as
    \href{https://github.com/hyperopt/hyperopt}{HyperOpt} or
    \href{https://deepmind.com/blog/article/population-based-training-neural-networks}{Population Based Training}.

\end{document}


\chapter{Approach}\label{chap:approach}

\section{Soft Actor-Critic}

The Soft Actor-Critic Algorithm \cite{haarnoja2018soft} is based on the maximizing entropy objective

\begin{equation}
\label{eq:maxent_objective}
  J(\pi)  = \sum_{t=0}^{T} \mathbb{E}(s_{t}, a_{t}) \sim \rho_{pi}[r(s_{t},a_{t}) + \alpha \mathcal{H}(\pi(\cdot|s_{t}))]
\end{equation}

reparameterization trick

\section{Truncated Quantile Critics TQC}

For training the agent the algorithm proposed in \cite{} was used.

Estimating a distribution of the Q-value and an ensemble of critics
enables it to controll  over and under estimation bias

\subsection{Distributional Reinforcement Learning with
Quantile Regression}

With Distributional Reinforcement Learning the objectiv changes to
$ Z^{\pi}(s, a) := \sum_{t=0}^{\infty} \gamma^{t} R\left(s_{t}, a_{t}\right)$
and the Q Function to $Q^{\pi}(s, a):=  \mathbb{E} {Z^{\pi}(s, a)}$.
In the work QR-DQN % \cite{dabney2018distributional}%
the distribution $Z^\pi(s,a)$ is approximated with
$ Z_{\psi}(s, a):=\frac{1}{M} \sum_{m=1}^{M} \delta ( \theta^{m}_\psi(s, a) )$,
a mixture of atoms---Dirac delta functions at locations $\theta^1_\psi(s,a), \dots ,\theta^M_\psi(s,a)$
given by a parametric model  $\theta_{\psi}: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}^M$.

\begin{equation}
  Z_{\psi_n}(s, a) := \frac{1}{M} \sum_{m=1}^{M} \delta \left(\theta_{\psi_n}^m(s, a) \right) ,
\end{equation}

supported on atoms $ \theta_{\psi_n}^1(s,a), \dots, \theta_{\psi_n}^M(s,a) $

\subsection{Loss Function}

The author chooses to use the Wasserstein distance as a metric to minimize
the differences between each of the Q-value distribution
and the temporal difference target distribution Y (s, a).

\begin{equation}
  J_Z (\psi_n) = \mathbb{E}_{\mathcal{D}, \pi} \mathcal{L}^k(s_t,a_t; \psi_n)]
\end{equation}

over the parameters $\psi_n$, where

\begin{equation}
  \mathcal{L}^k(s,a; \psi_n) = \frac{1}{kNM} \sum_{m=1}^M \sum_{i=1}^{kN} \rho_{ \tau_m}^H(y_i(s, a) - \theta_{\psi_n}^>
\end{equation}

The TQC pseudo code is given in Algorithm \ref{alg:main}.

\input{figures/approach/Tqc.tex}


\section{Asset Allocation with Transaction Costs}
\label{sec:asset_allocation}

\begin{frame}{Asset Allocation with Transaction Costs}

	\begin{block}{Goal}
		How to dynamically invest the available capital in a portfolio of different assets in order to maximize the expected total return or another relevant performance measure.
	\end{block}

	\begin{block}{\textbf{Reward function}: portfolio log-return with transaction costs}
		\vspace{-0.5cm}
		\begin{equation*}
		\resizebox{.9 \textwidth}{!}{
		 $R_{t+1} = \log \left\{ 1 + \sum^{I}_{i=0} \left[ a_t^i X_{t+1}^i - \delta_i
		 			 	\left| a_t^i - \tilde{a}_t^i \right| - \delta_s {(a_t^i)}^- \right] -
		 			 	\delta_f \mathbf{1}_{{a}_t \neq \tilde{{a}}_{t-1}}\right\}$
		 }
		 \end{equation*}
	\end{block}

	\begin{block}{\textbf{Actions}: Portfolio weights}
		\begin{equation*}
			\{a_t^i\}_{i=0}^I \;\;\; \text{s.t.}\;\;\; \sum^{I}_{i=0} a_t^i = 1 \;\;\;\;\; \forall t \in \{0, 1, 2, \ldots\}
		\end{equation*}
	\end{block}

	\begin{block}{\textbf{State}: assets past returns and current allocation}
		\begin{equation*}
			S_t = \{X, X_t, X_{t-1}, \ldots, X_{t-P}, \tilde{a}_t\}
		\end{equation*}
	\end{block}

\end{frame}


\documentclass{standalone}
\begin{document}

\subsection{Clustering}

	Clustering approach is similar to the classifiers one, but in an unsupervised fashion, so does not require a training dataset.
	Clustering iteratively alternates between the image segmentation and its classes properties characterization. In this way, we can say that clustering approach trains itself by using the available data information.
	We can identify 3 main clustering algorith
	\begin{itemize}

		\item \textbf{k-means clustering: } partition $n$ observations in $k$ clusters iteratively computing a mean intensity for each class. It segments the image by classifying each pixel in the class with the closest mean

		\item \textbf{Fuzzy C-means: } this algorithm generalizes the K-means clustering in order to achieve soft-segmentation;

		\item \textbf{Expectation Maimization:} uses the same clustering principle as k-means by assuming that each observation belongs to a Gaussian mixture model. It is an iterative method that seek to find maximum likelihood estimates means, covariances and mixing coefficients of the mixture model.
	\end{itemize}

	It does not require training data, but it suffers from high sensitivity to the initial parameters and it does not incorporate spatial models: it is a pixel classification technique~\cite{ART:Pham}.

	The most used algorithm for clustering is the k-means clustering, which seeks to assign each pixel to a particular cluster aiming to minimize the average square distance between pixels in the same cluster~\cite{Arthur2007}. Each cluster is described by a vector representing its centre, so this technique is described as a centroid model~\cite{ART:Morisette}. The labelling is performed by assigning each point to the cluster with the nearest centroid. Each point is assigned to the cluster with the nearest centre.

	Given an integer $k$ and a set of $n$ data points from $\mathbb{R}^d$, the k-means clustering seeks to find the $k$ centres that minimize a potential function given by the sum of squares:
	\begin{equation}
		\Phi = \sum_{x\in S}\min\| x - c\|^2
	\end{equation}
	Where $S\subset \mathbb R^d$ is a set of points. In this work, $\mathbb{R}^d$ will be the colours space and $S$ is the space of colour of each voxel.

	The steps of the algorithm are the following:
		\begin{enumerate}
			\item Randomly initialize the values of the $k$ centres,
			\item Generates the $k$ clusters associating each point to its nearest centroid
			\item Re-compute the centroid for each cluster until the centroid does not change.

		\end{enumerate}

	Arthur and Vassilvitskii ~\cite{Arthur2007} have pointed out that this algorithm is not accurate and can produce arbitrarily bad clusters. So they have developed a method, the "k-means++" which improves the clustering accuracy by made an accurate choice of the initial cluster centres.

	They pointed out that the bad clustering is caused to the fact that $\frac{\Phi}{\Phi_{opt}}$ is unbounded even if the number of clusters and points are fixed, where $\Phi_{opt}$ is the potential function in the optimal centroids case. They proposed a variant for the choosing of the centroids: They randomly chose only the first centroid. To chose the other, they weight the initial points according to the distance square ($D(x)^2$) from the closest centre already chosen. So the final algorithm is equal to the k-means except for the initial centroids selection that is made as follows:
	\begin{enumerate}
			\item Take one centre $c_1$, chosen uniformly at random from $S$.

			\item  Take a new centre $c_i$, choosing $x \in Si$ with probability $\frac{D(x)^2}{\sum _{x \in S} D(x)^2}$

			\item Repeat the step 2 till $k$ centres are chosen

			\item Proceed like a classical k-means clustering.

		\end{enumerate}

	They have proved that this approach leads to better results in less time. For more details refer to ~\cite{Arthur2007}.

\end{document}


\hypertarget{regression}{%
\chapter{Regression}\label{regression}}

In the previous chapter we used simple regression to quantify the
relationship between two variables. In this chapter we'll get farther
into regression, including multiple regression and one of my all-time
favorite tools, logistic regression.

These tools will allow us to explore relationships among sets of
variables. As an example, we will use data from the General Social
Survey (GSS) to explore the relationship between income, education, age,
and sex. But first let's understand the limits of simple regression.

\hypertarget{limits-of-simple-regression}{%
\section{Limits of Simple
Regression}\label{limits-of-simple-regression}}

In a previous exercise, you made a scatter plot of vegetable consumption
as a function of income, and plotted a line of best fit. Here's what it
looks like:

\includegraphics{figs/fig09-01.png}

The slope of the line is 0.07, which means that the difference between
the lowest and highest income brackets is about 0.49 servings per day.
So that's not a very big difference.

But it was an arbitrary choice to plot vegetables as a function of
income. We could have plotted it the other way around, like this.

\includegraphics{figs/fig09-02.png}

The slope of this line is about 0.2, which means that the difference
between 0 and 10 servings per day is about 2 income levels, roughly from
level 5 to level 7.

And the difference between income levels 5 and 7 is about \$30,000 per
year, which is substantial.

So if we use vegetable consumption to predict income, we see a big
difference. But when we used income to predict vegetable consumption, we
saw a small difference.

This example shows that regression is not symmetric; the regression of A
onto B is not the same as the regression of B onto A.

We can see that more clearly by putting the two figures side by side and
plotting both regression lines on both figures.

\includegraphics{figs/fig09-03.png}

They are different because they are based on different assumptions.

\begin{itemize}
\item
  On the left, we treat income as a known quantity and vegetable
  consumption as random.
\item
  On the right, we treat vegetable consumption as known and income as
  random.
\end{itemize}

When you run a regression model, you make decisions about how to treat
the data, and those decisions affect the results you get.

This example demonstrates another point, which is that regression
doesn't tell you much about causation.

\begin{itemize}
\item
  If you think people with lower income can't afford vegetables, you
  might look at the figure on the left and conclude that it doesn't make
  much difference.
\item
  If you think better diet increases income, the figure on the right
  might make you think it does.
\end{itemize}

But in general, regression can't tell you what causes what. If you see a
relationship between any two variables, A and B, the reason for the
relationship might be that A causes B, or B causes A, or there might be
other factors that cause both A and B. Regression alone can't tell you
which way it goes.

However, we have tools for teasing apart relationships among multiple
variables; one of the most important is multiple regression.

SciPy doesn't do multiple regression, so we'll to switch to a new
library, StatsModels. Here's the import statement.

\begin{lstlisting}[language=Python,style=source]
import statsmodels.formula.api as smf
\end{lstlisting}

For the first example, we'll load data from the Behavioral Risk Factor
Surveillance Survey (BRFSS), which we saw in the previous chapter.

\begin{lstlisting}[language=Python,style=source]
import pandas as pd

brfss = pd.read_hdf('brfss.hdf5', 'brfss')
\end{lstlisting}

Now we can use StatsModels to fit a regression model. The name of the
function is \passthrough{\lstinline!ols!}, which stands for ``ordinary
least squares'', another name for regression.

\begin{lstlisting}[language=Python,style=source]
results = smf.ols('INCOME2 ~ _VEGESU1', data=brfss).fit()
\end{lstlisting}

The first argument is a \textbf{formula string} that specifies that we
want to regress income as a function of vegetable consumption.

The second argument is the BRFSS \passthrough{\lstinline!DataFrame!}.
The names in the formula correspond to columns in the
\passthrough{\lstinline!DataFrame!}.

The result from \passthrough{\lstinline!ols!} represents the model; then
we run \passthrough{\lstinline!fit!} to get the results.

Results is a \passthrough{\lstinline!RegressionResultsWrapper!}, which
contains a lot of information; the first thing we'll look at is the
attribute \passthrough{\lstinline!params!}, which contains the estimated
intercept and the slope associated with
\passthrough{\lstinline!\_VEGESU1!}.

\begin{lstlisting}[language=Python,style=source]
results.params
\end{lstlisting}

\begin{tabular}{lr}
\toprule
{} &         0 \\
\midrule
Intercept &  5.450700 \\
\_VEGESU1  &  0.204935 \\
\bottomrule
\end{tabular}

And we get the same results we got from SciPy, so that's good!

In the next section we'll move on to multiple regression. But first,
some exercises.

\textbf{Exercise:} In the BRFSS dataset, there is a strong relationship
between vegetable consumption and income. The income of people who eat 8
servings of vegetables per day is double the income of people who eat
none, on average.

Which of the following conclusions can we draw from this data?

\begin{enumerate}
\def\labelenumi{\Alph{enumi}.}
\item
  Eating a good diet leads to better health and higher income.
\item
  People with higher income can afford a better diet.
\item
  People with high income are more likely to be vegetarians.
\end{enumerate}

\textbf{Exercise:} Let's run a regression using SciPy and StatsModels,
and confirm we get the same results.

\begin{itemize}
\item
  Compute the regression of \passthrough{\lstinline!\_VEGESU1!} as a
  function of \passthrough{\lstinline!INCOME2!} using SciPy's
  \passthrough{\lstinline!linregress()!}.
\item
  Compute the regression of \passthrough{\lstinline!\_VEGESU1!} as a
  function of \passthrough{\lstinline!INCOME2!} using StatsModels'
  \passthrough{\lstinline!smf.ols()!}.
\end{itemize}

Note: \passthrough{\lstinline!linregress!} does not handle
\passthrough{\lstinline!NaN!} values, so you will have to use
\passthrough{\lstinline!dropna!} to select the rows with valid data.

\hypertarget{multiple-regression}{%
\section{Multiple Regression}\label{multiple-regression}}

Now that we have StatsModels, getting from simple to multiple regression
is easy. As an example, we'll use data from the General Social Survey
(GSS) and we'll explore variables that are related to income.

First, let's load the GSS data.

\begin{lstlisting}[language=Python,style=source]
import pandas as pd

gss = pd.read_hdf('gss_eda.hdf', 'gss')
\end{lstlisting}

Here are the first few rows of \passthrough{\lstinline!gss!}:

\begin{lstlisting}[language=Python,style=source]
gss.head()
\end{lstlisting}

\begin{tabular}{lrrrrrrrr}
\toprule
{} &  YEAR &  ID\_ &   AGE &  EDUC &  SEX &  GUNLAW &  GRASS &  REALINC \\
\midrule
0 &  1972 &    1 &  23.0 &  16.0 &    2 &     1.0 &    NaN &  18951.0 \\
1 &  1972 &    2 &  70.0 &  10.0 &    1 &     1.0 &    NaN &  24366.0 \\
2 &  1972 &    3 &  48.0 &  12.0 &    2 &     1.0 &    NaN &  24366.0 \\
3 &  1972 &    4 &  27.0 &  17.0 &    2 &     1.0 &    NaN &  30458.0 \\
4 &  1972 &    5 &  61.0 &  12.0 &    2 &     1.0 &    NaN &  50763.0 \\
\bottomrule
\end{tabular}

We'll start with another simple regression, estimating the parameters of
real income as a function of years of education.

\begin{lstlisting}[language=Python,style=source]
results = smf.ols('REALINC ~ EDUC', data=gss).fit()
results.params
\end{lstlisting}

\begin{tabular}{lr}
\toprule
{} &             0 \\
\midrule
Intercept & -13054.459834 \\
EDUC      &   3464.463066 \\
\bottomrule
\end{tabular}

On the left side of the formula string,
\passthrough{\lstinline!REALINC!} is the variable we are trying to
predict; on the right, \passthrough{\lstinline!EDUC!} is the variable we
are using to inform the predictions.

The estimated slope is about \passthrough{\lstinline!3450!}, which means
that each additional year of education is associated with an additional
\$3450 of income. But income also depends on age, so it would be good to
include that in the model, too. Here's how:

\begin{lstlisting}[language=Python,style=source]
results = smf.ols('REALINC ~ EDUC + AGE', data=gss).fit()
results.params
\end{lstlisting}

\begin{tabular}{lr}
\toprule
{} &             0 \\
\midrule
Intercept & -16152.855386 \\
EDUC      &   3514.291894 \\
AGE       &     54.008253 \\
\bottomrule
\end{tabular}

On the right side of the formula string, you can list as many variables
as you like, in this case, education and age. The
\passthrough{\lstinline!plus!} sign indicates that we expect the
contributions of the two variables to be additive, which is a common
assumption for models like this.

The estimated slope for \passthrough{\lstinline!EDUC!} is a little less
than what we saw before, about \$3514 per year.

The estimated slope for \passthrough{\lstinline!AGE!} is only about \$54
per year, which is surprisingly small. To see what's going on, let's
look more closely at the relationship between income and age.

\hypertarget{groupby}{%
\section{Groupby}\label{groupby}}

I'll use \passthrough{\lstinline!groupby()!}, which is a Pandas feature
we have not seen before, to divide the DataFrame into age groups. The
result is a \passthrough{\lstinline!GroupBy!} object that contains one
group for each value of \passthrough{\lstinline!age!}.

\begin{lstlisting}[language=Python,style=source]
grouped = gss.groupby('AGE')
type(grouped)
\end{lstlisting}

\begin{lstlisting}[style=output]
pandas.core.groupby.generic.DataFrameGroupBy
\end{lstlisting}

The \passthrough{\lstinline!GroupBy!} object behaves like a DataFrame in
many ways. You can use brackets to select a column, like
\passthrough{\lstinline!REALINC!} in this example, and then invoke a
method like \passthrough{\lstinline!mean()!}.

\begin{lstlisting}[language=Python,style=source]
mean_income_by_age = grouped['REALINC'].mean()
\end{lstlisting}

The result is a Pandas series that contains the mean income for each age
group, which we can plot like this.

\begin{lstlisting}[language=Python,style=source]
import matplotlib.pyplot as plt

plt.plot(mean_income_by_age, 'o', alpha=0.5)
plt.xlabel('Age (years)')
plt.ylabel('Income (1986 $)')
plt.title('Average income, grouped by age');
\end{lstlisting}

\begin{center}
\includegraphics[scale=0.75]{10_regression_files/10_regression_36_0.pdf}
\end{center}

Average income increases from age 20 to age 50, then starts to fall.\\
And that explains why the estimated slope is so small, because the
relationship is non-linear. Remember that correlation and simple
regression can't measure non-linear relationships.

But multiple regression can! To describe a non-linear relationship, one
option is to add a new variable that is a non-linear combination of
other variables.

As an example, I'll create a new variable called
\passthrough{\lstinline!AGE2!} that equals \passthrough{\lstinline!AGE!}
squared.

\begin{lstlisting}[language=Python,style=source]
gss['AGE2'] = gss['AGE']**2
\end{lstlisting}

Now we can run a regression with both \passthrough{\lstinline!age!} and
\passthrough{\lstinline!age2!} on the right side.

\begin{lstlisting}[language=Python,style=source]
model = smf.ols('REALINC ~ EDUC + AGE + AGE2', data=gss)
results = model.fit()
results.params
\end{lstlisting}

\begin{tabular}{lr}
\toprule
{} &             0 \\
\midrule
Intercept & -49865.446557 \\
EDUC      &   3293.454914 \\
AGE       &   1758.622812 \\
AGE2      &    -17.341566 \\
\bottomrule
\end{tabular}

Now the slope associated with \passthrough{\lstinline!AGE!} is
substantial, about \$1760 per year.

The slope associated with \passthrough{\lstinline!AGE2!} is about -\$17,
but that's harder to interpret.

In the next section, we'll see methods to interpret multivariate models
and visualize the results. But first, let's practice multiple
regression.

\textbf{Exercise:} To get a closer look at the relationship between
income and education, let's use the variable
\passthrough{\lstinline!EDUC!} to group the data, then plot mean income
in each group.

\begin{itemize}
\item
  Group \passthrough{\lstinline!gss!} by \passthrough{\lstinline!EDUC!}.
  Store the result in \passthrough{\lstinline!grouped!}.
\item
  From \passthrough{\lstinline!grouped!}, extract
  \passthrough{\lstinline!REALINC!} and compute the mean.
\item
  Plot mean income in each education group as a scatter plot.
\end{itemize}

What can you say about the relationship between education and income?
Does it look like a linear relationship?

\textbf{Exercise:} The graph in the previous exercise suggests that the
relationship between income and education is non-linear. So let's try
fitting a non-linear model.

\begin{itemize}
\item
  Add a column named \passthrough{\lstinline!EDUC2!} to the
  \passthrough{\lstinline!gss!} DataFrame; it should contain the values
  from \passthrough{\lstinline!EDUC!} squared.
\item
  Run a regression model that uses \passthrough{\lstinline!EDUC!},
  \passthrough{\lstinline!EDUC2!}, \passthrough{\lstinline!age!}, and
  \passthrough{\lstinline!age2!} to predict
  \passthrough{\lstinline!REALINC!}.
\end{itemize}

\hypertarget{visualizing-regression-results}{%
\section{Visualizing regression
results}\label{visualizing-regression-results}}

In the previous section we ran a multiple regression model to
characterize the relationships between income, age, and education.
Because the model includes quadratic terms, the parameters are hard to
interpret. For example, you might notice that the parameter for
\passthrough{\lstinline!EDUC!} is negative, and that might be a
surprise, because it suggests that higher education is associated with
lower income.

But the parameter for \passthrough{\lstinline!EDUC2!} is positive, and
that makes a big difference. In this section we'll see a way to
interpret the model visually and validate it against data.

Here's the model from the previous exercise.

\begin{lstlisting}[language=Python,style=source]
gss['EDUC2'] = gss['EDUC']**2

model = smf.ols('REALINC ~ EDUC + EDUC2 + AGE + AGE2', data=gss)
results = model.fit()
results.params
\end{lstlisting}

\begin{tabular}{lr}
\toprule
{} &             0 \\
\midrule
Intercept & -26080.884938 \\
EDUC      &   -522.032930 \\
EDUC2     &    153.405410 \\
AGE       &   1711.838648 \\
AGE2      &    -17.128130 \\
\bottomrule
\end{tabular}

Sometimes we can understand a model by looking at its parameters, but
often it is better to look at its predictions.

The regression results provide a method called
\passthrough{\lstinline!predict!} that uses the model to generate
predictions. It takes a \passthrough{\lstinline!DataFrame!} as a
parameter and returns a \passthrough{\lstinline!Series!} with a
prediction for each row in the \passthrough{\lstinline!DataFrame!}. To
use it, I'll create a new \passthrough{\lstinline!DataFrame!} with
\passthrough{\lstinline!AGE!} running from 18 to 89, and
\passthrough{\lstinline!AGE2!} set to \passthrough{\lstinline!AGE!}
squared.

\begin{lstlisting}[language=Python,style=source]
import numpy as np

df = pd.DataFrame()
df['AGE'] = np.linspace(18, 89)
df['AGE2'] = df['AGE']**2
\end{lstlisting}

Next, I'll pick a level for \passthrough{\lstinline!EDUC!}, like 12
years, which is the most common value. When you assign a single value to
a column in a \passthrough{\lstinline!DataFrame!}, Pandas makes a copy
for each row.

\begin{lstlisting}[language=Python,style=source]
df['EDUC'] = 12
df['EDUC2'] = df['EDUC']**2
\end{lstlisting}

Then we can use \passthrough{\lstinline!results!} to predict the average
income for each age group, holding education constant.

\begin{lstlisting}[language=Python,style=source]
pred12 = results.predict(df)
\end{lstlisting}

The result from \passthrough{\lstinline!predict!} is a
\passthrough{\lstinline!Series!} with one prediction for each row. So we
can plot it with age on the \(x\)-axis and the predicted income for each
age group on the \(y\)-axis. And we can plot the data for comparison.

\begin{lstlisting}[language=Python,style=source]
plt.plot(mean_income_by_age, 'o', alpha=0.5)

plt.plot(df['AGE'], pred12, label='High school', color='C4')

plt.xlabel('Age (years)')
plt.ylabel('Income (1986 $)')
plt.title('Income versus age, grouped by education level')
plt.legend();
\end{lstlisting}

\begin{center}
\includegraphics[scale=0.75]{10_regression_files/10_regression_53_0.pdf}
\end{center}

The dots show the average income in each age group. The line shows the
predictions generated by the model, holding education constant. This
plot shows the shape of the model, a downward-facing parabola.

We can do the same thing with other levels of education, like 14 years,
which is the nominal time to earn an Associate's degree, and 16 years,
which is the nominal time to earn a Bachelor's degree.

\begin{lstlisting}[language=Python,style=source]
df['EDUC'] = 16
df['EDUC2'] = df['EDUC']**2
pred16 = results.predict(df)

df['EDUC'] = 14
df['EDUC2'] = df['EDUC']**2
pred14 = results.predict(df)
\end{lstlisting}

\begin{lstlisting}[language=Python,style=source]
plt.plot(mean_income_by_age, 'o', alpha=0.5)

plt.plot(df['AGE'], pred16, ':', label='Bachelor')
plt.plot(df['AGE'], pred14, '--', label='Associate')
plt.plot(df['AGE'], pred12, label='High school', color='C4')

plt.xlabel('Age (years)')
plt.ylabel('Income (1986 $)')
plt.title('Income versus age, grouped by education level')
plt.legend();
\end{lstlisting}

\begin{center}
\includegraphics[scale=0.75]{10_regression_files/10_regression_56_0.pdf}
\end{center}

The lines show mean income, as predicted by the model, as a function of
age, for three levels of education. This visualization helps validate
the model, since we can compare the predictions with the data. And it
helps us interpret the model since we can see the separate contributions
of age and education.

In the exercises, you'll have a chance to run a multiple regression,
generate predictions, and visualize the results.

\textbf{Exercise:} At this point, we have a model that predicts income
using age, education, and sex.

Let's see what it predicts for different levels of education, holding
\passthrough{\lstinline!AGE!} constant.

\begin{itemize}
\item
  Create an empty \passthrough{\lstinline!DataFrame!} named
  \passthrough{\lstinline!df!}.
\item
  Using \passthrough{\lstinline!np.linspace()!}, add a variable named
  \passthrough{\lstinline!EDUC!} to \passthrough{\lstinline!df!} with a
  range of values from \passthrough{\lstinline!0!} to
  \passthrough{\lstinline!20!}.
\item
  Add a variable named \passthrough{\lstinline!AGE!} with the constant
  value \passthrough{\lstinline!30!}.
\item
  Use \passthrough{\lstinline!df!} to generate predicted income as a
  function of education.
\end{itemize}

\textbf{Exercise:} Now let's visualize the results from the previous
exercise!

\begin{itemize}
\item
  Group the GSS data by \passthrough{\lstinline!EDUC!} and compute the
  mean income in each education group.
\item
  Plot mean income for each education group as a scatter plot.
\item
  Plot the predictions from the previous exercise.
\end{itemize}

How do the predictions compare with the data?

\textbf{Optional Exercise:} Extend the previous exercise to include
predictions for a few other age levels.

\hypertarget{categorical-variables}{%
\section{Categorical variables}\label{categorical-variables}}

Most of the variables we have used so far --- like income, age, and
education --- are numerical. But variables like sex and race are
categorical; that is, each respondent belongs to one of a specified set
of categories.

With StatsModels, it is easy to include a categorical variable as part
of a regression model. Here's an example:

\begin{lstlisting}[language=Python,style=source]
formula = 'REALINC ~ EDUC + EDUC2 + AGE + AGE2 + C(SEX)'
results = smf.ols(formula, data=gss).fit()
results.params
\end{lstlisting}

\begin{tabular}{lr}
\toprule
{} &             0 \\
\midrule
Intercept   & -24567.566164 \\
C(SEX)[T.2] &  -4626.727813 \\
EDUC        &   -303.398385 \\
EDUC2       &    143.954306 \\
AGE         &   1702.199322 \\
AGE2        &    -16.995151 \\
\bottomrule
\end{tabular}

In the formula string, the letter \passthrough{\lstinline!C!} indicates
that \passthrough{\lstinline!SEX!} is a categorical variable.

The regression treats the value \passthrough{\lstinline!SEX=1!}, which
is male, as the default, and reports the difference associated with the
value \passthrough{\lstinline!SEX=2!}, which is female. So this result
indicates that income for women is about \$4156 less than for men, after
controlling for age and education.

\hypertarget{logistic-regression}{%
\section{Logistic Regression}\label{logistic-regression}}

In the previous section, we added a categorical variables on the right
side of a regression formula; that is, we used it as a predictive
variables.

But what if the categorical variable is on the left side of the
regression formula; that is, it's the value we are trying to predict? In
that case, we can use \textbf{logistic regression}.

As an example, one of the questions in the General Social Survey asks
``Would you favor or oppose a law which would require a person to obtain
a police permit before he or she could buy a gun?'' The responses are in
a column called \passthrough{\lstinline!GUNLAW!}; here are the values.

\begin{lstlisting}[language=Python,style=source]
gss['GUNLAW'].value_counts()
\end{lstlisting}

\begin{tabular}{lr}
\toprule
{} &  GUNLAW \\
\midrule
1.0 &   32038 \\
2.0 &    9975 \\
\bottomrule
\end{tabular}

\passthrough{\lstinline!1!} means yes and \passthrough{\lstinline!2!}
means no, so most respondents are in favor.

To explore the relationship between this variable and factors like age,
sex, and education, we can use StatsModels, which provides a function
that does logistic regression.

To use it, we have to recode the variable so \passthrough{\lstinline!1!}
means ``yes'' and \passthrough{\lstinline!0!} means ``no''. We can do
that by replacing \passthrough{\lstinline!2!} with
\passthrough{\lstinline!0!}.

\begin{lstlisting}[language=Python,style=source]
gss['GUNLAW'].replace([2], [0], inplace=True)
\end{lstlisting}

Now we can run the regression. Instead of
\passthrough{\lstinline!ols()!}, we use
\passthrough{\lstinline!logit()!}, which is named for the logit
function, which is related to logistic regression.

\begin{lstlisting}[language=Python,style=source]
formula = 'GUNLAW ~ AGE + AGE2 + EDUC + EDUC2 + C(SEX)'
results = smf.logit(formula, data=gss).fit()
\end{lstlisting}

\begin{lstlisting}[style=output]
Optimization terminated successfully.
         Current function value: 0.532878
         Iterations 6
\end{lstlisting}

Estimating the parameters for the logistic model is an iterative
process, so the output contains information about the number of
iterations. Other than that, everything is the same as what we have seen
before. And here are the results.

\begin{lstlisting}[language=Python,style=source]
results.params
\end{lstlisting}

\begin{tabular}{lr}
\toprule
{} &         0 \\
\midrule
Intercept   &  1.347289 \\
C(SEX)[T.2] &  0.771791 \\
AGE         & -0.021314 \\
AGE2        &  0.000222 \\
EDUC        & -0.075406 \\
EDUC2       &  0.004867 \\
\bottomrule
\end{tabular}

The parameters are in the form of \textbf{log odds}, which you may or
may not be familiar with. I won't explain them in detail here, except to
say that positive values are associated with things that make the
outcome more likely, and negative values make the outcome less likely.

For example, the parameter associated with
\passthrough{\lstinline!SEX=2!} is 0.75, which indicates that women are
more likely to support this form of gun control. To see how much more
likely, we can generate predictions, as we did with linear regression.

As an example, I'll generate predictions for different ages and sexes,
with education held constant.

First we need a \passthrough{\lstinline!DataFrame!} with
\passthrough{\lstinline!AGE!} and \passthrough{\lstinline!EDUC!}.

\begin{lstlisting}[language=Python,style=source]
df = pd.DataFrame()
df['AGE'] = np.linspace(18, 89)
df['EDUC'] = 12
\end{lstlisting}

Then we can compute \passthrough{\lstinline!AGE2!} and
\passthrough{\lstinline!EDUC2!}.

\begin{lstlisting}[language=Python,style=source]
df['AGE2'] = df['AGE']**2
df['EDUC2'] = df['EDUC']**2
\end{lstlisting}

We can generate predictions for men like this.

\begin{lstlisting}[language=Python,style=source]
df['SEX'] = 1
pred1 = results.predict(df)
\end{lstlisting}

And for women like this.

\begin{lstlisting}[language=Python,style=source]
df['SEX'] = 2
pred2 = results.predict(df)
\end{lstlisting}

Now, to visualize the results, I'll start by plotting the data. As we've
done before, we'll divide the respondents into age groups and compute
the mean in each group. The mean of a binary variable is the fraction of
people in favor.

Then we can plot the predictions, for men and women, as a function of
age.

\begin{lstlisting}[language=Python,style=source]
grouped = gss.groupby('AGE')
favor_by_age = grouped['GUNLAW'].mean()
plt.plot(favor_by_age, 'o', alpha=0.5)

plt.plot(df['AGE'], pred2, label='Female')
plt.plot(df['AGE'], pred1, '--', label='Male')

plt.xlabel('Age')
plt.ylabel('Probability of favoring gun law')
plt.title('Support for gun law versus age, grouped by sex')
plt.legend();
\end{lstlisting}

\begin{center}
\includegraphics[scale=0.75]{10_regression_files/10_regression_83_0.pdf}
\end{center}

According to the model, people near age 50 are least likely to support
gun control (at least as this questions was posed). And women are more
likely to support it than men, by almost 15 percentage points.

Logistic regression is a powerful tool for exploring relationships
between a binary variable and the factors that predict it. In the
exercises, you'll explore the factors that predict support for
legalizing marijuana.

\textbf{Exercise:} Let's use logistic regression to predict a binary
variable. Specifically, we'll use age, sex, and education level to
predict support for legalizing cannabis (marijuana) in the U.S.

In the GSS dataset, the variable \passthrough{\lstinline!GRASS!} records
the answer to the question ``Do you think the use of marijuana should be
made legal or not?''

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\item
  First, use \passthrough{\lstinline!replace!} to recode the
  \passthrough{\lstinline!GRASS!} column so that
  \passthrough{\lstinline!1!} means yes and \passthrough{\lstinline!0!}
  means no. Use \passthrough{\lstinline!value\_counts!} to check.
\item
  Next, use \passthrough{\lstinline!smf.logit()!} to predict
  \passthrough{\lstinline!GRASS!} using the variables
  \passthrough{\lstinline!AGE!}, \passthrough{\lstinline!AGE2!},
  \passthrough{\lstinline!EDUC!}, and \passthrough{\lstinline!EDUC2!},
  along with \passthrough{\lstinline!SEX!} as a categorical variable.
  Display the parameters. Are men or women more likely to support
  legalization?
\item
  To generate predictions, start with an empty DataFrame. Add a column
  called \passthrough{\lstinline!AGE!} that contains a sequence of
  values from 18 to 89. Add a column called
  \passthrough{\lstinline!EDUC!} and set it to 12 years. Then compute a
  column, \passthrough{\lstinline!AGE2!}, which is the square of
  \passthrough{\lstinline!AGE!}, and a column,
  \passthrough{\lstinline!EDUC2!}, which is the square of
  \passthrough{\lstinline!EDUC!}.
\item
  Use \passthrough{\lstinline!predict!} to generate predictions for men
  (\passthrough{\lstinline!SEX=1!}) and women
  (\passthrough{\lstinline!SEX=2!}).
\item
  Generate a plot that shows (1) the average level of support for
  legalizing marijuana in each age group, (2) the level of support the
  model predicts for men as a function of age, and (3) the level of
  support predicted for women as a function of age.
\end{enumerate}

\hypertarget{summary}{%
\section{Summary}\label{summary}}

At this point, I'd like to summarize the topics we've covered so far,
and make some connections that might clarify the big picture.

A central theme of this book is \textbf{exploratory data analysis},
which is a process and a set of techniques for working with data,
especially in the early stages of a project, or when you are working
with a new data set.

The last four chapters demonstrate the steps of this process:

\begin{itemize}
\item
  Chapter 7 is about importing and cleaning data, and checking for
  errors and other special conditions. This might not be the most
  exciting part of the process, but time spent validating data can save
  you from embarrassing errors.
\item
  Chapter 8 is about exploring variables one at a time, visualizing
  distributions using PMFs, CDFs, and KDE, and choosing appropriate
  summary statistics.
\item
  In Chapter 9 we explored relationships between variables two at a
  time, using scatter plots and other visualizations; and we quantified
  those relationships using correlation and simple regression.
\item
  Finally, in this chapter, we explored multivariate relationships using
  multiple regression and logistic regression.
\end{itemize}

In Chapter 7, we looked at the distribution of birth weights from the
National Survey of Family Growth. If you only remember one thing,
remember the 99 pound babies, and how much it can affect your results if
you don't validate the data.

In Chapter 8 we looked at the distributions of age, income, and other
variables from the General Social Survey. I recommended using CDFs as
the best way to explore distributions. But when you present to audiences
that are not familiar with CDFs, you can use PMFs if there are a small
number of unique values, and KDE if there are a lot.

In Chapter 9 we looked at heights and weights from the BRFSS, and
developed several ways to visualize relationships between variables,
including scatter plots, violin plots, and box plots.

We used the coefficient of correlation to quantify the strength of a
relationship. We also used simple regression to estimate slope, which is
often what we care more about, not correlation.

But remember that both of these methods only capture linear
relationships; if the relationship is non-linear, they can be
misleading. Always look at a visualization, like a scatter plot, before
computing correlation or simple regression.

In Chapter 10 we used multiple regression to add control variables and
to describe non-linear relationships. And finally we used logistic
regression to explain and predict binary variables.

We moved through a lot of material quickly, but if you practice and
apply these methods to other questions and other datasets, you will
learn more as you go.


\section{Application to Color Transfer}
\label{sec-appli-color}

This section shows how the relaxed and regularized OT formulation can be applied to imaging problems, more specifically to color transfer, and how the regularization and the relaxation improve the results obtained by previous methods. The color transfer problem consists in modifying an input image $X^0$ so that its colors match the colors of another input image $Y^0$.

\begin{figure*}[h]
\centering
\begin{tabular}{@{}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{}}
\includegraphics[width=.32\linewidth]{colorization/parrot_1} &
\includegraphics[width=.32\linewidth]{colorization/parrot_2} &
\includegraphics[width=.32\linewidth]{colorization/asymmetricparrot_l00008_KX1_KY1_nn4} \\
$X^0$ & $Y^0$ & ${\tilde X^0}$ \\
\includegraphics[width=.32\linewidth]{colorization/histo3Dx} &
\includegraphics[width=.32\linewidth]{colorization/histo3Dy} &
\includegraphics[width=.32\linewidth]{colorization/histo3Du}\\
$\mu_{X^0}$ & $\mu_{Y^0}$ & $\mu_{\tilde X^0}$
\end{tabular}
\caption{Example of the colorization problem. Given images $X^0$ and $Y^0$ with their corresponding 3-D color distributions $\mu_{X^0}$ and $\mu_{Y^0}$ (represented here using their 2-D projection on the RG plane), the goal of colorization methods is to define an image ${\tilde X^0}$ that has the geometry of $X^0$ and a histogram $\mu_{{\tilde X}^0}$  that is similar to $\mu_{Y^0}$.}\label{imcolorization}
\end{figure*}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Color Images and Histograms}

In the following, an image is stored as a vector $X^0 \in \RR^{N_0 \times d}$ where $d=3$ is the number of channels (here $d=3$ since we handle color images, with R, G and B color channels) and where $N_0=N_1 N_2$ is the number of pixels ($N_1$ being horizontal and $N_2$ vertical dimensions). The color histogram of such an image $X^0$ can be estimated using the empirical distribution $\mu_{X^0}$. The goal of color transfer algorithms is to compute a transformation $T^0$ such that $(\tilde X^0)_i = T^0(X^0_i)$, where the new empirical distribution $\mu_{\tilde X^0}$ is close (or equal) to $\mu_{Y^0}$. Figure~\ref{imcolorization} shows an example where $X^0$, $Y^0$ are the original input images, the second row displays the 2-D projection of the 3-D distribution of pixels $\mu_{X^0}$ and $\mu_{Y^0}$, and in the third column, we show the $\mu_{\tilde X^0}$ which is the result of applying $T^0$ to $X^0$, where $T^0$ is computed using the method described below. The associated image ${\tilde X^0}$ has the geometry of $X^0$ and the color palette (3-D histogram) of $Y^0$.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Regularized OT Color Transfer}

As exposed in Section~\ref{subsec-ot-imaging}, OT is now routinely used to perform color palette modification, and in particular color transfer. As we illustrate below in the numerical examples, relaxing the mass conservation constraint is crucial in order to better match the modes (i.e. the dominant colors) of each distribution. Regularizing the transport is also important to reduce colorization artifacts.

To make the optimization problem~\eqref{eq-symm-reg-energy} tractable for histograms obtained from large scale images, we apply the method on a sub-sampled point cloud. That is to say, before computing the relaxed and regularized transport, we define two smaller point clouds $X$ and $Y$ from $X^0$ and $Y^0$. These clouds are created such that their respective distributions $\mu_X$ and $\mu_Y$ are close to the two original distributions $\mu_{X^0}$ and $\mu_{Y^0}$. The mapping $T$ between these small clouds is then extended by interpolation to the original clouds. The complete algorithm for regularized OT color transfer between a pair of images $(X^0,Y^0)$ is exposed in Algorithm~\ref{alg-rot}. We now detail each step of the method.

\begin{algorithm}[ht!]
\caption{Regularized OT Color Transfer}
\label{alg-rot}
% \begin{algorithmic}[1]
\Require Images $X^0,Y^0 \in \RR^{N_0 \times d}$, $\la_X,\la_Y \in \RR^+ $, and $k_X,K_X,k_Y,K_Y \in \RR^+$, where $k_X \le K_X$ and $k_Y \le K_Y$.

\Ensure Image $\tilde X^0 \in \RR^{N_0 \times d}$.
% \Statex
\begin{enumerate}
	\algostep{Histogram down-sample} Compute $X,Y$ from $X^0,Y^0$ respectively \\ using K-means clustering.
	\algostep{Compute Mapping} Compute the optimal $\Sig$ such that  $T(X) = \diag(\Sig \U)^{-1} \Sig Y$ by solving eq.~\eqref{eq-symm-reg-energy} with algorithm~\eqref{eq-frankwolfe-update} or the linear program~\eqref{eq-symm-TV} solving with an interior point algorithm.
	%\algostep{Transport up-sample} Compute $\tilde T^0$ by solving eq.~\eqref{eq-upsample}, where  $T(X) = diag(\Sig \U)^{-1} \Sig Y$.
	\algostep{Obtain high resolution result} Compute $\tilde X^0$ with eq.~\eqref{eq-upsample}.
\end{enumerate}
% \end{algorithmic}
\end{algorithm}

%%%%%%%%
\paragraph{Pixels down-sampling}

We construct a smaller data set $X \in \RR^{N \times d}$ by clustering the set $X^0$ into $N$ clusters with the K-means algorithm (see~\cite{Lloyd57}).
% and~\cite{Quantization} for its application to image quantization).
Each cluster corresponds to a point $X_i$ in our smaller data set $X$.  The same procedure is done for $Y^0$ to obtain $Y \in \RR^{N \times d}$.%, and we compute

%%%%%%%%
\paragraph{Graph and $(G_X,G_Y)$ operator}

As exposed in Section~\ref{sec:regsymme}, the regularization is defined using gradient operators $(G_X,G_Y)$ on graphs $(\Gg_X,\Gg_Y)$ connecting the points in $X$ and $Y$. Inspired by several recent works on manifold learning (see Section~\ref{subsec-regul-intro}), we use here a $n$-nearest neighbor graph, where $n$ is the number of edges adjacent to each vertex, i.e. $\abs{\enscond{j}{(i,j) \in E_X}} = n$ where $E_X$ is the set of edges of $X$. The weights of the graphs are defined as $w_{i,j}=\norm{X_i - X_j}^{-1}$ (same applies for $Y$), which is consistent with the computation of the directional derivatives. An example of this graph can be observed in Figure~\ref{im:matching}. Note that this graph does not need to be fully connected.

%%%%%%%%
\paragraph{Transport map computation}

The regularized transport map $T$ between the sub-sampled data $(X,Y)$ is computed as
\eq{
	T(X_i)=\left(\diag(\Sig \U)^{-1}\Sig Y\right)_i \qforq i={1,\ldots,N}
}
where $\Sig$ is a solution of~\eqref{eq-symm-reg-energy}.

%%%%%%%%
\paragraph{Transport map up-sampling}

The transport map $T$ is extended to the whole space using a nearest neighbor interpolation
% to every point the transport map of the nearest point in $X$:
\eql{\label{eq-upsample}
	% (\tilde X^0)_{i} =
	\foralls x \in \RR^d, \quad
	T^0(x) = T(X_{i(x)}) + x - X_{i(x)},
	\qwhereq
	i(x) = \uargmin{1 \leq i \leq N} \norm{ x - X_i }.
}
Note that this interpolation scheme contains an additive term $x-X_{i(x)}$. This corresponds to adding back the quantization error (due to the K-means sub-sampling) to the nearest neighbors interpolation, which helps to restore small scale textural details, and improves the visual quality of the result.  Note also that one could use smoother interpolation schemes (for instance linear or natural neighbors) to help reducing the quantization error. These interpolations however require to compute a Delaunay tetrahedrization of the 3-D color space, which leads to a significant computational overhead.

The resulting transport computed in~\eqref{eq-upsample} can now be  applied to the input image $X^0$ to obtain the new pixel values $(\tilde X^0)_{i} = T^0(X^0_i)$.

\begin{figure}[ht]
\centering
\begin{tabular}{@{}c@{\hspace{1mm}}c}
\includegraphics[width=.4\linewidth]{barycenter/flowers/flowers-1.jpg} &
\fbox{\includegraphics[width=.55\linewidth,height=5.1cm]{coloredgraph}} \\
(a) & (b)\vspace{-0.3cm}
\end{tabular}
\caption{(a) Flower image ; (b) its empirical distribution projected on the Red-Blue plane. The line segments represent the edges $E_X$ of the $n$-nearest neighbor graph computed with $n=4$. \vspace{-0.1cm}}
\label{im:matching}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Parameters Selection}

The color transfer method has several parameters. We explain below their influence and how they were selected in the numerical examples shown in the following section.

%%%%%%%%
\paragraph{Parameters $(p,q)$: Sobolev vs. total variation}

All the numerical results reported in the remaining of the paper are computed using the total variation (TV) regularization, which corresponds to using the parameters $(p,q)=(1,1)$. We found empirically on the color transfer application that both methods produce visually similar results. The main difference is that the resulting optimal $\Sig$ is somehow sparser with the total variation prior, i.e. the resulting weights are more concentrated on a few entries.

%%%%
\paragraph{Relaxation parameter $\kappa$ }

The impact of this parameter has already been discussed in Section~\ref{subsec-relaxed-transport}~on synthetic 2-D examples.

%%%%
\paragraph{Regularization parameters $(\la_X,\la_Y)$}

%While the impact of these parameters has already been explored in Section~\ref{sec:regsymme} on 2-D examples, we further illustrate it for color transfer of synthetic images.
Figure~\ref{im:synth} shows an example of color transfer between two synthetic images $X^0$ and $Y^0$ shown in Figure~\ref{im:synth}~(a). We apply Algorithm~\ref{alg-rot} to obtain the image ${\tilde X^0}$ with a color palette close to $Y^0$, but with the geometry of the original $X^0$. We now study the influence of the parameters $\la_X$ and $\la_Y$. Figure~\ref{exlk} shows a 2-D projection in the Red-Green plane of $X$ and $Y$, displayed using respectively red and blue, and $\tilde X$ in green. As already pointed out in Section~\ref{secalgosymm},  a low value of $\la_X$ and $\la_Y$ (zero for the first column) tends to match the points in X to the closest point in Y. This behavior can be observed in the map of the column (b). Many points in the big cluster of $X$ are mapped to very few points in the small cluster of $Y$, which corresponds in the images to mapping many red values of $X^0$ to very few brown values in $Y^0$. The consequence is that the color resolution of $\tilde X$ is reduced, the brown area of Figure~\ref{im:synth}~(b) is flat, unlike the original brown values in $Y$. As we increase the value of $\la_X$ and $\la_Y$, the mapping spreads within the small cluster of $Y$ in Figure~\ref{exlk}(b) and we gain color resolution, as can be observed in Figure~\ref{im:synth}~(c). It should be observed that, even in this case, the contrast rendition of $\tilde{X}$ and $Y$ are still not the same. Comparing the convex hull size of the color histograms $\mu_{Y^0}$, $\mu_{{\tilde X}^0}$, we see that the convex hull of $\mu_{{\tilde X}^0}$ is smaller, thus the contrast is reduced.

On the other hand, if we increase too much the values of $\la_X$ and $\la_Y$, many points in $X$ get matched to the big cluster in $Y$ in Figure~\ref{exlk}~(c) which leads to a single dominant color in the final image $\tilde X^0$, in Figure~\ref{im:synth}~(d).

\newlength{\mylenX}\settowidth{\mylenX}{\includegraphics[width=1.75cm]{../images/syntheticexamples/X} } % Widest element
\newcommand{\sidecapX}[1]{ {\begin{sideways}\parbox{1.65cm}{\centering #1}\end{sideways}} }

\begin{figure}[ht]
\centering
\begin{tabular}{@{}c@{}}
\sidecapX{\scriptsize $X^0$} \includegraphics[width=.13\linewidth]{../images/syntheticexamples/X} \\
\sidecapX{\scriptsize $Y^0$} \includegraphics[width=.13\linewidth]{../images/syntheticexamples/Y} \\
\hspace{0.4cm} (a)\vspace{-0.25cm}
\end{tabular}
\begin{tabular}{@{}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{}}
\includegraphics[width=.27\linewidth]{../images/syntheticexamples/symmetricsyntheticinv_l0_KX8_KY8_nn4} &
\includegraphics[width=.27\linewidth]{../images/syntheticexamples/symmetricsyntheticinv_l0001_KX8_KY8_nn4} &
\includegraphics[width=.27\linewidth]{../images/syntheticexamples/symmetricsyntheticinv_l10_KX8_KY8_nn4} \\
(b) &  (c) & (d) \vspace{-0.25cm}
\end{tabular}
\caption{Effect of changing the parameters $\la_X$ and $\la_Y$ of the relaxed and regularized OT formulation presented in section~\ref{sec:regsymme}, using parameters $\kappa=(0.1,8,0.1,8)$.
	{(a)} original input images,
	{(b)} relaxed OT, $\la_X=\la_Y=0$;
	{(c)} $\la_X=\la_Y=0.001$;
	{(d)} $\la_X=\la_Y=10$.
	Each of these mappings can be observed in Figure~\ref{exlk}.\vspace{-0.1cm}}
  \label{im:synth}
\end{figure}

\paragraph{Number $N$ of quantization points}

In our numerical experiments, we have used $N=400$, which corresponds to a tradeoff between computation time and visual quality. In practice, of course, this number should be set as a function of the complexity (e.g. number of modes) of the input color distributions, but we found that $N=400$ was sufficient for all considered images in our tests.  Figure~\ref{changeN} shows some experiments to support this claim and to show that even setting $N$ to a smaller value did not affect the results much.

 % For images with a small number of homogeneous regions, where we could do with less clusters, the points just cluster around a single value. Then, for images with a huge amount of regions, it seems that 400 is enough to summarize the information, of course this value may depend on the size of the image.

 % Given the moderate impact of a change in this parameter, we decided to not introduce these experiments in the paper.

\newcommand{\myfigb}[1]{\includegraphics[width=.25\linewidth,height=.25\linewidth]{./images/influence-quantiz/#1}}

\begin{figure}[h!]
\centering
\begin{tabular}{@{}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{}}
\myfigb{parrot_1.jpg} &
\myfigb{parrotX_N64} &
\myfigb{parrotX_N144} &
\myfigb{parrotX_N400} \\
\myfigb{parrot_2.jpg} &
\myfigb{parrotY_N64} &
\myfigb{parrotY_N144} &
\myfigb{parrotY_N400} \\
Original & $N=64$ & $N=144$ & $N=400$
\end{tabular}
\vspace{0.1in}
\begin{tabular}{@{}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{}}
\myfigb{flowers-1} &
\myfigb{flowersX_N64} &
\myfigb{flowersX_N144} &
\myfigb{flowersX_N400} \\
\myfigb{flowers-2} &
\myfigb{flowersY_N64} &
\myfigb{flowersY_N144} &
\myfigb{flowersY_N400} \\
Original & $N=64$ & $N=144$ & $N=400$\vspace{-0.1cm}
\end{tabular}
\caption{Results obtained computing the relaxed-regularized OT between the original images, changing the value of $N$. The parameters were set as $n=4$, $M=N$, for all experiments. Then, for the first and second row $\kappa=(0.1,1 ,0.1,1)$, and $\la_X=\la_Y=0.001$. For the second experiment that involves the third and forth row, $\kappa=(0.1,1.1 ,0.1,1.1)$ and $\la_X=\la_Y=9 \cdot 10^{-4}$. Each column corresponds to a single experiment which differ on the value of $N$.\vspace{-0.1cm}}

%\caption{Results obtained computing the relaxed-regularized OT between the image in the first and third row ($X$) with the image in the second and forth row ($Y$), respectively. The parameters were set as $n=4$, $M=N$, for all experiments. Then, for the first and second row $\kappa=(0.1,1 ,0.1,1)$, and $\la_X=\la_Y=0.001$. For the second experiment that involves the third and forth row, $\kappa=(0.1,1.1 ,0.1,1.1)$ and $\la_X=\la_Y=9 \cdot 10^{-4}$. Each column corresponds to a single experiment which differ on the value of $N$.\vspace{-0.1cm}}
\label{changeN}
\end{figure}

%%%%
\paragraph{Number $n$ of nearest neighbors}

As for the other parameters of the method, the value of $n$ is set in order to obtain the best visual result.  This parameter is related to the connectivity of the graphs $\mathcal{G}_X$ and $\mathcal{G}_Y$. There are two behaviors that we need to avoid when setting it:
\begin{itemize}
\item Too small $n$: The graph has very few edges. So, an area that is almost homogeneous in the spatial domain could end up being disconnected. In this case, this homogeneous area in the original image could be transported to different locations, generating color artifacts. In practice, this situation has been observed only for very small values of $n$. An example can be seen in  the first row of Figure~\ref{imn}, for $n=1$ and $n=2$ there is an area in the background that after the color matching gets two different colors. For greater values of $n$ this artifact disappears.
\item Too large  $n$: The graph has a lot of edges and all the points are very interconnected, so the regularization term plays a very strong role by maintaining the structure of the graph in the original cloud of points, which reduces the flexibility of the map. This effect is quite difficult to observe in the resulting images.
\end{itemize}
\newcommand{\myfig}[1]{\includegraphics[width=.25\linewidth]{./images/influence-nn/#1}}

\begin{figure}[h]
\centering
\begin{tabular}{@{}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{\hspace{1mm}}c@{}}
\myfig{parrotX_nnx1} &
\myfig{parrotX_nnx2} &
\myfig{parrotX_nnx4} &
\myfig{parrotX_nnx10} \\
\myfig{parrotY_nnx1} &
\myfig{parrotY_nnx2} &
\myfig{parrotY_nnx4} &
\myfig{parrotY_nnx10} \\
\myfig{flowersX_nnx1} &
\myfig{flowersX_nnx2} &
\myfig{flowersX_nnx4} &
\myfig{flowersX_nnx10} \\
\myfig{flowersY_nnx1} &
\myfig{flowersY_nnx2} &
\myfig{flowersY_nnx4} &
\myfig{flowersY_nnx10} \\
$n=1$ & $n=2$ & $n=4$ & $n=10$
\end{tabular}
\caption{Results obtained by changing the $n$ parameter. The original images are presented in Fig.~\ref{changeN}}
\label{imn}
\end{figure}

As pointed out before, in our experiments we saw that setting $n=4$ avoids situations related to both cases. In Fig.~\ref{imn}, we can see how the final image evolves with a change in the value of the parameter $n$. Note that the only noticeable difference between the images can be observed on the results obtained with very small $n$ values ($n=1,2$) where homogeneous areas get divided. %Above a certain value, the results do not change.

%%%%

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Results}

Figure~\ref{star} shows some results on natural images and compare them with the methods of Piti\'e et al.~\cite{Pitie07}, Rabin et al.~\cite{Rabin_ip11}\footnote{We thank Julien Rabin for providing us these results} and Papadakis et. al~\cite{Papadakis_ip11}. Note that the method of~\cite{Pitie07} includes a post-processing step to reduce the color artifacts, but that it is applied here without the post-processing (it thus corresponds to the application of an approximate optimal transport).
 The goal of the experiment is to transfer the color palette of the images in the second row to the image on the first row.  Note that the methods in the state of the art introduce color artifacts (in the first column there is violet outside the flower, and in the second column the wheat is blueish), which can be avoided with the proposed method by an appropriate choice of $\la_X$, $\la_Y$ and $\kappa$. These results were obtained setting $N=400$ and constructing the graph as a $4$-nearest neighbor graph. By column, the values of $\la_X=\la_Y$ are $9\times 10^{-4}, 5\times 10^{-4}$, and $10^{-3}$ and $\kappa$ was set to (0.1,1.1,0.1,1.1), (0.1,1.3,0.1,1.3), and (0.1,1,0.1,1), respectively.

\newlength{\mylen}\settowidth{\mylen}{\includegraphics[width=3.8cm,height=3.3cm]{star/fleur_1}} % Widest element

\newcommand{\sidecap}[1]{{\begin{sideways}\parbox{3.3cm}{\centering #1}\end{sideways}} \hspace{-4mm}}

\newcommand{\myimg}[1]{\includegraphics[width=.25\linewidth,height=.22\linewidth]{#1}}

\begin{figure}[!ht]
\centering
\begin{tabular}{@{}@{}cc@{\hspace{1mm}}c@{\hspace{1mm}}c@{}}
\sidecap{ Original $X^0$ }  &
\myimg{star/fleur_1} &
\myimg{star/wheat_1.jpg} &
\myimg{star/parrot_1.jpg} \vspace{-0.45cm}\\
\sidecap{ Original $Y^0$ } &
\myimg{star/fleur_2} &
\myimg{star/wheat_2.jpg} &
\myimg{star/parrot_2.jpg}\vspace{-0.45cm} \\
\sidecap{ Piti\'e et al.~\cite{Pitie07} } &
\myimg{star/fleur_pitie} &
\myimg{star/wheat_pitie} &
\myimg{star/parrot_pitie}\vspace{-0.45cm} \\
\sidecap{ Rabin et al.~\cite{Rabin_ip11} } &
\myimg{star/fleur_Rabin} &
\myimg{star/wheat_Rabin} &
\myimg{star/parrot_Rabin}\vspace{-0.45cm} \\
\sidecap{ Papadakis et al.~\cite{Papadakis_ip11} } &
\myimg{star/fleur_papadakis} &
\myimg{star/wheat_papadakis}  &
\myimg{star/parrot_papadakis} \vspace{-0.45cm}\\
\sidecap{ Proposed method } &
\myimg{symmetric/symmetricfleur_l00009_KX11_KY11_nn4.eps} &
\myimg{symmetric/symmetricwheat_l00005_KX13_KY13_nn4.eps} &
\myimg{symmetric/symmetricparrot_l0001_KX1_KY1_nn4.eps} \\
 & (a) & (b) &(c)\vspace{-0.25cm}
\end{tabular}
\caption{Comparison between the results obtained with our method and with the methods of~\cite{Pitie07}, ~\cite{Rabin_ip11}  and~\cite{Papadakis_ip11} for image color transfer. Note how the proposed method is able to generate results without color artifacts for example, in \textbf{(a)} the violet color of the flower is not spread outside the flower, in \textbf{(b)} the wheat does not become bluish and in \textbf{(c)} the result does not enhance or colorize differently the flat areas of the background.\vspace{-0.25cm}}
\label{star}
\end{figure}


\section{Proof of ELBO lower bound}
\label{sec:nvmp-proof}

\begin{proof}
(This proof is very similar to that in \citet{Johnson2016}.)
In each step of mean-field variational inference,
we maximize the ELBO with respect to a single variational factor.
Consider a latent variable $\z_\n$.
Choosing a form for $\q(\z_\n)$
that does not capture the optimal $\q^*(\z_\n)$
will result
in a lower maximum than the maximizing
over all possible $\q(\z_\n)$ distributions.
We express this as
\begin{equation}
    \max_{\q(\z_\n)}\L[\q] \ge\q  \L(\eta_{\z_\n}^*)
\end{equation}
where
\[
\eta_{\z_\n}^* \triangleq \argmax_{\eta_{\z_\n}} \L[\q]
\]
is the optimal parameter of some chosen form of $\q(\z_\n)$.

In NVMP, we further modify this local optimization
by modifying the ELBO to include recognition networks
using the trick from \cite{Johnson2016}.
This results in a surrogate objective $\hat{\L}_\recparam[\q]$,
and maximizing this objective can do no better than maximizing the
parameterized objective $\L[\q]$.
\begin{equation}
    \max_{\q(\z_\n)}\L[\q] \ge\q  \L(\eta_{\z_\n}^*) \ge\q
    \L(\hat{\eta}_{\z_\n}^*)
\end{equation}
where
\[
\hat{\eta}_{\z_\n}^* \triangleq \argmax_{\eta_{\z_\n}} \hat{\L}_\recparam[\q]
\]

As each local optimization lower-bounds the ELBO,
the NVMP optimization holistically
lower-bounds the ELBO.

This bound is tight in that
if there exists a set of recognition
network weights $\recparam$ such that
\begin{equation}
\log \p(c \given \z_\n, \parents_c \not{\z_\n}) = \psi(c, \z_\n, \parents_{c} \not{\z_\n}; \recparam) \\
\end{equation}
we have
\begin{equation}
    \max_{\q(\z_\n)}\L[\q] \geq  \L(\eta_{\z_\n}^*) =
    \L(\hat{\eta}_{\z_\n}^*)
\end{equation}
\end{proof}

\section{Extension: stochastic variational inference}
\label{sec:nvmpsvi}
SVI \citep{Hoffman2013} treats local and global variational factors differently,
first performing VMP to optimize local
factors to completion. Note that these factors are only
optimized over mini-batches of data.
Global variational factors are then updated
with natural gradients $\tilde{\nabla}_{\globals}\L$ of the ELBO,
which take a simple form in conjugate exponential models. Specifically,
for the natural parameter for the global variables
$\eta_{\globals}^\q$,
the natural gradient update is
\begin{equation}
    \tilde{\nabla}_{\eta^\q_{\globals}} \L =  f_{\globals}\left(\{\vmpmessage{p}{\globals}\}_{p \in \pi_{\globals}}\right) + B \left(\sum_{c \in \children_{\globals}}\vmpmessage{c}{\globals}\right) - \eta^\q_{\globals}
\end{equation}
where $B$ is the number of minibatches in the dataset.

Importantly,
this natural gradient update natural gradient can be computed from VMP messages.
Thus, integrating SVI into NVMP requires no extra steps,
as NVMP simply modifies the messages.
NVMP can thus be treated as the inner-loop
in stochastic VMP, where natural gradients are computed
from NVMP messages when necessary. \footnote{This results in natural gradients computed over the NVMP surrogate objective.}

\section{Extension: amortized inference}
\label{sec:amortizeddetail}
NVMP produces approximates previously messages.
Thus, the role
of the recognition network is to assist in inference,
rather than directly amortizing inference.
This distinction is analogous to the difference
between the VAE and SVAE. In the VAE,
the recognition network directly outputs
the model parameters of the latent variables,
whereas in SVAE, the recognition network
outputs a message.
This difference plays a part in practice,
when we may desire a network that directly acts as a variational posterior,
avoiding the use of message passing in answering an inference query.

NVMP can be modified to include
recognition networks that directly output parameters.
Previously, where the optimal natural parameter for a factor $\q(\z_\n)$
was computed as a combination of messages
\begin{equation}
   \eta^\q_{\z_\n} = f_{\z_\n}\left(\{\vmpmessage{p}{\z_\n}\}_{p \in \parents_{\z_\n}}\right) + \sum_{c \in \children_{\z_\n}} \vmpmessage{c}{\z_\n}
\end{equation}
we replace the entire natural parameter with the output of a neural network
that takes children and coparent samples as input
\begin{equation}
   \eta^{\text{amort.}}_{\z_\n} = r^{\z_\n}_\recparam(\left\{c'\sim \q(c)\right\}_{c \in \children_{\z_\n}}, \{v' \sim \q(v)\}_{v \in \parents_c\not{\z_\n}})
\end{equation}
This produces a VAE-style algorithm, where the recognition networks
directly produce parameters and thus have utility outside of message passing.
However, we do lose the coordinate-ascent style of the algorithm, as
$\eta^{\text{amort.}}_{\z_\n}$ is not the optimal parameter
of a surrogate objective, but the natural parameter of a variational
distribution parameterized by $\recparam$.


\documentclass[letterpaper]{article}

\usepackage[utf8]{luainputenc}
\usepackage{fullpage}
\usepackage{nopageno}
\usepackage{amsmath}
\usepackage{amssymb}
\allowdisplaybreaks

\newcommand{\abs}[1]{\left\lvert #1 \right\rvert}

\begin{document}
\title{Notes}
\date{13 février, 2015}
\maketitle
\section*{weierstrass m-test}
if $f_{k}:S\to \mathbb{R}^n$ and for all $k$ there is $M_k$ with $||f_k||_\infty\le M_k$ then if $\sum\limits{M_k}$ converges then the series $\sum\limits{f_n}$ converges uniformly.

that is to say, if we can bound $f_n$ by  $M_n$ and the series of $M_n$ converges, then $f_n$ converges.
\subsection*{example}
if $\sum\limits_{n=1}^\infty{|a_n|}<\infty$ then $\sum\limits_{n=1}^\infty{a_n\cos(nx)}$ converges uniformly on $\mathbb{R}$.

$|a_n\cos(nx)|\le |a_n|$ for any  $x$. $a_n=M_n$\dots$\Box$

note that $\sum a_n\cos(nx)$ is continuous because $a_n\cos(nx)$ is continuous and $\sum a_n\cos(nx)$ converges uniformly.

\section*{theorem}
if $f_n(x)$ is continuous and $\sum\limits_{n=1}^\infty{f_n(x)}$ converges uniformly to $F(x)$ then $F(x)$ is continuous (the sums are continuous if $f_n$ is continuous)

\subsection*{example}
$\sum\limits_{n=1}^\infty{x^ne^{-nx}}$ converges uniformly  on $[0,A]$ for any $A>0$.

$|x^ne^{-nx}|\le |x^n|\le A^n$. $f'(x)=(nx^{n-1}e^{-nx})-nx^ne^{-nx}=ne^{-nx}x^{n-1}(1-x)=0$ so critical points are $0,1,A$.

$f''$ shows us that $1$ is concave down so it is a max. and so $x^ne^{-nx}\le e^{-n}$ on $[0,\infty)$.

so $\sum\limits{e^{-n}}=\sum\limits{\left(\frac{1}{e}\right)^n}$ which is geometric and so it converges to $\left(\frac{1}{1-\frac{1}{e}}\right)-1$ by m-test we converge uniformly on $[0,\infty)$
\end{document}