\chapter{On-policy control with approximation}

\section{Summary}

\subsection{Episodic Semi-gradient control}

The general formulation of episodic semi-gradient control equation~\ref{eq:general formula on-policy control with approximation} can be combined with one-step SARSA leading to equation~\ref{eq:episodic semi-gradient one step sarsa}. The algorithm itself will still need to use a $\epsilon$ greedy policy to allow exploration.

\begin{equation}
	w_{t+1} = w_t + \alpha \big[ U_t - \hat{q}(S_t, A_t, W_t) \big]
\label{eq:general formula on-policy control with approximation}
\end{equation}

\begin{equation}
w_{t+1} = w_t + \alpha \big[R_{t+1} + \gamma\hat{q}(S_{t+1}, A_{t+1},W_t) - \hat{q}(S_t, A_t, W_t) \big] \nabla \hat{q}(S_t, A_t, W_t)
\label{eq:episodic semi-gradient one step sarsa}
\end{equation}

\subsection{Semi-gradient n-step sarsa}
The n-step SARSA leads to update of equation~\ref{eq:semi-gradient n-step sarsa update}.

\begin{equation}
\begin{split}
G_{t:t+n} & = R_{t+1} + \gamma R_{t+2} + ... + \gamma^{n-1} R_{t+n} \\
w_{t+1} & = w_t + \alpha \big[G_{t:t+n} + \gamma \hat{q}(s_{t}, A_{t}, W_{t+n-1}) \big] \nabla \hat{q}(S_{t}, A_{t}, W_{t+n-1})
\end{split}
\label{eq:semi-gradient n-step sarsa update}
\end{equation}

\subsection{Average Reward: A new problem setting for continuing tasks}
The previous definitions of reward work great with episodic tasks, but turn out to be problematic with continuous tasks. Equation~\ref{eq:average reward definition}
introduces the \textbf{reward rate}, which represents the average amount you will have in a certain state.

\begin{equation}
\begin{split}
r(\pi) & = \lim\limits_{h \rightarrow \infty} \frac{1}{h}\sum_{t=1}^{h} \EX[R_t | S_0, A_{0:t-1} \sim \pi] \\
& = \lim_{t \rightarrow \infty} \EX[R_t | S_0, A_{0:t-1} \sim \pi] \\
& = \sum_s \mu_{\pi}(s)\sum_a \pi(a|s) \sum_{s', r}p(s', r| s, a)
\end{split}
\label{eq:average reward definition}
\end{equation}

The steady state distribution $\mu_{\pi}$, is a special case for which equation~\ref{eq:the steady state distribution} holds. The MDP must be \textbf{ergodic}, eg. the starting state and any early decisions made only have a short term effect. Otherwise the limit of equation~\ref{eq:average reward definition} is not guaranteed to exist. The policies that have a maximum $r(\pi)$ value are called the optimal policies.

\begin{equation}
\sum_s \mu_{\pi}(s)\sum_a \pi(a|s) p(s'| s, a)=\mu(s')
\label{eq:the steady state distribution}
\end{equation}

\begin{equation}
\mu(s) = \lim_{t \rightarrow \infty} \Pr\{S_t = s | A_{0:t-1} \sim \pi\}
\end{equation}

\begin{equation}
G_t = R_{t+1} - r(\pi) + R_{t+2} - r(\pi) + R_{t+3} - r(\pi) + ...
\label{eq:differential return}
\end{equation}

The differential return can be used to setup the values functions (equation~\ref{eq:difference value function update}).

\begin{equation}
\begin{split}
v_{\pi}(s) & = \sum_a \pi(a|s) \sum_{s',r}p(s',r|s, a)\big[ r - r(\pi) + v_{\pi}(s') \big] \\
q_{\pi}(s, a) & = \sum_{r, s'} p(s', r | s, a)\Big[ r - r(\pi) + \sum_{a'}\pi(a' | s')q_{\pi}(s', a') \Big] \\
v_*(s) & = \max_a \sum_{r, s'} p(s', r | s, a)\Big[ r - \max_{\pi}r(\pi) + v_*(s') \Big] \\
q_*(s, a) & = \sum_{r, s'} p(s', r | s, a)\Big[ r - \max_{\pi}r(\pi) + \max_{a'}q_{\pi}(s', a') \Big] \\
\end{split}
\label{eq:differential value functions}
\end{equation}

The TD errors then become equation~\ref{eq:TD errors differential form}. The weight update of semi-gradient SARA then becomes equation~\ref{eq:semi-gradient Sarsa differential form update rule}.

\begin{equation}
\begin{split}
\delta_t & = R_{t+1} - \bar{R_t} + \hat{v}(S_{t+1}, w_t) - \hat{v(S_t, w_t)} \\
\delta_t & = R_{t+1} - \bar{R_t} + \hat{q}(S_{t+1}, A_{t+1}, w_t) - \hat{q}(S_t, A_t, w_t) \\
\end{split}
\label{eq:TD errors differential form}
\end{equation}

\begin{equation}
w_{t+1} = w_t + \alpha \delta_t \nabla \hat{q}(S_t, A_t, w_t)
\label{eq:semi-gradient Sarsa differential form update rule}
\end{equation}

\subsection{Deprecating the discount setting}

The discount problem works really well in the tabular case, as states are easy to identify. With an approximated value function, it might be that we \textbf{don't know the explicit state}. We only have actions and rewards, in that case \textbf{the discounted return is a scaled version of the undiscounted}.

If we have a continuous process that gives a reward after every output, the \textbf{discounted} reward we would have the weights $1 + \gamma + \gamma^2 + \gamma^3 + ... = \frac{1}{1-\gamma}$. As we only have 1 explicit state. The total reward then becomes $\frac{r(\tau)}{1-\gamma}$, witch is a scaled version of the \textbf{undiscounted} $r(\tau)$.

A more formal proof can be found on page 254. The root cause of the difficulties with the discounted control setting is that with function approximation we have lost the policy improvement. This is further discussed in the chapter on \textbf{policy gradient descent}.

\subsection{Differential Semi-Gradient n-step SARSA}
The n-step differential TD error is defined in equation~\ref{eq:n-step differential TD-error}, algorithm~\ref{alg:differential semi-gradient n-step sarsa} illustrates the differential semi-gradient n-step Sarsa.

\begin{equation}
\begin{split}
G_{t:t+n} & = R_{t+1} - \bar{R}_{t+n-1} + ... + R_{t+n} + \bar{R}_{t+n-1} + \hat{q}(S_{t+n}, A_{t+n}, W_{t+n-1}) \\
\delta_t & = G_{t:t+n} - \hat{q}(S_t, A_t, W)
\end{split}
\label{eq:n-step differential TD-error}
\end{equation}

\begin{algorithm}
\begin{algorithmic}
	\State $W, S_0 \gets \text{Init}$
	\State $\pi(s)=\epsilon - \text{greedy}$
	\Loop \space \textbf{for each step t in the episode}
	\State $A_t = \pi(S_t)$
	\State $S_{t+1}, R_t = system(S_t, A_t)$
	\State $\tau = t -n + 1$ ($\tau$ is the time whose estimate is being updated)
	\If {$\tau \ge 0$}
		\State $\delta \gets \sum_{i=\tau+1}^{\tau + n} (R_i - \bar{R}) + \hat{q}(S_{\tau + n}, A_{r+n}, W) - \hat{q}(S_{\tau}, A_{\tau}, W)$
		\State $\bar{R} \gets \bar{R} + \beta \delta$
		\State $w \gets \alpha \delta \nabla \hat{q}(S_{\tau}, A_{\tau}, w)$
	\EndIf
	\EndLoop
\end{algorithmic}
\label{alg:differential semi-gradient n-step sarsa}
\caption{Differential semi-gradient n-step sarsa}\end{algorithm}
\section{Exercises}

\subsection{Exercise 10.1 page 248}
\textbf{We have no explicitly considered or given pseudo code for any Monte-Carlo methods in this chapter. what would they be like? Why is it reasonable not to give pseudo code for them? How would they perform on the Mountain Car Task.}

The monte carlo algorithm replaces $U_t$ with the actual return of an episode $G_t$ in equation~\ref{eq:general formula on-policy control with approximation}. This is by far the simplest way, and the algorithm is trivial.

\subsection{Exercise 10.2 page 248}
\textbf{Give pseudo code for semi-gradient one-step Expected Sarsa for control.}
\begin{algorithmic}
\State $W \gets \text{Init}$
\State $\pi(s)=\epsilon - \text{greedy}$
\Loop \space \textbf{for each episode}
	\State $S, A \gets S_{init}, \pi(S_{init})$
	\Loop \space \textbf{for each step in the episode}
		\State Take action $A$, observe $R$ and $S'$
		\State $A_{exp} \gets \EX_{\pi}[A' | S']$
		\State $W \gets W + \alpha \big[R + \gamma\hat{q}(S', A_{exp}, W) - \hat{q}(S, A, W) \big] \nabla \hat{q}(S, A, W)$
		\State $S, A \gets S', \pi(S')$
	\EndLoop
\EndLoop
\label{alg:approximated one step exptected SARSA}
\end{algorithmic}

\subsection{Exercise 10.3 page 248}
\textbf{Why do the results show in Figure 10.4 (book page 248) have higher standard errors at large n than at small n?}
The standard error $SE=\frac{\sigma}{\sqrt{n}}$ increase as $n$ goes up. As the policy considers more actions. The lower graph of figure 10.4 in the book page 248 has smoother lines on the lower n as the variance on Q is lower.

\subsection{Exercise 10.4 page 250}
\textbf{Give pseudocode for a differential version of semi-gradient Q-learning}

\begin{algorithmic}
\State $W \gets \text{Init}$
\State $\pi(s)=\epsilon - \text{greedy}$
\Loop \space \textbf{for each episode}
\State $S \gets S_{init}$
\State $\bar{R} \gets 0$
\Loop \space \textbf{for each step in the episode}
\State $A = \pi(S')$
\State Take action $A$, observe $R$ and $S'$
\State $\delta = R -\bar{R} + \max_a\hat{q}(S', a, W) - \hat{q}(S, A, W)$
\State $W \gets W + \alpha \delta \nabla \hat{q}(S, A, W)$
\State $S \gets S'$
\State $\bar{R} = \bar{R} + \beta \delta$
\EndLoop

\EndLoop
\end{algorithmic}

\subsection{Exercise 10.5 page 250}
 \textbf{What equations are needed (beyond equation~10.10 from the book page 250) to specify the differential version of TD(0)}

 Equation 10.10 from the book $\delta_t = R_{t+1} - \bar{R_t} + \hat{v}(S_{t+1}, w_t) - \hat{v(S_t, w_t)} $, can be combined with the update rule $w_{t+1} = w_t + \alpha \delta_t \nabla \hat{q}(S_t, A_t, w_t)$ to form the full TD(0) algorithm.

\subsection{Exercise 10.6 page 251}

\begin{itemize}
	\item MDP rewards +1,0,+1,0,... for any policy
	\item ergodicity is violated these is no stationary distribution $\mu$
	\item Average reward is well defined, what is it? \textbf{0.5 times the number of steps taken, as the average keeps going up}.
	\item As the average return is not static, the limit of the value function is not well defined. An alternative value function (equation~\ref{eq: ex10.6 alternative value function}) can be used.
	\item A, B are both MDP's with +1,0 returns as before. But A starts with 1 while B starts with zero. What are the differential value s of states A and B.
\end{itemize}

\begin{equation}
v_{\pi} = \lim_{\gamma \rightarrow 1} \lim_{h \rightarrow \infty} \sum_{t=0}^h \gamma^t \big( \EX[R_{t+1} | S_0=s] - r(\tau) \big)
\label{eq: ex10.6 alternative value function}
\end{equation}

If $A = [1, 0, 1, 0, 1, 0 ]$ then $v(A)=(1-0.5)+(0-0.5)*\gamma+(1-0.5)*\gamma^2...$ if $\gamma=1$ then $v=0.5 - 0.5 + 0.5 - 0.5 + 0.5 - 0.5$. This is the geometric series of equation~\ref{eq: ex10.6 alternative value function}, with $a = 0.5$ and $r = -1$. Resulting in $v=\frac{0.5}{1-(-1)}=\frac{1}{4}$

\begin{equation}
\sum_{k=0}^\infty ar^k = \frac{a}{1-r}
\label{eq:ex 10.6 geometric series}
\end{equation}

If $B = [0, 1, 0, 1, 0, 1, ... ]$ and $\gamma=1$ then $V(B) = (0-0.5) + (1-0.5) + (0-0.5)... = -0.5 + 0.5 -0.5 + 0.5$ notice that $V(A) = -V(B)$. So $V(B) = -\frac{1}{4}$.

\subsection{Exercise 10.7 page 251}
The average value is $1/3=r(\tau)$.

\begin{figure}
	\centering
	\begin{tikzpicture}

	\node at (2, 0) (a) {A};
	\node at (4, -3) (b) {B};
	\node at (0, -3) (c) {C};

	\draw [->, auto, bend left] (a) to node {+0} (b);
	\draw [->, auto, bend left] (b) to node {+0} (c);
	\draw [->, auto, bend left] (c) to node {+1} (a);
	\end{tikzpicture}
	\caption{MPD ex 10.7}
	\label{fig:mdp ex 10.7}
\end{figure}

$\gamma=1$ We get the following series: $v(A) = -\frac{1}{3} - \frac{1}{3} + \frac{2}{3} + ...$. We could split it up into 3 series, but notice that if we take $-\frac{1}{3} - \frac{1}{3}$ together into $-\frac{2}{3}$. It becomes one geometric series: $-\frac{2}{3} + \frac{2}{3} - ...$, with $a=-\frac{2}{3}$ and $r=\gamma=1$. So $v(A)=-\frac{1}{3}$

\begin{equation}
\begin{split}
v_{\pi}(B) & = \lim_{\gamma \rightarrow 1} \lim_{h \rightarrow \infty} \sum_{t=0}^h \gamma^t \big( \EX[R_{t+1} | S_0=B] - r(\tau) \big)\\
& = \lim_{\gamma \rightarrow 1} \lim_{h \rightarrow \infty} -\sum_{t=0}^h \gamma^{3t} \frac{1}{3} + \sum_{t=0}^h \gamma^{3t+1} \frac{2}{3} -\sum_{t=0}^h \gamma^{3t+2} \frac{1}{3} \\
& = \lim_{\gamma \rightarrow 1} \lim_{h \rightarrow \infty}  \sum_{t=0}^h \gamma^{3t} \Big( -\frac{1}{3} + \frac{2}{3}\gamma - \frac{1}{3}\gamma^2 \Big)\\
& =  0
\end{split}
\end{equation}

\begin{equation}
\begin{split}
v_{\pi}(C) & = \lim_{\gamma \rightarrow 1} \lim_{h \rightarrow \infty} \sum_{t=0}^h \gamma^t \big( \EX[R_{t+1} | S_0=C] - r(\tau) \big)\\
& = \lim_{\gamma \rightarrow 1} \lim_{h \rightarrow \infty}\sum_{t=0}^h \gamma^{3t} \frac{2}{3} - \sum_{t=0}^h \gamma^{3t+1} \frac{1}{3} -\sum_{t=0}^h \gamma^{3t+2} \frac{1}{3} \\
& = \lim_{\gamma \rightarrow 1} \lim_{h \rightarrow \infty}  \sum_{t=0}^h \gamma^{3t} \Big( -\frac{1}{3} + \frac{2}{3}\gamma - \frac{1}{3}\gamma^2 \Big)\\
& = \frac{1}{3}
\end{split}
\end{equation}

So: $V(A) = -1/3$, $V(B)=0$ and $V(C)=1/3$

\subsection{Exercise 10.8 page 251}

$\bar{R}$ is in steady state and fixed to $\frac{1}{3}$

\begin{table}
\begin{center}
	\begin{tabular}{c | c}
		Transition & $\delta=R_{t+1} - \bar{R}$ \\
		\hline
		A, B & $0 -\frac{1}{3}=-\frac{1}{3}$ \\
		B, C & $0 -\frac{1}{3}=-\frac{1}{3}$ \\
		C, A & $1 -\frac{1}{3}= \frac{2}{3}$ \\
	\end{tabular}
\end{center}
\caption{$\delta$ using a simple error}
\end{table}

Using equation $10.10$ from the book page 250: $\delta = R_{t+1} - \bar{R} + \hat{v}(S_{t+1}, W) - \hat{v}(S_t, W)$.

\begin{table}[H]
\begin{center}
	\begin{tabular}{c | c}
		$X$ & $V(X)$ \\
		\hline
		A & $-\frac{1}{3}$ \\
		B & $0$ \\
		C & $ \frac{1}{3}$ \\
	\end{tabular}
\end{center}
\label{tab:value function evaluated in previous exercise}
\caption{value function calculated in previous exercise}
\end{table}

\begin{table}[H]
\begin{center}
	\begin{tabular}{c | c}
		Transition & $\delta = R_{t+1} - \bar{R} + \hat{v}(S_{t+1}, W) - \hat{v}(S_t, W)$ \\
		\hline
		A, B & $0 - \frac{1}{3} + 0 + \frac{1}{3} = 0$ \\
		B, C & $0 - \frac{1}{3} + \frac{1}{3} - 0 = 0$ \\
		C, A & $1 - \frac{1}{3} + \frac{1}{3} + \frac{1}{3}=0$ \\
	\end{tabular}
	\caption{$\delta$ using the differential value function update}
\end{center}
\end{table}

The update using the $\delta$ with the value function has a stable update, as the estimation of the value function is in steady state.

\subsection{Exercise 10.9 page 255}
Page 35 of the hand describes the \textbf{exponential recency-weighted average without initial bias}, equation~\ref{eq:ex 10.9 Exponential recency-weighted average without initial bias}. Using this $\beta$ instead of the static one will get rid of the bias.

\begin{equation}
\begin{split}
\beta & = \frac{\alpha}{\bar{o}_n} \\
\bar{o}_n & = \bar{o}_{n-1} + \alpha (1-\bar{o}_{n-1})
\end{split}
\label{eq:ex 10.9 Exponential recency-weighted average without initial bias}
\end{equation}


\section{The \removecopyiii algorithm (final contract)}
\label{sec:removecopyiii}

In this section we extend the contracts of \specref{removecopy}
and \specref{removecopyii} by introducing a
logic function, which describes the relationship between the elements of
input range \inl{a[0..n-1]} and the output range \inl{b[0..\\result-1]}.
Note that we have shown in the previous section that
\inl{\\result} equals \inl{CountNotEqual(a, n, v)}.

\subsection{A closer look on the properties of \removecopy}
\label{sec:formal-view-remove}

Figure~\ref{fig:removecopy-trip} shows a modified version of the
Figure~\ref{fig:removecopy}. We left out the indices of values that were not copied into the
target array. Furthermore we have added a dashed arrow which points to the
index that corresponds to the \emph{one past the end} location of the input and
output range.

\begin{figure}[hbt]
\begin{center}
\includegraphics[width=0.70\textwidth]{Figures/remove_copy_partition.pdf}
\end{center}
\caption{\label{fig:removecopy-trip} Partitioning the input of \removecopy}
\end{figure}
\FloatBarrier

These arrows between the indices of the array~\inl{b} and array~\inl{a} define
the following sequence~$p$ of seven indices. The index of the \emph{one past the end} is underlined.
$p = (1, 2, 5, 7, 8, 10, \underline{11})$

%\clearpage

More generally, we refer to the sequence~$p$ as \emph{partitioning sequence} of \removecopy for the array \inl{a[0..n-1]}.
For the \textbf{length of a partitioning sequence} $m$ we get the following \textbf{strictly monotone increasing} sequence:
\begin{align}
  \label{eq:remove-monotone}
  0 &\leq  p_0 < ... < p_{m} = n \\
  \intertext{and the open index intervals}
  %\label{eq:remove-interval}
  \nonumber
  (p_i,&p_{i+1}) && \forall i: 0 \leq i < m\\
  \intertext{mark \textbf{consecutive ranges} of the value \inl{v}
  in the source array, that is,}
  \label{eq:between}
  a[k] &= v &&\forall k: p_i < k < p_{i+1}\\
  \intertext{Additionally, the half open interval}
  \nonumber
  [0,&p_{0})\\
  \intertext{also marks another \textbf{consecutive range} of the value \inl{v} in the
  source array:}
  \label{eq:beginning}
  a[k] &= v &&\forall k: 0 \leq k < p_{0}\\
  \intertext{Another observation is that}
  \label{eq:a_nv}
  a[p_i] &\neq v &&\forall i: 0 \leq i < m\\
  \intertext{holds. Finally, we have}
  \label{eq:b_eqa}
  a[p_i] &= b[i] &&\forall i: 0 \leq i < m\\
  \intertext{which, together with the inequality~\eqref{eq:a_nv}
  states, that the target does not contain the value~\inl{v}}
  \nonumber
  %\label{eq:final}
  b[i] &\neq v &&\forall i:0 \leq i < m
\end{align}

\subsection{More lemmas on \CountNotEqual}

Our formalization the properties of \S\ref{sec:formal-view-remove}
relies on the logic function \logicref{CountNotEqual}.
We also rely on the logic function \logicref{FindNotEqual} and
the lemmas of \logicref{CountFindNotEqual} in the following listing that provide more
facts about \CountNotEqual and \FindNotEqual.

\input{Listings/CountFindNotEqual.acsl.tex}

\clearpage

\subsection{Formalizing the properties of the partitions}

The function~\RemovePartition, whose axiomatic definition is given in
Listings~\ref{logic:RemovePartition-1} and~\ref{logic:RemovePartition-2}
defines the partitioning sequence~$p$ from \S\ref{sec:formal-view-remove}.

\begin{logic}[hbt]
\begin{minipage}{\textwidth}
\lstinputlisting[linerange={1-53}, style=acsl-block, frame=single]{Source/RemovePartition.acsl}
\end{minipage}
\caption{\Label{logic:RemovePartition-1} The logic function \RemovePartition (1)}
\input{Listings/RemovePartition.acsl.labels.tex}
\input{Listings/RemovePartition.acsl.index.tex}
\end{logic}

\FloatBarrier

Before we begin to relate the various lemmas
to the formulas from \S\ref{sec:formal-view-remove} we want to remind
the reader that logic functions (and predicates) must be total that is they must
be defined for all possible argument values.

\begin{logic}[hbt]
\begin{minipage}{\textwidth}
\lstinputlisting[linerange={54-103}, style=acsl-block, frame=single]{Source/RemovePartition.acsl}
\end{minipage}
\caption{\Label{logic:RemovePartition-2}The logic function \RemovePartition (2)}
\end{logic}

\FloatBarrier

The lemmas for \RemovePartition are related to the properties of
\S\ref{sec:formal-view-remove} in the following way.

\begin{itemize}
  \item Property \eqref{eq:remove-monotone} is expressed by the lemmas
  \RemovePartitionEmpty, \RemovePartitionLeft
  \RemovePartitionRight, and \RemovePartitionStrictlyWeakIncreasing

  \item Properties~\eqref{eq:between} and~\eqref{eq:beginning}
  are described by lemmas \RemovePartitionSegment.

  \item Property~\eqref{eq:a_nv} is expressed by lemma \RemovePartitionNotEqual.

  \item Property~\eqref{eq:b_eqa} is formulated using the predicate \logicref{Remove}.
\end{itemize}

\clearpage

We would like to point out lemma \RemovePartitionCore which subsumes
the statements of the subsequent lemmas \RemovePartitionUpper,
\RemovePartitionNotEqual,\\
and \RemovePartitionCount.
While these three lemmas add nothing new
we have kept them because they correspond directly to individual properties
of \S\ref{sec:formal-view-remove}.
The question may arise why there is the lemma \RemovePartitionCore in the first place.
The answer is that we found the individual properties so intertwined
that we were not able to verify them separately but only their joint embodiment.

\subsection{The predicate \Remove}

The predicate \logicref{Remove} primarily serves
in order to improve the readability of our specification \specref{removecopyiii}.
As mentioned before this predicate encapsulates the Property~\eqref{eq:b_eqa}
from \S\ref{sec:formal-view-remove}.
Note that \logicref{Remove} also contains an overloaded version of
\Remove which will be used for the specification of the \emph{in-place}
variant \specref{remove} of \removecopy.

\input{Listings/Remove.acsl.tex}

\clearpage

\subsection{Formal specification of \removecopyiii}

The following listing shows the formal specification of \specref{removecopy}.
The additional postcondition \inl{remove} makes use of the predicate \logicref{Remove}
which we have just described.
Furthermore, we have again the postcondition~\inl{unchanged} which states that
the source array~\inl{a[0..n-1]} does not change.

\input{Listings/remove_copy3.h.tex}

\subsection{Implementation of \removecopyiii}
\label{sec:removecopyiii:impl}

We discuss now some aspects of the implementation of \implref{removecopyiii}.
We introduce the loop invariant~\inl{mapping}.
This invariant states that the variable~\inl{i} will
always be smaller or equal to the result of \inl{RemovePartition(a, n, v, k)}.
We also add the assertion~\inl{mapping} to our implementation as stepping stone
for the provers to verify the correctness of this loop invariant.

Somewhat surprisingly, in order to reduce excessive verification times we had to add
an else-branch to our implementation that besides the assertion \inl{unchanged}
is empty.

Regarding the assertion \inl{update}, one might wonder why we do not simply write
\inl{\\at(a[i], Pre)}.
However, this expression would be wrong because the index~\inl{i}
would then be interpreted as
\inl{\\at(i,Pre)} which doesn't makes sense for a local variable.
\wpframac consequently rejects this expression with the following error message.

\begin{small}
\begin{verbatim}
       Warning: unbound logic variable i. Ignoring code annotation
\end{verbatim}
\end{small}

\clearpage

We could explicitly refer to the current value of~\inl{i} by using
the subexpression \inl{\\at(i,Here)} inside the assertion~\inl{update}.
We felt, however, tow introduce the predicate \logicref{At}
to simplify the comparison of array elements in programme states
where the particular index variable isn't visible.

\input{Listings/At.acsl.tex}

The second argument \At is interpreted at the programme point here it appears,
that is, \inl{Here}.
Using this auxiliary logic function the assertion \inl{update}
is arguably more readable.

\input{Listings/remove_copy3.c.tex}

\clearpage


\documentclass[../report/main.tex]{subfiles}

\begin{document}

% The asterix after \subsection disables section numbering
\subsection*{Problem 1 Part B}
Determine the number of refrigerators to be shipped from plants to warehouses, and then warehouses to retailers to minimize the cost.  \textcolor{red}{For part B warehouse 2 is closed along with all associated routes.  Changes to the problem statement or solution relative to Problem 1 Part A are highlighted in red.}

\subsection*{Solution}
\textcolor{red}{There is no solution when warehouse 2 is closed.  The following is the error message that is returned by Matlab function linprog():\\\\Exiting: One or more of the residuals, duality gap, or total relative error has grown 100000 times greater than its minimum value so far:\\\indent the primal appears to be infeasible (and the dual unbounded).\\ \indent(The dual residual $<$ TolFun=1.00e-08.)\\\\So Matlab tells us that there is no feasible solution.  Why is that?  If you look back at the network diagram and the supply and demand tables, you'll note that with warehouse 2 out of commission, retailers 5, 6, and 7 can only receive shipments from warehouse 3 and their total demand is 450 units.  At the same time, warehouse 3 can only receive shipments from plant 3 and 4.  The total supply capacity of those plants is only 400 units.  Therefore, warehouse 3 gets 50 less units from plants 3 and 4 than are demanded from retailers 5, 6, and 7.}
\subsection*{Linear Program Formulation}
\begin{enumerate}[1.]
	\item Overall idea of problem
	\begin{itemize}
		\item Refrigerators moving from $n=4$ plants to $q=\textcolor{red}{2}$ warehouses to $m=7$ retailers.
		\item Not all plants deliver to all warehouses.
		\item Not all warehouses deliver to all retailers.
		\item Costs of shipping from plants to warehouses vary by pair.
		\item Costs of shipping from warehouses to retailers vary by pair.
		\item Each plant has a capacity in terms of number of refrigerators it can supply.
		\item Each retailer has a capacity in terms of number of refrigerators it demands.
		\item\textcolor{red}{Warehouse 2 has closed and all associate routes have been eliminated.}
	\end{itemize}
	\item What is the goal?  What are you trying to achieve?
	\begin{itemize}
		\item Unchanged from part A.
	\end{itemize}
	\item Identify variables
	\begin{itemize}
		\item Unchanged from part A.
	\end{itemize}
	\item Identify constraints
	\begin{itemize}
		\item All constraints from part A remain in effect with the addition of two new constraints:
		\item\textcolor{red}{$np_{12} + np_{22} + np_{32} + np_{42} = 0$}
		\item\textcolor{red}{$nw_{23} + nw_{24} + nw_{25} + nw_{26} = 0$}
	\end{itemize}
	\item Identify inputs and outputs that you can control
	\begin{itemize}
		\item Unchanged from part A.
	\end{itemize}
	\item Specify all quantities mathematically
	\begin{itemize}
		\item Unchanged from part A.
	\end{itemize}
	\item Check the model for completeness and correctness
	\begin{itemize}
	\item All variables are positive.
	\end{itemize}
\end{enumerate}
\subsection*{ Matlab Code}
\textcolor{red}{Code minimally changed from part A.  Only changes are 2 additional constraints (16 total equations) in the linear equality matrix and vector.  Identical code from part A is not shown below (to save space).}
\lstinputlisting{../problem_one/partB_changes_fromA.m}
\end{document}


% ---------------------------------------------------------------------
\documentclass{article}

\usepackage{nberpreamble}
\title{\bfseries Notes on Simulating Power}
\author{\sffamily Prepared by Mauricio C\'aceres}
\date{\sffamily \today}

\usepackage{pgfplots}  % awesome plotting
\usepackage{tikz}      % vector graphics!
\usetikzlibrary{
  arrows,
  patterns,
  positioning,
  calc,
  fit,
  intersections,
  decorations.text,
  decorations.markings,
  decorations.pathmorphing,
  shadows.blur
}

\pgfplotsset{
  compat      = newest,
  axis x line = middle,
  axis y line = center,
  tick align  = outside,
  yticklabels = {,,},
  xticklabels = {,,},
  xtick       = {0},
  ytick       = {0}
}

\renewcommand{\displayoptions}{
  \maketitle
  \pagenumbering{arabic}
}

% ---------------------------------------------------------------------
\begin{document}
\displayoptions

%----------------------------------------------------------------------
\section{Parametric Power}
\label{sec:parametric_power}

We typically consider the model
\begin{equation}
Y_i = \alpha + \beta T_i + \gamma X_i + \varepsilon_i
\end{equation}

for some treatment $T_i$ at the individual level and the OLS estimator $\widehat{\beta}_{OLS}$ the difference in outcomes for the treatment and control groups. For $H_0: \beta = 0$, consider the rejection probability function
\begin{equation}
\pi_{N}(\beta) = P\set{\text{reject $H_0$} | \beta}
\end{equation}

For $\beta = 0$, this is $\alpha$ the probability of Type I error or \textit{significance}: How likely are we to make a mistake? For $\beta = \widetilde{\beta} \ne 0$ this is \textit{power}, the probability of rejecting the null: How likely are we to get it right? $\widehat{\beta}_{OLS}$ is $\sqrt{n}$-consistent, that is,
\[
  \sqrt{n} \left(\widehat{\beta}_{OLS} - \beta\right)
  \xrightarrow{D} N(0, V_{\widehat{\beta}})
\]

where $\beta$ is the true mean. Relying on large-sample asymptotics, we can visualize $\pi_N(\beta)$,
\begin{figure}[H]
  \centering
  \caption{Power of a Test}
  \begin{tikzpicture}
    \begin{axis}[name=plot1
      %,title=
      ,width=9cm
      ,height=5cm
      ,ymin=0
      ,ymax=0.5
      ,xmin=-1.5
      ,xmax=5.5
      ,domain=-6:6
      ,xticklabels = {-1,0,1,2,3,4}
      ,xtick = {-1,0,1,2,3,4}
      ]
      \addplot [fill=black!20,smooth,domain=1.96:6] {exp( - (x - 2.8)^2 / 2) / sqrt(2 * pi * 1)} \closedcycle;
      \addplot [smooth,domain=-6:1.96] {exp( - (x - 2.8)^2 / 2) / sqrt(2 * pi * 1)};
      \draw [-] (axis cs:2.8, 0) -- (axis cs:2.8, 0.399);
      \draw [<->] (axis cs:2, 0.1) -- (axis cs:2.76, 0.1);
      \node[above] at (axis cs:2.4, 0) {$t_{\kappa}$};
      \node[above] at (axis cs:2.8, 0.4) {$\widetilde{\beta} / SE$};
      \draw [->] (axis cs:4.75, 0.3) to[bend left=25] (axis cs:4.2, 0.115)
        node[above] at (axis cs:4.75, 0.3) {$\pi_N(\widetilde{\beta}) = \kappa$};
    \end{axis}
    \begin{axis}[name=plot2
      ,title={}
      ,at={($(plot1.south) + (-2.125cm, 4cm)$)}
      ,anchor=south
      ,width=9cm
      ,height=5cm
      ,ymin=0
      ,ymax=0.5
      ,xmin=-3.5
      ,xmax=3.5
      ,domain=-4:4
      ,xticklabels = {-2,-1,0,1}
      ,xtick = {-2,-1,0,1}
      ]
      \addplot [fill=black!20,smooth,domain=-1.96:1.96] {exp( - (x - 0)^2 / 2) / sqrt(2 * pi * 1)} \closedcycle;
      \addplot [smooth,domain=-4:-1.96] {exp( - (x - 0)^2 / 2) / sqrt(2 * pi * 1)};
      \addplot [smooth,domain=1.96:4] {exp( - (x - 0)^2 / 2) / sqrt(2 * pi * 1)};
      \draw [smooth] (axis cs:0, 0) -- (axis cs:0, 0.5);
      \node[above] at (axis cs:1.96, 0.1) {$t_{1 - \frac{\alpha}{2}}$};
      \node[above] at (axis cs:0.9, 0.4) {$H_0: \beta = 0$};
      \draw [->] (axis cs:-2.1, 0.2) to[bend right=25] (axis cs:-1.3, 0.115)
        node[above] at (axis cs:-2.1, 0.2) {$\pi_N(0) = 1 - \alpha$};
    \end{axis}
    \draw [dashed] (3.66, 4) -- (3.66, 1.5);
  \end{tikzpicture}
  \label{fig:power_of_a_test}
\end{figure}

A \textit{parametric} approach to power estimates $\widetilde{\beta}$ given $\alpha, \kappa, N, SE$ and terms it the \textit{minimum detectable effect}, MDE, or it estimates $N$ given $\alpha, \kappa, SE, MDE$.

%----------------------------------------------------------------------
\section{Simulated Confidence Interval}
\label{sec:simulated_confidence_interval}

However, it is possible to follow a \textit{non-parametric} approach to estimating power. Note there are $C = \left(\begin{smallmatrix} N \\ NP \end{smallmatrix}\right)$ ways to treat $PN$ individuals. If we estimate $\widehat{\beta}_{OLS}$ for each $c = 1, \ldots, C$, then we would know the exact distribution of our estimator for the treatment effect under the null given the data. Thus we could compute an exact $p$-value and determine whether to reject the null.

Even for modestly-sized data, $C$ will be intractably large. Hence we simulate $K$ draws from the possible treatment-control arrangements, $T_{ik}$ such that $\sum^{N}_{i = 1} T_{ik} = PN$, and estimate
\begin{equation}
Y_i = \alpha + \beta_k T_{ik} + \gamma X_i + \varepsilon_i
\label{eq:ri_ci_reg}
\end{equation}

Here $\widehat{\beta}_k$ will be distributed around $0$ and a $1 - \alpha$ CI under the null is given by
\begin{equation}
\widehat{CI}_{1 - \alpha} = \left(\widehat{F}^{-1}(\alpha / 2), \widehat{F}^{-1}(1 - \alpha / 2)\right)
\label{eq:ri_ci_hat}
\end{equation}

with $\widehat{F}(\widehat{\beta}_k)$ the empirical cdf of $\widehat{\beta}_k$. This approach is appealing compared to a parametric approach because it naturally takes into account the correlation structure of the errors, whereas a parametric approach requires making an assumption about $V_{\widehat{\beta}}$. If we had historical data on our study population (we can think of historical data as data on a population where our treatment had no effect, or the counterfactual of what would happen were we to treat our population with no effect) then we can simulate the CI above, and say that we expect to be able to reject effects outside the confidence interval.

%----------------------------------------------------------------------
\section{Simulated Power}
\label{sec:simulated_power}

Note, however, that this says nothing about power. In fact, power at either end of the confidence interval should be about $0.5$ (if $\widehat{\beta}_{OLS}$ were to be symmetrically distributed). Typically we look for a power level of $0.8$ or $0.9$. We could assume that the true effect of $T_i$ is $\widetilde{\beta}$, and estimate
\begin{equation}
  \begin{array}{r@{\hskip 4.5pt}l}
    \widetilde{Y}_{ik} & = Y_{i} + \widetilde{\beta} T_{ik} \\
    \widetilde{Y}_{ik} & = \alpha + \beta_k T_{ik} + \gamma X_{i} + \varepsilon_{i}
  \end{array}
\label{eq:ri_power_reg}
\end{equation}

In this case, $\widehat{\beta}_k$ will be distributed around $\widetilde{\beta}$ but the shape of the distribution would not have changed. Thus we can estimate power as
\begin{equation}
\widehat{\kappa} = \dfrac{1}{K} \sum^{}_{k} 1\left(\widehat{\beta}_k \notin \widehat{CI}_{1 - \alpha}\right)
\label{eq:ri_power_hat}
\end{equation}

and we we can search for $\widetilde{\beta}$ such that $\widehat{\kappa} \approx \kappa$ for some desired power level $\kappa$ (note $\widehat{\kappa} \xrightarrow{P} P\left(\beta_k \notin CI_{1 - \alpha}\right) = \kappa$). That is, we look for a $\widetilde{\beta}$ that causes us to reject the null $\kappa$ portion of the time. This is more complicated if $Y_i$ is binary. The approach works to obtain a CI under the null but the subsequent search does not map trivially. One idea is to randomly swap successes to failures (or the converse) based on $\widetilde{\beta}$. Consider
\begin{equation}
  \begin{array}{r@{\hskip 4.5pt}l}
    \widetilde{Y}_{ik} & = Y_{i} (1 - T_{ik}) + (Y_i + S_{ik}) T_{ik}  = Y_i + S_{ik} T_{ik} \\
    \widetilde{Y}_{ik} & = \alpha + \beta_k T_{ik} + \gamma X_{i} + \varepsilon_{i}
  \end{array}
\label{eq:ri_power_reg_binary}
\end{equation}

where $S_{ik}$ is constructed as follows
\begin{itemize}
  \item Let
    \begin{align*}
    T_k & = \sum^{}_{i} T_{ik}
    \quad\quad
    S_k = \sum^{}_{i} T_{ik} Y_i
    \\
    \widetilde{\beta} & \in \left[-\dfrac{S_k}{T_k}, \dfrac{T_k - S_k}{T_k}\right] \\
    S^1_k & = \widetilde{\beta} T_k \\
    S^2_k & =
      \begin{cases}
      \dfrac{S_k}{T_k} - S_{1k} & \widetilde{\beta} > 0  \\[9pt]
        \dfrac{T_k - S_k}{T_k} - S_{1k} & \widetilde{\beta} < 0
    \end{cases} \\
    \end{align*}

  \item Construct the set $\varsigma_k$ with $S^1_k$ entries equal to $1(\widetilde{\beta} > 0) - 1(\widetilde{\beta} < 0)$ and $S^2_k$ entries equal to $0$.

  \item For $i$ such that $T_{ik} = 1$, draw $s$ from $\varsigma_k$ \textit{without} replacement and set $S_{ik} = s$ ($S_{ik} = 0$ otherwise).
\end{itemize}

Note that $T^{-1}_k \sum^{}_{i} S_{ik} T_{ik} = \widetilde{\beta}$, hence
\begin{align*}
E\widehat{\beta}_k & = E\left[\widetilde{Y}_{ik} | T_{ik} = 1\right] - E\left[\widetilde{Y}_{ik} | T_{ik} = 0\right] \\
& = E\left[Y_i + S_{ik} | T_{ik} = 1\right] - E\left[Y_i | T_{ik} = 0\right] \\
& = EY_i + E\left[S_{ik} | T_{ik} = 1\right] - EY_i \\
& = \widetilde{\beta}
\end{align*}

So $\widehat{\beta}_k$ will be distributed around $\widetilde{\beta}$. Now we can outline a general simulation procedure:
\begin{enumerate}
\item Estimate a $1 - \alpha$ CI for $\widehat{\beta}_{OLS}$ under $H_0: \beta = 0$ using \Cref{eq:ri_ci_reg} and \Cref{eq:ri_ci_hat}.

\item Choose a starting MDE, $\widetilde{\beta}$, and estimate $\widehat{\beta}_k$ for $k = 1, \ldots, K$ using \Cref{eq:ri_power_reg} if $Y_i$ is continuous or \Cref{eq:ri_power_reg_binary} if $Y_i$ is binary.

\item Estimate power using \Cref{eq:ri_power_hat}.

\item If $\widehat{\kappa} < \kappa$ then increase $\widetilde{\beta}$; if $\widehat{\kappa} > \kappa$ then decrease $\widetilde{\beta}$.

\item Continue until $\left|\widehat{\kappa} - \kappa\right| < \epsilon$ for $\epsilon$ small.
\end{enumerate}

%----------------------------------------------------------------------
\section{Monte Carlo Simulations}
\label{sec:monte_carlo_simulations}

Consider some data-generating process (DGP) for $w_i = (y_i, x_i, T_i)$,
\[
  x_i \sim F_x
  \quad\quad
  T_i \sim \text{Bernoulli}(P)
  \quad\quad
  T_i \indep x_i
  \quad\quad
  y_i | x_i, T_i \sim F_y
\]

where
\[
\beta = E(y_i | T_i = 1, x_i) - E(y_i | T_i = 0, x_i)
\]

The aim is to compute the power of testing $\widehat{\beta}_{OLS}$ against our simulated confidence interval. For $m = 1, \ldots, M$:
\begin{enumerate}
\item Generate two draws from the DGP, $w_{1m}$ with $P = 0$ and $w_{2m}$ $P \in (0, 1)$.

\item Compute $\widehat{\kappa}_{1m}, \widehat{\kappa}_{2m}$ from our power simulation procedure and $\widehat{\beta}_{1m}, \widehat{\beta}_{2m}$ from OLS.

\item Construct a $1 - \alpha$ confidence interval for $\beta$, $\widehat{CI}^M_{1 - \alpha}$, using \Cref{eq:ri_ci_hat} and compute
\[
  \overline{\kappa}^M = \dfrac{1}{M} \sum^{}_{m} 1\left(
      \widehat{\beta}_{2m} \notin \widehat{CI}^M_{1 - \alpha}
  \right)
\]

\item It should be the case that $\widehat{\kappa}_{1m}$ are distributed around $\alpha$ and $\widehat{\kappa}_{2m}$ are distributed around $\overline{\kappa}^M$.
\end{enumerate}

% We use the procedure above to see if we can empirically match what power should be under known conditions. Consider the standard formula for MDE
% \begin{equation}
% MDE = (z_{\alpha / 2} + z_{\kappa}) \sqrt{\dfrac{\sigma_\varepsilon^2}{P (1 - P) N}}
% \end{equation}
%
% We can recover power as
% \begin{equation}
% \kappa = \Phi\left(MDE \sqrt{\dfrac{P (1 - P) N}{\sigma^2_\varepsilon}} - t_{\alpha / 2}\right)
% \end{equation}

% TODO: Finish writing this out. // 2016-09-02 14:33 EDT
% TODO: Actually code this up // 2016-09-02 14:33 EDT

% ---------------------------------------------------------------------
\end{document}


%%Note: You can only use \section command, you are not allowed, per TTU Graduate School, use
%%\subsection command for ghigher level subheadings. At most level 2 subheadings are allowed.
\chapter{Cyber-Physical Systems}
\label{Cyber-Physical Systems Chapter}

\section[URE Generation Mechanisms]{URE Generation Mechanisms}

Electronic digital circuitry utilizes many periodic signals generated by components such as clocks and oscillators. As part of their inherent operation, these signals are often directly coupled or mixed and modulated both 1) Intentionally for proper operation of the device and 2) Unintentionally resulting from imperfect filtering of high frequency signals or the operation of nonlinear components. This behavior in digital devices leads to generated URE being conducted on to the power infrastructure as shown in Figure \ref{fig:ure_model}. All periodic signals used in the operation of an electronic device are potentially, and independently, filtered by some process before being directly conducted as URE outside the device often through powerline emissions.  Additionally, each periodic signal may mix (multiply) with other periodic signals before being conducted as URE.

\begin{figure}[tb]
	\includegraphics[width=\textwidth]{./model_results/URE_Model.jpg}
	\centering
	\caption{Model of URE generation mechanisms within an electronic device.  URE is composed of the direct conduction components and the intermodulation components of periodic signals utilized within the URE generating device.}
	\label{fig:ure_model}
\end{figure}

The simplified model in Figure \ref{fig:ure_model} does not take in to account the continued mixing and addition of previously multiplied signals which could theoretically continue nearly indefinitely resulting in a significantly increased model complexity.  Given that no assumption was made with regard to the periodic signals ($P_n$), the additional frequency components generated from continued addition and multiplication can be folded in to additional $P$ components or different periodic signal structures.

\section[Model Development]{Model Development}

Analytically, the received complex URE time domain signal, $r$, can be written as the sum of the direct and intermodulation responses
\begin{equation}
	r ={} c \ast \left(\sum^N_{n=1} p_{n} \ast h_{n} + \sum^N_{i\neq{}j} \left(p_{i} \ast g_{i,j}\right)\left(p_{j} \ast g_{j,i}\right)\right) \label{eq:uremodel1}
\end{equation}
where $r$ is the time domain received signal, $c$ is the URE channel gain, $p_{n}$ are periodic signals used within the devices, $h_{n}$ are the direct power gains for each signal, and $g_{i,j}$ are mixing product gains for component cross-talk and intermodulation, $i$ and $j$ are sequences $1$ to $N$, and $N$ is the number of periodic signals. Note that Equation \ref{eq:uremodel1} assumes no signal mixes with itself.

Using the Convolution Theorem to transform $r$ into its Power Spectral Density (PSD), $\hat{r}$,
\begin{equation}
    \hat{r} ={} \hat{c} \left( \underbrace{\sum^{N}_{n=1} \hat{p}_{n}\hat{h}_{n}}_\text{Direct Conduction} + \underbrace{\sum^N_{i\neq{}j} \hat{p}_{i}\hat{g}_{i,j} \ast  \hat{p}_{j}\hat{g}_{j,i}}_\text{Mixing Products} \right) \label{eq:uremodel2}
\end{equation}
where $\hat{p}$ is the PSD of $p$ and $\hat{g}$, $\hat{h}$, and $\hat{c}$ are the Fourier transforms of $g$, $h$, and $c$ filter responses, respectively.

Evaluation of the trivial case where $p_1 = \sin(2\pi f_1 t)$, $p_2 = \sin(2\pi f_2 t)$, $c = h_1 = h_2 = g_1 = g_2 = \delta(t)$, and $f_1 \gg f_2$ shows the following
\begin{equation} \label{eq:uremodelsimpletime}
\begin{split}
	r & = \sin(2\pi f_1 t) + \sin(2\pi f_2 t) + \sin(2\pi f_1 t)\sin(2\pi f_2 t) \\
		& = \sin(2\pi f_1 t) + \sin(2\pi f_2 t) + \frac{1}{2}\cos(2\pi(f_1 - f_2)t) +  \frac{1}{2}\cos(2\pi(f_1 + f_2)t)\\
\end{split}
\end{equation}
which returns the following positive frequency spectral components
\begin{equation} \label{eq:uremodelsimplefreq}
	\hat{r} = \underbrace{\frac{1}{2}\delta(f - f_1) + \frac{1}{2}\delta(f - f_2)}_\text{Unmodulated Frequency Peaks} + \underbrace{\frac{1}{4}\delta\left(f - (f_1 - f_2)\right) + \frac{1}{4}\delta\left(f - (f_1 + f_2)\right)}_\text{Modulation Sidebands}\\
\end{equation}

Examination of Equation \ref{eq:uremodelsimplefreq} shows that for the simple case of two periodic signals, two unmodulated frequency tones and two modulation sidebands are present.  The number of potential frequency peaks within a URE spectrum grows at a rate of $\mathcal{O}(n^2)$ for each additional periodic signal and $\mathcal{O}(n^2)$ for each additional harmonic associated with the periodic signals, resulting in combined growth of $\mathcal{O}(n^2) \times \mathcal{O}(n^2) = \mathcal{O}(n^4)$.  For instance, $10$ carriers, with $10$ harmonics each, can generate nearly $10000$ peaks within the frequency domain. The number of potential peaks within the spectrum of a URE signal can be significant, resulting in an overdetermined system with a very large feature space, which DASP is presented to address.

The continuous-time Fourier Series expansion of a periodic signal shows that it is comprised of an infinite sum of its respective harmonics; therefore, aligning and summing its harmonic content maximizes detection of the signal.  Although the fundamental frequencies of the periodic signals comprising $r$ are unknown, the HASP algorithm provides a method of aligning harmonics of all periodic signals within a given band regardless of the fundamental frequency.  Additionally, the trigonometric product identities show that the intermodulation and mixing of periodic signals results in the sum and differences of the mixing frequencies, as shown in Equation \ref{eq:uremodelsimplefreq}.  The MASP, CMASP, and SCAP algorithms provide a method for detecting these modulations and aligning the frequency mixing sums and differences when the underlying carrier and modulation frequencies are unknown.  As shown in Sections \ref{Harmonically Aligned Signal Projection}, \ref{Modulation Aligned Signal Projection}, \ref{Spectral Correlation Aligned Projection}, \ref{Cross-Modulation Aligned Signal Projection} the HASP, MASP, SCAP, and CMASP algorithms provide a method for aligning and summing harmonics and their cross-products regardless of their underlying fundamental frequency, carrier frequency, and modulation frequency and therefore maximize features relevant to device detection and classification.  Additionally, Section \ref{Frequency Aligned Signal Projection} describes the Frequency Aligned Signal Projection which provides a method for aligning frequencies over time for further characterization of signal harmonic components and their respective time and frequency modulations.

\section[DASP Methodology]{DASP Methodology}

Alignment of signal components, such as harmonics, modulations, and frequencies, with the DASP algorithms result in 2-D images which need to be further processed for extraction of features and eventually used for testing and training in a machine learning framework.   As shown in Figure \ref{fig:dasp_methodology}, raw time domain signal captures are transformed through a DASP process in to a 2-D structure and further processed through an image scaling, segmentation, and summing process.   Statistical features and processed DASP images are then utilized for training and testing using LDA, k-NN, and CNN machine learning processes.

\begin{figure}[tb]
	\includegraphics[width=\textwidth]{./model_results/model.jpg}
	\centering
	\caption{DASP processing and feature generation methodology.  The DASP algorithms transform a 1-D vector into a 2-D image which is subsequently transformed, segmented, or processed through an image processing algorithm.  The DASP images are then utilized to classify URE from electrical devices through a variety of feature extraction and learning methods.}
	\label{fig:dasp_methodology}
\end{figure}


\addchap[Notation]{Notation}\label{chapter:notation}

\section*{Problems}

\nopagebreak\begin{tabularx}{\textwidth}{p{100pt}X}
    $T \in \mathbb{N}$ & number of time slots \\
    $\delta \in \mathbb{R}_{>0}$ & length of a time slot \\
    $d \in \mathbb{N}$ & number of dimensions \\
    $\mathcal{X} \subset \mathbb{R}^d$ & decision space \\
    $m_k \in \mathbb{N}$ & upper bound of dimension $k$ \\
    $M_k \subset \mathbb{N}_0$ & allowed values of dimension $k$ (in the discrete case) \\
    $\mathcal{M} = \mathcal{X}$ & set of all configurations (in the discrete case) \\
    $f_t(x) \in \mathbb{R}_{\geq 0}$ & hitting cost of action $x \in \mathcal{X}$ during time slot $t$ \\
    $\beta_k \in \mathbb{R}_{>0}$ & switching cost of dimension $k$ \\
    $\lambda_t \in \mathbb{N}_0^e$ & load profile during time slot $t$ \\
    $c_k \in \mathbb{R}_{\geq 0}$ & time-independent hitting cost of dimension $k$ \\
\end{tabularx}

\section*{Data Center Model}

\subsection*{Dispatching}

\nopagebreak\begin{tabularx}{\textwidth}{p{100pt}X}
    $m_k \in \mathbb{N}$ & maximum number of servers of type $k$ \\
    $l_k^{max} \in \mathbb{N}$ & maximum number of jobs a server of type $k$ can process in a single time slot \\
    $e \in \mathbb{N}$ & number of load types \\
    $\lambda_{t,i} \in \mathbb{N}_0$ & number of jobs of type $i$ during time slot $t$ \\
    $\lambda_t \in \mathbb{N}_0$ & total load during time slot $t$ \\
    $\lambda^{max} \in \mathbb{N}$ & maximum total load of feasible load profiles \\
    $\mathcal{Z}$ & set of job assignments of jobs of individual load types to server types \\
    $z_{t,k,i} \in [0,1]$ & fraction of jobs of type $i$ assigned to servers of type $k$ during time slot $t$ \\
    $l_{t,k,i} \in [0,\lambda^{max}]$ & (fractional) number of jobs of type $i$ assigned to servers of type $k$ during time slot $t$ \\
    $l_{t,k} \in [0,\lambda^{max}]$ & (fractional) number of jobs assigned to servers of type $k$ during time slot $t$ \\
    $s_k(l) \in [0,1]$ & utilization (or speed) of a server of type $k$ with load $l$ \\
    $\theta_k \in [0,1]$ & maximum allowed utilization of a server of type $k$ \\
    $g_{t,k}(l) \in \mathbb{R}_{\geq 0}$ & operating cost of a server of type $k$ during time slot $t$ when $l$ jobs are processed on the server \\
    $q_{t,k,i}(l) \in \mathbb{R}_{\geq 0}$ & cost of processing a job of type $i$ on a server of type $k$ during time slot $t$ when $l$ jobs are processed on the server \\
\end{tabularx}

\subsection*{Energy Cost}

\nopagebreak\begin{tabularx}{\textwidth}{p{100pt}X}
    $e_{t,k}(s) \in \mathbb{R}_{\geq 0}$ & energy cost of a server of type $k$ with utilization $s$ during time slot $t$ \\
    $\nu_{t,k}(p) \in \mathbb{R}_{\geq 0}$ & energy cost of a server of type $k$ with energy consumption $p$ during time slot $t$ \\
    $\phi_k(s) \in \mathbb{R}_{\geq 0}$ & energy consumption of a server of type $k$ with utilization $s$ \\
    $\Phi_k(s) \in \mathbb{R}_{\geq 0}$ & power consumption of a server of type $k$ with utilization $s$ \\
    $\Phi_k^{max} \in \mathbb{R}_{\geq 0}$ & power consumption of a server of type $k$ on full load \\
    $\Phi_k^{min} \in \mathbb{R}_{\geq 0}$ & power consumption of an idling server of type $k$ \\
    $c_{t,i} \in \mathbb{R}_{\geq 0}$ & energy cost of energy source $i$ per unit of energy during time slot $t$ \\
\end{tabularx}

\subsection*{Revenue Loss}

\nopagebreak\begin{tabularx}{\textwidth}{p{100pt}X}
    $\gamma \in \mathbb{R}_{\geq 0}$ & revenue loss factor, i.e., lost revenue per unit of delay \\
    $r_{t,i}(d) \in \mathbb{R}_{\geq 0}$ & revenue loss of jobs of type $i$ with an average delay $d$ during time slot $t$ \\
    $d_{t,k,i}(l) \in \mathbb{R}_{\geq 0}$ & average delay of a job of type $i$ processed on a server of type $k$ during time slot $t$ where the total load on the server is $l$ \\
    $\delta_{t,k,i} \in \mathbb{R}_{\geq 0}$ & constant delay when processing a job of type $i$ on a server of type $k$ during time slot $t$ \\
    $\mu_k \in \mathbb{R}_{\geq 0}$ & service rate of a server of type $k$ \\
    $\delta_i \in \mathbb{R}_{\geq 0}$ & minimal detectable delay of jobs of type $i$ \\
    $\eta_{k,i} \in \mathbb{R}_{\geq 0}$ & processing time of a job of type $i$ on a server of type $k$ \\
\end{tabularx}

\subsection*{Switching Cost}

\nopagebreak\begin{tabularx}{\textwidth}{p{100pt}X}
    $\epsilon_k \in \mathbb{R}_{\geq 0}$ & additional energy consumed by toggling a server of type $k$ on and off \\
    $\delta_k \in \mathbb{R}_{\geq 0}$ & delay in migrating connections or data of a server of type $k$ before it can be powered down \\
    $\tau_k \in \mathbb{R}_{\geq 0}$ & wear-and-tear costs of toggling a server \\
    $\rho_k \in \mathbb{R}_{\geq 0}$ & perceived risk associated with toggling a server of type $k$ \\
    $\xi_k \in \mathbb{R}_{>0}$ & normalized switching cost measuring the minimum duration a server of type $k$ must be asleep to outweigh the switching cost \\
\end{tabularx}

\subsection*{Networks}

\nopagebreak\begin{tabularx}{\textwidth}{p{100pt}X}
    $\iota \in \mathbb{N}$ & number of data centers \\
    $\zeta \in \mathbb{N}$ & number of geographically centered request sources \\
    $\delta_{t,j,s} \in \mathbb{R}_{\geq 0}$ & network delay incurred by routing a request from source $s$ to data center $j$ during time slot $t$ \\
    $\xi$ & number of energy sources \\
    $p_{t,i,j} \in \mathbb{R}_{\geq 0} \cup \{\infty\}$ & energy from source $i$ available at data center $j$ during time slot $t$ \\
    $u_{t,i} \in \mathbb{R}_{\geq 0}$ & average profit per unit of energy from source $i$ during time slot $t$ \\
    $q_{t,i} \in [0,1]$ & minimum fraction for energy from source $i$ during time slot $t$ \\
    $\delta_{t,i,j} \in \mathbb{R}_{\geq 0}$ & remaining energy requirement of data center $j$ during time slot $t$ after all energy sources up to source $i$ were used \\
\end{tabularx}

\section*{Complexity}

\nopagebreak\begin{tabularx}{\textwidth}{p{100pt}X}
    $\mathcal{O}(C)$ & complexity of computing the hitting costs $f_t$ \\
    $\mathcal{O}(O_{\epsilon}^d)$ & convergence rate of a convex optimization finding an $\epsilon$-optimal solution in $d$ dimensions \\
    $\mathcal{O}(R_{\epsilon})$ & convergence rate of a root finding method finding an $\epsilon$-optimal root \\
    $\mathcal{O}(I_{\epsilon})$ & convergence rate of a quadrature method finding an $\epsilon$-optimal integral \\
\end{tabularx}

\section*{Algorithms}

\nopagebreak\begin{tabularx}{\textwidth}{p{100pt}X}
    $\tau$ & current time slot \\
    $\hat{x}$ & optimal value of $x$ with respect to some optimization \\
    $X \in \mathcal{X}^t$ & schedule, i.e., sequence of configurations over the time horizon $t$ \\
    $x, y \in \mathcal{X}$ & configuration \\
    $i, j \in [m_k]_0$ & value of dimension $k$ \\
    $x_{k \gets j}$ & configuration $x$ after updating dimension $k$ to $j$ \\
\end{tabularx}

\section*{Implementation}

\nopagebreak\begin{tabularx}{\textwidth}{p{100pt}X}
    $\chi_{\tau}(t, x) \in \bigcup_{n=1}^{\infty} \mathbb{R}_{\geq 0}^n$ & hitting cost of action $x \in \mathcal{S}$ at time slot $t \in [T]$ given the information from time slot $\tau$ \\
    $\mathcal{P}_t \in \left(\bigcup_{n=1}^{\infty} \mathbb{N}_0^n\right)^e$ & predicted load profile for time slot $t$, i.e., vector of predicted loads for each job type\\
\end{tabularx}

\section*{Miscellaneous}

\nopagebreak\begin{tabularx}{\textwidth}{p{100pt}X}
    $[n] := \{1, \dots, n\}$ & range of natural numbers from $1$ to $n$ \\
    $[n]_0 := \{0\} \cup [n]$ & range of integers from $0$ to $n$ \\
    $[a : b] :=\newline \{a, a+1, \dots, b\}$ & range of integers from $a$ to $b$ \\
    $(x)_a^b :=\newline \max\{a, \min\{b, x\}\}$ & uni-dimensional projection of $x \in \mathbb{R}$ onto $[a,b]$ \\
    $(x)^+ := \max\{0, x\}$ & uni-dimensional projection of $x \in \mathbb{R}$ onto $[0, \infty)$ \\
    $x_{\text{min}}$ & smallest entry of the vector $x$ \\
    $x_{\text{max}}$ & largest entry of the vector $x$ \\
\end{tabularx}


\iffalse
It's unfair to say that mathematicians aren't real doctors, we perform surgeries all the time. In this class we'll introduce the notion of a topological manifold via simplicial (delta) complexes. Spend a day or two doing examples and go over several notions like orientation, cobordism and of course surgery.

Keywords: simplicial complex, manifold, orientation, cobordism, surgery

Type: Lecture
Homework: Recommended
Prereqs: None
\fi

\input{../preamble}
\rhead{\scshape Mathcamp 2017 : All things Manifoldy}
\begin{document}
\title{Modular origami}
\author{Apurva Nakade}
\thispagestyle{fancy}
\maketitle

\section{Orientation of surfaces}
Let us start with surfaces. A surface is called orientable if has two sides. A sphere or a torus are easily seen to be orientable. In fact any surface that can be \textit{embedded} in $\R^3$ is orientable. A Mobius strip on the other hand (which is not a surface as it has a boundary) is not orientable. With a little more mental effort one might be able to see that a Klein bottle or a Projective space are not orientable. We want an equivalent definition which can be used on gluing diagrams.

We begin by defining orientation of a polygon. An \textbf{orientation of a polygon} is simply a cyclic ordering of it's vertices. For example this is one of the two possible ways to orient a triangle with vertices $(0,1,2)$.

\begin{center}
	\begin{tabular}{c}
		\centering \includegraphics[height=3cm]{../noImageAvailable}
	\end{tabular}
\end{center}

\begin{exercise}
	This is related to the fact that the polygon has two sides via the right hand rule in physics. Do you see the connection?
\end{exercise}

An \textbf{orientation} of (a gluing diagram of) a surface is a compatible choice of orientation for each triangle, where the orientations of two adjacent triangles need to be compatible in the following way,
\begin{center}
	\begin{tabular}{c c c}
		\centering \includegraphics[height=3cm]{../noImageAvailable} & \: & \centering \includegraphics[height=3cm]{../noImageAvailable}
	\end{tabular}
\end{center}
This forces an edge to be directed in two different directions in adjacent triangles.

\begin{exercise}
	Add the diagonals to the standard gluing diagrams to obtain triangulations and try to find orientations for them. Conclude that $S^2,T$ are orientable and $\R\P^2,K$ are not.
\end{exercise}

\begin{exercise}
	Use gluing diagrams to show that $T\#T$ is orientable. Generalize this to argue that $T ^{\# n}$ are orientable.
\end{exercise}

This same method also allows us to understand manifolds in higher dimensions as well.

\section{Simplices}
As we go to higher dimensions polygons (or is it polytopes?) become whacky are themselves quite hard to understand. So instead of looking at arbitrary polygons we look at the simplest polygons: triangles. We'll upgrade the definition of a triangle to a simplex which can live in arbitrary dimensions.

\begin{definition}
	An $n$ dimensional \textbf{simplex} $\Delta^n$ is any set which is homeomorphic to the following region in $\R^{n}$.
	\begin{align}
		\{ (x_0, x_1, \cdots, x_n) : x_0 + \cdots + x_n \le 1 \mbox{ and each } x_i \ge 0  \}
	\end{align}
\end{definition}
A 1 dimensional simplex is a segment and a 2 dimensional simplex is a triangle.
Simplices are topologists best friend.

Note that by adding extra diagonal lines we could have made the gluing diagrams entirely out of triangles.
We represent a simplex by its set of vertices.

\begin{figure}[h]
	\centering \includegraphics{../noImageAvailable}
	\caption{Gluing diagram with labeled simplices}
	\label{}
\end{figure}

As with origami you can glue these simplices any way you want and get relaly interesting objects. We're going to glue them to create manifolds.

\section{Manifolds from simplices}
A delta complex is a collection of simplices glued along the faces.

\begin{definition}
	Faces of a simplex followed by examples.
\end{definition}

\begin{definition}
	Definition of a delta complex followed by several examples.
\end{definition}

Talk about spheres, tori and projective spaces in 3 dimensions.

\begin{ques}
	When is a delta complex a manifold?
\end{ques}

Examples of manifolds and non-manifolds.

\end{document}

01 All things manifoldy/04 All things manifoldy.tex


\section{Communication}
\label{sec:comm}

Each of the two MPI implementations consists of two communication steps, but they have different dimensions in each one.

\begin{itemize}
	\item \textbf{Bucket count communication}. In the first implementation, each process has to send each of their local bucket sizes to a single process. So for the complete matrix of all bucket sizes (this matrix has size $P \times B$) each element requires a single Send/Receive. This makes the complexity for this step $\Theta(P \cdot B)$
		With the addition of load balance control, this entire matrix has to be sent to every processor, making complexity becomes $\Theta(P^2 \cdot B)$. Although this complexity is higher for the version that should provide the best results, this is because the main overhead is not in the bucket size communication, but in the communication of the elements themselves. The number of buckets is usually really small (only 256 buckets for $g=8$), and the input can consist in millions of elements, this difference is clear.

	\item \textbf{Keys communication}. Each process has to send their local keys to the apropriate destination. So for $N$ keys, the complexity of communication becomes $O(N)$. Notice that this is $O$ notation and not $\Theta$ because some of the keys may already be in the appropriate process, requiring only a local copy in memory. The difference is in the balance of the keys. For the first version, the number of keys that a single process sends or receives can be anything between 0 and $N$, and it may happen that the weight of communication will be only in a subset of all processes, increasing communication delay. But in the second implementation, each process is garanteed to send exactly $N/P$ keys and receive that same amount back from other processes. This may have a huge impact in communication delay, and consequentely, in the overall performance of the algorithm.

\end{itemize}

Also note that this analysis refers only to a single iteration of radix sort. Every iteration has the same complexity, so the overall complexity will be equal to the one of a single iteration, multiplied by the number of iterations.


\section{Electric charges, fields, and potentials}

Biological tissue contains a high number of mobile electric charges (mostly ions, with \ce{Na+}, \ce{Cl-}, and \ce{K+} the most abundant ones \cite{Martinsen2015a}). Each charge is an electric monopole that emits an electric field, whose magnitude decays as $1/r^2$ (where $r$ is the distance from the charge). The electric field vectors point outwards for positive charges, and inwards for negative charges. All these electric fields summate linearly to form the total electric field vector $\E(\loc,t)$ at location $\loc$ and time $t$.

These last facts (and more) are concisely described by the first of Maxwell's equations, also known as Gauss's law \cite{Feynman2013}:
%
\begin{equation}
\label{eq:gauss}
\div{\E(\loc,t)} = \frac{\rho(\loc,t)}{\eps_0},
\end{equation}
%
where $\rho(\loc,t)$ (in \si{\coulomb\per\metre^3}) is the charge density at location $\loc$, and $\eps_0 \approx \SI{10}{\pF\per\m}$ is the distributed capacitance (permitivity) of free space. (For an intuitive explanation of the $\div$ notation, see \cref{sec:div_curl}).

In the complete Maxwell's equations (\cref{appendix_maxwell}), the electric and the magnetic fields are coupled, and cannot be described separately. When neither field changes too quickly over time however, the equations become decoupled. The electric field is then determined only by the locations of electric charges, and the existence of the magnetic field can be safely ignored. In electrophysiological conditions, this so called ``quasi-static'' assumption is met (\cite{Nunez2006,Plonsey2007}). The electric field is then completely described by the equations of electrostatics, namely \cref{eq:gauss} and the following \cref{eq:irrotational} -- even under normal time-varying conditions.
%
\begin{equation}
\label{eq:irrotational}
\curl{\E(\loc,t)} = \vb{0}
\end{equation}
%
A consequence of \cref{eq:irrotational} is that a potential function can be defined for the electric field \cite{Feynman2013}. This is the electric field potential, $\phi(\loc,t)$. It is a scalar field that is defined such that
%
\begin{equation}
\label{eq:potential}
\E(\loc,t) = -\grad{\phi(\loc,t)};
\end{equation}
%
i.e. such that the electric field points from locations of high potential to locations of lower potential. This is useful because the scalar field $\phi(\loc,t)$ is easier to reason about than the vector field $\E(\loc,t)$, while it contains the same amount of information. From \cref{eq:potential}, it is clear that $\phi(\loc,t)$ is only defined up to a constant. A natural choice of absolute electric potential is to shift the electric potential $\phi(\loc,t)$ so that a point with no charges around (or equivalently, surrounded by positive and negative charges that balance each other out) has $\phi(\loc,t) = \SI{0}{\volt}$ (see \cref{sec:appendix_potential}).

In neuroscience, the electric potential $\phi(\loc,t)$ outside cells is often called the `local field potential' (LFP) -- especially when only frequencies below about \SI{500}{\hertz} are considered.\footnotemark{} The adjective `local' is a bit misleading: the LFP is not more `local' than any other electric field potential. The LFP, $\phi(\loc,t)$, is the quantity that we measure in sharp wave-ripple detection.\phantomsection\label{def:LFP}

\footnotetext{The naming for $\phi(\loc,t)$ is in general inconsistent. It is variably denoted by `voltage', `electric field potential', `electric potential', `field potential', `potential field', or simply `potential'.}

Although the above equations are complete, it is not clear from them how the LFP arises from charge distributions in the brain. The following section derives a more explicit formula for $\phi(\loc,t)$ in terms of free charges (i.e. mostly ions).

\section{Electric potential in neural tissue}

\Cref{eq:gauss,eq:irrotational} are valid from the scale of atoms to the scale of the brain and beyond; material (or tissue) properties appear implicitly as nanoscale variations in the charge density term. When working at macroscopic scales (as we do), it is easier however to use a slightly different but equivalent formulation, where material properties are explicitly built into the equation. This formulation rests on two ideas. The first is the separation of charges $\rho$ into bound charges $\rho_\bound$ and free charges $\rho_\free$ (which are mostly ions in the brain): $\rho = \rho_\bound + \rho_\free$. The second is the assumption that the polarisation of materials is directly proportional to the electric field strength. This is a common assumption, that is largely valid for brain tissue in normal conditions \cite{Nunez2006}. It can then be shown \cite{Feynman2013} that \cref{eq:gauss,eq:irrotational} are equivalent to:
%
\begin{align}
\div(\eps_r(\loc,t)\; \E(\loc,t))
    &= \frac{\rho_\free(\loc,t)}{\eps_0}   \label{eq:gauss_brain} \\
\curl{\E(\loc,t)}
    &= \vb{0},                             \label{eq:irrot_brain}
\end{align}
%
where $\eps_r(\loc,t)$ is a dimensionless material property. $\eps_r(\loc,t) = \eps(\loc,t) / \eps_0$, with $\eps(\loc,t)$ the absolute permittivity of the material at location $\loc$ and $\eps_r(\loc,t)$ the relative permittivity of that material (also known as the dielectric constant; sometimes denoted by $\kappa$). $\eps$ and $\eps_r$ are scalars for a isotropic materials, and $3 \cross 3$ matrices for anisotropic materials.

Broadly, the brain tissue relevant for sharp wave-ripples can be divided into two tissue types with different material properties. (See \cref{fig:neuropil}). The first type is the seawater-like fluid inside and in between cells. It has a relative permittivity $\eps_r \approx 75$ and is moderately conductive due to the free ions it contains (conductivity $\sigma \approx \SI{1}{\siemens\per\metre}$; compare with $\approx \SI{1e7}{\siemens\per\metre}$ for most metals) \cite{Michel2017,Martinsen2015,Marszalek1991,Nunez2006}. The other type are the lipid bilayer membranes, separating the insides from the outside of cells, organelles, and vesicles. They have a relative permittivity $\eps_r \approx 5$ and are ordinarily not conductive ($\sigma \approx \SI{1e-6}{\siemens\per\metre}$)\footnotemark{} \cite{Marszalek1991,Weaver2003}.

\footnotetext{Translated to the scale of neural tissue, a patch of membrane (which is $\approx \SI{5}{\nano\metre}$ thick \cite{Goodsell2014}) of area $1 \cross 1 \si{\micro\metre^2}$ has a resistance of $\approx \SI{5}{\giga\ohm}$. A similar `slice' of extracellular fluid with a volume of $1 \si{\micro\metre^2} \cross \SI{5}{\nano\metre}$ has a resistance of $\approx \SI{5}{\kilo\ohm}$}
%
% Conductivity σ [S/m = S m^2 / m^3]
% Resistivity ρ = 1 / σ [Ω m]
% Conductance G [S = A/V]
%
% wikipedia:
% ρ = R A / l  <=>  R  = ρ l / A  = l / (A σ)
% => 1 / σ = A / (G l)
% or G = σ * A / l
% The longer, the less conductive. OK.

% Extracellular space occupies 0.02 - 0.2 of cortex
% Neurons: 0.4 - 0.5
% Glia: 0.3 - 0.5

\begin{figure}
\includegraphics[width=\textwidth]{EM}
\caption{\textbf{The environment in which recordings are made.} \emph{Figure
adapted from \cite{Knott2008}}. Scanning electron micrograph of a slice of mouse cortex. (Rat hippocampal tissue looks similar, see e.g. \cite{Martin2017}). This type of tissue is called ``neuropil''. It consists of the long and narrow excrescences of neurons (dendrites and axons) and of glial cells, seen here in cross and through section. Note that cell bodies are much larger than the cross sections of dendrites and axons as seen here; a typical neural cell body (``soma'') of \SI{15}{\micro\metre} wide would be larger than this image (which is \SI{7}{\micro\metre} wide). Some landmarks are the many mitochondria (two of which are annotated with red *); two myelinated axons (green *); and all the lipid bilayers separating cells from their environment (thin black lines, two of which are highlighted in blue). The pink inset (\SI{1}{\micro\meter} by \SI{1.24}{\micro\meter}) shows two ``boutons'' (axon terminals) synapsing onto a dendrite. Note the synaptic vesicles (the many small dark circles), which encapsulate neurotransmitter molecules, ready for release in the synaptic clefts (black and white arrows).}
\label{fig:neuropil}
\end{figure}

The relative permittivity $\eps_r(\loc,t)$ is therefore not uniform even throughout small regions in the brain. As a consequence, \cref{eq:gauss_brain,eq:irrot_brain} together with \cref{eq:potential} do not have a simple solution expressing the electric potential in terms of excess ion charges, and must instead be solved numerically.

We can however get some sense for the behavior of the electric potential $\phi(\loc,t)$ by assuming a uniform permittivity $\eps$ throughout the neural tissue. (This assumption is often implicitly made -- and is rarely even mentioned -- in the electrophysiology literature \cite{Plonsey2007,Nunez2006,Destexhe2013}). It can then be shown that \cref{eq:gauss_brain,eq:irrot_brain,eq:potential} has an intuitive and straightforward solution for the electric potential \cite{Feynman2013}. When we divide the tissue in many tiny volumes $\dd{V}$ that each contain a net free charge density $\rho_\free(t)$ at time $t$, the electric potential at any point and time can be calculated as
%
\begin{equation}
\label{eq:charge_sum}
\phi(\loc,t) = \frac{1}{4 \pi \eps} \int_V \frac{\rho_\free(t)}{r} \dd{V},
\end{equation}
%
where $r$ is the distance from the measurement location $\loc$ to each tiny volume $\dd{V}$, and where we sum over the whole region $V$ where free charges are present.

\Cref{eq:charge_sum} is a symbolical translation of the claim made in the introduction of this chapter, which I summarise here: Each free charge emits a potential field that decays as $1/r$. This potential field is positive around positive ions, and negative around negative ions. All these potential fields summate linearly, to form the total electric potential $\phi(\loc,t)$.

This description is already useful to `explain' multiple phenomena observed in actual $\phi(\loc,t)$-recordings in the brain, as the next section shows.