\documentclass[12pt]{scrartcl}

\input{preamble}

\makeatletter
\title{Hack 12.0}\let\Title\@title
\subtitle{Computer Science I\\
Recursion \& Memoization\\
{\small
\vskip1cm
Department of Computer Science \& Engineering \\
University of Nebraska--Lincoln}
\vskip-3cm}
%\author{Dr.\ Chris Bourke}
\date{~}
\makeatother

\begin{document}

\maketitle

\hrule

\input{instructions.tex}

Correctness: proportionally for test cases; it \textbf{must run on the grader} to
receive any points.

\section*{Problem Statement}

A binomial coefficient, ``$n$ choose $k$'' is a number that corresponds
to the number of ways to \emph{choose} $k$ items from a set of $n$ distinct
items.  You may be familiar with some the notations, $C(n,k)$ or $C_n^k$
or ${}_{n}C_k $, but most commonly this is written as
  $${n \choose k}$$
and read as ``$n$ choose $k$''.  There is an easy to compute formula involving
factorials:
  $${n \choose k} = \frac{n!}{(n-k)!k!}$$
For example, if we have $n = 4$ items, say $\{a, b, c, d\}$ and want to choose
$k=2$ of them, then there are
  $${4 \choose 2} = \frac{4!}{(4-2)!2!} = 6$$
ways of doing this.  The six ways are:
  $$\{a, b\}, \{a, c\}, \{a, d\}, \{b, c\}, \{b, d\}, \{c, d\}$$
There are a lot of other interpretations and applications for binomial
coefficients, but this hack will focus on computing their value using
a different formula, Pascal's Rule\footnote{Which can be used to generate
Pascal's Triangle, \url{https://en.wikipedia.org/wiki/Pascals_triangle}}:
  $${n \choose k} = {n-1 \choose k} + {n-1 \choose k-1}$$
which is a recursive formula.  The base cases for Pascal's Rule are when
$k = 0$ and $n = k$.  In both cases, the value is 1.  When $k = 0$, we are
not choosing any elements and so there is only one way of doing that (i.e.\
choose nothing).  When $n = k$ we are choosing every element, again there
is only one way of doing that.

\subsection*{Writing a Naive Recursion}

Implement and test the following function \emph{using a recursive}
solution:

\mintinline{c}{long choose(int n, int k);}

which takes $n$ and $k$ and computes ${n\choose k}$ using Pascal's Rule.
Note that the return type is a \mintinline{c}{long}\footnote{For those
using Windows, you may need to instead use a \mintinline{c}{long long}
data type to get a 64-bit integer.} which is a 64-bit
integer allowing you to compute values up to
  $$2^{63}-1 = 9,223,372,036,854,775,807$$
(a little over 9 quintillion).  Write a \mintinline{c}{main} function
that takes $n$ and $k$ as command line arguments and outputs the result
to the standard output so you can easily test it.

\subsection*{Benchmarking}

Run your program on values of $n, k$ in Table \ref{table:easyValues}
and time (roughly) how long it takes your program to execute.  You
can check your solutions with an online tool such as
\url{https://www.wolframalpha.com/}.

\begin{table}[ht]
\centering
\begin{tabular}{c|c}
$n$ & $k$ \\
\hline\hline
4 & 2 \\
10 & 5 \\
32 & 16 \\  %5 seconds
34 & 17 \\  %15 seconds
36 & 18 \\ %60 seconds
\end{tabular}
\caption{Test Values}
\label{table:easyValues}
\end{table}

Now formulate an estimate of how long your program would take to
execute with larger values.  You can make a \emph{rough} estimate
how many function calls are made using the binomial value itself.
That is, to compute ${n \choose k}$ using Pascal's Rule would make
\emph{about} ${n \choose k}$ function calls.

Use the running time of your program from the test values to
estimate how long your program would run for the values in
Table \ref{table:hardValues}.

\begin{table}[ht]
\centering
\begin{tabular}{c|r}
${n \choose k}$ &  value \\
\hline\hline
${54 \choose 27}$ & =     1,946,939,425,648,112 \\
${56 \choose 28}$ & =     7,648,690,600,760,440 \\
${58 \choose 29}$ & =    30,067,266,499,541,040 \\
${60 \choose 30}$ & =   118,264,581,564,861,424 \\
${62 \choose 31}$ & =   465,428,353,255,261,088 \\
${64 \choose 32}$ & = 1,832,624,140,942,590,534 \\
${66 \choose 33}$ & = 7,219,428,434,016,265,740 \\
\end{tabular}
\caption{Larger Values}
\label{table:hardValues}
\end{table}

\subsection*{Improving Performance with Memoization}

You'll now improve your program's performance using memoization
to avoid unnecessary repeated recursive calls.

\begin{enumerate}
  \item Write code (either in the \mintinline{c}{main} function or
  using another ``entry point'' function) to create a memoization table
  containing \mintinline{c}{long} values of dimension $(n+1) \times (k+1)$
  \item Initialize the values in the table to $-1$ as a flag value
  to indicate that the value in the table has not yet been set.
  \item Using your previous recursive implementation as a guide, write
  a new recursive function that also takes the table as a parameter.
  When the function needs to compute ${n \choose k}$ it checks the table
  first: if the value has already been computed (is not $-1$) then it
  returns that value.  Otherwise, it performs the recursive computation.
  Before returning the value, however, it should store it (\emph{cache}
  it) in the table so that subsequent computations avoid the recursion.
  \item Modify your \mintinline{c}{main} function to use this more
  efficient version and re-test it with the values above.  Compare the
  time it took using memoization versus the naive recursion.
  \item Rerun your program with the values in Tables \ref{table:easyValues}
  and \ref{table:hardValues} to verify they work and note the difference
  in running time.
\end{enumerate}

\section*{Instructions}

\begin{itemize}

  \item Place all of your function definitions in a source file named
  \mintinline{text}{binomial.c} and hand it in with your header file,
  \mintinline{text}{binomial.h}.  Place your \mintinline{c}{main}
  function in a file named \mintinline{text}{binomialDemo.c}

  \item You are encouraged to collaborate any number of students
  before, during, and after your scheduled hack session.

  \item Include the name(s) of everyone who worked together on
  this activity in your source file's header.

  \item Turn in all of your files via webhandin, making sure that
  it runs and executes correctly in the webgrader.  Each individual
  student will need to hand in their own copy and will receive
  their own individual grade.
\end{itemize}

\end{document}


\lab{Hash Tables}{Hash Tables}
\label{lab:HashTables}
\objective{Empower students with basic knowledge of the fundamental, canonical data structures in order to understand performance and runtime characteristics.}

\section*{Hash Tables}
A \emph{hash table} is a very simple data structure that trades memory space for speed.
The key advantage of a hash table is this speed; most of the operations of a hash table execute very efficiently, independent of its size.
As such, hash tables have very fast lookup times; they form the underlying data structure for Python's set and dictionary types.

One of the key components of a hash table is a \emph{hash function}.
A hash function takes a piece data as an input and outputs a natural number which corresponds to the index for that piece of data.
Since the hash function must be executed to perform any operation on the hash table (e.g. insertion or lookup), it is imperative that the hash function execute quickly.
Thus, the heart of a hash table is a good hash function.

We begin defining our \li{HashTable} class, then, by allocating space within a one-dimensional array (in this case, a Python list) and by taking both the array size and the hash function as arguments.
\begin{lstlisting}
class HashTable(object):
    def __init__(self, capacity):
        # Efficiently extend a list to the designated hash table size
        self.hashtable = [None] * capacity
        # Keeps track of the number of elements within the hash table
        self.size = 0
        # Indicates the size of the hash table
        self.capacity = capacity

    def load_factor(self):
        # The load factor tells how saturated a hash table is
        return float(self.size)/self.capacity
\end{lstlisting}

A good hash function will execute quickly and distribute items uniformly throughout the table.
In the hash tables in this lab, we will use Python's built-in \li{hash} function.
This code demonstrates the principle steps of inserting into a hash function.
Instead of using our \li{HashTable} class, however, we simply use a similar construct to show how the steps might be implemented in a specific instance:
\begin{lstlisting}
>>> capacity = 5
>>> table = [None] * capacity
>>> hash('three')
8354070563160704301
>>> hash('three') % capacity
1
>>> table[1] = 'one'
>>> print table
[None, 'three', None, None, None]
>>> table[hash('four') % capacity] = 'four'
>>> table[hash('five') % capacity] = 'five'
>>> print table
['four', 'three', 'five', None, None]

# We can locate 'five' quickly
# Notice that is faster than the list's builtin methods
>>> %timeit table.index('five')
1000000 loops, best of 3: 190 ns per loops
>>> %timeit table[hash('five') % capacity]
10000000 loops, best of 3: 111 ns per loop
>>> table[hash('two') % capacity] = 'two'
# Note that our load factor is 4/5.
>>> print table
['four', 'three', 'five', 'two', None]

# When our load factor passes a certain threshold, we need to make the hash table bigger
# Resizing the hash table means rehashing everything in it
>>> def resize(table, new_cap):
        new_table = [None] * new_cap
        for i in table:
            new_table[hash(i) % new_cap] = i
        return new_table
>>> table = resize(table, 7)
>>> print table
[None, None, None, 'two', None, None, 'four']
\end{lstlisting}

The ideal hash function maps unique inputs to unique outputs that are uniformly distributed over the hash space.
However, it is extremely difficult to create an ideal hash function; most hash functions will experience hash collisions.
Hash collisions are when two unique inputs are mapped to the same index in the hash table (in our example above, resizing the hash table resulted in hash collisions that lost half the strings we stored in the table).
Fortunately, there are ways to handle hash collisions.

\paragraph{Probing.}
One method that we can use to resolve hash collisions is probing.
A short example will be succintly illustrate this method.
\begin{lstlisting}
>>> capacity = 5
>>> table = [None] * capacity
# Now we add the words 'one' and 'two'
# Notice what happens when we add 'two'
>>> table[hash('one') % capacity] = 'one'
>>> print table
[None, None, None, 'one', None]
>>> table[hash('two') % capacity]
'one'

# We can resolve the collision by looking for the next available index
# Starting at index 3, we look at index 3+1 % capacity, 3+2 % capacity, 3+3 % capacity, ..., until we find and empty index
# Seeing that index 4 is None, we store 'two' there
>>> table[4] = 'two'
>>> print table
[None, None, None, 'one', 'two']
\end{lstlisting}
This is called linear probing.
There are other variations of this idea which are more optimal.

\paragraph{Chaining.}
Another method for resolving hash collisions is chaining.
This method slightly alters the structure of our hash table, so that at each index, a list is stored.
When an item is mapped to an index, it is appeneded to the list at that index.
\begin{lstlisting}
>>> capacity = 5
>>> table = [list() for i in xrange(capacity)]

# Now we add the words 'one' and 'two'
# Notice what happens when we add 'two'
>>> table[hash('one') % capacity].append('one')
>>> print table
[[], [], [], ['one'], []]
>>> table[hash('two') % capacity].append('two')
>>> print table
[[], [], [], ['one', 'two'], []]
\end{lstlisting}

\begin{problem}
Using the concepts illustrated in the examples above, and an insert method to your \li{HashTable} class.
Don't forget to update your size for the number of elements within the hashtable!
Use chaining to handle hash collisions (namely, implement your insert method via chaining).

Then, create a method for your hash table to resize.
Implement this method such that, when your hash table exceeds a load factor of $.8$, it will resize the hash table so that the load factor is below $.3$.
Don't forget to rehash the elements already within your table!

Initialize a hash table of size 4 and insert the following felines: `lion', `tiger', `cheetah', `cougar', `colocolo', `cat', `clouded leopard', and `jaguar'.
Print your hash table.
\label{prob:hash_table}
\end{problem}

\begin{info}
Usually exisiting implementations of data structures are better to use.
The implementations of linked lists, trees, and hash tables in this lab were designed to familiarize you with the data structure's mechanics through example and practice.
In real situations, you will likely use Python's deques, sets, or dictionaries.
Python doesn't have a built-in implementation of trees, but there are several Python libraries that have highly optimized implementations of trees.

Performance tuning for data structures can be tricky even for professionals.
For this reason, it is better to avoid writing your implementation when one already exists.
\end{info}


\chapter{Radial Basis Function}

\section{Least Square Estimator}
Linear Regression Equations:
\begin{equation*}
\begin{split}
y &= \sum_{i=1}^{n} w_i f_i(\mathbf{x}) \\
&= w_1 f_1(\mathbf{x}) + w_2 f_2(\mathbf{x}) + \ldots + w_n f_n(\mathbf{x})  \\
\mathbf{y} &= \mathbf{Aw}
\end{split}
\end{equation*}
The squared error:
\begin{equation*}
\begin{split}
E &=\| \mathbf{y -d} \|^{2} \\
&= \| \mathbf{d - Aw} \|^{2} \\
&= \mathbf{(d-Aw)^{T}(d-Aw)} \\
&= \mathbf{d^{T}d - d^{T}Aw - (Aw)^{T}d + (Aw)^{T}(Aw)} \\
&= \mathbf{d^{T}d - 2w^{T}A^{T}d + w^{T}A^{T}Aw}
\end{split}
\end{equation*}
For least-square estimator:
\begin{equation*}
\begin{split}
\frac{\partial E}{\partial \mathbf{w}} &= 0 \\
2\mathbf{A^{T}d} - 2\mathbf{A^{T}Aw} &= 0 \\
\mathbf{A^{T}Aw} &= \mathbf{A^{T}d} \\
\mathbf{w} &= \mathbf{(A^{T}A)^{-1}A^{T}d}
\end{split}
\end{equation*}
If A is non-singular, $\mathbf{(A^{T}A)^{-1}A^{T}}$ is pseudoinverse of A. \\
If A is singular, pseudoinverse is same as inverse of matrix. \\
Singular matrix: Square matrix that is not invertible.

\section{Cover's Separability Theorem}
A complex pattern-classification problem cast in a high-dimensional hidden space non-linearly is more likely to be linearly separable than in a low-dimensional input space.

\section{Basis Function Network}
\begin{figure}[h]
\centering
\includegraphics[width=10cm]{chapter5_1}
\end{figure}
$$f(\mathbf{x}) = y = \sum_{i=1}^{N} w_i \phi_i(\mathbf{x})$$
The radial basis functions technique consists of choosing a basis function of the following form:
$$f(\mathbf{x}) = \sum_{i=1}^{N} w_i \phi(\| \mathbf{\mathbf{x-\mu_{i}}}\|)$$

\section{Typical Radial Basis Functions}
Piecewise linear approximation:
$$\phi(\| \mathbf{x} - \pmb{\mu_{i}}\|) = \| \mathbf{x} - \pmb{\mu_{i}} \|$$
Cubic approximation:
$$\phi(\| \mathbf{x} - \pmb{\mu_{i}}\|) = \| \mathbf{x} - \pmb{\mu_{i}} \|^{3}$$
Gaussian function:
$$\phi(\| \mathbf{x} - \pmb{\mu_{i}}\|) = exp(-\frac{\| \mathbf{x} - \pmb{\mu_{i}} \|^{2}}{2 \sigma^{2}})$$
Thin plate splines:
$$\phi(\| \mathbf{x} - \pmb{\mu_{i}}\|) = \| \mathbf{x} - \pmb{\mu_{i}} \|^{2} log(\| \mathbf{x} - \pmb{\mu_{i}} \|)$$
Multiquadratic function:
$$\phi(\| \mathbf{x} - \pmb{\mu_{i}}\|) = (\| \mathbf{x} - \pmb{\mu_{i}} \|^{2} + \sigma^{2})^{1/2}$$
Inverse multiquadratic function:
$$\phi(\| \mathbf{x} - \pmb{\mu_{i}}\|) = (\| \mathbf{x} - \pmb{\mu_{i}} \|^{2} + \sigma^{2})^{-1/2}$$

\section{RBF as Interpolation Networks}
In the interpolation approach,RBF technique consists of choosing a function such that:
\begin{equation}
f(\mathbf{x}) = y = \sum_{i=1}^{L} w_i \phi(\| \mathbf{x - x_{i}} \|^{2})
\label{interpolation}
\end{equation}
And the interpolation condition is:
\begin{equation}
f(\mathbf{x}) = \mathbf{d}
\label{interpolation_condition}
\end{equation}
Let:
\begin{equation}
\phi_{li} = \phi(\| \mathbf{x_l - x_i} \|)
\label{temp4}
\end{equation}
Substituting \ref{interpolation_condition} and \ref{temp4} to \ref{interpolation}
\begin{equation}
\begin{split}
\sum_{i=1}^{L} w_i \phi(\| \mathbf{x_l - x_{i}} \|^{2}) &= d_l \\
\sum_{i=1}^{L} w_i \phi_{li} &= d_l
\end{split}
\end{equation}
Rewrite the equation in matrix form:
$$
\begin{bmatrix}
\phi_{11}  & \cdots & \phi_{1N} \\
\vdots     & \ddots & \vdots \\
\phi_{N1}  & \cdots & \phi_{NN}
\end{bmatrix}
\begin{bmatrix}
w_1 \\
\vdots \\
w_N
\end{bmatrix} =
\begin{bmatrix}
d_1 \\
\vdots \\
d_N
\end{bmatrix}
$$
$$\pmb{\phi w = d}$$
Assuming \emph{Gram Matrix} $\phi$ is non-singular:
$$\pmb{w = \phi^{-1} d}$$
\textit{Note: The number of hidden nodes is equal to the number of data points}

\section{RBF as Approximation Networks}
In a generalized Gaussian RBF network, each hidden node represents a cluster of patterns.
\begin{equation}
\begin{split}
\phi(\| \mathbf{x} - \pmb{\mu_{i}}\|) &= exp\Big(-\frac{\| \mathbf{x} - \pmb{\mu_{i}} \|^{2}}{2 \sigma^{2}} \Big) \\
&= exp\Big(- \frac{1}{2} (\mathbf{x} - \pmb{\mu_i})^{T} \mathbf{C_i^{-1}} (\mathbf{x} - \pmb{\mu_i}) \Big)
\end{split}
\end{equation}
The centroid $\pmb{\mu}$ and covariance matrix $\mathbf{C}$:
$$\pmb{\mu_{i}} = \frac{1}{L} \sum_{j=1}^{L} \mathbf{x_j}$$
$$\mathbf{C_i} = \frac{1}{L-1} \sum_{j=1}^{L} (\mathbf{x_j} - \pmb{\mu_i})(\mathbf{x_j} - \pmb{\mu_i})^{T}$$
RBF use k-means clustering algorithm to learn centroids of hidden layer neurons.


\begin{enumerate}

\item If $\underline{s}$ is a maximum length LFSR sequence of an LFSR of length
$L$, then it has period $2^L-1$ (as covered in lectures). This means that the
LFSR goes through $2^L-1$ distinct states before repeating a state.
Specifically, these states are the members of $\mathbb{F}_2^L \setminus
{(0,\dots,0)}$. Also, the first bits of these states are the bits of the
sequence in a single period.

To check whether $\underline{s}$ is balanced, we just need to check whether the
first bits of the members of $\mathbb{F}_2^L \setminus {(0,\dots,0)}$ (the
states) are balanced. Clearly, there are $2^{L-1}$ states that start with 1 and
$2^{L-1}-1$ states that start with 0 (since including $(0,\dots,0)$ as a state
implies that it is the only state). Thus, the number of 0s and 1s differ by one
and therefore the sequence is balanced: given a large $L$, a randomly chosen bit
will be 0 or 1 each with probability approximately $1/2$.

\item In a maximum length LFSR, the state of the LFSR register runs through all
of the members of $\mathbb{F}_2^L \setminus {(0,\dots,0)}$ in a single period
(\cite{slides}, Slide 36). A run of length $i$ occurs at a time $t$ with state
$s_t$ if the preceding state $s_{t-1}$ (at time $t-1$) starts with a known
substring of $i+2$ bits (eg. (0, 1, \dots, 1, 0) - 1 repeated $i$ times - or (1,
0, \dots, 0, 1) (0 repeated $i$ times)). There are $2^{L-i-2}$ such states that
precede the start of a run of $i$ 0s or 1s. Accounting for both runs of ones and
of zeros, the total number of states that precede the start of any run is $2
\times 2^{L-i-2} = 2^{L-i-1}$.

Any state that precedes the start of a run either starts with 01 or 10. This
shows that approximately half of the states precede the start of a run: the half
that begins with 01 or 10 rather than 11 or 00. If the total number of states is
$tot_{states} = 2^L-1$ (excluding (0,\dots,0) ), then the number of states that
precede the start of a run is $tot_{runs} = 2^{L-1}$ - half of the bit strings
of length L.

This shows that the number of runs of length $k$ for $1 \le k \le L-1$ is

\[2^{L-1-k} = 2^{-k} \times 2^{L-1}\]

which means that $2^{-k}$ of all runs are of length k.

This breaks down when you consider runs of length L. However, there can only be
one run of length L: the run of L 1s. The run of L 0s is the 0 state, and is
never reached in a maximum-length LFSR whose initial state is non-zero, as only
non-zero states are found in such an LFSR (\cite{slides}, Slide 37).

\item We will prove the result with linear algebra.

\paragraph{Lemma:} Let $k$ be an integer with $1 \le k < 2^L-1$ and let $T$
be a linear transformation $\mathbb{F}^L_2 \to \mathbb{F}^L_2$ which
generates an m-sequence $(s_i)_{i=1}^\infty$. Then $\text{ker}(T^k+\mathbb{I})=\{0\}$.

\paragraph{Proof of Lemma:} Define $V$ as $\text{im}(T^k+\mathbb{I})$ and let
$c_T$ be the characteristic (and, in this case, minimal) polynomial of $T$.
Suppose this were not the case, that
$\text{dim ker}(T^k+\mathbb{I})>0$. Then $\text{dim}\ V<L$ by Rank-Nullity Theorem.
But $V$ is $T$ invariant, since, for $v \in \mathbb{F}_2^L$,
$T(T^k+\mathbb{I})v=(T^k+\mathbb{I})Tv$,
so let $P$ be the characteristic polynomial of $T$ restricted to $V$.
If $v$ is a vector in $\mathbb{F}^L_2$, then note $P(T)(T^k+\mathbb{I})v=0$,
by Cayley-Hamilton theorem on $V$. So $P(T)(T^k+\mathbb{I})=0$, and hence
$c_T(x) | P(x)(x^k+1)$ ($c_T$ is the minimal polynomial of the sequence
since it is irreducible). But $c_T$ is irreducible since $T$ generates an
m-sequence, and $\deg P<L=\deg c_T$, so $c_T(x) | x^k+1$, which implies
$k\ge 2^L-1$ (\cite{slides}, Slide 37), a contradiction. This concludes
the proof of the lemma.

\paragraph{Main Proof:} The $2^L-1$ $L$-vectors starting at each position
in an m-sequence consist of every non-zero $L$-vector. Since
$\text{ker}(T^k+\mathbb{I})=\{0\}$, these vectors are permuted in this map.
Specifically, there are $2^{L-1}-1$ vectors starting with $0$ and $2^{L-1}$ vectors
starting with $1$ in the image of this map. However, $s_x$ and $s_{x+k}$
disagree iff $s_{x+k}\oplus s_x=1$, or iff $T^k{\bf s_x}+{\bf s_x}$ begins with
$1$, where ${\bf s_x}$ is the $L$-vector starting with $s_x$ in the m-sequence.
However, the probability of $T^k{\bf s_x}+{\bf s_x}=(T^k+\mathbb{I}){\bf s_x}$
starting with $1$ is approximately $1/2$, since $T^k+\mathbb{I}$ maps about half the non-zero
$L$-vectors to vectors starting with $0$ and about half to vectors starting with
$1$. So the m-sequence
is approximately uncorrelated with itself shifted $k$ places whenever $k$ is not
a multiple of $2^L-1$.

\end{enumerate}