%TODOS FOR THIS LAB
%1.  Maybe merge mst and ImgSegMST

\lab{Algorithm}{Kruskal's and Prim's Algorithm}{Kruskal's and Prim's Algorithm}
\label{Ch:Kruskal}

\objective{Find a minimum spanning tree for a connected, weighted graph using Kruskal's Algorithm and Prim's Algorithm.}

\section*{Weighted Graphs and Spanning trees}

Remember that a graph is composed of two sets: a set of nodes and a set of edges that connect the nodes.

\begin{figure}[H]
\includegraphics[width = .4\textwidth]{graph1.pdf}
\caption{An example of an undirected graph}
\label{mst:graph1}
\end{figure}

A graph is directed if connections are uni-directional.
A graph is undirected if connections are bi-directional.
Figure 7.1 shows an example of an undirected graph.
A weighted graph is a graph where each edge has a value associated with it.
Usually these values represent some sort of cost or distance.
A connected graph is a graph where there is a path, or a set of edges, that connects every two nodes together.
We can write a matrix that describes this type of graph.
Each row of our matrix represents a starting point and each column represents a destination.
If an edge from one node to another exists, we put the weight of the edge.
If there is no edge, we put a 0.
For the above graph in Figure 7.1 we generate the following matrix:

\[
A = \begin{pmatrix}
0 & 1 & 0 & 0 & 0 & 0\\
1 & 0 & 1 & 0 & 0 & 1\\
0 & 1 & 0 & 1 & 1 & 1\\
0 & 0 & 1 & 0 & 1 & 0\\
0 & 0 & 1 & 1 & 0 & 1\\
0 & 1 & 1 & 0 & 1 & 0\\
\end{pmatrix}
\]

This matrix is called an adjacency matrix.
Note that since the graph is undirected, then this matrix is symmetric.
Now consider the graph in Figure 7.4.  This graph is the same as the graph in Figure 7.1, except now there is a weight attached to each edge.  The matrix for this graph is
\[
A = \begin{pmatrix}
0 & 3 & 0 & 0 & 0 & 6\\
3 & 0 & 5 & 0 & 0 & 4\\
0 & 5 & 0 & 1 & 1 & 5\\
0 & 0 & 1 & 0 & 2 & 0\\
0 & 0 & 1 & 2 & 0 & 4\\
6 & 4 & 5 & 0 & 4 & 0\\
\end{pmatrix}
\]

Another way to store the information from a graph is to make a list of the edges with their corresponding weights.
For an unweighted, undirected graph, this would just mean making a list of the pairs of nodes that correspond to each edge.
For a weighted, undirected graph, a third value could be added to the end of each edge representing the corresponding weight of that edge.
A list like this for the graph in Figure 7.1 would look like this:

\begin{align*}
[('A', 'B'),
 ('B', 'C'),
 ('B', 'F'),
 ('C', 'D'),\\
 ('C', 'E'),
 ('C', 'F'),
 ('D', 'E'),
 ('E', 'F')]
\end{align*}

For this lab, we will be focusing on undirected, weighted graphs.

A spanning tree of a connected, undirected graph $G$ is an undirected graph that contains all the nodes of $G$, a subset of the edges, and no cycles.
A cycle, for undirected graphs, is a path where you start and end on the same node without crossing any edge more than once.
The red in Figure 7.2 is an example of a cycle in an undirected graph.

\begin{figure}[H]
\includegraphics[width = .4\textwidth]{graph3.pdf}
\caption{A cycle in an undirected graph}
\label{mst:graph3}
\end{figure}

The minimum spanning tree (MST) of a weighted, undirected graph is a spanning tree where the total weight is less than or equal to the total weight of every other spanning tree.
Both Kruskal's and Prim's Algorithms are methods that find the minimum spanning tree of a weighted, undirected graph.
Figure 7.3 shows a spanning tree of the graph shown in Figure 7.1.
Figure 8.5 shows a minimum spanning tree of the graph shown in Figure 7.4.

\begin{figure}[H]
\includegraphics[width = .4\textwidth]{graph2.pdf}
\caption{A spanning tree with no cycles for the graph in Figure 7.1.}
\label{mst:graph2}
\end{figure}

\begin{figure}[H]
\includegraphics[width = .4\textwidth]{graph4.pdf}
\caption{A weighted, undirected graph}
\label{mst:graph4}
\end{figure}

\begin{figure}[H]
\includegraphics[width = .4\textwidth]{graph5.pdf}
\caption{The MST of the graph in Figure 7.4.}
\end{figure}

\section*{Kruskal's algorithm}

Given a weighted, directed graph G with $n$ nodes, Kruskal's algorithm finds a minimum spanning tree by first sorting the edges from smallest to largest.
Then starting with the smallest, the algorithm adds edges to the tree as long as the addition of each new edge does not create a cycle.
When $n-1$ edges have been added, the algorithm stops.

In order to avoid creating cycles while building the tree, it is necessary to keep track of which portions of the tree currently lie in connected groups.
This can be done by creating a dictionary where, when the algorithm starts, each node points to itself as the "root" of its own tree.
As you add edges to the tree you will change which nodes are the "root" nodes. This tracks which nodes are currently connected.
Doing this allows us to run the algorithm by iterating over the edges by weight in ascending order, and adding them to the tree if they connect nodes which are not already connected by the current tree.

Consider the graph in Figure 7.4 and apply Kruskal's algorithm.
First, we initialize our tree to be an empty list: \li{[]}.
To track the root nodes of each tree, we will also initialize a dictionary where each node points to itself.
It will look like this: \li{\{A:A, B:B, C:C, D:D, E:E, F:F\}}.
Since there are 6 nodes in the graph, we will continue until we have 5 edges in the tree.
Next, sort the edges by weight.
The result is something like the list \li{[(C, D, 1), (C, E, 1), (D, E, 2), (A, B, 3), (B, F, 4), (E, F, 4), (B, C, 5), (C, F, 5), (A, F, 6)]}.

Now we begin iterating through the edges to build the tree.
The first edge in the list is \li{(C,D,1)}.
The root for \li{C} is \li{C} and the root for \li{D} is \li{D}, so we add this edge to the tree.
The tree becomes \li{[(C, D, 1)]}.
We then change the root node of \li{D} to be \li{C}.
The dictionary of root nodes now looks like \li{\{A:A, B:B, C:C, D:C, E:E, F:F\}}.

Now we process the next edge, \li{(C, E, 1)}.
The root node of \li{C} is \li{C}, and the root node of \li{E} is \li{E}, so adding this edge does not create a cycle.
We add this edge into the tree, so the tree is now \li{[(C, D, 1), (C, E, 1)]}.
Then we change the root of \li{E} so that it is \li{C}, so the dictionary is \li{\{A:A, B:B, C:C, D:C, E:C, F:F\}}.

The next edge is \li{(D, E, 2)}.
The root node of the tree containing \li{D} is \li{C} and the root node of the tree containing \li{E} is \li{C}, so these nodes are already connected, so we do not add this edge to the tree.

The next edge is \li{(A, B, 3)}.
The root node for \li{A} is \li{A}, and the root node for \li{B} is \li{B}, so we add the edge to the tree and update the dictionary.
The dictionary becomes \li{\{A:A, B:A, C:C, D:C, E:C, F:F\}}

The next edge is \li{(B, F, 4)}.
The root node for \li{B} is \li{A} and the root node for \li{F} is \li{F}, so we add the edge to the tree and change the root node of \li{F} to be \li{A}.
The dictionary becomes \li{\{A:A, B:A, C:C, D:C, E:C, F:A\}}

The next edge is \li{(E, F, 4)}.
The root node for \li{E} is \li{C} and the root node for \li{F} is \li{A}.
We add the edge to the tree and end the algorithm since there are now 5 edges in the tree.
(If we were to continue the algorithm, we would update the root node of \li{A} to be \li{C}.)

Notice how we updated the root of the root node to one of the nodes in the edge added.
We manage the root nodes in this manner in order to find the root node by tracing back through the dictionary until we find a node that points to itself.
For example, to find the root node of \li{F} we first need to get the value for \li{F} from the dictionary.  This is \li{A}.
Then we get the value for \li{A} from the dictionary which is \li{C}.
Since the value of \li{C} in the dictionary is itself, the \li{C} is the root node of the graph containing \li{F}.
It is necessary to trace through the dictionary in this manner each time because we avoid iterating over all of the nodes and updating their root values everytime.

Below is the pseudocode for the algorithm as we have described it above:

\begin{itemize}

% Note: The comments in the solutions match the pseudocode.
% When making changes, please keep that in mind.

\item Initialize an empty list of edges for the minimum spanning tree.

\item Make a dictionary that points each node toward its root (not always directly to it).
Start with each node pointing to itself.

\item Initialize the number of nodes that still need to be processed to the number of nodes minus 1.

\item Define a helper function that, given a node, traces through the dictionary to find the root of its tree.
This can be done like this:

	\begin{itemize}

	\item Initialize a temporary variable to be the node for which we are finding the root.

	\item While the temporary node does not point to itself in the dictionary:

		\begin{itemize}

		\item Update the temporary node to be the node it currently points to in the dictionary.

		\end{itemize}

	\item Return the temporary node.

	\end{itemize}

\item Iterate over the edges by ascending weight.
Use a \li{for} loop for this and return the tree when it is big enough which breaks the loop for you.

	\begin{itemize}

	\item Trace through the dictionary to find the root node of each of the nodes in the edge you are processing.

	\item If the roots are not the same (i.e. if adding the edge doesn't form a cycle):

		\begin{itemize}

		\item Add the edge to the tree.

		\item Lower the number of edges remaining by one.

		\item If the number of edges remaining is 0, return the tree (which also breaks the loop).

		\item Update the root of the root of the second node in the edge to be the root of the first node in the edge.
			This lets us record that the two subtrees are now connected.

		\end{itemize}

	\end{itemize}

\end{itemize}
Note on implementation: You can iterate over a sorted copy of a list using the built in \li{sorted} function.
You can sort by the third value in each tuple using the \li{itemgetter} function that is part of the \li{operator} library included with Python.
For example:
\begin{lstlisting}
from operator import itemgetter
...
for n1, n2, weight in sorted(edges, key=itemgetter(2)):
    ...
\end{lstlisting}

\begin{problem}
Implement Kruskal's algorithm.
Test your algorithm on random symmetric arrays.
You can generate a random symmetric array by multiplying a random array by its transpose.
Use the data from MSTdata.npy to test your tree.
Use np.load("MSTdata.npy") to get it.
Use the \li{formChanger} function below to put it in the right form.
\begin{lstlisting}
def formChanger(oldData):
    newData=[]
    for i in oldData: newData.append((i[0],i[1],int(i[2])))
    return newData
\end{lstlisting}
\end{problem}
\section*{Prim's algorithm}

Prim's is a similar algorithm for finding minimum spanning trees.
While it is much slower than Kruskal's algorithm for sparse graphs, it is much faster for dense graphs because Prim's algorithm avoids sorting the edges.

Again, consider the example shown in Figure 7.4.
We first initialize a dictionary with all the nodes as keys in order to track which nodes have not been processed.
At the beginning it will be \li{\{A:False, B:False, C:False, D:False, E:False, F:False\}}.
We then form the dictionary that maps each node to the edges that contain it.
Since we will already know one node of each edge while we use the dictionary, we only need to store the node that is not being looked up.
It should end up looking like \li{\{A:[(B, 3), (F, 6)], B:[(A, 3), (C, 5), (F, 4)], C:[(B, 5), (D, 1), (E, 1), (F, 5)], D:[(C, 1), (E, 2)], E:[(C, 1), (D, 2), (F, 4)], F:[(A, 6), (B, 4), (C, 5), (E, 4)]\}}.
We will also initialize an empty dictionary to track the shortest edges that run between nodes we have processed and nodes that we haven't.
As we iterated over the edges in our initialization step, we are also able to find the shortest edge.
In this case, it is \li{(D, C, 1)}.
Let's start with \li{(D, C, 1)} and initialize our tree as the list \li{[(D, C, 1)]}.
Next, we mark \li{D} and \li{C} as processed in the dictionary that tracks which nodes have been processed.
We now start to build our dictionary of nodes that are one edge away from our processed nodes.
In this case, after adding the shortest edges between processed and unprocessed nodes to the dictionary, the dictionary becomes \li{\{B:(C, B, 5), E:(C, E, 1), F:(C, F, 5)\}}.
Notice we did not include \li{E:(D, E, 2)} because there is a shorter edge to \li{E} from \li{C}.

Of the edges in the dictionary of edges that can be processed next, the shortest is \li{(C, E, 1)}, so we add that edge to the tree and mark \li{E} as processed.
With \li{E} being processed, we can now reach \li{F} at a cost of 4.
After making this change, the dictionary of edges to process becomes \li{\{B:(C, B, 5), E:(C, E, 1), F:(E, F, 4)\}}.
Since \li{E} no longer needs to be processed, we can remove it from consideration.
So this dictionary now becomes \li{\{B:(C, B, 5), F:(E, F, 4)\}}.

Of the edges to be processed next, \li{(E, F, 4)} is the shortest, so we add it to the tree and mark \li{F} as processed.
After performing the appropriate modifications to the dictionary of edges to be processed next, it becomes \li{\{B:(F, B, 4), A:(F, A, 6)\}}.

Of the edges to be processed next, \li{(F, B, 4)} is the shortest, so we add it to the tree and mark \li{B} as processed.
After performing the appropriate modifications to the dictionary of edges to be processed next, it becomes \li{\{A:(B, A, 3)\}}.

The only edge to be considered is \li{(B, A, 3)}, so we add it to the tree.
The tree is long enough that it it spans the nodes, so the algorithm is finished.

Here's pseudocode for a version of Prim's algorithm.
While it is not a perfectly optimized version, it is pretty good.

\begin{itemize}

% Note: The comments in the solutions match the pseudocode.
% When making changes, pleas keep that in mind.

\item Initialize a dictionary to track which nodes have been processed.

\item Initialize an empty dictionary of lists to track the edges containing each node.

\item Fill the edge list.
	Be sure to add each edge to the list corresponding to both of its nodes.

\item Get the first edge to add (The shortest edge from any given node is a good pick).

\item Mark the nodes in the first edge as processed.

\item Initialize the tree to be the list containing the first edge.

\item Initialize an empty dictionary that will be used to contain the edges that can be processed next.

\item Define a helper function to insert an edge into the dictionary (if that insertion is needed).
	This can be done as follows:

	\begin{itemize}

	\item Get the value of the node that is reached by the edge.

	\item If that node isn't in the dictionary, set its value to be the edge passed to the functions.

	\item If it is in the dictionary already, set its value to be the shorter of the edges being processed and the edges already in the dictionary.

	\end{itemize}

\item Use the helper function to insert the edges reached by the first two processed nodes into the dictionary of edges to be processed.

\item Until the tree contains enough edges to span all the nodes:

	\begin{itemize}

	\item Find the shortest edge in the dictionary of edges to be processed.

	\item Remove the shortest edge from the dictionary.

	\item Add it to the tree.

	\item Mark the node reached by the new edge as processed.

	\item Use the helper function to insert the edges reached by the newly processed node into the dictionary of edges to be processed.

	\end{itemize}

\item Return the completed tree.

\end{itemize}

\begin{problem}
Write a Python function that uses Prim's algorithm to find the minimum spanning tree of a graph.
Test your implementation with the same data as the previous problem.
Compare the speed of Prim's algorithm with the speed of Kruskal's algorithm.
Create a function which prints these two times.
\end{problem}

Many graph packages have functions for finding the minimal spanning tree. One of these is NetworkX. You begin as follows
\begin{lstlisting}
import networkx as nx
G=nx.Graph()
\end{lstlisting}
You can add a node $x$ with \li{G.add_node(x)} and add a edge from $x$ to $y$ with weight n with \li{G.add_edge(x,y,weight=n)}. Then \li{nx.minimum_spanning_tree(G)} returns the minimal spanning tree using Kruskals algorithm.

\begin{problem}
Use NetworkX to find the minimal spanning tree of the data in MSTdata.npy.
Compare the output with that of your algorithm.
Create a function which prints your comparison.
\end{problem}

%Add specifications

\section*{Image Segmentation}

%Lab \ref{MSTImgSeg}

One application of Minimal Spanning Trees (MSTs) is image segmentation.
Kruskal's algorithm is especially good at this.
You can convert an image into graph. Each pixel is a vertex and you define weights between the nodes. You then build the minimal spanning tree. If you take away the edge with the greatest weight you will get two forests (removing an edge from any graph with no cycles will create two forests). Then you can continue doing this to each forest to split up those forests. Each forest corresponds to a segment of the image.
Let $k$ be the number of divisions that is wanted and $n$ be the number of nodes.
Kruskal's algorithm is performed until $n-(k+1)$ edges are added which is the same as taking out the $k$ edges with greatest weights.

There are many different ways to turn an image into a graph and weight the edges.
A simple, yet effective, version is to make every pixel a node and the edges are the difference in intensities in the four cardinal directions.

This means that there are less than $4n$ edges.
Other image segmentation algorithms have to use $n^2$ space.
This gives the MST algorithm a critical advantage over other image segmentation algorithms.

\begin{problem}
Write a function that takes a black and white image as input and outputs a list of the edges and a list of nodes.
Store the edges using the form \li{(node,node,weight)}.
\end{problem}

The provided Kruskal's algorithm takes as inputs the list of nodes, the list of edges and the number of divisions desired.
The number of divisions often has to be higher than the actual number that is needed because sometimes one or two pixels form a division because the difference between them and the pixels around them is so great.
You will have to adjust the number of divisions until the desired result is found.  See Figure 9.1.

\begin{problem}
Perform the image segmentation algorithm on the image, then graph the original image and the three largest divisions.
(Use the Counter class from collections to find the number of pixels in each division.)
\end{problem}

\begin{problem}
Make a division of the image a different color.
\end{problem}

\vfill
\begin{figure}[ht]
\begin{minipage}[b]{0.47\linewidth}
\centering
\includegraphics[width=\textwidth]{MSTseg1.jpg}
\end{minipage}
\hspace{0.5cm}
\begin{minipage}[b]{0.47\linewidth}
\centering
\includegraphics[width=\textwidth]{MSTseg2.jpg}
\end{minipage}
\begin{minipage}[b]{0.47\linewidth}
\centering
\includegraphics[width=\textwidth]{MSTseg3.jpg}
\end{minipage}
\hspace{0.5cm}
\begin{minipage}[b]{0.47\linewidth}
\centering
\includegraphics[width=\textwidth]{MSTseg4.jpg}
\end{minipage}
\caption{The original image is in the top left hand corner. The three larges segments are shown in the other corners. The original image was 498x498 and 50000 divisions were used.}
\end{figure}
\vfill


\section{Yuyan}

Please present main results on manifold ...

\subsection{Manifold Learning}
Suppose there a dataset $\mathbf X =(\mathbf x_1,...,\mathbf x_n)$ and each $\mathbf x_i\in \mathbb R^d$. In many  practical problem, the $\mathbf x_i$ locates on a low dimension manifold $\mathcal M$ in $\mathbb R^d$. The goal of Manifold Learning is to find the parameter coordinate of $\mathcal M$, so we can present the points $\mathbf x_1,...,\mathbf x_n$ in a essential form.
\subsubsection{Isomap}
The main idea of isomap is to keep the geodesic distance of $\mathcal M$ during dimensionality reduction.

The Figure \ref{fig:isomapmnist} is the effect of Isomap for MNist, it shows different digit in the MNist dataset is separable.
%\begin{figure}
	%\centering
	%\includegraphics[width=0.7\linewidth]{../figures/IsomapMnist}
	%\caption{The effect of Isomap for MNist}
	%\label{fig:isomapmnist}
%\end{figure}


\section{Generic Time Triggered Cyclic Executive Scheduler}
Let $f$ denote the frame length, $P$ the full period, $D(k)$ the relative deadline of task $k$,
$C(k)$ its execution time, and $p(k)$ the period of task $k$ (how often it occurs).
Then the following conditions have to be satisfied:
\begin{itemize}
	\item $\forall k . f \leq p(k)$ (at most one execution within a frame)
	\item $P = \lcm_k\left(p(k)\right)$
	\item $\forall k . f \geq C(k)$ (processes start and complete within single frame)
	\item $\forall k . 2f - \gcd\left(p(k),f\right) \leq D(k)$ (between release time and deadline of every task there is at least one frame boundary)
\end{itemize}


\section{Weight biased leftist heaps}\label{sec:wblh}

A heap is a tree-based data structure used to implement priority queue. Each node in a heap satisfies \textit{priority property}: priority of element at the node is not lower than priority of the children nodes\footnote{I will also refer to children of a node as ``subtrees'' or ``subheaps''.}. Therefore element with the highest priority is stored at the root. Access to it has $O(1)$ complexity.

Weight biased leftist tree \cite{ChoSah96} is a binary tree that satisfies \textit{rank property}: for each node rank of its left child is not smaller than rank of its right child. Rank of a tree is defined as its size (number of nodes). Weight biased leftist tree that satisfies priority property is called a weight biased leftist heap.

\textit{Right spine} of a node is the rightmost path from that node to an empty node. From priority property it follows that right spine of a weight biased leftist heap is an ordered list (in fact, any path from root to a leaf is!). Two weight biased leftist heaps can be merged in $O(\log n$) time by merging their right spines in the same way one merges ordered lists and then swapping children along the right spine of merged heap to restore rank property \cite{Oka99}. Inserting new element into weight biased leftist heap can be defined as merging existing heap with a newly created singleton heap (ie. a heap storing exactly one element). Deleting element with the highest priority can be defined as merging children of the root element.


\lab{Image Segmentation}{Image Segmentation}

\objective{Understand some basic applications of eigenvalues to graph theory.  Learn how to calculate the Laplacian matrix of a graph.  Apply the Laplacian matrix to determine connectivity of a graph and segment an image.}
\label{lab:ImgSeg_eigenvalues}

\section*{Graph Theory}
\begin{figure}

 \begin{tikzpicture}[auto,node distance=1.5cm,
 thick,main node/.style={circle,draw}]

  \node[main node] (5) [] {6};
  \node[main node] (2) [below right of=5] {3};
  \node[main node] (3) [above right of=5] {4};
  \node[main node] (4) [right of=3] {5};
  \node[main node] (1) [right of=2] {2};
  \node[main node] (0) [below right of=4] {1};

  \foreach \s/\t in {5/3, 3/4, 4/0, 0/1, 1/2, 2/3, 1/4, 5/0} {
   \path[draw] (\s) edge (\t);}
\end{tikzpicture}
\caption{An undirected graph that is connected.}
\label{fig:example_graph}
\end{figure}

\begin{figure}

 \begin{tikzpicture}[auto,node distance=2cm,
 thick,main node/.style={circle,draw}]

  \node[main node] (0) [] {1};
  \node[main node] (1) [below of=0] {2};
  \node[main node] (2) [right of=0] {3};
  \node[main node] (3) [below of=2] {4};
  \node[main node] (4) [right of=2] {5};
  \node[main node] (5) [right of=3] {6};

   \path[draw] (0) edge node [left] {3} (1);
   \path[draw] (2) edge node [left] {1} (3);
   \path[draw] (4) edge node{-1} (5);
   \path[draw] (3) edge node{2} (4);
   \path[draw] (3) edge node [below]{.5} (5);
\end{tikzpicture}
\caption{A weighted undirected graph that is not connected.}
\label{fig:example_graph2}
\end{figure}

% \begin{tikzpicture}[auto,node distance=1.5cm,
% thick,main node/.style={circle,draw}]
%
%  \node[main node] (2) [] {2};
%  \node[main node] (1) [below left of=2] {1};
%  \node[main node] (0) [below right of=2] {0};
%
%  \foreach \s/\t in {1/1, 1/2, 1/3, 2/3} {
%   \path[draw] (\s) edge (\t);}
%\end{tikzpicture}
%\caption{An undirected graph that is not simple.}
%\label{fig:example_graph}
%\end{figure}

%\begin{figure}
% \begin{tikzpicture}[->,>=stealth',shorten >=1pt,auto,node distance=1.5cm,
% thick,main node/.style={circle,draw}]
%
%  \node[main node] (A) [] {A};
%  \node[main node] (B) [below of=A] {B};
%  \node[main node] (C) [right of=A] {C};
%  \node[main node] (D) [below of=C] {D};
%  \node[main node] (E) [right of=C] {E};
%  \node[main node] (F) [right of=D] {F};
%
%  \foreach \s/\t in {A/C, B/A, B/D, C/E, C/F, D/C} {
%   \path[draw] (\s) edge (\t);}
%\end{tikzpicture}
%\caption{A simple directed graph}
%\end{figure}

Graph theory is a branch of mathematics dealing with mathematical structures called graphs.  Graphs represent relationships between objects.
%For example, the transition diagram in Figure \ref{fig:markov1} in Lab \ref{lab:EigSolve} shows the relationship between states in a Markov chain.
An \emph{undirected graph} is a set of nodes (or vertices) and edges, where each edge connects exactly two nodes (see Figure \ref{fig:example_graph}).
In a \emph{directed graph}, edges are directional. Each edge only goes one way, usually visualized as an arrow pointing from one node to another.
In this lab, we will only consider undirected graphs, which we will simply call graphs (unless we wish to emphasize the fact that they are undirected).

%A graph is \emph{simple} if no edge connects a node to itself.
%The graph in Figure [TODO!] is simple, but the graph in Figure [TODO!] is not.

A \emph{weighted} graph is a graph with a weight attached to each edge.
For example, a weighted graph could represent a collection of cities with roads connecting them.
The vertices would be cities, the edges would be roads, and weight of an edge would be the length of a road.
Such a graph is depicted in Figure \ref{fig:example_graph2}.

Any unweighted graph can be thought of as a weighted graph by assigning a weight of 1 to each edge.

\subsection*{Adjacency, Degree, and Laplacian Matrices}
We will now introduce three matrices associated with a graph.
Throughout this section, assume we are working with a weighted undirected graph with $N$ nodes, and that $w_{ij}$ is the weight attached to the edge connecting node $i$ and node $j$.
We first define the adjacency matrix.

\begin{definition} The \emph{adjacency matrix} is an $N \times N$ matrix whose $(i,j)$-th entry is
\begin{center}
	$ \begin{cases}  w_{ij} & \mbox{if an edge connects node i and node j} \\ 0 & \mbox{otherwise.} \end{cases}$
\end{center}
% If the graph is not simple, there are differing conventions for how to define the diagonal of the adjacency matrix.
\end{definition}

For example, the graph in Figure \ref{fig:example_graph} has the adjacency matrix $A_1$ and the graph in Figure \ref{fig:example_graph2} has the adjacency matrix $A_2$, where
\[
A_1 = \begin{pmatrix}
0 & 1 & 0 & 0 & 1 & 1\\
1 & 0 & 1 & 0 & 1 & 0\\
0 & 1 & 0 & 1 & 0 & 0\\
0 & 0 & 1 & 0 & 1 & 1\\
1 & 1 & 0 & 1 & 0 & 0\\
1 & 0 & 0 & 1 & 0 & 0
\end{pmatrix} \qquad A_2 =
 \begin{pmatrix}
0 & 3 & 0 & 0 & 0 & 0\\
3 & 0 & 0 & 0 & 0 & 0\\
0 & 0 & 0 & 1 & 0 & 0\\
0 & 0 & 1 & 0 & 2 & .5\\
0 & 0 & 0 & 2 & 0 & -1\\
0 & 0 & 0 & .5 & -1 & 0
\end{pmatrix}.
\]
Notice that these adjacency matrices are symmetric. This will always be the case for undirected graphs.

\begin{comment}
Raising the adjacency matrix to a power yields some very interesting information.
We can discover the number of paths of length $n$ between two nodes by raising a graph's adjacency matrix to the $n$th power.
For example, by squaring $A$, we can find the number of paths of length two between every pair of nodes.
\begin{lstlisting}
>>> A = np.array([[0,1,0,0,1,0],[1,0,1,0,1,0],
                  [0,1,0,1,0,0],[0,0,1,0,1,1],
                  [1,1,0,1,0,0],[0,0,0,1,0,0]])

>>> np.linalg.matrix_power(A,2)
array([[2, 1, 1, 1, 1, 0],
       [1, 3, 0, 2, 1, 0],
       [1, 0, 2, 0, 2, 1],
       [1, 2, 0, 3, 0, 0],
       [1, 1, 2, 0, 3, 1],
       [0, 0, 1, 0, 1, 1]])
\end{lstlisting}
We can see that no paths of length two exist between node 0 and node 5 because $A^2_{0,5} = 0$.
By calculating $A^6$ we can find the number of paths of length six from node 3 to itself.
\begin{lstlisting}
>>> np.linalg.matrix_power(A, 6)
array([[45, 54, 38, 45, 54, 16],
       [54, 86, 29, 77, 51, 11],
       [38, 29, 55, 15, 70, 27],
       [45, 77, 15, 75, 31,  4],
       [54, 51, 70, 31, 93, 34],
       [16, 11, 27,  4, 34, 14]])
\end{lstlisting}
We see that there are 75 unique paths of length six from node 3 to itself.
Imagine trying to count all of those paths by hand!
It would be very easy to count incorrectly.
This method makes it very simple to count paths without mistakes.

Adjacency matrices can also be composed of \li{True} and \li{False} values.
In this case, the $n$th power of such a matrix (using boolean arithmetic)
is again a matrix of
boolean values which simply indicate whether there exists a path of length $n$ between the given pair of nodes, rather than indicating the number of such
paths.

\begin{problem}
Let the following matrix represent a directed graph
\[
\begin{pmatrix}
0 & 0 & 1 & 0 & 1 & 0 & 1 \\
1 & 0 & 0 & 0 & 0 & 1 & 0 \\
0 & 0 & 0 & 0 & 0 & 1 & 0 \\
1 & 0 & 0 & 0 & 1 & 0 & 0 \\
0 & 0 & 0 & 1 & 0 & 0 & 0 \\
0 & 0 & 1 & 0 & 0 & 0 & 1 \\
0 & 1 & 0 & 0 & 0 & 0 & 0
\end{pmatrix}
\]
Between which pair of nodes does there exist the greatest number of paths
of length five?
From which node to which node is there no path of length seven?
\end{problem}
\end{comment}

The second matrix is the degree matrix.
\begin{definition} The \emph{degree matrix} is an $N \times N$ diagonal matrix whose $(i,i)$-th entry is
\[
\sum_{j=1}^N w_{ij}.
\]
This quantity is the sum of the weight of each edge leaving node $i$.
\end{definition}
%For a directed graph, each node has an \emph{out-degree} (the number of edges directed away from a node) and an \emph{in-degree} (the number edges directed toward a node).
We call the $(i, i)$-th entry of the degree matrix the \emph{degree} of node $i$. As an example, the degree matrices of the graphs in Figures \ref{fig:example_graph} and \ref{fig:example_graph2} are $D_1$ and $D_2$, respectively.

\[
D_1 = \begin{pmatrix}
3 & 0 & 0 & 0 & 0 & 0\\
0 & 3 & 0 & 0 & 0 & 0\\
0 & 0 & 2 & 0 & 0 & 0\\
0 & 0 & 0 & 3 & 0 & 0\\
0 & 0 & 0 & 0 & 3 & 0\\
0 & 0 & 0 & 0 & 0 & 2
\end{pmatrix}. \qquad D_2 =
 \begin{pmatrix}
3 & 0 & 0 & 0 & 0 & 0\\
0 & 3 & 0 & 0 & 0 & 0\\
0 & 0 & 1 & 0 & 0 & 0\\
0 & 0 & 0 & 3.5 & 0 & 0\\
0 & 0 & 0 & 0 & 1 & 0\\
0 & 0 & 0 & 0 & 0 & -.5
\end{pmatrix}
\]

Finally, we can combine the degree matrix and the adjacency matrix to get the Laplacian matrix.
% Wikipedia defines the Laplacian of a simple graph only. I don't know why.
% The graph in our application is NOT simple.
% However, the non-simple parts cancel out, meaning that the Laplacian of the graph is the same as if you removed all self edges and then computed the Laplacian.
% So I just define the Laplacian this way and don't talk about simple graphs.
\begin{definition}
The \emph{Laplacian matrix} of a graph is
\[D - A \]
where $D$ is the degree matrix and $A$ is the adjacency matrix of the graph.
\end{definition}

For example, the Laplacian matrix of the graphs in Figures \ref{fig:example_graph} and \ref{fig:example_graph2} are $L_1$ and $L_2$, respectively, where

\[
L_1 = \begin{pmatrix}
3 & -1 & 0 & 0 & -1 & -1\\
-1 & 3 & -1 & 0 & -1 & 0\\
0 & -1 & 2 & -1 & 0 & 0\\
0 & 0 & -1 & 3 & -1 & -1\\
-1 & -1 & 0 & -1 & 3& 0\\
-1 & 0 & 0 & -1 & 0 & 2
\end{pmatrix}. \qquad L_2 =
 \begin{pmatrix}
3 & -3 & 0 & 0 & 0 & 0\\
-3 & 3 & 0 & 0 & 0 & 0\\
0 & 0 & 1 & -1 & 0 & 0\\
0 & 0 & -1 & 3.5 & -2 & -.5\\
0 & 0 & 0 & -2 & 1 & 1\\
0 & 0 & 0 &- .5 & 1 & -.5
\end{pmatrix}
\]

In this lab we will learn about graphs by studying their Laplacian matrices.
While the Laplacian matrix seems simple, we can learn surprising things from its eigenvalues.

\begin{problem}
Write a function that accepts the adjacency matrix of a graph as an argument and returns the Laplacian matrix.
Test your function on the graphs in Figures \ref{fig:example_graph} and \ref{fig:example_graph2}.

Hint: You can compute the diagonal of the degree matrix in one line by summing over an axis (see Lab \ref{lab:NumpyIntro}).
\label{prob:laplacian}
\end{problem}

\subsection*{Connectivity: First Application of Laplacians}

A \emph{connected graph} is a graph where every vertex is connected to every other vertex by at least one path.
The graph in Figure \ref{fig:example_graph} is connected, whereas the graph in Figure \ref{fig:example_graph2} is not.
It is often important to know if a graph is connected.
A naive approach to determine connectivity of a graph is to search every possible path from each vertex.
While this works for very small graphs, most interesting graphs will have thousands of vertices, and for such graphs this approach is not feasible.

Instead of the naive approach, we can use an interesting result from algebraic graph theory.
This result relates the connectivity of a graph to its Laplacian.

The Laplacian of any graph always has at least one zero eigenvalue.
Why is this true?  If $L$ is the Laplacian matrix of a graph, then the rows of $L$ must sum to 0.
(Think about how L was created.)
Since this is true, L cannot have full rank, so $\lambda = 0$ must be an eigenvalue of $L$.

Furthermore, if $L$ represents a graph that is \textit{not} connected, more than one of the eigenvalues of $L$ will be zero.
To see this, let $J \subset \{1 \dots N\}$ such that the vertices $\{v_j\}_{j \in J}$ form a connected component of the graph.
Define a vector $\x$ such that
 \[
    \x_j = \begin{cases}
        1, & j \in J \\
        0, & j \not\in J
        \end{cases}
  \]

Then $\x$ is an eigenvector of $L$ corresponding to the eigenvalue $\lambda = 0$.
(Look at the Laplacian matrix in Figure \ref{fig:example_graph2} and consider the product $L_2 \x$.)
In other words, for each connected component, 0 appears at least once as an eigenvalue.

In fact, it can be rigorously proven that the number of zero eigenvalues of the Laplacian exactly \textit{equals} the number of connected components.
If we can solve for the eigenvalues of $L$, this makes it simple to calculate how many connected components are in the graph.

$L$ is always a positive semi-definite matrix, so all of its eigenvalues are greater than or equal to 0.
The second smallest eigenvalue of $L$ is known as the \textit{algebraic connectivity} of the graph.
It is clearly 0 for non-connected graphs.
For connected graphs, the algebraic connectivity can give us useful information about the sparsity or ``connectedness'' of a graph.
A higher algebraic connectivity indicates that the graph is more strongly connected.

\begin{comment}
Thus, it will have real eigenvalues.
Surprisingly, a graph is connected if the second smallest eigenvalue of its Laplacian matrix is positive.  By second smallest, we mean the second eigenvalue when they are ordered smallest to largest.  The eigenvalues of the Laplacian matrix are never negative; thus, if the second one is not zero, the graph is connected.
In many applications, the Laplacian matrix is sparse, so by taking advantage of this sparsity, we can cheaply determine if a graph is connected.
\end{comment}

\begin{problem}
\leavevmode
Compute the number of connected components in a graph.
Write a function that accepts the adjacency matrix of a graph and returns two arguments: the number of connected components, and the algebraic connectivity of the graph (second smallest eigenvalue of the Laplacian).

Use the \li{scipy.linalg} package to compute the eigenvalues.
Note that this package will return complex eigenvalues (with negligible imaginary parts). Keep only the real parts.
Your function should also accept a tolerance value, such that all eigenvalues less than this value are assumed to be zero.
This should default to \li{tol=1e-8}.

\begin{comment}
%Old part of problem, removed because no great way to evaluate it
Here is a function that creates a random symmetric matrix of Boolean values with sparsity determined by the input \li{c}.
\begin{lstlisting}
def sparse_generator(n, c):
    ''' Return a symmetric nxn matrix with sparsity determined by c.
    Inputs:
        n (int): dimension of matrix
        c (float): a float in [0,1]. Larger values of c will produce
             matrices with more entries equal to zero.
    '''
    A = np.random.rand(n**2).reshape((n, n))
    A = ( A > c**(.5) )
    return A.T.dot(A)
\end{lstlisting}

Test your function on matrices created by \li{sparse_generator} with inputs $n = 10, 100$ and $c = .25, .5, .95$.
What do you notice about the likelihood that a random graph is connected?
\end{comment}

\end{problem}

\section*{Image Segmentation: Second Application of Laplacians}

\begin{figure}
    \centering
    \begin{subfigure}{0.31\textwidth}
        \includegraphics[width=\textwidth]{RegMon.png}
    \end{subfigure}
   \hspace*{\fill}
    \begin{subfigure}{0.31\textwidth}
        \includegraphics[width=\textwidth]{NegMon.png}
    \end{subfigure}
    \hspace*{\fill}
    \begin{subfigure}{0.31\textwidth}
        \includegraphics[width=\textwidth]{PosMon.png}
    \end{subfigure}

\caption{An image and its segments.}
\label{fig:monument}
\end{figure}
Image segmentation is the process of finding natural boundaries in an image (see Figure \ref{fig:monument}).
This is a simple task for humans, who can easily pick out portions of an image that ``belong together.''
In this lab, you will learn one way for a computer to segment images using graph theory.

The algorithm we will present comes from a paper by Jianbo Shi and Jitendra Malik in 2000 (\cite{Shi2000}).
They segment an image using the following steps.
\begin{enumerate}
	\item \textbf{Represent the image as a weighted graph of pixels.}
An image is made up of individual pixels, each having a brightness and a location.
(Here, \emph{brightness} is equivalent to the grayscale value of the pixel.)
To define a graph representing the image, we let each pixel be a vertex.
%We say that two pixels are connected in the graph if they are close together, i.e. within some distance $r$ of each other, in the image.
The weight of the edges between two pixels is determined by their distance apart and their similarity in brightness.
We define the graph so that two similar pixels (i.e., with similar brightness and location) will be connected by a strong edge.
	\item \textbf{Calculate the Laplacian.}
We calculate the adjacency and degree matrices of the graph representing the image, and use these to obtain the Laplacian.
	\item \textbf{Choose the best cut.}
The graph we have created is connected, but not all edges have the same weight.
Dissimilar pixels will be connected by edges that have a low weight.
We can split the graph into two connected components by ``cutting" it along the low-weight edges.
These components are the image segments we are looking for.
(We can also cut an image multiple times to segment it into more than two pieces.)
As we will see later, Shi and Malik's algorithm uses spectral information (eigenvectors) of the Laplacian to minimize the weight of the cut edges.
\end{enumerate}

\begin{comment}
% Previous explanation which was refactored into the list form above
Their idea is to represent an image as a weighted graph as follows.
To a computer, an \emph{image} is a collection of \emph{pixels}.
Each pixel has a brightness and coordinates describing its location in the image.
To define a graph representing this image, we let every pixel be a vertex.
Two pixels are connected if the distance between their coordinates is small (less than $r$).
The weight of the edge connecting two pixels is related to their similarity in brightness, where a low weight means they are very different.

After defining this graph, we will segment the image by ``cutting'' (or removing) edges with low weights, which represent lines of high contrast in the image.
The ``cut'' is the total weight of the edges removed. Thus, to segment an image, we wish to minimize the ``cut.''
We can ``cut'' an image multiple times to segment an image into more than two pieces.
\end{comment}

\subsection*{Defining the Graph and Adjacency Matrix}

%New
We now define the graph that represents an image. In our $M \times N$ image, the associated graph will have $MN$ nodes, one representing each pixel. We let $w_{ij}$ be the weight of the edge connecting pixels $i$ and $j$, and define

 \begin{equation}
 \label{eq:adjacency}
w_{ij} = \begin{cases} \exp(-\frac{|I(i) - I(j)|}{\sigma_I^2}-\frac{d(i,j)}{\sigma_d^2}) & \mbox{ for $d(i,j) < r$} \\ 0 & \mbox{ otherwise,} \end{cases}
\end{equation}
where
\begin{itemize}
	\item$d(i,j)$ is the Euclidean distance between pixel $i$ and pixel $j$
	\item $|I(i) - I(j)|$ is the difference in brightness of pixels $i$ and $j$
	\item $r$, $\sigma_I$ and $\sigma_d$ are constants that we choose
\end{itemize}

With this definition, pixels that are farther apart than radius $r$ will never be connected.
Pixels within $r$ will be strongly connected if they are similar in brightness ($|I(i) - I(j)|$ is small) and close together ($d(i,j)$ is small).
Highly contrasting pixels ($|I(i) - I(j)|$ is large) will be weakly connected.
This gives us a graph with the properties that we intuitively want.

The adjacency matrix $W$ is $(MN) \times (MN)$ and has $w_{ij}$ as its $ij$th entry.
$W$ will be sparse as long as $r$ is small.
Figure \ref{fig:adjacency} shows a visualization of the adjacency matrix for a $4 \times 4$ image when $r=1.2$.

\begin{comment}
Now let us define the adjacency matrix of the graph associated to an image.
Since an $M \times N$ image has $M \times N$ pixels, the adjacency matrix will be $(M \times N) \times (M \times N)$.
After choosing a radius $r$ and some scaling factors $\sigma_I$ and $\sigma_d$, we define the adjacency matrix to be $W = (w_{ij})$, where

\begin{equation}
\label{eq:adjacency}
w_{ij} = \begin{cases} \exp(-\frac{|I(i) - I(j)|}{\sigma_I^2}-\frac{d(i,j)}{\sigma_d^2}) & \mbox{ for $d(i,j) < r$} \\ 0 & \mbox{ otherwise,} \end{cases}
\end{equation}
where
\begin{itemize}
	\item$d(i,j)$ is the Euclidean distance between pixel $i$ and pixel $j$.
	\item $|I(i) - I(j)|$ is the difference in brightness of pixels $i$ and $j$.
\end{itemize}

$W$ will be sparse as long as $r$ is small.
Figure \ref{fig:adjacency} shows what the adjacency matrix looks like for a $4x4$ image when $r=1.2$.
\end{comment}

\begin{figure}
\begin{tikzpicture}[dot/.style={circle,fill=black,minimum
	size=4pt,inner sep=0pt,outer sep=-1pt}, >=stealth]
%scale=.85, transform shape,

%image
\draw[step=.75,thick](2.999,0)grid(6,3);
%numbers 1-16
\foreach \x in {1,2,3,4}
	\foreach \y in {4}
		\node[draw=none, anchor=south west]at(\x*.75+2.6, \y-1.4){\x};
\foreach \x [evaluate=\x as \r using int(\x+4)]in {1,2,3,4}
	\foreach \y in {3}
		\node[draw=none, anchor=south west]at(\x*.75+2.6, \y-1.2){\r};
\foreach \x [evaluate=\x as \r using int(\x+8)] in {1,2,3,4}
	\foreach \y in {2}
		\node[draw=none, anchor=south west]at(\x*.75+2.5, \y-.95){\r};
\foreach \x [evaluate=\x as \r using int(\x+12)] in {1,2,3,4}
	\foreach \y in {1}
		\node[draw=none, anchor=south west]at(\x*.75+2.5, \y-.7){\r};

\node[draw=none](image)at(4.5, -.5){$image$};
\node[draw=none](flattened)at(7.25,-3){\textit{flattened image}};
\node[draw=none](adjacency)at(12,-3){\textit{adjacency matrix}};

%dots within grid
\foreach \x in {1,2,3,4}
	\foreach \y in {1,2,3,4}
		\node[draw, dot]at(\x*.75+2.6,\y*.75-.4){};

%color fill
\foreach \x/\y in {2/3.75, 1.25/3, 2/3, 2.75/3, 2/2.25} {\node[draw, minimum
	size=.75cm, fill=green!30!black, fill opacity=.25]at(\x+2.118,\y-1.118){};}

%circle in image
\node[draw, circle, minimum size=2cm,thick](circle)at(4.13,1.86){};

%flattened image
\draw[step=.5, thick](6.999,-2.5)grid(7.5,5.5);
\foreach \x in {7}
	\foreach \y in {1,...,16}
		\node[draw=none]at(\x+.25,\y*-.5+5.75){\y};

\draw[->,thick](6.1, 1.5)--(6.9,1.5);

%adjancey matrix
\draw[step=.5](7.9999,-2.5)grid(16,5.5);
%\draw[step=2,thick](7.999,-2.5)grid(16,5.5);

%outside labels
\foreach \x in {1,5,9,13} {\node[draw=none]at(\x*.5+7.75,5.8){\x};}
\foreach \y in {1,5,9,13}{\node[draw=none]at(16.3,\y*-.5+5.8){\y};}

%shading of boxes
\foreach \x/\y in {8.5/5.5, 9/5.5, 8.5/5, 9/5, 9.5/5,9/4.5,9.5/4.5,10/4.5,
	9.5/4, 10/4, 10.5/5.5, 11/5,11.5/4.5, 12/4, 12.5/3.5,
	13.5/2.5, 14/2, 14.5/1.5, 15/1, 15.5/.5,16/0, 10.5/3.5, 11/3.5, 11/2.5,
	11.5/2.5,12/2.5, 11.5/2,12/2, 12.5/1.5, 13/1.5, 12.5/1, 13/1,
	13.5/1,13/.5,13.5/.5,14/.5, 13.5/0,14/0, 14.5/-.5,15/-.5,14.5/-1,
	15/-1,15.5/-1, 15/-1.5, 15.5/-1.5, 16/-1.5, 15.5/-2, 16/-2, 8.5/3.5,
	9.5/2.5,10/2,10.5/1.5,11/1,11.5/.5,12/0, 12.5/-.5, 13/-1, 13.5/-1.5, 14/-2}
	{\node[draw, minimum size=.5cm, fill=black, fill opacity=.25]
	at(\x-.25,\y-.25){};}

%green shaded boxes
\foreach \x/\y in {9/3,11/3, 11.5/3, 10.5/3, 13/3} {\node
	[draw, minimum size=.5cm, fill=shadecolor]
	at(\x-.25,\y-.25){};}

\node[draw=none]at(8.75,2.75){2};
\node[draw=none]at(10.25,2.75){5};
\node[draw=none]at(10.75,2.75){6};
\node[draw=none]at(11.25,2.75){7};
\node[draw=none]at(12.75,2.75){10};

\end{tikzpicture}

\caption{The grid on the left represents a $4\times4$ (or $M \times N$) image with 16 pixels.
At right is the corresponding $16 \times 16$ (or $(MN) \times (MN)$) adjacency matrix with all nonzero entries shaded.
For example, in the $6^{th}$ row, entries 2, 5, 6, 7, and 10 are nonzero because those pixels are within radius $r$ of pixel 6 (here $r = 1.2$).}
%For example, the $6^{th}$ row corresponds to the $6^{th}$ pixel.
%Within that row, entries are nonzero if they correspond to pixels that are within radius $r$ of pixel 6 ($r = 1.2$ was used here).}
\label{fig:adjacency}
\end{figure}

\subsection*{Computing the Adjacency Matrix}
We will now write a function to compute the adjacency matrix for a given image.
%The function will also accept constants \li{radius}, \li{sigma_I}, and \li{sigma_d} to use in \ref{eq:adjacency}.
The function will accept a filename and constants \li{radius}, \li{sigma_I}, and \li{sigma_d}, and return the adjacency matrix and the diagonal of the corresponding degree matrix.

The basic approach is straightforward.
For each pixel in the image, compare it to every other pixel, use (\ref{eq:adjacency}) to calculate the weight of the edge between them, and fill in the weight in the adjacency matrix.
This section will discuss how to implement this efficiently in Python.

First, load an image and convert it to grayscale.
The included function \li{getImage} helps handle color images.
It accepts a filename and returns a 2-D grayscale image.

It will be useful to flatten the $M \times N$ image.
This converts it into a 1-D array of pixels of length $MN$, essentially giving each pixel an index.
We can use \li{img.flatten} to flatten the array \li{img}.
\begin{lstlisting}
>>> A = np.array([[1,2],[3,4]])
>>> A.flatten()
array([1,2,3,4])
\end{lstlisting}

Next, initialize empty adjacency and degree matrices to fill in.
As in Figure \ref{fig:adjacency}, the adjacency matrix will be sparse, so initialize it as a sparse matrix \li{W}.
Use the sparse matrix type \li{lil_matrix}, which is optimized for filling in a matrix one entry at a time.
Since we only need to store the diagonal of the degree matrix, initialize this diagonal as a regular 1-dimensional NumPy array \li{D}.

We now fill in \li{W} according to our definition in Equation \ref{eq:adjacency}.
Each pixel in the image corresponds to one row in \li{W}.
For each pixel:
\begin{itemize}
\item Find its neighbors (the pixels in the image that are within distance $r$ of it)
\item Use \ref{eq:adjacency} to calculate the weights connecting the pixel to each of its neighbors
\item Fill in these weights in \li{W}, leaving the rest of the values in the row as zeros. \footnote{Note that \li{W} will be a symmetric matrix. We could potentially speed up this algorithm by taking advantage of this fact.}
\end{itemize}
The sum of the entries of a row in \li{W} will be the corresponding entry in \li{D}.

You may choose to use the provided helper function \li{getNeighbors} to find the neighbors of a given pixel.
The function accepts the index of a pixel in the flattened image, along with a value for $r$ and the original image dimensions.
It returns two flat arrays: \li{indices} and \li{distances}.
The array \li{indices} contains the indices of the neighbor pixels within distance $r$ of the input pixel.
The array \li{distances} contains the corresponding distances of those pixels from the input pixel.
%According to (\ref{eq:adjacency}), the array \li{indices} contains exactly the indices of the nonzero entries of the current row of \li{W}.
Using Figure \ref{fig:adjacency} as an example, with the inputs 6, 1.2, 4, and 4 the outputs would be \li{indices = array([2,5,6,7,10])} and \li{distances = array([1,1,0,1,1])}.
Try running the function with different inputs to build intuition about what it does.

Finally, convert \li{W} to the sparse matrix type \li{csc_matrix}, which is faster for computations. Then return \li{W} and \li{D}.

\begin{problem}
Write the function \li{adjacency} described in this section.
Accept an image a filename and constants \li{radius}, \li{sigma_I}, and \li{sigma_d}.
Return the corresponding sparse adjacency matrix \li{W} and the diagonal of the degree matrix \li{D}.
Use (\ref{eq:adjacency}) to compute the weights in the adjacency matrix.
For speed, try to compute an entire row of \li{W} at once, instead of filling in \li{W} entry by entry.

%Notice that for each pixel you can save time by only checking the pixels $r$ rows and columns away.
%For that you'll have to handle the pixels on the edges and corners of the image carefully.
%I gave them new helper code, which abstracts away the edge cases.
\label{prob:adjacency_dream}
\end{problem}

\subsection*{Minimizing the `Cut'}
As stated earlier, the goal is to split the image into two segments, while minimizing the weight of the edges that are `cut' in the corresponding graph.
Let $L$ be the Laplacian of the adjacency matrix defined in \ref{eq:adjacency} and let $D$ be the degree matrix.
Shi and Malik proved that using the second smallest eigenvalue of $D^{-1/2}LD^{-1/2}$, we can minimize the `cut' .
Both $D$ and $L$ will be symmetric matrices, so all eigenvalues of $D^{-1/2}LD^{-1/2}$ will be real, therefore the second smallest one is well-defined.
(Note that $D^{-1/2}$ refers not to matrix, but element-wise, exponentiation.)

The eigenvector associated to the second smallest eigenvalue is the key to segmenting the image.
This eigenvector will have $MN$ entries.
Shi and Malik proved that the indices of its \emph{positive} entries are the indices of the pixels in the flattened image which belong in one segment.
Likewise, the indices of its \emph{negative} entries are the indices of the pixels which belong in the other segment.

To compute the segments, reshape this eigenvector as an $M \times N$ array, and set the positive entries to True and the negative entries to False.
We can multiply this True-False mask entry-wise by the image.
This zeros out the pixels in the image corresponding to the \li{False} entries in the mask, without affecting the pixels corresponding to \li{True} entries.
We can negate the mask using the tilde operator, which lets us compute the other segment of the image.
Finally we return the two segments.

\begin{comment}
Here is the definition of a function that will segment an image.
\begin{lstlisting}
1. def segment(img):
\end{lstlisting}

Use the function \li{adjacency} from Problem \ref{prob:adjacency_dream} to compute the adjacency matrix and the diagonal of the degree matrices of the image.
\begin{lstlisting}
2.     W, D = adjacency(img)
\end{lstlisting}

Next we create sparse matrices corresponding to $D$ and $D^{-1/2}$ in Shi and Malik's algorithm. Remember that the sparse matrix type \li{csc_matrix} is best for computations.
\begin{lstlisting}
3.     Dsq = # calculate the square root of D
4.     D_matrix = spar.spdiags(D, 0, D.shape[1], D.shape[1], format = 'csc')
5.     Dsq_matrix = # create a sparse matrix with diagonal Dsq
\end{lstlisting}
Now it is simple to compute $D^{-1/2}LD^{-1/2}$. We call this matrix \li{P}.
\begin{lstlisting}
6.     L = # compute the Laplacian
7.     P = # compute D^{-1/2}*L*D^{-1/2} as in Shi and Malik's algorithm
\end{lstlisting}
According to Shi and Malik, we need the eigenvector corresponding to the second smallest eigenvalue of \li{P}. We compute this with the \li{eigs()} method of the \li{scipy.sparse.linalg} module. We set the parameter \li{which='SR'} in the function call in order to compute the eigenvalues with Smallest Real part and their corresponding eigenvectors. The parameter \li{k} in the function call can be used to specify how many eigenvalues the method computes.
\begin{lstlisting}
8.     e = # compute the two smallest eigenvalues of P and their eigenvectors
9.     eigvec = # eigenvector of the second smallest eigenvalue
\end{lstlisting}

Next we create a mask that is \li{True} wherever \li{eigvec} is positive and reshape it to be the size of \li{img}.
\begin{lstlisting}
10.    mask = # create mask
\end{lstlisting}

\begin{lstlisting}
11.    pos = # compute positive segment
12.    neg = # compute negative segment
13.    return pos, neg
\end{lstlisting}

\end{comment}

\begin{problem} Write the function \li{segment} to segment an image.
The function should accept an image filename and return both of the segments.
You should call the code you wrote in Problem \ref{prob:adjacency_dream}.
Use sparse matrices where possible.

Test on the image \li{dream.png}.
Your segments should look like the segments in Figure \ref{fig:dream_solution} (the original image is on the left).

Hints:
\begin{enumerate}
\item After defining $D^{-1/2}$, convert $D$ and $D^{-1/2}$ into sparse matrices using \li{scipy.sparse.spdiags}.
\item Since are now dealing with sparse instead of dense matrices, you shouldn't use your solution to Problem \ref{prob:laplacian} to calculate the Laplacian.

\item Multiply sparse matrices with \li{A.dot(B)}.

\item Use \li{scipy.sparse.eigsh} to calculate the eigenvector. This is a sparse eigenvalue solver optimized for symmetric matrices.
Set the keyword \li{which = "SM"} to return the smallest eigenvalues.

\item The provided function \li{displayPosNeg} can be used to plot your images.

\end{enumerate}

\end{problem}

\begin{figure}
\centering
    \centering
    \begin{subfigure}{0.31\textwidth}
        \includegraphics[width=\textwidth]{RegDream.png}
    \end{subfigure}
    \hspace*{\fill}
    \begin{subfigure}{0.31\textwidth}
        \includegraphics[width=\textwidth]{NegDream.png}
    \end{subfigure}
    \hspace*{\fill}
    \begin{subfigure}{0.31\textwidth}
        \includegraphics[width=\textwidth]{PosDream.png}
    \end{subfigure}
\caption{Segments of \li{dream.png}}
\label{fig:dream_solution}
\end{figure}

\begin{comment} %Old stuff from middle of lab
Here is the function definition, which includes some default values for the constants.
\begin{lstlisting}
1.	def adjacency(img, radius=5.0, sigma_I = .15, sigma_d = 1.7):
\end{lstlisting}
\begin{lstlisting}
4.     W = spar.lil_matrix((flat_img.size, flat_img.size), dtype=float)
5.     D = np.zeros((1, flat_img.size))
\end{lstlisting}

Later, our function will iterate through the rows of the adjacency matrix, initializing one row at a time. Each row corresponds to a pixel of the original image. Thus, we begin by flattening the image (see Figure \ref{fig:adjacency}). We also store the dimensions of \li{img} for later.
\begin{lstlisting}
2.     flat_img = img.flatten()
3.     height, width = img.shape
\end{lstlisting}

\begin{lstlisting}
6.     for pixel in xrange(flat_img.size):
7.         indices, distances = getNeighbors(pixel, radius, height, width)
8.         weights = # weights[j] should be W[pixel, indices[j]]
9.         W[pixel, indices] = weights
10.        D[0,pixel] = weights.sum()
\end{lstlisting}
\end{comment}

\begin{comment}
\begin{lstlisting}
11.    W = W.tocsc()
12.    return W, D
\end{lstlisting}
\end{comment}

\begin{comment}
\section*{Appendix: helper code for Problem \ref{prob:adjacency_dream}}
Here is the function \li{getNeighbors} which you can use to compute the adjacency matrix of an image, as in Problem \ref{prob:adjacency_dream}.

\lstinputlisting[style=fromfile]{getNeighbors.py}
\end{comment}


\section{Operational Laws}
\label{sec:Operational-Laws}

Operational laws are laws that hold for any open/closed system \footnote{they often are most powerful when applied to closed ones.} or part of it.
They are distribution independent and depends only on mean quantities.
Operational laws can be put together to prove asymptotic bounds on closed networks behavior.
These asymptotic bounds are useful to answer what-if questions.
They are fundamental for the modification analysis.

In the following sections, we present the following operational laws:
\textit{Little's Law},
\textit{Utilization Law},
\textit{Forced Flow Law}, and
\textit{Bottleneck Law}.

\subsection{Little's Law}
\label{sec:Little-Law}

The Little's Law \cite{little1961proof} is the most important operational law. It relates mean number of system jobs to the mean response time.

\begin{theorem}[Little's Law for open systems]
\label{thm:Little-Law-Open-Systems}
	For any ergodic open system, we have that

	\begin{equation}
	\label{eqn:Little-Law-Open-Systems}
	\expected{N} = \lambda \cdot \expected{T}
	\end{equation}

	where $\expected{N}$ is the expected number of jobs in the system, $\lambda$ is the average arrival rate into the system, and $\expected{T}$ is the mean time jobs spend in the system.
\end{theorem}

\begin{theorem}[Little's Law for closed systems]
\label{thm:Little-Law-Closed-Systems}
	For any ergodic closed system, we have that

	\begin{equation}
	\label{eqn:Little-Law-Closed-Systems}
	N = X \cdot \expected{T}
	\end{equation}

	where $N$ is the multiprogramming level, $X$ is the throughput, and $\expected{T}$ is the mean time jobs spend in the system.
\end{theorem}

It is important to note that Little's Law makes no assumption on arrival process, service process, network topology or anything else. It only requires the system to be in steady state with finite $\expected{N}$ and $\expected{T}$.

The Little's Law holds for higher moments only under very restrictive conditions (e.g. it requires FCFS \cite{bertsimas1995distributional,brumelle1972generalization}).

The Little's Law take into account the \textit{effective arrival rate}. Be aware to consider the three quantities in Little's Law on the same request flow.

The following is an alternative expression of the Little's Law for closed systems.

\begin{corollary}
\label{cor:Response-Time-Law-Closed-Systems}

	For every ergodic closed system, we have that

	\begin{equation}
	\label{eqn:Response-Time-Law-Closed-Systems}
	\expected{R} = \frac{N}{X} - \expected{Z}
	\end{equation}

	where
	$\expected{R}$ is the expected response time,
	$N$ is the multiprogramming level,
	$X$ is the system throughput, and
	$\expected{Z}$ is the expected think time.

	\begin{proof}
		The corollary directly follows from the Little's Law.
		\begin{equation*}
		\begin{split}
		N & = X \cdot \expected{T} \\
		& = X \cdot (\expected{R} + \expected{Z})
		\end{split}
		\end{equation*}
		So we have
		\begin{equation*}
		\expected{R} = \frac{N}{X} - \expected{Z}
		\end{equation*}
	\end{proof}
\end{corollary}

\subsection{Utilization-Law}
\label{sec:Utilization-Law}

\begin{theorem}[Utilization Law]
\label{thm:Utilization-Law-2}
	For every stable queue node we have that

	\begin{equation}
	\varrho = \frac{\lambda}{\mu}
	\end{equation}

	where $\varrho$ is the long-run fraction of time that the server is busy.

	\begin{proof}
		Let us consider only the server of the queue node, without the associated queue. The number of jobs in the node is 1 with probability $\varrho$ and 0 with probability $(1-\varrho)$. Hence the the expected number of jobs in the node is $\varrho$.
		So, applying Little's Law, we have
		\begin{equation*}
			\begin{split}
			\varrho & = \expected{N} \\
					& = \lambda \cdot \expected{T} \\
					& = \lambda \cdot \expected{S} \\
					& = \lambda \cdot \frac{1}{\mu}
			\end{split}
		\end{equation*}
	\end{proof}
\end{theorem}

Notice that

\begin{equation}
\varrho_{i} = \probability{server \; i \; busy} = \expected{requests \; to \; server \; i}
\end{equation}
That is, the utilization is the probability to have the server busy and the expected number of requests being served.

\subsection{Forced Flow Law}
\label{sec:Forced-Flow-Law}

\begin{theorem}[Forced Flow Law]
\label{thm:Forced-Flow-Law}
	For every stable network of queues, we have that

	\begin{equation}
	\label{eqn-Forced-Flow-Law}
	X_{i} = \expected{V_{i}} \cdot X
	\end{equation}

	where
	$X$ is the system throughput,
	$X_{i}$ is the throughput at the \textit{i}-th server, and
	$V_{i}$ is the visit ratio for the \textit{i}-th server, that is the number of visits per job to the \textit{i}-th server.

	\begin{proof}
		Let us first consider a intuitive demonstration.
		For every system completion, there are on average $\expected{V_{i}}$ completions at the \textit{i}-th server. Hence the rate of completions at the \textit{i}-th server is $\expected{V_{i}}$ times the rate of system completions.

		Let us now consider a more formal demonstration.
		Let $C(t)$ denote the system completions during time $t$, $C_{i}(t)$ denote the completions at the \textit{i}-th server during time $t$, and $V_{i}^{(i)}$ denote the number of visits the the \textit{j}-th job makes to the \textit{i}-th server. Then,
		\begin{equation*}
			\begin{split}
				C_{i}(t) & = \sum_{j \in C(t))}V_{i}^{(j)} \\
				\frac{C_{i}(t)}{t} & = \frac{\sum_{j \in C(t))}V_{i}^{(j)} }{C(t)} \cdot \frac{C(t)}{t} \\
				\lim_{t \to \infty} \frac{C_{i}(t)}{t} & = \lim_{t \to \infty} \frac{\sum_{j \in C(t))}V_{i}^{(j)} }{C(t)} \cdot \lim_{t \to \infty} \frac{C(t)}{t} \\
				X_{i} & = \expected{V_{i}} \cdot X
			\end{split}
		\end{equation*}
	\end{proof}
\end{theorem}

\subsection{Bottleneck Law}
\label{sec:Bottleneck-Law}

We define $D_{i}$ to be the total service demand on the \textit{i}-th server

\begin{equation}
D_{i} = \sum_{j=1}^{V_{i}}S_{i}^{(j)}
\end{equation}

Since $V_{i}$ and $S_{i}^{(j)}$ are independent, we have that

\begin{equation}
\label{eqn:Service-Demand}
\expected{D_{i}} = \expected{V_{i}} \cdot \expected{S_{i}}
\end{equation}

In practice, we measure $\expected{D_{i}}$ as follows

\begin{equation}
\expected{D_{i}} = \frac{B_{i}}{C}
\end{equation}

where
$B_{i}$ is the busy time for the \textit{i}-th server, and
$C$ is the system completions.

Let $m$ be the number of servers in our system. Let

\begin{equation}
\label{eqn:Total-Demand}
D = \sum_{i=1}^{m}\expected{D_{i}}
\end{equation}

and

\begin{equation}
\label{eqn:Maximum-Demand}
D_{max} = \max\{\expected{D_{i}}\}
\end{equation}

Previous definitions will be useful when discussing \textit{Bottleneck Law}, \textit{asymptotic bounds} and \textit{modification analysis}.

\begin{theorem}[Bottleneck Law]
\label{thm:Bottleneck-Law}
	For every stable network of queues, we have that

	\begin{equation}
	\label{eqn:Bottleneck-Law}
	\varrho_{i} = X \cdot \expected{D_{i}}
	\end{equation}

	where
	$\varrho_{i}$ is the utilization of the \textit{i}-th server,
	$X$ is the system throughput, and
	$\expected{D_{i}}$ is the expected service demand on the \textit{i}-th server for all visits of a single job.

	\begin{proof}
		Let us first consider an intuitive demonstration.
		Let $X$ denote the job/sec arrival rate to the whole system and let each arrival contribute $\expected{D_{i}}$ seconds of work for the \textit{i}-th server. We have that the \textit{i}-th server is busy for $X \cdot \expected{D_{i}}$ seconds out of every second.
		Thus $X \cdot \expected{D_{i}}$ represents the utilization of the \textit{i}-th server.

		Let us now consider a more formal demonstration.
		Let us consider
		the Utilization Law ($\varrho = X \cdot \expected{S}$),
		the Forced Flow Law ($X_{i} = \expected{V_{i}} \cdot X$), and
		the demand definition ($D_{i} = \sum_{j=1}^{V_{i}}S_{i}^{(j)} \Rightarrow \expected{D_{i}} = \expected{V_{i}} \cdot \expected{S_{i}}$).
		We have that
		\begin{equation*}
			\begin{split}
				\varrho_{i} & = X_{i} \cdot \expected{S_{i}} \\
							& = X \cdot \expected{V_{i}} \cdot \expected{S_{i}} \\
							& = X \cdot \expected{D_{i}}
			\end{split}
		\end{equation*}
	\end{proof}
\end{theorem}