\documentclass[../main.tex]{subfiles}
\begin{document}
\begin{figure}[h!]
    \centering
    \includegraphics[width=0.7\columnwidth]{fig/two_pointers.png}
    \caption{Two pointer Technique}
    \label{fig:two pointer}
\end{figure}
On linear data structures, or on implicit linear state space, either a particular targeted item or a consecutive substructure such as a subarray and substring can be searched.

To find a single item on linear space, we can apply linear search in general, or binary search if the data structure is ordered/sorted with logarithmic cost. In this chapter, we introduce two pointer techniques that are commonly used to solve two types of problems:
\begin{enumerate}
    \item Searching: To search for an item such as median,  a predefined substructure, and a substructure that satisfy certain conditions such as finding the minimum subarray length wherein the subarray equals to a targeted sum. Or find a substructure satisfy a string pattern.
    \item Adjusting: To adjust ordering or arrangement of items in the data structure such as removing duplicates from sorted array.
\end{enumerate}

% \section{Introduction to Two Pointers}
As the name suggests, Two pointers technique involves two pointers that start and move with the following two patterns:
\begin{enumerate}
    \item Equi-directional:  Both pointers start from the beginning of the array, and usually one moves faster and the other slower. Sliding window algorithm can be put into this category.
    \item Opposite-directional: One pointer start at the start position and conversely the other pointer starts at the end. These two oppositely posed pointers  move toward each other and usually meet in the middle.
\end{enumerate}
In the following sections, we will detail on two-pointer technique exemplified on real interview questions.

\section{Slow-Faster Pointers}
Suppose we have two pointers, $i$ and $j$, which may or may not start at the start position in the linear data structures, but one move slower ($i$) and the other faster ($j$). Two pointers can decide either a pair or a subarray to solve related problems.  For the case of subarray, the algorithm is called sliding window algorithm. On the span of the array, and at most of three potential sub-spaces exist: from start index to $i$ ($[0, i]$), from $i$ to $j$ ($[i, j]$), and from $j$ to the end index ($[j, n]$).

Even though slow-faster pointers technique rarely given formal introduction in book, it is widely used in algorithms. In sorting, Lumuto's partition in the QuickSort used the slow-faster pointers to divide the whole region into three parts according the comparison result to the pivot: Smaller Items region, Larger Items region, and the unrestricted region. In string pattern matching, fixed sliding window and one we will introduce in this chapter.

In this section, we explain how two pointers work on two types of linear data structures: Array and Linked List.
\subsection{Array}
\subsubsection{Remove Duplicates from Sorted Array(L26)} Given a sorted array $a=[0,0,1,1,1,2,2,3,3,4]$, remove the duplicates in-place such that each element appears only once and return the new length. Do not allocate extra space for another array, you must do this by modifying the input array in-place with O(1) extra memory. In the given example, there are in total of 5 unique items and 5 is returned.
\paragraph{Analysis} We set both slower pointer $i$ and the faster pointer $j$ at the first item in the array. Recall that slow-fast pointers cut the space of the sorted array into three parts, we can define them as:
\begin{enumerate}
    \item unique items in region $[0, i]$,
    \item untouched items in region $[i+1, j]$,
    \item and unprocessed items in region $[j+1, n)$.
\end{enumerate}
In the process, we compare the items pointed by two pointers, once these two items does not equal, we find an new unique item. We copy this unique item at the faster pointer right next to the position of the slower pointer. Afterwards, we move the slow pointer by one position to remove duplicates of our copied value.

With our example, at first, $i=j=0$, region one has one item which is naively unique and region two has zero item. Part of the process is illustrated as:
\begin{lstlisting}[numbers=none]
i  j   [0, i]  [i+1, j]   process
0  0   [0]     []         item 0==0, j+1=1
0  1   [0]     [0]        item 0==0, j+1=2
0  2   [0]     [0, 1]     item 0!=1, i+1=1, copy 1 to index 1, j+1=3
1  3   [0, 1]  [1, 1]     item 1==1, j+1=4
1  4   [0, 1]  [1, 1, 1]  item 1==1, j+1=5
1  5   [0, 1]  [1, 1, 1, 2]item 1==2, i+1=2, copy 2 to index 2, j+1=6
2  6   [0, 1, 2] [1, 1, 2, 2]
\end{lstlisting}
The code is given as:
\begin{lstlisting}[language=Python]
def removeDuplicates(nums) -> int:
    i, j = 0, 0
    while j < len(nums):
        if nums[i] != nums[j]:
            # Copy j to i+1
            i += 1
            nums[i] = nums[j]
        j += 1
    return i + 1
\end{lstlisting}
After calling the above function on our given example, array $a$ becomes $[[0, 1, 2, 3, 4, 2, 2, 3, 3, 4]$. Check the source code for the whole visualized process.

\subsubsection{Minimum Size Subarray Sum(L209)} Given an array of $n$ positive integers and a positive integer $s$, find the minimal length of a contiguous subarray of which the $sum \geq s$. If there isn't one, return 0 instead.
\begin{lstlisting}[numbers=none]
Example:

Input: s = 7, nums = [1,4,1,2,4,3]
Output: 2
Explanation: the subarray [4,3] has the minimal length under the problem constraint.
\end{lstlisting}
\paragraph{Analysis} In this problem, we need to secure a substructure--subarray--that not only satisfies a condition($sum \geq s$) but also  has the minimal length. Naively, we can enumerate all subarrays and search through them to find the minimal length, which requires at least $O(n^2)$ time complexity using prefix sum. The code is as:
\begin{lstlisting}[language=Python]
\end{lstlisting}
However, we can use two pointers $i$ and $j$ ($i\leq j)$ and both points at the first item. In this case, these two pointers defines a subarray  $a[i:j+1]$ and we care the region $[i, j]$. As we increase pointer $j$, we keep adding positive item into the sum of the subarray, making the subarray sum monotonically increasing. Oppositely, if we increase pointer $i$, we remove positive item away from the subarray, making the sum of the subarray monotonically decreasing. The detailed steps of two pointer technique in this case is as:
\begin{enumerate}
    \item Get the optimal subarray for all subproblems(subarries) that start from current $i$, which is $0$ at first. We accomplish this by forwarding $j$ pointer to include enough items until $sum \geq s$ that we pause and go to the next step. Let's assume pointer $j$ stops at $e_0$.
    \item Get the optimal subarray for all subproblems(subarries) that end with current j, which is $e_0$ at the moment. We do this by forwarding pointer $i$ this time to shrink the window size until $sum\geq s$ no longer holds. Let's assume pointer $i$ stops at index $s_0$. Now, we find the optimal solution for subproblems $a[0:i,0:j]$( denoting subarries with the start point in range $[0, i]$ and the end point in range $[0,j]$.
    \item Now that $i=s_0$ and $j=e_0$, we repeat step 1 and 2.
\end{enumerate}
In our example, we first move $j$ until $j=3$ with a subarray sum of 8. Then we move pointer $i$ until $i=1$ when the subarray sum is less than 7. For subarray $[1, 4, 1, 2]$, we find its optimal solution to have a length 3. The Python code is given as:
\begin{lstlisting}[language=Python]
def minSubArrayLen(s: int, nums) -> int:
    i, j = 0, 0
    acc = 0
    ans = float('inf')
    while j < len(nums):
        acc += nums[j]
        # Shrink the window
        while acc >= s:
            ans = min(ans, j - i + 1)
            acc -= nums[i]
            i += 1
        j += 1

    return ans if ans < float('inf') else 0
\end{lstlisting}

Because both pointer $i$ and $j$  move at most $n$ steps, with the total operations to be at most $2n$, making the time complexity as $O(n)$. The above question would be trivial if the maximum subarray length is asked.

\subsection{Minimum Window Substring (L76, hard)}
 Given a string $S$ and a string $T$, find all the minimum windows in $S$ which will contain all the characters in $T$ in complexity $O(n)$.
\begin{lstlisting}[numbers=none]
Example:
Input: S = "AOBECDBANC", T = "ABC"
Output: ["CDBA", "BANC"]
\end{lstlisting}
\begin{figure}[h!]
    \centering
    \includegraphics[width=0.5\columnwidth]{fig/minimum_window_substring.png}
    \caption{The data structures to track the state of window.}
    \label{fig:minimum_window}
\end{figure}
\paragraph{Analysis} Applying two pointers, with the region between pointer $i$ and $j$ to be our testing substring. For this problem, the condition for the window $[i, j]$ it will at most have all characters from $T$. The intuition is we keep expanding the window by moving forward $j$ until all characters in $T$ is found. Afterwards, we contract the window so that we can find the minimum window with the condition satisfied. Instead of using another data structure to track the state of the current window, we can depict the pattern $T$ as a dictionary data structure where all unique characters comprising the keys and with the number of occurrence of each character as value. We use another variable \texttt{count} to track how the number of unique characters. In all, they are used to track the state of the moving window in $[i, j]$, with the value of the dictionary to indicate how many occurrence is short of, and the \texttt{count} represents how many unique characters is not fully found, and we depict the state  in Fig.~\ref{fig:minimum_window}.

\begin{figure}[h!]
    \centering
    \includegraphics[width=1.1\columnwidth]{fig/minimum_window_substring_process.png}
    \caption{The partial process of applying two pointers. The grey shaded arrow indicates the pointer that is on move.}
    \label{fig:minimum_window_process}
\end{figure}

Along the expanding and shrinking of the window that comes with the movement of pointer $i$ and $j$, we track the state with:
\begin{itemize}
    \item When forwarding $j$, we encompass $S[j]$ in the window. If $S[j]$ is a key in the dictionary, decrease the value by one. Further, if the value reaches to the threshold $0$, we decrease \texttt{count} by one, meaning we are short of one less character in the window.
    \item Once \texttt{count=0}, our window satisfy condition for contracting. We then forward $i$, removing $S[i]$ from the window if it is existing key in the dictionary by increasing this key's value, meaning the window is short of one more character. Once the value reaches to the threshold of $1$, we increase $count$.
\end{itemize}
Part of this process with our example is shown in Fig.~\ref{fig:minimum_window_process}. And the Python code is given as:
\begin{lstlisting}[language=Python]
from collections import Counter
def minWindow(s, t):
  dict_t = Counter(t)
  count = len(dict_t)
  i, j = 0, 0
  ans = []
  minLen = float('inf')
  while j < len(s):
    c = s[j]
    if c in dict_t:
      dict_t[c] -= 1
      if dict_t[c] == 0:
        count -= 1
    # Shrink the window
    while count == 0 and i < j:
      curLen = j - i + 1
      if curLen < minLen:
        minLen = j - i + 1
        ans = [s[i:j+1]]
      elif curLen == minLen:
        ans.append(s[i:j+1])

      c = s[i]
      if c in dict_t:
        dict_t[c] += 1
        if dict_t[c] == 1:
          count += 1
      i += 1

    j += 1
  return ans
\end{lstlisting}

% until When the window has all the desired characters, we contract (if possible) and save the smallest window till now. The only difference compared with the above problem is the definition of desirable: we need to compare the state of current window with the required state in T. They can be handled as a hashmap with character as key and frequency of characters as value.
% \begin{lstlisting}[language=Python]
% def minWindow(self, s, t):
%     dict_t = Counter(t)
%     state = Counter()
%     required = len(dict_t)

%     # left and right pointer
%     i, j = 0, 0

%     formed = 0
%     ans = float("inf"), None # min len, and start pos

%     while j < len(s):
%         char = s[j]
%         # record current state
%         if char in dict_t:
%             state[char] += 1
%             if state[char] == dict_t[char]:
%                 formed += 1

%         # Try and contract the window till the point where it ceases to be 'desirable'.
%         # bPrint = False
%         while i<=j and formed == required:
%             # if not bPrint:
%             #     print('found:', s[i:j+1], i, j)
%             #     bPrint = True
%             char = s[i]
%             if j-i+1 < ans[0]:
%                 ans = j - i + 1, i
%             # change the state
%             if char in dict_t:
%                 state[char] -= 1
%                 if state[char] == dict_t[char]-1:
%                     formed -= 1

%             # Move the left pointer ahead,
%             i += 1

%         # Keep expanding the window
%         j += 1
%         # if bPrint:
%         #     print('move to:', s[i:j+1], i, j)
%     return "" if ans[0] == float("inf") else s[ans[1] : ans[1] + ans[0]]
% \end{lstlisting}

% The process would be:
% \begin{lstlisting}[numbers=none]
% found: ADOBEC 0 5
% move to: DOBECO 1 6
% found: DOBECODEBA 1 10
% move to: ODEBAN 6 11
% found: ODEBANC 6 12
% move to: ANC 10 13
% \end{lstlisting}
\subsection{When Two Pointers do not work} Two pointer does not always work on subarray related problems.
\begin{bclogo}[couleur = blue!30, arrondi=0.1,logo=\bccrayon,ombre=true]{What happens if there exists negative number in the array? } Since the sum of the subarray is no longer monotonically increasing with the number of items between two pointers, we can not figure out how to move two pointers each step. Instead (1) we can use prefix sum and organize them in order, and use binary search to find all possible start index.  (2) use monotone stack (see  LeetCode probelm: 325. Maximum Size Subarray Sum Equals k, 325. Maximum Size Subarray Sum Equals k (hard)))
\end{bclogo}

\begin{bclogo}[couleur = blue!30, arrondi=0.1,logo=\bccrayon,ombre=true]{What if we are to check the maximum average subarray? }
644. Maximum Average Subarray II (hard). Similarly, the average of subarray does not follow a certain order with the moving of two pointers at each side, making it impossible to decide how to make the two pointers.

\end{bclogo}

\subsection{Linked List}
The complete code to remove cycle is provided in google colab together with running examples.
\subsubsection{Middle of the Linked List(L876)}
%The simplest example of slow-fast pointers on linked list is to get the middle node of a given linked list.
Given a non-empty, singly linked list with head node $head$, return a middle node of linked list. When the linked list is of odd length, there exists one and only middle node, but when it is of even length, two exists and we return the second middle node.
\begin{lstlisting}[numbers=none]
Example 1 (odd length):

Input: [1,2,3,4,5]
Output: Node 3 from this list (Serialization: [3,4,5])

Example 2 (even length):

Input: [1,2,3,4,5,6]
Output: Node 4

 from this list (Serialization: [4,5,6])
\end{lstlisting}

\paragraph{Analysis} If the data structure is array, we can compute the position of the middle item simply with the total length. Following this method, if only one pointer is applied, we can first iterate over the whole linked list in $O(n)$ time to get the length. Then we do another iteration to obtain the middle node. $n + \frac{n}{2}$ times of operations  needed, making the time complexity $O(n)$.

However, we can apply two pointers simultaneously at the head node, each one moves at different paces: the slow pointer moves one step at a time and the fast moves two steps instead. When the fast pointer reached the end, the slow pointer will stop at the middle. This slow-faster pointers technique requires only $\frac{n}{2}$ times of operations, which is three times faster than our naive method, although the big Oh time complexity still remains $O(n)$.
\paragraph{Implementation}
\begin{figure}[h]
    \centering
    \includegraphics[width = 0.98\columnwidth]{fig/middle_of_linked_list.png}
    \caption{Slow-fast pointer to find middle}
    \label{fig:slow-faster}
\end{figure}
Simply, we  illustrate the process of running the two pointers technique on our two examples in Fig.~\ref{fig:slow-faster}. As we can see, when the slow pointer reaches to item 3, the faster pointer is at item 5, which is the last item in the first example that comes with odd length. Further, when the slow pointer reaches to item 4, the faster pointer reaches to the empty node of the last item in the second example that comes with even length. Therefore, in the implementation, we check two conditions in the \texttt{while} loop:
\begin{enumerate}
    \item For example 1: if the fast pointer has no successor (\texttt{fast.next==None}), the loop terminates.
    \item For example 1: if the fast pointer is invalid (\texttt{fast==None}), the loop terminates.
\end{enumerate}
The Python code is as:
\begin{lstlisting}[language=Python]
def middleNode(head):
    slow = fast = head
    while fast and fast.next:
        fast = fast.next.next
        slow = slow.next
    return slow
\end{lstlisting}

\subsubsection{Floyd's Cycle Detection (Floyd's Tortoise and Hare)}
\begin{figure}[h!]
    \centering
    \includegraphics[width=0.8\columnwidth]{fig/circular_linked_list.png}
    \caption{Circular Linked List}
    \label{fig:circular_linked_list}
\end{figure}
When a linked list which has a cycle, as shown in Fig.~\ref{fig:circular_linked_list}, iterating items over the list will make the program stuck into infinite loop. The pointer starts from the heap, traverse to the start of the loop, and then comes back to the start of the loop again and continues this process endlessly. To avoid being stuck into a ``trap'', we have to possibly solve the following three problems:
\begin{enumerate}
    \item Check if there exists a cycle.
    \item Check where the cycle starts.
    \item Remove the cycle once it is detected.
\end{enumerate}
The solution encompasses the exact way of slow faster pointers traversing through the linked list as our last example. With the slow pointer iterating one item at a time, and the faster pointer in double pace, these two pointers will definitely meet at one item in the loop. In our example, they will meet at node 6. So, is it possible that it will meet at the non-loop region starts from the heap and ends at the start node of the loop? The answer is No, because the faster pointer will only traverse through the non-loop region once and it is always faster than the slow pointer, making it impossible to meet in this region.  This method is called Floyd's Cycle Detection, aka Floyd's Tortoise and Hare Cycle Detection. Let's see more details at how to solve our mentioned three problems with this method.
\paragraph{Check Linked List Cycle(L141)}
Compared with the code in the last example, we only need to check if the \texttt{slow} and \texttt{fat} pointers are pointing at the same node: If it is, we are certain that there must be a loop in the list and return \texttt{True}, otherwise  return \texttt{False}.
\begin{lstlisting}[language=Python]
def hasCycle(head):
    slow = fast = head
    while fast and fast.next:
        slow = slow.next
        fast = fast.next.next
        if slow == fast:
            return True
    return False
\end{lstlisting}
\paragraph{Check Start Node of Linked List Cycle(L142)} Given a linked list, return the node where the cycle begins. If there is no cycle, return \texttt{None}.
\begin{figure}[h!]
    \centering
    \includegraphics[width=0.6\columnwidth]{fig/TQoyH.png}
    \caption{Floyd's Cycle finding Algorithm}
    \label{fig:floyd_cycle_1}
\end{figure}

For a given linked list, assume the slow and fast pointers meet at node somewhere in the cycle. As shown in Fig.~\ref{fig:floyd_cycle_1}, we denote  three nodes: head ($h$, start node of cycle($s$), and meeting node in the cycle($m$). we denote the distance between $h$ and $s$ to be $x$, the distance between $s$ and $m$ to be $y$, and the distance between $m$ and $s$ to be $z$.  Because the faster pointer traverses through the list in double speed, when it meets up with the slow pointer, the distance that it traveled($x+y+z+y$) to be two times of the distance traveled by the slow pointer ($x+y$).
\begin{align}
    x + y + z + y &= x + y \\
    x = z
\end{align}
From the above equation, we obtain the equal relation between $x$ and $z$. the starting node of the cycle from the head is $x$, and $y$ is the distance from the start node to the slow and fast pointer's node, and $z$ is the remaining distance from the meeting point to the start node.  Therefore, after we have detected the cycle from the last example, we can reset the slow pointer to the head of the linked list after. Then we make the slow and the fast pointer both traverse at the same pace--one node at a time--until they meet at a node we stop the traversal. The node where they stop at is the start node of the cycle. The code is given as:

% the meeting point, and making both slow and fast pointer to move one node at a time, they will meet at the starting node of the cycle.

% Now, let's try to device the algorithm. Both slow and fast pointer starts at position 0, the node index they travel each step is: [0,1,2,3,...,k] and [0,2,4,6,...,2k] for slow and fast pointer respectively. Therefore, the total distance traveled by the slow pointer is half of the distance travelled by the fat pointer. From the above figure, we have the distance travelled by slow pointer to be $d_s = x+y$, and for the fast pointer $d_f = x+y+z+y = x+2y+z$. With the relation $2*d_s = d_f$. We will eventually get $x = z$. Therefore, by moving slow pointer to the start of the linked list after the meeting point, and making both slow and fast pointer to move one node at a time, they will meet at the starting node of the cycle. (LeetCode problem: 142. Linked List Cycle II (medium)).
\begin{lstlisting}[language=Python]
def detectCycle(head):
    slow = fast = head

    def getStartNode(slow, fast, head):
      # Reset slow pointer
      slow = head
      while fast and slow != fast:
          slow = slow.next
          fast = fast.next
      return slow

    while fast and fast.next:
        slow = slow.next
        fast = fast.next.next
        # A cycle is detected
        if slow == fast:
            return getStartNode(slow, fast, head)

    return None
\end{lstlisting}

\paragraph{Remove Linked List Cycle}
We can remove the cycle by recirculing the last node in the cycle, which in example in Fig.~\ref{fig:circular_linked_list} is node 6 to an empty node. Therefore, we have to modify the above code to make the \texttt{slow} and {fast} pointers stop at the last node instead of the start node of the loop. This subroutine is implemented as:
\begin{lstlisting}[language=Python]
def resetLastNode(slow, fast, head):
    slow = head
    while fast and slow.next != fast.next:
        slow = slow.next
        fast = fast.next
    fast.next = None
\end{lstlisting}
The complete code to remove cycle is provided in google colab together with running examples.
\begin{bclogo}[couleur = blue!30, arrondi=0.1,logo=\bccrayon,ombre=true]{What if there has not only one, but multiple cycles in the Linked List? }
\end{bclogo}

\section{Opposite-directional Pointers}
Another variant of two pointers technique is to place these two pointers oppositely: one at the beginning and the other at the end of the array. Through the process, they move toward each other  until they meet in the middle. Details such as how much each pointer moves or which pointer to move at each step decided by our specific problems to solve. We just have to make sure when we are applying this technique, we have considered its whole state space, and will not miss out some area which makes the search incomplete.

The simplest example of this two pointers method is to reverse an array or a string around. For example, when the list $a=[1, 2, 3, 4, 5]$ is reversed, it becomes $[5, 4, 3, 2, 1]$. Of course we can simply assign a new list and copy the items in reversed orders. But, with two pointers, we are able to reverse it in-place and using only $O(\frac{n}{2})$ times of operations through the following code:
\begin{lstlisting}[language=Python]
def reverse(a):
  i, j = 0, len(a) - 1
  while i < j:
    # Swap items
    a[i], a[j] = a[j], a[i]
    i += 1
    j -= 1
\end{lstlisting}
Moreover, binary search can be viewed as an example of opposite-directional pointers. At first, these two pointers are the first and the last item in the array. Then depends on which side of the target compared with the item in the middle, one of the pointers move either forward or backward to the middle point, reducing the search space to half of where it started at each step.  We also explore another example with this technique.
\subsubsection{Two Sum on Sorted Array(L167)}
Given an array of integers that is already sorted in ascending order, find two numbers such that they add up to a specific target number.
\begin{lstlisting}[numbers=none]
Input: numbers = [2,7,11,15], target = 9
Output: [1,2]
Explanation: The sum of 2 and 7 is 9. Therefore index1 = 0, index2 = 1.
\end{lstlisting}
\paragraph{Analysis} If we simply put enumerate all possible pairs, we have to take $O(n^2)$ to solve this problem. However, with the opposite-directional two pointers, it gives out linear performance.

Denote the list as $A=[a_1, a_2, ..., a_{n-1}, a_{n}]$, and for the sorted array we have $a_1\leq a_2 \leq...\leq  a_{n-1} \leq a_n$. The range of the sum of any two items in the array is within two possible ranges: $[a_1+a_2, a_1+a_n]$ and $[a_1+a_n, a_{n-1}+a_n]$.  By placing one pointer $i$ at $a_1$ and the other $j$ at $a_n$ to start with, we can get $a_1+a_n$ as the sum. Pointer $i$ can only move forward, accessing larger items. On the other hand, pointer $j$ can only backward, accessing smaller items. Now there are three scenarios according to the comparison between the target and the current sum of the two pointers:
\begin{enumerate}
    \item  If $t == a[i] + a[j]$, target sum found.
    \item  If $t > a[i] + a[j]$, we have to increase the sum, we can only do this by moving pointer $i$ forward.
    \item If $t > a[i] + a[j]$, we have to decrease the sum, we can only do this by moving pointer $j$ backward.
\end{enumerate}
The Python code is as:
\begin{lstlisting}[language=Python]
def twoSum(a, target):
    n = len(a)
    i, j  = 0, n-1
    while i < j:
        temp = a[i] + a[j]
        if temp == target:
            return [i, j]
        elif temp < target:
            i += 1
        else:
            j -= 1
    return []
\end{lstlisting}
\section{Follow Up: Three Pointers}
Sometimes, manipulating two pointers is not even enough to distinguish different subspaces, we might need to the assistant of one another pointer to make things  work.
\subsubsection{Binary Subarrays With Sum (L930)} In an array $A$ of $0$s and $1$s, how many non-empty subarrays have sum $S$?
\begin{lstlisting}[numbers=none]
Example 1:
Input: A = [1,0,1,0,1], S = 2
Output: 4
Explanation:
The 4 subarrays are listed below:
[1,0,1], index (0, 2)
[1,0,1,0], index (0, 3)
[0,1,0,1], index (1, 4)
[1,0,1], index (2, 4)
\end{lstlisting}
\paragraph{Analysis}
This problem is highly similar to the minimum length subarray problem we encountered before. We naturally start with two pointers $i$ and $j$, and restrict the subarray in range $[i, j]$ to satisfy condition $sum\leq S$. The window is contracted when the condition is violated. We would have write the following code:
\begin{lstlisting}[language=Python]
def numSubarraysWithSum(a, S):
  i, j = 0, 0
  win_sum = 0
  ans = 0
  while j < len(a):
    win_sum += a[j]
    while i < j and win_sum > S:
      win_sum -= a[i]
      i += 1
    if win_sum == S:
      ans += 1
      print('({}, {})'.format(i, j))
    j += 1
  return ans
\end{lstlisting}
However, the above code only returns $3$, instead of $4$ as shown in the example. By printing out pointers $i$ and $j$, we can see the above code is missing case $(2, 4)$. Why? Because we are restricting the subarray sum in range $[i, j]$ to be smaller than or equal to $S$, with the occruence of $0$s that might appear in the front or in the rear of the subarray:
\begin{itemize}
\item In the process of expanding the subarray, pointer $j$ is moved one at a time. Thus, even though $0$s appear in the rear of the subarray, the counting is correct.
\item However, in the process of shrinking the subarray while the restriction is violated($sum > S$), we stop right away once $sum \leq S$. And in the code, we end up only counting it as one occurrence. With $0$s at the beginning of the subarray, such as the subarray $[0, 1, 0, 1]$ with index $1$ and $4$, there count should be two instead of one.
\end{itemize}
The solution is to add another pointer $i_h$ to handle the missed case: When the $sum=S$, count the total occurrence of $0$ in the front. Compared with the above solution, the code only differs slightly with the additional pointer and one extra \texttt{while} loop to deal the case. Also we need to pay attention that $i_h \leq j$, otherwise, the \texttt{while} loop would fail with example with only zeros and a targeting sum $0$.
\begin{lstlisting}[language=Python]
def numSubarraysWithSum(a, S):
  i, i_h, j = 0, 0, 0
  win_sum = 0
  ans = 0
  while j < len(a):
    win_sum += a[j]
    while i < j and win_sum > S:
      win_sum -= a[i]
      i += 1
    # Move i_h to count all zeros in the front
    i_h = i
    while i_h < j and win_sum == S and a[i_h] == 0:
      ans += 1
      i_h += 1

    if win_sum == S:
      ans += 1
    j += 1
  return ans
\end{lstlisting}

We noticed that in this case, we have to explicitly restrict $i < j$ and $i_h < j$ due to the special case,  while in all our previous examples, we do not have to.

\section{Summary}
Two pointers is a powerful tool for solving problems on liner data structures, such as ``certain'' subarray and substring problems as we have shown in the examples. The ``window'' secluded between the two pointers can be viewed as sliding window: It can move slide forwarding with the forwarding the slower pointer. Two important properties are generally required for this technique to work:
\begin{enumerate}
\begin{figure}[h!]
    \centering
    \includegraphics[width=0.7\columnwidth]{fig/Sliding_window_property.png}
    \caption{Sliding Window Property}
    \label{fig:slide_window}
\end{figure}
    \item Sliding window property: Either we move the faster pointer $j$ forward by one, or move the slower pointer $i$, we can get the state of current window in $O(1)$ cost knowing the state of the last window.

    For example, given an array, imagine that we have a fixed size window as shown in Fig.~\ref{fig:slide_window}, and we can slide it forward one position at a time, compute the sum of each window. The bruteforce solution would be of $O(kn)$ complexity where $k$ is the window size and $n$ is the array size by using two nested \texttt{for} loops: one to set the starting point, and the other to compute the sum in $O(k)$. However, the sum of the current window ($S_c$) can be computed from the last window ($S_l$), and  the items that just slid out and in as $a_j$ and $a_i$ respectively. Then $S_c = S_l-a_i+a_j$. Getting the state of of the window between two pointers in $O(1)$ as shown in the example is our called Sliding Window Property.

    Usually, for an array with numerical value, it satisfies the sliding window property if we are to compute its sum or product. For substring, as shown in our minimum window substring example, we can get the state of current window referring to the state of the last window in $O(1)$ with the assist of dictionary data structure. In substring, this is more obscure, and the general requirement is that the state of the substring does not relate to the order of the characters(anagram-like state).

    \item Monotonicity: For subarray sum/product, the array should only comprise all positive/negative values so that the prefix sum/product has monotonicity: moving the faster pointer and the slower pointer forward results into opposite change to the state.  The same goes for the substring problems where we see from the minimum window substring example the change of the state: \texttt{count} and the value of the dictionary is monotonic, and each either increases or decreases with the moving of two pointers.
% \begin{lstlisting}[language=Python]
% def fixedSlideWindow(A, k):
%     n = len(A)
%     if k >= n:
%         return sum(A)
%     # compute the first window
%     acc = sum(A[:k])
%     ans = acc
%     # slide the window
%     for i in range(n-k): # i is the start point of the window
%         j = i + k # j is the end point of the window
%         acc = acc - A[i] + A[j]
%         ans = max(ans, acc)
%     return ans
% \end{lstlisting}
\end{enumerate}

% The steps of using sliding windows:
% \begin{enumerate}
%     \item Initialize the left and right pointer;
%     \item Handle the right pointer and record the state of the current window;
%     \item While the window is in the state of desirable: record the optimal solution, move the left pointer and record the state (change or stay unchanged).
%     \item Up till here, the state is not desirable.  Move the right pointer in order to find a desirable window;
% \end{enumerate}
\section{Exercises}
\begin{enumerate}
\item 3. Longest Substring Without Repeating Characters
\item 674. Longest Continuous Increasing Subsequence (easy)
\item 438. Find All Anagrams in a String
\item 30. Substring with Concatenation of All Words
\item 159. Longest Substring with At Most Two Distinct Characters
\item 567. Permutation in String
\item 340. Longest Substring with At Most K Distinct Characters
\item 424. Longest Repeating Character Replacement
\end{enumerate}
% that data strArray Search is to find a \textbf{sub-structure} on a given linear data structure( Chapter~\ref{chapter_linear_data_structure}) or a virtual linear search space. Categorized by the definition of sub-structure:
% \begin{itemize}
%     \item Define the sub-structure as a \textbf{particular item}: Usually the worst and average performance is $O(n)$.  \textbf{Binary search} (Section~\ref{sec_binary_search}) finds an item within an ordered data structure, each time, the search space is elimilated by half in size, which makes the worst time complexity $O(\log n)$. Using hashmap can gain us the best complexity of $O(1)$.
%     \item Define the sub-structure as a \textbf{consecutive substructure} indexed by a start and end index (subarray) in the linear data structure, we introduce the \textbf{Sliding Window Algorithm} (Section~\ref{sec_pointer_sliding_window}). Compared with the brute force solution, it decrease the complexity from $O(n^2)$ to $O(n)$. If the sub-structure is \textbf{predefined pattern}, we need pattern matching algorithms. This usually exists in string data structure, and we do string pattern matching (Section~\ref{}).
% \end{itemize}

% \subfile{chapters/learning/search/sliding_window}

 \end{document}


%-------------------1.1
\section{Auto-resizing equation}
\begin{tabular}{l | c}
\begin{minipage}[m]{0.4\textwidth}
\enum{
\resizebox{.6\textwidth}{!}{$\dot{\rho}=
\dfrac{x^3}{45a^9-23b}$}}{1.1}
\end{minipage}
& \begin{minipage}[m]{0.5\textwidth}
\renewcommand\textminus{\mbox{-}}%<<<<<<<<<<<
\begin{lstlisting}[numberstyle=\zebra{black!5}{blue!15},numbers=left,basicstyle=\footnotesize]{tex}
\begin{equation*}\label{eq1}
\resizebox{.4\textwidth}{!}{ % change .4 to 0.5...
$\dot{\rho}=\dfrac{x^3}{45a^9-23b}$}
\end{equation*}
    \end{lstlisting}
\end{minipage}
\end{tabular}

%-------------------1.2
\section{Form for simplest calculation}
\begin{tabular}{l | c}
\begin{minipage}[m]{0.4\textwidth}
\enum{ \newcommand{\sss}[1]{this.getField("#1").value}
\begin{Form}
\noindent%
Fill with number \\
\small{\mybox[red]{if it does't work try another PDF viewer}}\\

\TextField[name=a]{a:} \\

\TextField[name=b]{b:} \\

\TextField[name=c]{c:} \\

\noindent%
$\sum = $ \TextField[name=AvgStat, calculate={
  event.value = (
    \sss{a} +
    \sss{b} +
    \sss{c}) ;
}, readonly, value=0]{}
\end{Form}}{1.2}
\end{minipage}
& \begin{minipage}[m]{0.5\textwidth}
\renewcommand\textminus{\mbox{-}}%<<<<<<<<<<<
\begin{lstlisting}[numberstyle=\zebra{black!5}{blue!15},numbers=left,basicstyle=\footnotesize]{tex}
\documentclass{article}
\usepackage{hyperref}
\begin{document}
\newcommand{\sss}[1]{this.getField("#1").value}
\begin{Form}
\noindent%
Fill with number\\

\TextField[name=a]{a:} \\

\TextField[name=b]{b:} \\

\TextField[name=c]{c:} \\
\noindent%
$\sum = $ \TextField[name=AvgStat, calculate={
  event.value = (
    \sss{a} +
    \sss{b} +
    \sss{c}) ;
}, readonly, value=0]{}
\end{Form}
\end{document}
\end{lstlisting}
\end{minipage}
\end{tabular}

%-------------------1.3
\section{Equation in the form of steps}
\begin{tabular}{l | c}
\begin{minipage}[m]{0.4\textwidth}
\enum{ \resizebox{.4\textwidth}{!}{$  \frac{n_0}{n_1} = q_1 + \dfrac{\makebox[\mywd][l]{$1$}}
  {\makebox[\mywd][l]{$q_2 + \dfrac{\makebox[\mywd][l]{$1$}}
  {\makebox[\mywd][l]{$q_3 + \dfrac{\makebox[\mywd][l]{$1$}}
  {\makebox[\mywd][l]{$q_4 +
   \raisebox{-6pt}{$\ddots$}
   \raisebox{-12pt}{+$\dfrac{\makebox[\mywd][l]{$1\kern30pt$}}
  {q_{k-1} + \dfrac{1}
  {q_k}}$}$}}$}}$}} $}}{1.3}
\end{minipage}
& \begin{minipage}[m]{0.5\textwidth}
\renewcommand\textminus{\mbox{-}}%<<<<<<<<<<<
\begin{lstlisting}[numberstyle=\zebra{black!5}{blue!15},numbers=left,basicstyle=\footnotesize]{tex}
\documentclass{article}
\usepackage{amsmath}
\def\mywd{35pt}
\begin{document}
\[
  \frac{n_0}{n_1} = q_1 + \dfrac{\makebox[\mywd][l]{$1$}}
  {\makebox[\mywd][l]{$q_2 + \dfrac{\makebox[\mywd][l]{$1$}}
  {\makebox[\mywd][l]{$q_3 + \dfrac{\makebox[\mywd][l]{$1$}}
  {\makebox[\mywd][l]{$q_4 +
   \raisebox{-6pt}{$\ddots$}
   \raisebox{-12pt}{+$\dfrac{\makebox[\mywd][l]{$1\kern30pt$}}
  {q_{k-1} + \dfrac{1}
  {q_k}}$}$}}$}}$}}
\]
\end{document}
\end{lstlisting}
\end{minipage}
\end{tabular}

%-------------------1.4
\section{One number for multiline equation}
\begin{tabular}{l | c}
\begin{minipage}[m]{0.4\textwidth}
\enum{ \begin{equation}
\begin{aligned}
x_{ij} &= d_{ijk}E_k, \\
x_{ij} &= \varsigma_{ijk}H_k,\\
x_{ij} &= s_{ijkl}X_{kl},\\
x_{ij} &= \xi_{ij}\delta p,\\
x_{ij} &= \alpha_{ij}\delta T
\end{aligned}
\end{equation}}{1.4}
\end{minipage}
& \begin{minipage}[m]{0.5\textwidth}
\renewcommand\textminus{\mbox{-}}%<<<<<<<<<<<
\begin{lstlisting}[numberstyle=\zebra{black!5}{blue!15},numbers=left,basicstyle=\footnotesize]{tex}
\documentclass{article}
\usepackage{amsmath}
\begin{document}
\begin{equation}
\begin{aligned}
x_{ij} &= d_{ijk}E_k, \\
x_{ij} &= \varsigma_{ijk}H_k,\\
x_{ij} &= s_{ijkl}X_{kl},\\
x_{ij} &= \xi_{ij}\delta p,\\
x_{ij} &= \alpha_{ij}\delta T
\end{aligned}
\end{equation}
\end{document}
\end{lstlisting}
\end{minipage}
\end{tabular}

%-------------------1.5
\section{Matrix in \textbf{standalone} documentclass}
\begin{tabular}{l | c}
\begin{minipage}[m]{0.4\textwidth}
\enum{ \begin{equation*}
\begin{matrix}
a_{11} & a_{12} & a_{13}  \\
a_{21} & a_{22} & a_{23}  \\
a_{31} & a_{32} & a_{33}  \\
\end{matrix}
\end{equation*} }{1.5}
\end{minipage}
& \begin{minipage}[m]{0.5\textwidth}
\renewcommand\textminus{\mbox{-}}%<<<<<<<<<<<
\begin{lstlisting}[numberstyle=\zebra{black!5}{blue!15},numbers=left,basicstyle=\footnotesize]{tex}
\documentclass[preview,border={-5cm 0cm -5cm -0.1cm}]{standalone}
\usepackage{amsmath}
\begin{document}
\begin{equation*}
\begin{matrix}
a_{11} & a_{12} & a_{13}  \\
a_{21} & a_{22} & a_{23}  \\
a_{31} & a_{32} & a_{33}  \\
\end{matrix}
\end{equation*}
\end{document}
\end{lstlisting}
\end{minipage}
\end{tabular}

%-------------------1.6
\section{Multiple lines, one centered label}
\begin{tabular}{l | c}
\begin{minipage}[m]{0.4\textwidth}
\enum{ \begin{equation} \label{eq1}
\begin{split}
A & = \frac{\pi r^2}{2} \\
 & = \frac{1}{2} \pi r^2
\end{split}
\end{equation} }{1.6}
\end{minipage}
& \begin{minipage}[m]{0.5\textwidth}
\renewcommand\textminus{\mbox{-}}%<<<<<<<<<<<
\begin{lstlisting}[numberstyle=\zebra{black!5}{blue!15},numbers=left,basicstyle=\footnotesize]
\begin{equation} \label{eq1}
\begin{split}
A & = \frac{\pi r^2}{2} \\
 & = \frac{1}{2} \pi r^2
\end{split}
\end{equation}
\end{lstlisting}
\end{minipage}
\end{tabular}


\chapter{Chromodynamics}

In this last chapter we want to give a quick introduction to chromodynamics or the theory of strong interaction. This is really just a small overview of how the theory is built and how we can apply our Hamilton formalism.
Let's start with a fundamental observation:

\begin{itemize}
\item \textbf{Electrodynamics:} \\
If we want to seperate an electron and a positron, we have to invest a finite energy.
\item \textbf{Chromodynamics:} \\
If we want to seperate a quark and an antiquark, we would need an infinite amount of energy. The field lines form a tube and get closer and closer the more we seperate the particles. The field energy and the fluctuations raise, so that we create new quarks which bind again before we can seperate the old ones. This process is called "color confinement".
\end{itemize}

So the physical properties of these two interactions are very different but the mathematical description is quite similar, as we will see.
One often starts with the postulation of gauge invariance to introduce the theory of chromodynamics. We want to show a different approach with the same result: a matrix-generalization of electrodynamics.
In the end, we want to find a Lagrangian again and apply the Hamiltonian method. \\

Let us introduce matrices $\hat{T}^a$, where $a = 1, \dots, n$ with the following properties:
\begin{enumerate}
\item $[ \hat{T}^a , \hat{T}^b ] = t^{abc} \hat{T}^c$ \ with real-valued coefficients $t^{abc}$.
\item $\text{tr}(\hat{T}^a \hat{T}^b) = - 2 \delta^{ab}$.
\item $( \hat{T}^a )^\dag = - \hat{T}^a$.
\end{enumerate}

For $n=3$, we see that the following $3 \times 3$ matrices satisfy the required properties:
\begin{align}
\hat{T}^1 =
\begin{pmatrix}
0 & 0 & 0 \\
0 & 0 & - 1 \\
0 & 1 & 0 \\
\end{pmatrix}, \ \ \
\hat{T}^2 =
\begin{pmatrix}
0 & 0 & 1 \\
0 & 0 & 0 \\
- 1 & 0 & 0 \\
\end{pmatrix}, \ \ \
\hat{T}^3 =
\begin{pmatrix}
0 & - 1 & 0 \\
1 & 0 & 0 \\
0 & 0 & 0 \\
\end{pmatrix}.
\end{align}

In group theory, this is often referred to the three-dimensional representation of SU(2). \footnote{Note that one can define $i \hat{T}^a \equiv \hat{\tilde{T}}^a$ and get the standard property of hermitian matrices: $( \hat{\tilde{T}}^a )^\dag = - \hat{\tilde{T}}^a$.} \\

In nature, we observe the case $n=8$ which corresponds to the group SU(3). It turns out that this group describes strong interaction very well. The number 3 can be referred to the 3 colors of chromodynamics. \\

The following developement of the theory doesn't depend on the number $n$ of matrices. For different $n$, we can get a different description. Just remember that $n=8$ is the case of strong interaction (chromodynamics).

\pagebreak

With the help of our matrices, we generalize the theory of electrodynamics in the following way. Instead of one electromagnetic potential field $A_{\mu}$, we take $n$ different electromagnetic potentials $A_{\mu}^a$:
\begin{align*}
\arraycolsep=10pt\def\arraystretch{1.6}
\begin{array}{ccccc}
 &  & Electrodynamics & & Chromodynamics \\
q_i(t) & \longrightarrow & A_{\mu}(\bar{x},t) & \longrightarrow & A_{\mu}^a(\bar{x},t) \\
i & \longrightarrow & \{ \bar{x},\mu \} & \longrightarrow & \{ \bar{x},a,\mu \}
\end{array}
\end{align*}
and define the "gluon field" $\hat{A}_{\mu}(\bar{x},t)$:
\begin{align}
\hat{A}_{\mu}(\bar{x},t) = \sum_{a=1}^n \ A_{\mu}^a(\bar{x},t) \hat{T}^a.
\end{align}

A simple generalization would be:

\begin{itemize}
\item \textbf{Electrodynamics:}
\begin{align}
L_{el}[A_{\mu}(\bar{x}), \dot{A}_{\mu}(\bar{x})] = - \frac{1}{4} \displaystyle\int d^3 x \ F_{\mu \nu}(\bar{x}) F^{\mu \nu}(\bar{x})
\end{align}
with the electromagnetic field tensor $F_{\mu \nu} = \partial_{\mu} A_{\nu} - \partial_{\nu} A_{\mu}$.
\item \textbf{Chromodynamics:}
\begin{align}
L_{ch}[A_{\mu}^a(\bar{x}), \dot{A}_{\mu}^a(\bar{x})] = \frac{1}{8} \displaystyle\int d^3 x \ \text{tr}\left(\hat{F}_{\mu \nu}(\bar{x}) \hat{F}^{\mu \nu}(\bar{x}) \right)
\end{align}
with the "gluon field strength tensor" $\hat{F}_{\mu \nu} = \partial_{\mu} \hat{A}_{\nu} - \partial_{\nu} \hat{A}_{\mu}$.
\end{itemize}

When we write it in the form
\begin{align}
\hat{F}_{\mu \nu}(\bar{x}) = \sum_{a=1}^n F_{\mu \nu}^a(\bar{x}) \hat{T}^a ,
\end{align}
we get for the Lagrangian of chromodynamics:
\begin{align}
L_{ch} = - \frac{1}{4} \displaystyle\int d^3 x \ \sum_{a=1}^n \left( F_{\mu \nu}^a F^{a, \mu \nu} \right) .
\end{align}

But such a simple generalization doesn't lead to chromodynamics. We have to add another term and take
\begin{align}
\hat{F}_{\mu \nu} &= \partial_{\mu} \hat{A}_{\nu} - \partial_{\nu} \hat{A}_{\mu} + g [ \hat{A}_{\mu},\hat{A}_{\nu} ] \\
F_{\mu \nu}^a &= \partial_{\mu} A_{\nu}^a - \partial_{\nu} A_{\mu}^a + g t^{abc} A_{\mu}^b A_{\nu}^c
\end{align}
as the gluon field strength tensor analog to the electromagnetic field tensor. \\
This looks very similar to the curvature tensor of differential geometry. In fact, this extra term conserves the antisymmetry and vanishes in the case of electrodynamics ($n=1$). \\
The equations of motion are non-linear now and the principle of superposition doesn't hold anymore (gluon fields interact with each other).

\pagebreak

The Hamiltonian procedure is the same, just a bit more complicated. We define
\begin{align}
A_{\mu}^a(\bar{x}) \ \ \ \longrightarrow \ \ \ \pi_{\mu}^a(\bar{x}) = \frac{\delta L}{\delta \dot{A}^{a, \mu}(\bar{x})}.
\end{align}
Then we build the Hamiltonian and get again a primary constraint.
It turns out that the primary constraint has the same form like in electrodynamics:
\begin{align}
\phi_1 = \pi_0^a(\bar{x}) = 0.
\end{align}

The secondary constraint reads now:
\begin{align}
\phi_2 = \nabla \cdot \bar{\pi}^a + g t^{abc} \left( \bar{A}^b \bar{\pi}^c \right) = 0.
\end{align}

The procedure ends here and both constraints turn out to be first-class.
The last term can be interpreted like the appearance of a charge in analogy to electrodynamics. If this term isn't zero, then we have interaction between different gluons. \\

Which transformations do they generate?
\begin{align}
\delta g = \varepsilon_a \{ g,\phi_a \}
\end{align}

We generalize this to arbitrary matrices $\hat{\omega}(\bar{x})$ by letting $\omega^a(\bar{x})$ be arbitrary functions:
\begin{align}
\hat{\omega}(\bar{x}) = \sum_{a=1}^n \ \omega^a(\bar{x}) \hat{T}^a.
\end{align}

After calculating the Poisson bracket with the constraints, we see that the space part changes like
\begin{align}
\hat{\bar{A}} \ \ &\longrightarrow \ \ \hat{\bar{A}}' = e^{- \hat{\omega}} \hat{\bar{A}} e^{\hat{\omega}} + e^{- \hat{\omega}} \nabla e^{\hat{\omega}} \\
\hat{\bar{E}} \ \ &\longrightarrow \ \ \hat{\bar{E}}' = e^{- \hat{\omega}} \hat{\bar{E}} e^{\hat{\omega}}
\end{align}
and $\hat{\omega}^\dag = - \hat{\omega}$ is antihermite because of the properties of $\hat{T}^a$. \\
This is quite similar to electrodynamics, where we had:
\begin{align}
\bar{A}(\bar{x}) \ \ \longrightarrow \ \ \bar{A}'(\bar{x}) = \bar{A}(\bar{x}) + \nabla \omega(\bar{x}) .
\end{align}

One realizes that the field strength cannot be a physical quantity anymore but
\begin{align}
\text{tr} ( (\hat{\bar{E}}')^2 ) = \text{tr} ( e^{- \hat{\omega}} \hat{\bar{E}} e^{\hat{\omega}} e^{- \hat{\omega}} \hat{\bar{E}} e^{\hat{\omega}} ) = \text{tr} ( \hat{\bar{E}}^2 ),
\end{align}
because of the gauge invariance. \\

Our small overview ends with this short introduction. It is left to say that the theory turns out to be very complicated. Because of the non-linearity, gluons can interact with each other and create new gluons. Curious readers are advised to get familiar with the Yang-Mills theory.


\section{Multi-armed Bandits}
\subsection{Exercise 2.1}
\subsubsection*{Q}
In $\varepsilon$-greedy action selection, for the case of two actions and $\varepsilon = 0.5$, what is the probability that the greedy action is selected?

\subsubsection*{A}
0.75.

\subsection{Exercise 2.2: Bandit example}
\subsubsection*{Q}
Consider a $k$-armed bandit problem with $k = 4$ actions, denoted 1, 2, 3, and 4. Consider applying to this problem a bandit algorithm using $\varepsilon$-greedy action selection, sample-average action-value estimates, and initial estimates of $Q_1(a) = 0$, for all $a$. Suppose the initial sequence of actions and rewards is $A_1 = 1$, $R_1 =1$, $A_2 =2$, $R_2 =1$, $A_3 =2$, $R_3 =2$, $A_4 =2$, $R_4 =2$, $A_5 =3$, $R_5 =0$. On some of these time steps the $\varepsilon$ case may have occurred, causing an action to be selected at random. On which time steps did this definitely occur? On which time steps could this possibly have occurred?

\subsubsection*{A}
$A_2$ and $A_5$ were definitely exploratory. Any of the others \emph{could} have been exploratory.

\subsection{Exercise 2.3}
\subsubsection*{Q}
In the comparison shown in Figure 2.2, which method will perform best in the long run in terms of cumulative reward and probability of selecting the best action? How much better will it be? Express your answer quantitatively.

\subsubsection*{A}
The $\varepsilon = 0.01$ will perform better because in both cases as $t \to \infty$ we have $Q_t \to q_*$. The total reward and probability of choosing the optimal action will therefore be 10 times larger in this case than for $\varepsilon = 0.1$.

\subsection{Exercise 2.4}
\label{ex:2.4}
\subsubsection*{Q}
If the step-size parameters, $\alpha_n$, are not constant, then the estimate $Q_n$ is a weighted average of previously received rewards with a weighting different from that given by (2.6). What is the weighting on each prior reward for the general case, analogous to (2.6), in terms of the sequence of step-size parameters?

\subsubsection*{A}
Let $\alpha_0 = 1$, then
\begin{equation}
    Q_{n + 1} = \left(\prod_{i=1}^n (1 - \alpha_i) \right) Q_1 + \sum_{i = 1}^{n}  \alpha_{i} R_{i} \prod_{k = i + 1}^n
(1 - \alpha_k).
\end{equation}
Where $\prod_{i=x}^y f(i) \doteq 1$ if $x > y$.

\subsection{Exercise 2.5 (programming)}
\subsubsection*{Q}
Design and conduct an experiment to demonstrate the difficulties that sample-average methods have for non-stationary problems. Use a modified version of the 10-armed testbed in which all the $q_*(a)$ start out equal and then take independent random walks (say by adding a normally distributed increment with mean zero and standard deviation 0.01 to all the $q_*(a)$ on each step). Prepare plots like Figure 2.2 for an action-value method using sample averages, incrementally computed, and another action-value method using a constant step-size parameter, $\alpha$ = 0.1. Use $\varepsilon$ = 0.1 and longer runs, say of 10,000 steps.

\subsubsection*{A}
\ProgrammingExercise

\includegraphics[width=\textwidth]{\ProjectDir/data/exercise_output/ex_2_5/learning_curve.png}

\subsection{Exercise 2.6: Mysterious Spikes}
\subsubsection*{Q}
The results shown in Figure 2.3 should be quite reliable because they are averages over 2000 individual, randomly chosen 10-armed bandit tasks. Why, then, are there oscillations and spikes in the early part of the curve for the optimistic method? In other words, what might make this method perform particularly better or worse, on average, on particular early steps?

\subsubsection*{A}
At some point after step 10, the agent will find the optimal value. It will then choose this value greedily. The small step-size parameter (small relative to the initialisation value of 5) means that the estimate of the optimal value will converge slowly towards its true value.\\

It is likely that this true value is less than 5. This means that, due to the small step size, one of the sub-optimal actions will still have a value close to 5. Thus, at some point, the agent begins to act sub-optimally again.

\subsection{Exercise 2.7: Unbiased Constant-Step-Trick}
\subsubsection*{Q}
In most of this chapter we have used sample averages to estimate action values because sample averages do not produce the initial bias that constant step sizes do (see the analysis in (2.6)). However, sample averages are not a completely satisfactory solution because they may perform poorly on non-stationary problems. Is it possible to avoid the bias of constant step sizes while retaining their advantages on non-stationary problems? One way is to use a step size of
\begin{equation}
    \beta_t \doteq \alpha / \bar{o}_t,
\end{equation}
where $\alpha > 0$ is a conventional constant step size and $\bar{o}_t$ is a trace of one that starts at 0:
\begin{equation}
    \bar{o}_{t+1} = \bar{o}_t + \alpha (1 - \bar{o}_t)
\end{equation}
for $t \geq 1$ and with $\bar{o}_1 \doteq \alpha$.\\

Carry out an analysis like that in (2.6) to show that $\beta_t$ is an exponential recency-weighted average \emph{without initial bias}.

\subsubsection*{A}
Consider the answer to \hyperref[ex:2.4]{Exercise 2.4}. There is no dependence of $Q_k$ on $Q_1$ for $k > 1$ since $\beta_1 = 1$. Now it remains to show that the weights in the remaining sum decrease as we look further into the past. That is
\begin{equation}
    w_i = \beta_i \prod_{k = i + 1}^{n} (1 - \beta_k)
\end{equation}
increases with $i$ for fixed n. For this, observe that
\begin{equation}
    \frac{w_{i+1}}{w_i} = \frac{\beta_{i+1}}{\beta_i(1 - \beta_{i + 1})} = \frac{1}{1 - \alpha} > 1
\end{equation}
where we have assumed $\alpha < 1$. If $\alpha = 1$ then $\beta_t = 1 \,\, \forall \, t$.

\subsection{Exercise 2.8: UCB Spikes}
\subsubsection*{Q}
In Figure 2.4 the UCB algorithm shows a distinct spike in performance on the 11th step. Why is this? Note that for your answer to be fully satisfactory it must explain both why the reward increases on the 11th step and why it decreases on the subsequent steps. Hint: if $c = 1$, then the spike is less prominent.

\subsubsection*{A}
In the first 10 steps the agent cycles through all of the actions because when $N_t(a) = 0$ then $a$ is considered maximal. On the 11th step the agent will most often then choose greedily. The agent will continue to choose greedily until $\mathrm{ln}(t)$ overtakes $N_t(a)$ for one of the other actions, in which case the agent begins to explore again hence reducing rewards.\\

Note that, in the long run, $N_t = O(t)$ and $\mathrm{ln}(t) / t \to 1$. So this agent is `asymptotically greedy'.

\subsection{Exercise 2.9}
\subsubsection*{Q}
Show that in the case of two actions, the soft-max distribution is the same as that given by the logistic, or sigmoid, function often used in statistics and artificial neural networks.

\subsubsection*{A}
Let the two actions be denoted by 0 and 1. Now
\begin{equation}
    \P{}(A_t = 1) = \frac{e^{H_t(1)}}{e^{H_t(1)} + e^{H_t(0)}} = \frac{1}{1 + e^{-x}},
\end{equation}
where $x = H_t(1) - H_t(0)$ is the relative preference of 1 over 0.

\subsection{Exercise 2.10}
\subsubsection*{Q}
Suppose you face a 2-armed bandit task whose true action values change randomly from time step to time step. Specifically, suppose that, for any time step, the true values of actions 1 and 2 are respectively 0.1 and 0.2 with probability 0.5 (case A), and 0.9 and 0.8 with probability 0.5 (case B). If you are not able to tell which case you face at any step, what is the best expectation of success you can achieve and how should you behave to achieve it? Now suppose that on each step you are told whether you are facing case A or case B (although you still don’t know the true action values). This is an associative search task. What is the best expectation of success you can achieve in this task, and how should you behave to achieve it?

\subsubsection*{A}
I assume the rewards are stationary.\\

One should choose the action with the highest expected reward. In the first case, both action 1 and 2 have expected value of 0.5, so it doesn't matter which you pick.\\

In the second case one should run a normal bandit method separately on each colour. The expected reward from identifying the optimal actions in each case is 0.55.

\subsection{Exercise 2.11 (programming)}
\subsubsection*{Q}
Make a figure analogous to Figure 2.6 for the non-stationary case outlined in Exercise 2.5. Include the constant-step-size $\varepsilon$-greedy algorithm with $\alpha=0.1$. Use runs of 200,000 steps and, as a performance measure for each algorithm and parameter setting, use the average reward over the last 100,000 steps.

\subsubsection*{A}
\ProgrammingExercise

\includegraphics[width=\textwidth]{\ProjectDir/data/exercise_output/ex_2_11/action_values.png}

\includegraphics[width=\textwidth]{\ProjectDir/data/exercise_output/ex_2_11/parameter_study.png}


\section{Output}
\lstinputlisting[style=Plain]{../src/lab_01_output.txt}

\newpage
% \begin{lstlisting}[style=Plain]
\begin{verbatim}
\section{Basics of \LaTeX{}}

\subsection{Simplifying Fractions}
Consider the function
$$
f(x) = \frac{x^2 - 1}{x + 1}.
$$
To simplify this funciton we can factor the numerator and cancel like terms
\begin{align*}
    f(x)
       & = \frac{x^2 - 1}{x + 1}
    \\ & = \frac{(x - 1) (x + 1)}{x + 1}
    \\ & = x - 1.
\end{align*}

\subsection{Matrix}
A general $3 \times 3$ matrix $A$ has the form
$$
A =
\begin{bmatrix}
    1 & 2 & 3 \\
    4 & 5 & 6 \\
    7 & 8 & 9 \\
\end{bmatrix}.
$$

\subsection{The Millennium Prize Problems}
\emph{The Millennium Prize Problems} are seven problems in mathematics that were stated by
the \textbf{Clay Mathematics Institute} on May 24, 2000. The problems are
\begin{enumerate}[(1)]
  \item Birch and Swinnerton-Dyer conjecture,
  \item Hodge conjecture,
  \item Navier–Stokes existence and smoothness,
  \item P versus NP problem,
  \item Poincar\'{e} conjecture,
  \item Riemann hypothesis,
  \item Yang–Mills existence and mass gap.
\end{enumerate}
\textsc{The Riemann zeta function} is defined for complex $s$ with real part greater than $1$
by the absolutely convergent infinite series
$$
\zeta(s) = \sum_{n=1}^{\infty} \frac{1}{n^s} = \frac{1}{1^s} + \frac{1}{2^s} + \frac{1}{3^s} + \cdots
$$
The practical uses of the Riemann hypothesis include many propositions known true under the
Riemann hypothesis, and some that can be shown to be equivalent to the Riemann hypothesis:
\begin{itemize}
  \item Distribution of prime numbers,
  \item Growth of arithemtic functions,
  \item Large prime gap conjecture,
  \item Criteria equivalent to the Riemann hypothesis.
\end{itemize}
\end{verbatim}
% \end{lstlisting}


%----------------------------------------------------------
%

\documentclass[10pt,letterpaper,oneside,notitlepage]{article}
%\documentclass{report}%
\usepackage{algorithm}
\usepackage{algpseudocode}
\usepackage{enumitem}
\usepackage{nomencl}
\usepackage{amsmath}%
%\usepackage{amsfonts}%
%\usepackage{amssymb}%
%\usepackage{graphicx}
%----------------------------------------------------------
\makenomenclature
%\theoremstyle{plain}
%\newtheorem{acknowledgement}{Acknowledgement}
%\newtheorem{definition}{Definition}
%\newtheorem{remark}{Remark}
%\numberwithin{equation}{section}
%-----------------------------------------------------------
\begin{document}
\title{Interpolation of DCMs}
\author{Bonnie Jonkman}
\maketitle

\section{Logarithmic maps for DCMs}

For any direction cosine matrix (DCM), $\Lambda$,
the logarithmic map for the matrix is a skew-symmetric matrix, $\lambda$:

\begin{equation}
\label{EqLog}
\lambda
= \log( \Lambda )
= \begin{bmatrix}
		0          &  \lambda_3 & -\lambda_2 \\
		-\lambda_3 &  0         &  \lambda_1 \\
		 \lambda_2 & -\lambda_1 &  0
	\end{bmatrix}
\end{equation}

\section{Matrix exponentials}

The angle of rotation for the skew-symmetric matrix, $\lambda$ is
\begin{equation}
\label{EqRotationAng}
\theta = \left\|\lambda\right\| = \sqrt{{\lambda_1}^2+{\lambda_2}^2+{\lambda_3}^2}
\end{equation}

The matrix exponential is
\begin{equation}
\label{EqExp}
 \Lambda = \exp(\lambda) =
 \left\{
 \begin{matrix}
	I                                                                              &  \theta = 0 \\
	I + \frac{\sin\theta}{\theta}\lambda + \frac{1-\cos\theta}{\theta^2}\lambda^2  &  \theta > 0 \\
 \end{matrix}
 \right.
\end{equation}

\section{Solving for $\lambda$}

If the logarithmic map and matrix exponential are truly inverses, we need
\begin{equation}
\exp(\log(\Lambda)) = \Lambda.
\end{equation}
Using the expression for $\lambda$ from Equation \ref{EqLog}, we get
\begin{equation}
\label{EqExpMatrix}
	\exp\left(
 \begin{bmatrix}
		0          &  \lambda_3 & -\lambda_2 \\
		-\lambda_3 &  0         &  \lambda_1 \\
		 \lambda_2 & -\lambda_1 &  0
	\end{bmatrix}
\right)	= \Lambda =
 \begin{bmatrix}
  \Lambda_{11} & \Lambda_{12} & \Lambda_{13} \\
  \Lambda_{21} & \Lambda_{22} & \Lambda_{23} \\
  \Lambda_{31} & \Lambda_{32} & \Lambda_{33} \\
 \end{bmatrix}
\end{equation}

Doing a little algebra for $\theta > 0$, Equation \ref{EqExpMatrix} becomes
\begin{equation}
\label{EqMatrixAlgebra}
\Lambda =
 \begin{bmatrix}
  1-\frac{1-\cos\theta}{\theta^2}\left( \lambda_3^2 + \lambda_2^2\right)
&   \frac{\sin\theta}{\theta}\lambda_3+\frac{1-\cos\theta}{\theta^2}\lambda_1\lambda_2
&  -\frac{\sin\theta}{\theta}\lambda_2+\frac{1-\cos\theta}{\theta^2}\lambda_1\lambda_3 \\
   -\frac{\sin\theta}{\theta}\lambda_3+\frac{1-\cos\theta}{\theta^2}\lambda_1\lambda_2
&  1-\frac{1-\cos\theta}{\theta^2}\left( \lambda_3^2 + \lambda_1^2\right)
&  \frac{\sin\theta}{\theta}\lambda_1+\frac{1-\cos\theta}{\theta^2}\lambda_2\lambda_3  \\
   \frac{\sin\theta}{\theta}\lambda_2+\frac{1-\cos\theta}{\theta^2}\lambda_1\lambda_3
& -\frac{\sin\theta}{\theta}\lambda_1+\frac{1-\cos\theta}{\theta^2}\lambda_2\lambda_3
&  1-\frac{1-\cos\theta}{\theta^2}\left( \lambda_2^2 + \lambda_1^2\right)                    \\
 \end{bmatrix}
\end{equation}
It follows that the trace is
\begin{eqnarray*}
\mathrm{Tr}(\Lambda) &=& 3 - 2\frac{1-\cos\theta}{\theta^2}(\lambda_1^2 + \lambda_2^2 + \lambda_3^2) \\
               &=& 3 - 2\left(1-\cos\theta\right) \\
               &=& 1 + 2\cos\theta
\end{eqnarray*} or
\begin{equation}
\theta= \begin{matrix} \cos^{-1}\left(\frac{1}{2}\left(\mathrm{Tr}(\Lambda)-1\right)\right) & \theta \in \left[0,\pi\right]\end{matrix}
\end{equation}
It also follows that
\begin{equation}
\Lambda - \Lambda^T=\frac{2\sin\theta}{\theta}
 \begin{bmatrix}
  0  &   \lambda_3  &  -\lambda_2 \\
   -\lambda_3 &  0 &  \lambda_1  \\
   \lambda_2 & -\lambda_1 &  0 \\
 \end{bmatrix}
\end{equation} or, when $\sin\theta \neq 0$
\begin{equation}
\label{EqLambdaSinNot0}
\lambda = \frac{\theta}{2\sin\theta} \left( \Lambda - \Lambda^T\right)
\end{equation}
We need an equation that works when $\sin\theta$ approaches 0, that is, when $\theta$ approaches 0 or $\theta$ approaches $\pi$. When $\theta$ approaches 0, Equation \ref{EqLambdaSinNot0} actually behaves well:
\begin{equation}
\lim_{\theta \to 0}\frac{\theta}{2\sin\theta} = \frac{1}{2}
\end{equation}
and since $\theta$ is the $l_2$ norm of the individual components of $\lambda$, it follows that they approach zero, and we get
\begin{equation}
\label{EqLambdaTheta0}
\lambda = 0
\end{equation}
However, when $\theta$ approaches $\pi$, $\Lambda - \Lambda^T$ approaches 0, and
\begin{equation}
\lim_{\theta \to \pi}\frac{\theta}{2\sin\theta} = \infty
\end{equation}
We need a different method to find $\lambda$. Going back to Equations \ref{EqExpMatrix} and \ref{EqMatrixAlgebra}, we can compute the following:
\begin{equation}
\Lambda_{11}+\Lambda_{22}-\Lambda_{33} = 1 - \frac{2\lambda_3^2(1-\cos\theta)}{\theta^2}
\end{equation}
%\begin{equation}
%\Lambda_{11}-\Lambda_{22}+\Lambda{33} = 1 - \frac{1-\cos\theta}{\theta^2}\left( 2\lambda_2^2\right)
%\end{equation}
%\begin{equation}
%-\Lambda_{11}+\Lambda_{22}+\Lambda{33} = 1 - \frac{1-\cos\theta}{\theta^2}\left( 2\lambda_1^2\right)
%\end{equation}
or
\begin{equation}
\label{EqLambda3}
\lambda_3 = \pm \theta\sqrt{ \frac{\left(1 + \Lambda_{33} - \Lambda_{11} - \Lambda_{22}\right)}{2\left(1-\cos\theta\right)}   }
\end{equation}
Equations for the other two components of $\lambda$ are similar:
\begin{equation}
\label{EqLambda1}
\lambda_1 = \pm \theta\sqrt{ \frac{\left(1 + \Lambda_{11} - \Lambda_{22} - \Lambda_{33}\right)}{2\left(1-\cos\theta\right)}   }
\end{equation}
\begin{equation}
\label{EqLambda2}
\lambda_2 = \pm \theta\sqrt{ \frac{\left(1 + \Lambda_{22} - \Lambda_{11} - \Lambda_{33}\right)}{2\left(1-\cos\theta\right)}   }
\end{equation}
Equations \ref{EqLambda3}-\ref{EqLambda2} give us the magnitude, not the sign of the vector we are looking for. Here is one possibility for choosing the sign:
If $(\lambda_1) \neq 0$, choose $\mathrm{sign}(\lambda_1)$ positive.
\begin{equation}
\Lambda_{12}+\Lambda_{21} = \frac{2\left(1-\cos\theta\right)}{\theta^2}\lambda_1\lambda_2
\end{equation}
so
\begin{equation}
\mathrm{sign}(\lambda_2) = \mathrm{sign}(\Lambda_{12}+\Lambda_{21})
\end{equation}
and similarly,
\begin{equation}
\mathrm{sign}(\lambda_3) = \mathrm{sign}(\Lambda_{13}+\Lambda_{31})
\end{equation}
If $(\lambda_1) = 0$, similar arguments can be used to choose $\mathrm{sign}(\lambda_2)$ positive, and
\begin{equation}
\mathrm{sign}(\lambda_3) = \mathrm{sign}(\Lambda_{23}+\Lambda_{32})
\end{equation}
At this point, the relationships between the components of $\lambda$ are set, so we have computed $\pm\lambda$. If $\theta=\pi$, both values are a solution, so this good enough.

If $\theta$ is close to $\pi$, we will need to determine if we have the negative of the solution. This is required for numerical stability of the solution.
In this case, $\Lambda-\Lambda^T$ is not exactly zero, so we can look at the sign of the solution we would have computed if we had used Equation \ref{EqLambdaSinNot0}:
\begin{equation}
\Lambda_{23}-\Lambda_{32} = \left|\frac{\sin\theta}{\theta}\right|\lambda_1
\end{equation}
\begin{equation}
\Lambda_{31}-\Lambda_{13} = \left|\frac{\sin\theta}{\theta}\right|\lambda_2
\end{equation}
\begin{equation}
\Lambda_{12}-\Lambda_{21} = \left|\frac{\sin\theta}{\theta}\right|\lambda_3
\end{equation}
For numerical reasons, we don't want to use these equations to get the magnitude of the components of $\lambda$, but we can look at the sign of the element with largest magnitude and ensure our $\lambda$ has the same sign.

\section{Interpolation}
\subsection{Periodicity of solutions}

Given $\lambda_k = \lambda \left( 1 + \frac{2k\pi}{\left\| \lambda \right\|}\right)$ for any integer $k$, it follows that
\begin{equation}
\theta_k = \left| 1 + \frac{2k\pi}{\left\|\lambda\right\|}\right| \theta
\end{equation}
or
\begin{equation}
\theta_k =  \left|\theta + 2k\pi\right|
\end{equation}

\begin{eqnarray*}
\Lambda_k
&=&  \exp(\lambda_k) \\
&=&  I + \frac{\sin\theta_k}{\theta_k}\lambda_k + \frac{1-\cos\theta_k}{\theta_k^2}\lambda_k^2  \\
&=&  I + \frac{\sin\left|\theta + 2k\pi\right|}{\left|\theta + 2k\pi\right|}\left( \frac{\theta+2k\pi}{\theta}\right)\lambda + \frac{1-\cos\left|\theta + 2k\pi\right|}{\left|\theta + 2k\pi\right|^2}\left(\frac{\theta+2k\pi}{\theta}\right)^2\lambda^2  \\
&=&  I + \frac{\sin\left|\theta + 2k\pi\right|}{\theta}
         \frac{\theta + 2k\pi}{\left|\theta + 2k\pi\right|}\lambda + \frac{1-\cos\left|\theta + 2k\pi\right|}{\theta^2}\lambda^2\\
&=&  I + \frac{\sin\theta}{\theta} \lambda + \frac{1-\cos\theta}{\theta^2}\lambda^2\\
&=& \exp(\lambda) \\
&=& \Lambda \\
\end{eqnarray*}

Thus, if $\lambda$ is one solution to $\log(\Lambda)$, then so is
$\lambda_k = \lambda \left( 1 + \frac{2k\pi}{\left\| \lambda \right\|}\right)$ for any integer k.

\subsection{Finding values of $\lambda$ for interpolation}
Given a set of $\lambda^j$ to be interpolated, find equivalent $\tilde{\lambda}^j$ for integers $j=1,2,...n$:
Set $\tilde{\lambda}^1 = \lambda^1$.
For each $j\in\left[2,n\right]$,
check to see if $\tilde{\lambda}^{j-1}$ is closer (in the $l_2$-norm sense) to
$\lambda^j$ or $\lambda^j \left( 1 + \frac{2\pi}{\left\| \lambda^j \right\|}\right)$.
If the latter, set $\tilde{\lambda}^{j}=\lambda^j \left( 1 + \frac{2\pi}{\left\| \lambda^j \right\|}\right)$ and continue checking if we need to add more $2\pi$ periods.
Otherwise, check to see if $\tilde{\lambda}^{j-1}$ is closer to
$\lambda^j$ or $\lambda^j \left( 1 - \frac{2\pi}{\left\| \lambda^j \right\|}\right)$.
If the latter, set $\tilde{\lambda}^{j}=\lambda^j \left( 1 - \frac{2\pi}{\left\| \lambda^j \right\|}\right)$ and continue checking if we need to subtract more $2\pi$ periods.
Otherwise set $\tilde{\lambda}^{j} = \lambda^j$.

Interpolation must occur on the $\tilde{\lambda}^{j}$ and not the $\lambda^j$.

\end{document}


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% Copyright (c) 2003-2018 by The University of Queensland
% http://www.uq.edu.au
%
% Primary Business: Queensland, Australia
% Licensed under the Apache License, version 2.0
% http://www.apache.org/licenses/LICENSE-2.0
%
% Development until 2012 by Earth Systems Science Computational Center (ESSCC)
% Development 2012-2013 by School of Earth Sciences
% Development from 2014 by Centre for Geoscience Computing (GeoComp)
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Seismic Wave Propagation in Two Dimensions}

\sslist{example08a.py}
We will now expand upon the previous chapter by introducing a vector form of
the wave equation. This means that the waves will have not only a scalar
magnitude as for the pressure wave solution, but also a direction. This type of
scenario is apparent in wave types that exhibit compressional and transverse
particle motion. An example of this would be seismic waves.

Wave propagation in the earth can be described by the elastic wave equation
\begin{equation} \label{eqn:wav} \index{wave equation}
\rho \frac{\partial^{2}u_{i}}{\partial t^2} - \frac{\partial
\sigma_{ij}}{\partial x_{j}} = 0
\end{equation}
where $\sigma$ is the stress given by
\begin{equation} \label{eqn:sigw}
 \sigma _{ij} = \lambda u_{k,k} \delta_{ij} + \mu (
u_{i,j} + u_{j,i})
\end{equation}
and $\lambda$ and $\mu$ represent Lame's parameters. Specifically for seismic
waves, $\mu$ is the propagation materials shear modulus.
In a similar process to the previous chapter, we will use the acceleration
solution to solve this PDE. By substituting $a$ directly for
$\frac{\partial^{2}u_{i}}{\partial t^2}$ we can derive the
acceleration solution. Using $a$ we can see that \autoref{eqn:wav} becomes
\begin{equation} \label{eqn:wava}
\rho a_{i} - \frac{\partial
\sigma_{ij}}{\partial x_{j}} = 0
\end{equation}
Thus the problem will be solved for acceleration and then converted to
displacement using the backwards difference approximation as for the acoustic
example in the previous chapter.

Consider now the stress $\sigma$. One can see that the stress consists of two
distinct terms:
\begin{subequations}
\begin{equation} \label{eqn:sigtrace}
\lambda u_{k,k} \delta_{ij}
\end{equation}
\begin{equation} \label{eqn:sigtrans}
\mu (u_{i,j} + u_{j,i})
\end{equation}
\end{subequations}
One simply recognizes in \autoref{eqn:sigtrace} that $u_{k,k}$ is the
trace of the displacement solution and that $\delta_{ij}$ is the
kronecker delta function with dimensions equivalent to $u$. The second term
\autoref{eqn:sigtrans} is the sum of $u$ with its own transpose. Putting these
facts together we see that the spatial differential of the stress is given by the
gradient of $u$ and the aforementioned operations. This value is then submitted
to the \esc PDE as $X$.
\begin{python}
g=grad(u); stress=lam*trace(g)*kmat+mu*(g+transpose(g))
mypde.setValue(X=-stress) # set PDE values
\end{python}
The solution is then obtained via the usual method and the displacement is
calculated so that the memory variables can be updated for the next time
iteration.
\begin{python}
accel = mypde.getSolution() #get PDE solution for acceleration
u_p1=(2.*u-u_m1)+h*h*accel #calculate displacement
u_m1=u; u=u_p1 # shift values by 1
\end{python}

Saving the data has been handled slightly differently in this example. The VTK
files generated can be quite large and take a significant amount of time to save
to the hard disk. To avoid doing this at every iteration a test is devised which
saves only at specific time intervals.

To do this there are two new parameters in our script.
\begin{python}
# data recording times
rtime=0.0 # first time to record
rtime_inc=tend/20.0 # time increment to record
\end{python}
Currently the PDE solution will be saved to file $20$ times between the start of
the modelling and the final time step. With these parameters set, an if
statement is introduced to the time loop
\begin{python}
if (t >= rtime):
	saveVTK(os.path.join(savepath,"ex08a.%05d.vtu"%n),displacement=length(u),\
    	acceleration=length(accel),tensor=stress)
    	rtime=rtime+rtime_inc #increment data save time
\end{python}
\verb!t! is the time counter. Whenever the recording time \verb!rtime! is less
then \verb!t! the solution is saved and \verb!rtime! is incremented. This
limits the number of outputs and increases the speed of the solver.

\section{Multi-threading}
The wave equation solution can be quite demanding on CPU time. Enhancements can
be made by accessing multiple threads or cores on your computer. This does not
require any modification to the solution script and only comes into play when
\esc is called from the shell. To use multiple threads \esc is called using
the \verb!-t! option with an integer argument for the number of threads
required. For example
\begin{verbatim}
$escript -t 4 example08a.py
\end{verbatim}
would call the script in this section and solve it using 4 threads.

The computation times on an increasing number of cores is outlined in
\autoref{tab:wpcores}.

\begin{table}[ht]
\begin{center}
\caption{Computation times for an increasing number of cores.}
\label{tab:wpcores}
\begin{tabular}{| c | c |}
\hline
Number of Cores & Time (s) \\
\hline
1 & 691.0 \\
2 & 400.0 \\
3 & 305.0 \\
4 & 328.0 \\
5 & 323.0 \\
6 & 292.0 \\
7 & 282.0 \\
8 & 445.0 \\ \hline
\end{tabular}
\end{center}
\end{table}

\section{Vector source on the boundary}
\sslist{example08b.py}
For this particular example, we will introduce the source by applying a
displacement to the boundary during the initial time steps. The source will
again be
a radially propagating wave but due to the vector nature of the PDE used, a
direction will need to be applied to the source.

The first step is to choose an amplitude and create the source as in the
previous chapter.
\begin{python}
U0=0.01 # amplitude of point source
# will introduce a spherical source at middle left of bottom face
xc=[ndx/2,0]

############################################FIRST TIME STEPS AND SOURCE
# define small radius around point xc
src_length = 40; print("src_length = ",src_length)
# set initial values for first two time steps with source terms
xb=FunctionOnBoundary(domain).getX()
y=source[0]*(cos(length(x-xc)*3.1415/src_length)+1)*\
                whereNegative(length(xb-src_length))
src_dir=numpy.array([0.,1.]) # defines direction of point source as down
y=y*src_dir
\end{python}
where \verb xc  is the source point on the boundary of the model. Note that
because the source is specifically located on the boundary, we have used the
\verb!FunctionOnBoundary! call to ensure the nodes used to define the source
are also located upon the boundary. These boundary nodes are passed to
source as \verb!xb!. The source direction is then defined as an $(x,y)$ array and multiplied by the
source function. The directional array must have a magnitude of $\left| 1
\right| $ otherwise the amplitude of the source will become modified. For this
example, the source is directed in the $-y$ direction.
\begin{python}
src_dir=numpy.array([0.,-1.]) # defines direction of point source as down
y=y*src_dir
\end{python}
The function can then be applied as a boundary condition by setting it equal to
$y$ in the general form.
\begin{python}
mypde.setValue(y=y) #set the source as a function on the boundary
\end{python}
The final step is to qualify the initial conditions. Due to the fact that we are
no longer using the source to define our initial condition to the model, we
must set the model state to zero for the first two time steps.
\begin{python}
# initial value of displacement at point source is constant (U0=0.01)
# for first two time steps
u=[0.0,0.0]*wherePositive(x)
u_m1=u
\end{python}

If the source is time progressive, $y$ can be updated during the
iteration stage. This is covered in the following section.

\begin{figure}[htp]
\centering
\subfigure[Example 08a at 0.025s ]{
\includegraphics[width=3in]{figures/ex08pw50.png}
\label{fig:ex08pw50}
}
\subfigure[Example 08a at 0.175s ]{
\includegraphics[width=3in]{figures/ex08pw350.png}
\label{fig:ex08pw350}
} \\
\subfigure[Example 08a at 0.325s ]{
\includegraphics[width=3in]{figures/ex08pw650.png}
\label{fig:ex08pw650}
}
\subfigure[Example 08a at 0.475s ]{
\includegraphics[width=3in]{figures/ex08pw950.png}
\label{fig:ex08pw950}
}
\label{fig:ex08pw}
\caption{Results of Example 08 at various times.}
\end{figure}
\clearpage

\section{Time variant source}

\sslist{example08b.py}
Until this point, all of the wave propagation examples in this cookbook have
used impulsive sources which are smooth in space but not time. It is however,
advantageous to have a time smoothed source as it can reduce the temporal
frequency range and thus mitigate aliasing in the solution.

It is quite
simple to implement a source which is smooth in time. In addition to the
original source function the only extra requirement is a time function. For
this example the time variant source will be the derivative of a Gaussian curve
defined by the required dominant frequency (\autoref{fig:tvsource}).
\begin{python}
#Creating the time function of the source.
dfeq=50 #Dominant Frequency
a = 2.0 * (np.pi * dfeq)**2.0
t0 = 5.0 / (2.0 * np.pi * dfeq)
srclength = 5. * t0
ls = int(srclength/h)
print('source length',ls)
source=np.zeros(ls,'float') # source array
ampmax=0
for it in range(0,ls):
    t = it*h
    tt = t-t0
    dum1 = np.exp(-a * tt * tt)
    source[it] = -2. * a * tt * dum1
    if (abs(source[it]) > ampmax):
        ampmax = abs(source[it])
    time[t]=t*h
\end{python}
\begin{figure}[ht]
\centering
\includegraphics[width=3in]{figures/source.png}
\caption{Time variant source with a dominant frequency of 50Hz.}
\label{fig:tvsource}
\end{figure}

We then build the source and the first two time steps via;
\begin{python}
# set initial values for first two time steps with source terms
y=source[0]
*(cos(length(x-xc)*3.1415/src_length)+1)*whereNegative(length(x-xc)-src_length)
src_dir=numpy.array([0.,-1.]) # defines direction of point source as down
y=y*src_dir
mypde.setValue(y=y) #set the source as a function on the boundary
# initial value of displacement at point source is constant (U0=0.01)
# for first two time steps
u=[0.0,0.0]*whereNegative(x)
u_m1=u
\end{python}

Finally, for the length of the source, we are required to update each new
solution in the iterative section of the solver. This is done via;
\begin{python}
# increment loop values
t=t+h; n=n+1
if (n < ls):
	y=source[n]**(cos(length(x-xc)*3.1415/src_length)+1)*\
                   whereNegative(length(x-xc)-src_length)
        y=y*src_dir; mypde.setValue(y=y) #set the source as a function on the
boundary
\end{python}

\section{Absorbing Boundary Conditions}
To mitigate the effect of the boundary on the model, absorbing boundary
conditions can be introduced. These conditions effectively dampen the wave energy
as they approach the boundary and thus prevent that energy from being reflected.
This type of approach is typically used when a model is shrunk to decrease
computational requirements. In practise this applies to almost all models,
especially in earth sciences where the entire planet or a large enough
portional of it cannot be modelled efficiently when considering small scale
problems. It is impractical to calculate the solution for an infinite model and thus ABCs allow
us the create an approximate solution with small to zero boundary effects on a
model with a solvable size.

To dampen the waves, the method of \citet{Cerjan1985}
where the solution and the stress are multiplied by a damping function defined
on $n$ nodes of the domain adjacent to the boundary, given by;
\begin{equation}
\gamma =\sqrt{\frac{| -log( \gamma _{b} ) |}{n^2}}
\end{equation}
\begin{equation}
y=e^{-(\gamma x)^2}
\end{equation}
This is applied to the bounding 20-50 pts of the model using the location
specifiers of \esc;
\begin{python}
# Define where the boundary decay will be applied.
bn=30.
bleft=xstep*bn; bright=mx-(xstep*bn); bbot=my-(ystep*bn)
# btop=ystep*bn # don't apply to force boundary!!!

# locate these points in the domain
left=x[0]-bleft; right=x[0]-bright; bottom=x[1]-bbot

tgamma=0.98   # decay value for exponential function
def calc_gamma(G,npts):
    func=np.sqrt(abs(-1.*np.log(G)/(npts**2.)))
    return func

gleft  = calc_gamma(tgamma,bleft)
gright = calc_gamma(tgamma,bleft)
gbottom= calc_gamma(tgamma,ystep*bn)

print('gamma', gleft,gright,gbottom)

# calculate decay functions
def abc_bfunc(gamma,loc,x,G):
    func=exp(-1.*(gamma*abs(loc-x))**2.)
    return func

fleft=abc_bfunc(gleft,bleft,x[0],tgamma)
fright=abc_bfunc(gright,bright,x[0],tgamma)
fbottom=abc_bfunc(gbottom,bbot,x[1],tgamma)
# apply these functions only where relevant
abcleft=fleft*whereNegative(left)
abcright=fright*wherePositive(right)
abcbottom=fbottom*wherePositive(bottom)
# make sure the inside of the abc is value 1
abcleft=abcleft+whereZero(abcleft)
abcright=abcright+whereZero(abcright)
abcbottom=abcbottom+whereZero(abcbottom)
# multiply the conditions together to get a smooth result
abc=abcleft*abcright*abcbottom
\end{python}
Note that the boundary conditions are not applied to the surface, as this is
effectively a free surface where normal reflections would be experienced.
Special conditions can be introduced at this surface if they are known. The
resulting boundary damping function can be viewed in
\autoref{fig:abconds}.

\section{Second order Meshing}
For stiff problems like the wave equation it is often prudent to implement
second order meshing. This creates a more accurate mesh approximation with some
increased processing cost. To turn second order meshing on, the \verb!rectangle!
function accepts an \verb!order! keyword argument.
\begin{python}
domain=Rectangle(l0=mx,l1=my,n0=ndx, n1=ndy,order=2) # create the domain
\end{python}
Other pycad functions and objects have similar keyword arguments for higher
order meshing.

Note that when implementing second order meshing, a smaller timestep is required
then for first order meshes as the second order essentially reduces the size of
the mesh by half.

\begin{figure}[ht]
 \centering
 \includegraphics[width=5in]{figures/ex08babc.png}
 \label{fig:abconds}
 \caption{Absorbing boundary conditions for example08b.py}
\end{figure}

\begin{figure}[htp]
\centering
\subfigure[Example 08b at 0.03s ]{
\includegraphics[width=3in]{figures/ex08sw060.png}
\label{fig:ex08pw060}
}
\subfigure[Example 08b at 0.16s ]{
\includegraphics[width=3in]{figures/ex08sw320.png}
\label{fig:ex08pw320}
} \\
\subfigure[Example 08b at 0.33s ]{
\includegraphics[width=3in]{figures/ex08sw660.png}
\label{fig:ex08pw660}
}
\subfigure[Example 08b at 0.44s ]{
\includegraphics[width=3in]{figures/ex08sw880.png}
\label{fig:ex08pw880}
}
\label{fig:ex08pw}
\caption{Results of Example 08b at various times.}
\end{figure}
\clearpage

\section{Pycad example}
\sslist{example08c.py}
To make the problem more interesting we will now introduce an interface to the
middle of the domain. In fact we will use the same domain as we did fora
different set of material properties on either side of the interface.

\begin{figure}[ht]
\begin{center}
\includegraphics[width=5in]{figures/gmsh-example08c.png}
\caption{Domain geometry for example08c.py showing line tangents.}
\label{fig:ex08cgeo}
\end{center}
\end{figure}

It is simple enough to slightly modify the scripts of the previous sections to
accept this domain. Multiple material parameters must now be defined and assigned
to specific tagged areas. Again this is done via
\begin{python}
lam=Scalar(0,Function(domain))
lam.setTaggedValue("top",lam1)
lam.setTaggedValue("bottom",lam2)
mu=Scalar(0,Function(domain))
mu.setTaggedValue("top",mu1)
mu.setTaggedValue("bottom",mu2)
rho=Scalar(0,Function(domain))
rho.setTaggedValue("top",rho1)
rho.setTaggedValue("bottom",rho2)
\end{python}
Don't forget that the source boundary must also be tagged and added so it can
be referenced
\begin{python}
# Add the subdomains and flux boundaries.
d.addItems(PropertySet("top",tblock),PropertySet("bottom",bblock),\
                                     PropertySet("linetop",l30))
\end{python}
It is now possible to solve the script as in the previous examples
(\autoref{fig:ex08cres}).

\begin{figure}[ht]
\centering
\includegraphics[width=4in]{figures/ex08c2601.png}
\caption{Modelling results of example08c.py at 0.2601 seconds. Notice the
refraction of the wave front about the boundary between the two layers.}
\label{fig:ex08cres}
\end{figure}


\documentclass[12pt]{article}
\usepackage[utf8]{inputenc}
\usepackage{xcolor}
\usepackage[T1]{fontenc}
\usepackage{pagecolor}
\usepackage{amssymb}
\usepackage{lmodern}
\usepackage{mathtools, nccmath}
\usepackage{courier}
\usepackage[overload]{empheq}
\usepackage[inline, shortlabels]{enumitem}
\usepackage{amsmath}
\usepackage{mathtools}
\color{white}
\title{Solution to Number theory \#2}
\author{@all.about.mathematics}

\pagecolor{black}
\begin{document}
\maketitle
\section{Problem}
\noindent\fbox{%
    \parbox{\textwidth}{%
        Is there a power of 2 ($>8$) such that that its digits can be rearranged to form another power of 2? (zeros are not allowed in leading digits, e.g. 032 is not allowed)
    }%
}
\section{solution}
We list out some simple and obvious facts here to help us with the problem:

1. Rearranging digits does not change the total number of digits.

2. At most 4 consecutive powers of 2 can have the same number of digits.

3. A number is equal to its digit sum modulo 9.

\noindent{Using these, we rephrase the question:}

\noindent\fbox{%
    \parbox{\textwidth}{%
    Is there a power of 2 such that there is another power of 2 with the same number of digits and the same remainder modulo 9?
    }%
}

Observing the remainders modulo 9 of some powers of 2, we conjecture that the remainders repeat themselves every 6 powers.
$$\because 64\equiv1 \mod 9$$
$$\therefore 2^{n+6}\equiv2^n\times 64\equiv 2^n \mod 9$$
Since the remainders modulo 9 repeat themselves every 6 powers and there can only be at most 4 powers of 2 can have the same digits, the answer to the question is NO.
\end{document}

Number theory/all.about.mathematics' questions/Number theory 2.tex


 %%%%%%%%%%%%%%%%%%%%%%
\section{Numerical Experiments}
\label{sec:NumExp}

For our numerical experiments, we calculated normalized geodesic lengths for a variety of regression and classification tasks.  In practice, this involved training a pair of randomly initialized models to the desired test loss value/accuracy/perplexity, and then attempting to connect that pair of models via the Dynamic String Sampling algorithm.  We also tabulated the average number of ``beads'', or the number intermediate models needed by the algorithm to connect two initial models.  For all of the below experiments, the reported losses and accuracies are on a restricted test set.  For more complete architecture and implementation details, see our \href{github.com/danielfreeman11/convex-nets}{GitHub page}.

The results are broadly organized by increasing model complexity and task difficulty, from easiest to hardest.  Throughout, and remarkably, we were able to easily connect models for every dataset and architecture investigated except the one explicitly constructed counterexample discussed in Appendix \ref{symdisc}.  Qualitatively, all of the models exhibit a transition from a highly convex regime at high loss to a non-convex regime at low loss, as demonstrated by the growth of the normalized length as well as the monotonic increase in the number of required ``beads'' to form a low-loss connection.

%\begin{table}[ht]
%  \centering
%  \begin{tabular}{c@{\quad}cc}
%    & a & b \\
%    1 & \includegraphics[width=.3\textwidth]{../Plots/normlengthquadratics}\fixedlabel{QUADRATICSfigsA}{1a}
%      & \includegraphics[width=.3\textwidth]{../Plots/numbeadsquadratics}\fixedlabel{QUADRATICSfigsB}{1b}  \\ \\
%    2 & \includegraphics[width=.3\textwidth]{../Plots/normlengthcubics}\fixedlabel{CUBICSfigsA}{2a}
%      & \includegraphics[width=.3\textwidth]{../Plots/numbeadscubics} \fixedlabel{CUBICSfigsB}{2b} \\ \\
%    3 & \includegraphics[width=.3\textwidth]{../Plots/normlengthMNIST}\fixedlabel{MNISTfigsA}{3a}
%      & \includegraphics[width=.3\textwidth]{../Plots/numbeadsMNIST}\fixedlabel{MNISTfigsB}{3b} \\ \\
%    4 & \includegraphics[width=.3\textwidth]{../Plots/normlengthCIFAR}\fixedlabel{CIFARfigsA}{4a}
%      & \includegraphics[width=.3\textwidth]{../Plots/numbeadsCIFAR}\fixedlabel{CIFARfigsB}{4b} \\ \\
%    5 & \includegraphics[width=.3\textwidth]{../Plots/normlengthPTB}\fixedlabel{PTBfigsA}{5a}
%      & \includegraphics[width=.3\textwidth]{../Plots/numbeadsPTB}\fixedlabel{PTBfigsB}{5b}
%
%  \end{tabular}
%  \caption{(Column a) Average normalized geodesic length and (Column b) average number of beads versus loss. (1) \href{github.com/danielfreeman11/convex-nets/tree/master/LaunchScripts/QUADRATIC.py}{A quadratic regression task}. (2) \href{github.com/danielfreeman11/convex-nets/tree/master/LaunchScripts/CUBIC.py}{A cubic regression task}. (3) A convnet for \href{github.com/danielfreeman11/convex-nets/tree/master/LaunchScripts/MNIST.py}{MNIST}. (4) A convnet inspired by \href{www.cs.toronto.edu/\%7Ekriz/cifar.html}{Krizhevsky} for \href{github.com/danielfreeman11/convex-nets/tree/master/LaunchScripts/CIFAR10.py}{CIFAR10}. (5) \href{github.com/danielfreeman11/convex-nets/tree/master/LaunchScripts/PTBRNN.py}{A RNN} inspired by \href{arxiv.org/pdf/1409.2329.pdf}{Zaremba} for PTB next word prediction.}
%  \label{FigTable}
%\end{table}

\subsection{Polynomial Regression}
\label{sec:PolyFuncs}
%%%%%%%%%%%%%%%%%%%%%%

 We studied a 1-4-4-1 fully connected multilayer perceptron style architecture with sigmoid nonlinearities and RMSProp/ADAM optimization.  For ease-of-analysis, we restricted the training and test data to be strictly contained in the interval $x\in[0,1]$ and $f(x)\in[0,1]$.  The number of required beads, and thus the runtime of the algorithm, grew approximately as a power-law, as demonstrated in Table \ref{FigTable} Fig. 1.  We also provide a visualization of a representative connecting path between two models of equivalent power in Appendix \ref{visualization}.

\begin{figure}
%\begin{figure}[h]
  \centering
%\begin{center}
%  \begin{tabular}{c@{\quad}cc}
%    & a & b \\
  \includegraphics[width=.34\textwidth]{../Plots/normlengthquadratics}%\fixedlabel{QUADRATICSfigsA}\caption{(1a)}
   \includegraphics[width=.34\textwidth]{../Plots/numbeadsquadratics} \\ %\fixedlabel{QUADRATICSfigsB}\caption{(1b)}  \\
    \includegraphics[width=.34\textwidth]{../Plots/normlengthcubics} %\fixedlabel{CUBICSfigsA}\caption{(2a)}
    \includegraphics[width=.34\textwidth]{../Plots/numbeadscubics}\\% \fixedlabel{CUBICSfigsB}\caption{(2b)} \\
    \includegraphics[width=.34\textwidth]{../Plots/normlengthMNIST}%\fixedlabel{MNISTfigsA}\caption{(3a)}
     \includegraphics[width=.34\textwidth]{../Plots/numbeadsMNIST}\\% \fixedlabel{MNISTfigsB}\caption{(3b)} \\
   \includegraphics[width=.34\textwidth]{../Plots/normlengthCIFAR}% \fixedlabel{CIFARfigsA}\caption{(4a)}
     \includegraphics[width=.34\textwidth]{../Plots/numbeadsCIFAR} \\ %\fixedlabel{CIFARfigsB}\caption{(4b)} \\
   \includegraphics[width=.34\textwidth]{../Plots/normlengthPTB}% \fixedlabel{PTBfigsA}\caption{(5a)}
   \includegraphics[width=.34\textwidth]{../Plots/numbeadsPTB} \\ %\fixedlabel{PTBfigsB}\caption{(5b)}
  \caption{(Column a) Average normalized geodesic length and (Column b) average number of beads versus loss. (1) \href{github.com/danielfreeman11/convex-nets/tree/master/LaunchScripts/QUADRATIC.py}{A quadratic regression task}. (2) \href{github.com/danielfreeman11/convex-nets/tree/master/LaunchScripts/CUBIC.py}{A cubic regression task}. (3) A convnet for \href{github.com/danielfreeman11/convex-nets/tree/master/LaunchScripts/MNIST.py}{MNIST}. (4) A convnet inspired by \href{www.cs.toronto.edu/\%7Ekriz/cifar.html}{Krizhevsky} for \href{github.com/danielfreeman11/convex-nets/tree/master/LaunchScripts/CIFAR10.py}{CIFAR10}. (5) \href{github.com/danielfreeman11/convex-nets/tree/master/LaunchScripts/PTBRNN.py}{A RNN} inspired by \href{arxiv.org/pdf/1409.2329.pdf}{Zaremba} for PTB next word prediction.}
  \label{FigTable}
\end{figure}
%\end{center}

 The cubic regression task exhibits an interesting feature around $L_0=.15$ in Table \ref{FigTable} Fig. 2, where the normalized length spikes, but the number of required beads remains low.  Up until this point, the cubic model is strongly convex, so this first spike seems to indicate the onset of non-convex behavior and a concomitant radical change in the geometry of the loss surface for lower loss.

%%%%%%%%%%%%%%%%%%%%%%
\subsection{Convolutional Neural Networks}
\label{sec:CNN}
%%%%%%%%%%%%%%%%%%%%%%

 To test the algorithm on larger architectures, we ran it on the MNIST hand written digit recognition task as well as the CIFAR10 image recognition task, indicated in Table \ref{FigTable}, Figs. 3 and 4.  Again, the data exhibits strong qualitative similarity with the previous models: normalized length remains low until a threshold loss value, after which it grows approximately as a power law.  Interestingly, the MNIST dataset exhibits very low normalized length, even for models nearly at the state of the art in classification power, in agreement with the folk-understanding that MNIST is highly convex and/or ``easy''.  The CIFAR10 dataset, however, exhibits large non-convexity, even at the modest test accuracy of 80\%.

%%%%%%%%%%%%%%%%%%%%%%
\subsection{Recurrent Neural Networks}
%%%%%%%%%%%%%%%%%%%%%%

 To gauge the generalizability of our algorithm, we also applied it to an LSTM architecture for solving the next word prediction task on the PTB dataset, depicted in Table \ref{FigTable} Fig. 5.  Noteably, even for a radically different architecture, loss function, and data set, the normalized lengths produced by the DSS algorithm recapitulate the same qualitative features seen in the above datasets---i.e., models can be easily connected at high perplexity, and the normalized length grows at lower and lower perplexity after a threshold value, indicating an onset of increased non-convexity of the loss surface.


\section{Problems to try before some fixed day}

\prob{}
{Thue's Note}{}{
    Let $S$ be a set of all positive integers which can be represented as
    $a^2+ 5b^2$ for some coprime integers $a,b$. Let $p$ be a prime number
    such that $p= 4n+ 3$ for some integer $n$. Show that if for some positive
    integer $k$ the numberk $p$ is in $S$, then $2p$ is in $S$ as well.
}

Konig's theorem, Dilworth theorem, Max flow Min cut