\chapter{Loops and strings}

Computers are often used to automate repetitive tasks, such as searching for text in documents.
Repeating tasks without making errors is something that computers do well and people do poorly.

In this chapter, we'll learn how to use \java{while} and \java{for} loops to add repetition to your code.
We'll also take a first look at \java{String} methods and solve some interesting problems.

%We have seen methods, like \java{countdown} and \java{factorial}, that use recursion to iterate.
%Although recursion is elegant and powerful, it takes some getting used to.
%Java provides language features that make iteration much easier: the \java{while} and \java{for} statements.

\section{The while statement}
\label{The_while_statement}

\index{while}
\index{loop!while}
\index{statement!while}

Using a \java{while} statement, we can repeat the same code multiple times:

%slr: 8-16-19
%\begin{code}
%int n = 3;
%while (n > 0) {
%    System.out.println(n);
%    n = n - 1;
%}
%System.out.println("Blastoff!");
%\end{code}
% height = 130 + 11 * num_lines
\begin{trinket} [230] {Blastoff.java}
public class Blastoff {

    public static void main(String[] args) {
       int n = 3;
       while (n > 0) {
          System.out.println(n);
          n = n - 1;
       }
       System.out.println("Blastoff!");
    }
}
\end{trinket}

Run the program.  What do you see?

Reading the code in English sounds like: ``Start with \java{n} set to 3.
While \java{n} is greater than zero, print the value of \java{n}, and reduce the value of \java{n} by 1.
When you get to zero, print Blastoff!''
%So the output is:
%
%\begin{stdout}
%3
%2
%1
%Blastoff!
%\end{stdout}
%slr: end 8-16-19

The flow of execution for a \java{while} statement is:

\begin{enumerate}

\item Evaluate the condition in parentheses, yielding \java{true} or \java{false}.

\item If the condition is \java{false}, skip the following statements in braces.

\item If the condition is \java{true}, execute the statements and go back to step 1.

\end{enumerate}

\index{loop}

This type of flow is called a {\bf loop}, because the last step ``loops back around'' to the first.
Figure~\ref{fig.while} shows this idea using a flowchart.

\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.9]{figs/while.pdf}
\caption{Flow of execution for a \java{while} loop.}
\label{fig.while}
\end{center}
\end{figure}

\index{loop body}
\index{infinite loop}
\index{loop!infinite}

The {\bf body} of the loop should change the value of one or more variables so that, eventually, the condition becomes \java{false} and the loop terminates.
Otherwise the loop will repeat forever, which is called an {\bf infinite loop}.

\begin{code}
int n = 3;
while (n > 0) {
    System.out.println(n);
    // n never changes
}
\end{code}

This example will print the number \java{3} forever, or at least until you terminate the program.
An endless source of amusement for computer scientists is the observation that the directions on shampoo, ``Lather, rinse, repeat,'' are an infinite loop.

In the first example, we can prove that the loop terminates when \java{n} is positive.
But in general, it is not so easy to tell whether a loop terminates.
For example, this loop continues until \java{n} is 1 (which makes the condition \java{false}):

%slr: 8-16-19
%\begin{code}
%while (n != 1) {
%    System.out.println(n);
%    if (n % 2 == 0) {         // n is even
%        n = n / 2;
%    } else {                  // n is odd
%        n = 3 * n + 1;
%    }
%}
%\end{code}
% height = 130 + 11 * num_lines
\begin{trinket} [270] {Collatz.java}
public class Collatz {

    public static void main(String[] args) {
       int n = 3;
       while (n != 1) {
          System.out.println(n);
          if (n % 2 == 0) {         // n is even
             n = n / 2;
          } else {                  // n is odd
             n = 3 * n + 1;
          }
       }
    }
}
\end{trinket}
%slr: end 8-16-19

Each time through the loop, the program displays the value of \java{n} and then checks whether it is even or odd.
If it is even, the value of \java{n} is divided by two.
If it is odd, the value is replaced by $3n+1$.
For example, if the starting value is 3, the resulting sequence is 3, 10, 5, 16, 8, 4, 2, 1.

Since \java{n} sometimes increases and sometimes decreases, there is no obvious proof that \java{n} will ever reach 1 and that the program will ever terminate.
For some values of \java{n}, such as the powers of two, we can prove that it terminates.
The previous example ends with such a sequence, starting when \java{n} is 16 (or $2^4$).

The hard question is whether this program terminates for {\em all} values of n.
So far, no one has been able to prove it {\em or} disprove it!
For more information, see \url{https://en.wikipedia.org/wiki/Collatz_conjecture}.
%The field of computer science is interested in these types of questions, because their answers give insight to the limits of what computers can and cannot do.

\section{Increment and decrement}

Here is another \java{while} loop example; this one displays the numbers 1 to 5.

%slr 8-16-19
%\begin{code}
%int i = 1;
%while (i <= 5) {
%    System.out.println(i);
%    i++;  // add 1 to i
%}
%\end{code}
% height = 130 + 11 * num_lines
\begin{trinket} [220] {Increment.java}
public class Increment {

    public static void main(String[] args) {
       int i = 1;
       while (i <= 5) {
          System.out.println(i);
          i++;  // add 1 to i
       }
    }
}
\end{trinket}
%slr: end 8-16-19
\index{increment}
\index{decrement}

Assignments like \java{i = i + 1} don't often appear in loops, because Java provides a more concise way to add and subtract by one.
Specifically, \java{++} is the {\bf increment} operator; it has the same effect as \java{i = i + 1}.
And \java{--} is the {\bf decrement} operator; it has the same effect as \java{i = i - 1}.

%So far in this book we have only used (\java{=}) to assign values to variables.
%For convenience, Java provides other assignment operators that increase or decrease the value of a variable.

If you want to increment or decrement a variable by an amount other than \java{1}, you can use \java{+=} and \java{-=}.
For example, \java{i += 2} increments \java{i} by \java{2}.

%slr:  8-16-19
%\begin{code}
%int i = 2;
%while (i <= 8) {
%    System.out.print(i + ", ");
%    i += 2;  // add 2 to i
%}
%System.out.println("Who do we appreciate?");
%\end{code}
%
%And the output is:
%
%\begin{stdout}
%2, 4, 6, 8, Who do we appreciate?
%\end{stdout}
% height = 130 + 11 * num_lines
\begin{trinket} [230] {Increment2.java}
public class Increment2 {

    public static void main(String[] args) {
       int i = 2;
       while (i <= 8) {
          System.out.print(i + ", ");
          i += 2;  // add 2 to i (same as i = i + 2;
       }
       System.out.println("Who do we appreciate?");
    }
}
\end{trinket}

\textbf{Section Exercises}
\begin{enumerate}
\item See if you can predict the output of the program before runnning it.  Were you correct?  Mixing \java{print} and \java{println} statements can be tricky!
\item Change the \java{print} statement on line 6.  Instead, do the equivalent with a \java{printf} statement.
\end{enumerate}
%slr: end 8-16-19

\section{The for statement}

\index{for}
\index{loop!for}
\index{statement!for}

The loops we have written so far have several elements in common:
\begin{enumerate}
\item They start by \textit{initializing} a variable
\item They have a \textit{condition} that depends on that variable
\item Inside the loop they do something to \textit{update} that variable
\end{enumerate}

\index{iteration}

Running the same code multiple times is called {\bf iteration}.
This type of loop is so common that there is another statement, the \java{for} loop, that expresses it more concisely.
For example, we can rewrite the 2-4-6-8 loop this way:

%slr:  8-16-19
%\begin{code}
%for (int i = 2; i <= 8; i += 2) {
%    System.out.print(i + ", ");
%}
%System.out.println("Who do we appreciate?");
%\end{code}
% height = 130 + 11 * num_lines
\begin{trinket} [210] {For.java}
public class For {

    public static void main(String[] args) {

       for (int i = 2; i <= 8; i += 2) {
          System.out.print(i + ", ");
       }
       System.out.println("Who do we appreciate?");
    }
}
\end{trinket}
%slr: end 8-16-19

\java{for} loops have these three components in parentheses, separated by semicolons: the \textbf{initializer}, the \textbf{condition}, and the \textbf{update}.

\begin{enumerate}

\item The {\em initializer} runs once at the very beginning of the loop.
(It is equivalent to the line before the \java{while} statement.)

\item The {\em condition} is checked each time through the loop.
If it is \java{false}, the loop ends.
Otherwise, the body of the loop is executed (again).

\item At the end of each iteration, the {\em update} runs, and we go back to step~2.

\end{enumerate}

The \java{for} loop is often easier to read because it puts all the loop-related statements at the top of the loop.
Doing so allows you to focus on the statements in the loop body.
Figure~\ref{fig.for} illustrates \java{for} loops with a flowchart.

\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.9]{figs/for.pdf}
\caption{Flow of execution for a \java{for} loop.}
\label{fig.for}
\end{center}
\end{figure}

There is another difference between \java{for} loops and \java{while} loops: if you declare a variable in the initializer, it only exists {\em inside} the \java{for} loop.
For example:

%slr:  8-16-19
%\begin{code}
%for (int n = 3; n > 0; n--) {
%    System.out.println(n);
%}
%System.out.println("n is now " + n);  // compiler error
%\end{code}
% height = 130 + 11 * num_lines
\begin{trinket} [200] {ForScope.java}
public class ForScope {

    public static void main(String[] args) {
       for (int n = 3; n > 0; n++) {
          System.out.println(n);
       }
       System.out.println("n is now " + n);
    }
}
\end{trinket}

Line 7 tries to display \java{n} (for no reason other than demonstration) but it won't compile - try it and see.  This is because \java{n} has \textbf{local scope}: it is declared within the \java{for} loop and is not visible outside the for loop.
%slr: end 8-16-19
If you need to use a loop variable outside the loop, you have to declare it {\em outside} the loop, like this:

\begin{code}
int n;
for (n = 3; n > 0; n--) {
    System.out.println(n);
}
System.out.println("n is now " + n);
\end{code}

Notice that the \java{for} statement does not say \java{int n = 3}.
Rather, it simply initializes the existing variable \java{n}.
%slr: 8-16-19

\textbf{Section Exercises}
\begin{enumerate}
\item Fix the syntax error in the program as shown above.  Now \java{n} is visible throughout the \java{main} method.
\end{enumerate}
%slr: end 8-16-19

\section{Nested loops}
\label{nested}

\index{loop!nested}
\index{nested!loops}

Like conditional statements, loops can be nested one inside the other.
Nested loops allow you to iterate over two variables.
For example, we can generate a ``multiplication table'' like this:

%slr:  8-16-19
%\begin{code}
%for (int x = 1; x <= 10; x++) {
%    for (int y = 1; y <= 10; y++) {
%        System.out.printf("%4d", x * y);
%    }
%    System.out.println();
%}
%\end{code}
% height = 130 + 11 * num_lines
\begin{trinket} [230] {MultTable.java}
public class MultTable {
    public static void main(String[] args) {

       for (int x = 1; x <= 10; x++) {
          for (int y = 1; y <= 10; y++) {
             System.out.printf("%4d", x * y);
          }
          System.out.println();
       }
    }
}
\end{trinket}
%slr: end 8-16-19

\index{loop variable}
\index{variable!loop}
\index{inner loop}
\index{outer loop}

Variables like \java{x} and \java{y} are called {\bf loop variables}, because they control the execution of a loop.
In this example, the first loop (\java{for x}) is known as the ``outer loop'', and the second loop (\java{for y}) is known as the ``inner loop''.

Each loop repeats their corresponding statements 10 times.
The outer loop iterates from 1 to 10 only once, but the inner loop iterates from 1 to 10 each of those 10 times.
As a result, the \java{printf} method is invoked 100 times.

\index{format specifier}

The format specifier \java{\%4d} displays the value of \java{x * y} padded with spaces so it's four characters wide.
Doing so causes the output to align vertically, regardless of how many digits the numbers have:

\begin{stdout}
   1   2   3   4   5   6   7   8   9  10
   2   4   6   8  10  12  14  16  18  20
   3   6   9  12  15  18  21  24  27  30
   4   8  12  16  20  24  28  32  36  40
   5  10  15  20  25  30  35  40  45  50
   6  12  18  24  30  36  42  48  54  60
   7  14  21  28  35  42  49  56  63  70
   8  16  24  32  40  48  56  64  72  80
   9  18  27  36  45  54  63  72  81  90
  10  20  30  40  50  60  70  80  90 100
\end{stdout}

It's important to realize that the output is displayed row by row.
The inner loop displays a single row of output, followed by a newline.
The outer loop iterates over the rows themselves.
Another way to read nested loops, like the ones in this example, is ``for each row \java{x}, and for each column \java{y}, \ldots''

\section{Characters}

Some of the most interesting problems in computer science involve searching and manipulating text.
In the next few sections, we'll discuss how to apply loops to strings.
Although the examples are short, the techniques work the same whether you have one word or one million words.

\index{charAt}
\index{char}
\index{type!char}

Strings provide a method named \java{charAt}.
It returns a \java{char}, a java primitive data type that stores an individual character (as opposed to strings of them).

\begin{code}
String fruit = "banana";
char letter = fruit.charAt(0);
\end{code}

The argument \java{0} means that we want the character at {\bf index} 0.
String indexes range from 0 to $n-1$, where $n$ is the length of the string.
So the character assigned to \java{letter} is \java{b}.

\begin{center}
\ttfamily
\begin{tabular}{cccccc}
\hline
\multicolumn{1}{|l|}{b} & \multicolumn{1}{l|}{a} & \multicolumn{1}{l|}{n} & \multicolumn{1}{l|}{a} & \multicolumn{1}{l|}{n} & \multicolumn{1}{l|}{a} \\ \hline
0                       & 1                      & 2                      & 3                      & 4                      & 5
\end{tabular}
\end{center}

Characters work like the other data types we have seen.
You can compare them using relational operators:

\begin{code}
if (letter == 'a') {
    System.out.println('?');
}
\end{code}

\index{quote mark}
\index{escape sequence}

Character literals, like \java{'a'}, appear in single quotes.
Unlike string literals, which appear in double quotes, character literals can only contain a single character.
Escape sequences, like \java{'\t'}, are legal because they represent a single character.

The increment and decrement operators also work with characters.
So this loop displays the letters of the alphabet:

%slr:  8-16-19
%\begin{code}
%System.out.print("Roman alphabet: ");
%for (char c = 'A'; c <= 'Z'; c++) {
%    System.out.print(c);
%}
%System.out.println();
%\end{code}
% height = 130 + 11 * num_lines
\begin{trinket} [195] {Alphabet.java}
public class Alphabet {
    public static void main(String[] args) {

       for (char c = 'A'; c <= 'Z'; c++) {
          System.out.print(c);
       }
       System.out.println();
    }
}
\end{trinket}
\index{Unicode}
\java{char}s are thinly disguised \java{int}s.  The integer values are encoded as characters.
%slr: end 8-16-19
Java uses {\bf Unicode} to represent characters, so strings can store text in other alphabets like Cyrillic and Greek, and non-alphabetic languages like Chinese.
You can read more about it at \url{http://unicode.org/}.

In Unicode, each character is represented by a ``code point'', which you can think of as an integer.
The code points for uppercase Greek letters run from 913 to 937, so we can display the Greek alphabet like this:

%slr:  8-16-19
%\begin{code}
%System.out.print("Greek alphabet: ");
%for (int i = 913; i <= 937; i++) {
%    System.out.print((char) i);
%}
%System.out.println();
%\end{code}
% height = 130 + 11 * num_lines
\begin{trinket} [210] {GreekAlphabet.java}
public class GreekAlphabet {
    public static void main(String[] args) {

       System.out.println("Greek alphabet: ");
       for (int i = 913; i <= 937; i++) {
          System.out.print((char) i);
       }
       System.out.println();
    }
}
\end{trinket}
%slr:  end 8-16-19

This example uses a type cast to convert each integer (in the range) to the corresponding character.
Try running the code and see what happens.
%slr:  8-16-19

\textbf{Section Exercises}
\begin{enumerate}
\item Try printing out (a portion of) the Thai alphabet:  code point 3585 through 3642
\item Read up on Unicode blocks at \url{https://en.wikipedia.org/wiki/Unicode_block}.  See if you can print other alphabets.  (You'll have to convert from Hex to decimal.)
\end{enumerate}
%slr: end 8-16-19

\section{String iteration}

\index{iteration}

The following loop iterates the characters in \java{fruit} and displays them, one on each line:
%slr:  8-16-19

%\begin{code}
%for (int i = 0; i < fruit.length(); i++) {
%    char letter = fruit.charAt(i);
%    System.out.println(letter);
%}
%\end{code}
% height = 130 + 11 * num_lines
\begin{trinket} [210] {StringIter.java}
public class StringIter {
    public static void main(String[] args) {

       String fruit = "banana";
       for (int i = 0; i < fruit.length(); i++) {
          char letter = fruit.charAt(i);
          System.out.println(letter);
       }
    }
}
\end{trinket}
%slr: end 8-16-19
\index{string!length}
\index{length!string}

Strings provide a method called \java{length} that returns the number of characters in the string.
Because it is a method, you have to invoke it with the empty argument list, \java{()}.
When \java{i} is equal to the length of the string, the condition becomes \java{false} and the loop terminates.

To find the last letter of a string, you might be tempted to do something like:

\begin{code}
int length = fruit.length();
char last = fruit.charAt(length);      // wrong!
\end{code}

\index{StringIndexOutOfBoundsException}
\index{exception!StringIndexOutOfBounds}

This code compiles and runs, but invoking the \java{charAt} method throws a \java{StringIndexOutOfBoundsException}.
The problem is that there is no sixth letter in \java{"banana"}.
Since we started counting at 0, the 6 letters are indexed from 0 to 5.
To get the last character, you have to subtract 1 from \java{length}.

\begin{code}
int length = fruit.length();
char last = fruit.charAt(length - 1);  // correct
\end{code}

Many string algorithms involve reading one string and building another.
For example, to reverse a string, we can add one character at a time:

%slr:  8-16-19
%\begin{code}
%public static String reverse(String s) {
%    String r = "";
%    for (int i = s.length() - 1; i >= 0; i--) {
%        r += s.charAt(i);
%    }
%    return r;
%}
%\end{code}
% height = 130 + 11 * num_lines
\begin{trinket} [290] {StringReverse.java}
public class StringReverse {

    public static String reverse(String s) {
       String r = ""; //empty String
       for (int i = s.length() - 1; i >= 0; i--) {
          r += s.charAt(i);  //means r = r + s.charAt(i);  + is concatenation for Strings
       }
       return r;
    }

    public static void main(String[] args) {
       String fruit = "banana";
       String yiurf = reverse(fruit);
       System.out.println(yiurf);
    }
}
\end{trinket}

\index{empty string}

The initial value of \java{r} is \java{""}, which is the {\bf empty string}.
The loop iterates the letters of \java{s} in reverse order.
Each time through the loop, it creates a new string and assigns it to \java{r}.
When the loop exits, \java{r} contains the letters from \java{s} in reverse order.
So the result of \java{reverse("banana")} is \java{"ananab"}.

\section{The indexOf method}

\index{indexOf}

To search for a specific character in a string, you could write a \java{for} loop and use \java{charAt} like in the previous section.
However, the \java{String} class already provides a method for doing just that.

\begin{code}
String fruit = "banana";
int index = fruit.indexOf('a');     // returns 1
\end{code}

This example finds the index of \java{'a'} in the string.
But the letter appears three times, so it's not obvious what \java{indexOf} should do.
According to the documentation, it returns the index of the {\em first} appearance.

To find subsequent appearances, you can use another version of \java{indexOf}, which takes a second argument that indicates where in the string to start looking.

\begin{code}
int index = fruit.indexOf('a', 2);  // returns 3
\end{code}

To visualize how \java{indexOf} and other \java{String} methods work, it helps to draw a picture like Figure~\ref{fig.banana}.
The previous code starts at index 2 (the first \java{'n'}) and finds the next \java{'a'}, which is at index 3.

\index{memory diagram}
\index{diagram!memory}

\begin{figure}[!ht]
\begin{center}
\includegraphics{figs/banana.pdf}
\caption{Memory diagram for a \java{String} of six characters.}
\label{fig.banana}
\end{center}
\end{figure}

%\begin{center}
%\begin{tabular}{c|c|c|c|c|c}
%%\hline
%b & a & n & a & n & a \\
%\hline
%0 & 1 & 2 & 3 & 4 & 5 \\
%%\hline
%\end{tabular}
%\end{center}

If the character happens to appear at the starting index, the starting index is the answer.
So \java{fruit.indexOf('a', 5)} returns \java{5}.
If the character does not appear in the string, \java{indexOf} returns \java{-1}.
Since indexes cannot be negative, this value indicates the character was not found.

You can also use \java{indexOf} to search for an entire string, not just a single character.
For example, the expression \java{fruit.indexOf("nan")} returns \java{2}.

\section{String comparison}
\label{strcmp}

\index{equals}
\index{string!comparing}

To compare two strings, it may be tempting to use the \java{==} and \java{!=} operators.

\begin{code}
String name1 = "Alan Turing";
String name2 = "Ada Lovelace";
if (name1 == name2) {                 // wrong!
    System.out.println("The names are the same.");
}
\end{code}

This code compiles and runs, and sometimes it gets the answer right.
But sometimes it gets the answer wrong.
If you give it two different strings that contain the same letters, the condition will be \java{false}.

%slr 8-16-19
Here is an example of that situation.  When prompted, enter the string "foo" and see what happens:

% height = 130 + 11 * num_lines
\begin{trinket} [220] {StringEquals.java}
import java.util.Scanner;
public class StringEquals {
   public static void main(String[] args) {

      Scanner kbd = new Scanner(System.in);
      String s = "foo";
      System.out.print("Enter a string: ");
      String u = kbd.next();
      System.out.println(s == u); //don't do this!
   }
}
\end{trinket}

The \java{String}s have the same content but are not the \textit{same object}.
%slr:  end 8-16-19
The problem is that the \java{==} operator checks whether the two variables refer to the {\em same object} by comparing the references.
We'll learn more about references in the next chapter.
The correct way to compare strings is with the \java{equals} method, like this:

%slr:  8-16-19
% height = 130 + 11 * num_lines
\begin{trinket} [220] {StringEquals.java}
import java.util.Scanner;
public class StringEquals {
    public static void main(String[] args) {

       Scanner kbd = new Scanner(System.in);
       String s = "foo";
       System.out.print("Enter a string: ");
       String u = kbd.next();
       System.out.println(s.equals(u)); //correct
    }
}
\end{trinket}

Now run it and enter "foo" when prompted. The \java{String}s have the same content and the \java{equals} method returns \java{true}.
%\begin{code}
%if (name1.equals(name2)) {
%    System.out.println("The names are the same.");
%}
%\end{code}

%This example invokes \java{equals} on \java{name1} and passes \java{name2} as an argument.
%The \java{equals} method returns \java{true} if the strings contain the same characters; otherwise it returns \java{false}.
%slr: end 8-16-19

\index{compareTo}

If the strings differ, we can use \java{compareTo} to see which comes first in alphabetical order:

%slr:  8-16-19
%\begin{code}
%int diff = name1.compareTo(name2);
%if (diff == 0) {
%    System.out.println("The names are the same.");
%} else if (diff < 0) {
%    System.out.println("name1 comes before name2.");
%} else if (diff > 0) {
%    System.out.println("name2 comes before name1.");
%}
%\end{code}
% height = 130 + 11 * num_lines
\begin{trinket} [260] {StringCompareTo.java}
public class StringCompareTo {
    public static void main(String[] args) {
       String name1 = "Alan Turing";
       String name2 = "Ada Lovelace";
       int diff = name1.compareTo(name2);
       if (diff == 0) {
          System.out.println("The names are the same.");
       } else if (diff < 0) {
          System.out.println("name1 comes before name2.");
       } else if (diff > 0) {
          System.out.println("name2 comes before name1.");
       }
    }
}
\end{trinket}
%slr: end 8-16-19

The return value from \java{compareTo} is the difference between the first characters in the strings that are not the same.
In the preceding code, \java{compareTo} returns positive 8, because the second letter of \java{"Ada"} comes before the second letter of \java{"Alan"} by 8 letters.

If the strings are equal, their difference is zero.
If the first string (the one on which the method is invoked) comes first in the alphabet, the difference is negative.
Otherwise, the difference is positive.

\index{case-sensitive}

Both \java{equals} and \java{compareTo} are case-sensitive.
In Unicode, uppercase letters come before lowercase letters.
So \java{"Ada"} comes before \java{"ada"}.
%slr:  8-16-19

\textbf{Section Exercises}
\begin{enumerate}
\item Change the \java{StringCompareTo} program above so as to print out the value of \java{diff}.  Do you get the value 8?
\item Change name1 and name2.  Predict what the value of \java{diff} will be.  Run the program to check.
\end{enumerate}
%slr: end 8-16-19

\section{Substrings}

\index{substring}

The \java{substring} method returns a new string that copies letters from an existing string, starting at the given index.

\begin{itemize}
\item \java{fruit.substring(0)} returns \java{"banana"}
\item \java{fruit.substring(2)} returns \java{"nana"}
\item \java{fruit.substring(6)} returns \java{""}
\end{itemize}

The first example returns a copy of the entire string.
The second example returns all but the first two characters.
As the last example shows, \java{substring} returns the empty string if the argument is the length of the string.

Like most string methods, \java{substring} is \textbf{overloaded}.
That is, there are other versions of \java{substring} that have different parameters.
If it's invoked with two arguments, they are treated as a start and end index:

\begin{itemize}
\item \java{fruit.substring(0, 3)} returns \java{"ban"}
\item \java{fruit.substring(2, 5)} returns \java{"nan"}
\item \java{fruit.substring(6, 6)} returns \java{""}
\end{itemize}

Notice that the character indicated by the end index is {\em not} included.
Defining \java{substring} this way simplifies some common operations.
For example, to select a substring with length \java{len}, starting at index \java{i}, you could write \java{fruit.substring(i, i + len)}.
%So \java{fruit.substring(2, 2 + 3)} returns \java{"nan"}.
%slr:  8-16-19

\textbf{Section Exercises}

Use the following to complete the exercises:

% height = 130 + 11 * num_lines
\begin{trinket} [180] {Substring.java}
public class Substring {
    public static void main(String[] args) {
       String name = "Ada Lovelace";
       System.out.println(name.substring(4));
       System.out.println(name.substring(4, 7));
    }
}
\end{trinket}

\begin{enumerate}
\item Predict the output of the program prior to running it.  Were you correct?
\item Play with the parameter passed to \java{substring} in line 4.  Make certain you understand how this version of \java{substring} sorks!
\item Play with the parameter\textbf{s} passed to \java{substring} in line 5.  Make certain you understand how this version of \java{substring} sorks!
\end{enumerate}
%slr: end 8-16-19

\section{String formatting}

\index{printf}

In Section~\ref{printf}, we learned how to use \java{System.out.printf} to display formatted output.
Sometimes programs need to create strings that are formatted a certain way, but not display them immediately, or ever.
For example, the following program uses a method that returns a time string in 12-hour format:

%slr:  8-16-19
%\begin{code}
%public static String timeString(int hour, int minute) {
%    String ampm;
%    if (hour < 12) {
%        ampm = "AM";
%        if (hour == 0) {
%            hour = 12;  // midnight
%        }
%    } else {
%        ampm = "PM";
%        hour = hour - 12;
%    }
%    return String.format("%02d:%02d %s", hour, minute, ampm);
%}
%\end{code}
% height = 130 + 11 * num_lines
\begin{trinket} [350] {Formatting.java}
public class Formatting {

    public static String timeString(int hour, int minute) {
       String ampm;
       if (hour < 12) {
           ampm = "AM";
           if (hour == 0) {
               hour = 12;  // midnight
           }
       } else {
           ampm = "PM";
           hour = hour - 12;
       }
       return String.format("%02d:%02d %s", hour, minute, ampm);
    }

    public static void main(String[] args) {
       String formattedTime = timeString(13, 46);
       System.out.println(formattedTime);
    }
}
\end{trinket}

\index{string!format}

\java{String.format}, a static method of the \java{String} class, takes the same arguments as \java{System.out.printf}: a format specifier followed by a sequence of values.
The main difference is that \java{System.out.printf} displays the result on the screen.
\java{String.format} creates a new string, but does not display anything.

In this example, the format specifier \java{\%02d} means ``two digit integer padded with zeros'', so \java{timeString(19, 5)} returns the string \java{"07:05 PM"}.
As an exercise, try writing two nested \java{for} loops (in \java{main}) that invoke \java{timeString} and display all possible times over a 24-hour period.

%slr: 8-16-19
At some point today, skim through the documentation for \java{String} found in the \href{https://docs.oracle.com/javase/8/docs/api/}{Java API}.
Knowing what other methods are there will help you avoid reinventing the wheel.
%The easiest way to find documentation for Java classes is to do a web search for ``Java'' and the name of the class.
%slr:  end 8-16-19

\section{Vocabulary}

\begin{description}

\term{loop}
A statement that executes a sequence of statements repeatedly.

\term{loop body}
The statements inside the loop.

\term{infinite loop}
A loop whose condition is always true.

\term{increment}
Increase the value of a variable.

\term{decrement}
Decrease the value of a variable.

\term{iteration}
Executing a sequence of statements repeatedly.

\term{loop variable}
A variable that is initialized, tested, and updated in order to control a loop.

\term{index}
An integer variable or value used to indicate a character in a string.

\term{Unicode}
An international standard for representing characters in most of the world's languages.

\term{empty string}
The string \java{""}, which contains no characters and has a length of zero.

\end{description}

\section{Exercises}

The code for this chapter is in the {\tt ch06} directory of {\tt ThinkJavaCode2}.
See page~\pageref{code} for instructions on how to download the repository.
Before you start the exercises, we recommend that you compile and run the examples.

If you have not already read Appendix~\ref{debugger}, now might be a good time.
It describes the DrJava debugger, which is a useful tool for visualizing the flow of execution through loops.

\begin{exercise}  %%V6 Ex7.1

Consider the following methods:

\begin{code}
public static void main(String[] args) {
    loop(10);
}

public static void loop(int n) {
    int i = n;
    while (i > 1) {
        System.out.println(i);
        if (i % 2 == 0) {
            i = i / 2;
        } else {
            i = i + 1;
        }
    }
}
\end{code}

\begin{enumerate}

\item Draw a table that shows the value of the variables \java{i} and \java{n} during the execution of \java{loop}.
The table should contain one column for each variable and one line for each iteration.

\item What is the output of this program?

\item Can you prove that this loop terminates for any positive value of \java{n}?

% If i is odd and we increment by 1, the result is even.  So the second
% branch is always followed by the first branch.
% If i is even and we divide by 2, the result might be odd.  So in the
% worst case, we might alternate between the branches.
% But we can't do more of the second branch than the first.
% So we divide at least as often as we add.

% If i is 1, we're done.
% If i is 2, we divide by 2 and we're done.
% If i is greater than 2, the first branch decreases more than the
% second branch increases.
% So if we do one of each, the net effect is a decrease.
% Therefore, the value of i has to decrease after any two steps.

\end{enumerate}

\end{exercise}

\begin{exercise}  %%V6 Ex7.2

Let's say you are given a number, $a$, and you want to find its square root.
One way to do that is to start with a rough guess about the answer, $x_0$, and then improve the guess using this formula:
%
\[ x_1 =(x_0 + a/x_0) / 2 \]
%
For example, if we want to find the square root of 9, and we start with $x_0 = 6$, then $x_1 = (6 + 9/6) / 2 = 3.75$, which is closer.
We can repeat the procedure, using $x_1$ to calculate $x_2$, and so on.
In this case, $x_2 = 3.075$ and $x_3 = 3.00091$.
So it converges quickly on the correct answer.

Write a method called \java{squareRoot} that takes a \java{double} and returns an approximation of the square root of the parameter, using this technique.
You should not use \java{Math.sqrt}.

As your initial guess, you should use $a/2$.
Your method should iterate until it gets two consecutive estimates that differ by less than 0.0001.
%In other words, return when the absolute value of $x_n - x_{n-1}$ is less than 0.0001.
You can use \java{Math.abs} to calculate the absolute value of the difference.

\end{exercise}

\begin{exercise}  %%V6 Ex7.6

One way to evaluate $\exp(-x^2)$ is to use the infinite series expansion:
%
\[ \exp(-x^2) = 1 - x^2 + x^4/2 - x^6/6 + \ldots \]
%
The $i$th term in this series is $(-1)^i x^{2i} / i!$.
Write a method named \java{gauss} that takes \java{x} and \java{n} as arguments and returns the sum of the first \java{n} terms of the series.
You should not use \java{factorial} or \java{pow}.

\end{exercise}

\begin{exercise}  %%V6 Ex9.5

\index{abecedarian}

A word is said to be ``abecedarian'' if the letters in the word appear in alphabetical order.
For example, the following are all six-letter English abecedarian words:

\begin{quote}
abdest, acknow, acorsy, adempt, adipsy, agnosy, befist, behint, %\\
beknow, bijoux, biopsy, cestuy, chintz, deflux, dehors, dehort, %\\
deinos, diluvy, dimpsy %\\
\end{quote}

Write a method called \java{isAbecedarian} that takes a \java{String} and returns a \java{boolean} indicating whether the word is abecedarian.
%Your method can be iterative or recursive.

\end{exercise}

\begin{exercise}  %%V6 Ex9.6
\label{doubloon}

\index{doubloon}

A word is said to be a ``doubloon'' if every letter that appears in the word appears exactly twice.
Here are some example doubloons found in the dictionary:

\begin{quote}
Abba, Anna, appall, appearer, appeases, arraigning, beriberi, bilabial, boob, Caucasus, coco, Dada, deed, Emmett, Hannah, horseshoer, intestines, Isis, mama, Mimi, murmur, noon, Otto, papa, peep, reappear, redder, sees, Shanghaiings, Toto
\end{quote}

Write a method called \java{isDoubloon} that takes a string and checks whether it is a doubloon.
To ignore case, invoke the \java{toLowerCase} method before checking.
\end{exercise}

\begin{exercise}  %%V6 Ex9.8

\index{Scrabble}

In Scrabble\footnote{Scrabble is a registered trademark owned in the USA and Canada by Hasbro Inc., and in the rest of the world by J.\ W.\ Spear \& Sons Limited of Maidenhead, Berkshire, England, a subsidiary of Mattel Inc.} each player has a set of tiles with letters on them.
The object of the game is to use those letters to spell words.
The scoring system is complex, but longer words are usually worth more than shorter words.

Imagine you are given your set of tiles as a string, like \java{"quijibo"}, and you are given another string to test, like \java{"jib"}.

Write a method called \java{canSpell} that takes two strings and checks whether the set of tiles can spell the word.
You might have more than one tile with the same letter, but you can only use each tile once.

\end{exercise}


\hypertarget{implicitmesh}{%
\section{ImplicitMesh}\label{implicitmesh}}

The \texttt{implicitmesh} module allows you to build meshes from
implicit functions. For example, the unit sphere could be specified
using the function \texttt{x\^{}2+y\^{}2+z\^{}2-1\ ==\ 0}.

To use the module, first import it:

\begin{lstlisting}
import implicitmesh
\end{lstlisting}

To create a sphere, first create an ImplicitMeshBuilder object with the
implict function you'd like to use:

\begin{lstlisting}
var impl = ImplicitMeshBuilder(fn (x,y,z) x^2+y^2+z^2-1)
\end{lstlisting}

You can use an existing function (or method) as well as an anonymous
function as above.

Then build the mesh,

\begin{lstlisting}
var mesh = impl.build(stepsize=0.25)
\end{lstlisting}

The \texttt{build} method takes a number of optional arguments:

\begin{itemize}

\item
  \texttt{start} - the starting point. If not provided, the value
  Matrix({[}1,1,1{]}) is used.
\item
  \texttt{stepsize} - approximate lengthscale to use.
\item
  \texttt{maxiterations} - maximum number of iterations to use. If this
  limit is exceeded, a partially built mesh will be returned.
\end{itemize}


\section{Correlation Estimation}

\begin{enumerate}[label=\alph*), leftmargin=*]
%% a)
\item
%

Figure \ref{fig:2_1_a} shows the biased (red) and unbiased (blue) estimates of the autocorrelation function (ACF) as well as the correlogram spectral estimates obtained for:
white gaussian noise (WGN), a noisy sinusoidal signal ($\sigma^{2} = 1$) and filtered white gaussian noise.

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/parametric-and-line-spectra/correlation-estimation/assets/a/acf-WGN}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/parametric-and-line-spectra/correlation-estimation/assets/a/psd-WGN}
    \end{subfigure}
    ~
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/parametric-and-line-spectra/correlation-estimation/assets/a/acf-Filtered_WGN}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/parametric-and-line-spectra/correlation-estimation/assets/a/psd-Filtered_WGN}
    \end{subfigure}
    ~
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/parametric-and-line-spectra/correlation-estimation/assets/a/acf-Noisy_Sinewave}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/parametric-and-line-spectra/correlation-estimation/assets/a/psd-Noisy_Sinewave}
    \end{subfigure}
    \caption{ACF and Correlogram: biased and unbiased estimates of various signals.}
    \label{fig:2_1_a}
\end{figure}

Regarding the ACF estimates, we verify that for small lags, $k \lessapprox 150$, the two estimates are very similar, but for larger lags
the unbiased estimates increase in value while the biased estimates fade out to zero. We now show that the biased and unbiased ACF
estimates are obtained by windowing the ideal ACF $r_{xx}(k)$ with the Bartlett window, $w_{B}(k)$, and the rectangular window, $w_{R}(k)$, respectively.

\begin{align}
    \E[ \hat{r}_{biased}(k) ] &= \sum_{n=k+1}^{N} \frac{N - k}{N} r_{xx}(k) = w_{B}(k) r_{xx}(k) \\
    \E[ \hat{r}_{unbiased}(k) ] &= \sum_{n=k+1}^{N} \frac{N - k}{N - K} r_{xx}(k) = w_{R}(k) r_{xx}(k)
\end{align}

Thus, using the Fourier transform pair ACF-PSD, we obtain the expected values of the correlograms $\E[\hat{P}_{biased}]$ and $\E[\hat{P}_{unbiased}]$ as the convolution of
of the true power spectrum with the Fourier transform of the Bartlett and rectangular window, respectively. The Fourier transform of the rectangular window
is the $sinc$ function, which introduces \textbf{negative} values, and does not preserve the positive semi-definiteness of the PSD. On the other hand, the
Fourier transform of the Bartlett window is strictly non-negative\footnote{see figure \ref{fig:1_3_a_2} from Assignment 1}, guaranteeing positive semi-definiteness.
The correlograms in figure \ref{fig:2_1_a} verify our theoretical argument, where the unbiased estimates lead to negative PSD values while this is not the case for
the biased estimates.

%% b)
\item
%

Figure \ref{fig:2_1_b} illustrates the periodogram of 100 realisations of the random process $x(n)$,
as well as the ensemble mean and standard deviation, where:

\begin{equation}
    x(n) = 1.5 sin(2 \pi 0.3 n) + sin(2 \pi 0.6 n) + 2 sin(2 \pi 0.9 n) + w(n) \quad w \sim \mathcal{N}(0, 1)
\end{equation}

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/parametric-and-line-spectra/correlation-estimation/assets/b/psd_mean}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/parametric-and-line-spectra/correlation-estimation/assets/b/psd_std}
    \end{subfigure}
    \caption{Correlogram: mean and standard deviation of $x(n)$.}
    \label{fig:2_1_b}
\end{figure}

When the biased ACF estimate is used, the obtained correlogram is an \textbf{inconsistent estimator} since:

\begin{equation}
    \Var[\hat{P}_{biased}(f)] = P_{xx}^{2}(f) \bigg[ 1 + \bigg(\frac{sin(2 \pi N f)}{N sin(2 \pi f)}\bigg)^{2} \bigg]
\end{equation}

and as a result when $N \rightarrow \infty$, $\Var[\hat{P}_{biased}(f)] \rightarrow P_{xx}^{2}(f) \gg 0$. This also explains the increased standard deviation of the periodogram
at the peak frequencies of the ideal PSD, $P_{xx}(f)$, of the process.

%% c)
\item
%

The experiments are repeated and the periodograms are expressed in $dB$ this time, as illustrated in figure \ref{fig:2_1_c}.
Interestingly, the variance close to the peak frequencies, $f = 0.3,\ 0.6,\ 0.9\ Hz$, is decreased.
This is a counter-intuitive result, but it can be explained by the logarithmic function's gradient ($\frac{1}{x}$).
Fluctuations around zero are significantly amplified, while values greater than $1$ are attenuated (sub-linear trend). Consequently, since the power at peak frequencies is much greater
than $1$, its variance is squeezed, while at all other frequencies power is concentrated around zero, leading to amplified variance.

Clearly, this is an advantageous representation, since at frequencies of interest the spread-out is reduced and the interpretation of the periodograms is easier.

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/parametric-and-line-spectra/correlation-estimation/assets/c/psd_mean}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/parametric-and-line-spectra/correlation-estimation/assets/c/psd_std}
    \end{subfigure}
    \caption{Correlogram: mean and standard deviation of $x(n)$ in $dB$.}
    \label{fig:2_1_c}
\end{figure}

%% d)
\item
%

Figure \ref{fig:2_1_d} shows the periodograms of complex exponential signals (two sine waves) in noise for different number of samples and therefore frequency resolutions.
As expected, for small number of samples ($n \leq 40$) the two peaks cannot be distinguished but for larger values ($n \geq 45$) the two peaks are visible.
This observation agrees with theory, since the resolution of the (standard) periodogram is given by $\Delta f = \frac{0.89}{N}$ and the two complex signals have
frequencies $f_{1} = 0.3\ Hz$ and $f_{2} = 0.32\ Hz$, hence discrimination is possible when:

\begin{equation}
    \Delta f \leq f_{2} - f_{1} \Longrightarrow N \geq \frac{0.89}{f_{2} - f_{1}} = \frac{0.89}{0.32 - 0.3} = 44.5
\end{equation}

\begin{figure}[h]
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/parametric-and-line-spectra/correlation-estimation/assets/d/complex_exponentials-1}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/parametric-and-line-spectra/correlation-estimation/assets/d/complex_exponentials-2}
    \end{subfigure}
    \caption{Periodogram: frequency resolution and peak identification.}
    \label{fig:2_1_d}
\end{figure}

Hence, despite the simplicity of the signals under investigation (two sine waves), the periodogram fails to adequately estimate their power spectral density when a small number
of samples is available. Therefore, alternative methods (i.e subspace methods) must be used in these cases.

%% e)
\item
%

The MUltiple SIgnal Classification algorithm (MUSIC) estimates the spectral density of a signal, using a \textbf{subspace} method. It assumes that the signal, $x(n)$,
consists of $p$ complex exponentials in the presence of Gaussian white noise. Given an autocorrelation matrix, $\mathbf{R}_{xx} \in \sR^{M \times M}$,
if its eigenvalues are sorted in decreasing order, the eigenvectors corresponding to the $p$ largest eigenvalues (i.e. directions of largest variability) span the signal subspace, $\mathbf{R}_{s}$.
The remaining $M-p$ eigenvectors span the orthogonal space, $\mathbf{R}_{n}$, where there is only noise.
Let the noise eigenvectors $\vv_{i}$ and the helper vector $\ve$ such that:

\begin{equation}
    \vv_{i}, \quad i = p+1, \ldots, M \qquad \text{and} \qquad
    \ve =
    \begin{bmatrix}
        1 & e^{jw} & e^{j2w} & \cdots &  e^{j(M-1)w}
    \end{bmatrix}^{T}
\end{equation}

then the MUSIC spectral estimate $\hat{P}_{MU}$ is given by:

\begin{equation}
    \hat{P}_{MU}(e^{jw}) = \frac{1}{\sum_{i=p+1}^{M} |\ve^{H} \vv_{i}|^{2}}
\end{equation}

where at signal frequencies $w_{1},\ \ldots,\ w_{k},\ \ldots,\ w_{p}$ the noise eigenvectors $\vv_{i}$ and the signal eigenvectors $\ve_{k}$ will be orthogonal (since $\mathbf{R}_{xx}$ is Hermitian)
and therefore $\hat{P}_{MU}$ will have $p$ peaks, as expected for a spectrum estimator of $p$ complex exponentials.

The modified autocorrelation matrix $\mathbf{R}_{xx}$ with $M = 14$ is obtained using MATLAB command:

\begin{equation*}
    \mathtt{[X,Rxx] = corrmtx(x, 14, "mod")}
\end{equation*}

which is used by the MUSIC algorithm for spectral estimation

\begin{equation*}
    \mathtt{[S,F] = pmusic(Rxx, p, [ ], 1, "corr")}
\end{equation*}

where \texttt{p} is the signal subspace dimensionality, a hyperparameter that must be tuned/selected.

In figure \ref{fig:2_1_e}, the 100 realisations overlay of a two complex exponentials signal ($f_{1} = 0.3\ Hz$ and $f_{2} = 0.32\ Hz$) MUSIC spectrum estimate is provided,
alogn with their standard deviation, for different values of hyperparameter $p$.

\begin{figure}[h]
    \centering
    \centering
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/parametric-and-line-spectra/correlation-estimation/assets/e/MUSIC_mean-p_1}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/parametric-and-line-spectra/correlation-estimation/assets/e/MUSIC_std-p_1}
    \end{subfigure}
    ~
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/parametric-and-line-spectra/correlation-estimation/assets/e/MUSIC_mean-p_2}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/parametric-and-line-spectra/correlation-estimation/assets/e/MUSIC_std-p_2}
    \end{subfigure}
    ~
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/parametric-and-line-spectra/correlation-estimation/assets/e/MUSIC_mean-p_3}
    \end{subfigure}
    ~
    \begin{subfigure}{0.49\textwidth}
        \centering
        \includegraphics[height=1.5in]{report/parametric-and-line-spectra/correlation-estimation/assets/e/MUSIC_std-p_3}
    \end{subfigure}
    \caption{MUSIC: spectral estimation of two complex exponentials and signal space size $p$.}
    \label{fig:2_1_e}
\end{figure}

Despite the small number of samples available ($n = 30$) and thus the poor frequency resolution, the algorithm successfully identifies the two
frequency components at $f_{1}$ and $f_{2}$, when $p=2$ is selected, unlike the periodogram that requires a larger amount of samples.
Similar to the periodogram, the standard deviation of the estimator increases closer to the peak values, but most importantly, the reliability
of the estimator deteriorates significantly when $p$ is not matched with the nature of signal $x(n)$.

To sum up, MUSIC method is very powerful when a small number of samples is available but its chief disadvantage is that it requires the number of components $p$ to be known in advance,
so \textbf{it cannot be used in more general cases}, when prior knowledge of the signal is not provided. On the other hand, the periodogram needs
a finer frequency resolution to detect closely-spaced sine waves (larger $n$), but it does not require any knowledge about the signal $x(n)$.

%
\end{enumerate}


\section{Updates: 04/28/2020}%
\label{sec:updates_2020_04_28}
%
\subsection{Additional changes to loss function}%
\label{subsec:additional_changes_to_loss_fn}
%
Instead of working with the mixed loss function,
\(\ell_{\lambda_{\mathcal{Q}}}(\xip, \xi, A(\xip|\xi))\) from
Eq.\ref{eq:ell_lambda}, we can focus exclusively on the second term, which is
directly related to the tunneling rate.
%
The equation then becomes
%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% TODO: Include both ell_p and ell_q functions with separate weights in
% `master` loss function
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{align}
  \ell_{\lambda_{\Q}}\left(\xip, \xi, A(\xip|\xi)\right)
  &= - \frac{\delta_{\Q}\cdot A(\xi^{\prime}|\xi)}{\lambda_{\Q}^{2}}\\
  &= - {\left(\frac{{\mathcal{Q}^{\prime} -
  \mathcal{Q}}}{\lambda_\mathcal{Q}}\right)}^{2}\cdot
    A(\xi^{\prime}|\xi)
\end{align}


\section{Functions} % (fold)
\label{sec:functions}

% This sheet is a little long

\begin{questions}

\titledquestion{Hello} % (fold)
\label{sub:hello_world}

\begin{parts}
    \part Write a function \verb\hello_world\ that prints \verb\'Hello, world!'\
    \part Write a function \verb\hello_name(name)\ that prints \verb\'Hello, name!'\
    where \texttt{name} is a string.
    \part Explain the difference between the \texttt{print} and \texttt{return}
    keywords.
    What would change if instead of \texttt{print} you would use \texttt{return}?
\end{parts}

% titledquestion hello_world (end)

\titledquestion{Polynomial} % (fold)
\label{sub:polynomial}

Write a function that evaluates the polynomial $3x^2 - x + 2$.

% titledquestion polynomial (end)

\titledquestion{Maximum} % (fold)
\label{sub:maximum}

Write a function \verb\my_max(x,y)\ that returns the maximum of $x$ and $y$.
Do not use the \texttt{max} function, but use \texttt{if} instead in following two ways:
\begin{parts}
    \part Use both \texttt{if} and \texttt{else}.
    \part Use \texttt{if} but not \texttt{else} (nor \texttt{elif}).
\end{parts}

% titledquestion maximum (end)

\titledquestion{Primes} % (fold)
\label{sub:primes}

\begin{parts}
    \part Write a function \verb\is_prime(n)\ that returns \texttt{True} only if $n$ is prime.
    \part Note that apart from 2 and 3, all primes are of the form $6k \pm 1$
        (though not all numbers of the form $6k \pm 1$ are prime of course).
        Using this, we can improve the computation time by a factor $3$.
        Update your function to use this.
    \part Write a function that returns all primes up to $n$.
    \part Write a function that returns the first $n$ primes.
\end{parts}

% titledquestion primes (end)

\titledquestion{Root finding} % (fold)
\label{sub:root_finding}

Suppose $f$ is a continuous function and $f(a) < 0$ and $f(b) > 0$ for some known $a$ and $b$.
For simplicity, assume $a < b$.
Then, there must exist some $c$ such that $f(c) = 0$.

\begin{parts}
    \part Write a function \verb|root(f, a, b)| that takes a function \verb|f| and two floats
\verb|a| and \verb|b| and returns the root \verb|c|. Hint: check the sign at the midpoint of the interval.
    \part Remove the assumption that $a < b$, and that $f(a) < 0$ and $f(b) > 0$, if
    your current code relies on them.
    \part Add a check that prints\\ \verb|'function evals have same sign'|\\ if $f(a) >0$
    and $f(b) >0$ or if $f(a) <0 $ and $f(b) < 0$.
\end{parts}

% titledquestion root_finding (end)

\end{questions}

% section functions (end)


%===============================================================================
% COVID-19 Reproduction Numbers in South Africa
% Kevin Durant
% May 2020
%===============================================================================

\documentclass[12pt,a4paper]{article}

% \usepackage{booktabs} % better looking tables.
\usepackage{mathtools} % also loads amsmath.
\usepackage{microtype}
\usepackage{iftex}
\ifPDFTeX
  \usepackage[T1]{fontenc}
  \usepackage[utf8]{inputenc}
  \usepackage{amssymb}
\else
  % Unicode-math should be loaded after other maths- or font-related packages.
  % It loads amsmath and fontspec if necessary, and enables a custom version of
  % the Latin Modern Math font by default.
  \usepackage{unicode-math}
%   \setmainfont{STIX Two Text}
%   \setmathfont{STIX Two Math}
\fi
\usepackage{biblatex}
\usepackage{svg}
\usepackage{hyperref}
\addbibresource{cvza.bib}

% General maths commands %======================================================

\DeclarePairedDelimiter\lr{\lparen}{\rparen}  % sized parentheses.
\DeclareMathOperator\Pb{P}                    % probability.
\DeclareMathOperator\B{B}                     % beta distribution.
\DeclareMathOperator\BP{BP}                   % beta prime distribution.
\DeclareMathOperator\NB{NB}                   % negative binomial distribution.

% Title %=======================================================================

\title{Model Summary}
\author{Kevin Durant}
\date{}

% Document %====================================================================

\begin{document}

\maketitle

\section{Definitions} %=========================================================

$k_t$: number of new infections on day $t$. \\
$\lambda_t$: underlying daily rate of infection (and expected number of new
infections) on day $t$.

\section{Assumptions} %=========================================================

Firstly, that $k_t$ depends on $\lambda_t$ via a negative binomial distribution
with dispersion parameter $r$:
\begin{equation*}
  \Pb(k_t \mid r, \lambda_t)
  = \NB\lr*{k_t \Bigm\vert r, \frac{\lambda}{r + \lambda} = p_t}.
\end{equation*}

Secondly, that the posterior on $p_t$ is a beta distribution (which is the
conjugate prior for the negative binomial with known dispersion):
\begin{align*}
  \Pb(p_t \mid k_1, \dots, k_t) &= \B(p_t \mid \alpha_t, \beta_t) \\
  \Leftrightarrow \Pb\lr*{\frac{\lambda_t}{r} \Bigm\vert k_1, \dots, k_t}
    &= \BP\lr*{\frac{\lambda_t}{r} \Bigm\vert \alpha_t, \beta_t}.
\end{align*}
As indicated here, this is equivalent to the assumption of a beta prime prior
on $\lambda_t/r$.

Thirdly, we assume that the predictive prior on $p_t$ is related to the
posterior on $p_{t-1}$ as follows:
\begin{align*}
  \Pb(p_t \mid k_1, \dots, k_{t-1})
    &= \B\lr*{p_t \Bigm\vert \frac{\alpha_{t-1}}{c}, \frac{\beta_{t-1}}{c}} \\
  \Leftrightarrow \Pb\lr*{\frac{\lambda_t}{r} \Bigm\vert k_1, \dots, k_{t-1}}
    &= \BP\lr*{\frac{\lambda_t}{r} \Bigm\vert
    \frac{\alpha_{t-1}}{c}, \frac{\beta_{t-1}}{c}}.
\end{align*}

\section{Results} %=============================================================

The parameters of the posterior distributions satisfy
\begin{align*}
  \alpha_t &= \frac{\alpha_{t-1}}{c} + k_t
    = \frac{a_1}{c^{t-1}} + \sum_{i=0}^{t-1} \frac{k_{t-i}}{c^i}, \\
  \beta_t &= \frac{\beta_{t-1}}{c} + r
    = \frac{b_1}{c^{t-1}} + \sum_{i=0}^{t-1} \frac{r}{c_i},
\end{align*}
and the marginal likelihood of the model parameters $r$ and $c$ is
\begin{align*}
  \Pb(k_1, \dots, k_t \mid r, c)
    &= \prod_{t=1}^{t} \Pb(k_t \mid k_1, \dots, k_{t-1}, r, c) \\
  &= \prod_{t=1}^{t} \frac{1}{k_t\B(k_t, r)}
    \frac{\B(a_t + k_t, b_t + r)}{\B(a_t, b_t)} \quad (\text{if } k_t > 0).
\end{align*}

\end{document}


\documentclass{article}

\title{Neural Network Initialization}
\author{Vladimir Feinberg}

\input{../old/defs}
\usepackage{animate}

\begin{document}

\maketitle

Initialization is an important aspect of ANN training. It contributes to improvements both in terms of training speed and generalization. Content is mostly from Dr. Goodfellow's \nurl{http://www.deeplearningbook.org/}{Deep Learning Book}.

Parameters should not be initialized uniformly so as to break symmetry between hidden units; otherwise, the parameters will remain the same during training. This motivates random initialization. Biases, however, are typically constants, heuristically chosen.

Weights are typically Gaussian or uniform, but scale is important. Large scale, which breaks symmetry strongly, must be balanced with stability of the learning algorithm. ReLU activations, since they're linear, suggest that that the scale of the output for a basic ReLU MLP with $\norm{y}\sim \norm{\vx}\prod_i\norm{W^{(i)}}$. In general, the scale of weights should be treated as a hyperparameter. Importantly, loss (such as softmax) not scale-invariant; keep an eye out for initialization.

Output layer bias should be initialized with the marginal statistics of the training data. Certain initialization schemes may be compatible with positive constant bias initialization, which lets gradients start on the linear part of a ReLU, but it is unclear if this yields improvement over zero bias initialization. (\nurl{https://research.google.com/pubs/pub45473.html}{Jozefowicz et al 2015}). If a unit acts as a control gate for letting other information pass (like a forget gate in an LSTM), then bias should be initialized as positive there.

\section{Uniform Random Initialization}

(\nurl{http://proceedings.mlr.press/v9/glorot10a.html}{Glorot and Bengio 2010}). Normed initialization suggests that a fully connected layer initializes an $n\times m$ weight matrix entrywise with $\Uniform\pa{-u, u}$ where $u=\sqrt{\frac{6}{m+n}}$. This approach intends to keep gradient variance the same across layers, and it is very commonly used because of its simplicity.

\section{Orthogonal Matrix Initialization}

(\nurl{https://arxiv.org/abs/1312.6120}{Saxe et al 2013}, \nurl{https://arxiv.org/abs/1412.6558}{Susillo 2014}). Orthogonal matrix initialization with a scaling factor dependent on layer activation is motivated by making hidden units track a varied set of functions, resulting in training time independent of depth (if nonlinearities are the identity).

\subsection{Greedy Pre-training}

(\nurl{https://papers.nips.cc/paper/3048-greedy-layer-wise-training-of-deep-networks}{Bengio et al 2007}). Greedy pre-training is a form of initialization where a FFNN is trained by first training a shallow network to completion, then appending new layers after lower layers are trained. This greedy form of optimization isn't optimal from a training loss perspective, but in addition to speeding up convergence it can be viewed as a form of regularization. However, it has fallen out of favor as other modern techniques (ReLU, dropout, BN, unsupervised pre-training) have been developed.

\subsection{Unsupervised Pre-training}

(\nurl{http://www.jmlr.org/papers/v11/erhan10a.html}{Erhan et al 2010}, \nurl{https://arxiv.org/abs/1412.6597}{Paine et al 2014}). Unsupervised approaches have found success in both improving generalization error and optimization time. Here, layers from unsupervised learners, such as autoencoders or GANs, might be used (\nurl{https://arxiv.org/abs/1606.03498}{Salimans et al 2017}). An early technique was to use TICA (\nurl{http://people.ee.duke.edu/~lcarin/Iulian9.22.06b.pdf}{Hyv\:{a}rinen et al 2001}), introduced by \nurl{https://papers.nips.cc/paper/4136-tiled-convolutional-neural-networks}{Le et al 2010}.

\subsection{Layer-sequential Unit-variance (LSUV)}

(\nurl{https://arxiv.org/abs/1511.06422}{Mishkin and Matas 2015}). Tested out on FFNNs with results comparable to BN, greedy pre-training, and other initialization schemes, LSUV offers a cheap initialization procedure which effectively does a single round of BN at initialization time. There's not a lot of background on LSUV use, nor its extensions to RNNs.

\end{document}
% LocalWords: Dumoulin Visin Boureau Ngiam autoplay Theano Hyv rinen Erhan


Previously, the idea of using \textit{optimal control}\cite{Kirk2004} techniques (also referred to as \textit{trajectory optimization}) for robot motion planning and control was presented. In this chapter, the optimal control problem is revisited in more detail, including a brief discussion on the use of both \textit{indirect} and \textit{direct} methods.

\notessection{Optimal Control and Trajectory Optimization}
Consider an optimal control problem (OCP) formulated as the following optimization problem:
\begin{equation} \label{eq:ocp}
\begin{split}
\underset{\bm{u},\x}{\text{minimize}} \:\: & h(\x(t_f),t_f) + \int_{t_0}^{t_f} g(\x(t),\bm{u}(t),t) dt,\\
\text{s.t.} \:\:& \dot{\x}(t) = a(\x(t),\bm{u}(t),t), \\
&\x(t_0) = \x_0,
\end{split}
\end{equation}
where $\x \in \R^n$ is the robot state, $\bm{u} \in \R^m$ is the control input, $x_0$ is a known robot initial condition, $a(\x, \bm{u}, t)$ is a function describing the robot's dynamics, and the functions $h(\x(t_f), t_f)$ and $g(\x(t), \bm{u}(t), t)$ define the cost function\footnote{State constraints $\x(t) \in \mathcal{X}$ and control constraints $\bm{u}(t) \in \mathcal{U}$ are also often included in practice, but for simplicity are not included here.}.
The goal is to solve the optimal control problem \eqref{eq:ocp} in order to define an \textit{optimal} open-loop control law of the form
\begin{equation*}
    \bm{u}^*(t) = f(\x(t_0), t).
\end{equation*}

Unfortunately, this optimization problem is particularly challenging to solve since it is \textit{infinite-dimensional}\footnote{It is referred to as infinite-dimensional because it is an optimization over functions and not just a finite set of parameters.}. Methods for solving \eqref{eq:ocp} can be categorized as either \textit{indirect} or \textit{direct}. Both types of methods (almost always) require some form of discretization, such that the problem can be solved numerically. However, the way in which the problem is discretized is what makes each method unique.

\begin{enumerate}
    \item \textit{Indirect methods} follow a ``first optimize, then discretize" approach. These methods first derive conditions for optimality of the original infinite-dimensional problem. A solution is then recovered by discretizing the optimality conditions.
    \item \textit{Direct methods} follow a ``first discretize, then optimize" approach. These methods first discretize the original problem into a finite-dimensional problem (called a \textit{nonlinear program}), which is then solved numerically to recover an optimal solution.
\end{enumerate}

\subsection{Indirect Methods}
As previously mentioned, indirect methods solve the optimal control problem \eqref{eq:ocp} by deriving \textit{necessary optimality conditions} (NOC). A numerical procedure is then used to find solutions that satisfy these conditions of optimality, thereby ``indirectly'' solving the original OCP. As a brief example, for unconstrained finite-dimensional optimization problems the classic first-order necessary optimality condition\footnote{It is important to note that these conditions are called necessary because they are ``necessary'', but they may not be ``sufficient''. In other words there may exist solutions that satisfy the NOCs but do not solve the original problem.} is that the gradient of the function must be zero (e.g. minimize $f(x) = x^2$ with $x \in \R$ has NOC $\frac{df}{dx} = 0$).

\subsubsection{Constrained Finite-Dimensional Optimization} \label{subsubsec:constfiniteopt}
Before discussing techniques to derive necessary optimality conditions for the infinite-dimensional OCP \eqref{eq:ocp}, it is useful to briefly examine analogous conditions in finite-dimensional optimization\cite{BoydVandenberghe2004}. Consider the equality-constrained finite-dimensional optimization problem:
\begin{equation} \label{eq:finiteopt}
\begin{split}
\underset{\x}{\text{minimize}} \:\: &f(\x), \\
\text{s.t.} \:\:&h_i(\x) = 0, \quad i = 1,\dots,m \\
\end{split}
\end{equation}
with variable $\x \in \R^n$. \\

Necessary optimality conditions for \eqref{eq:finiteopt} are derived by first forming a function called the Lagrangian $L(\x, \blam)$, which augments the objective function with a weighted sum of the constraint functions:
\begin{equation} \label{eq:lagrangian}
    L(\x,\blam) = f(\x) + \sum_{i=1}^m \lambda_i h_i(\x),
\end{equation}
where $\blam \in \R^m$ is a vector of \textit{Lagrange multipliers}. The NOCs are then given as:
\begin{equation} \label{eq:finitenoc}
\begin{split}
\nabla_{\x} L(\x^*,\blam^*) &= 0, \\
\nabla_{\blam} L(\x^*,\blam^*) &= 0,
\end{split}
\end{equation}
which are the gradients of the Lagrangian with respect to the variables $\x$ and the multipliers $\blam$. Note that the NOCs \eqref{eq:finitenoc} are a set of $n+m$ \textit{algebraic} equations with $n+m$ unknowns. In contrast, it will be seen next that the NOCs for infinite-dimensional problems are not algebraic, but rather differential.

\subsubsection{Necessary Optimality Conditions}
Analogously to the Lagrangian \eqref{eq:lagrangian} in finite-dimensional optimization, the first step to defining the NOCs for the infinite-dimensional OCP \eqref{eq:ocp} is to define a function called the \textit{Hamiltonian}:
\begin{equation} \label{eq:hamiltonian}
H(\x(t), \bu(t), \p(t), t) := g(\x(t), \bu(t), t) + \p^\top (t)a(\x(t), \ac(t), t),
\end{equation}
where $\p(t) \in \R^n$ is a multiplier referred to as a \textit{costate}. The NOCs are then given by a set of differential and algebraic equations:
\begin{equation} \label{eq:optnoc}
\begin{split}
\dot{\x}^*(t) &= \frac{\partial{H}}{\partial{\p}}(\x^*(t), \bu^*(t), \p^*(t), t), \\
\dot{\p}^*(t) &= -\frac{\partial{H}}{\partial{\x}}(\x^*(t), \bu^*(t), \p^*(t), t), \\
0 &= \frac{\partial{H}}{\partial{\bu}}(\x^*(t), \bu^*(t), \p^*(t), t),
\end{split}
\end{equation}
which must be satisfied for all $t \in [t_0, t_f]$. These NOCs consist of $2n$ first order differential equations and $m$ algebraic equations. Identifying unique solutions to the $2n$ differential equations requires $2n$ boundary conditions (actually $2n+1$ if the final time $t_f$ is not fixed). The initial condition $\x^*(t_0) = \x_0$ specifies $n$ of these conditions, and the remaining conditions are given by
\begin{equation} \label{eq:boundarycond}
\begin{split}
&\big(\frac{\partial{h}}{\partial{\x}}(\x^*(t_f), t_f) - \p^*(t_f))\big)^\top \delta{\x_f}\\\
& + \big(H(\x^*(t_f), \ac^*(t_f), \p^*(t_f), t_f) +\frac{\partial{h}}{\partial{t}}(\x^*(t_f), t_f)\big)\delta t_f = 0,\\
\end{split}
\end{equation}
where $\delta \x_f$ and $\delta t_f$ are referred to as \textit{variations}. If either the final time or final state is fixed in the optimal control problem the corresponding variation is forced to be zero, which changes the boundary conditions \eqref{eq:boundarycond}. The resulting boundary conditions for the four possible scenarios are now summarized:

\paragraph{Fixed Final Time and Fixed Final State:} If both $t_f$ and $\x(t_f)$ are fixed, both variations $\delta t_f$ and $\delta \x_f$ are set to zero. In this case the boundary conditions \eqref{eq:boundarycond} are trivially satisfied, and the remaining boundary conditions on the NOCs \eqref{eq:optnoc} are given by:
\begin{equation*}
\begin{split}
&\x^*(t_0) = \x_0, \\
&\x^*(t_f) = \x_f.
\end{split}
\end{equation*}

\paragraph{Fixed Final Time and Free Final State:} If only $t_f$ is fixed, then only the variation $\delta t_f = 0$. In this case the conditions \eqref{eq:boundarycond} simplify and the boundary conditions for the NOCs \eqref{eq:optnoc} are given by:
\begin{equation*}
\begin{split}
&\x^*(t_0) = \x_0, \\
&\frac{\partial h }{\partial \x} (\x^* (t_f), t_f) - \p^*(t_f) = 0.
\end{split}
\end{equation*}

\paragraph{Free Final Time and Fixed Final State:} If only $\x_f$ is fixed, then only the variation $\delta \x_f = 0$. In this case the conditions \eqref{eq:boundarycond} simplify and the boundary conditions for the NOCs \eqref{eq:optnoc} are given by:
\begin{equation*}
\begin{split}
&\x^*(t_0) = \x_0, \\
&\x^*(t_f) = \x_f, \\
&H(\x^*(t_f), \ac^*(t_f), \p^*(t_f), t_f) +\frac{\partial{h}}{\partial{t}}(\x^*(t_f), t_f) = 0.
\end{split}
\end{equation*}
Note that in this case since the final time is free an additional boundary condition is added, so there are now $2n + 1$ total conditions.

\paragraph{Free Final Time and Free Final State:} If neither $t_f$ or $\x(t_f)$ is fixed, then the boundary conditions for the NOCs \eqref{eq:optnoc} are given by:
\begin{equation*}
\begin{split}
&\x^*(t_0) = \x_0, \\
&\frac{\partial h }{\partial \x} (\x^* (t_f), t_f) - \p^*(t_f) = 0, \\
&H(\x^*(t_f), \ac^*(t_f), \p^*(t_f), t_f) +\frac{\partial{h}}{\partial{t}}(\x^*(t_f), t_f) = 0.
\end{split}
\end{equation*}
Again, since the final time is free an additional boundary condition is added such that there are $2n+1$ total. Note that last two conditions are both extracted from \eqref{eq:boundarycond} because the variations $\delta \x_f$ and $\delta t_f$ are independent.

\subsubsection{Two-Point Boundary Value Problems}
Finding solutions that satisfy the necessary optimality conditions \eqref{eq:optnoc} for the optimal control problem is challenging. In particular, any solution must satisfy a set of $2n$ differential equations with boundary conditions specified at both $t_0$ and $t_f$. The problem of finding solutions to differential equations with boundary conditions specified at two points is called a \textit{two-point boundary value problem}.
Luckily, numerical procedures have been developed for solving these types of problems. For example the \texttt{scikits.bvp\_solver} package in Python or the function \texttt{bvp4c} in Matlab implement schemes for solving these problems.

Most solvers for two-point boundary value problems typically assume the NOCs \eqref{eq:optnoc} and their boundary conditions are expressed in the standard form:
\begin{equation} \label{eq:standardtpbvp}
\dot{\z} = g(\z, t), \quad l(\z(t_0), \z(t_f)) = 0.
\end{equation}
However, some types of problems may not directly fit into this standard form. For such instances, it is sometimes possible to convert a non-standard form problem into the standard form \eqref{eq:standardtpbvp} \cite{AscherRussel1981}.

In optimal control settings one common case where the two-point boundary value problem cannot directly be expressed in standard form is free final time problems, where $t_f$ needs to be determined but does not have any associated dynamics. A useful trick in this case is to define a new variable $\tau = \frac{t}{t_f} \in [0,1]$ to replace the time variable $t$ (since before $t_f$ wasn't known but now $\tau_f = 1$ is known). With this new variable the following changes can be made:
\begin{enumerate}
    \item Replace all derivatives with respect to $t$ with derivatives with respect to $\tau$, using $\frac{d(\cdot)}{d\tau} = t_f \frac{d(\cdot)}{dt}$ (chain rule).
    \item Introduce a ``dummy'' state $r$ that corresponds to $t_f$ with dynamics $\dot{r} = 0$.
    \item Replace $t_f$ with $r$ in all NOCs and in all boundary conditions.
\end{enumerate}
The ``dummy'' state $r$ can then be included in the vector $\z$ and the NOCs expressed in the standard form \eqref{eq:standardtpbvp}. In summary, this approach can be thought of as ``tricking'' the standard-form solver to think that the final time is 1 and that $t_f$ is actually a state with dynamics (although the dynamics are $\dot{t}_f = 0$).

\begin{example}[Free Final Time OCP] \label{ex:ocp}
\theoremstyle{definition}
Consider a double integrator system
\begin{equation*}
    \ddot{x} = u,
\end{equation*}
where $x \in \R$ is the state and $u \in \R$ is the control input where the control task is to find a trajectory that minimizes the cost function
\begin{equation*}
J = \frac{1}{2}\alpha t_f^2 + \int_{0}^{t_f} \frac{1}{2}\beta u^2(t) \mathrm{d}t,
\end{equation*}
and satisfies the boundary conditions
\begin{equation*}
x(0) = 10,\quad \dot{x}(0) = 0,\quad x(t_f) = 0,\quad \dot{x}(t_f) = 0.
\end{equation*}
This problem is a free final time problem with a fixed final state, and the cost is formulated to find a trajectory that minimizes a combination of the time to reach the final state and the amount of control effort required to get there. A trade-off between minimizing final time and minimizing control effort is made by adjusting the weighting parameters\footnote{What does intuition suggest the optimal behavior would be for $\alpha = 0$ or for $\beta=0$?} $\alpha$ and $\beta$. From the cost function it is apparent that:
\begin{equation*}
\begin{split}
h(\x(t_f), t_f) &= \frac{1}{2}\alpha t_f^2, \quad g(\x(t), \bu(t), t) = \frac{1}{2}\beta u^2(t),
\end{split}
\end{equation*}
and the dynamics equation can be equivalently expressed as a first-order system of ODEs by setting $x_1 = x$ and $x_2 = \dot{x}$:
\begin{equation*}
\begin{split}
\dot{x}_1 &= x_2, \\
\dot{x}_2 &= u, \\
\end{split}
\end{equation*}
such that $\x = [x_1, \: x_2]^\top $ and the boundary conditions become:
\begin{equation*}
x_1(0) = 10, \quad x_2(0) = 0,\quad x_1(t_f) = 0,\quad x_2(t_f) = 0.
\end{equation*}

Now that the problem has been introduced, the first step is to derive the Hamiltonian:
\begin{equation*}
H = \frac{1}{2}\beta u^2 + p_1x_2 + p_2u,
\end{equation*}
where $p_1$ and $p_2$ are the costates. Next, the NOCs \eqref{eq:optnoc} can be derived by taking the partial derivatives of $H$ with respect to $p$, $x$, and $u$:
\begin{equation*}
\begin{split}
\dot{x}_1^* &= x_2^*, \\
\dot{x}_2^* &= u^*, \\
\dot{p}_1^* &= 0, \\
\dot{p}_2^* &=-p_1^*, \\
0 & = \beta u^* + p_2^*.
\end{split}
\end{equation*}
The next step is then to determine appropriate boundary conditions for the NOCs. As mentioned before, this problem is a free final time and fixed final state problem. Therefore the boundary conditions are given by
\begin{equation*}
\begin{split}
&x_1^*(0) = 10, \\
&x_2^*(0) = 0, \\
&x_1^*(t_f) = 0, \\
&x_2^*(t_f) = 0, \\
&\frac{1}{2}\beta u^*(t_f)^2 + p_1^*(t_f)x_2^*(t_f)+p_2^*(t_f)u^*(t_f) + \alpha t_f = 0.
\end{split}
\end{equation*}

Now, from the last NOC it can be seen that the optimal control $u^*$ can be solved for in terms of the costate $p_2^*$:
\begin{equation*}
    u^* = -\frac{1}{\beta}p_2^*.
\end{equation*}
This expression can then be substituted into the second NOC and into the boundary conditions. At this point the resulting two-point boundary value problem can be expressed in the standard form \eqref{eq:standardtpbvp} (by using the free final time trick previously discussed), and solved numerically. However, it also turns out that this problem is simple enough to solve analytically as well.

\paragraph{Analytical Solution:}
Integrating the differential equations for the costates $p_1$ and $p_2$ gives:
\begin{equation*}
\begin{split}
p_1^* &= C_1, \\
p_2^* &= -C_1t + C_2, \\
\end{split}
\end{equation*}
where $C_1$ and $C_2$ are constants. Therefore, the optimal control $u^*$ can be expressed as $u^* = \frac{C_1}{\beta}t - \frac{C_2}{\beta}$ and the states $x_1$ and $x_2$ can be integrated to yield:
\begin{equation*}
\begin{split}
x_2^* &= \frac{C_1}{2\beta}t^2 - \frac{C_2}{\beta}t + C_3, \\
x_1^* &= \frac{C_1}{6\beta}t^3 - \frac{C_2}{2\beta}t^2 + C_3t + C_4, \\
\end{split}
\end{equation*}
where $C_3$ and $C_4$ are additional constants. There are now five unknown quantities, $C_1$, $C_2$, $C_3$, $C_4$, and $t_f$, which can be determined by leveraging the five boundary conditions. In particular from the condition $x_1^*(0) = 10$ and $x^*_2(0) = 0$ it is easy to see that $C_3 = 0$ and $C_4 = 10$. The remaining boundary conditions can then be used to analytically solve for the remaining constants, and in particular:
\begin{equation*}
t_f = (1800 \frac{\beta}{\alpha})^{1/5}.
\end{equation*}

For a couple of interesting insights, it can be noted that as $\beta \xrightarrow{} 0$ the cost function penalizes the final time more, and from the expression for $t_f$ we can see that $t_f \xrightarrow{} 0$. Additionally, as $\alpha \xrightarrow{} 0$ the cost function penalizes control inputs, and correspondingly it can be seen in the expression for $t_f$ that $t_f \xrightarrow{} \infty$. Further, note that the optimal control takes the form
\begin{equation*}
    u^*(t) = \frac{C_1}{\beta}t - \frac{C_2}{\beta}.
\end{equation*}
Thus the control input is linear in time and its magnitude is inversely proportional to $\beta$.
\end{example}

\subsection{Direct Methods}
Unlike indirect methods, direct methods do not require a derivation of the necessary optimality conditions. Instead these methods directly discretize the original optimal control problem \eqref{eq:ocp} to turn it into a finite-dimensional constrained optimization problem called a \textit{nonlinear programming problem}.

While several approaches for discretizing the OCP exist, one simple approach is to just use a forward Euler time discretization. Recall that the forward Euler time discretization method (the simplest of the Runge-Kutta methods) can be used to numerically solve differential equations. In particular, with the choice of a time step $h_i$ the differential equations $\dot{\x} = a(\x, \bu, t)$ are discretized as:
\begin{equation} \label{eq:hdynamics}
\x_{i+1} = \x_{i} + h_ia(\x_i, \bu_i, t_i),
\end{equation}
where $\x_i = \x(t_i)$, $\bu_i = \bu(t_i)$, and $t_{i+1} - t_i = h_i$. With this recursive expression \eqref{eq:hdynamics}, an initial condition $\x(t_0)$, and a sequence of inputs $\bu(t_i)$ for $i \geq 0$, the states $\x(t_i)$ can be computed easily.
Suppose the optimal control problem \eqref{eq:ocp} was defined over the time interval $[t_0, t_f]$. Applying a forward Euler time discretization essentially partitions this interval into a finite set of $N$ times $\{t_0, t_1, \dots, t_N\}$ where $t_N = t_f$ and the time step between each is $h_i = t_{i+1} - t_i$. Then the parameters of the optimization problem will simply become the state and controls at these times, $\x_i = \x(t_i)$ and $\bu_i = \bu(t_i)$ for $i = 0,\dots,N$.

Rewriting the original OCP \eqref{eq:ocp} as a function of the discrete set of parameters $t_i$, $\x_i$, and $\bu_i$ will require modifications to both the constraints and to the cost function. First, the recursive formula \eqref{eq:hdynamics} is used to replace the dynamics constraint $\dot{\x} = a(\x, \bu, t)$ in the OCP\footnote{The original dynamics model $\dot{\x} = a(\x, \bu, t)$ is sometimes called the \textit{continuous time} model and the recursive formula $\x_{i+1} = \x_{i} + ha(\x_i, \bu_i, t_i)$ is called the \textit{discrete time} model.}. Updating the cost function is going to require a numerical approximation of the integral, such as by using one of the Newton-Cotes formulas. The simplest of which would yield the approximation:
\begin{equation*}
\int_{t_0}^{t_f} g(\x(t),\bm{u}(t),t) dt \approx \sum_{i=0}^{N-1} h_i g(\x_i,\bm{u}_i,t_i).
\end{equation*}
The OCP \eqref{eq:ocp} can now be expressed completely as the finite-dimensional nonlinear program (NLP):
\begin{equation} \label{eq:nlp}
\begin{split}
\underset{\bm{u}_i,\x_i}{\text{minimize}} \:\: & h(\x_N,t_N) + \sum_{i=0}^{N-1} h_i g(\x_i,\bm{u}_i,t_i),\\
\text{s.t.} \:\:& \x_{i+1} = \x_{i} + h_ia(\x_i, \bu_i, t_i), \quad i = 0, \dots, N-1, \\
&\x_0 = \x(t_0).
\end{split}
\end{equation}

\subsection{Consistency of Time Discretization}
The finite-dimensional problem \eqref{eq:nlp} is only an \textit{approximation} of the original problem \eqref{eq:ocp}, so it is important to justify that this approximation method is \textit{consistent} with the original problem. This is accomplished by taking a look at the necessary optimality conditions for the NLP \eqref{eq:nlp} and comparing them to the necessary optimality conditions for the original OCP \eqref{eq:ocp}.

Recall that the necessary conditions of optimality for equality-constrained finite-dimensional optimization problems have previously been discussed in Section \ref{subsubsec:constfiniteopt}. In particular, the Lagrangian is first formulated, which for \eqref{eq:nlp} takes the form:
\begin{equation*}
L = h(\x_N,t_N) + \sum_{i=0}^{N-1} h_i g(\x_i,\bm{u}_i,t_i) + \sum_{i=0}^{N-1} \blam_i^\top (\x_{i} + h_ia(\x_i, \bu_i, t_i) - \x_{i+1}).
\end{equation*}
Note that even though the initial condition constraint is included in \eqref{eq:nlp} it can be ignored in the Lagrangian by simply assuming $\x_0$ is not actually a decision variable in the optimization problem (since it is fixed).
The NOCs are then given by:
\begin{equation} \label{eq:dirnoc}
\begin{split}
\nabla_{\x_i} L = h_i \frac{\partial g}{\partial \x}(\x_i, \bu_i) + h_i\big(\frac{\partial a}{\partial \x}(\x_i, \bu_i)\big)^\top  \blam_i + (\blam_i - \blam_{i-1}) &= 0, \quad i = 1,\dots,N-1 \\
\nabla_{\x_N} L = \frac{\partial h}{\partial \x}(\x_N) - \blam_{N-1} &= 0, \\
\nabla_{\bu_i} L = h_i \frac{\partial g}{\partial \bu}(\x_i, \bu_i) + h_i\big(\frac{\partial a}{\partial \bu}(\x_i, \bu_i)\big)^\top  \blam_i &= 0, \quad i = 0, \dots, N-1\\
\x_{i} + h_ia(\x_i, \bu_i, t_i) - \x_{i+1} &= 0, \quad i = 0,\dots,N-1\\
\end{split}
\end{equation}

Now, from the indirect method with equations \eqref{eq:hamiltonian}, \eqref{eq:optnoc}, and boundary conditions \eqref{eq:boundarycond} with fixed final time and free final state, the NOCs for the infinite-dimensional OCP can be written as:
\begin{equation} \label{eq:indinoc}
\begin{split}
\frac{\partial g}{\partial \x}(\x(t), \bu(t)) + \big(\frac{\partial a}{\partial \x}(\x(t), \bu(t))\big)^\top  \p(t) + \dot{\p}(t) &= 0, \quad t \in [t_0,t_f] \\
\frac{\partial h}{\partial \x}(\x(t_f)) - \p(t_f) &= 0, \\
\frac{\partial g}{\partial \bu}(\x(t), \bu(t)) + \big(\frac{\partial a}{\partial \bu}(\x(t), \bu(t))\big)^\top  \p(t) &= 0, \quad t \in [t_0, t_f]\\
\dot{\x}(t) - a(\x(t), \bu(t), t) &= 0, \quad t\in[t_0,t_f]\\
\x_0 - \x(t_0) &= 0.
\end{split}
\end{equation}

The NOCs \eqref{eq:dirnoc} for the discretized problem and the NOCs for the original OCP \eqref{eq:indinoc} are remarkably similar. In fact, the NOCs \eqref{eq:dirnoc} can be seen as themselves simply the discretized versions of \eqref{eq:indinoc}. To see this, simply perform a forward Euler discretization of the equations in \eqref{eq:indinoc} with:
\begin{equation*}
\begin{split}
\dot{\p}(t) = \frac{\blam_{i} - \blam_{i-1}}{h_i}, \quad \p(t_i) = \blam_i, \quad i = 0, \dots, N-1, \\
\dot{\x}(t) = \frac{\x_{i+1} - \x_{i}}{h_i}, \quad \x(t_i) = \x_i, \quad \bu(t_i) = \bu_i, \quad i = 0, \dots, N-1.
\end{split}
\end{equation*}
Therefore, as the time step $h_i \xrightarrow{} 0$ the NOCs for the discretized (direct method) problem converge to the NOCs derived directly for the original infinite-dimensional OCP (indirect method)!

\subsection{Exercises}
\subsubsection{Optimal Control and Trajectory Optimization}
Complete \textit{Extra Problem:  Optimal Control and Trajectory Optimization} located in the online repository:

\vspace{\baselineskip}

\url{https://github.com/PrinciplesofRobotAutonomy/AA274A_HW1},

\vspace{\baselineskip}

where you will compute a dynamically feasible and \textit{optimal} trajectory for a unicycle robot by using an indirect method to set up the necessary optimality conditions and solve them using a two-point boundary value solver.


\documentclass[11pt]{amsart}
\usepackage{geometry}                % See geometry.pdf to learn the layout options. There are lots.
\geometry{letterpaper}                   % ... or a4paper or a5paper or ...
%\geometry{landscape}                % Activate for for rotated page geometry
%\usepackage[parfill]{parskip}    % Activate to begin paragraphs with an empty line rather than an indent
\usepackage{graphicx}
\usepackage{amssymb}
\usepackage{epstopdf}
\DeclareGraphicsRule{.tif}{png}{.png}{`convert #1 `dirname #1`/`basename #1 .tif`.png}

\title{Category cheat sheet for Haskellers}
\author{Mark Hopkins}
%\date{}                                           % Activate to display a given date or no date
\DeclareMathOperator{\End}{End}
\DeclareMathOperator{\Coend}{Coend}
\DeclareMathOperator{\Hom}{Hom}

\newcommand{\myvec}[1]{\vec{#1}}
\newcommand{\cat}[1]{\mathbf{#1}}
\newcommand{\op}[1]{#1^{\text{op}}}

\newcommand{\blank}{{-}}

%\def\cat{Cat}
\begin{document}
%\newcommand{cat}[1]{\boldmath{#1}}
    \maketitle

    \subsection*{Products}

    $\cat{C}$ has products if
    $
    \cat{C}(C, A \times B) \cong \cat{C}(C, A) \times \cat{C}(C, B)
    $

    \subsection*{Coproducts}

    $\cat{C}$ has coproducts or sums if
    $
    \cat{C}(A + B, C) \cong \cat{C}(A,C) \times \cat{C}(B,C)
    $

    \subsection*{Quotient}
    A quotient of a set is a way of identifying elements.

    \subsection*{Enriched categories}
    This means doing category theory with a category $\mathcal V$ in place of $\cat{Set}$.
    This means there is a $\mathcal V$-object of object of objects and hom-objects instead of hom-sets.

    $\mathcal V$ should be a complete, cocomplete, biclosed monoidal category.
%\section{}
%\subsection{}

    \subsection*{Complete category}
    A category is complete if it has all limits.

    \subsection*{Cocomplete category}
    A category is cocomplete if it has all colimits.

    \subsection*{Profunctor}
    A profunctor on $\cat C$ is a functor $\op{\cat C} \times {\cat c} \to \cat(Set)$

    \subsection*{Ends}
    If $F:\op{\cat C} \times \cat C \to \cat D$
    then we can form the ``diagonal'' product $\prod_{c\in\cat C} F(c,c)$ in $\cat D$.

    Given a morphism $f: c \to c'$ in $\cat C$, there are two ways we can apply it to factors in this product:
    we can apply it covariantly

    \[
        F(c, f): F(c,c) \to F(c', c')
    \]

    or contravariantly

    \[
        F(f, c'): F(c',c') \to F(c', c').
    \]

    $\End F = \int_{\cat C} F = \int_{c\in\cat C} F(c,c)$
    is the subobject for which these give the same answer for all $f$.

    It comes with projection morphisms
    $\int_{c\in\cat C} F(c,c) \to F(c, c)$ for every $c\in \cat{C}$
    satisfying a universal property.

    The universal quantifier lets us approximate the end of a profunctor in Haskell.

    \begin{verbatim}
        data End f = End (forall c. f c c)

        proj :: End f -> f c c
        proj (End fcc) = fcc
    \end{verbatim}

    \subsection*{Coends}
    If $F:\cat C^{\text{op}} \times \cat C \to \cat D$
    then we can form the ``diagonal'' sum $\sum_{c\in\cat C} F(c,c)$ in $\cat D$.

    Given a morphism $f: c \to c'$ in $\cat C$, there are two ways we can apply it to terms in this sum:
    we can apply it covariantly

    \[
        F(1_c, f): F(c,c) \to F(c', c')
    \]

    or contravariantly

    \[
        F(f, 1_{c'}): F(c',c') \to F(c', c').
    \]

    $\Coend F = \int^{\cat C} F = \int^{c\in\cat C} F(c,c)$
    is the quotient of the sum which identifies the two answers for all $f$.

    It comes with injection morphisms
    $F(c, c) \to \int^{c\in\cat C} F(c,c)$ for every $c\in \cat{C}$
    satisfying a universal property.

    The existential quantifier lets us approximate the coend of a profunctor in Haskell.

    \begin{verbatim}
        data Coend f = forall c. Coend (f c c)

        inj :: f c c -> Coend c
        inj fcc = Coend fcc
    \end{verbatim}

    \subsection*{Presheaf}
    A presheaf on $\cat C$ is a functor $\op{\cat C} -> \cat{Set}$.

    \subsection*{Representable functor}
    A covariant representable functor is one isomorphic to $\cat{C}(c, \blank)$.

    In Haskell this means a datatype defined using only products (no sums).
    For instance

    \begin{verbatim}
        data Pair a = Pair a a deriving (Eq, Show, Functor)
    \end{verbatim}

    \verb|Pair a| is isomorphic to \verb|Bool -> a|, so it's representable, and represented by \verb|Bool|.

    You can think of it as an indexable data structure, with $c$ playing the role of the index.
    We also call this the reader monad.

    A contravariant representable functor is one isomorphic to $\cat{C}(\blank, c)$.
    It's an example of a presheaf

    \subsection*{Yoneda lemma}
    The Yoneda lemma says that natural transformations \textit{out of} a representable functor (covariant or
    contravariant)
    are tightly controlled.

    \[
        [\op{\cat{C}}, \cat{Set}]\left(\cat{C}(c, \blank), G\right) \cong G(c)
    \]

    In words, if $F$ and $G$ are presheaves on $\cat C$, and $F$ is represented by $c$, then the natural transformations
    from $F$ to $G$ line up one-to-one with the elements of the set $G(c)$.
    \subsection*{Kan extensions}
    \subsection*{}

\end{document}