\hypertarget{bootstrap-sampling}{%
\chapter{Bootstrap Sampling}\label{bootstrap-sampling}}

In the previous chapter we used resampling to compute standard errors
and confidence intervals, which quantify the variability in an estimate
due to random sampling. As one of the examples, we used data from the
General Social Survey (GSS) to explore changes in support for gun
control over time and to compute confidence intervals for the estimated
proportions.

In this chapter, we'll use GSS data to estimate average income and the
10th percentile of income. We'll see that the resampling method we used
in the previous chapter works for the average income but not for the
10th percentile.

To solve this problem, we'll use another kind of resampling, called
bootstrapping or bootstrap sampling (see
\url{https://en.wikipedia.org/wiki/Bootstrapping_(statistics)}). Then
we'll use bootstrapping to compute sampling distributions and confidence
intervals for other statistics, including the coefficient of correlation
and the parameters of linear regression. Finally, I'll point out a
problem with bootstrap resampling when there are not enough different
values in a dataset, and a way to solve it with KDE resampling.

\hypertarget{estimating-average-income}{%
\section{Estimating Average Income}\label{estimating-average-income}}

As a first example, we'll use data from the General Social Survey to
estimate average family income. The following cell loads the data, which
I have stored in an HDF file.

\begin{lstlisting}[language=Python,style=source]
import pandas as pd

gss = pd.read_hdf('gss_eda.hdf', 'gss')
gss.head()
\end{lstlisting}

\begin{tabular}{lrrrrrrrr}
\toprule
{} &  YEAR &  ID\_ &   AGE &  EDUC &  SEX &  GUNLAW &  GRASS &  REALINC \\
\midrule
0 &  1972 &    1 &  23.0 &  16.0 &    2 &     1.0 &    NaN &  18951.0 \\
1 &  1972 &    2 &  70.0 &  10.0 &    1 &     1.0 &    NaN &  24366.0 \\
2 &  1972 &    3 &  48.0 &  12.0 &    2 &     1.0 &    NaN &  24366.0 \\
3 &  1972 &    4 &  27.0 &  17.0 &    2 &     1.0 &    NaN &  30458.0 \\
4 &  1972 &    5 &  61.0 &  12.0 &    2 &     1.0 &    NaN &  50763.0 \\
\bottomrule
\end{tabular}

The column \passthrough{\lstinline!REALINC!} records family income,
converted to 1986 dollars. We can use \passthrough{\lstinline!notna!}
and \passthrough{\lstinline!sum!} to count the number of valid
responses.

\begin{lstlisting}[language=Python,style=source]
n_realinc = gss['REALINC'].notna().sum()
n_realinc
\end{lstlisting}

\begin{lstlisting}[style=output]
58293
\end{lstlisting}

And we can compute the mean and standard deviation of the valid
responses.

\begin{lstlisting}[language=Python,style=source]
mean_realinc = gss['REALINC'].mean()
std_income = gss['REALINC'].std()
print(mean_realinc, std_income)
\end{lstlisting}

\begin{lstlisting}[style=output]
31742.563372816312 29526.067896235618
\end{lstlisting}

The average family income in this dataset is \$31,742. But notice that
the standard deviation is almost equal to the mean. That's because the
distribution of income is long-tailed; many families have incomes below
the mean, and there are a few families with incomes much higher than the
mean. Nevertheless, we can use this function from the previous chapter
to simulate the sampling process.

\begin{lstlisting}[language=Python,style=source]
import numpy as np

def simulate_sample_mean(n, mu, sigma):
    sample = np.random.normal(mu, sigma, size=n)
    return sample.mean()
\end{lstlisting}

\passthrough{\lstinline!simulate\_sample\_mean!} takes as parameters the
sample size, \passthrough{\lstinline!n!}, and the presumed mean and
standard deviation of income. It generates a sample from a normal
distribution with the given mean and standard deviation, and returns the
mean of the sample. If we call it once, we get a random value from the
sampling distribution of the mean.

\begin{lstlisting}[language=Python,style=source]
simulate_sample_mean(n_realinc, mean_realinc, std_income)
\end{lstlisting}

\begin{lstlisting}[style=output]
31679.30633347414
\end{lstlisting}

If we call it many times, we get a random sample from the sampling
distribution.

\begin{lstlisting}[language=Python,style=source]
t1 = [simulate_sample_mean(n_realinc, mean_realinc, std_income)
     for i in range(1000)]
\end{lstlisting}

I'll use the following function to summarize the sampling distribution.

\begin{lstlisting}[language=Python,style=source]
def summarize(t, digits=2):
    table = pd.DataFrame(columns=['Estimate', 'SE', 'CI90'])
    est = np.mean(t).round(digits)
    SE = np.std(t).round(digits)
    CI90 = np.percentile(t, [5, 95]).round(digits)
    table.loc[''] = est, SE, CI90
    return table
\end{lstlisting}

\begin{lstlisting}[language=Python,style=source]
summary1 = summarize(t1, digits=1)
summary1
\end{lstlisting}

\begin{tabular}{lrrl}
\toprule
{} &  Estimate &     SE &                CI90 \\
\midrule
{} &   31741.5 &  118.3 &  [31542.9, 31934.5] \\
\bottomrule
\end{tabular}

The result is a table that shows the mean of the sampling distribution,
the standard error, and a 90\% confidence interval. The mean of the
sampling distribution is consistent with the mean of the data, which is
\$31,743.

\hypertarget{estimating-percentiles}{%
\section{Estimating Percentiles}\label{estimating-percentiles}}

The method in the previous section uses a normal distribution to
simulate the sampling process. In the previous chapter, when we
estimated average height, this worked well because human heights are
well modeled by a normal distribution. But income is not. We can see
that by computing the CDF of a normal distribution with the same mean
and standard as the income data.

\begin{lstlisting}[language=Python,style=source]
from scipy.stats import norm

xs = np.linspace(-40000, 150000)
ys = norm(mean_realinc, std_income).cdf(xs)
\end{lstlisting}

And comparing it to the CDF of the data.

\begin{lstlisting}[language=Python,style=source]
from empiricaldist import Cdf

cdf_realinc = Cdf.from_seq(gss['REALINC'])
\end{lstlisting}

\begin{lstlisting}[language=Python,style=source]
import matplotlib.pyplot as plt

plt.plot(xs, ys, color='0.7', label='normal model')
cdf_realinc.plot(label='GSS data')

plt.xlabel('Family income (1986 $)')
plt.ylabel('CDF')
plt.title('Distribution of income')
plt.legend();
\end{lstlisting}

\begin{center}
\includegraphics[scale=0.75]{12_bootstrap_files/12_bootstrap_29_0.pdf}
\end{center}

The normal distribution is not a good model for the data. As we saw in
the previous section, it is good enough to compute the sampling
distribution of the mean. But for other statistics, like percentiles, it
is not good enough. To demonstrate, here's a function that generates a
sample from a normal distribution and returns the 10th percentile.

\begin{lstlisting}[language=Python,style=source]
def simulate_sample_percentile(n, mu, sigma):
    sample = np.random.normal(mu, sigma, size=n)
    return np.percentile(sample, 10)
\end{lstlisting}

We can use it to generate a sample from the sampling distribution of the
10th percentile.

\begin{lstlisting}[language=Python,style=source]
t2 = [simulate_sample_percentile(n_realinc, mean_realinc, std_income)
      for i in range(1000)]
\end{lstlisting}

Here's a summary of the results.

\begin{lstlisting}[language=Python,style=source]
summary2 = summarize(t2)
summary2
\end{lstlisting}

\begin{tabular}{lrrl}
\toprule
{} &  Estimate &      SE &                 CI90 \\
\midrule
{} &  -6101.04 &  203.16 &  [-6421.5, -5749.55] \\
\bottomrule
\end{tabular}

The mean of the sampling distribution is negative, which is not
consistent with the actual 10th percentile of the data, which is \$5631.

In this example, resampling based on a normal model doesn't produce
sensible results. Fortunately there is a simple alternative that is more
robust: bootstrapping.

\hypertarget{bootstrapping}{%
\section{Bootstrapping}\label{bootstrapping}}

Bootstrapping is a kind of resampling, based on the framework we saw in
the previous chapter:

\includegraphics{figs/resampling.png}

The idea is that we treat the sample as if it were the entire
population, and simulate the sampling process by choosing random rows
with replacement. \passthrough{\lstinline!DataFrame!} provides a method
called \passthrough{\lstinline!sample!} we can use to select a random
sample of the rows.

\begin{lstlisting}[language=Python,style=source]
bootstrapped = gss.sample(n=n_realinc, replace=True)
bootstrapped.shape
\end{lstlisting}

\begin{lstlisting}[style=output]
(58293, 8)
\end{lstlisting}

The argument \passthrough{\lstinline!n=n\_realinc!} means that the
bootstrapped sample has the same size as the original.

\passthrough{\lstinline!replace=True!} means that sampling is done with
replacement; that is, the same row can be chosen more than once. To see
how many times each row appears in the bootstrapped sample, we can use
\passthrough{\lstinline!value\_counts!} and the
\passthrough{\lstinline!\_ID!} column, which contains a unique
identifier for each respondent.

\begin{lstlisting}[language=Python,style=source]
repeats = bootstrapped['ID_'].value_counts()
repeats.head()
\end{lstlisting}

\begin{tabular}{lr}
\toprule
{} &  ID\_ \\
\midrule
1464 &   46 \\
832  &   46 \\
1243 &   45 \\
700  &   45 \\
1424 &   45 \\
\bottomrule
\end{tabular}

Since some rows appear many times, other rows don't appear at all. To
see how many, we can use \passthrough{\lstinline!set!} subtraction to
count the values of \passthrough{\lstinline!ID\_!} that appear in the
original dataset but not the bootstrapped sample.

\begin{lstlisting}[language=Python,style=source]
unused = set(gss['ID_']) - set(bootstrapped['ID_'])
len(unused)
\end{lstlisting}

\begin{lstlisting}[style=output]
536
\end{lstlisting}

Now we can use bootstrapping to generate a sampling distribution. For
example, the following function takes a
\passthrough{\lstinline!DataFrame!}, generates a bootstrapped sample,
and returns the average income.

\begin{lstlisting}[language=Python,style=source]
def bootstrap_mean(df, varname):
    bootstrapped = df.sample(n=len(df), replace=True)
    return bootstrapped[varname].mean()
\end{lstlisting}

If we run it many times, we get a sample from the sampling distribution
of the mean.

\begin{lstlisting}[language=Python,style=source]
t3 = [bootstrap_mean(gss, 'REALINC')
      for i in range(1001)]
\end{lstlisting}

Here's a summary of the results, compared to the results based on the
normal model.

\begin{lstlisting}[language=Python,style=source]
summary3 = summarize(t3)
table = pd.concat([summary1, summary3])
table.index=['normal model', 'bootstrapping']
table
\end{lstlisting}

\begin{tabular}{lrrl}
\toprule
{} &  Estimate &      SE &                  CI90 \\
\midrule
normal model  &  31741.50 &  118.30 &    [31542.9, 31934.5] \\
bootstrapping &  31733.17 &  126.22 &  [31532.82, 31947.12] \\
\bottomrule
\end{tabular}

The results from bootstrap sampling are consistent with the results
based on the normal model. Now let's see what happens when we estimate
the 10th percentile.

The following function generates a bootstrapped sample and returns the
10th percentile. Instead of \passthrough{\lstinline!percentile!} from
Numpy, it uses \passthrough{\lstinline!quantile!} from Pandas, which
drops \passthrough{\lstinline!NaN!} values. The parameter of
\passthrough{\lstinline!quantile!} is a probability between 0 and 1,
rather than a percentage between 0 and 100.

\begin{lstlisting}[language=Python,style=source]
def bootstrap_percentile(df):
    bootstrapped = df.sample(n=len(df), replace=True)
    return bootstrapped['REALINC'].quantile(0.1)
\end{lstlisting}

We can use it to generate a sample from the sampling distribution of the
10th percentile.

\begin{lstlisting}[language=Python,style=source]
t4 = [bootstrap_percentile(gss)
      for i in range(1001)]
\end{lstlisting}

Here are the results along with the results based on the normal model.

\begin{lstlisting}[language=Python,style=source]
summary4 = summarize(t4)
table = pd.concat([summary2, summary4])
table.index=['normal model', 'bootstrapping']
table
\end{lstlisting}

\begin{tabular}{lrrl}
\toprule
{} &  Estimate &      SE &                 CI90 \\
\midrule
normal model  &  -6101.04 &  203.16 &  [-6421.5, -5749.55] \\
bootstrapping &   5646.98 &   86.05 &     [5512.5, 5806.0] \\
\bottomrule
\end{tabular}

The mean of the sampling distribution is consistent with the 10th
percentile of the data, which is \$5631. So the results from
bootstrapping are more sensible than the results based on the normal
model.

In general, bootstrapping is robust; that is, it works well with many
different distributions and many different statistics. However, at the
end of the chapter, we'll see one example where it fails.

\hypertarget{bigger-data}{%
\section{Bigger Data}\label{bigger-data}}

As sample size increases, errors due to random sampling get smaller. To
demonstrate this effect, I'll use data from the Behavioral Risk Factor
Surveillance System (BRFSS).

In previous chapters, we used BRFSS data to explore the relationship
between height and weight, and the relationship between income and
vegetable consumption. In this section, we'll use it to estimate the
average height for men in the United States.

First, let's read the 2019 data, which I have stored in an HDF file.

\begin{lstlisting}[language=Python,style=source]
import pandas as pd

brfss = pd.read_hdf('brfss.hdf', 'brfss')
brfss.shape
\end{lstlisting}

\begin{lstlisting}[style=output]
(418268, 9)
\end{lstlisting}

This dataset contains 418 268 rows, one for each respondent, and 11
columns, one for each variable I selected. Here are the first few rows.

\begin{lstlisting}[language=Python,style=source]
brfss.head()
\end{lstlisting}

\begin{tabular}{lrrrrrrrrr}
\toprule
{} &       SEQNO &   HTM4 &  WTKG3 &  \_SEX &  \_AGEG5YR &  \_VEGESU1 &  \_INCOMG &      \_LLCPWT &   AGE \\
\midrule
0 &  2019000001 &  157.0 &  69.85 &     2 &      13.0 &     114.0 &        2 &   135.304080 &  82.0 \\
1 &  2019000002 &  163.0 &  48.99 &     2 &      11.0 &     121.0 &        3 &  1454.882220 &  72.0 \\
2 &  2019000003 &  165.0 &  86.18 &     2 &      10.0 &     164.0 &        5 &   215.576852 &  67.0 \\
3 &  2019000004 &  165.0 &  55.34 &     2 &      13.0 &       NaN &        4 &   261.282838 &  82.0 \\
4 &  2019000005 &  152.0 &  49.90 &     2 &      13.0 &     178.0 &        9 &   535.270103 &  82.0 \\
\bottomrule
\end{tabular}

Here's a Boolean \passthrough{\lstinline!Series!} that's
\passthrough{\lstinline!True!} for male respondents.

\begin{lstlisting}[language=Python,style=source]
male = (brfss['_SEX'] == 1)
male.sum()
\end{lstlisting}

\begin{lstlisting}[style=output]
189835
\end{lstlisting}

The column \passthrough{\lstinline!HTM4!} contains their heights in
centimeters.

\begin{lstlisting}[language=Python,style=source]
height = brfss['HTM4']
\end{lstlisting}

We can use \passthrough{\lstinline!notna!} and
\passthrough{\lstinline!sum!} to count the number of rows with valid
height data.

\begin{lstlisting}[language=Python,style=source]
n_height = height[male].notna().sum()
n_height
\end{lstlisting}

\begin{lstlisting}[style=output]
182269
\end{lstlisting}

There are 182 269 male respondents with known heights. Here is the mean
and standard deviation of these values.

\begin{lstlisting}[language=Python,style=source]
mean_height = height[male].mean()
std_height = height[male].std()
mean_height, std_height
\end{lstlisting}

\begin{lstlisting}[style=output]
(178.0768644146838, 7.966455313471575)
\end{lstlisting}

The average height for men in the U.S. is about 178.1 cm; the standard
deviation is about 8 cm. We can use bootstrapping to generate values
from the sampling distribution of the mean. To reduce computation time,
I reduced the number of iterations to 201.

\begin{lstlisting}[language=Python,style=source]
t5 = [bootstrap_mean(brfss[male], 'HTM4')
      for i in range(201)]

summarize(t5, digits=3)
\end{lstlisting}

\begin{tabular}{lrrl}
\toprule
{} &  Estimate &     SE &                CI90 \\
\midrule
{} &   178.076 &  0.019 &  [178.047, 178.107] \\
\bottomrule
\end{tabular}

Because the sample size is so large, the standard error is small and the
confidence interval is narrow. This result suggests that our estimate is
very precise, which is true in the sense that the error due to random
sampling is small.

But there are other sources of error. For example, the heights and
weights in this dataset are based on self-reports, so they are
vulnerable to \textbf{social desirability bias} (see
\url{https://en.wikipedia.org/wiki/Social-desirability_bias}). It's also
possible that there are errors in recording the data. In a previous year
of the BRFSS, there are a suspicious number of heights recorded as 60 or
61 centimeters. I suspect that many of them are six feet, or six feet
and one inch, and something went wrong in the preparation of the data.

And that brings us to the first point of this example:

\begin{quote}
With large sample sizes, error due to random sampling is small, but with
real-world data, that usually means that there are other sources of
error that are bigger. So we can't be sure that the estimate is
accurate.
\end{quote}

In fact, there is another source of error in this example that we have
not taken into account: oversampling.

\hypertarget{weighted-bootstrapping}{%
\section{Weighted Bootstrapping}\label{weighted-bootstrapping}}

By design, the BRFSS oversamples some groups; that is, people in some
groups are more likely than others to appear in the sample. If people in
these groups are taller than others on average, or shorter, our
estimated mean would not be accurate.

We encountered this issue in Chapter 7, where we used data from the
National Survey of Family Growth (NSFG) to compute the average birth
weight for babies in the United States. In that example, we corrected
for oversampling by computing a weighted mean.

In this example, we will use a different method, \textbf{weighted
bootstrapping}, to estimate the mean and compute a confidence interval.

The BRFSS dataset includes a column, \passthrough{\lstinline!\_LLCPWT!},
that contains sampling weights. The sampling weight for each respondent
is the number of people in the population they represent. People in
oversampled groups have lower sampling weights; people in undersampled
groups have higher sampling weights. Here's what the range of values
looks like.

\begin{lstlisting}[language=Python,style=source]
brfss['_LLCPWT'].describe()
\end{lstlisting}

\begin{tabular}{lr}
\toprule
{} &        \_LLCPWT \\
\midrule
count &  418268.000000 \\
mean  &     603.513276 \\
std   &    1082.430311 \\
min   &       1.016173 \\
25\%   &     111.160528 \\
50\%   &     272.869258 \\
75\%   &     654.211787 \\
max   &   42066.730900 \\
\bottomrule
\end{tabular}

The lowest sampling weight is about 1; the largest is 42 066. So that's
a very wide range!

We can take these weights into account by passing them as an argument to
\passthrough{\lstinline!sample!}. That way, the probability that any row
is selected is proportional to its sampling weight.

\begin{lstlisting}[language=Python,style=source]
n = len(brfss)
bootstrapped = brfss.sample(n=n, replace=True, weights='_LLCPWT')
\end{lstlisting}

As we saw with unweighted bootstrapping, the same row can appear more
than once. To see how many times, we can use
\passthrough{\lstinline!value\_counts!} and the
\passthrough{\lstinline!SEQNO!} column, which contains a unique
identifier for each respondent.

\begin{lstlisting}[language=Python,style=source]
repeats = bootstrapped['SEQNO'].value_counts()
repeats.head()
\end{lstlisting}

\begin{tabular}{lr}
\toprule
{} &  SEQNO \\
\midrule
2019000108 &    139 \\
2019000851 &    119 \\
2019006167 &    118 \\
2019000047 &    116 \\
2019000011 &    115 \\
\bottomrule
\end{tabular}

Some rows appear more than 100 times. Most likely, these are the rows
with the highest sampling rates, which correspond to people from
undersampled groups.

To see how many rows don't appear at all, we can use
\passthrough{\lstinline!set!} subtraction to count the values of
\passthrough{\lstinline!SEQNO!} that appear in the original dataset but
not the sample.

\begin{lstlisting}[language=Python,style=source]
unused = set(brfss['SEQNO']) - set(bootstrapped['SEQNO'])
len(unused)
\end{lstlisting}

\begin{lstlisting}[style=output]
826
\end{lstlisting}

There are several hundred rows that don't appear in this sample, but
they are not dropped altogether; when we repeat this process, they will
appear in other samples.

We can use weighted bootstrapping to generate values from the sampling
distribution of the mean. The following function uses
\passthrough{\lstinline!sample!} and the
\passthrough{\lstinline!\_LLCPWT!} column to generate a bootstrapped
sample, then returns the average height.

\begin{lstlisting}[language=Python,style=source]
def weighted_bootstrap_mean(df):
    n = len(df)
    sample = df.sample(n=n, replace=True, weights='_LLCPWT')
    return sample['HTM4'].mean()
\end{lstlisting}

I'll test this function with a \passthrough{\lstinline!DataFrame!} that
contains only male respondents. If we run it once, we get a random value
from the sampling distribution of the weighted mean.

\begin{lstlisting}[language=Python,style=source]
male_df = brfss[male]
weighted_bootstrap_mean(male_df)
\end{lstlisting}

\begin{lstlisting}[style=output]
177.6302436908912
\end{lstlisting}

If we run it many times, we get a random sample from the sampling
distribution.

\begin{lstlisting}[language=Python,style=source]
t6 = [weighted_bootstrap_mean(male_df)
      for i in range(201)]

summarize(t6, digits=3)
\end{lstlisting}

\begin{tabular}{lrrl}
\toprule
{} &  Estimate &     SE &                CI90 \\
\midrule
{} &   177.638 &  0.019 &  [177.604, 177.668] \\
\bottomrule
\end{tabular}

The mean of the sampling distribution estimates the average height for
men in the U.S., corrected for oversampling. If we compare it to the
unweighted mean we computed, it is a little lower.

\begin{lstlisting}[language=Python,style=source]
print(np.mean(t6), mean_height)
\end{lstlisting}

\begin{lstlisting}[style=output]
177.63797698644524 178.0768644146838
\end{lstlisting}

So it seems like people in the oversampled groups are taller than
others, on average, by enough to bring the unweighted mean up by about
half a centimeter.

The difference between the weighted and unweighted means is bigger than
the width of the confidence interval. So in this example the error if we
fail to correct for oversampling is bigger than variability due to
random sampling.

\hypertarget{correlation-and-regression}{%
\section{Correlation and Regression}\label{correlation-and-regression}}

Bootstrap resampling can be used to estimate other statistics and their
sampling distributions. For example, in Chapter 9 we computed the
correlation between height and weight, which is about 0.48.

\begin{lstlisting}[language=Python,style=source]
var1, var2 = 'HTM4', 'WTKG3'
corr = brfss[var1].corr(brfss[var2])
corr
\end{lstlisting}

\begin{lstlisting}[style=output]
0.47715146283881427
\end{lstlisting}

That correlation does not take into account oversampling. We can correct
it with this function, which generates a weighted bootstrapped sample
and uses it to compute the correlation of the columns with names
\passthrough{\lstinline!var1!} and \passthrough{\lstinline!var2!}.

\begin{lstlisting}[language=Python,style=source]
def weighted_bootstrap_corr(df, var1, var2):
    n = len(df)
    sample = df.sample(n=n, replace=True, weights='_LLCPWT')
    corr = sample[var1].corr(sample[var2])
    return corr
\end{lstlisting}

\textbf{Exercise:} Use this function to draw 101 values from the
sampling distribution of the correlation between height and weight. What
is the mean of these values? Is it substantially different from the
correlation we computed without correcting for oversampling? Compute the
standard error and 90\% confidence interval for the estimated
correlation.

\textbf{Exercise:} In Chapter 9 we also computed the slope of the
regression line for weight as a function of height. Here's the result
with with 2019 data.

\begin{lstlisting}[language=Python,style=source]
from scipy.stats import linregress

subset = brfss.dropna(subset=['WTKG3', 'HTM4'])
res = linregress(subset['HTM4'], subset['WTKG3'])
res.slope
\end{lstlisting}

\begin{lstlisting}[style=output]
0.932929241334935
\end{lstlisting}

The estimated slope is 0.93 kg/cm, which means that we expect someone 1
cm taller than average to be about 0.93 kg heavier than average.

Write a function called
\passthrough{\lstinline!weighted\_bootstrap\_slope!} that takes a
\passthrough{\lstinline!DataFrame!}, generates a weighted bootstrapped
sample, runs \passthrough{\lstinline!linregress!} with height and
weight, and returns the slope of the regression line.

Run it 101 times and collect the results. Use the sampling distribution
to compute the mean of the slope estimates, standard error, and a 90\%
confidence interval.

\hypertarget{limitations-of-bootstrapping}{%
\section{Limitations of
Bootstrapping}\label{limitations-of-bootstrapping}}

One limitation of bootstrapping is that it can be computationally
expensive. With small datasets, it is usually fast enough that we can
generate 1000 values from the sampling distribution, which means that we
can compute standard errors and confidence intervals precisely. With
larger datasets, we can cut the computation time by generating fewer
values. With 100-200 values, the standard errors we get are usually
precise enough, but the bounds of the confidence intervals might be
noisier.

The other limitation, which can be more problematic, is that bootstrap
sampling does not work well with datasets that contain a small number of
different values. To demonstrate, I'll select data from the GSS for one
year, 2018:

\begin{lstlisting}[language=Python,style=source]
gss2018 = gss[gss['YEAR']==2018]
\end{lstlisting}

And I'll use bootstrapping to generate values from the sampling
distribution of income.

\begin{lstlisting}[language=Python,style=source]
t9 = [bootstrap_percentile(gss2018)
      for i in range(1001)]
\end{lstlisting}

Here are the results.

\begin{lstlisting}[language=Python,style=source]
summary9 = summarize(t9)
summary9
\end{lstlisting}

\begin{tabular}{lrrl}
\toprule
{} &  Estimate &      SE &              CI90 \\
\midrule
{} &   5169.52 &  254.68 &  [5107.5, 6129.0] \\
\bottomrule
\end{tabular}

The mean of the sampling distribution and the standard error look
plausible at first glance, but the width of the confidence interval is
0, which suggests that something has gone wrong!

The problem is that \passthrough{\lstinline!REALINC!} is not really a
numerical variable; it is a categorical variable in disguise. Using
\passthrough{\lstinline!value\_counts!}, we can see that there are only
26 distinct values in this column.

\begin{lstlisting}[language=Python,style=source]
len(gss2018['REALINC'].value_counts())
\end{lstlisting}

\begin{lstlisting}[style=output]
26
\end{lstlisting}

The reason is that the GSS does not ask respondents to report their
incomes. Instead, it gives them a list of ranges and asks them to pick
the range their income falls in. The ranges are described in the
documentation of the related variable
\href{https://gssdataexplorer.norc.org/variables/104/vshow}{\passthrough{\lstinline!INCOME!}}.

Then GSS analysts compute the midpoint of each range and convert to 1986
dollars by adjusting for inflation. Details of the methodology are in
available from
\url{https://gss.norc.org/Documents/reports/methodological-reports/MR101\%20Getting\%20the\%20Most\%20Out\%20of\%20the\%20GSS\%20Income\%20Measures.pdf}.

As a result, there are only 26 distinct values in
\passthrough{\lstinline!REALINC!}. When we generate a bootstrapped
sample and compute the 10th percentile, we get a small subset of them.
Here are the values that appear in our sample:

\begin{lstlisting}[language=Python,style=source]
pd.Series(t9).value_counts().sort_index()
\end{lstlisting}

\begin{tabular}{lr}
\toprule
{} &    0 \\
\midrule
5107.5 &  944 \\
5561.5 &    1 \\
5675.0 &    1 \\
5788.5 &    2 \\
6015.5 &    1 \\
6129.0 &    2 \\
6242.5 &   50 \\
\bottomrule
\end{tabular}

There are only 5 different values, and one of them appears more than
95\% of the time. When we compute a 90\% confidence interval, this value
is both the 5th and the 95th percentile.

Bootstrapping works well for most distributions and most statistics; the
one thing it can't handle is lack of diversity in the data. However,
even this problem can be solved. The fundamental cause is that the data
have been discretized excessively, so the solution is to smooth it.
Jittering is one option. Another is to use kernel density estimation
(KDE).

\hypertarget{resampling-with-kde}{%
\section{Resampling with KDE}\label{resampling-with-kde}}

We have used KDE several times to estimate and plot a probability
density based on a sample. We can also use it to smooth data that have
been discretized.

In Chapter 7 we saw that the distribution of income is well modeled by a
lognormal distribution, so if we take the log of income, it is well
modeled by a normal distribution. Here are the logarithms of the income
data.

\begin{lstlisting}[language=Python,style=source]
log_realinc = np.log10(gss2018['REALINC'].dropna())
\end{lstlisting}

And here's what the estimated density looks like.

\begin{lstlisting}[language=Python,style=source]
import seaborn as sns

sns.kdeplot(log_realinc)

plt.xlabel('Income (log10 1986 dollars)')
plt.ylabel('Probability density')
plt.title('Estimated distribution of income');
\end{lstlisting}

\begin{center}
\includegraphics[scale=0.75]{12_bootstrap_files/12_bootstrap_115_0.pdf}
\end{center}

To draw samples from this distribution, we'll use a Scipy function
called \passthrough{\lstinline!gaussian\_kde!}, which takes the data and
returns an object that represents the estimated density.

\begin{lstlisting}[language=Python,style=source]
from scipy.stats import gaussian_kde

kde = gaussian_kde(log_realinc)
\end{lstlisting}

\passthrough{\lstinline!kde!} provides a method called
\passthrough{\lstinline!resample!} that draws random values from the
estimated density. As we've done in previous examples, we'll generate a
resampled dataset with the same size as the original.

\begin{lstlisting}[language=Python,style=source]
n = gss2018['REALINC'].notna().sum()
n
\end{lstlisting}

\begin{lstlisting}[style=output]
2152
\end{lstlisting}

Now we can draw a sample, compute the 10th percentile, and convert from
a logarithm to a dollar value.

\begin{lstlisting}[language=Python,style=source]
sample = kde.resample(n)
10 ** np.percentile(sample, 10)
\end{lstlisting}

\begin{lstlisting}[style=output]
4745.466287228786
\end{lstlisting}

The result is a random value from the sampling distribution of the 10th
percentile. The following function encapsulates these steps.

\begin{lstlisting}[language=Python,style=source]
def resample_kde_percentile(kde):
    sample = kde.resample(kde.n)
    return 10 ** np.percentile(sample, 10)
\end{lstlisting}

Now we can generate a sample from the sampling distribution.

\begin{lstlisting}[language=Python,style=source]
t10 = [resample_kde_percentile(kde)
       for i in range(1000)]

summary10 = summarize(t10)
\end{lstlisting}

The following table compares the results with KDE resampling to the
previous result with bootstrapping.

\begin{lstlisting}[language=Python,style=source]
table = pd.concat([summary9, summary10])
table.index=['bootstrapping', 'KDE resampling']
table
\end{lstlisting}

\begin{tabular}{lrrl}
\toprule
{} &  Estimate &      SE &               CI90 \\
\midrule
bootstrapping  &   5169.52 &  254.68 &   [5107.5, 6129.0] \\
KDE resampling &   5087.77 &  248.81 &  [4699.39, 5507.4] \\
\bottomrule
\end{tabular}

The means and standard errors are about the same with either method. The
difference is that the confidence interval we get from KDE resampling is
much more reasonable.

\hypertarget{summary}{%
\section{Summary}\label{summary}}

There are ten examples in this chapter so far; let's review them:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\item
  First we used resampling based on a normal model to estimate average
  family income in the GSS and compute a confidence interval.
\item
  Then we used the same method to estimate the 10th percentile of
  income, and we found that the width of the confidence interval was 0.
  The problem is that the normal model does not fit the distribution of
  income.
\item
  To solve this problem, we switched to bootstrap sampling. First we
  estimated average family income and confirmed that the results are
  consistent with the results based on the normal model.
\item
  Then we used bootstrapping to estimate the 10th percentile of income.
  The results are much more plausible.
\item
  Next we used data from the BRFSS to estimate the average height of men
  in the U.S. Since this dataset is large, the confidence interval is
  very small. That means that the estimate is precise, in the sense that
  variability due to random sampling is small, but we don't know whether
  it is accurate, because there are other possible sources of error.
\item
  One of those sources of error is oversampling; that is, some people
  are more likely to appear in the sample than others. In the BFRSS,
  each respondent has a sampling weight that indicates how many people
  in the population they represent. We used these weighted to do
  weighted bootstrapping, and found that the error due to oversampling
  is larger than the variability due to random sampling.
\item
  In one exercise you used weighted bootstrapping to estimate the
  correlation of height and weight and compute a confidence interval.
\item
  In another exercise you estimated the slope of a regression line and
  computed a confidence interval.
\item
  Finally, I demonstrated a problem with bootstrap sampling when the
  dataset has only a few different values,
\item
  And presented a solution using KDE to smooth the data and draw samples
  from an estimated distribution.
\end{enumerate}

In the exercise below, you can work on one more example.

\textbf{Exercise:} In Chapter 10 we used logistic regression to model
support for legalizing marijuana as a function of age, sex, and
education level. Going back to that example, let's explore changes in
support over time and generate predictions for the next decade.

To prepare the data for logistic regression, we have to recode the
\passthrough{\lstinline!GRASS!} column so \passthrough{\lstinline!1!}
means in favor of legalization and \passthrough{\lstinline!0!} means not
in favor.

\begin{lstlisting}[language=Python,style=source]
gss['GRASS'].replace(2, 0, inplace=True)
gss['GRASS'].value_counts()
\end{lstlisting}

\begin{tabular}{lr}
\toprule
{} &  GRASS \\
\midrule
0.0 &  25662 \\
1.0 &  11884 \\
\bottomrule
\end{tabular}

As explanatory variables we'll use \passthrough{\lstinline!YEAR!} and
\passthrough{\lstinline!YEAR!} squared, which I'll store in a column
called \passthrough{\lstinline!YEAR2!}.

\begin{lstlisting}[language=Python,style=source]
gss['YEAR2'] = (gss['YEAR']-1990) ** 2
\end{lstlisting}

Now we can run the model like this:

\begin{lstlisting}[language=Python,style=source]
import statsmodels.formula.api as smf

formula = 'GRASS ~ YEAR + YEAR2'
results = smf.logit(formula, data=gss).fit(disp=False)
\end{lstlisting}

To generate predictions, I'll create a
\passthrough{\lstinline!DataFrame!} with a range of values of
\passthrough{\lstinline!YEAR!} up to 2030, and corresponding values of
\passthrough{\lstinline!YEAR2!}.

\begin{lstlisting}[language=Python,style=source]
years = np.linspace(1972, 2030)
df_pred = pd.DataFrame()
df_pred['YEAR'] = years
df_pred['YEAR2'] = (df_pred['YEAR']-1990)**2

pred = results.predict(df_pred)
\end{lstlisting}

I'll use \passthrough{\lstinline!groupby!} to compute the fraction of
respondents in favor of legalization during each year.

\begin{lstlisting}[language=Python,style=source]
grass_by_year = gss.groupby('YEAR')['GRASS'].mean()
\end{lstlisting}

The following function plots the data and decorates the axes.

\begin{lstlisting}[language=Python,style=source]
def plot_data():
    grass_by_year.plot(style='o', alpha=0.5, label='data')
    plt.xlabel('Year')
    plt.ylabel('Fraction in favor')
    plt.title('Support for legalization of marijuana')
    plt.legend(loc='upper left');
\end{lstlisting}

Here's what the predictions look like, plotted along with the data.

\begin{lstlisting}[language=Python,style=source]
plt.plot(years, pred, label='logistic model', color='gray')
plot_data()
\end{lstlisting}

\begin{center}
\includegraphics[scale=0.75]{12_bootstrap_files/12_bootstrap_144_0.pdf}
\end{center}

The model fits past data reasonably well and makes plausible predictions
for the next decade, although we can never be sure that trends like this
will continue.

This way of representing the results could be misleading because it does
not show our uncertainty about the predictions. Random sampling is just
one source of uncertainty among many, and for this kind of prediction it
is certainly not the biggest. But it is the easiest to quantify, so
let's do it, if only as an exercise.

Write a function called
\passthrough{\lstinline!bootstrap\_regression\_line!} that takes a
\passthrough{\lstinline!DataFrame!} as a parameter, uses
\passthrough{\lstinline!sample!} to resample the rows, runs the logistic
regression model, generates predictions for the rows in
\passthrough{\lstinline!df\_pred!}, and returns the predictions.

Call this function 101 times and save the results as a list of
\passthrough{\lstinline!Series!} objects. To visualize the results, you
have two options:

\begin{enumerate}
\def\labelenumi{\arabic{enumi}.}
\item
  Loop through the list and plot each prediction using a gray line with
  a low value of \passthrough{\lstinline!alpha!}. The overlapping lines
  will form a region showing the range of uncertainty over time.
\item
  Pass the list of \passthrough{\lstinline!Series!} to
  \passthrough{\lstinline!np.percentile!} with the argument
  \passthrough{\lstinline!axis=0!} to compute the 5th and 95th
  percentile in each column. Plot these percentiles as two lines, or use
  \passthrough{\lstinline!plt.fill\_between!} to plot a shaded region
  between them.
\end{enumerate}


\documentclass{article}

\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}

% Main maths packages
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathrsfs}

% Required to declare unnumbered theorems. It enhances the style of the theorems as well.
% Required to use proofs.
\usepackage{amsthm}

% The syntax is \newtheorem{envname}[theorem name to share the numbering]{caption}[section type/theorem name to number it accordingly].
\newtheorem{theorem}{Theorem}[subsection]
\newtheorem{corollary}{Corollary}[theorem]
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem*{definition}{Definition}

\begin{document}

\section{Theorems, corollaries and lemmas}

\subsection{Declare theorems}

First, the theorem must be declared in the preamble. It can be used in the document afterwards. The name of the author can be specified between brackets.

\begin{theorem}[Pythagore]
	\label{pythagore}
	the square of the hypotenuse is equal to the sum of the squares of the other two sides.
\end{theorem}

\begin{corollary}
	In any right triangle, the hypotenuse is greater than any one of the other sides, but less than their sum.
\end{corollary}

\begin{theorem}
	Another theorem...
\end{theorem}

\begin{corollary}
	Another corollary...
\end{corollary}

\begin{lemma}
	A lemma, which shares the same numbering as the theorem.
\end{lemma}

A theorem can be referenced easily, such as \ref{pythagore}.

\subsection{Unnumbered theorems}

\begin{definition}
	This is an unnumbered definition.
\end{definition}

\subsection{Proofs}

\begin{proof}
	The two large squares shown in the figure each contain four identical triangles, and the only difference between the two large squares is that the triangles are arranged differently. Therefore, the white space within each of the two large squares must have equal area. Equating the area of the white space yields the Pythagorean Theorem.
\end{proof}

Note the white square at the end of the proof, it stands for Q.E.D. The abbreviation "Q.E.D." is written to indicate the end of a proof. This abbreviation stands for "Quod Erat Demonstrandum", which is Latin for "that which was to be demonstrated". A more common alternative is to use a square or a rectangle.

\end{document}


\subsection{Computing the moments of the mRNA and protein distributions}
\label{sec_moments}

Finding analytical solutions to chemical master equations is often fraught with
difficulty. An alternative approach is to to approximate the distribution. One
such scheme of approximation, the maximum entropy principle, makes use of the
moments of the distribution to approximate the full distribution. In this
section we will demonstrate an iterative algorithm to compute the mRNA and
protein distribution moments.

The kinetic model for the simple repression motif depicted in
\fref{fig2_minimal_model}(A) consists of an infinite system of ODEs for each
possible pair of mRNA and protein copy number, $(m, p)$. To compute any moment
of the distribution, we define a vector
\begin{equation}
	\ee{\bb{m^x p^y}} \equiv (\ee{m^x p^y}_A, \ee{m^x p^y}_I, \ee{m^x p^y}_R)^T,
\end{equation}
where $\ee{m^x p^y}_S$ is the expected value of $m^x p^y$ in state $S \in \{A,
I, R\}$ for $x, y \in \mathbb{N}$. In other words, just as we defined the
vector $\PP(m, p)$, here we define a vector to collect the expected value of
each of the promoter states. By definition, any of these moments $\ee{m^x
p^y}_S$ can be computed as
\begin{equation}
  \ee{m^x p^y}_S \equiv \sum_{m=0}^\infty \sum_{p=0}^\infty m^x p^y P_S(m, p).
  \label{eq_mom_def}
\end{equation}
Summing over all possible values for $m$ and $p$ in \eref{eq_cme_matrix}
results in an ODE for any moment of the distribution of the form (See
\siref{supp_moments} for full derivation)
\begin{equation}
  \begin{aligned}
    \dt{\bb{\ee{m^x p^y}}} &=
    \Km \bb{\ee{m^x p^y}}\\
    &+ \Rm \bb{\ee{p^y \left[ (m + 1)^x -m^x \right]}}
     + \Gm \bb{\ee{m p^y \left[ (m - 1)^x - m^x \right]}}\\
    &+ \Rp \bb{\ee{m^{(x + 1)} \left[ (p + 1)^y - p^y \right]}}
     + \Gp \bb{\ee{m^x p \left[ (p - 1)^y - p^y \right]}}.
    \label{eq_gral_mom}
  \end{aligned}
\end{equation}

Given that all transitions in our stochastic model are first order reactions,
\eref{eq_gral_mom} has no moment-closure problem \cite{Voliotis2014a}. This
means that the dynamical equation for a given moment only depends on lower
moments (See \siref{supp_moments} for full proof). This feature of our model
implies, for example, that the second moment of the protein distribution
$\ee{p^2}$ depends only on the first two moments of the mRNA distribution
$\ee{m}$ and $\ee{m^2}$, the first protein moment $\ee{p}$, and the
cross-correlation term $\ee{mp}$. We can therefore define $\bmu$ to be a vector
containing all moments up to $\bb{\ee{m^x p^y}}$ for all promoter states,
\begin{equation}
\bmu = \left[ \bb{\ee{m^0 p^0}},
							\bb{\ee{m^1 p^0}},
							\ldots, \bb{\ee{m^x p^y}} \right]^T.
\end{equation}
Explicitly for the three-state promoter model depicted in
\fref{fig2_minimal_model}(A) this vector takes the form
\begin{equation}
	\bmu = \left[ \ee{m^0 p^0}_A, \ee{m^0 p^0}_I, \ee{m^0 p^0}_R,
	\ldots, \ee{m^x p^y}_A, \ee{m^x p^y}_I, \ee{m^x p^y}_R \right]^T.
\end{equation}

Given this definition we can compute the general moment dynamics as
\begin{equation}
\dt{\bmu} = \bb{A} \bmu,
\label{eq_mom_dynamics}
\end{equation}
where $\bb{A}$ is a square matrix that contains all the numerical coefficients
that relate each of the moments. We can then use \eref{eq_gral_mom} to build
matrix $\bb{A}$ by iteratively substituting values for the exponents $x$ and
$y$ up to a specified value. In the next section, we will use
\eref{eq_mom_dynamics} to numerically integrate the dynamical equations for our
moments of interest as cells progress through the cell cycle. We will then use
the value of the moments of the distribution to approximate the full gene
expression distribution. This method is computationally more efficient than
trying to numerically integrate the infinite set of equations describing the
full probability distribution $\bb{P}(m, p)$, or using a stochastic algorithm
to sample from the distribution.


\section{Boolean network}
\label{sect:background-boolean-network}

A \textit{(combinational) Boolean network} is a directed acyclic graph $G=(V,E)$,
with a set $V$ of vertices and a set $E\subseteq V \times V$ of edges.
Two non-empty disjoint subsets $V_I$ and $V_O$ of $V$ are identified:
a vertex $v \in V_I$ (resp. $V_O$) is referred to as a \textit{primary input} (PI) (resp. \textit{primary output} (PO)).
Each vertex $v \in V$ is associated with a Boolean variable $b_v$.
Each vertex $v \in V \setminus V_I$ is associated with a Boolean function $f_v$.
An edge $(u,v)\in E$ indicates $f_v$ refers to $b_u$ as an input variable;
$u$ is called a \textit{fanin} of $v$, and $v$ a \textit{fanout} of $u$.
The valuation of the Boolean variable $b_v$ of vertex $v$ is as follows:
if $v$ is a PI, $b_v$ is given by external signals; otherwise, $b_v$ equals the value of $f_v$.
To ease readability, we will not distinguish a vertex $v$ and its corresponding Boolean variable $b_v$.
We will simply denote $b_v$ with $v$.

Note that a Boolean network can be converted in linear time to a Boolean formula in CNF through Tseitin transformation~\cite{Tseitin1983}.


\chapter{Recursive Methods}

\index{iterative}
\index{recursive}

Up to this point, we've been using \java{while} and \java{for} loops whenever we've needed to repeat something.
Methods that use iteration are called {\bf iterative}.
They are straight-forward, but sometimes there are more elegant solutions.

In this chapter, we explore one of the most magical things that a method can do: invoke {\em itself} to solve a smaller version of the {\em same} problem.
A method that invokes itself is called {\bf recursive}.

\section{Recursive Void Methods}
\label{recursion}

\index{countdown}
%slr:  11/17/19
Consider the following iterative example from Section~\ref{The_while_statement}:

% height = 130 + 11 * num_lines
\begin{trinket} [230] {Blastoff.java}
public class Blastoff {

    public static void main(String[] args) {
       int n = 3;
       while (n > 0) {
          System.out.println(n);
          n = n - 1;
       }
       System.out.println("Blastoff!");
    }
}
\end{trinket}

We can rewrite this using \emph{recursion}.  In the following trinket, the method \java{countdown} calls \emph{itself}:

% height = 130 + 11 * num_lines
\begin{trinket} [280] {RecursiveBlastoff.java}
public class RecursiveBlastoff {

    public static void main(String[] args) {
       countdown(3);
    }

    public static void countdown(int n) {
       if (n == 0) {  //'base case' - ends the sequence of recursive calls
           System.out.println("Blastoff!");
       } else {
           System.out.println(n);
           countdown(n - 1);
       }
    }
}
\end{trinket}

%\begin{code}
%public static void countdown(int n) {
%    if (n == 0) {
%        System.out.println("Blastoff!");
%    } else {
%        System.out.println(n);
%        countdown(n - 1);
%    }
%}
%\end{code}
%slr: end 11/17/19

The name of the method is \java{countdown}; it takes a single integer as a parameter.
If the parameter is zero, it displays the word ``Blastoff''.
Otherwise, it displays the number and then invokes itself, passing \java{n - 1} as the argument.

What happens if we invoke \java{countdown(3)} from \java{main}?

\vspace{-1ex}
\begin{quote}
The execution of \java{countdown} begins with \java{n == 3}, and since \java{n} is not zero, it displays the value 3, and then invokes itself...
\begin{quote}
The execution of \java{countdown} begins with \java{n == 2}, and since \java{n} is not zero, it displays the value 2, and then invokes itself...
\begin{quote}
The execution of \java{countdown} begins with \java{n == 1}, and since \java{n} is not zero, it displays the value 1, and then invokes itself...
\begin{quote}
The execution of \java{countdown} begins with \java{n == 0}, and since \java{n} is zero, it displays the word ``Blastoff!'' and then returns.
\end{quote}
The \java{countdown} that got \java{n == 1} returns.
\end{quote}
The \java{countdown} that got \java{n == 2} returns.
\end{quote}
The \java{countdown} that got \java{n == 3} returns.
\end{quote}
\vspace{-1ex}

And then you're back in \java{main}.
So the total output looks like:

\begin{stdout}
3
2
1
Blastoff!
\end{stdout}

%slr:  11/17/19
The java classes (i.e. programs) \java{Blastoff} and \java{RecursiveBlastoff} produce the same result, but with very different strategies.  It can be shown that any iterative algorithm can be written as a recursive one and conversely.  In practice, some tasks can be performed much more easily with one scheme than the other.
%slr: end 11/17/19

As a second example, we'll rewrite the methods \java{newLine} and \java{threeLine} from Section~\ref{adding_methods}.
Here they are again:

\begin{code}
public static void newLine() {
    System.out.println();
}

public static void threeLine() {
    newLine();
    newLine();
    newLine();
}
\end{code}

\index{newline}

Although these methods work, they would not help if we wanted to display two newlines, or maybe 100.
A more general \textbf{iterative} alternative would be:

\begin{code}
public static void nLines(int n) {
    for(int i = 0; i < n; i++) {
       System.out.println();
    }
}
\end{code}

And a more general \textbf{recursive} alternative would be:

\begin{code}
public static void nLines(int n) {
    if (n == 0) { //'base case' - ends the sequence of recursive calls
       return;
    } else {
       System.out.println();
       nLines(n - 1);
    }
}
\end{code}

This recursive method takes an integer, \java{n}, as a parameter and displays \java{n} newlines.
The structure is similar to the \java{countdown} method in class RecursiveCountdown above.
As long as $n$ is greater than zero, it displays a newline and then invokes itself to display $(n-1)$ additional newlines.
The total number of newlines is $1 + (n - 1)$, which is just what we wanted: $n$.

%slr: 11/17/19
Here is a complete Java trinket class:

% height = 130 + 11 * num_lines
\begin{trinket} [300] {RecursiveNewLines.java}
public class RecursiveNewLines {

    public static void main(String[] args) {
       System.out.println("Here come 3 newlines:");
       nLines(3);
       System.out.println("Goodbye!");
    }

    public static void nLines(int n) {
       if (n == 0) {  //'base case' - ends the sequence of recursive calls
          return;
       } else {
          System.out.println();
          nLines(n - 1);
       }
    }
}
\end{trinket}

\textbf{Section Exercises:}
\begin{enumerate}
\item Modify the trinket so that it prints a character (perhaps a period or an asterisk) on each of the newlines.
\item Modify the trinket so that it produces the same output, but with base case condition \java{n == 1} instead of \java{n == 0}.
\end{enumerate}

%slr:  end 11/17/19

\section{Recursive Stack Diagrams}

\index{stack diagram}
\index{diagram!stack}

In Section~\ref{stack}, we used a stack diagram to represent the state of a program during a method invocation.
The same kind of diagram can make it easier to interpret a recursive method.

Remember that every time a method gets called, Java creates a new frame that contains the method's parameters and variables.
Figure~\ref{fig.stack2} is a stack diagram for \java{countdown}, called with \java{n == 3}.

\begin{figure}[!ht]
\begin{center}
\includegraphics{figs/stack2.pdf}
\caption{Stack diagram for the \java{countdown} program.}
\label{fig.stack2}
\end{center}
\end{figure}

By convention, the frame for \java{main} is at the top, and the stack of other frames grows down.
That way, we can draw stack diagrams on paper without needing to guess how far they will grow.
The frame for \java{main} is empty because \java{main} does not have any variables.
(It has the parameter \java{args}, but since we're not using it, we left it out of the diagram.)

\index{base case}

There are four frames for \java{countdown}, each with a different value for the parameter \java{n}.
The last frame, with \java{n == 0}, is called the {\bf base case}.
It does not make a recursive call, so there are no more frames below it.

\index{StackOverflowError}
\index{exception!StackOverflow}

If there is no base case in a recursive method, or if the base case is never reached, the stack would grow forever---at least in theory.
In practice, the size of the stack is limited.
If you exceed the limit, you get a \java{StackOverflowError}.

For example, here is an example of a recursive method without a base case:
%slr:  11/17/19
%\begin{code}
%public static void forever(String s) {
%    System.out.println(s);
%    forever(s);
%}
%\end{code}

% height = 130 + 11 * num_lines
\begin{trinket} [230] {InfiniteRecursiveMethod.java}
public class InfiniteRecursiveMethod {

    public static void main(String[] args) {
       forever("and again,");
    }

    public static void forever(String s) {
       System.out.println(s);
       forever(s);
    }
}
\end{trinket}

\index{call stack}

Run the program and see what happens. This recursive method repeatedly displays the given string since there is no base case ending the recursion - an infinite recursion.  The program does terminate, however, when the stack overflows and an exception is thrown.

\textbf{Section Exercises:}
\begin{enumerate}
\item Modify the trinket so that it is no longer infinite.
\end{enumerate}
%slr:  end 11/17/19

\section{Value-Returning Methods}
\label{factorial}

%\index{Turing complete}
%\index{language!complete}
%
%\index{Turing, Alan}
%\index{Church, Alonzo}
%
%Now that we have methods that return values, we have a {\bf Turing complete} programming language.
%That means Java can compute anything computable, for any reasonable definition of ``computable''.
%This idea was developed by Alonzo Church and Alan Turing, so it is known as the Church-Turing thesis.
%You can read more about it at \url{https://en.wikipedia.org/wiki/Turing_thesis}.

To give you an idea of what you can do with the tools we have learned, let's look at methods that evaluate recursively-defined mathematical functions which therefor return a value.

A recursive definition is similar to a ``circular'' definition, in the sense that the definition refers to the thing being defined.
Of course, a truly circular definition is not very useful:

\begin{quote}
{\bf recursive:} \\
An adjective used to describe a method that is recursive.
\end{quote}

\index{recursion}

If you saw that definition in the dictionary, you might be annoyed.
Then again, if you search for ``recursion'' on Google, it displays ``Did you mean: recursion'' as an inside joke.
People fall for that link all the time.

\index{factorial}

Many mathematical functions are defined recursively, because that is often the simplest way.
For example, the {\bf factorial} of an integer $n$, which is written $n!$, is defined like this:
%
\begin{eqnarray*}
&&  0! = 1 \\
&&  n! = n \cdot(n-1)!
\end{eqnarray*}
%
Don't confuse the mathematical symbol $!$, which means {\em factorial}, with the Java operator \java{!}, which means {\em not}.
This definition says that \java{factorial(0)} is \java{1}, and \java{factorial(n)} is \java{n * factorial(n - 1)}.

So \java{factorial(3)} is \java{3 * factorial(2)}; \java{factorial(2)} is \java{2 * factorial(1)}; \java{factorial(1)} is \java{1 * factorial(0)}; and \java{factorial(0)} is \java{1}.
Putting it all together, we get \java{3 * 2 * 1 * 1}, which is 6.

If you can formulate a recursive definition of something, you can easily write a Java method to evaluate it.
The first step is to decide what the parameters and return type are.
Since factorial is defined for integers, the method takes an \java{int} as a parameter and returns an \java{int}.
%So here's a good starting place:

\begin{code}
public static int factorial(int n) {
    return 0;  // stub
}
\end{code}

Next, we think about the base case.
If the argument happens to be zero, we return 1.

\begin{code}
public static int factorial(int n) {
    if (n == 0) {
        return 1;
    } else {
        return 0;  // stub
    }
}
\end{code}

Otherwise, and this is the interesting part, we have to make a recursive call to find the factorial of $n-1$, and then multiply it by $n$.

\begin{code}
public static int factorial(int n) {
    if (n == 0) {
        return 1; //base case
    } else {
        int recurse = factorial(n - 1);
        int result = n * recurse;
        return result;
    }
}
\end{code}

To illustrate what is happening, we use the temporary variables \java{recurse} and \java{result}.
In each method call, \java{recurse} stores the factorial of $n - 1$, and \java{result} stores the factorial of $n$.

%slr:  11/19/19
The trinket below is a complete, executable program illustrating a recursive algorithm for computing 3!:

% height = 130 + 11 * num_lines
\begin{trinket} [280] {RecursiveFactorial.java}
public class RecursiveFactorial {

    public static void main(String[] args) {
       System.out.println("3! = " + factorial(3));
    }

    public static int factorial(int n) {
       if (n == 0) {
          return 1; //base case
       } else {
          int recurse = factorial(n - 1);
          int result = n * recurse;
          return result;
       }
    }
}
\end{trinket}
%slr:  end 11/19/19

The flow of execution for this program is similar to \java{countdown} from Section~\ref{recursion}.
If we invoke \java{factorial} with the value 3:

\vspace{-1ex}
\begin{quote}
Since 3 is not zero, we skip the first branch and calculate the factorial of $n-1$...
\begin{quote}
Since 2 is not zero, we skip the first branch and calculate the factorial of $n-1$...
\begin{quote}
Since 1 is not zero, we skip the first branch and calculate the factorial of $n-1$...
\begin{quote}
Since 0 {\em is} zero, we take the first branch and return the value 1 immediately.
% without making any more recursive invocations.
\end{quote}
The return value (1) gets multiplied by \java{n}, which is 1, and the result is returned.
\end{quote}
The return value (1) gets multiplied by \java{n}, which is 2, and the result is returned.
\end{quote}
The return value (2) gets multiplied by \java{n}, which is 3, and the result, 6, is returned to whatever invoked \java{factorial(3)}.
\end{quote}
\vspace{-1ex}

\index{stack diagram}
\index{diagram!stack}

Figure~\ref{fig.stack3} shows what the stack diagram looks like for this sequence of method invocations.
The return values are shown being passed up the stack.
Notice that \java{recurse} and \java{result} do not exist in the last frame, because when \java{n == 0} the code that declares them does not execute.

\begin{figure}[!ht]
\begin{center}
\includegraphics{figs/stack3.pdf}
\caption{Stack diagram for the \java{factorial} method.}
\label{fig.stack3}
\end{center}
\end{figure}

%slr:  11/21/19 omit leap of faith section - incorporate into previous section
%\section{The Leap of Faith}
%\label{leap_of_faith}
%
%\index{leap of faith}
%
%Following the flow of execution is one way to read programs, but it can quickly become overwhelming.
%Another way to understand recursion is the {\bf leap of faith}:
%when you come to a method invocation, instead of following the flow of execution, you {\em assume} that the method works correctly and returns the appropriate value.
%
%In fact, you are already practicing this leap of faith when you use methods in the Java library.
%When you invoke \java{Math.cos} or \java{System.out.println}, you don't think about the implementations of those methods.
%You just assume that they work properly.
%
%The same is true of other methods.
%For example, consider the method from Section~\ref{boolmeth} that determines whether an integer has only one digit:
%
%\begin{code}
%public static boolean isSingleDigit(int x) {
%    return x > -10 && x < 10;
%}
%\end{code}
%
%Once you convince yourself that this method is correct---by examining and testing the code---you can just use the method without ever looking at the implementation again.
%
%Recursive methods are no different.
%When you get to a recursive call, don't think about the flow of execution.
%Instead, {\em assume} that the recursive call produces the desired result.
%
%For example, ``Assuming that I can find the factorial of $n-1$, can I compute the factorial of $n$?''
%Yes you can, by multiplying by $n$.
%slr:  end 11/21/19 omit leap of faith section
Here's an implementation of \java{factorial} without use of the temporary variables \java{recurse} and \java{result}:

\begin{code}
public static int factorial(int n) {
    if (n == 0) {
        return 1;
    } else {
       return n * factorial(n - 1);
    }
}
\end{code}

Notice how similar this version is to the original mathematical definition:
%There is essentially a one-to-one correspondence.
%
\begin{eqnarray*}
&&  0! = 1 \\
&&  n! = n \cdot(n-1)!
\end{eqnarray*}
%
%Of course, it is strange to assume that the method works correctly when you have not finished writing it.
%But that's why it's called a leap of faith!

Following the flow of execution of a recursive method is one way to understand it, but this can become overwhelming.  Another way to understand a recursive method is to read and understand it the same way we read the recursive definition itself.

%\section{Fibonacci Sequence}
\label{fibonacci}

\index{fibonacci}

As another example consider the recursively-defined Fibonacci sequence, a function which has the following definition:
%
\begin{eqnarray*}
&& fibonacci(1) = 1 \\
&& fibonacci(2) = 1 \\
&& fibonacci(n) = fibonacci(n-1) + fibonacci(n-2)
\end{eqnarray*}
%

Notice that each Fibonacci number is the sum of the two preceding Fibonacci numbers.  Thus the terms of the sequence are: 1, 1, 2, 3, 5, 8, 13, 21, ...

We see from the definition of the Fibonacci sequence that the \textbf{base case} is evaluating the function at \java{1} or at \java{2}:

\begin{code}
public static int fibonnacci(int n) {
   if (n == 1 || n == 2) {
      return 1;  //base case(s)
   } else {
      return 0;  //stub for recursive case
   }
}
\end{code}

What about the recursive case?  If n neither 1 nor 2, we want to invoke the recursive method \textbf{twice}, once at \java{n-1} and once at \java{n-2} and add the results per the definition above.  Translated into Java, this function is:

\begin{code}
public static int fibonacci(int n) {
    if (n == 1 || n == 2) {
        return 1;
    } else {
       return fibonacci(n - 1) + fibonacci(n - 2);
    }
}
\end{code}

If you try to follow the flow of execution here, even for small values of \java{n}, your head will explode.  Each call to the method \java{fibonacci} may result in \textbf{two more} invocations of \java{fibonacci}!
But it is clear, looking at the definition, that our implementation is correct once we understand recursive method calls.

Here is a complete trinket implentation for computing a term of the Fibonacci sequence:

% height = 130 + 11 * num_lines
\begin{trinket} [270] {Fibonacci.java}
public class Fibonacci {

    public static void main(String[] args) {
       System.out.printf("5th term of the Fibonacci sequence is %d\n", fibonacci(5));
    }

    public static int fibonacci(int n) {
       if (n == 1 || n == 2) {
          return 1; //base case
       } else {
          return fibonacci(n-1) + fibonacci(n-2);
       }
    }
}
\end{trinket}

Run the trinket and see what you get.

\textbf{Section Exercises:}
\begin{enumerate}
\item Modify the trinket above containing the class \java{RecursiveFactorial} so that it in no longer uses the temporary variables \java{recurse} and \java{result}.  Verify that it works correctly.
\item Modify the trinket above containing the class \java{Fibonacci} to compute and display the first 10 terms of the sequence.
\item Modify the trinket above containing the class \java{Fibonacci} to compute the 50th term of the sequence.  (Be patient!)  What happens?  Why?
\item Modify the trinket above containing the class \java{Fibonacci} to compute the term \textit{iteratively} rather than \textit{recursively}.
\end{enumerate}

\section{Counting Up Recursively}
\label{countup}

The \java{countdown} example in Section~\ref{recursion} has three parts: (1) it checks the base case, (2) it displays something, and (3) it makes a recursive call.
What do you think happens if you reverse steps (2) and (3), making the recursive call {\em before} displaying?  Run the trinket below and see:

%slr:  11/22/19
% height = 130 + 11 * num_lines
\begin{trinket} [280] {CountUp.java}
public class CountUp {

    public static void main(String[] args) {
       countup(3);
    }

    public static void countup(int n) {
        if (n == 0) {
            System.out.println("Blastoff!");
        } else {
            countup(n - 1);
            System.out.println(n);
        }
    }
}
\end{trinket}

%\begin{code}
%public static void countup(int n) {
%    if (n == 0) {
%        System.out.println("Blastoff!");
%    } else {
%        countup(n - 1);
%        System.out.println(n);
%    }
%}
%\end{code}
%slr:  end 11/22/19

The stack diagram is the same as before, and the method is still called $n$ times.
But now the \java{System.out.println} happens just before each recursive call returns.
As a result, it counts {\em up} instead of down:

\begin{stdout}
Blastoff!
1
2
3
\end{stdout}

Keep this in mind for the next example, which displays numbers in binary.

\section{Binary Number System}
\label{binary}

You are probably aware that computers can only store 1's and 0's.
That's because processors and memory are made up of billions of tiny on-off switches.

The value 1 means a switch is on; the value 0 means a switch is off.
All types of data, whether integer, floating-point, text, audio, video, or something else, are represented by 1's and 0's.

\index{binary}

Fortunately, we can represent any integer as a {\bf binary} number.
The following table shows the first eight numbers in binary and decimal.

\begin{table}[!ht]
\begin{center}
\begin{tabular}{|c|c|}
\hline
Binary & Decimal \\
\hline
0 & 0 \\
\hline
1 & 1 \\
\hline
10 & 2 \\
\hline
11 & 3 \\
\hline
100 & 4 \\
\hline
101 & 5 \\
\hline
110 & 6 \\
\hline
111 & 7 \\
\hline
\end{tabular}
\caption{The first eight binary numbers.}
\label{tab:binary}
\end{center}
\end{table}

%In the decimal system, each part of a number is referred to as a %``digit''.
%For example, the number 456 has three digits.
%In the binary system, each part of a number is referred to as a ``bit''.
%The number 10111 in binary has five bits.

%When you hear the phrase ``64-bit computer'', it means that the %processors and memory use 64 bits to store integers.
%That is where the limits for data types like \java{int} and \java{long} %come from.

% ABD: I think we can leave bits out of the discussion, and just talk
% about different written representations of the same number.

In decimal there are 10 digits, and the written representation of numbers is based on powers of 10.
For example, the number 456 has 4 in the 100's place, 5 in the 10's place, and 6 in the 1's place.
So the value is 400 + 50 + 6.

\begin{center}
\begin{tabular}{|c|c|c|}
\hline
4 & 5 & 6 \\
\hline
$10^2$ & $10^1$ & $10^0$ \\
\hline
\end{tabular}
\end{center}

In binary there are 2 digits, and the written representation of numbers is based on powers of 2.
For example, the number 10111 has 1 in the 16's place, 0 in the 8's place, 1 in the 4's place, 1 in the 2's place, and 1 in the 16's place.
So the value is 16 + 0 + 4 + 2 + 1, which is 23 in decimal.

\begin{center}
\begin{tabular}{|c|c|c|c|c|}
\hline
1 & 0 & 1 & 1 & 1 \\
\hline
$2^4$ & $2^3$ & $2^2$ & $2^1$ & $2^0$ \\
\hline
\end{tabular}
\end{center}

To get the digits of a decimal number, we can use repeated division.
For example, if we divide 456 by 10, we get 45 with remainder 6.
The remainder is the rightmost digit of 456.

If we divide the result again, we get 4 with remainder 5.
The remainder is the second rightmost digit of 456.
And if we divide again, we get 0 with remainder 4.
The remainder is the third rightmost digit of 456, and the result, 0, tells us that we're done.

We can do the same thing in binary if we divide by two.
When you divide by two, the remainder is the right-most digit, either 0 or 1.
If you divide the result again, you get the second rightmost digit.
If you keep going, and write down the remainders, you'll have your number in binary.

\begin{stdout}
23 / 2 is 11 remainder 1
11 / 2 is  5 remainder 1
 5 / 2 is  2 remainder 1
 2 / 2 is  1 remainder 0
 1 / 2 is  0 remainder 1
\end{stdout}

Reading these remainders from bottom to top, 23 in binary is 10111.

\section{Recursive Binary Method}

Now, to display a number in binary we can combine the algorithm from the previous section and the ``count up'' pattern from Section~\ref{countup}.

Here is a recursive method that displays any positive integer in binary:

%slr:  11/22/19
\begin{code}
public static void displayBinary(int value) {
    if (value == 0) {
       return;  //base case
    } else {
       displayBinary(value / 2); //recursive call
       System.out.print(value % 2); //print afterwards
    }
}
\end{code}

Here is a shorter version of the method that is equivalent:

\begin{code}
public static void displayBinary(int value) {
    if (value > 0) {
       displayBinary(value / 2); //recursive call
       System.out.print(value % 2); //print afterwards
    }
}
\end{code}

If \java{value} is zero, this shorter version of \java{displayBinary} does nothing (that's the base case).
If the argument is positive, the method divides it by two and calls \java{displayBinary} recursively.
When the recursive call returns, the method displays one digit of the result and returns (again).
Figure~\ref{fig.stack4} illustrates this process.

\index{stack diagram}
\index{diagram!stack}

\begin{figure}[!ht]
\begin{center}
\includegraphics[width=190pt]{figs/stack4.pdf}
\caption{Stack diagram for the \java{displayBinary} method.}
\label{fig.stack4}
\end{center}
\end{figure}

The leftmost digit is near the bottom of the stack, so it gets displayed first.
The rightmost digit, near the top of the stack, gets displayed last.
%slr:  11/22/19
After invoking \java{displayBinary}, we use \java{println} to complete the output as shown in the trinket below:

%\begin{code}
%displayBinary(23);      // output is 10111
%System.out.println();
%\end{code}
% height = 130 + 11 * num_lines
\begin{trinket} [270] {Binary.java}
public class Binary {

    public static void main(String[] args) {
       displayBinary(23);
       System.out.println();
    }

    public static void displayBinary(int value) {
        if (value > 0) {
           displayBinary(value / 2); //recursive call
           System.out.print(value % 2); //print afterwards
        }
    }
}
\end{trinket}

Run the trinket to verify we obtain the result \java{10111}.

\section{Two Recursion Examples (from CodingBat)}

In the past several chapters you've seen methods, conditions, loops, strings, arrays, and recursion.
A great resource for practicing all of these concepts is \url{https://codingbat.com/}.

\index{CodingBat}

CodingBat is a free website of programming problems developed by Nick Parlante, a Computer Science lecturer at Stanford University.
As you work on these problems, CodingBat saves your progress (if you create an account).

To conclude this chapter, we consider two problems in the {\sf Recursion-1} section of CodingBat.
One of them deals with strings, and the other deals with arrays.
Both of them have the same recursive idea: check the base case, look at the current index, and recursively handle the rest.

The first problem is available at \url{https://codingbat.com/prob/p118230}:

\begin{quote}
\textbf{Recursion-1 ~noX}

Given a string, recursively compute a new string where all the \java{'x'} chars have been removed.

\ttfamily
noX("xaxb") $\rightarrow$ "ab" \\
noX("abc") $\rightarrow$ "abc" \\
noX("xx") $\rightarrow$ ""
\end{quote}

When solving recursive problems, it helps to think about the base case first.
The base case is the easiest version of the problem; for \java{noX}, it's the empty string.
If the argument is an empty string, there are no {\tt x}'s to be removed.

\begin{code}
if (str.length() == 0) {
    return "";
}
\end{code}

% NOTE: Some people like isEmpty(), but I think this is the right
% idiom for our purposes
% https://www.techiedelight.com/check-string-empty-or-null-java/

Next comes the more difficult part.
To solve a problem recursively, you need to think of a simpler instance of the same problem.
For \java{noX}, it's removing all the {\tt x}'s from a shorter string.

So let's split the string into two parts, the first letter and the rest. Since we know the string has length greater than 0, we are assured we can do this:

\begin{code}
char first = str.charAt(0);
String rest = str.substring(1);
\end{code}

Now we can make a recursive call to remove the {\tt x}'s from {\tt rest}:

\begin{code}
String recurse = noX(rest);
\end{code}

That takes care of {\tt rest} and places the result into {\tt recurse},

To take care of {\tt first} we have two possibilities: if it happens to be an {\tt x}, we're done; we just have to return {\tt recurse}.  Otherwise, we have to concatenate {\tt first} and {\tt recurse}.  Here's the {\tt if} statement we need.

\begin{code}
if (first == 'x') {
    return recurse;
} else {
    return first + recurse;
}
\end{code}

%slr:  11/22/19
%You can run this solution on CodingBat by pasting these snippets into the provided method definition.
Here is a complete Java class trinket with a stubbed version of the method \java{noX()}:

% height = 130 + 11 * num_lines
\begin{trinket} [270] {NoX.java}
public class NoX {

    public static void main(String[] args) {
       String s = noX("xaxb");
       System.out.println(s);
    }

    public static String noX(String str) {
        if (str.length() == 0) {
           return ""; //empty String
        } else {
           return ""; //stub for recursive case
        }
    }
}
\end{trinket}

Complete the implementation of the \java{noX()} method and test your code.  This is section exercise 1 below.

The second problem is available at \url{https://codingbat.com/prob/p135988}:

\begin{quote}
\textbf{Recursion-1 ~array11}

Given an array of ints, compute recursively the number of times that the value 11 appears in the array.

\ttfamily
array11([1, 2, 11], 0) $\rightarrow$ 1 \\
array11([11, 11], 0) $\rightarrow$ 2 \\
array11([1, 2, 3, 4], 0) $\rightarrow$ 0
\end{quote}

This problem uses the convention of passing the index as an argument.  In this way, we can pass the entire array and the index in each recursive call instead of computing and passing a subarray.

The base case is when we've reached the end of the array.  At that point, we know there are no more {\tt 11}'s.

\begin{code}
if (index >= nums.length) {  //at the end of the array
    return 0;
}
\end{code}

Next we look at the current array element (based on the given index), and check if it's an {\tt 11}.
After that, we can recursively check the rest of the array.
Similar to the noX problem, we look at only one array entry per method call.

\begin{code}
int recurse = array11(nums, index + 1);  //recursive call with rest of array
if (nums[index] == 11) {
    return recurse + 1;  //increment the count of 11s
} else {
    return recurse;
}
\end{code}

%slr:  11/22/19
%Again, you can run this solutions on CodingBat by pasting the snippets into the method definition.
Here is a complete Java class trinket with a stubbed version of the method \java{array11()}:

% height = 130 + 11 * num_lines
\begin{trinket} [290] {Elevens.java}
public class Elevens {

    public static void main(String[] args) {
       int a = {0, 23, -4, 11, 1, 11, 35};
       int numElevens = array11(a, 0);
       System.out.printf("There are %d elevens in the array\n", numElevens);
    }

    public static int array11(int[] nums, int index) {
        if (index >= nums.length) {
           return 0; //empty String
        } else {
           return 0; //stub for recursive case
        }
    }
}
\end{trinket}

Complete the implementation of the \java{array11()} method and test your code.  This is section exercise 2 below.
%slr:  end 11/22/19

\index{Java Tutor}

To see how these solutions actually work, you might find it helpful to step through them with a debugger (see Appendix~\ref{debugger}) or Java Tutor (\url{https://thinkjava.org/javatutor}).
Then try solving other CodingBat problems on your own.

Learning to think recursively is an important part of learning to think like a computer scientist.
Many algorithms can be written concisely with recursive methods that perform computations on the way down, on the way up, or both.

\textbf{Section Exercises:}
\begin{enumerate}
\item Complete the implementation of the recursive method in the trinket above containing the class \java{NoX}.  Verify that it works correctly.
\item Complete the implementation of the recursive method in the trinket above containing the class \java{Elevens}.  Verify that it works correctly.
\end{enumerate}

\section{Vocabulary}

\begin{description}

%\term{void method}
%A method that does not return a value.

%\term{value method}
%A method that returns a value.

\term{iterative}
A method or algorithm that repeats steps using one or more loops.

\term{recursive}
A method or algorithm that invokes itself one or more times with different arguments.

%\term{recursion}
%The process of invoking (and restarting) the same method that is currently executing.

\term{base case}
A condition that causes a recursive method {\em not} to make another recursive call.

%\term{Turing complete}
%A programming language that can implement any theoretically possible algorithm.

\term{factorial}
The product of all the integers up to and including a given integer.

%\term{leap of faith}
%A way to read recursive programs by assuming that the recursive call works, rather than following the flow of execution.

\term{binary}
A system that uses only zeros and ones to represent numbers.
Also known as ``base 2''.

\end{description}

\section{Exercises}

The code for this chapter is in the {\tt ch08} directory of {\tt ThinkJavaCode2}.
See page~\pageref{code} for instructions on how to download the repository.
Before you start the exercises, we recommend that you compile and run the examples.

If you have not already read Appendix~\ref{JUnit}, now might be a good time.
It describes JUnit, a standard framework for writing test code.

\begin{exercise}  %%V6 Ex5.5

The purpose of this exercise is to take a problem and break it into smaller problems, and to solve the smaller problems by writing simple methods.
Consider the first verse of the song ``99 Bottles of Beer'':

\begin{quote}
99 bottles of beer on the wall,\\
99 bottles of beer,\\
ya' take one down, ya' pass it around,\\
98 bottles of beer on the wall.
\end{quote}

Subsequent verses are identical except that the number of bottles gets smaller by one in each verse, until the last verse:

\begin{quote}
No bottles of beer on the wall,\\
no bottles of beer,\\
ya' can't take one down, ya' can't pass it around,\\
'cause there are no more bottles of beer on the wall!
\end{quote}

And then the song (finally) ends.

Write a program that displays the entire lyrics of ``99 Bottles of Beer''.
Your program should include a recursive method that does the hard part, but you might want to write additional methods to separate other parts of the program.
As you develop your code, test it with a small number of verses, like \java{3}.

\end{exercise}

\begin{exercise}  %%V6 Ex6.7

Write a recursive method named \java{oddSum} that takes a positive odd integer \java{n} and returns the sum of odd integers from 1 to n.
Start with a base case, and use temporary variables to debug your solution.
You might find it helpful to print the value of \java{n} each time \java{oddSum} is invoked.

\end{exercise}

\begin{exercise}  %%V6 Ex6.6

In this exercise, you will use a stack diagram to understand the execution of the following recursive method.

\begin{code}
public static void main(String[] args) {
    System.out.println(prod(1, 4));
}

public static int prod(int m, int n) {
    if (m == n) {
        return n;
    } else {
        int recurse = prod(m, n - 1);
        int result = n * recurse;
        return result;
    }
}
\end{code}

\begin{enumerate}

\item Draw a stack diagram showing the state of the program just before the last invocation of \java{prod} completes.

\item What is the output of this program?
(Try to answer this question on paper first, then run the code to check your answer.)

\item Explain in a few words what \java{prod} does (without getting into the details of how it works).

\item Rewrite \java{prod} without the temporary variables \java{recurse} and \java{result}.
{\it Hint:} You only need one line for the \java{else} branch.

\end{enumerate}

\end{exercise}

\begin{exercise}  %%V6 Ex6.8

The goal of this exercise is to translate a recursive definition into a Java method.
The Ackermann function is defined for non-negative integers as follows:
\begin{eqnarray*}
A(m, n) = \begin{cases}
              n+1 & \mbox{if } m = 0 \\
        A(m-1, 1) & \mbox{if } m > 0 \mbox{ and } n = 0 \\
A(m-1, A(m, n-1)) & \mbox{if } m > 0 \mbox{ and } n > 0
\end{cases}
\end{eqnarray*}

Write a recursive method called \java{ack} that takes two \java{int}s as parameters and that computes and returns the value of the Ackermann function.

Test your implementation of Ackermann by invoking it from \java{main} and displaying the return value.
Note the return value gets very big very quickly.
You should try it only for small values of $m$ and $n$ (not bigger than 3).

\end{exercise}

\begin{exercise}  %%V6 Ex6.9
\label{ex.power}

Write a recursive method called \java{power} that takes a double \java{x} and an integer \java{n} and returns $x^n$.

{\it Hint:} A recursive definition of this operation is $x^n = x \cdot x^{n-1}$.
Also, remember that anything raised to the zeroth power is 1.

Optional challenge: you can make this method more efficient, when \java{n} is even, using $x^n = \left( x^{n/2} \right)^2$.

\end{exercise}

\begin{exercise}  %%V6 Ex8.8

Many of the patterns we have seen for traversing arrays can also be written recursively.
It is not common, but it is a useful exercise.

\begin{enumerate}

\item Write a method called \java{maxInRange} that takes an array of integers and two indexes, \java{lowIndex} and \java{highIndex}, and finds the maximum value in the array, but only considering the elements between \java{lowIndex} and \java{highIndex}, including both.

This method should be recursive.
If the length of the range is 1, that is, if \java{lowIndex == highIndex}, we know immediately that the sole element in the range must be the maximum.
So that's the base case.

If there is more than one element in the range, we can break the array into two pieces, find the maximum in each of the pieces, and then find the maximum of the maxima.

\item Methods like \java{maxInRange} can be awkward to use.
To find the largest element in an array, we have to provide the range for the entire array.

\begin{code}
double max = maxInRange(a, 0, a.length - 1);
\end{code}

Write a method called \java{max} that takes an array and uses \java{maxInRange} to find and return the largest element.

\end{enumerate}

\end{exercise}

\newpage
\begin{exercise}  %%V6 Ex9.4

Create a program called {\tt Recurse.java} and type in the following methods:

\begin{code}
/**
 * Returns the first character of the given String.
 */
public static char first(String s) {
    return s.charAt(0);
}
\end{code}

\begin{code}
/**
 * Returns all but the first letter of the given String.
 */
public static String rest(String s) {
    return s.substring(1);
}
\end{code}

\begin{code}
/**
 * Returns all but the first and last letter of the String.
 */
public static String middle(String s) {
    return s.substring(1, s.length() - 1);
}
\end{code}

\begin{code}
/**
 * Returns the length of the given String.
 */
public static int length(String s) {
    return s.length();
}
\end{code}

\begin{enumerate}

\item Write some code in \java{main} that tests each of these methods.
Make sure they work, and you understand what they do.

\item Using these methods, and without using any other \java{String} methods, write a method called \java{printString} that takes a string as a parameter and that displays the letters of the string, one on each line.
It should be a void method.

\item Again using only these methods, write a method called \java{printBackward} that does the same thing as \java{printString} but that displays the string backward (again, one character per line).

\item Now write a method called \java{reverseString} that takes a string as a parameter and that returns a new string as a return value.
The new string should contain the same letters as the parameter, but in reverse order.

\begin{code}
String backwards = reverseString("coffee");
System.out.println(backwards);
\end{code}

The output of this example code should be:

\begin{stdout}
eeffoc
\end{stdout}

\index{palindrome}

\item A palindrome is a word that reads the same both forward and backward, like ``otto'' and ``palindromeemordnilap''.
Here's one way to test whether a string is a palindrome:

\begin{quotation}
\noindent
A single letter is a palindrome, a two-letter word is a palindrome if the letters are the same, and any other word is a palindrome if the first letter is the same as the last and the middle is a palindrome.
\end{quotation}

Write a recursive method named \java{isPalindrome} that takes a \java{String} and returns a \java{boolean} indicating whether the word is a palindrome.

\end{enumerate}

\end{exercise}


\section{Multiple Random Variables}

\subsection{Facts}
\begin{itemize}[+]
    \item RVs are independent if and only if their pdfs factorise
    \item Functions of independent RVs are independent
    \item Expectations of products (and hence mgfs, etc. of sums) of independent RVs factor
    \item Variance of sum of independent RVs is sum of variances.
    \item Independent RVs have vanishing covariance/correlation, but the converse is not true in general.
\end{itemize}

\subsection{Bivariate Relations}
\begin{theorem}[Conditional Expectation]
    \[
        \E{}[\E{}[X|Y]] = \E{}[X]
    \]
    provided the expectations exist.
\end{theorem}

\begin{theorem}[Conditional variance]
    \[
        \Var{}[X] = \E{}[\Var{}[X \vert{} Y]] + \Var{}[\E{}[X \vert{} Y]]
    \]
    provided the expectations exist.
\end{theorem}

\begin{definition}[Covariance]
    \[
        \Cov{}[X, Y] = \E{}[(X - \mu_X)(Y - \mu_Y)]
    \]
\end{definition}

\begin{theorem}
    \[
        \Cov{}[X, Y] = \E{}[XY] - \mu_X\mu_Y
    \]
\end{theorem}

\begin{theorem}
    \[
        \Var[aX + bY] = a^2\Var{}[X] + b^2\Var{}[Y] + 2ab\Cov[X, Y]
    \]
\end{theorem}

\begin{definition}[Correlation]
    \[
        \rho_XY = \frac{\Cov[X, Y]}{\sigma_X \sigma_Y}
    \]
\end{definition}

\begin{remark}
    The correlation measures the strength of \emph{linear} relation between two RVs. It is possible to have strong non-linear relationships but with $\rho = 0$.
\end{remark}

We can use an argument similar to the standard proof of Cauchy-Schwarz to show the following
\begin{theorem}
    Let $X$ and $Y$ be any RVs, then
    \begin{enumerate}[a.]
        \item $-1 \leq \rho_{XY} \leq 1$,
        \item $\abs{\rho_{XY}} = 1$ if and only if there are constants $a \neq 0, b$ such that $\P{}(Y = aX + b) = 1$. If $\abs{\rho_{XY}} = 1$ then $\text{sign}(\rho) = \text{sign}(a)$.
    \end{enumerate}
\end{theorem}

\subsection{Inequalities}

\subsubsection{Numerical Inequalities}

\begin{theorem}
    Let $a$ and $b$ be any positive numbers and let $p, q > 1$ satisfy $1/p + 1/q = 1$, then
    \[
        \frac1p a^p + \frac1q b^q \geq ab
    \]
    with equality if and only if $a^p = b^q$.
\end{theorem}

\begin{theorem}[H{\"o}lder's Inequality]
    Let $X$ and $Y$ be any random variables and let $p, q > 1$ satisfy $1/p + 1/q = 1$, then
    \[
        \abs{\E{}[XY]} \leq \E{}[\abs{XY}] \leq \E{}[\abs{X}^p]^{1/p} \E{}[\abs{Y}^q]^{1/q}
    \]
\end{theorem}

\begin{corollary}\mbox{}
    \begin{itemize}
        \item Cauchy-Schwarz is the special case $p = q = 2$
        \item $\Cov[X, Y]^2 \leq \sigma_X^2 \sigma_Y^2$
        \item $\E{}[\abs{X}] \leq \E{}[\abs{X}^p]^{1/p}$
        \item \emph{Liapounov's Inequality} $\E{}[\abs{X}^r]^{1/r} \leq \E{}[\abs{X}^s]^{1/s}$ where $1 < r < s < \infty$.
    \end{itemize}
\end{corollary}

\subsubsection{Functional Inequalities}

\begin{definition}[Convex Function]
    A function $g(x)$ is \emph{convex} on a set $S$ if for all $x, y \in S$ and $0< \lambda < 1$
    \[
        g(\lambda x + (1 - \lambda)y) \leq \lambda g(x) + (1 - \lambda)g(y).
    \]
    \emph{Strictly convex} is when the inequality is strict. $g$ is \emph{concave} if $-g$ is convex.
\end{definition}

\begin{lemma}
    $g(x)$ is convex on $S$ if $g''(x) \geq 0$ $\forall x \in S$.
\end{lemma}

\begin{theorem}[Jensen's Inequality]
    If $g(x)$ is convex, then for any random variable $X$
    \[
        \E{}[g(X)] \leq g(\E{}[X]).
    \]
    Equality holds if and only if, for every line $a + bx$ that is tangent to $g(x)$ at $x = \E{}[X]$, $\P{}\{g(X) = a + bX\} = 1$. (So if and only if $g$ is affine with probability 1.)
\end{theorem}

\begin{corollary}
    \mbox{}
    \begin{itemize}
        \item $\E{}[X^2] \geq \E{}[X]^2$
        \item $\E{}[1/X] \geq 1/ \E{}[X]$
    \end{itemize}
\end{corollary}


Integer Linear Programming is the first of the two methods that has been used in this project (see Introduction for more details about the problem). This method always finds the optimal solution without taking into account the amount of computational
resources needed. This model is developed in CPLEX and the model implemented is described in the next sections.

\subsection{Decision variables}

\begin{itemize}

\item $ w_{n,h} (\mathbb{B})  : \text{This boolean variable specifies whether the nurse n works at the hour h (1) or not (0)}  $
\item $ z_{n} (\mathbb{B})  : \text{This boolean variable specifies whether the nurse n works during the shift or not.} $
	\begin{itemize}[label=$\star$]
 	\item $ z{n} = 1  \Rightarrow \text{ The nurse n works at least 1 hour, } \exists h, w_{n,h} = 1 $
 	\item $ z{n} = 0 \Rightarrow \forall h, w_{n,h} = 0 $
 	\end{itemize}
\item $  s_{n} (\mathbb{N}) : \text{Positive integer variable specifying the hour in which the nurse n starts working,} \\ \text{  such that $ w_{n,s_{n}}=1 $ and  $ w_{n,s_{n}-i}=0 $,  $ \forall i: 1 \leq s_{n} - i < s_{n} $ } $
\item $  e_{n} (\mathbb{N}) : \text{Positive integer variable specifying the hour in which the nurse n stops working,} \\ \text{such that $ w_{n,e_{n}}=1 $ and  $ w_{n,e_{n}+i}=0,\forall i : e_{n} < e_{n}+i \leq 24 $  } $
\end{itemize}

\subsection{Instance parameters}

\begin{itemize}
\item  $ demand_h $: Array of integers, specifying the required number of nurses at hour h
\item  $ nNurses $: Integer that specifies the number of available nurses to assign.
\item  $minHours$: Integer that specifies the minimum number of hours that a nurse must work if she works.
\item  $maxHours$: Integer that specifies the maximum number of hours that a nurse must work if she works.
\item  $maxConsec$: Integer that specifies the maximum number of consecutive hours that a nurse can work.
\item  $maxPresence$: Integer that specifies the maximum number of hours that a nurse can stay at the hospital.
\end{itemize}

\subsection{Objective function}
\begin{center}
Minimize $ \sum\limits_{n=1}^{nNurses} z_{n}  $ \\
\end{center}

This objective function aims to minimize the number of working nurses, this means, minimize the number of $z_n$ variables that are activated (with a value of 1), which is the main goal for our problem.

\subsection{Constraints}

\begin{itemize}
\item  Set the $z_n$ values correctly: \\ \\
\begin{equation}
 \forall n: 1 \leq n \leq nNurses,  \\
	24 \cdot z_{n}  \geq \sum\limits_{1 \leq h \leq 24} w_{n,h} \\
   z_{n} \leq \sum\limits_{1 \leq h \leq 24} w_{n,h}
 \end{equation}

\item  At any hour h, at least demandh nurses must be working: \\ \\
\begin{equation}
\forall h : 1 \leq h \leq 24, \\
 \sum\limits_{1 \leq n \leq nNurses} w_{n,h} \geq demand_{h} \\
\end{equation}

\item  Each nurse that works, must work at least minHours: \\ \\
\begin{equation}
\forall n: 1 \leq n \leq nNurses \\
	\sum\limits_{1 \leq h \leq 24} w_{n,h} \geq minHours \cdot z_{n} \\
\end{equation}

\item  Each nurse that works, must work at most maxHours: \\ \\
\begin{equation}
\forall n: 1 \leq n \leq nNurses \\
	\sum\limits_{1 \leq h \leq 24} w_{n,h} \leq maxHours \cdot z_{n} \\
\end{equation}

\item  Each nurse works at most maxConsec consecutive hours: \\ \\
\begin{equation}
\forall n:  1 \leq n \leq nNurses, \\
	\forall h_{1}:  1 \leq h_{1} \leq 24 - maxConsec, \\
	\sum\limits_{ h_{1} \leq h \leq h_{1} + maxConsec} w_{n,h} \leq maxConsec
\end{equation}

\item  Each nurse can stay in the hospital at most maxPresence hours: \\ \\
\begin{equation}
\begin{aligned}
&\forall n:  1 \leq n \leq nNurses, e_{n} \leq 24 \cdot z_{n}, \\
 &\forall n:  1 \leq n \leq nNurses, \forall h: 1 \leq h \leq 24, e_{n} \geq h \cdot w_{n,h}, \\
 &\forall n:  1 \leq n \leq nNurses, s_{n} \geq 0, \\
 &\forall n:  1 \leq n \leq nNurses, \forall h: 1 \leq h \leq 24, s_{n} \leq (h - 24) \cdot w_{n,h} + 24 \cdot z_{n}, \\
  &\forall n:  1 \leq n \leq nNurses: \\ &e_{n} - s_{n} + 1 - (2 \cdot 24)\cdot(1 - z_{n}) \leq maxPresence \cdot z_{n}
\end{aligned}
\end{equation}

\item  Each nurse can rest at most one consecutive hour:

\begin{equation}
\begin{aligned}
\forall n:  1 \leq n \leq nNurses, \forall h: 2 \leq h \leq 22, \forall M: M \geq 24:  \\ M - M \cdot w_{n,h-1} + M \cdot w_{n,h} + M \cdot w_{n,h+1}  \geq \sum\limits_{h+1 \leq h_{i} \leq 24 }  w_{n,h_{i}}
 \end{aligned}
\end{equation}

\end{itemize}

\pagebreak


\newpage
\sectionnonum{Appendix}

\begin{appendices}

\section{Descent Lemma}
\label{app:DL}
\begin{lem}
    Let $ f: \mathbb{R}^n \rightarrow R $ be continuous differentiable,
    and $ x,\ y\in \mathbb{R}^n $. Suppose that:
    \begin{equation}
        \forall t \in [0,\ 1]\ \ \|\nabla f(x+t y)-\nabla f(x)\| \leq L t\|y\|
    \end{equation}
    where $ L $ is a scalar, then:
    \begin{equation}
        f(x+y) \leq f(x)+y^{T} \nabla f(x)+\frac{L}{2}\|y\|^{2}
    \end{equation}
\end{lem}

\begin{prf}
    Let $ g(t) = f(x+ty) $, then: \\
\begin{align}
    f(x+y)-f(x) & =g(1)-g(0) \\
    &=\int_{0}^{1} \frac{d g}{d t}(t) d t \\
    &=\int_{0}^{1} y^{T} \nabla f(x+t y) d t \\
    & \leq \int_{0}^{1} y^{T} \nabla f(x) d t+\left|\int_{0}^{1} y^{T}(\nabla f(x+t y)-\nabla f(x)) d t\right| \\
    & \leq \int_{0}^{1} y^{T} \nabla f(x) d t+\int_{0}^{1}\|y\| \cdot\|\nabla f(x+t y)-\nabla f(x)\| d t \\
    & \leq y^{T} \nabla f(x)+\|y\| \int_{0}^{1} L t\|y\| d t \\
    &=y^{T} \nabla f(x)+\frac{L}{2}\|y\|^{2}
\end{align}
\end{prf}

\section{PReLU}

He et al. \parencite{he2015delving} first propose the variant
PReLU, The definition is:
\begin{align}
g\left(z_{j}\right)=\left\{\begin{array}{ll}{z_{j},} & {\text { if } z_{j}>0} \\ {a_{j} z_{j},} & {\text { if } z_{j} \leq 0}\end{array}\right.
\end{align}
or in a more compact form:
\begin{equation}
    g(z_j) = \max \left(0, z_{j}\right)+a_{j} \min \left(0, z_{j}\right)
\end{equation}
$ a_j $ is the coefficient that can be adaptively learnt,
subscript $ j $ means the coefficient can vary on the
different channel in the same layer. Though the shape of
PReLU and Leaky ReLU (LReLU) look the same, PReLU drastically
avoids the overfitting by introducing a few parameters.
In practice, if $ a_j $ is shared by the same layers the
performance will be better.

\section{Numerous Zero-error Minimum}
\begin{thm}
    \label{thm:bezout}
    Suppose $ \mathbb{F} $ is a field and polynomials
    $ P,\ Q\in \mathbb{F} $ have no common factor of degree $ d\geq 1 $
    \footnote{A more frank expression is independent.}. Let
    $ Z(P,\ Q) = \{(x,\ y)\in \mathbb{F}^2|P(x,\ y) = Q(x,\ y)\} $.
    Then the number of points is upper bounded by $ d(P)d(Q) $.
\end{thm}
Via Bezout theorem \autoref{thm:bezout}, we can obtain the upper bound of a system
with $l$ layers and $ N $ data: $ d^{lN} $, where $ d $ is the
degree of polynomial equations. In the language of neural networks,
the number of zero points of cost functions is $ d^{lN} $. The theorem can
not strictly apply to neural networks with ReLU. But experimental
in Figure 2 in \parencite{poggio2017theory}
results show the similar behavior of ReLU and polynomial approximation.
Therefore, we give a proposition as follow:
\begin{pro}
    There are a large number of zero-error minimum.
\end{pro}

\end{appendices}