\chapter{Inner product spaces}
%%fakesection Cheerleading
It will often turn out that our vector spaces which look more like $\RR^n$
not only have the notion of addition, but also a notion of \emph{orthogonality}
and the notion of \emph{distance}.
All this is achieved by endowing the vector space with a so-called \textbf{inner form},
which you likely already know as the ``dot product'' for $\RR^n$.
Indeed, in $\RR^n$ you already know that
\begin{itemize}
	\ii $v \cdot w = 0$ if and only if $v$ and $w$ are perpendicular, and
	\ii $|v|^2 = v \cdot v$.
\end{itemize}
The purpose is to quickly set up this structure in full generality.
Some highlights of the chapter:
\begin{itemize}
	\ii We'll see that the high school ``dot product''
	formulation is actually very natural:
	it falls out from the two axioms we listed above.
	If you ever wondered why $\sum a_i b_i$ behaves as nicely as it does,
	now you'll know.
	\ii We show how the inner form can be used to make $V$ into a \emph{metric space},
	giving it more geometric structure.
	\ii A few chapters later,
	we'll identify $V \cong V^\vee$ in a way that wasn't possible before,
	and as a corollary deduce the nice result that
	symmetric matrices with real entries
	always have real eigenvalues.
\end{itemize}

Throughout this chapter, \emph{all vector spaces are over $\CC$ or $\RR$},
unless otherwise specified.
We'll generally prefer working over $\CC$ instead of $\RR$ since
$\CC$ is algebraically closed
(so, e.g.\ we have Jordan forms).
Every real matrix can be thought of as a matrix
with complex entries anyways.

\section{The inner product}
\prototype{Dot product in $\RR^n$.}
\subsection{For real numbers: bilinear forms}

First, let's define the inner form for real spaces.
Rather than the notation $v \cdot w$ it is most customary
to use $\left< v,w \right>$ for general vector spaces.
\begin{definition}
	Let $V$ be a real vector space.
	A \vocab{real inner form}\footnote{Other
		names include ``inner product'', ``dot product'',
		``positive definite nondegenerate symmetric bilinear form'', \dots}
	is a function
	\[ \left< \bullet, \bullet \right> : V \times V \to \RR \]
	which satisfies the following properties:
	\begin{itemize}
		\ii The form is \vocab{symmetric}: for any $v,w \in V$ we have
		\[ \left< v,w \right> = \left< w,v\right>. \]
		Of course, one would expect this property from a product.

		\ii The form is \vocab{bilinear}, or \textbf{linear in both arguments},
		meaning that $\left< -, v\right>$
		and $\left< v, -\right>$ are linear functions for any fixed $v$.
		Spelled explicitly this means that
		\begin{align*}
			\left< cx, v \right> &= c \left< x,v \right> \\
			\left< x+y, v \right> &= \left< x,v \right> + \left< y,v \right>.
		\end{align*}
		and similarly if $v$ was on the left.
		This is often summarized by the single equation
		$\left< cx+y, z \right> = c \left< x,z \right> + \left< y,z \right>$.

		\ii The form is \vocab{positive definite}, meaning $\left<v,v\right> \ge 0$
		is a nonnegative real number, and equality takes place only if $v = 0_V$.
	\end{itemize}
\end{definition}
\begin{exercise}
	Show that linearity in the first argument plus symmetry
	already gives you linearity in the second argument,
	so we could edit the above definition
	by only requiring $\left< -, v\right>$ to be linear.
\end{exercise}

\begin{example}
	[$\RR^n$]
	As we already know, one can define the inner form on $\RR^n$ as follows.
	Let $e_1 = (1, 0, \dots, 0)$, $e_2 = (0, 1, \dots, 0)$,
	\dots, $e_n = (0, \dots, 0, 1)$ be the usual basis.
	Then we let
	\[
		\left< a_1 e_1 + \dots + a_n e_n, b_1 e_1 + \dots + b_n e_n \right>
		\defeq a_1 b_1 + \dots + a_n b_n.
	\]
	It's easy to see this is bilinear
	(symmetric and linear in both arguments).
	To see it is positive definite,
	note that if $a_i = b_i$
	then the dot product is $a_1^2 + \dots + a_n^2$,
	which is zero exactly when all $a_i$ are zero.
\end{example}

\subsection{For complex numbers: sesquilinear forms}
The definition for a complex product space is similar, but has one difference:
rather than symmetry we instead have \emph{conjugate symmetry}
meaning $\left< v, w \right> = \ol{ \left< w,v \right> }$.
Thus, while we still have linearity in the first argument,
we actually have a different linearity for the second argument.
To be explicit:
\begin{definition}
	Let $V$ be a complex vector space.
	A \vocab{complex inner product} is a function
	\[ \left< \bullet, \bullet \right> : V \times V \to \CC \]
	which satisfies the following properties:
	\begin{itemize}
		\ii The form has \vocab{conjugate symmetry}, which means that
		for any $v,w \in V$ we have
		\[ \left< v,w \right> = \ol{\left< w,v\right>}. \]

		\ii The form is \vocab{sesquilinear}
		(the name means ``one-and-a-half linear'').
		This means that:
		\begin{itemize}
		\ii The form is \textbf{linear in the first argument}, so again we have
		\begin{align*}
			\left< x+y, v \right> &= \left< x,v \right> + \left< y,v \right> \\
			\left< cx, v \right> &= c\left< x,v \right>.
		\end{align*}
		Again this is often abbreviated to the single line
		$\left< cx+y, v \right> = c \left< x,v \right> + \left< y,v \right>$
		in the literature.

		\ii However, it is now \textbf{\vocab{anti-linear}
		in the second argument}:
		for any complex number $c$ and vectors $x$ and $y$ we have
		\begin{align*}
			\left< v, x+y\right> &= \left< v, x\right> + \left< v,y \right> \\
			\left< v, cx \right> &= \ol c \left< v, x\right>.
		\end{align*}
		Note the appearance of the complex conjugate $\ol c$,
		which is new!
		Again, we can abbreviate this to just
		$\left< v, cx+y \right> = \ol c \left< v,x \right> + \left< v,y\right>$
		if we only want to write one equation.
		\end{itemize}

		\ii The form is \vocab{positive definite},
		meaning $\left<v,v\right>$ is a nonnegative real number,
		and equals zero exactly when $v = 0_V$.
	\end{itemize}
\end{definition}
\begin{exercise}
	Show that anti-linearity follows
	from conjugate symmetry plus linearity in the first argument.
\end{exercise}

\begin{example}
	[$\CC^n$]
	The dot product in $\CC^n$ is defined as follows:
	let $\ee_1$, $\ee_2$, \dots, $\ee_n$ be the standard basis.
	For complex numbers $w_i$, $z_i$ we set
	\[
		\left< w_1 \ee_1 + \dots + w_n \ee_n, z_1 \ee_1 + \dots + z_n \ee_n \right>
		\defeq w_1\ol{z_1} + \dots + w_n \ol{z_n}.
	\]
\end{example}
\begin{ques}
	Check that the above is in fact a complex inner form.
\end{ques}

\subsection{Inner product space}
It'll be useful to treat both types of spaces simultaneously:
\begin{definition}
	An \vocab{inner product space} is either a real vector space
	equipped with a real inner form,
	or a complex vector space equipped with a complex inner form.

	A linear map between inner product spaces
	is a map between the underlying vector spaces
	(we do \emph{not} require any compatibility with the inner form).
\end{definition}

\begin{remark}
	[Why sesquilinear?]
	The above example explains one reason why we want
	to satisfy conjugate symmetry rather than just symmetry.
	If we had tried to define the dot product as $\sum w_i z_i$,
	then we would have lost the condition of being positive definite,
	because there is no guarantee that
	$\left< v,v \right> = \sum z_i^2$ will even be a real number at all.
	On the other hand, with conjugate symmetry
	we actually enforce $\left< v,v \right> = \ol{\left< v,v \right>}$,
	i.e.\ $\left< v,v \right> \in \RR$ for every $v$.

	Let's make this point a bit more forcefully.
	Suppose we tried to put a bilinear form $\left< -, -\right>$,
	on a \emph{complex} vector space $V$.
	Let $e$ be any vector with $\left< e, e \right> = 1$ (a unit vector).
	Then we would instead get
	$\left< ie, ie \right> = - \left< e,e \right> = -1$;
	this is a vector with length $-1$, which is not okay!
	That's why it is important that,
	when we have a complex inner product space,
	our form is sesquilinear, not bilinear.
\end{remark}

Now that we have a dot product,
we can talk both about the norm and orthogonality.

\section{Norms}
\prototype{$\RR^n$ becomes its usual Euclidean space with the vector norm.}

The inner form equips our vector space with a notion of distance, which we call the norm.
\begin{definition}
	Let $V$ be an inner product space.
	The \vocab{norm} of $v \in V$ is defined by
	\[ \norm{v} = \sqrt{\left<v,v\right>}. \]
	This definition makes sense because
	we assumed our form to be positive definite,
	so $\left< v,v\right>$ is a nonnegative real number.
\end{definition}

\begin{example}[$\RR^n$ and $\CC^n$ are normed vector spaces]
	When $V = \RR^n$ or $V = \CC^n$ with the standard dot product norm,
	then the norm of $v$ corresponds to the absolute value that we are used to.
\end{example}

Our goal now is to prove that
\begin{moral}
	With the metric $d(v,w) = \norm{v-w}$, $V$ becomes a metric space.
\end{moral}
\begin{ques}
	Verify that $d(v,w) = 0$ if and only if $v = w$.
\end{ques}
So we just have to establish the triangle inequality.
Let's now prove something we all know and love,
which will be a stepping stone later:
\begin{lemma}
	[Cauchy-Schwarz]
	Let $V$ be an inner product space.
	For any $v,w \in V$ we have
	\[ \left\lvert \left< v,w\right> \right\rvert
	\le \norm{v} \norm{w} \]
	with equality if and only if $v$ and $w$ are linearly dependent.
\end{lemma}
\begin{proof}
	The theorem is immediate if $\left< v,w\right> = 0$.
	It is also immediate if $\norm{v} \norm{w} = 0$,
	since then one of $v$ or $w$ is the zero vector.
	So henceforth we assume all these quantities are nonzero
	(as we need to divide by them later).

	The key to the proof is to think about the equality case:
	we'll use the inequality $\left< cv-w, cv-w\right> \ge 0$.
	Deferring the choice of $c$ until later, we compute
	\begin{align*}
		0 &\le \left< cv-w, cv-w \right> \\
		&= \left< cv, cv\right> - \left< cv, w\right> - \left< w, cv\right> + \left< w,w \right> \\
		&= |c|^2 \left< v,v \right> - c \left< v,w \right> - \ol c \left< w,v \right> + \left< w,w \right> \\
		&= |c|^2 \norm{v}^2 + \norm{w}^2 - c \left< v,w \right> - \ol{c \left< v,w\right> } \\
		2 \Re \left[ c \left< v,w \right> \right] &\le |c|^2 \norm{v}^2 + \norm{w}^2  \\
		\intertext{At this point, a good choice of $c$ is}
		c &= \frac{ \norm w}{\norm v} \cdot \frac{|\left< v,w\right>|}{\left< v,w\right>} \\
		\intertext{since then}
		c \left< v,w \right> &= \frac{\norm w}{\norm v} \left\lvert \left< v,w\right> \right\rvert \in \RR \\
		|c| &= \frac{\norm w}{\norm v} \\
		\intertext{whence the inequality becomes}
		 2\frac{\norm w}{\norm v} \left\lvert \left< v,w\right> \right\rvert &\le 2 \norm{w}^2  \\
		 \left\lvert \left< v,w\right> \right\rvert &\le \norm v \norm w. \qedhere
	\end{align*}
\end{proof}
Thus:
\begin{theorem}
	[Triangle inequality]
	We always have
	\[ \norm v + \norm w \ge \norm{v+w} \]
	with equality if and only if $v$ and $w$ are linearly dependent.
\end{theorem}
\begin{exercise}
	Prove this by squaring both sides, and applying Cauchy-Schwarz.
\end{exercise}

In this way, our vector space now has a topological structure of a metric space.

\section{Orthogonality}
\prototype{Still $\RR^n$!}
Our next goal is to give the geometric notion of ``perpendicular''.
The definition is easy enough:
\begin{definition}
	Two nonzero vectors $v$ and $w$ in an inner product space
	are \vocab{orthogonal} if $\left< v,w \right> = 0$.
\end{definition}

As we expect from our geometric intuition in $\RR^n$,
this implies independence:
\begin{lemma}[Orthogonal vectors are independent]
	Any set of pairwise orthogonal vectors $v_1$, $v_2$, \dots, $v_n$,
	with $\norm{v_i} \ne 0$ for each $i$,
	is linearly independent.
\end{lemma}
\begin{proof}
	Consider a dependence
	\[ a_1 v_1 + \dots + a_n v_n = 0 \]
	for $a_i$ in $\RR$ or $\CC$.
	Then \[ 0_V = \left< v_1, \sum a_i v_i \right> = \ol{a_1} \norm{v_1}^2. \]
	Hence $a_1 = 0$, since we assumed $\norm{v_1} \neq 0$.
	Similarly $a_2 = \dots = a_m = 0$.
\end{proof}

In light of this, we can now consider a stronger condition on our bases:
\begin{definition}
	An \vocab{orthonormal} basis of a
	\emph{finite-dimensional} inner product space $V$
	is a basis $e_1$, \dots, $e_n$ such that
	$\norm{e_i} = 1$ for every $i$ and
	$\left< e_i, e_j \right> = 0$ for any $i \neq j$.
\end{definition}
\begin{example}[$\RR^n$ and $\CC^n$ have standard bases]
	In $\RR^n$ and $\CC^n$ equipped with the standard dot product,
	the standard basis $\ee_1$, \dots, $\ee_n$ is also orthonormal.
\end{example}
This is no loss of generality:
\begin{theorem}[Gram-Schmidt]
	Let $V$ be a finite-dimensional inner product space.
	Then it has an orthonormal basis.
\end{theorem}
\begin{proof}[Sketch of Proof]
	One constructs the orthonormal basis explicitly from any basis
	$e_1$, \dots, $e_n$ of $V$.
	Define $\opname{proj}_u(v) = \frac{\left< v,u\right>}{\left< u,u\right>} u$.
	Then recursively define
	\begin{align*}
		u_1 &= e_1 \\
		u_2 &= e_2 - \opname{proj}_{u_1}(e_2) \\
		u_3 &= e_3 - \opname{proj}_{u_1}(e_3) - \opname{proj}_{u_2}(e_3) \\
		&\vdotswithin{=} \\
		u_n &= e_n - \opname{proj}_{u_1}(e_n) - \dots - \opname{proj}_{u_{n-1}}(e_n).
	\end{align*}
	One can show the $u_i$ are pairwise orthogonal and not zero.
\end{proof}
Thus, we can generally assume our bases are orthonormal.

Worth remarking:
\begin{example}[The dot product is the ``only'' inner form]
	Let $V$ be a finite-dimensional inner product space,
	and consider \emph{any} orthonormal basis $e_1, \dots, e_n$.
	Then we have that
	\[ \left< a_1 e_1 + \dots + a_n e_n,
		b_1 e_1 + \dots + b_n e_n \right>
		= \sum_{i,j=1}^n a_i\ol{b_j} \left< e_i, e_j \right>
		= \sum_{i=1}^n a_i \ol{b_i} \]
	owing to the fact that the $\{e_i\}$ are orthonormal.
\end{example}
And now you know why the dot product expression is so ubiquitous.

\section{Hilbert spaces}
In algebra we are usually scared of infinity,
and so when we defined a basis of a vanilla vector space many chapters ago,
we only allowed finite linear combinations.
However, if we have an inner product space,
then it is a metric space and we \emph{can}
sometimes actually talk about convergence.

Here is how it goes:
\begin{definition}
	A \vocab{Hilbert space} is a inner product space $V$,
	such that the corresponding metric space is complete.
\end{definition}

In that case, it will now often make sense to take infinite linear combinations,
because we can look at the sequence of partial sums and let it converge.
Here is how we might do it.
Let's suppose we have $e_1$, $e_2$, \dots an infinite sequence
of vectors with norm $1$ and which are pairwise orthogonal.
Suppose $c_1$, $c_2$, \dots, is a sequence of real or complex numbers.
Then consider the sequence
\begin{align*}
	v_1 &= c_1 e_1 \\
	v_2 &= c_1 e_1 + c_2 e_2 \\
	v_3 &= c_1 e_1 + c_2 e_2 + c_3 e_3 \\
	&\vdotswithin=
\end{align*}

\begin{proposition}
	[Convergence criteria in a Hilbert space]
	The sequence $(v_i)$ defined above
	converges if and only if $\sum \left\lvert c_i \right\rvert^2 < \infty$.
\end{proposition}
\begin{proof}
	This will make more sense if you read \Cref{ch:calc_limits},
	so you could skip this proof if you haven't read the chapter.
	The sequence $v_i$ converges if and only if it is Cauchy,
	meaning that when $i < j$,
	\[ \norm{v_j - v_i}^2 = |c_{i+1}|^2 + \dots + |c_j|^2 \]
	tends to zero as $i$ and $j$ get large.
	This is equivalent to the sequence
	$s_n = |c_1|^2 + \dots + |c_n|^2$ being Cauchy.

	Since $\RR$ is complete, $s_n$ is Cauchy
	if and only if it converges.
	Since $s_n$ consists of nonnegative real numbers,
	converges holds if and only if $s_n$ is bounded,
	or equivalently if $\sum \left\lvert c_i \right\rvert^2 < \infty$.
\end{proof}

Thus, when we have a Hilbert space, we change our definition slightly:
\begin{definition}
	An \vocab{orthonormal basis} for a Hilbert space $V$
	is a (possibly infinite) sequence $e_1$, $e_2$, \dots,
	of vectors such that
	\begin{itemize}
		\ii $\left< e_i, e_i \right> = 1$ for all $i$,
		\ii $\left< e_i, e_j \right> = 0$ for $i \ne j$,
		i.e.\ the vectors are pairwise orthogonal
		\ii every element of $V$ can be expressed uniquely as an
		infinite linear combination
		\[ \sum_i c_i e_i \]
		where $\sum_i \left\lvert c_i \right\rvert^2 < \infty$,
		as described above.
	\end{itemize}
\end{definition}
That's the official definition, anyways.
(Note that if $\dim V < \infty$, this agrees with our usual definition,
since then there are only finitely many $e_i$.)
But for our purposes you can mostly not worry about it and instead think:
\begin{moral}
	A Hilbert space is an inner product space
	whose basis requires infinite linear combinations,
	not just finite ones.
\end{moral}
The technical condition $\sum \left\lvert c_i \right\rvert^2 < \infty$
is exactly the one which ensures the infinite sum makes sense.

\section{\problemhead}

\begin{problem}
	[Pythagorean theorem]
	Show that if $\left< v,w \right> = 0$ in an inner product space,
	then $\norm{v}^2 + \norm{w}^2 = \norm{v+w}^2$.
\end{problem}

\begin{sproblem}
	[Finite-dimensional $\implies$ Hilbert]
	Show that a finite-dimensional inner product space
	is a Hilbert space.
	\begin{hint}
		Fix an orthonormal basis $e_1$, \dots, $e_n$.
		Use the fact that $\RR^n$ is complete.
	\end{hint}
\end{sproblem}

%\begin{problem}
%	Let $V$ be a complex inner product space
%	with orthonormal basis $e_1$, \dots, $e_n$.
%	Suppose $x = a_1 e_1 + \dots + a_n e_n \in V$
%	and $y = b_1 e_1 + \dots + b_n e_n \in V$.
%	Show:
%	\begin{align*}
%		\left< x,x \right> &= \sum_\xi |a_i|^2 \\
%		a_1 &= \left< x, e_1 \right> \\
%		\left< x,y \right> &= \sum_i a_i \ol{b_i}.
%	\end{align*}
%\end{problem}

\begin{problem}[Taiwan IMO camp]
	\gim
	In a town there are $n$ people and $k$ clubs.
	Each club has an odd number of members,
	and any two clubs have an even number of common members.
	Prove that $k \le n$.
	\begin{hint}
		Dot products in $\FF_2$.
	\end{hint}
	\begin{sol}
		Interpret clubs as vectors in the vector space $\mathbb F_2^n$.
		Consider a ``dot product'' to show that all $k$ vectors are linearly independent.
		Thus $k \le \dim \mathbb F_2^n = n$.
	\end{sol}
\end{problem}

\begin{sproblem}[Inner product structure of tensors]
	\label{prob:inner_prod_tensor}
	Let $V$ and $W$ be finite-dimensional inner product spaces over $k$,
	where $k$ is either $\RR$ or $\CC$.
	\begin{enumerate}[(a)]
		\ii Find a canonical way to make $V \otimes_k W$ into an inner product space too.
		\ii Let $e_1$, \dots, $e_n$ be an orthonormal basis of $V$
		and $f_1$, \dots, $f_m$ be an orthonormal basis of $W$.
		What's an orthonormal basis of $V \otimes W$?
	\end{enumerate}
	\begin{hint}
		Define it on simple tensors then extend linearly.
	\end{hint}
	\begin{sol}
		The inner form given by
		\[ \left< v_1 \otimes w_1 , v_2 \otimes w_2 \right>_{V \otimes W}
		= \left< v_1,v_2 \right>_V \left< w_1,w_2\right>_W \]
		on pure tensors, then extending linearly.
		For (b) take $e_i \otimes f_j$ for $1 \le i \le n$, $1 \le j \le m$.
	\end{sol}
\end{sproblem}

\begin{problem}[Putnam 2014]
	\gim
	Let $n$ be a positive integer.
	What is the largest $k$ for which there exist
	$n\times n$ matrices $M_1,\dots,M_k$ and $N_1,\dots,N_k$
	with real entries such that for all $i$ and $j$,
	the matrix product $M_i N_j$ has a zero entry somewhere
	on its diagonal if and only if $i \ne j?$
	\begin{hint}
		$k = n^n$.
		Endow tensor products with an inner form.
		Note that ``zero entry somewhere on its diagonal''
		is equivalent to the product of those entries being zero.
	\end{hint}
\end{problem}

\begin{problem}
	[Sequence space]
	Consider the space $\ell^2$ of infinite sequences of real numbers
	$a = (a_1, a_2, \dots)$ satisfying $\sum_i a_i^2 < \infty$.
	We equip it with the dot product
	\[ \left< a, b\right> = \sum_i a_i b_i. \]
	Is this a Hilbert space?
	If so, identify a Hilbert basis.
\end{problem}

\begin{problem}
	[Kuratowski embedding]
	A \vocab{Banach space} is a normed vector space $V$,
	such that the corresponding metric space is complete.
	(So a Hilbert space is a special case of a Banach space.)

	Let $(M,d)$ be any metric space.
	Prove that there exists a Banach space $X$
	and an injective function $f \colon M \injto X$
	such that $d(x,y) = \norm{f(x)-f(y)}$ for any $x$ and $y$.
\end{problem}


\chapter{Introduction}\label{chapter:introduction}

Two of the most important results in geometric group theory are Gromov's classification of groups with \emph{polynomial volume growth}~\cite{gromov1981}, and Grigorchuk's construction of a group with \emph{intermediate volume growth}~\cite{grigorchuk1983}.
These results answer two questions originally posed by \citeauthor{milnor1968a}, namely, whether the \emph{volume growth} of groups must always be either exponential, or polynomial of an integer degree; and if there is an algebraic classification of groups with polynomial volume growth~\cite{milnor1968a}.
In this thesis, we consider the analogous questions for \emph{geodesic growth}.

The \emph{volume growth function} of a group counts the number of elements that can be represented using words up to a given length.
It was shown by \citeauthor{gromov1981} that a group has polynomial volume growth if and only if it is \emph{virtually nilpotent}~\cite{gromov1981}.
The \emph{geodesic growth function} of a group counts the number of \emph{geodesic words} (i.e.\ minimal-length representatives of group elements) with a given upper bound on their lengths.
The \emph{volume (resp.~geodesic) growth series} is then the power series whose coefficients are the values of the volume (resp.~geodesic) growth function.
Since each element of a group has at least one corresponding geodesic, we see that the geodesic growth is bounded from below by the volume growth.
From this and Gromov's theorem, we see that only \emph{virtually nilpotent} groups may have polynomial geodesic growth.
It is well known that the class of \emph{nilpotent} groups come in a sequence of \textit{steps}, the first \textit{step} being the abelian groups.
Thus, to obtain a classification of polynomial geodesic growth, it is natural to start with the \emph{virtually abelian} groups.

The study of geodesic growth for abelian groups began in \citeyear{shapiro1997} when \citeauthor{shapiro1997} considered the function $p_S\colon G \to \mathbb{N}$ which counts the geodesics corresponding to a given element of a group $G$~\cite{shapiro1997}.
The function $p_S$ is referred to as the \emph{Pascal function} as, in the case of free-abelian groups, it resembles a Pascal triangle.
The geodesic growth of virtually abelian groups was considered by \citeauthor{bridson2012} who provided the first example of a group with polynomial geodesic growth that is not \emph{virtually cyclic}~\cite{bridson2012}.
Moreover, they provided a sufficient condition for a virtually abelian group to have polynomial geodesic growth with respect to some generating set, a condition for a group to have exponential geodesic growth with respect to every generating set, and proved that the geodesic growth function of a virtually cyclic group is either exponential, or polynomial of an integer degree.
In this thesis, we extend their results by completely characterising the geodesic growth for virtually abelian groups.

\begin{restatable*}{theoremx}{TheoremGeodesicGrowth}\label{thm:geodesic-growth}
	Let $G$ be a virtually abelian group with a finite weighted monoid generating set $S$.
	Then the geodesic growth with respect to $S$ is either polynomial of integer degree with rational geodesic growth series, or exponential with holonomic geodesic growth series.
\end{restatable*}

We provide the first example of a group with polynomial geodesic growth that is not virtually abelian.
In particular, we show that the \emph{virtually Heisenberg group}
\[
	\vH
	=
	\left\langle
		a,b,c,t
	\,\,\middle|\,\,
		[a,b] = c,\,
		[a,c] = [b,c] = t^2 = 1,\,
		a^t = b
	\right\rangle
\]
has a polynomial upper bound on its geodesic growth function with respect to the generating set $S = \{a,a^{-1},t\}$.
We prove this result in \cref{thm:main}.
This example shows that a classification of polynomial geodesic growth is more complicated than just a subclass of virtually abelian groups.

\begin{restatable*}{theoremx}{TheoremVirtuallyHeisenberg}\label{thm:main}
	The geodesic growth function of $\vH$ with respect to $S = \{a,a^{-1},t\}$ is bounded from above by a polynomial of degree $8$.
\end{restatable*}

It was shown by \citeauthor{duchin2019} that the volume growth series of the Heisenberg group $H_3$ is rational with respect to every generating set~\cite{duchin2019}.
\Cref{thm:geodesic-growth} shows us that if the geodesic growth of a virtually abelian group is polynomial, then its geodesic growth series is rational.
However, it is not clear if this holds for the geodesic growth function of our virtually Heisenberg example $H_3 \rtimes C_2$.
In fact, computational experiments suggest that the geodesic growth series is not rational (see~\cite{githubcode}).
Thus, this may be the first example with polynomial geodesic growth and non-rational geodesic growth series.

The question of the existence of a group with intermediate geodesic growth was considered as early as 1993 by Grigorchuk and Shapiro (see~\cite[756]{grigorchuk2014}).
In the PhD thesis of \citeauthor{broennimann2016} %Theorem~3.5,3.8,3.10 and 3.14
most of the groups known to have intermediate volume growth, at the time, were shown to have exponential geodesic growth with respect to their standard generating sets~\cite[Chapter~3]{broennimann2016};
these results were an extension of an unpublished work of \citeauthor{elder2006} where this was shown only for the \emph{first Grigorchuk group}~\cite{elder2006}.
We cannot, at this time, eliminate the possibility of there being a virtually nilpotent group with intermediate geodesic growth.
Thus, the results in this thesis also have application to the search of a group with intermediate geodesic growth.

Most of the literature on geodesic growth has been concerned with either showing that the language of geodesics is regular (see~\cref{sec:grammars-and-automata}), or that the geodesic growth series is rational (see~\cref{sec:formal-language/generating-functions/rational}) with respect to some particular generating sets.
It is known that the language of geodesics for a \emph{hyperbolic group} is regular with respect to any finite generating set~\cite[Theorem~3.4.5]{epstein1992}.
This result was generalised by \citeauthor{neumann1995} to any group and generating set with the \emph{falsification by fellow traveller property}~\cite[Proposition~4.2~on~p.~267]{neumann1995}.
There are many results for particular generating sets of certain \emph{Coxeter groups}, \emph{Artin groups} \cite{antolin2021,mairesse2006,kolpakov2020,ciobanu2016,antolin2013,holt2012,antolin2021,athreya2014}, and \emph{Garside groups}~\cite{sabalka2004,charney2004}.
\Citeauthor{shapiro1997} studied the Pascal function for abelian and hyperbolic groups~\cite{shapiro1997}.
It was shown by \citeauthor{loeffler2002} that having a regular language of geodesics is preserved by graph product~\cite{loeffler2002}.
\Citeauthor{hermiller2008} studied groups whose languages of geodesics are \emph{locally testable} (which is a proper subfamily of the regular languages)~\cite{hermiller2008}.
\Citeauthor{cleary2006} showed that the language of geodesics for the \emph{lamplighter group} is \emph{context-free} and \emph{counter} for some generating sets; and that the language of geodesics for \emph{Thompson's group $F$} is not regular for any generating set~\cite{cleary2006}.

We prove \cref{thm:geodesic-growth} by constructing an algorithm that converts geodesics into \emph{patterned words}.
From this algorithm, we find a bijection from the set of geodesics of a virtually abelian group to a certain formal language with a \emph{holonomic} growth series.
It is then natural to ask if there is a formal-language characterisation for the language of geodesics for a virtually abelian group.
In \cref{thm:virtually-abelian-are-blind-counter}, we obtain such a characterisation by implementing this algorithm using \emph{blind multicounter automata}.

\begin{restatable*}{theoremx}{TheoremBlindMulticounter}\label{thm:virtually-abelian-are-blind-counter}
	The language of geodesics of a virtually abelian group with respect to any finite weighted monoid generating set $S$ is blind multicounter.
\end{restatable*}

A \emph{formal language over a group} is a set of words whose letters are taken from the generating set.
So far we have discussed our results on the language of geodesics for a group.
In this thesis, we also study the \emph{co-word problem}, that is, the language of words that do not correspond to the group identity.
Characterisations of such languages provide us with one measure for the computational difficulty involved with computing in a group.

The \emph{word problem} of a group $G$ with respect to a finite monoid generating set $S$, denoted $\WP_S$, is the set of all words in $S^*$ that correspond to the group identity.
We see that the word problem completely specifies a group as $\left\langle S \mid \WP_S \right\rangle$ is a presentation for $G$.
This formal language is one characterisation of the difficulty of computing within a group, i.e., checking if two words $u,v \in S^*$ represent the same group element is equivalent to checking if the word $uv^{-1}$ is in the word problem $\WP_S$.
The \emph{co-word problem} of a group, denoted $\coWP_S$, is the complement of the word problem in the sense that $\coWP_S = S^* \setminus \WP_S$.

There are groups for which it is not possible to decide membership to the word problem.
Interestingly, there are such groups for which the word problem is \emph{recursively enumerable} but not \emph{recursive}, that is, there is an algorithm that lists every word in the word problem (with no guarantee of order) but no such algorithm which lists out every word in the co-word problem.
In particular, there are finitely-presented groups with unsolvable word problems.
The existence of such examples was shown independently by \textcite{novikov1955} (see \cite{novikov1955-translated} for an English translation) and \textcite{boone1959}.
An explicit example with 10 generators and 27 relators was given by \textcite{collins1986}, then later a simpler example with 2 generators and 27 relators was given by \textcite{wang2016}.
We see that the word problem for any finitely-presented group is recursively enumerable (see \cref{prop:appendix/wp-fp}).

Suppose that $\mathcal{F}$ is a family of formal languages that is closed under \emph{inverse word homomorphism}.
Then, if the word or co-word problem with respect to some generating set belongs to $\mathcal{F}$, it belongs to $\mathcal{F}$ for all generating sets (see~\cref{lem:well-defined-coword}).
Thus, for such a family it is well defined to state that a group has a word or co-word problem in $\mathcal{F}$.
Examples of such families are the \emph{regular}, \emph{context-free} and \emph{context-sensitive} languages; the family of \emph{ET0L languages} which are a type of L-system, introduced by \textcite{rozenberg1973}, that generalises context-free languages, and form a subfamily of context-sensitive languages; and the family of \emph{blind multicounter languages}~\cite{greibach1978} which generalise counter languages and are equivalent to the class of \emph{reversal bounded multicounter languages}~\cite{baker1974} and \emph{Parikh languages}~\cite{klaedtke2003}.
If $\mathcal{F}$ is a family of formal languages that is closed under inverse word homomorphism, and the word (resp.~co-word) problem of a group lies in $\mathcal{F}$, then we say that the group is an $\mathcal{F}$ (resp.~co-$\mathcal{F}$) group.

It is interesting to find classifications of groups whose languages $\WP_S$ and $\coWP_S$ belong to certain language families.
The study of this problem began with \citeauthor{anisimov1971} who showed that a group is finite if and only if $\WP_S$ is a regular language~\cite{anisimov1971}.
Further classifications were obtained by \citeauthor{muller1983} who showed that a group is \emph{virtually free} if and only if its word problem is a context-free language~\cite{muller1983}; and \citeauthor{elder2008} who showed that a group is virtually abelian if and only if its word problem is a blind multicounter language~\cite{elder2008}.

The study of the group co-word problem, $\coWP_S$, gives us an additional source of such classifications.
The class of groups for which $\coWP_S$ is context-free was first studied by \textcite{holt2005}.
Their results were then extended by \textcite{lehnert2007} who showed that Thompson's group $V$ has a context-free co-word problem.
Combining a result of \textcite{bleak2016} with a remark in \citeauthor{lehnert2008}'s thesis~\cite[\S\,4.2]{lehnert2008}, it is conjectured that every group with context-free co-word problem is a subgroup of Thompson's group $V$.
Moreover, it is conjectured that Grigorchuk's group does not have a context-free co-word problem~\cite{bleak2016}.

The class of \emph{bounded automata groups} includes important examples such as Grigorchuk's group of intermediate growth, the Gupta-Sidki groups, and many more \cite{grigorchuk1980,gupta1983,nekrashevych2005,sidki2000}.
It was shown by \textcite{holt2006} that bounded automata groups have \emph{indexed} co-word problems.
ET0L languages form a proper subfamily of the indexed languages introduced by \textcite{aho1968} (see Corollary~4.1 in~\cite{culik1974} and Proposition~4.5 in~\cite{ehrenfeucht1976}).
For the case of Grigorchuk's group, it was later shown by \citeauthor{ciobanu2018} that the co-word problem is ET0L~\cite{ciobanu2018}.
They proved this result by explicitly constructing an ET0L grammar to recognise the co-word problem.
In \cref{thm:bounded automata is ET0L} we use an equivalent machine model to generalise this result to all bounded automata groups.

\begin{restatable*}{theoremx}{TheoremBoundedAutomata}\label{thm:bounded automata is ET0L}
	Every finitely-generated bounded automata group is co-ET0L.
\end{restatable*}

All results in this thesis are with respect to monoid generating sets for groups.
In particular, this means that we do not assume that our generating sets are \emph{symmetric}.
For example, the group of integers $\mathbb{Z} = \left\langle a \mid -\right\rangle$ is generated by $\{a^{-1}, a^3\}$.
Moreover, we prove \cref{thm:geodesic-growth,thm:virtually-abelian-are-blind-counter} for each finite weighted monoid generating set.
That is, for each generator we associate a positive integer weight, and we say that a word is a geodesic if it represents an element with minimal weight.
Notice that we may recover the usual definition of a geodesic by choosing the weight of each generator to be one.

\section{Structure}

This thesis is structured as follows.
In \cref{chp:formal-language-and-automata}, we provide background on formal language theory and define the families of formal languages that are used in our proofs.
In \cref{chp:coword-problems} we study the co-word problem for bounded automata groups, and prove \cref{thm:bounded automata is ET0L}.
We then return to formal language theory in \cref{chp:polyhedral} where we define the family of polyhedrally constrained languages.
Then, in \cref{chapter:polynomial-geodesic-growth} we prove provide a characterisation of the language of geodesics and geodesic growth of virtually abelian groups by proving \cref{thm:geodesic-growth,thm:virtually-abelian-are-blind-counter}.
Finally, in \cref{chapter:virtually-heisenberg} we consider virtually nilpotent groups and prove \cref{thm:main}.

\section{Attribution of Results}

\Cref{thm:geodesic-growth,thm:virtually-abelian-are-blind-counter} are published in the single-authored paper~\cite{bishop2021}.
\Cref{thm:main} is joint work with my supervisor, Murray Elder, a preprint of this work is available in~\cite{bishop2020}.
\Cref{thm:bounded automata is ET0L} is also joint work with Elder and has appeared in the conference proceedings of LATA (Language and Automata Theory and Applications) 2019~\cite{bishop2019}.

\section{Notation}\label{chapter:notation}

Let $\mathbb{N} = \{0,1,2,\ldots\}$ denote the set of nonnegative integers, including zero, and $\mathbb{N}_+ = \{1,2,3,\ldots\}$ the set of positive integers.

Let $G$ be a group, and let $g,h \in G$, then $[g,h] = ghg^{-1}h^{-1}$ and $g^h = h g h^{-1}$.
Given two subgroups $H,K \leqslant G$, we write $[H,K]$ for the subgroup
\[
	[H,K] = \left\langle\left\{ [h,k] \mid h \in H,\ k \in K \right\}\right\rangle.
\]
A group $G$ is \emph{$k$-step nilpotent} if there is a finite sequence of subgroups
\[
	G_0 = G,\
	G_1 = [G,G_0],\
	G_2 = [G,G_1],\
	G_3 = [G,G_2],\
	\ldots,\
	G_k = [G,G_{k-1}] = \{1\}.
\]
Notice that the $1$-step nilotent groups are precisely the abelian groups.
Moreover, for each $k$, the set of $(k+1) \times (k+1)$ invertible upper-triangular integer matrices with 1's on their diagonal form a $k$-step nilpotent group.
In fact, it is a result of \citeauthor{auslander1967} that each nilpotent (or more generally each \emph{polycyclic}) group has a faithful representation in $\mathrm{SL}(n,\mathbb{Z})$ for some $n$~\cite[Theorem~2]{auslander1967}.

Let $\mathcal{P}$ be some group property, e.g., being abelian, nilpotent, or free.
Then, we say that a group is \emph{virtually $\mathcal{P}$} is it has a finite-index subgroup with the property $\mathcal{P}$.
For example, we say that a group is virtually nilpotent if it has a finite-index nilpotent subgroup.

Let $G$ be a group with a finite generating set $S$, then we write $S^*$ for the set of all words, including the empty word $\varepsilon \in S^*$, in the letters of $S$; and $\overline{\sigma} \in G$ for the group element corresponding to the word $\sigma \in S^*$.
We endow $S$ with a weighting, that is, for each generator $s \in S$ we assign a positive integer weight $\omega(s) \in \mathbb{N}_+$.
We then say that $S$ is a finite weighted generating set for the group $G$.
The \emph{weight} of a word $\sigma = \sigma_1 \sigma_2 \cdots \sigma_k \in S^*$ is then given by
$
	\omega(\sigma)
	=
	\sum_{i=1}^k
	\omega(\sigma_i)
$.
Moreover, we write $|\sigma|_S = k$ for the \emph{word length} of $\sigma$.
The \emph{weighted length} of an element $g \in G$ is then defined as the minimum weight required to represent it as a word, that is,
\[
	\ell_S(g)
	=
	\min\{
		\omega(\sigma)
	\mid
		\overline{\sigma} = g
		\text{ where }
		\sigma \in S^*
	\}.
\]
We may now define the volume growth function $a_S \colon \mathbb{N} \to \mathbb{N}$ as follows.

\begin{definition}\label{defn:volume-growth}
	The \emph{volume growth function} $a_S\colon \mathbb{N} \to \mathbb{N}$ is defined as
	\[
		a_S(n) = \#\{
			g \in G
		\mid
			\ell_S(g) \leq n
		\}.
	\]
	That is, $a_S(n)$ counts the elements which can be represented by a word of length $n$ or less.
\end{definition}

We say that a word $\sigma \in S^*$ is a \emph{geodesic} if it represents $\overline{\sigma}$ with minimal weight, that is, if $\omega(\sigma) = \ell_S(\overline{\sigma})$.
We write $\GeodesicWords_S$ for the set of all geodesic words with respect to the generating set $S$, that is,
\[
\GeodesicWords_S
=
\{
\sigma \in S^*
\mid
\omega(\sigma) = \ell_S(\overline{\sigma})
\}.
\]
We then define the \emph{geodesic growth function} $\gamma_S\colon \mathbb{N} \to \mathbb{N}$ as follows.

\begin{definition}\label{defn:geodesic-growth}
	The \emph{geodesic growth function} $\gamma_S \colon \mathbb{N} \to \mathbb{N}$ is defined as
	\[
	\gamma_{S}(n)
	=
	\#
	\{
	\sigma \in \GeodesicWords_S
	\mid
	\omega_S(\sigma) \leq n
	\}
	\]
	This function counts the number of geodesic words of length $n$ or less.
\end{definition}

Notice that the volume and geodesic growth functions can be at most exponential as
\[
		a_S(n)
	\leq
		\gamma_S(n)
	\leq
		\sum_{i=0}^n |S|^{i}
	\leq
		|S|^{n+1}.
\]
We say that a (volume/geodesic) growth function $f \colon \mathbb{N} \to \mathbb{N}$ has
\begin{itemize}
	\item \emph{polynomial growth} if there is some $\beta,d \in \mathbb{N}_+$ such that $f(n) \leq \beta n^d$ for each $n \geq 1$;
	\item \emph{exponential growth} if there is an $\alpha \in \mathbb{R}$ with $\alpha > 1$ such that $f(n) \geq \alpha^n$; and
	\item \emph{intermediate growth} if its growth is neither polynomial nor exponential.
\end{itemize}
Notice that the volume and geodesic growth functions are submultiplicative, that is, if $f\colon \mathbb{N} \to \mathbb{N}$ is a growth function, then $f(n+m) \leq f(n) f(m)$ for each $n,m \in \mathbb{N}$.
Thus, we may apply the following result.

\begin{lemma}[\citeauthor{fekete1923}'s lemma~\cite{fekete1923}]\label{lemma:fekete}
	If $f\colon \mathbb{N} \to \mathbb{N}$ is submultiplicative, then the \emph{growth rate}
	$
		\alpha_f = \lim_{n \to \infty} \sqrt[n]{f(n)}
	$ is defined.
\end{lemma}

From \cref{lemma:fekete}, we see that a function $f \colon \mathbb{N} \to \mathbb{N}$ has exponential growth if and only if the growth rate $\alpha_f > 1$.

In this thesis, we are interested in studying the asymptotics of growth functions by considering their associated generating functions.
We write $A_S$ and $\Gamma_S$ for the generating functions associated with $a_S$ and $\gamma_S$, respectively.

\begin{definition}\label{defn:growth-series}
	We write
	\[
	A_S(z) = \sum_{n = 0}^\infty a_S(n) z^n
	\quad\text{and}\quad
	\Gamma_S(z) = \sum_{n = 0}^\infty \gamma_S(n) z^n
	\]
	for the \emph{volume} and \emph{geodesic growth series}, respectively.
\end{definition}

We write $\mathbf{x} = (x_1,x_2,\ldots,x_m)$ for a finite list of variables.
Then, for each vector $\mathbf{n} = (n_1,n_2,\ldots,n_m)$, we then write $\mathbf{x}^\mathbf{n} = x_1^{n_1} x_2^{n_2} \cdots x_m^{n_m}$.
We may write a multivariate generating series as $f(\mathbf{x}) = \sum_{\mathbf{n} \in \mathbb{N}^m} c_\mathbf{n} \mathbf{x}^{\mathbf{n}}$ where each $c_\mathbf{n}$ is a constant.
We write $\mathbb{C}[[\mathbf{x}]]$, $\mathbb{C}[\mathbf{x}]$, $\mathbb{C}((\mathbf{x}))$, and $\mathbb{C}(\mathbf{x})$ for the class of formal power series, polynomials, formal Laurent series, and rational functions, respectively, over the variables $\mathbf{x} = (x_1,x_2,\ldots,x_m)$.
Moreover, we write $\partial_{x_i} f(\mathbf{x})$ for the formal partial derivative of $f(\mathbf{x})$ with respect to $x_i$.
We use this notation in \cref{sec:generating-functions} where we define multivariate generating functions and the classes of rational, algebraic and holonomic power series.

Let $G$ be a group with finite monoid generating set $S$, then the \emph{word} and \emph{co-word} problem of $G$ with respect to $S$ are given as
\[
	\WP(G,S) = \{ w \in S^* \mid \overline{w} = 1_G \}
	\quad\text{and}\quad
	\coWP(G,S) = \{ w \in S^* \mid \overline{w} \neq 1_G \},
\]
respectively.
Notice here that $\coWP(G,S) = S^* \setminus \WP(G,S)$, that is, the two sets are complements of each another with respect to the set $S^*$.


\chapter{Low Thrust Optimization Using Orthogonal Collocation} \label{Ch:OrthogonalCollocation}

\section{Nomenclature}

\begin{tabbing}
12345678 \= Reynolds number based on length $s$ \kill\\
$t$  \>   Time in dimensional units \\
$\tau$  \>  Time in nondimensional units $(-1 \leq \tau \leq 1)$\\
$t_I$  \>   Initial time \\
$t_F$  \>  Final time\\
$\mathbf{r}(t)$        \> Position vector in continous time form\\
$\mathbf{v}(t)$        \> Velocity vector in continous time form\\
$m(t)$        \> Mass in continous time form\\
$\mathbf{x}(t)$   \>  State Vector in continous time form\\
$\mathbf{r}_k$        \> Position vector in discrete time form at $k^{th}$ point\\
$\mathbf{v}_k$        \> Velocity vector in discrete time form at $k^{th}$ point\\
$m_k$        \> Mass in discrete time form at $k^{th}$ point\\
$\mathbf{x}_k$   \>  State Vector in discrete time form at $k^{th}$ point\\
$n_x$              \> Number of elements in state vector\\
$n_u$              \> Number of elements in control vector\\
$N$              \> Number of collocation points\\
$n_k$              \> Number of quadrature points \\
$L_i$              \> $i^{th}$ Lagrange polynomial \\
\end{tabbing}

\section{Dynamics Model}

\section{Continuous Time Problem}

\begin{equation}
     J = \Phi \left(t_I,\mbx(t_I),t_F,\mbx(t_F)\right) + \int_{t_I}^{t_F} \mathcal{L}\left( \mbx(t),\mbu(t),t;\mbq \right)dt
\end{equation}
%
subject to
%
\begin{equation}
   \frac{\partial \mathbf{x}}{\partial t} = \mathbf{f}\left( \mbx(t),\mbu(t),t;\mbq \right)
\end{equation}

\section{Discretization of State and Dynamics}

\begin{equation}
    \mbx(\tau_k) \approx \mbx_k =  \sum^{N}_{j = 1,j \neq i} \mbx_i L_i(\tau_k)
\end{equation}
%
where
%
\begin{equation}
    L_i = \prod^{N}_{j = 1,j \neq i} \frac{\tau - \tau_j}{\tau_i - \tau_j}
\end{equation}
%
%
\begin{equation}
   \dot{\mbx}(t_k) \approx \dot{\mbx}_k =  \sum^{N}_{j = 1,j \neq i} \mbx_i \dot{L}_i(\tau_k)
\end{equation}

Define the matrix $\mathbf{X}$ as follows
%
\begin{equation}
     \mathbf{X} = \left[
     \begin{array}{ccc}
         \mbx_1^T \\
         \mbx_2^T \\
         \vdots \\
         \mbx_N^T
     \end{array}
     \right]
     %
     =
     %
     \left[
     \begin{array}{ccc}
         \mbr_1^T & \mbv_1^T & m_1 \\
         \mbr_2^T & \mbv_2^T & m_2\\
         \vdots \\
         \mbr_N^T & \mbv_N^T & m_N
     \end{array}
     \right]
     %
\end{equation}
%
Similarly, define the matrix $\mathbf{F}$ as
%
\begin{equation}
   \mathbf{F} = \left[
     \begin{array}{ccc}
         \mathbf{f}_1^T \\
         \mathbf{f}_2^T \\
         \vdots \\
         \mathbf{f}_N^T
     \end{array}
     \right]
\end{equation}
%
and
%
\begin{equation}
   \mathbf{U} = \left[
     \begin{array}{ccc}
         \mathbf{u}_1^T \\
         \mathbf{u}_2^T \\
         \vdots \\
         \mathbf{u}_N^T
     \end{array}
     \right]
\end{equation}

then,
%
\begin{equation}
    \dot{\mathbf{X}} = \mathbf{D}\mathbf{X}
\end{equation}
%
\begin{equation}
 \mathbf{D}\mathbf{X} - \mathbf{F}(\mathbf{X},\mathbf{U},\mathbf{Q}) = \mathbf{0}
\end{equation}
%
\begin{equation}
    \dot{\mathbf{x}}_k^T = \mathbf{D}_{(k,:)}\mathbf{X}
\end{equation}
%

\begin{equation}
     \frac{\partial \dot{\mbx}_k}{\partial \mbx_{\ell}} = \left[ \mathbf{0}_{n_x,\ell-1}
     \hspace{.1 in} \mathbf{D}_{(k,:)}^T \hspace{.1 in}  \mathbf{0}_{n_x,n_x - \ell+1} \right]
\end{equation}
%
\begin{equation}
     \frac{\partial \dot{\mbx}_k}{\partial \mathbf{u}_{\ell}} = \mathbf{0}_{n_x,n_u}
\end{equation}

\begin{equation}
     \frac{\partial \mathbf{f}_k}{\partial \mathbf{x}_{\ell}} =
     \left\{ \begin{array}{ccc}
    &\mathbf{A}(t_\ell) \hspace{.1 in} & k = \ell \\
    &\mathbf{0}_{n_x,n_x}  \hspace{.1 in} & k \neq \ell  \\
    \end{array} \right.
\end{equation}
%
\begin{equation}
     \frac{\partial \mathbf{f}_k}{\partial \mathbf{u}_{\ell}} =
          \left\{ \begin{array}{ccc}
          & \displaystyle\frac{\partial \mbf(t_\ell)}{\partial \mbu} \hspace{.1 in} & k = \ell \\
          &\mathbf{0}_{n_x,n_u}  \hspace{.1 in} & k \neq \ell  \\
          \end{array} \right.
\end{equation}

\section{Discretization of Cost Function}


\chapter{Transfer Function}
\label{chap:tf}
The transfer function was calculated from the system equation
\eqref{eq:equationmotion} analysed in the section
\ref{subsec:equationofomotion}.
%
\begin{equation}
\label{eq:gs}
	G(s) = \big([M]s^2+[C]s+[K]\big)^{-1}
\end{equation}
%
Given \(F(s) = g_{\text{v}}V(s)\)  the Laplace transform of the force signal,
thus the transfer function can be write:
\begin{align}
\label{eq:tf}
  {X(s)} &= G(s)\frac{F(s)}{g_{\text{v}}}\\
  \frac{X(s)}{F(s)} &= \frac{1}{g_{\text{v}}}\,{G(s)}
\end{align}
%
\begin{figure}[htb]
	\centering
	\includegraphics[width=0.66\textwidth]{bodediagram1}
	\caption{Bode diagram of the transfer functions as in \eqref{eq:tf}}
	\label{fig:bodeplot1}
\end{figure}
%
The bode diagram are shown in Figure \ref{fig:bodeplot1}.


\documentclass{report}
\usepackage{amsmath}
\usepackage{amssymb}
\begin{document}
\section{Expectation and Variance}
\subsection{Abstract}
In this issue, we mainly study some properties of Gaussian distribution
\subsection{Assumption}
Now given a bunch of data:
$$
X=(x_1, x_2, ..., x_N)^T
$$
$$
x_i \in \mathcal{R}^p
$$
First, we assume our model: the Gauss linear model.\\\\
To simplify the derivation of formula, we set $p$ equals $1$, so
$$
x \backsim N(\mu, \sigma^2)
$$
$$
\theta=(\mu, \sigma)
$$
Next, we use maximum likelihood estimation ($MLE $) to get the expectation and variance based on this bunch of data\\\\
The likelihood function is given below:
$$
\begin{aligned}
p(X|\theta)&=log(\prod_{i=1}^N \frac{1}{\sqrt{2\pi}\sigma} \exp(-\frac{(x_i-\mu)^2}{2\sigma^2})) \\
&=\sum_{i=1}^N log(\frac{1}{\sqrt{2\pi}\sigma} \exp(-\frac{(x_i-\mu)^2}{2\sigma^2}))\\
&=\sum_{i=1}^N log(\frac{1}{\sqrt{2\pi}}) - log(\sigma) - \frac{(x_i-\mu)^2}{2\sigma^2}
\end{aligned}
$$
\subsection{Expectation}
Next, we first use the maximum likelihood estimation to obtain the estimated value of the expected $\mu $

$$
\begin{aligned}
\mu_{MLE}
&=argmax(p(X|\theta))\\
&=argmin(\sum_{i=1}^N (x_i-\mu)^2)
\end{aligned}
$$
By deriving the formula:
$$
\begin{aligned}
\sum_{i=1}^N 2(x_i-\mu)&=0\\
\sum_{i=1}^N x_i - N \mu&=0\\
\mu_{MLE}&=\frac{1}{N} \sum_{i=1}^N x_i\\
\end{aligned}
$$

\subsection{Variance}
Similarly, we use maximum likelihood estimation to estimate the variance
$\sigma $

$$
\begin{aligned}
\sigma_{MLE}
&=argmax(p(X|\theta))\\
&=argmin(\sum_{i=1}^N log(\sigma)+\frac{(x_i-\mu)^2}{2\sigma^2})
\end{aligned}
$$
Similarly, we derive the formula:
$$
\sum_{i=1}^N(\frac{1}{\sigma}-\frac{(x_i-\mu)^2}{\sigma^3})=0
$$
Finally, we get the estimated value:
$$
\sigma_{MLE}^2 = \Sigma_{MLE} = \frac{1}{N} \sum_{i=1}^N (x_i-\mu)^2
$$
\subsection{Bias Estimation}
To verify whether an estimate is biased or unbiased, we only need to calculate the expectation of the estimate.
\subsubsection{$\mu$}
$$
\begin{aligned}
E[\mu_{MLE}]
&=E[\frac{1}{N}\sum_{i=1}^N x_i]\\
&=\frac{1}{N}\sum_{i=1}^N E[x_i]\\
&=\mu
\end{aligned}
$$
So $\mu_{MLE}$ is an unbiased estimation
\subsubsection{$\sigma$}
First we deform the estimate of $\sigma$
$$
\begin{aligned}
\sigma_{MLE}^2
&=\frac{1}{N} \sum_{i=1} ^N (x_i - \mu_{MLE})^2\\
&=\frac{1}{N} \sum_{i=1} ^N (x_i^2 - 2x_i\mu_{MLE} + \mu_{MLE}^2)\\
&=\frac{1}{N} \sum_{i=1}^N x_i^2 - 2(\frac{1}{N} \sum_{i=1} ^N x_i) \mu_{MLE} + \mu_{MLE}^2\\
&=\frac{1}{N} \sum_{i=1}^N x_i^2 - 2\mu_{MLE}^2 + \mu_{MLE}^2\\
&=\frac{1}{N} \sum_{i=1}^N x_i^2 - \mu_{MLE}^2\\
&=(\frac{1}{N} \sum_{i=1}^N x_i^2-\mu^2) - (\mu_{MLE}^2-\mu^2)\\
\end{aligned}
$$
set $f_1=(\frac{1}{N} \sum_{i=1}^N x_i^2-\mu^2)$ , $f_2=(\mu_{MLE}^2-\mu^2)$\\\\
so:
$$
\begin{aligned}
E[f_1]
&=E[\frac{1}{N} \sum_{i=1}^N x_i^2 - \mu^2]\\
&=E[\frac{1}{N} \sum_{i=1}^N (x_i^2 - \mu^2)]\\
&=\frac{1}{N} \sum_{i=1}^N E[x_i^2] - E[\mu^2]\\
&=\frac{1}{N} \sum_{i=1}^N E[x_i^2] - \mu^2\\
&=\frac{1}{N} \sum_{i=1}^N E[x_i^2] - (E[x_i])^2\\
&=\sigma^2
\end{aligned}
$$
similarly:
$$
\begin{aligned}
E[f_2]
&=E[\mu_{MLE}^2 - \mu^2]\\
&=E[\mu_{MLE}^2 - (E[\mu_{MLE}])^2]\\
&=Var[\mu_{MLE}]\\
&=Var[\frac{1}{N} \sum_{i=1} ^N x_i]\\
&=\frac{1}{N^2} \sum_{i=1} ^N Var[x_i]\\
&=\frac{1}{N} \sigma^2
\end{aligned}
$$
finally, adding $f_1$ and $f_2$ , we get:
$$
E[\sigma_{MLE}^2]=\frac{N-1}{N} \sigma^2
$$
So our estimate of $\sigma$ from the maximum likelihood estimate is slightly smaller than the true value, so it is biased.\\\\
The unbiased estimate of $\sigma^2$ is $\frac{1}{N-1}\sum_{i=1}^N (x_i-\mu_{MLE})^2$
\end{document}