\chapter{Appendix}
THE ELEMENTARY TRANSCENDENTAL FUNCTIONS

A*l. On certain results assumed in Chapters I-IV.

It was convenient, in the first four chapters of this work, to assume
some of the properties of the elementary transcendental functions,
namely the exponential, logarithmic and circular functions; it was
also convenient to make use of a number of results which the reader
would be prepared to accept intuitively by reason of his familiarity
with the geometrical representation of complex numbers by means of
points in a plane.

To take two instances, (i) it was assumed \hardsectionref{2}{7}) that lim (exp i) =
exp (lim *, and (ii) the geometrical concept of an angle in the
Argand diagram made it appear plausible that the argument of a complex
number was a many-valued function, possessing the property that any
two of its values diffei-ed by an integer multiple of ir.

The assumption of results of the first type was clearly illogical; it
wjis also illogical to base ai'ithmetical results on geometrical
reasoning. For, in order to put the foundations of geometry on a
satisfactory basis, it is not only desirj ble to employ the axioms of
arithmetic, but it is also necessary to utilise a further set of
axioms of a more definitely geometrical character, concerning
properties of points, straight lines and planes*. And, fm-ther, the
arithmetical theory of the logarithm of a complex nimiber appears to
be a necessary preliminary to the development of a logical theory of
angles.

Apart from this, it seems unsatisfcictory to the c esthetic taste of
the mathematician to employ one branch of mathematics as an essential
constituent in the structui-e of another; particularly when the
former has, to some extent, a material basis whereas the latter is of
a purely abstract nature f.

The reasons for pursuing the somewhat illogical and unaesthetic
procedure, adopted in the earlier part of this work, were, firstly,
that the properties of the elementary transcen- dental functions were
required gradually in the com"se of Chapter n, and it seemed
imdesirable that the coiu e of a general development of the various
infinite processes should be frequently interrupted in oi"der to prove
theoremsf (with which the reader was, in all probabihty, already
famihar concerning a single particuliU* function; and, secondly, that
(in counexioji with the assumption of results based on geometrical
considei-ations) a pui'ely arithmetical mode of development of
Chapters l-iv, deriving no help or illus- trations from geometrical
processes, would have very greatly inci-eased the difliculties of the
reader unacquainted with the methods and the spirit of the analyst,

* It is not our object to give any account of the foundations of
geometry in this work. They are investigated by various writei-s. such
as Tiitehead. Axioms of Projective Geometry (Cambridge Math. Tracts,
no. i. 1906) and Mathews, Projectile Geometry London, 1914). A perusal
of Chapters i, xx, xxii and xxv of the latter work will convince the
reader that it is even more laborious to develop geometiy in a logical
manner, from the minimum number of axioms, than it is to evolve the
theory of the circular functions by purely analytical methods. A
complete account of the elements both of arithmetic and of geometry
has been given by Whitehead and Bussell, Principia Mathematica
(1910-1913).

t Cf. Merz, History of European Thought in the Nineteenth Century, n.
(London, 1903), pp. 631 (note 2) and 707 (note 1), where a letter from
Weierstrass to Schwarz is quoted. See also Sylvester, P/ii7. Mag. (5),
ii. (1876), p. 307 [Math. Papers, lu. (1909). p. -50].

%
% 580
%

A'll. Summary of the Appendix.

The general course of the Appendix is as follows :

In v5§ A*2-A-22, the exponential function is defined by a power
series. From this definition, combined with results contained in
Chapter ll, are derived the elementary properties (apart from the
periodic properties) of this function. It is then easy to deduce
corresponding properties of logarithms of positive numbers (§§
A'3-A'33).

Next, the sine and cosine are defined by power series from which
follows the connexion of these functions with the exponential
function. A brief sketch of the manner in which the formulae of
elementary trigonometry may be derived is then given ( § A'4-A-42).

The results thus obtained render it possible to discuss the
periodicity of the exponential and circular functions by purely
arithmetical methods (§§ A"5, A*51).

In §§ A"52-A'522, we consider, substantially, the continuity of the
inverse circular functions. When these functions have been
investigated, the theory of logarithms of complex numbers (§ A'6)
presents no further difficulty.

Finally, in § A"7, it is shewn that an angle, defined in a purely
analytical manner, possesises properties which are consistent with the
ordinary concept of an angle, based on our experience of the material
world.

It will be obvious to the reader that we do not profess to give a
complete account of the elementary transcendental functions, but we
have confined ourselves to a brief sketch of the logical foundations
of the theory*. The developments have been given by writers- of
various treatises, such as Hobson, Plane Trigonometry; Hardy, A
course of Pure Mathematics; and Bromwich, Theory of Infinite Series.

A'12. -1 logical order of development of the elements of Analysis.

The reader will find it instructive to read Chapters i-iv and the
Appendix a second time in the following order :

Chapter i (omitting + all of § 1 "5 except the first two paragraphs).

Chapter ii to the end of \hardsubsectionref{2}{6}{1} (omitting the examples inf §,
2-31-2-61).

Chapter iii to the end of \hardsubsectionref{3}{3}{4} and §§ 3-5-3-73.

The Appendix, §§ A-2-A-6 (omitting §§ A-32, A-33).-

Chapter ii, the examples of v \hardsubsectionref{2}{3}{1}-2'61.

Chapter in, 5\hardsubsubsectionref{3}{3}{4}{1}-3-4.

Chapter iv, inserting §§ A"32, A-33, Aw after \hardsubsectionref{4}{1}{3}.

Chapter ii, §§ 2-7-2-82.

He should try thus to convince himself that (in that order) it is
possible to elaborate a purely arithmetical development of the
subject, in which the graphic and familiar language of geometry J is
to be regarded as merely conventional.

* In writing the Appendix, frequent reference has been made to the
article on Algebraic Analysis in the Encijklopiidie der Math.
Wissenschaften by Pringsheim and Faber, to the same article translated
and revised by Molk for the Encyclopedic des Sciences Math., and to
Tannery, Introduction h la Theorie dis Fonctions d'uue Variable
(Paris, 1904).

t The properties of the argument (or phase) of a complex number are
not required in the text before Chapter v.

i E.p. 'a point ' for 'an ordered number-pair,' ' the circle of unit
radius with centre at the origin' for 'the set of ordered number-pairs
(.r, y) which satisfy the condition .x'- + y-=l,' 'the points of a
straight line ' for ' the set of ordered number-pairs (x, y) which
satisfy a relation of the type Ax + By + C =0,' and so on.

%
% 581
%

A "2. The exponential function exp z.

The exponential function, of a complex variable 2, is defined bj the
series*

exp.= l + f; + | + | + ... = l+J |;.

This series converges absolutely for all values of z (real and
complex) by D'Alembert's ratio test \hardsubsectionref{2}{3}{6}) since lim | z\ n) | = 0<1
; so the definition is valid for all values of z.

Further, the series converges uniformly throughout any bounded domain
of values of z; for, if the domain be such that j 2 j /iJ when z is
in the domain, then

j(sVm!)I R"/nl,

and the uniformity of the convergence is a consequence of the test of
Weierstrass \hardsubsectionref{3}{3}{4}),

X

by reason of the convergence of the series 1+2 R"i'n !), in which the
terms are indepen-

n = l

dent of z.

Moreover, since, for any fixed value of n, s'V ! is a continuous
function of z, it follows from \hardsubsectionref{3}{3}{2} tha t the exponential function
is continuous for all values of z; and hence (cf. § 3 '2), if z be a
variable which tends to the limit (, we have

lim exp 2= exp f.

A*21. The addition-theorem for the exponential function, and its
consequences. From Cauchy's theorem on multiplication of absolutely
convergent series \hardsubsectionref{2}{5}{3}), it follows thatf

(exp 2i)(exp2,) = (l+- V|', + -)(l + + !;' + ..)

2i + 22 Zi + 2z Z2 + Z-2,

= 1 + -Yj-+ 2", +...

= exp (21 + 22),

so that exp (21 + 22) can be expressed in terms of exponential
functions of 21 and of 22 by the formula

exp (21 + 22) = (exp 2i) (exp z. .

This result is known as the addition-theorem for the exponential
function. From it,

we see by induction that

(exp2i) (exp 22) ... (exp2,i) = exp(2i + 22+...+2 ),

and, in particular,

 exp 2 exp ( - 2) =exp 0=1.

From the last equation, it is apparent that there is no value of 2 for
which exp 2 =; for, if there were such a value of 2, since exp (-2)
would exist for this value of 2, we should have 0=1.

It also follows that, when x is real, exp >0; for, from the series
definition, exp.r l when x O; and, when x 0, exp x= 1/exp ( - .r)>0.

* It was formerly customary to define exp 2 as lim ( 1 + - I, cf.
  Cauchy, Coins cV Analyse, i.

p. 167. Cauchy ibid. pp. 168, 309) also derived the properties of the
function from the series, but his investigation when z is not rational
is incomplete. See also \Schlomilch, Handbuch der alg. Analysis
(1889), pp. 29, 178, 246. Hardy has pointed out (Math. Gazette, in. p.
284) that the limit definition has many disadvantages.

t The reader will at once verify that the general term in the product
series is (2i + Ci2i"- 22 + C22i"-22.\,2 + . . . + z ' n ! = ( i + z
)''ln ! .

%
% 582
%

Further, exp .> is an increasing function of the real variable,v;
for, if X.'>0, exp x + k) - exp x = exp x . exp >l- - 1 > 0, because
exp .r>0 and exp />!.

Also, since expA- 1 /A = 1 +(A/2 :) + (/i2/3 !) + ...,

and the series on the right is seen (by the methods of § A"2) to be
continuous for all

values of h, we have

lim exp/i- 1 //; = 1,

rfexpa,. exp(2+A)-exp2 and so -y = hm - - = exp z.

A'22. Variom properties of the exponential function.

Returning to the formula (exp j) (exp 2.>) ... (exp 2- ) = exp (21+22
+ +2n) we see that,

when n is a positive integer,

(exp 2)" = exp (ns),

and (exp z)-" = 1 /(exp 2)" = 1 /exp nz) = exp ( - ?iz).

In particular, taking 2 = 1 and writing e in place of exp 1 =
2-71828..., we see that, when m is an integer, positive or negative,

e' = exp Hi = l + ( i/i:) + (? -/2 !) + ....

Also, if /Li be any rational number TODO, where// and q are integers,
q being positive)

(exp fi)'i = exp fiq = exp p = e'\

so that the th power of exp is et'; that is to say, expM is a value
of ef'ii=ei, and it is obviously (§ A-21) the real positive value.

If X be an irrational-real number (defined by a section in which Oj
and ao are typical members of the X-class and the ff-class
respectively), the irrational power e is most simply defined as exp x
; we thus have, for all real values of x, rational and irrational,

X x'

e =\ -I 1 1-

1 ! 2! '

an equation first given by Newton*.

It is, therefore, legitimate tg write e for exp x when x is real, and
it is customary to write e for exp 2 when z is complex. The function e
(which, of course, must not be regarded as being a power of e), thus
defined, is subject to the ordinary laws of indices,, viz.

[Note. Tannery, Lecons d'Algehre et d' Analyse (1906), I. p. 45,
practically defines e, when X is irrational, as the only number X
such that e'* X e"-, for every ! and ao. From the definition we have
given it is easily seen that such a unique number exists. For e\ ] )x(
= X) satisfies the inequality, and i( X' X) also did so, then

exj) 2 - exp r i = c'"- - /'' I X' - -V i,

so that, since the exponential function is continuous, a-i - a cannot
be chosen arbitrarily small, and so ( i, a-2) does not define a
section.]

* De Aitdlysi per aequat. nu)n. term. inf. (written before 16G9, but
not published till 1711); it was also given both by Newtou and by
Leibniz in letters to Oldenburg in 1676; it was first published by
Wallis in 1685 in his Treatise on Algebra, p. 343. The equation when x
is irrational was explicitly stated by \Schlomilch, Handbuch der alg.
Analysis (1889), p. 182.

%
% 583
%

A'3. Logarithms of positive numbers*.

It has been seen ( § A"2, A"21) that, when x is real, expA* is a
positive continuous increasing function of x, and obviously exp a- - +
x as x - - + oo, while

exp.'r=l/exp (- ')- >-0 as x- - cc.

If, then, a be any positive number, it follows from \hardsubsectionref{3}{6}{3} that the
equation in x,

exp X = a,

has one real root and only one. This root (which is, of course, a
function of a) will be written t LoggW or simply Log a; it is called
the Logarithm of the positive number a.

Since a one-one correspondence has been established between i- and a,
and since a is an increasing function of x, x must be an increasing
function of a; that is to say, the Logarithm is an increasing
function.

Example. Deduce from § A-21 that Log a + Log 6 = Log a6.

A "31. The continuity of the Logarithm.

It will now be shewn that, when a is positive. Log a is a continuous
function of a.

Let Log a = jp, Log (a + A) = .r + k

so that e =a, e'' =a-Vh, l + (hja) = e''.

First suppose that A>0, so that >0, and then

l + hla) = l+k+U' + ...>l + k, and so 0<k<hja,

that is to say 0<Log( + A) - Loga<A/a.

Hence, h being positive, Log (a + A) - Log can be mjide arbitrarily
small by taking k sufficiently small.

Next, suppose that A<0, so that Z-<0, and then a/ a + h) = e~''. Hence
(taking 0< -h< a, as is obviousl ' permissible) we get

al(a + h) = l + -k) + U' +...>l-k, and so - - < - 1+ a/ (a + A) = - h/
a + h)<- hja.

Therefore, whether h be positive or negative, if e be an arbitrary
positive number and if I A I be taken less than both \ a and |ae, we
have

I Log a + h) - Log a\ <e,

and so the condition for continuity \hardsectionref{3}{2}) is satisfied,

A'32. Diferentiation of the Logarithm.

Retaining the notation of § A-31, we see, from results there proved,
 that, if h- O a being fixed), then also k ~Q. Therefore, when a>0,

(ILosa,. k 11

Since Log 1=0, we have, by \hardsubsectionref{4}{1}{3} example 3,

Loga= I ~i dt.

* Many mathematicians define the Logarithm by the integral formula
given in § A.-32. The reader should consult a memoir by Hurwitz Math.
Ann. lxx. (1911), pp. 33-47) on the founda- tions of the theory of the
logarithm.

t This is in agreement with the notation of most text-books, in which
Log denotes the principal value (see § A-6) of the logarithm of a
complex number.

%
% 584
%

A'33. The expansion of Log (1 + ) in powers of a. From § A "32 we have

Log (l + a) =i" I + t)~idt J >

= r t+f -... + -y>-H'- + -)"(" i+t)- dt

= a - ia- + itf'' -... + (-)"-!- a" + /?,

n

where / = (\ )" /"% (i +<)-i c .

Now, if - l<a<l, we have

i nl i t" l-\ a ) - dt

= |a|" + i (n+l)a-| i) -' -- as ?t - ac .

feence, when - !< <!, Log(l + a) can be expanded into the convergent
series*

X

Log(l+ ) = o--ia2 + ia3-...= 2 (-)"-! "/?;.

n = l If a=+l,

|/?,J=/ i (H-;)-it (;< I i;"rf = (>i + l)-i- Oas n-9-x, ./ ./

so the expansion is valid when a= + 1; it is not valid when = - 1.
Example. Shew that lim (l+-j =e.

[We have Jm. n log (l + 1) = Jiu. ( -, + sTT " -)

= 1,

and the result required follows from the result of § A-2 that lim e- =
e ]

A*4. The definition of the sine and cosine.

The functions t sin z and cos z are defined analytically by means of
power series, thus

23 25 CO /\ .) 22 + l

these series converge absolutely for all values of z (real and
complex) by \hardsubsectionref{2}{3}{6}, and so the definitions are valid for all Vcilues
of z.

On comparing these series with the exponential scries, it is ap )arcnt
that the sine and cosine are not essentially new functions, but they
can be expressed in terms of exponential functions by the equations J

2i sin z = exp iz) - exp ( - iz), 2 cos 2 = exp (iz) + exp ( - iz).

* This method of obtaining the Logarithmic expansion is, in effect,
due to Walhs, P]iil. Trans. 11. (1668), p. 754.

t These series were given by Newtou, Dc Anahjsi... (1711), see § A'22
footnote. The other trigouometrical functions are defined in the
manner with which the reader is famiHar, as quotients and reciprocals
of sines and cosines.

X These equations were derived by Euler [they were given in a letter
to Jobann liernoulH in 1740 and pubhshed in the Hist. Acad. Berlin, v.
(1749), p. 279] from the geometrical definitions of the sine and
cosine, npon which the theory of the circular functions was tlien
universally based.

%
% 585
%

It is obvious that sin z and cos z are odd and even functions of z
respectively; that is to say

sin ( - 2)= -sin 2, cos ( - 2) = cos 2.

A*41. The fundaraental properties of sin z and cos z.

It may be proved, just as in the case of the exponential function (§
A"2), that the series for sin 2 and cos 2 conyerge uniformly in any
bounded domain of values of z, and con- sequently that sin 2 and cos 2
are continuous functions of 2 for all values of 2. Further, it may be
proved in a similar manner that the series

, 22 z* 1-3-1 + 5--- defines a continuous function of 2 for all values
of 2, and, in particular, this function is continuous at 2=0, and so
it follows that

lim (2~ sin2) = L

A "42. The additio7i-theorems for sin 2 and cos 2.

By using Euler's equations ( A*4), it is easy to prove from properties
of the exponential function that

sin (2 + 22) = sin z cos 23 + cos 2 sin z and cos z + 22) - cos 21 cos
22 - sin z sin 22;

these results are known as the additiorit-theorems for sin 2 and cos
2.

It may also be proved, by using Euler's equations, that

sin2 2 + cos-2 = l.

By means of this result, sin(2j + 22) can be expressed as an algebraic
function of sin2i and sin 22, while cos (21 + 22) can similarly be
expressed as an algebraic function of cos 21 and cos 22; so the
addition-formulae may be regarded as addition-theorems in the strict
sense (cf. §§ 20-3, 22-732 note).

By differentiating Euler's equations, it is obvious that

(/sin 2 dcoaz

- J - = cos 2, - 5 - = - sm 2. dz dz

Example. Shew that

sin 2i = 2 sin 2 cos 2, cos 22 = 2 cos- 2 - 1;

these results are known as the duplication-formulae.

A'5. The periodicity of the exponential function.

If 2i and 22 are such that exp2j = exp22, then, multiplying both sides
of the equation by exp( - 22), we get exp (21 - 22) = !; and writing
y for z - Z2, we see that, for all values of 2 and all integral values
of n,

exp (2 -f- ny) = exp 2 . (exp y)'' = exp 2.

The exponential function is then said to have period y, since the
effect of increasing z by y, or by an integral multiple thereof, does
not affect the value of the function.

It will now be shewn that such numbers y (other than zero) actually
exist, and that all the numbers y, possessing the property just
described, are comprised in the expression

2mvi, ( =±1, ±2, ±3, ...)

where it is a certain jjositive number* which happens to be greater
than 2v'2 and less than 4.

* The fact that tt is an irrational number, whose value is 3'141-59...
, is irrelevant to the present investigation. For an account of
attempts at determining the value of tt, concluding with a proof of
the theorem that it satisfies no algebraic equation with rational
coefficients, see Hobson's monograph Squaring the Circle (1913).

%
% 586
%

A*51. The sold tion of the equation e\ \ >y=.

Let y = a + ifi, where a and /3 are real; then the problem of solving
the equation expy=l is identical with that of solving the equation

exp . expi/3=l. Comparing the real and imaginary parts of each side of
this equation, we have

expa.cos =l, expo, sin j3=0. Squaring and adding these equations, and
using the identity cos-/3 + sin2/3= 1, we get

exp2a=l. Xow if a were positive, exp 2a would be greater than 1, and
if a were negative, exp 2a would be less than 1; and so the only
possible value for a is zero. It follows that cos /3 = 1, sin 3 = 0-

Now the equation sin/ii = is a necessary consequence of the equation
cos = l, on account of the identity cos2/3 + sin- i3=l. It is
therefore sufficient to consider solutions (if such solutions exist)
of the equation cos/3 = l.

Instead, however, of considering the equation cos/3 = l, it is more
convenient to consider the equation* cos 07=0.

It will now be shewn that the equation cos.r=0 has one root, and only
one, lying between and 2, and that this root exceeds 2; to prove
these statements, we make use of the following considerations :

(I) The function cos.r is certainly continuous in the range O a' 2.

(II) When A- '2, we have +

1- - >0 -\ \ >0 \ \ \ >o

2! ' 4! 6! ' 8! 10!= ' "'

and so, when .i- y'2, cos x > 0. (Ill) The value of cos 2 is

2<' / 4 \ 210 / 4 \

- + S-72o('-7:Ti)-IO!('-l-Tn-2)--=-i--< -

(IV) WhenO<.r 2,

sin.v .v"\ A''* x \, x',

- = 0-6) + 12oO-6r7) + ->l-6 ' and so, when x 4. 2, sin x \ x.

It follows from (II) and (III) combined with the results of (I) and of
3-63 that the equation cosa'=0 has at least one root in the range v 2
<.i'< 2, and it has no root in the range \$ x J±

Further, there is oiot more than, one root in the range J2<x<2; for,
suppose that there were two, Xi and X2 x.2>Xi); then
0<.r2-A'j<2-y'2<l, and sin (a% - Xij - sin .Cg cos x - sin Xi cos x =
0, and this is incompatible with (IV) which shews that sin (0: 2 - -
'i) M' 2~' 'i)-

The equation cos:r=0 therefore has one and onli/ one root lying
betiveen and 2. This root lies between 2 and 2, and it is called \ n;
and, as stated in the footnote to § A"5, its actual value happens to
be 1 "57079....

* If cos.r = 0, it is an immediate consecjuence of the
duplication-formulae that cos2.r= -1 and tht'uce that co8 4.t = 1, so,
if x is a solution of cos.c = 0, 4.t is a solutiou of cos/3 = l.

t The symbol may be replaced by > except when x =, 2 in the first
place where it occurs, and except when x = in the other places.

%
% 587
%

From the addition-formulae, it may be proved at once by induction that

cos ?;7r = ( - 1)", sin?i7r = 0, where ?i is any integer.

In particular, cos2)i7r = l, where n is any integer.

Moreover, there is no value of /3, other than those values which are
of the form 2?i7r, for which cosj3=l; for if there were such a value,
it must be real*, and so we can choose the integer ra so that

- TT 27mr -I3<7r.

We then have

sin I WITT - /3 I = ±sin(m -l/3)=±sini =±2~2(l\ cos/3)i = 0,

and this is inconsistent t with sin j wtt - /3 | J | j/itt - i/3 [
unless /3 = 2mTr.

Consequently the numbers 2n7r, (/i = 0, ±1, ±2,...), a/id no others,
hare their cosines equal to unity.

It follows that a positive number n exists such that exps has period
iri and that exp2 has no period fundamentally distinct from liri.

The formulae of elementary trigonometry concerning the periodicity of
the circular functions, with which the reader is already acquainted,
can now be proved by analytical methods without any difficulty.

Example . Shew that sini r is equal to 1,-not to - L

Example 2. Shew that tan.i->.z; when 0<x<\ Tr.

[For cos.i'>0 and

sm.r - orcosA':

---A n-'i- 7- 1, l)!i 4>i + lj'

71 = 1 (4

and every term in the series is positive.]

x 77 v 25 x x

Example 3. Shew that 1 - V + ir7 ~; i positive when x =,r, and that
1 - V + Jrr

2 24 / 20 lb z 24

vanishes when .r = (6 - 2 /3)2 = 1-5924...; and deduce that J

3-125 <7r<3-185.

A'52. The solution of a pair of trigonometncal equations. Let X, /x be
a pair of real numbers such that X + jit l. Then, if X=|= - 1, the
equations

cos X = X, sin X = i have an infinity of solutions of which one and
only one lies between § - tt and tt.

First, let X and fi be not negative; then \hardsubsectionref{3}{6}{3}) the equation cos.r
= X has at least one solution Xi such that O .x-j iTr, since cos = 1,
cos 7r=0. The equation has not two solutions in this range, for if Xi
and x.2, were distinct solutions we could prove (cf. § A-51) that sin
(.t'l - .i"2) = 0, and this would contradict § A-51 (IV), since

0<| 2-- i I <i7r<2. Further, sin.ri= + (1 -coh' X])= -1- /(1 -- ) = M5
so x is a solution of both equations.

* The equation cos/3=l implies that exp i(i - l, and we have seen that
this equation has no complex roots.

+ The inequality is true by (IV) since j irnr - |/3 | : i7r<2.

+ See De Morgan, A Budget of Purado.ves (London, 1872), pp. .S16 et.
sfq., for reasons for proving that 7r>3 .

§ If X=: - 1, ±7r are solutions and there are no others in the range (
- tt, tt).

%
% 588
%

The equations have no solutions iu the ranges ( - tt, 0) and ( tt, tt)
since, in these ranges, either sin x or cos x is negative. Thus the
equations have one soUition, and only one, in the range ( - rr, tt).

If X or fi (or both) is negative, we may investigate the equations in
a similar manner; the details are left to the rea<ier.

It is obvious that, if x- is a solution of the equations, so also is
Xy + 'iniY, where n is any integer, and therefore the equations have
an infinity of real solutions.

A'521. The principal solution of the trigonometrical equations.

The unique solution of the equations cos.t' = X, sin,t-=/i. (where X +
fx' = ) which lies between - it and n is called the principal
solution*, and any other solution differs from it by an integer
multiple of Stt.

The principal valuei of the argument of a complex number (=1=0) can
now be defined analytically as the principal solution of the equations

I z \ cos ( = / (2), j 2 1 sin = / z), and then, if = 1 2 1 . (cos + /
sin 6),

we must have d = (P + 2mr, and 6 is called a value of the argument of
j, and is written arg2 (cf i 1-5).

A "522. The continuity of the argument of a complex variable.

It will now be shewn that it is possible to choose such a value of the
argument 6 (z), of a complex variable z, that it is a continuous
function of z, provided that z does not pass through the value zero.

Let 2,1 6 given 'alue of 2 and let 6 be any value of its argument;
then, to prove that 6 (z) is continuous at Zq, it is sufficient to
shew that a number 61 exists such that i=arg2i and that j 1 - 0 1 can
be made less than an arbitrary positive number e by giving 1 21 - 20 1
any value less than some positive number t).

Let fo = '-"o + Vo > 1 = . 'i + >i

Also let j 2i - So 1 l e chosen to be so small that the following
inequalities are satisfied J : (I) I .Ti - 0 i < 2 I - 0 1 5 provided
that Xq =t= 0, (II) I ?/i - 2/0 i < 2 ! J/o I, provided that 3/0 =t=
0, (III) |Xi-.%|<Je|2(,|, i?/i-3/o|<ie|2ol. From (I) and (II) it
follows that x Vi and y f/i are not negative, and

.'ToA-i a'o /o i > i.yo so that xo .vi + ?/o2/i J i 20 j 2.

Now let that value of 61 be taken which differs from 0 by less tlian
it; then, since x, and Xi have not opposite signs and yo f 'id 1/1
have not opposite signs J, it follows from the solution of the
equations of A"52 that and 0 differ by less than n.

Now tan( i-,.) = ':i ",

* If \= - 1, we take +7r as the principal solution; of. p. 9.

t The term principal value was introduced in 1845 by Bjorling; see
the Archiv der Math, und Fhijs. ix. (1847), p. 408.

X (I) or (II) respectively is simply to be suppressed in the case when
(i) .Tq O, or when (ii) 2/0 = 0.

§ The gtometrical interpretation of these conditions is merely that r
and z are not in different quadrants of the plane.

%
% 589
%
aud so (§ A"51
example 2),

  I o(yi-yo)-y o(' i-- o) I

  2 I 2o |~ I 0 I  I yi- o ! + l 'o !  I - 'i - 0 ! . But I A'o 1 ko
! and also | yo j | :o |; therefore

I (9i - 6>o i =\$ 2 I 2o 1~ | yi- ?/o I + 1 1 - . 0 | <e-

Further, if we take | 2i - zq 1 l ss than  - ] J 'o |, (if o + 0)
and i [ yo i ? (if o + 0) and | e j g I, the inequalities (I), (II),
(III) above are satisfied; so that, if ri be the smallest of the
three numbers * i | a-o |, -I ! yo ! > i ko i j by taking j Sj - q 1
< '?, we have | i - o I < <; and this is the condition that 6 z)
should be a continuous function of the complex ariable z.

A '6. Logarithms of comflex numhers. The number f is said to be a
logarithm of z if 2 = e

To solve this equation in (, write = + '7? where and rj are real; and
then we have

z = e (cos Tj + i sin ?;).

Taking the modulus of each side, we see that | 3| = e, so that (§
A-3), = Log \ z\; and

then

z=\ z\ (cos 7/ + 1 sin/;),

so that T] must be a value of argz.

The logarithm of a complex number is consequently a many-valued
function, and it can be expressed in terms of more elementary
functions by the equation

log z = Log j 2 1 4- 1 arg z.

The continuity of logs (when s4=0) follows from § A'31 and § A'522,
since \ z\ is a continuous function of z.

The differential coefficient of any particular branch of logs \hardsectionref{5}{7})
may be determined as in § A*32; and the expansion of § A"33 may be
established for log (1 + ) when | a | < 1.

Corollary. If a be defined to mean e'iog, a is a continuous function
of z aud of a when a=t=0.

A'7. The analytical definition of an angle.

Let 1, 22, 23 be three complex numbers represented by the points Pj,
P., F3 in the Argand diagram. Then the angle between the lines (§
A"12, footnote) PiPo and P1P3 is defined to be any value of arg (23 -
Zj) - arg (22 - Zi).

It will now be shewn f that the area (defined as an integi'al), which
is bounded by two radii of a given circle and the arc of the circle
terminated by the radii, is proportional to one of the values of the
angle between the radii, so that an angle (in the analytical sense)
jjossesses the propei'ty which is given at the beginning of all
text-books on Trigonometry |.

* If any of these numbers is zero, it is to be omitted.

t The proof here given applies only to acute angles; the reader
should have no difficulty in extending the result to angles greater
than lir, and to the case when OX is not one of the bounding radii.

X Euclid's definition of an angle does not, in itself, afford a
measure of an angle; it is shewn in treatises on Trigonometry (cf.
Hobson, Plane Trigonometry (1918), Ch. i) that an angle is measured by
twice the area of the sector which the angle cuts off from a unit
circle whose centre is at the vertex of the angle.

%
% 590
%

Let (o-'i, yi) be any point (both of whose coordinates are positive)
of the circle x' - y- = a? a>0). Let 6 be the principal value of arg(
i + iyi), so that 0<6 <Itt. Then the area bounded by OX and the line
joining (0, 0) to x\, i/y) and the arc of the

circle joining .Vi, y ) to (o, 0) is / f x)dx, where*

./ '

/( )=a;tan (0 .r a cos ),

/ x) = a - x )- (a cos d x a), if an area be defined as meaning a
suitably chosen integral (cf. p. 61).

It remains to be proved that I f(x) dx is proportional to 6.

J "

fa /"a cose fa,

Now I Z f(x) dx = I X tan 6 dx + (a - x-)i dx

J I) Jo J (I cose

 ha- smecose + hr L a --x ')- + x a -x ) \ dx

f" - 1

= a-j a -x-) - dx

J o COS Q

= |a2 1 r (1 - -) 'id(- p' (1 - f') - - dt\

on writing x = at and using the example worked out on p. 64.

That is to say, the area of the sector is proportional to the angle of
the sector. To this extent, we have shewn that the popular conception
of an angle is consistent with the analytical definition.

* The reader will easily see the geometrical interpretation of the
integral by drawing a figure.


In this chapter, we will introduce some basic property of approximation theory and analyze the approximation property for some activation functions.

	The adaptive FEM we have studied before is indeed using linear
	subspaces. For a given and fixed basis, select the best n term to
	approximate a function. The non-linear approximation theory (by
	DeVore) is to relax the smoothness of function to achieve the optimal
	rate $n^{-1/d}$ but won't improve the rate.

\input{6DL/Fourier}
\input{6DL/PolyWeierstrass}
\input{6DL/DNN_Qualitative0}
%\input{6DL/DNN_Qualitative}  % this is actually for
%quantitative/asymptotic analysis
\section{Asymptotic approximation properties}
\input{6DL/SampleLemma}
\input{6DL/Representations}
\input{6DL/Barron-L2}
%\input{6DL/Barron-Hk}
\chapter{Optimal approximation estimates for minimally smooth functions}
\input{6DL/Entropy}
\chapter{High order approximation rates for cosine and ReLU$^k$ networks}
The presentation in this section follows the paper \cite{siegel2020high}.
\input{6DL/CosineApprox}
\input{6DL/ReLUkApprox}
\input{6DL/CosineLower}


\documentclass[12pt,a4paper]{article}
\usepackage[utf8]{inputenc}

\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}

\usepackage{graphicx}
\usepackage{caption}

\setlength{\parindent}{0em}
\setlength{\parskip}{1em}

\begin{document}
\title{Consensus Algorithm}
\author{Frederic Laudarin}
\date{November 1, 2020}
\maketitle

\section{Framework}

This document exposes an algorithm for decision making within a group. It is supported by the framework where:
\begin{itemize}
\item There is a finite set of alternatives an
\item The size of the group is finite
\item Each member of the group must rank every alternatives
\item Several alternatives may have the same rank
\end{itemize}
Once the ranking is carried out by every member. The algorithm must provide a set of alternative which is optimal in regard of all the individual ranks made in the group.

The base principle of the algorithm is for each member to assign payoffs to alternatives depending on their ranking and to sum the payoffs of every alternative from every player. The alternatives having the best payoff over the group constitute the optimal set of alternatives.

\section{Basic algorithm}

Let $\Omega=\{A_i\}_{i=1}^N$ denote the set of $N\in\mathbb{N}^*$ alternatives. The ranking assigned to the alternative $A_i$ by a group's member $(m)$ is noted $r_i^{(m)}$. In the context of a member $(m)$ the alternatives are classified by ranking values in classes $\{\mathcal{C}_k^{(m)}\}_{k=1}^{K^{(m)}}$ with $K^{(m)}\in\mathbb{N}^*$. The rank of every alternative in the class $\mathcal{C}_k^{(m)}$ is obviously $k$, the preferred rank being $k=1$. The cardinality of a class $\mathcal{C}_k^{(m)}$ is $\mathrm{card}(\mathcal{C}_k^{(m)}) = N_k^{(m)}$ so that:
\begin{equation}
\sum_{k=1}^{K^{(m)}}{N_k^{(m)}} = N
\end{equation}
For the sake of readability, the upper index $(m)$ will be avoided in the following, the notations previously introduced will implicitly refer to a member $(m)$.

Each class $\mathcal{C}_k$ has a frequency $f_k$:
\begin{equation}
f_k = \frac{N_k}{N}
\end{equation}
As shown in the set of pairs $\{(k,f_k), k=1..K\}$ can be considered as a probability distribution as:
\begin{equation}\label{eq_sum_frequencies}
\sum_{k=1}^K{f_k} = 1
\end{equation}

\begin{figure}[h]
\centering
\includegraphics[scale=0.6]{distribution.png}
\caption{Distribution of alternatives}
\label{fig:distribution}
\end{figure}
The payoff $v_k$ assigned to alternatives in class $\mathcal{C}_k$ is defined as the probability of getting an alternative $A_i\in\Omega$ such as $A_i\in\mathcal{C}_l$ with $l>k$:
\begin{equation}\label{eq_member_payoff}
v_l = v\left(A_i\in\mathcal{C}_l\right)=\mathrm{Pr}\left(\left\lbrace A_i\in\mathcal{C}_l, l>k\right\rbrace\right)=\sum_{l>k}{f_l}
\end{equation}
This is the payoff value $v^{(m)}(A_i)$ of the member $(m)$ for the alternative $A_i$.
As a consequence, an alternative being in the last class $\mathcal{C}_K$ has a payoff of 0. If a member of the group has no preference then they classify all the alternatives in one unique class $\mathcal{C}_1$ and every alternative is assigned a zero payoff value.

Eventually the payoff value of an alternative to the group is the sum of values assigned by each member:
\begin{equation}\label{eq_alternatives_payoff}
v\left(A_i\right)=\sum_{m}{v^{(m)}(A_i)}
\end{equation}

\section{Basic algorithm - Example}

The group is composed of two persons namely Manon and Martin. They must choose they need to decide on the location of their summer holidays. They consider the following alternatives:
\begin{itemize}
\item Australia
\item California US
\item Costa Rica
\item Germany
\item Hungary
\item Japan
\item Thailand
\item UK
\item Venezuela
\end{itemize}

\subsection{Ranking of alternatives}
Manon makes the following ranking:
\begin{table}[h]
\centering
\begin{tabular}{|c|l|}
\hline
\textbf{Rank}&\textbf{Alternatives}\\
\hline
1 & Thailand, Venezuela \\
2 & Australia, Costa Rica \\
3 & Hungary \\
4 & California US, Germany, UK \\
5 & Japan \\
\hline
\end{tabular}
\end{table}

and Martin this one:

\begin{table}[h]
\centering
\begin{tabular}{|c|l|}
\hline
\textbf{Rank}&\textbf{Alternatives}\\
\hline
1 & Japan, Germany, Hungary \\
2 & Australia, California US \\
3 & UK, Thailand, Venezuela, Costa Rica \\
\hline
\end{tabular}
\end{table}

\subsection{Classes}

The alternatives are classified by ranking for each member of the group. The frequency of each class is calculated.

For \textsl{Manon}:

\begin{table}[h]
\centering
\begin{tabular}{|c|c|c|}
\hline
\textbf{Class}&\textbf{Frequency} &\textbf{Alternatives}\\
$\mathcal{C}_k$ & $f_k$ & $A_i$ \\
\hline
$\mathcal{C}_1$ & $2/9$ & Thailand, Venezuela \\
$\mathcal{C}_2$ & $2/9$ & Australia, Costa Rica \\
$\mathcal{C}_3$ & $1/9$ & Hungary \\
$\mathcal{C}_4$ & $1/3$ & California US, Germany, UK \\
$\mathcal{C}_5$ & $1/9$ & Japan \\
\hline
\end{tabular}
\end{table}

\clearpage
For \textsl{Martin}:
\begin{table}[ht]
\centering
\begin{tabular}{|c|c|c|}
\hline
\textbf{Class}&\textbf{Frequency} &\textbf{Alternatives}\\
$\mathcal{C}_k$ & $f_k$ & $A_i$ \\
\hline
$\mathcal{C}_1$ & $1/3$ & Japan, Germany, Hungary \\
$\mathcal{C}_2$ & $2/9$ & Australia, California US \\
$\mathcal{C}_3$ & $4/9$ & UK, Thailand, Venezuela, Costa Rica \\
\hline
\end{tabular}
\end{table}

\subsection{Payoff}

The following table gives the payoff values of Manon's ranking classes:

\begin{table}[ht]
\centering
\begin{tabular}{|c|c|c|}
\hline
\textbf{Class}&\textbf{Frequency} &\textbf{Payoff}\\
$\mathcal{C}_k$ & $f_k$ & $v_k$ \\
\hline
$\mathcal{C}_1$ & $2/9$ & 7/9 \\
$\mathcal{C}_2$ & $2/9$ & 5/9 \\
$\mathcal{C}_3$ & $1/9$ & 4/9 \\
$\mathcal{C}_4$ & $1/3$ & 1/9 \\
$\mathcal{C}_5$ & $1/9$ & 0 \\
\hline
\end{tabular}
\end{table}

And this one the payoff values of Martin's ranking classes:

\begin{table}[ht]
\centering
\begin{tabular}{|c|c|c|}
\hline
\textbf{Class}&\textbf{Frequency} &\textbf{Payoff}\\
$\mathcal{C}_k$ & $f_k$ & $v_k$ \\
\hline
$\mathcal{C}_1$ & $1/3$ & 2/3  \\
$\mathcal{C}_2$ & $2/9$ & 4/9 \\
$\mathcal{C}_3$ & $4/9$ & 0 \\
\hline
\end{tabular}
\end{table}

\clearpage
Now the payoff value of alternatives for Manon and Martin can be combined as in equation (\ref{eq_alternatives_payoff}):
\begin{table}[ht]
\centering
\begin{tabular}{|c|c|c|c|}
\hline
\textbf{Alternative}&\textbf{Manon}&\textbf{Martin}&\textbf{Payoff}\\
$A_i$ & $v^{\mathrm{Manon}}(A_i)$ & $v^{\mathrm{Martin}}(A_i)$ & $v(A_i)$ \\
\hline
Hungary & $4/9$ & $2/3$ & $10/9$ \\
Australia & $5/9$ & $4/9$ & $1.$ \\
Thailand & $7/9$ & $0$ & $7/9$ \\
Venezuela & $7/9$ & $0$ & $7/9$ \\
Germany & $1/9$ & $2/3$ & $7/9$ \\
Japan & $0$ & $2/3$ & $2/3$ \\
California US & $1/9$ & $4/9$ & $5/9$ \\
Costa Rica & $5/9$ & $0$ & $5/9$ \\
UK & $1/9$ & $0$ & $1/9$ \\
\hline
\end{tabular}
\end{table}

The payoff is maximal for \textbf{Hungary} which achieves the consensus.

\section{Algorithm with scaling}\label{section_algorithm_with_scaling}

Now, in some cases ranking the alternatives with mere ordering is not fully representative of the perception of members in the group. A member may have a greater comparative preference between two classes of alternatives. The following scale of preference is used:
\begin{itemize}
\item moderate
\item strong
\item very strong
\item extreme
\end{itemize}
The \textsl{moderate} preference is the equivalent of the hierarchy induced by the \textsl{basic algorithm}. The \textbf{intensity} of comparison is based on that scale with a integer value $I_c\in[0..3]$. The moderate intensity value is $I_c=0$ and the extreme is $I_c=3$.

A comparison with a higher intensity than moderate adds a dummy class between the two compared classes. The frequency of this dummy class increases with the intensity.

A member of the group made a ranking $\left\lbrace\mathcal{C}_k, k=1..K\right\rbrace$. Equation (\ref{eq_member_payoff}) gives the difference of payoff value between two successive classes with:
\begin{equation}
v_k-v_{k+1} = f_{k+1}>0,\ \forall k\in[1..K-1]
\end{equation}
This relation assumes that the group's member shows a \textsl{moderate} preference between every consecutive pair of classes which they ranked. Now if the \textsl{intensity} of this preference is higher between two classes $\mathcal{C}_k$ and $\mathcal{C}_{k+1}$, the difference expressed by the previous equation must be increased. The additional term is noted $\varphi_{k,k+1}>0$ so that the relation becomes:
\begin{equation}
v_k-v_{k+1} = f_{k+1} + \varphi_{k,k+1}
\end{equation}
As stated before if the preference is \textsl{moderate} ($I_c=0$) then $\varphi_{k,k+1}=0$. Here, the additional term is modeled as proportional to the frequency $f_{k+1}$:
\begin{equation}\label{eq_add_term}
\begin{split}
&\varphi_{k,k+1} = \left(a\left(I_c\right)-1\right)f_{k+1} \\
\mathrm{with}\ &a\left(I_c\right) \geq 1,\ \forall I_c\in[0..3] \\
\mathrm{and}\ &a(0)=1
\end{split}
\end{equation}
As a consequence:
\begin{equation}
v_k-v_{k+1} = a\left(I_c\right) f_{k+1}
\end{equation}
The \textsl{extreme preference} ratio is noted $a_\mathrm{ext} = a(3)$. This value is subjective and could be for instance set to 10. The intermediate values $a(1)$ and $a(2)$ are calculated from $a_\mathrm{ext}$ assuming the following expression:
\begin{equation}\label{eq_intensity_ratio}
a\left(I_c\right) = \exp(\alpha I_c)
\end{equation}
Using a power law for modeling the intensity of human feeling is not a new idea. Coefficient $\alpha$ can be determined from $a_\mathrm{ext} = a(3) = e^{3\alpha}$:
\begin{equation}\label{eq_intensity_exponent}
\alpha = \frac13\ln(a_\mathrm{ext})
\end{equation}
The integration of the intensities of preferences gives the following expression of payoff for a class $\mathcal{C}_l,$ with $l<K$ :
\begin{equation}
v_l = \sum_{k>l}{f_k} + \sum_{l\leq k<K}{\varphi_{k,k+1}}
\end{equation}
This expression is not valid as is because members in the group would not provide payoff on the same scale. The payoff values would completely loss their objectivity and members providing rankings with higher intensities in comparison would see their opinion prevail over the group. This overrating stems from that the sum of class frequencies is 1. (see equation (\ref{eq_sum_frequencies})) but:
\begin{equation}
 S = \sum_{k=1}^K{f_k} + \sum_{k=1}^{K-1}{\varphi_{k,k+1}} = 1 + \sum_{k=1}^{K-1}{\varphi_{k,k+1}} \geq 1
\end{equation}
Once the values of intensity terms $\varphi_{k,k+1}$ are determined the actual payoff value is computed by applying the corrective ratio $1/S$:
\begin{equation}\label{eq_scaled_payoff_with_intensities}
v_l = \frac1S \left( \sum_{k>l}{f_k} + \sum_{l\leq k<K}{\varphi_{k,k+1}} \right)
\end{equation}

\section{Application case with scaling}

\subsection{Reference with moderate preferences}
There are 8 alternatives \textit{A}, \textit{B}, \textit{C}, \textit{D}, \textit{E}, \textit{F}, \textit{G}, \textit{H}. A member of the group makes the following ranking:

\begin{table}[ht]
\centering
\begin{tabular}{|c|c|c|}
\hline
\textbf{Class}&\textbf{Frequency} &\textbf{Alternatives}\\
$\mathcal{C}_k$ & $f_k$ & $A_i$ \\
\hline
$\mathcal{C}_1$ & 1/4  & \textit{A}, \textit{B} \\
$\mathcal{C}_2$ & 3/8  & \textit{C}, \textit{D}, \textit{E} \\
$\mathcal{C}_3$ & 1/8  & \textit{F} \\
$\mathcal{C}_1$ & 1/4  & \textit{G}, \textit{H} \\
\hline
\end{tabular}
\end{table}

The payoff with the basic approach where every comparison of classes is associated with a moderate preference is:

\begin{table}[ht]
\centering
\begin{tabular}{|c|c|c|}
\hline
\textbf{Class}&\textbf{payoff} &\textbf{Alternatives}\\
$\mathcal{C}_k$ & $v_k$ & $A_i$ \\
\hline
$\mathcal{C}_1$ & 3/4  & \textit{A}, \textit{B} \\
$\mathcal{C}_2$ & 3/8  & \textit{C}, \textit{D}, \textit{E} \\
$\mathcal{C}_3$ & 1/4  & \textit{F} \\
$\mathcal{C}_4$ & 0  & \textit{G}, \textit{H} \\
\hline
\end{tabular}
\end{table}

\subsection{Preferences with variable intensities}

Now, the member specifies intensities $I_c$ in the preferences between the class of alternatives:

\begin{table}[ht]
\centering
\begin{tabular}{|c|l|}
\hline
\textbf{Classes}&\textbf{Prefetence}\\
$\mathcal{C}_k$ / $\mathcal{C}_{k+1}$ & $I_c(k)$ \\
\hline
$\mathcal{C}_1$ / $\mathcal{C}_2$ & very strong, $I_c(1)=2$ \\
$\mathcal{C}_2$ / $\mathcal{C}_3$ & moderate, $I_c(2)=0$ \\
$\mathcal{C}_3$ / $\mathcal{C}_4$ & strong, $I_c(3)=1$ \\
\hline
\end{tabular}
\end{table}

The payoff of class $\mathcal{C}_4$ is unchanged and remains zero. The preference of class $\mathcal{C}_3$ over $\mathcal{C}_4$ is \textsl{strong}. The additional term reflecting this intensity is given by equations (\ref{eq_add_term}), (\ref{eq_intensity_ratio}) and (\ref{eq_intensity_exponent}) with an \textsl{extreme} ratio $a_\mathrm{ext}$:
\begin{equation}
\varphi_{3,4} = \left(a\left(1\right)-1\right)f_4 = \left(\exp(\frac13\ln(10)\times1)-1\right)\frac14\approx0.2886
\end{equation}
The preference of class $\mathcal{C}_2$ over $\mathcal{C}_3$ is \textsl{moderate} so there is no additional term ($\varphi_{2,3}=0$).
The payoff of class $\mathcal{C}_4$ is unchanged and remains zero. The preference of class $\mathcal{C}_3$ over $\mathcal{C}_4$ is \textsl{very strong}:
\begin{equation}
\varphi_{1,2} = \left(a\left(2\right)-1\right)f_2 = \left(\exp(\frac13\ln(10)\times2)-1\right)\frac38\approx1.3656
\end{equation}

\subsection{Payoff}

As stated in section \ref{section_algorithm_with_scaling} the less preferred class has a payoff of 0, then $v_4=0$. For the other classes, the equation (\ref{eq_scaled_payoff_with_intensities}) applies. The scaling ratio is $S$ is:
\begin{equation}
1 + \varphi_{1,2} + \varphi_{3,4} \approx 2.6542
\end{equation}
The payoff value of class $\mathcal{C}_3$ is:
\begin{equation}
v_3 = \frac1S \left( f_4 + \varphi_{3,4} \right) \approx 0.2029
\end{equation}
The payoff value of class $\mathcal{C}_2$ is:
\begin{equation}
v_2 = \frac1S \left( f_3 + f_4 + 0 + \varphi_{3,4}  \right)\approx 0.2500
\end{equation}
Eventually the payoff value of class $\mathcal{C}_2$ is:
\begin{equation}
v_1 = \frac1S \left(f_2 + f_3 + f_4 + \varphi_{1,0} + 0 + \varphi_{3,4}  \right)\approx 0.9058
\end{equation}

The table bellow compares the payoff with the \textsl{base algorithm} and with preferences with increased intensities:

\begin{table}[ht]
\centering
\begin{tabular}{|c|c|c|c|}
\hline
\textbf{Class}&\textbf{payoff (base)}&\textbf{payoff (base)} &\textbf{Alternatives}\\
$\mathcal{C}_k$ & $v_k$ & $v_k$ & $A_i$ \\
\hline
$\mathcal{C}_1$ & 0.75 & 0.906 & \textit{A}, \textit{B} \\
$\mathcal{C}_2$ & 0.375 & 0.250 & \textit{C}, \textit{D}, \textit{E} \\
$\mathcal{C}_3$ & 0.25 & 0.203 & \textit{F} \\
$\mathcal{C}_4$ & 0 & 0. & \textit{G}, \textit{H} \\
\hline
\end{tabular}
\end{table}

\end{document}


%\input{head.tex}

%\begin{document}

\section{Basic Concepts}

Most of the following concepts apply to arbitrary groups.

\subsection{Groups and Subgroups}

\defn{
group, Abelian, conjugate, power, finite, cyclic, subgroup, generate, order, complex product, coset, coset, transversal, p-group.
}

\prop{
bijective mappings, cyclic groups, subgroup, set-generate, complex product, order of c.p., Lagrange Thm, order of g and G, transversal, complement, Dedekind Identity.
}

\subsection{Homomorphisms and Normal Subgroups}

\defn{
homomorphism, im, ker, epi/mono/endo/iso/automorphism, normal, simple, factor group, subnormal, section.
}

\prop{
inverse homo, ker is normal, normal iff, natural homo, Homo Thm, Iso Thm*2, subnormal.
}

\subsection{Automorphisms}

\defn{
AutG, InnG, Z(G), characteristic, X-inv, X-homo.
}

\prop{
cyclic G/Z(G) implies Abelian G, char is trans.
}

\subsection{Cyclic Groups}

\defn{$C_n$}

\prop{
subgroups of $\Z$, finite cyclic groups and their subgroups, char, cyclic p-groups, Abelian simple groups.
}

\subsection{Commutators}

\defn{
commutator subgroup, perfect.
}

\prop{
comu with homo, smallest s.t. Abelian factor, perfect factor, $[x,yz]=[x,z][x,y]^z$, [X,Y] is normal in $<X,Y>$, Three-subgroups Lem.
}

\subsection{Products of Groups}

\defn{
internal/external direct product, central product, internal/external semidirect product, involutions, dihedral group.
}

\prop{
in/ex iso, on Z(G) G' G/N, on normal N, internal structures $G/\bigcap_i N_i \cong G/N_1\times\cdots\times G/N_n$, prod of normal subgs of relative prime order is direct, on elements, central product on G/Z[G], factorization and product of elements in a semiderict product, dihedral groups are semidirect products.
}

\subsection{Minimal Normal Subgroups}

\defn{
minimal normal subgroups.
}

\prop{
properties of a min-normal subg, product of min-normal subgs, factorization(structure for Abelian versus uniqueness for non-Abelian).
}

\subsection{Composition Series}

\defn{
refine normal(rep. subnormal)series to chief(esp. composition) series, solvable group, X-section, X-composition series, X-simple.
}

\prop{
Jordan-Holder Thm.
}

%%%%%%%%%% below are solutions of exercises %%%%%%%%

\begin{exerlist}

\exer{hahaha}{jajaja}

\end{exerlist}

%\end{document}


\graphicspath{{Pics/}}
\subsection{Parallel lines to the $ A $-median}

\den{Skeleton Diagram}{
    In a triangle $ABC$, $M$ is the center of $BC$. $B', C'$ are on $\odot
    ABC$ such that $BB'\parallel CC' \parallel AM$. Let $R$ be the point on
    $BC$ such that $\odot B'C'R$ is tangent to $BC$. Draw the two circles
    through $BB'$ and $CC'$ that are tangent to $BC$. Also let $S = BC \cap
    B'C'$, and let $G$ be such that $SG$ is tangent to $\odot ABC$ at $G$.
    Also let $ M_a, M_A $ be the minor and major arc midpoints of $ \widehat{BC} $.
}

\figdf{1}{apmo_2019_p3_and_IGO_2019_p4_0}{Skeleton}

\lem{}{
    $M_A, R, G$ are collinear
}

\begin{prooof}
    Let $R' = GM_A\cap BC$. Then do some angle chansing to show that $SR' = SG$.
\end{prooof}

\den{Further Construction}{
    Now let $ M_AR\cap AM = H, M_AR\cap\odot ABC = G $.
        Let $ \omega_b, \omega_c $ be the circles through $ BB' $ and $
            CC' $ and tangent to $ BC $.
        Let $ I $ be the intersection of the line through $ M_a $
            parallel to $ AM $ and $ \omega_x $.
        Let $ J=AM\cap\omega_x $.
        Let $ J\cap\omega_b=X, J\cap\omega_c=Y $.
        Let $ AS\cap \odot ABC = F $.
}

\lem{}{$ SG $ is tangent to $ \odot ABC $}

\lem{}{$ R, M, M_a, G $ is cyclic, call it $ \omega_x $}

\lem{}{$ I, G, O $ are collinear, and $ IG=IJ $}

\lem{}{$ BX, CY, AM, RM_A$ are concurrent.}

\figdf{1}{apmo_2019_p3_and_IGO_2019_p4}{}

\prob{}{IGO 2019 Advanced P4}{H}{
    $ XHYG $ is cyclic, and tangent to $
    \omega_b, \omega_c $ and $ \odot ABC $. And $ I $ is the center of this
    circle.
}

\figdf{1}{apmo_2019_p3_and_IGO_2019_p4_1}{}

\prob{https://artofproblemsolving.com/community/c6h1854153p12519663}{APMO 2019
    P3}{MH}{
    A variable point $P$ is selected in the line segment $AM$. The
    circumcircles of triangles $BPM$ and $CPM$ intersect $\Gamma$ again at points
    $D$ and $E$, respectively. The lines $DP$ and $EP$ intersect (a second time)
    the circumcircles to triangles $CPM$ and $BPM$ at $X$ and $Y$, respectively.
    Prove that $ AFXY $ is cyclic.
}


\problemname{Pipe Rotation}

\noindent The four ninja turtles: Leonardo, Donatello, Michelangelo, and Raphael are seeking a new home in Manhattan, New York City. The turtles don't like sudden dead-ends in their home. Fortunately, the government recently installed a new sewage system where pipes can be rotated! The turtles needs your help finding a suitable home, so they're willing to provide you a grid of the current layout of the sewage system.\\

The grid consists of $N$ rows and $M$ columns. The cell $G_{i,j}$ will consist of one of four pipes, encoded as a letter between ``\texttt{A}" and ``\texttt{D}". These pipes can be rotated by any multiple of $90$ degrees:\\

\noindent
\begin{minipage}{0.25\textwidth}
  \includegraphics[width=0.95\textwidth]{pipe_rotation_1.png}
\end{minipage}
\begin{minipage}{0.75\textwidth}
  \begin{itemize}
    \item (\texttt{A}) Nothing
    \item (\texttt{B}) Straight pipe (pipes leaving through two opposite edges)
    \item (\texttt{C}) Elbow-shaped pipe (pipes leaving through two adjacent edges)
    \item (\texttt{D}) Four-way pipe (pipes leaving through all four edges)
  \end{itemize}
\end{minipage}
\vspace{10pt}

Determine whether it's possible to rotate the cells such that the pipes all line up with one another. In particular, for each edge shared by a pair of adjacent cells, there must either be a pipe on both sides of that edge, or on neither side. And for each each of the $2 \cdot (N + M)$ outer edges of the grid, there must be no pipe leaving through that edge. Below are examples:

\begin{figure}[h]
  \centering
  \begin{minipage}[t]{0.3\textwidth}\centering
      \includegraphics[width=0.95\textwidth]{pipe_rotation_2}
      Fig 1. Invalid example,\\ two sudden dead-ends.
  \end{minipage}
  \hspace{30pt}
  \begin{minipage}[t]{0.3\textwidth}\centering
      \includegraphics[width=0.95\textwidth]{pipe_rotation_3}
      Fig 2. Valid example,\\ no sudden dead-ends.
  \end{minipage}
\end{figure}

\section*{Input}
The first line of input consists of two space-separated integers, $N$ and $M$ ($1 \leq N, M \leq 100$).\\
$N$ lines follow, the $i$th of which consists of $M$ characters $G_{i,1}, G_{i,2}, G_{i,j}, \dots, G_{i,M}$ and $G_{i,j} \in \{A,B,C,D\}$ (for $i = 1..N$).

\section*{Output}
Print, on a single line, a single string, either ``\texttt{Possible}" if it's possible to produce a valid configuration of pipes, or ``\texttt{Impossible}" otherwise.\\


\chapter{Conclusion}
In this thesis we have examined how Bayesian inference can be used in neural networks using Markov chain Monte Carlo sampling and how these and non-Bayesian neural networks perform regression and classification. The most essential difference between these two approaches of using neural networks is that Bayesian neural networks require a prior distribution for its weights and use sampling procedures to generate a distribution for its weights and predictions, while non-Bayesian neural networks use optimization algorithms, such as the gradient-based optimization algorithms described in section \ref{sec:gradient_optimization}, to learn a single estimate of the weights that result in predictions that provide the least amount of loss.
\\
\\
The focal point of the examination of the Bayesian neural networks has been how to efficiently sample from the posterior distribution for the weights using Markov chain Monte Carlo methods. We analyzed this by first providing a simple example of a Bayesian neural network that took weights sampled from a prior and accepted them with a probability proportional to the likelihood of the produced results, a sort of rejections sampling illustrated by \cite{neal2012bayesian}. This provided the motivation for examining more efficient sampling methods such as the Markov chain Monte Carlo methods. One such method was the Metropolis algorithm that has the inefficiency of exploring the distribution with a random walk behavior. For this reason we examined the Hamiltonian Monte Carlo, which combine the principles of the Metropolis algorithm with Hamiltonian dynamics for a better exploration of the distribution. This turns out to have the inefficiency of sometimes performing "U-turns", meaning that it has the possibility of returning to the previous sample-point of the Markov chain and thus make highly correlated samples.
The No-U-Turn Hamiltonian Monte Carlo is an extension to Hamiltonian Monte Carlo, which prevents such a U-turn and also make adaptive choices to the size and the number of steps taken by the Markov chain before sampling. We conclude our examination of Bayesian neural networks by covering the effect of the prior choice and by suggesting general schemes for choosing a prior distribution.
\\
\\
We end our thesis with an evaluation and illustration on how Bayesian and non-Bayesian neural networks work in practice. We do this with varying design choices to show how elements such as a hierarchical prior and different choice of regularization affects results. These different neural networks are applied to a regression task with the aim of predicting house prices and a classification task with the aim of predicting default probabilities of credit card clients. We furthermore show some of the benefits from the produced distributions of the predictive posterior from the Bayesian neural networks, and how to use these to evaluate models and make better predictions.


\clearpage
\phantomsection
\addcontentsline{toc}{subsection}{SRA}
\label{insn:sra}
\subsection*{SRA: Shift Right Arithmetic}

\subsubsection*{Format}

\textrm{SRA \%rd, \%r1, \%r2}

\begin{center}
\begin{bytefield}[endianness=big,bitformatting=\scriptsize]{32}
\bitheader{0,7,8,15,16,23,24,31} \\
\bitbox{8}{0x2E}
\bitbox{8}{r1}
\bitbox{8}{r2}
\bitbox{8}{rd}
\end{bytefield}
\end{center}

\subsubsection*{Description}

The \instruction{sra} instruction shifts the value in \registerop{r1} right by
the number of bits indicated in \registerop{r2}, placing the results in register
\registerop{rd}.  This instruction only operates on \textbf{signed} integers.

\subsubsection*{Pseudocode}

\begin{verbatim}
%rd = %r1 >> %r2
\end{verbatim}

\subsubsection*{Constraints}

\registerop{r1}, \registerop{r2}, and \registerop{rd} must be less than
\nregs{}.

\medskip
\noindent
\registerop{rd} cannot be \register{r0}.

\subsubsection*{Failure modes}

This instruction has no run-time failure modes beyond its constraints.


  \documentclass{mynotes}

%\geometry{showframe}% for debugging purposes -- displays the margins

\newcommand{\E}{\mbox{E}}
\newcommand{\MSE}{\mbox{MSE}}
\newcommand{\var}{\mbox{var}}
\newcommand{\by}{\textbf{y}}

\usepackage{amsmath,amssymb,amsfonts,amsthm} % amsmath package, useful for mathematical formulas
%\usepackage[garamond]{mathdesign}
\usepackage{hyperref}
\hypersetup{colorlinks}% uncomment this line if you prefer colored hyperlinks (e.g., for onscreen viewing)

\newtheorem{theorem}{Theorem}
\newtheorem{lemma}[theorem]{Lemma}

% Set up the images/graphics package
\usepackage{graphicx}
\setkeys{Gin}{width=\linewidth,totalheight=\textheight,keepaspectratio}
\graphicspath{{graphics/}}

\title[Exercises 2 $\cdot$ SDS 383D]{Exercises 2: Bayes and the Gaussian linear model}
%\author[ ]{ }
\date{}  % if the \date{} command is left out, the current date will be used

% The following package makes prettier tables.  We're all about the bling!
\usepackage{booktabs}

% The units package provides nice, non-stacked fractions and better spacing
% for units.
\usepackage{units}

% The fancyvrb package lets us customize the formatting of verbatim
% environments.  We use a slightly smaller font.
\usepackage{fancyvrb}
\fvset{fontsize=\normalsize}

% Small sections of multiple columns
\usepackage{multicol}

% Provides paragraphs of dummy text
\usepackage{lipsum}

% These commands are used to pretty-print LaTeX commands
\newcommand{\doccmd}[1]{\texttt{\textbackslash#1}}% command name -- adds backslash automatically
\newcommand{\docopt}[1]{\ensuremath{\langle}\textrm{\textit{#1}}\ensuremath{\rangle}}% optional command argument
\newcommand{\docarg}[1]{\textrm{\textit{#1}}}% (required) command argument
\newenvironment{docspec}{\begin{quote}\noindent}{\end{quote}}% command specification environment
\newcommand{\docenv}[1]{\textsf{#1}}% environment name
\newcommand{\docpkg}[1]{\texttt{#1}}% package name
\newcommand{\doccls}[1]{\texttt{#1}}% document class name
\newcommand{\docclsopt}[1]{\texttt{#1}}% document class option name

\newcommand{\N}{\mbox{N}}
\newcommand{\thetahat}{\hat{\theta}}
\newcommand{\sigmahat}{\hat{\sigma}}
\newcommand{\betahat}{\hat{\beta}}

\begin{document}

\maketitle% this prints the handout title, author, and date

\section{A simple Gaussian location model}

Take a simple Gaussian model with unknown mean and variance:
\begin{equation}
\label{eqn:normal_model}
(y_i \mid \theta, \sigma^2) \sim N(\theta, \sigma^2) \; , \quad i = 1, \ldots, n \, .
\end{equation}
Let $\bf{y}$ be the vector of observations $\textbf{y} = (y_1, \ldots, y_n)^T$.

Suppose we place conjugate normal and inverse-gamma priors on $\theta$ and $\sigma^2$, respectively:
\begin{align*}
(\theta \mid \sigma^2) &\sim N(\mu, \tau^2 \sigma^2) \\
\sigma^2 &\sim \mbox{Inv-Gamma}\left( \frac{d}{2}, \frac{ \eta}{2} \right) \, ,
\end{align*}
where $\mu$, $\tau > 0$, $d>0$ and $\eta > 0$ are fixed scalar hyperparameters.  Note a crucial choice here: the error variance appears in the prior for $\theta$.  This affects the interpretation of the hyperparameter $\tau$, which is not the prior variance of $\theta$, but rather the prior signal-to-noise ratio.  This is pretty common thing to do in setting up priors for location parameters: to scale the prior by the error variance.  There are a few good reasons to do this, but historically the primary one has been analytical convenience (as you'll now see).

Here's a sensible way to interpret each of these four parameters:
\begin{itemize}
\item $\mu$ is a prior guess for $\theta$.
\item $\tau$ is a prior signal-to-noise ratio---that is, how disperse your prior is for $\theta$, relative to the error standard deviation $\sigma$.
\item $d$ is like a ``prior sample size'' for the error variance $\sigma^2$.
\item $\eta$ is like a ``prior sum of squares'' for the error variance $\sigma^2$.  More transparently, $\eta/d$ is like a prior guess for the error variance $\sigma^2$.  It's not exactly the prior mean for $\sigma^2$, but it's close to the prior mean as $d$ gets larger, since the inverse-gamma(a,b) prior has expected value
$$
E(\sigma^2) = b/(a-1) =  \frac{\eta / 2}{d/2 - 1} = \frac{\eta}{d - 2} \approx \eta/d
$$
if $d$ is large.\footnote{This expression is only valid if $d > 2$.}
\end{itemize}

What is meant by the phrases ``prior sample size'' and ``prior sum of squares''?  Well, remember that conjugate priors always resemble the likelihood functions that they're intended to play nicely with.  The two relevant quantities in the likelihood function for $\sigma^2$ are the sample size and the sums of squares.  The prior here is designed to mimic the likelihood function for $\sigma^2$ that you'd get if you hallucinated a previous data set with sample size $d$ and sums of squares $\eta$.

\paragraph{Precisions are easier than variances.}  It's perfectly fine to work with this form of the prior, and it's easier to interpret this way.  But it turns out that we can make the algebra a bit cleaner by working with the precisions $\omega = 1/\sigma^2$ and $\kappa = 1/\tau^2$ instead.
\begin{align*}
(\theta \mid \omega) &\sim N(\mu, (\omega \kappa)^{-1}) \\
\omega &\sim \mbox{Gamma}\left( \frac{d}{2}, \frac{ \eta}{2} \right) \, .
\end{align*}

This means that the joint prior for $(\theta, \omega)$ has the form
\begin{equation}
\label{eqn:normal_gamma_prior1}
p(\theta, \omega) \propto \omega^{(d+1)/{2} - 1} \exp \left\{ - \omega \cdot \frac{\kappa (\theta - \mu)^2}{2}  \right\}
\cdot \exp\left\{ -  \omega \cdot \frac{\eta}{2}  \right\}
\end{equation}
This is often called the \textit{normal/gamma} prior for $(\theta, \omega)$ with parameters $(\mu, \kappa, d, \eta)$, and it's equivalent to a normal/inverse-gamma prior for $(\theta, \sigma^2)$.  (The interpretation of $\kappa$ is like a prior sample size for the mean $\theta$.)  Note: you can obviously write this joint density in Equation \ref{eqn:normal_gamma_prior1} in a way that combines the exponential terms, but this way keeps the bit involving $\theta$ separate, so that you can recognize the normal kernel.\footnote{The term ``kernel'' is heavily overloaded in statistics. \href{http://en.wikipedia.org/wiki/Kernel_(statistics)\#In_Bayesian_statistics}{See Wikipedia for the sense in which I mean the term.}  }

\begin{enumerate}[(A)]

\item By construction, we know that the marginal prior distribution $p(\theta)$ is a gamma mixture of normals.  Show that this takes the form of a centered, scaled $t$ distribution:
$$
p(\theta) \propto \left(1+ \frac{1}{\nu} \cdot \frac{(x - m)^2}{s^2}  \right)^{-\frac{\nu+1}{2}}
$$
with center $m$,  scale $s$, and degrees of freedom $\nu$, where you fill in the blank for $m$, $s^2$, and $\nu$ in terms of the four parameters of the normal-gamma family.  Note: you did a problem just like this on a previous exercise!  This shouldn't be a lengthy re-derivation.

\item Assume the normal sampling model in Equation \ref{eqn:normal_model} and the normal-gamma prior in Equation \ref{eqn:normal_gamma_prior1}.  Calculate the joint posterior density $p(\theta, \omega \mid \textbf{y})$, up to constant factors not depending on $\omega$ or $\theta$.  Show that this is also a normal/gamma prior in the same form as above:
\begin{equation}
\label{eqn:normal_gamma_post}
p(\theta, \omega \mid \by) \propto \omega^{(d^\star+1)/2 - 1} \exp \left\{ - \omega \cdot \frac{\kappa^\star (\theta - \mu^\star)^2}{2}  \right\}
\cdot \exp\left\{ -  \omega \cdot \frac{\eta^\star}{2}  \right\}
\end{equation}
From this form of the posterior, you should able to read off the new updated parameters, by pattern-matching against the functional form in Equation \ref{eqn:normal_gamma_prior1}:
\begin{itemize}
\item $\mu \longrightarrow \mu^\star =  \; ?$
\item $\kappa \longrightarrow \kappa^\star =  \; ?$
\item $d \longrightarrow d^{\star} =  \; ?$
\item $\eta \longrightarrow \eta^\star = \; ?$
\end{itemize}
You may notice that my parameterization of the normal-gamma in Equation \ref{eqn:normal_gamma_prior1} differs from, say, the one you might find in textbooks or on websites.  I've chosen this parameterization in order to make these four updates for the parameters, above, as simple-looking and intuitive as possible.

Tip: this one is a bit of an algebra slog, with a lot of completing the square, collecting common terms, and cancelling positives with negatives.  For example, to make the calculations go more easily, you might first show (or recall, from a previous exercise) that the likelihood can be written in the form
$$
p(\by \mid \theta, \omega) \propto \omega^{n/2} \exp \left\{ - \omega \cdot \left( \frac{S_y + n(\bar{y} - \theta)^2}{2} \right) \right\} \, ,
$$
where $S_y = \sum_{i=1}^n (y_i - \bar{y})^2$ is the sum of squares for the $\by$ vector.  This expresses the likelihood in terms of the two statistics $\bar{y}$ and $S_y$, which you may recall from your math-stat course are sufficient statistics for $(\theta, \sigma^2)$.

Take care in ignoring constants here: some term that is constant in $\theta$ may not be constant in $\omega$, and vice versa.  You're focusing on the joint posterior, so you can't ignore anything that has a $\theta$ or $\omega$ in it.

\item From the joint posterior you just derived, what is the conditional posterior distribution $p(\theta \mid \by, \omega)$?  Note: this should require no calculation---you should just be able to read it off directly from the joint distribution, since you took care to set up things so that the joint posterior was in the same form as Equation \ref{eqn:normal_gamma_prior1}.

\item From the joint posterior you calculated in (A), what is the marginal posterior distribution $p(\omega \mid \by)$?  Unlike the previous question, this one doesn't come 100\% for free---you have to integrate over $\theta$.  But it shouldn't be too hard, since you can ignore constants not depending on $\omega$ in calculating this integral.

\item Show that the marginal posterior $p(\theta \mid \by)$ takes the form of a centered, scaled $t$ distribution and express the parameters of this $t$ distribution in terms of the four parameters of the normal-gamma posterior for $(\theta, \omega)$.  Note: since you've set up the normal-gamma family in this careful conjugate form, this should require no extra work.  It's just part (A), except for the posterior rather than the prior.

\item True or false: in the limit as the prior parameters $\kappa$, $d$, and $\eta$ approach zero, the priors $p(\theta)$ and $p(\omega)$ are valid probability distributions.  (Remember that a valid probability distribution must integrate to 1 (or something finite, so that it can normalized to integrate to 1) over its domain.)

\item True or false: in the limit as the prior parameters $\kappa$, $d$, and $\eta$ approach zero, the posteriors $p(\theta \mid \by)$ and $p(\omega \mid \by)$ are valid probability distributions.

\item Your result in (E) implies that a Bayesian credible interval for $\theta$ takes the form
$$
\theta \in m \pm t^{\star} \cdot s \, ,
$$
where $m$ and $s$ are the posterior center and scale parameters from (E), and $t^\star$ is the appropriate critical value of the t distribution for your coverage level and degrees of freedom (e.g.~it would be 1.96 for a 95\% interval under the normal distribution).

True or false: In the limit as the prior parameters $\kappa$, $d$, and $\eta$ approach zero, the Bayesian credible interval for $\theta$ becomes identical to the classical (frequentist) confidence interval for $\theta$ at the same confidence level.

\end{enumerate}

\newpage

\section{The conjugate Gaussian linear model}

Now consider the Gaussian linear model
$$
(\by \mid \beta, \sigma^2) \sim \N(X\beta, (\omega \Lambda)^{-1} ) \, ,
$$
where $\by$ is an $n$ vector of responses, $X$ is an $n \times p$ matrix of features, and $\omega = 1/\sigma^2$ is the error precision, and $\Lambda$ is some known matrix.  A typical setup would be $\Lambda = I$, the $n \times n$ identity matrix, so that the residuals of the model are i.i.d.~normal with variance $\sigma^2$.  But we'll consider other setups as well, so we'll leave a generic $\Lambda$ matrix in the sampling model for now.

Note that when we write the model this way, we typically assume one of two things: either (1) that both the $y$ variable and all the $X$ variables have been centered to have mean zero, so that an intercept is unnecessary; or (2) that $X$ has a vector of 1's as its first column, so that the first entry in $\beta$ is actually the intercept.

We'll again work in terms of the precision $\omega = \sigma^2$, and consider a normal--gamma prior for $\beta$:
\begin{align}
(\beta \mid \omega) &\sim N(m, (\omega K)^{-1}) \\
\omega &\sim \mbox{Gamma}(d/2, \eta/2) \, .
\end{align}
Here $K$ is a $p \times p$ precision matrix in the multivariate normal prior for $\beta$, which we assume to be known.

The items below follow a parallel path to the derivations you did for the Gaussian location model---except for the multivariate case.  Don't reinvent the wheel if you don't have to: you should be relying heavily on your previous results about the multivariate normal distribution.\footnote{That is, if you find yourself completing the square over and over again with matrices and vectors, you should stop and revisit your Exercises 1 solutions.}

\subsection{Basics}

\begin{enumerate}[(A)]

\item Derive the conditional posterior $p(\beta \mid \by, \omega)$.

\item Derive the marginal posterior $p(\omega \mid \by)$.

\item Putting these together, derive the marginal posterior $p(\beta \mid \by)$.

\item Take a look at the data in ``gdpgrowth.csv'' from the class website, which has macroeconomic variables for several dozen countries.  In particular, consider a linear model (with intercept) for a country's GDP growth rate (GR6096) versus its level of defense spending as a fraction of its GDP (DEF60).

Fit the Bayesian linear model to this data set, choosing $\Lambda = I$ and something diagonal and pretty vague for the prior precision matrix $K = \mbox{diag}(\kappa_1, \kappa_2)$.  Inspect the fitted line (graphically).  Are you happy with the fit?  Why or why not?

\end{enumerate}

\subsection{A heavy-tailed error model}

Now it's time for your first ``real'' use of the hierarchical modeling formalism to do something cool.  Here's the full model you'll be working with:
\begin{align}
(\by \mid \beta, \omega, \Lambda) &\sim N(X \beta, (\omega \Lambda)^{-1}) \\
\Lambda &= \mbox{diag}(\lambda_1, \ldots, \lambda_n) \\
\lambda_i &\stackrel{iid}{\sim} \mbox{Gamma}(h/2, h/2) \\
(\beta \mid \omega) &\sim N(m, (\omega K)^{-1}) \\
\omega &\sim \mbox{Gamma}(d/2, \eta/2) \, .
\end{align}
where $h$ is a fixed hyperparameter.

\begin{enumerate}[(A)]

\item Under this model, what is the implied conditional distribution $p(y_i \mid X, \beta, \omega)$?  Notice that $\lambda_i$ has been marginalized out.  This should look familiar.

\item What is the conditional posterior distribution $p(\lambda_i \mid \by, \beta, \omega)$?

\item Combining these results with those from the ``Basics'' subsection above, code up a \href{http://en.wikipedia.org/wiki/Gibbs_sampling}{Gibbs sampler} that repeatedly cycles through sampling the following three sets of conditional distributions.
\begin{itemize}
\item $p(\beta \mid \by, \omega, \Lambda)$
\item $p(\omega \mid \by, \Lambda)$
\item $p(\lambda_i \mid \by, \beta, \omega)$
\end{itemize}
The first two should follow identically from your previous results, except that we are explicitly conditioning on $\Lambda$, which is now a random variable rather than a fixed hyperparameter.  If you cycle through these conditional posterior draws a few thousand times, you will build up a Markov-chain Monte Carlo (MCMC) sample from the joint posterior distribution $p(\beta, \omega, \Lambda \mid \by)$.  Why this technique works for getting posterior draws is the subject of a different course, but hopefully it is reasonably intuitive.

Now use your Gibbs sampler (with at least a few thousand draws) to fit this model to the GDP growth rate data for an appropriate choice of $h$.  Are you happier with the fit?  What's going on here (i.e.~what makes the model more or less appropriate for the data)?\footnote{An interesting plot will be the posterior mean of $1/\lambda_i$ for each country.}

\end{enumerate}

\end{document}


\subsection{Search Termination}\label{subsubsec:SeachTerminationMethodology}

At each discrete time-step, the agent can either choose to move to a new grid location to record a sensor measurement or it can decide to terminate the search based on its estimated state of the environment. There is a trade-off in terminating the search early, which means that less time and resources are spent on continuing the search, versus the possibility of drawing misinformed conclusions from the search due to a lack of information. For example, if the agent receives a series of false positive readings at a given location, it could mistakenly choose to conclude that the target is present at a given location rather than sample further to gain confidence that it has correctly found the location of the target. Following this line of thinking, it is clear that a strategy needs to be devised to minimize the probability of drawing false conclusions, which is described in the performance measure set out in Section \ref{sssection:PerfMeas}.\par
\citeauthor{Pollock1971SearchInterfaces} outlines three commonly used criteria that can be used to make a decision whether to terminate the search or not: the \textit{Bayes Criterion}, which minimizes the expected cost per decision, the \textit{Minimax Criterion}, which chooses a decision which minimises the maximum expected cost and the \textit{Neyman-Pearson Criterion}, which uses a likelihood ratio test to determine the optimal decision \cite{Pollock1971SearchInterfaces}. These tests are proposed in the context of a target that is definitely present in the search region, but can be extended to incorporate the decision problem where the target may or may not be present. In addition, related work by \citeauthor{Chung2007ASearch} has addressed this problem using methods that use heuristics to make a decision whether to terminate the search or not \cite{Chung2007ASearch}.

We ultimately choose to implement the Sequential Probability Ratio Test (SPRT), which is a hypothesis-testing framework developed by \citeauthor{Wald1950BayesProblems} to optimally deal with sequential decision problems, as opposed to traditional frameworks which assume that all the necessary data has been gathered prior to analysis \cite{Wald1950BayesProblems}. The background knowledge behind the SPRT can be found in Section \ref{subsec:SPRT}. An algorithm is also provided on how to perform this test in practice.
%The details of the proof of optimality of the SPRT is given in \cite{Wald1950BayesProblems} and we have outlined the details of how to perform hypothesis-testing using this framework in section <refer to the section>, along with the practical advantages and drawbacks of using it.
We applied the SPRT algorithm to our problem to provide a search termination criteria using the following quantities:
%In order to allow the agent to make a decision on whether to terminate the search or not, the following procedure was used: \note{Might be worthwhile simply outlining the algorithm}

\begin{gather}\label{eqn:SPRTQuantities}
H_0 : \text{The null hypothesis, the target is not present in the search region}\nonumber
\\ \nonumber
H_1 : \text{The alternative hypothesis, the target is present in the search region}\nonumber
\\ \nonumber
\alpha : \text{The maximum probability of making a type } \Romannum{1} \text{ error.} \nonumber
\\ \nonumber
\beta : \text{The maximum probability of making a type } \Romannum{2} \text{ error.}\nonumber
\\ \nonumber
\\ \nonumber
p_{0t} : \text{ The probability of observing the data $(e_1, ..., e_t)$ under the assumption of $H_0$} \nonumber
\\ \nonumber
p_{0t}=\sum_{loc=1}^{n} p(TargetLoc_t = loc, AgentLoc_t, SearchStatus_t| e_{1:t}, u_{1:t})\nonumber
\\ \nonumber
\\ \nonumber
p_{1t} : \text{ The likelihood of observing the data $(e_1, ..., e_t)$ under the assumption of $H_1$} \nonumber
\\ \nonumber
p_{1t}=p(TargetLoc_t = n+1, AgentLoc_t, SearchStatus_t | e_{1:t}, u_{1:t})\nonumber
\end{gather}

$p_{0t}$ and $p_{1t}$ are calculated by using the evidence likelihood algorithm, which is described in detail in Section \ref{subsubsec:EvLikelihood}. The SPRT algorithm was then used at each time-step to decide whether to
\begin{enumerate}
    \item Terminate the search accepting $H_0$, that the target is not present in the search region.
    \item Terminate the search accepting $H_1$, that the target is present in the search region. In this case, the target location with the highest estimated probability is returned as the target location.
    \item Continue the search, using the Action Selection Strategy described in Section \ref{subsubsec:ActionSelection}.
\end{enumerate}
In practice, we often took logarithmic transforms of the values used in the SPRT, to simplify some calculations and to be able to look at plots that are less heavily skewed and simpler to interpret.

\subsection{Analysis of Search Termination Criteria}
$\alpha$ and $\beta$, the two parameters that the user needs to specify to perform the SPRT, need to be chosen carefully and depend on the context of the search. As in the standard hypothesis testing context, it is important to consider the significance level and power of the test to ensure that they reflect the severity of drawing an incorrect conclusion \cite{IntroductionToMathematicalStatistics}. They can also help to perform analysis on how well the agent could perform, since setting a high threshold means that the agent may have to take a minimum number of samples at the correct target location in order to choose $H_0$ or $H_1$. \par

Figure \ref{fig:SPRTCutoffFunctionOfAlphaAndBeta} shows how A and B vary as functions of the parameters $\alpha$ and $\beta$ on a log scale
%\footnote{Tables of the values of A and B for varying for varying Type \Romannum{1} rates ($\alpha$) and Type \Romannum{2} ($\beta$) error rates may be referred to in Appendix \ref{chap:AppendixOne}.}.
Tables of the values of A and B for varying for varying Type \Romannum{1} rates ($\alpha$) and Type \Romannum{2} ($\beta$) error rates may be referred to in Appendix \ref{chap:AppendixOne}.
If the log-likelihood ratio of the data lies in between the red and blue surfaces on the graph, another sample is taken. A projection of this plot is shown in Figure \ref{fig:VaryingSPRTParametersProjected1}:
\begin{enumerate}[label=(\alph*)]
    \item shows a plot of how varying the Type \Romannum{1} error rate $\alpha$ for a fixed Type \Romannum {2} error rate $\beta = 0.07$ affects the decision boundaries defined by A and B.
    \item shows a plot of how varying the Type \Romannum{2} error rate $\beta$ for a fixed Type \Romannum{1} error rate $\alpha = 0.1$ affects the decision boundaries defined by A and B.
\end{enumerate}

Figures \ref{fig:SPRTCutoffFunctionOfAlphaAndBeta} and \ref{fig:VaryingSPRTParametersProjected1} reflect the intuition behind the decision boundaries. The lower the probability of making either a Type \Romannum{1} or Type \Romannum{2} error, the further apart the decision boundary becomes, meaning the likelihood ratio must be very far apart from 1, which appears as 0 on the log scale. A likelihood ratio of 1 indicates maximum uncertainty. It is also possible to see that the surfaces meet when $\alpha=0.5$, $\beta=0.5$, which reflects the fact that immediately terminating the search will give a 0.5 probability of returning a false positive or false negative. \par

\note{Might be better off editing these images with explicit labels showing regions in which H0 and H1 will be accepted.}
Figure \ref{fig:VaryingSPRTParametersProjected2} shows the regions in which the SPRT will elect to take another sample (the green shaded region), to accept $H_0$, the upper red shaded region, and to accept $H_1$, the lower red shaded region. The blue line shows the log-likelihood ratio of $\frac{(e_{1:t} | H_0)}{p(e_{1:t} | H_1)}$ as a function of the agent belief that the target is present in the search region.
\note{This is not really clear, should exaggerate this more}
Note that since the values of $\alpha$ and $\beta$ are significantly smaller in Figure \ref{fig:VaryingSPRTParametersProjected2} (b) than in Figure \ref{fig:VaryingSPRTParametersProjected2} (a), the acceptance region for $H_0$ and $H_1$ are significantly smaller in (b) than in (a).

\par
%varying the type \Romannum{1} error rate for a fixed type \Romannum{2} error rate affects the upper and lower threshold for cutting off the search. Figure \ref{fig:SPRTVaryingT2} shows how the varying the type \Romannum{2} error rate for a fixed type \Romannum{1} error rate affects the upper and lower threshold for cutting off the search. Note that as the varying error rate increases on the x-axis, the decision boundaries come closer together, reflecting the intuition that we are more willing to accept a mistaken conclusion.
%\begin{figure}
%    \centering
%    \includegraphics[width = 0.75\linewidth]{Chapters/MultiAgentTargetDetection/Figs/SearchTermination/SPRTDecisionThresholdVaryingT1ErrorRate.png}
%    \caption{The Log-likelihood upper and lower threshold for a varying type \Romannum{1} error rate and fixed type \Romannum{2} error rate.}
%    \label{fig:SPRTVaryingT1}
%\end{figure}

%\begin{figure}
%    \centering
%    \includegraphics[width = 0.75\linewidth]{Chapters/MultiAgentTargetDetection/Figs/SearchTermination/SPRTDecisionThresholdVaryingT2ErrorRate.png}
%    \caption{The Log-likelihood upper and lower threshold for a varying type \Romannum{2} error rate and fixed type \Romannum{1} error rate.}
%    \label{fig:SPRTVaryingT2}
%\end{figure}

%Show how A and B vary for a fixed T1 or T2 error rate.
\begin{figure}[H]
    \centering
    \subfloat[Varying T\Romannum{1} error rate for a fixed T\Romannum{2} error rate]{{\includegraphics[width=11cm]{Chapters/MultiAgentTargetDetection/Figs/SearchTermination/AAndBVaryingWithAlpha.png} }}%
    \qquad
    \subfloat[Varying T\Romannum{2} error rate and fixed T\Romannum{1} error rate]{{\includegraphics[width=11cm]{Chapters/MultiAgentTargetDetection/Figs/SearchTermination/AAndBVaryingWithBeta.png} }}%
    \caption{Upper and lower values of A and B for varying error rates. Values are shown on a log scale}%
    \label{fig:VaryingSPRTParametersProjected1}%
\end{figure}

%3-Dimensional plot to show how A and B vary with alpha and beta
\begin{figure}[h]
    \centering
    \includegraphics[width = 0.78\linewidth]{Chapters/MultiAgentTargetDetection/Figs/SearchTermination/AandBAsFunctionOfAlphaAndBetaLogTransform.png}
    \caption{The upper and lower SPRT bounds for acceptance and rejection of $H_0$, as functions of the significance and power of the test. Values are shown on a log scale.}
    \label{fig:SPRTCutoffFunctionOfAlphaAndBeta}
\end{figure}

%\begin{figure}
%    \centering
%    \includegraphics[width = 0.75\linewidth]{Chapters/MultiAgentTargetDetection/Figs/SearchTermination/CutoffRegions.png}
%    \caption{The cut-off regions of the SPRT for $\alpha=0.05$, $\beta=0.08$ as a function of agent belief in whether target is present or not. Values are shown on a log scale.}
%    \label{fig:SPRTLogLikelihoodRatio}
%\end{figure}

%showing how A and B create upper and lower limits for terminating the search
\begin{figure}[H]%
    \centering
    \subfloat[SPRT cut-off regions for $\alpha=0.05$, $\beta=0.08$]{{\includegraphics[width=11.5cm]{Chapters/MultiAgentTargetDetection/Figs/SearchTermination/CutoffRegionsAlphaPointZeroFiveBetaPointZeroEight.png} }}%
    \caption{} \label{fig:SPRTProjected1}
    %\qquad
\end{figure}
\begin{figure}[H]\ContinuedFloat
    \centering
    \subfloat[SPRT cut-off regions for $\alpha=0.01$, $\beta=0.02$ \label{fig:SPRTProjected2}]{{\includegraphics[width=11.5cm]{Chapters/MultiAgentTargetDetection/Figs/SearchTermination/CutoffRegionsAlphaPointOneBetaPointZeroTwo.png} }}%
    \caption{The cut-off regions of the SPRT for (a) $\alpha=0.05$, $\beta=0.08$ and (b) $\alpha=0.01$, $\beta=0.02$ as a function of agent belief in whether target is present or not. Values are shown on a log scale.}%
    \label{fig:VaryingSPRTParametersProjected2}%
\end{figure}

\note{maybe move tables to appendix}


\subsection{Motivation}

%\input{sankalpmot}
\input{pushkarmot}

\subsubsection{Why try to make better Fourier Transforms?} Fourier transforms
are the most important tools used in data processing. The relative amplitudes of
the signal at different frequencies give us a lot of characterising data of the
signal, and changing the signal in frequency space has a lot of applications,
like auto-tuning, removing noise etc. \cite{wiki:fourier}

\begin{equation}
    \tilde{f}(\omega) = \int_{-\infty}^{\infty} e^{-i\omega t}f(t) dt~.
\end{equation}

\subsubsection{Fourier is Overkill.}
Looking at the structure of the Fourier transform, it is easy to see that it is
closely resembles the idea of a correlation, essentially extracting from the
signals its \emph{similarity} to a given frequency. These coefficients, gathered
using all frequencies, can produce a one to one mapping to a function space in
the frequency realm. However, the entire function is far too much information.
For most tasks involving categorization and identification of sounds,
fingerprinting is more than good enough. That is, considering the delta response
of the system, its restriction to a tiny subset of points in frequency space.

As such, simply extracting the info for these frequencies alone is sufficient,
and can be done whilst incorporating several statistical approximations. This is
easily done with a correlation of the signal with a pure mode. \cite{wiki:coranddep}

However, when frequencies are aggregated, the phase difference is absorbed into
the coefficients of multiple frequencies, but in the correlation scenario, no
such hope exists. This is where cross-correlation comes in. \cite{wiki:crosscor}

\begin{equation}
    (f \ast g)(t) \triangleq \int_{-\infty}^{+\infty} \overline{f(\tau)} \cdot g(t+\tau) d\tau
\end{equation}

By considering the similarities of shifted signals, we can infer more precise
information about their similarity in space, by completely disregarding the
temporal dimension. This is immensely useful in matching an externally sourced
wave of unknown phase with a synthetic mode of known or unknown phase to
identify the wave.

In particular, the global maximum of the cross-correlation may be taken as the
phase independent correlation of two signals.

\begin{equation}
    corr(f, g) \triangleq \max((f \ast g) (t))_t
\end{equation}


% file: problems/convex-diameter.tex

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{The Convex Polygon Diameter Problem}	\label{section:convex-diameter}

%%%%%%%%%%%%%%%%%%%%
\subsection{Solution}

%%%%%%%%%%%%%%%
\begin{theorem}
  For a convex polygon, a pair of vertices determine the diameter.
\end{theorem}
%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%
\begin{proof}
\end{proof}
%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%
\begin{definition}[Line of Support]
\end{definition}
%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%
\begin{definition}[Antipodal]
\end{definition}
%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%
\begin{fact}
  Not all vertex pairs are antipodal.
\end{fact}
%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%
\begin{proof}
\end{proof}
%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%
\begin{theorem}[Yaglom]
  The diameter of a convex polygon is the greatest distance between parallel lines of support.
\end{theorem}
%%%%%%%%%%%%%%%

%%%%%%%%%%%%%%%
\begin{proof}
\end{proof}
%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%