\section{Scaling the GNK value}\label{sec:scaling}

Because the GNK value is difficult to compute at scale, we review two primary approaches to approximate the GNK value for larger and more realistic networks:

\begin{itemize}
    \item In sections \ref{sec:sampling_techniques} to \ref{subsection:selection_of_sampling_method} we describe and employ sampling techniques to reduce the number of minimax optimisations that need to be conducted to approximate the GNK value to sufficient accuracy.
    \item And in section \ref{sec:modified_gnk} we consider a polynomial-time computable proxy inplace of the minimax optimisations in the characteristic function \eqref{knvalue1} of the GNK value.
\end{itemize}

\subsection{Sampling techniques}\label{sec:sampling_techniques}
To compute the GNK value to a required accuracy, not all of the minimax optimisations $v(S)$ need to be performed, as sampling techniques may be used.
We consider two different approaches for bias free sampling of the GNK value:
\begin{enumerate}
    \item By the inspection of equation \ref{da_value_eq}, we see that the GNK value of any player $i$ is an average over $v_{i,k}$%(the average value $v(S)$ for coalitions $S$ of size $k$ that include player $i$)
, and that by randomly sampling coalitions of size $k$ which include $i$ we can sample for estimations of each $v_{i,k}$.
    \item By utilizing equation \ref{convert1} we convert the problem into a standard cooperative game, where we can then compute the GNK value via the many existing sampling techniques developed for approximating the Shapley Value.
\end{enumerate}

The first of these two is an uncomplicated approach consisting of randomly generating coalitions $S\subset N$, %(implicitly also calculating $v(N\setminus S)$ via equation \ref{myeq2})
and then approximating $v_{i,k}$ by averaging the appropriate $v(S)$, which are then averaged to approximate the GNK value $\varphi$ via equation \ref{da_value_eq}. We denote this method `\textsc{Simple}'.

The second of these two approaches is more complicated, since it involves converting the problem into a cooperative game and then a selecting a technique to sample the Shapley Value.
Some of the possible techniques include: Neyman Sampling (`\textsc{Neyman}') \citep{CASTRO2017180,1938.10503378}, sampling to minimize a Hoeffding-type inequality (`\textsc{Hoeffding}') \citep{2013arXiv1306.4265M}, as well as a random stratified join-order sampling method (`\textsc{Join}') \citep{CASTRO2017180}, and unstratified random join-order sampling `ApproShapley' (`\textsc{Appro}') \citep{DBLP:journals/cor/CastroGT09}.
We consider these alongside our own developed method, the Stratified Finite Empirical Bernstein Sampling method (`SEBM') (as developed in Section \ref{section:SEBB}).

We will compare the performance of these sampling approaches in section \ref{section:performance}, but first we will summarise some of the differences between Shapley Value sampling techniques first.

\subsection{Differences in sampling approaches}

All the Shapley Value sampling technique sample over marginal contributions in slightly different ways; but primarily, they differ in whether they employ stratified sampling or not.
If they employ stratified sampling, then they sample the marginal contributions by player $i$ and size $k$ and average them to estimate each $\hat{v}_{i,k}$ (ie. approximating the terms of \eqref{eq:shapley_value2} and then using \eqref{shap2}).
Conversely, if they do not employ stratified sampling, then they directly approximate the Shapley Value via equation \eqref{shapley_value3}.

Secondarily, they differ in whether or not they sample by a join-order process or not. Sampling over marginal contributions involves calculating the difference between $v(S)$ and $v(S\cup\{i\})$ for various players $i$ and coalitions $S$, and one particularly easy way of doing this is to start with the empty coalition $\emptyset$ and generate a permutation of players that sequentially join the coalition and each make a marginal contribution in turn, in this way $n+1$ evaluations of $v(S)$ can be used to calculate $n$ marginal contributions.
Conversely, if they do not use a join-order process, then the methods randomly select coalitions $S$ and player $i$ and calculate the marginal contribution $v(S\cup\{i\}) - v(S)$, thus taking two evaluations of $v(S)$ for one marginal contribution sample point.

Between the methods, as shown in Table \ref{table:stratified_sampling_methods}: \textsc{Appro} randomly samples in join orders without stratification, \textsc{Join} randomly samples in join orders with stratification, \textsc{Hoeffding} samples with stratification and without join orders, to minimise a sum of Hoeffding-type concentration inequalities on each of the estimates $\hat{v}_{i,k}$,
\textsc{Neyman} samples with stratification without join orders to the sample each $\hat{v}_{i,k}$ proportional to the sample variance of the marginal contributions which make up each,
and \textsc{SEBM} samples with stratification without join orders to sample $\hat{v}_{i,k}$ in order to maximally reduce a complicated concentration inequality on the resultant estimated Shapley Value itself.

\begin{table}[]
\centering
\begin{tabular}{|l|l|l|l|}
\hline
Method & Stratified & Join-Order & Sampling Choice \\ \hline
\textsc{Appro} & No & Yes & Random\\
\textsc{Join} & Yes & Yes & Random\\
\textsc{Hoeffding} & Yes & No & Hoeffding-type inequality \\
\textsc{Neyman} & Yes & No & by variance of the strata \\
\textsc{SEBM} & Yes & No & speciality inequality\\ \hline
\end{tabular}
\caption{Different Shapley Value sampling methods and their features}
\label{table:stratified_sampling_methods}
\end{table}

The full details about each of the methods can be found in their respective source documents \citep{CASTRO2017180,2013arXiv1306.4265M,DBLP:journals/cor/CastroGT09} (and Chapter \ref{chap:stratified_sampling_chapter}).
And the performance of using these different methods is evaluated in the next section \ref{section:performance}.

\subsection{Sampling the GNK value at scale}\label{section:performance}

To analyse the performance of approximating the GNK with different sampling techniques we calculated the average absolute error in the approximated GNK value for randomly generated electricity networks.
We used a known process of generating pseudo random meshed networks of buses and lines reminiscent of real electricity networks. The particular algorithm is called the `Simple minimum-distance graph' method as expounded by \cite{hines1} and is given as Algorithm \ref{alg1}.

\begin{algorithm}[]
\caption{Simple minimum-distance graph algorithm}
\label{alg1}
\begin{algorithmic}
    \REQUIRE number of nodes $N$, number of links $m$
    \REQUIRE natural numbers $n_i$, such that $\forall~i~n_i\leq i$ and also $\sum_{i=1}^Nn_i=m$
    \STATE $M=\emptyset$ is set of nodes
    \FOR{$a=1:N$}
        \STATE Randomly generate planar coordinates for node index $a$, $(x_a ,y_a)$ with uniform distribution
        \STATE $M_a=\emptyset$ is set of links for node index $a$
        \FOR{$k=1:n_a$}
            \STATE select a novel $b\in M$ to minimise the Euclidean distance to node index $a$:\\ $\quad\min_b\quad (x_a-x_b)^2+(y_a-y_b)^2\quad\text{s.t}\quad (a, b)\notin M_a$
            \STATE Add the link $a$-to-$b$:\\ $\quad M_a=M_a\cup \{(a, b)\}$\\ $\quad M_b=M_b\cup \{(b, a)\}$
        \ENDFOR
        \STATE Add the node $a$:\\ $\quad M=M\cup \{(x_a ,y_a)\}$,
    \ENDFOR
    \STATE Output $M$ and $M_a$
\end{algorithmic}
\end{algorithm}

Using this algorithm we considered networks which were randomly generated to have 10 nodes with 12 lines between them, and thus small enough for it to be possible to solve the GNK value exactly.
And in each of these networks each line had randomly generated line limits and each node was assigned to be either a generator or consumer of electricity randomly with a randomly generated linear utility function with randomly generated consumption/generation limits.

By computing the exact GNK value for these networks we were then able to compute the average absolute error achieved by each of the different sampling methods for different sample budgets.
For a given budget, all the algorithms called for the computation of different numbers of the bilevel optimisations $v(S)$ (per equation \ref{knvalue1}).
And by plotting the average absolute error achieved against the number of unique optimisations called, we can see the performance of the sampling methods as shown in Figure \ref{fig:performance_graph1}.

From this graph it is seen that the methods which utilised stratification (\textsc{Hoeffding}, \textsc{Neyman}, \textsc{SEBM} and \textsc{Join})  generally performed better than those that did not use stratification (particularly \textsc{Simple} and \textsc{Appro}).
And particularly that the \textsc{Join} method, which utilises stratification and join-order sampling is quite effective over a range of situations.

On this graph, the sampling methods which utilised stratification were warm-started with a budget of 200 samples (two from each strata), as methods like \textsc{Neyman} required atleast two samples per stratum minimum (totalling 200) for a bias-free estimate of the variances which they run on.
Conversely \textsc{Simple} and \textsc{Appro} were able to be run with a smaller sample budget.
It is noted that the methods \textsc{Hoeffding}, \textsc{Neyman}, \textsc{SEBM} and \textsc{Join} perform exactly the same at 200 samples, as they each have two samples per stratum and no extra budget for any difference in logic to operate on.
It was also recognised that because of our 10 bus networks, there are only 1024 unique optimsations $v(S)$ that can be called, and hence the x-axis stops at 1024.
Each of the methods were run multiple times with increasing sample budget allowances, up until a point where the computation time became prohibitive, particularly some of the methods which used sampling without replacement would only call a stochastic number of unique optimisations, and hence it became time-prohibitive to sample to perfect accuracy utilising them.

\subsection{Selection of sampling method}\label{subsection:selection_of_sampling_method}

From the Figure \ref{fig:performance_graph1} we witnessed that \textsc{Join} was seen to be reasonably effective and it was chosen for all further analysis.
The reason for this superiority is the additional power of sampling with stratification and by join-orders.
As already stated, by using join orders, it is possible to calculate $n$ marginal contributions using $n+1$ evaluations of $v(S)$ as opposed to taking two evaluations of $v(S)$ for one marginal contribution sample point.
This simple factor outweighed the advantage of the extra sophistication associated with \textsc{SEBM} and \textsc{Neyman} methods, which did not use join-order sampling - even though SEBM was seen to be more performant for larger sample budgets.
Another advantage that was witnessed in utilising the \textsc{Join} method was that it didn't have the computational overhead of using the formulas present in the \textsc{SEBM} method, and additionally the join-order sampling allowed each new optimisation that was called to be warm-started by the previous one as each new addition to the coalition would yield an optimisation similar to the previous one.
These factors made \textsc{Join} appear as simpler and more effective method for our purposes.

To evaluate the performance of the \textsc{Join} method in approximating the GNK value for larger networks, we generated random networks of different sizes and estimated the GNK value using 8 simultaneous estimations, and timed how long it took for the average magnitude of error between these estimations to reach an one percent of each other.
A scatter plot of the run-time performance of \textsc{Join} in approximating the GNK value for variously sized random networks by this process was generated and the results are shown in Figure \ref{fig:performance_graph2}.

From this figure it is witnessed that the GNK value appears to be doubly exponentially complex to approximate by sampling and quickly becomes intractable for networks consisting of more than $16$ busses.
This double-exponential complexity was somewhat expected as solving the GNK value exactly is identified as a calculation that involves exponentially many NP-hard computations.

In light of this realisation we sought to make simplifications to the GNK value itself to ease its intractability.
Specifically we considered a polynomial-time computable proxy inplace of the minimax optimisations in the characteristic function \eqref{knvalue1} of the GNK value.

\iffigures
\input{figs/absolute_error1.tex}
\fi

\iffigures
\input{figs/approximating_times1.tex}
\fi

\subsection{Sampling a modified GNK value (M-GNK) at scale}\label{sec:modified_gnk}

Given the intractable nature of the GNK value for large networks, we considered a proxy inplace of the minimax optimisations of the characteristic function that define the GNK value (equation \ref{knvalue1}).
Particularly we considered a relaxation:

\begin{align}
\label{knvalue22}
v(S) = &+ \frac{1}{2}\left(\sum_{i\in S} u_i(p_i) - \sum_{i\in N\setminus S}u_i(p_i)\right)\nonumber\\
&\quad\quad\quad\quad\quad~\text{s.t}~ \left[\{p_i\}_{i\in N}=\argmax \left(\sum_{i\in S} u_i(p_i) + \sum_{i\in N\setminus S}\epsilon u_i(p_i)\right)\right]\nonumber\\
&- \frac{1}{2}\left(\sum_{i\in N\setminus S}u_i(p_i) - \sum_{i\in S} u_i(p_i)\right)\nonumber\\
&\quad\quad\quad\quad\quad~\text{s.t}~ \left[\{p_i\}_{i\in N}=\argmax \left(\sum_{i\in N\setminus S}u_i(p_i) + \sum_{i\in S} \epsilon u_i(p_i) \right)\right]
\end{align}

Where $\epsilon$ is a small positive value, and where we assume all the DC power constraints apply in both $\argmax$.

Rather than equation \eqref{knvalue1}, this equation \eqref{knvalue22} encapsulates the expected payoff advantage of the coalition under a 50:50 coinflip of who goes first, where in each case the leader chooses the powerflows that strictly prioritise their own utility and then the follower's utility is maximised secondarily, we call this proxy GNK value the `M-GNK value'.\footnote{This dynamic is evident for sufficiently small $\epsilon$ in the formula, but the same dynamic could also be achieved by a two stage optimisation process.}
This expression encodes much of the same dynamic as the original expression (equation \ref{knvalue1}) but avoids much of the adversarial strategic counter-considerations that make the original expression NP-hard, as the leader is unwilling to sacrifice their utility to harm the follower's utility and vice versa.

This proxy replaces the two-part NP-hard bilevel problems of equation \ref{knvalue1}, into two-part single-level linear programming problems.
This transformation makes the M-GNK value much easier to compute at scale, but potentially at the cost of being a less perfect description of idealised minimax bargaining - even though no axioms (from section \ref{the_value_def}) would be lost.
Thus the use of this proxy might potentially see the introduction of artefacts - we discuss the similarity and the resulting numerical differences between the original GNK and M-GNK in section \ref{sec:GNK_extensions_discussion}.

Some of the runtime statistics of sampling this M-GNK value with \textsc{Join} sampling for randomly generated network sizes are shown in Figure \ref{fig:performance_graph4}.
From the figure it is seen that it is readily possible to calculate to about one percent accuracy this M-GNK value in three minutes of runtime for networks of up to the size of about 50 nodes, and with ten minutes of runtime up to about 80 nodes.\footnote{\label{note1} All calculations were performed on an Dell Optiplex 9020, with Intel i7 quad core 3.6GHz processor, and all source-code available at:\\
\href{https://github.com/Markopolo141/The\_Generalized\_N-K\_Value}{github.com/Markopolo141/The\_Generalized\_N-K\_Value}}

Using this ability to calculate, we can now turn our attention to what this M-GNK value actually looks like in comparison with LMP and VCG for larger networks.

%\iffigures
\input{figs/accuracy_graph1.tex}
%\fi

\section{Results and evaluation of the GNK value at scale}\label{sec:results_and_evaluation_of_GNK}

%\iffigures
\input{figs/experiment_table1.tex}
%\fi

In this section we display and discuss some of the behaviour exhibited by the M-GNK, VCG and LMP for randomly generated larger networks.
Particularly in figures \ref{fig:experiment_table_11}-\ref{fig:experiment_table_14} we show the results of these techniques applied to an example 90 bus network (generated by algorithm \ref{alg1}), consisting of a 50-50 split of small consumers and small generators of electricity.
From this exercise we are able to see some of the distinct features of these techniques when applied to a large number of players.

The most immediate result is that LMP and VCG are nearly identical (figures \ref{fig:experiment_table_11} and \ref{fig:experiment_table_12}) while GNK is distinctly different (\ref{fig:experiment_table_13}).
The similarity between VCG and LMP can be explained by a variety of means, but a most informal explanation is that both LMP and VCG implement the same electrical outcome, and allocate payments in proportional to marginal differences about the socially optimal point.
The confluence between VCG and LMP is not only witnessed by us, but also has been noted in a more general settings where there are many small participants \citep{NATH2019673, 8430852} (see section \ref{subsec:summary_discussion_VCG}).%\ref{subsec:summary_discussion_VCG}

In figure \ref{fig:experiment_table_11} and \ref{fig:experiment_table_12} we see that there are no negative utility imputations, but that positive utility is allocated to those generators who are able to provide power the cheapest, and those consumers who value power the greatest (high x-value \& low y-value, and low x-value \& high y-value respectively).
This is because the LMP creates prices for power such that the cheapest generators are scheduled to create power which is consumed by the most desiring consumers up until a marginal point where any further dispatch/consumption would be unmotivated, and because VCG schemes allocate non-negative utilities by their axiomatic construction (per axiom of individual rationality).
From these graphs we straightforwardly identify those consumers/generators which generate/receive power as they are rewarded with positive utility.

Conversely we notice in figure \ref{fig:experiment_table_13} a completely different result for the M-GNK value, particularly that the M-GNK imputes utility to those consumers who do not receive power (as already noted in section \ref{sec:features}), but furthermore, it allocates negative net utility to all but the cheapest generators, even those who generate power at the socially optimal point (as identified by the previous paragraph as those generators who have positive imputation under LMP, in figure \ref{fig:experiment_table_11}), and this dynamic is not particularly easy to explain.

One primary explanation, lies in considering the average additional playoff advantage to a coalition of a high-cost generator.
Particularly, if we consider the taking of a 50-50 coin flip about whether the coalition or its complement chooses their strategies first, and if the coalition goes first, then the generator will be idle (bringing no benefit to the coalition), whereas if the complement goes first, then it potentially will get dispatched (hence causing a loss to the coalition) because the complement will choose to consume power and power-constraints must be obeyed, giving them a greater pay off advantage.
Hence the high-cost generator only brings negative payoff-advantage which is reflected in the M-GNK value.

Or more succinctly, the way in which the generators get allocated negative utility is that they are targets of being forced to generate at their own deficit since power-conservation constraints must be obeyed, and this dynamic becomes a negotiating lever, in the selection of threat points.
This consideration works reversely for consumers, who can only consume at a positive utility to them, thus they are at a bargaining advantage which is then reflected in the positive utility they are rewarded with under M-GNK.

This behaviour in the context of large networks was somewhat unexpected; and we summarise the consequences in the following section \ref{sec:GNK_value_discussion}.

\subsection{Possible extensions}\label{sec:GNK_extensions_discussion}

One primary question is how much the behaviour exhibited in figure \ref{fig:experiment_table_13} is entirely the result of us using a modified characteristic function, per the M-GNK value (equation \ref{knvalue22}), over the more original characteristic function of the GNK value (equation \ref{knvalue1}).
In order to investigate this question, the error of using this modification across randomly generated networks was calculated and shown in Figure \ref{fig:performance_graph5}.
This graph shows the cumulative probability of the difference in payments between the GNK and M-GNK - and shows for instance, that 60\% of participants could expect to receive within 10\% what they would have between GNK and M-GNK values, and that this similarity increases for larger network sizes.

From this graph it is noticed that the GNK and M-GNK values feature similarity which seems to increase with the size of the network under consideration --- although for computational reasons, it is increasingly difficult to confirm for networks with a size greater than 13 nodes.
This limited observation coincides with expectations that the possible strategic counter-considerations that are discarded by using equation \ref{knvalue22} over equation \ref{knvalue1} become less relevant in the context of larger networks.

Another observation, is that while the GNK remains soluble for networks of less than 13 nodes and the M-GNK value remains soluble for up to 80 node networks (to about 1\% accuracy, as per figure \ref{fig:performance_graph4}), however these numbers might still be regarded as being too small for real-world electrical network modelling.
Larger networks are expected to be tractable for the M-GNK value with more computing power and/or execution time, however further methodological improvements may be necessary to make the GNK value (or anything similar to it) capable of bearing on larger and real world networks.
Some possible avenues of investigation include employing further approximations such as player clustering (such as implemented by \cite{DBLP:journals/corr/abs-1903-10965}), or transforming the problem into a non-atomic form, similar to non-atomic Shapley Value.
However, these technical considerations are secondary to the ethical considerations about the desirability of GNK-type solutions, which we consider in the following section \ref{sec:GNK_value_discussion}.

Another outstanding question, is what the likely outcome would be, under GNK, in the context of strategising network participants.
So, while VCG has some explicit consideration of the strategising of individual agents, and LMP is identified to be largely identical to it in the context of large numbers of small players, the consequences of agents strategising under GNK is not considered here.
As the GNK is incentive incompatible and also quite complex, this issue constitutes both a major consideration and a difficult question.
While there does exist some work on similar (but more complex) solution concepts like GNK that are incentive compatible (such as presented by \cite{myerson1,Salamanca2019}) their investigation and evaluation in the context of electricity networks remain a topic for further investigation.

%\iffigures
\input{figs/probability_comparrison1.tex}
%\fi

\section{Discussion and conclusion}\label{sec:GNK_value_discussion}

To consider the social and ethical value of the GNK value we must loop back to consider some of the topics explored in Chapter \ref{cha:background}.
Particularly we briefly considered some moral elements about the distribution of resources, including Equality, Efficiency, and the various normative reference points which may be pertinent.
We will consider the GNK in light of each of these in turn.

\subsection{Efficiency}
It is easy to realise that the GNK value is axiomatically efficient, specifically in terms of maximising the sum of utility, by axiom (equation \ref{myeq}); and as it is efficient in this regard it is also therefore Pareto optimal.
The way this efficiency is implemented is that the GNK designates the electrical outcome which maximises the sum of utility, and issues budget-balanced utility transfers between the players.
Neither LMP or VCG has a similar efficiency claim, as both allocate payments between network participants are not necessarily budget balanced.
However in order to consider whether this axiomatic efficiency is actually a good thing we need to consider it in terms of what is socially good.

While there are different and potentially competing philosophical notions of what social 'efficiency' should mean, the maximisation of the sum of utility is a very classical notion.
The implementation of the GNK value assumes that utility is transferable, and the easiest way of comprehending this is in terms of utility as measured by money. And hence, in this context the maximisation of utility might be seen as corresponding to the maximisation of the monetary value of outcomes.

However it is worth noting that this might not be socially desirable, as there may be some sense in thinking that maximising utility via monetary measurement could potentially be prone to social problems, specifically as there exists some agreement of the diminishing value for money itself, in that the rich are made only a little better off by an amount of money, that would be more appreciated by the poor.

The GNK value allocates the utilitarian outcome, however it is not necessarily required that utility be measured in money (even though this is the easiest interpretation).
It is also possible to note that there exist non-transferable-utility (NTU) modifications - such as also considered by \cite{value1} - which could be modified to yield a NTU GNK value.
Such solution concepts could be utilised where utility is not directly transferable and which could incorporate non-linear utilities over possible monetary transactions, potentially resulting in more egalitarian outcomes.

The potential for modifying the GNK value for application in NTU settings in order to account for different efficiency measures is a topic of potential future investigation.

\subsection{Formal-equality}

All the techniques considered (GNK, VCG and LMP, etc.) satisfy formal-equality, as they only treat different participants differently in relation to specific morally relevant characteristics --- particularly their utility preferences, and how their presence and actions can affect the utility of others.
In the electricity context, this involves the utility of electricity and capacity to deliver/generate electricity at their location in relation to the utility of that electricity for others.

However, whether-or-not these qualify as morally warranted bases for differential treatment is the subject of a wider discussion, particularly as differential treatment gives rise to differential incentives.
For instance, the idea of people being afforded different effective prices for electricity depending on the location of their electrical influence may be regarded as being ethically unfair to some people (especially perhaps between close neighbours whose grid connections are electrically different), however a negation of this also destroy any incentive for people to install additional generation capacity in electrically advantageous locations.

Differential treatment can be seen as the mirror image of differential incentives, and a comprehensive reflection of the incentives that would be given under LMP against VCG and GNK are beyond the scope of our consideration.
However we can speculate that there exists a primary difference between VCG and LMP (which give similar results) and GNK, in that LMP and VCG incentivse behaviour that could affect the network operating point; particularly affecting only those generators/consumers which would (or do) generate or consume power at the network operating point.
Wherease GNK is more comprehensive in its consideration, as it affords utility for every possible way that the network could degenerate into adversarial competition; thus we might speculate that the GNK value (or something similar to it) might incentives robustness in the network.

\subsection{Hetrogeny of normative points}

The essential feature of the GNK value is that it is an extension of bargaining solution concepts to multiple players over the restricted strategy space of a generalised non-cooperative games.
%And it inherits the underlying idea that the disagreement point in the non-cooperative game between two parties is the normative reference point from which the cooperative bargaining result is determined.
The GNK value inherits the assumption from Nash bargaining that the minimax of the zero-sum game is the point which is/should-be the disagreement point between any two coalitions for the system, and that all coalitions are equally weighted - however this construction can be questioned.

Firstly, the GNK can be seen to assign equal weight for every possible coalition that could form, irrespective of the likelihood that such coalitions would actually form in competition.
It may be possible to construct a weighted GNK value to account for differential likelihoods of different coalitions forming, but that is a remaining further consideration.

Secondly, the minimax of the zero-sum game identifies a point of maximum strategising to gain payoff advantage specifically over the opponent, irrespective of the absolute payoff to the player.
And in this way minimax of the zero-sum game identifies a point of maximally engaged competition between the players.
%the GNK value integrates this dynamic to consider the space of all possible coalitions and anti-coalitions which could form, and amalgamates all possible minimax's between them into a point of value for each participant in those formations.
Against this consideration, it is good to note, that while the GNK (and Nash bargaining solution concepts) can be viewed as a description of perfect competition between individuals, it can also directly account for the possibility that that groups (or subgroups) of players may be altruistic; in that altruism may be accounted for by a player's utility function including terms associated with positive utility for others.

However this might not be sufficient, and the GNK value contrasts against other similar solutions over non-cooperative games (as briefly mentioned in section \ref{relating_to_the_old}) such von Neumann and Morgenstern's solution (per equation \ref{knvalue3}), where the characteristic function is identified as minimax payoff, not minimax payoff advantage.
In this way von Neumann and Morgenstern's characteristic function can be considered as descriptive of a point of less totally engaged competition between coalitions - which might be more appropriate or pertinent for electricity networks.
Von Neumann and Morgenstern's characteristic function is not the only alternative way that the characteristic function could be constructed, but unfortunately these alternative constructions would not nessisarily yield Nash bargaining solution in the two player case.

%And the question outstanding is, should resources be distributed above the point of such a specific description of perfect competition?

%From this perspective the GNK contrasts against other solutions (such as LMP and VCG) in regards to the reference point which they build upon.
%The VCG solution builds upon the idea that the value of a player should be in reference to their 'absence' only at the solution point of the grand coalition, and therefore cannot allocate resources to thoes who do not interact with such a solution point; and a similar dynamic is present in LMP, in terms of marginal interaction.

%Between the various solutions (LMP, VCG, GNK and possible variants thereof) there are different choices of reference point from which the dynamic occurs, and the question which is pertinent, realistic and morally justified is still ultimately unaddressed.
%However, one particularly important reference point, which is axiomatic to VCG is the reference point of the utility of zero.
%Under VCG the axiom of individual rationality casts the dynamics such that each participant will be rewarded with a non-negative utility.
%If a zero utility is associated with not participating in the system, then individual rationality guarantees that it is better that any participant participates in the mechanism than not.
%This is an important reference point is not manifest in GNK and we discuss it in the next.

\subsection{Wider equality}\label{sec:wider_equality_gnk}

In the development of this research it was hoped that the GNK value would ultimately be witnessed to have a similar individual rationality property as VCG.
Particularly that no participant should be allocated less than zero utility, which might be interpreted as being what they would get if they did not participate in the mechanism.
The ethical importance of individual rationality cannot easily be overstated, particularly as a primary notion of ethical exchange is the concept of `euvoluntary' exchange (see section \ref{sec:reference_points}) which is (at least) where every party is not left worse-off for participating.

However, it is evident that GNK seems to violate this property, as it is possible for participants to be allocated with less less utility than zero.
We would expect that this particular absence of a guarantee for participants would be a major hindrance to GNK's application in electricity systems; hence an ethical failing.
%Indeed, is difficult to imagine how a system which would leave participants worse off than if they did not participate could be good for the wellbeing of society.

In the designing of the GNK value it was hoped that individual rationality would be a property which would be present for larger networks, particularly as it was suspected in section \ref{subsec:nash_bargaining_endogenous} that if it was possible for players to unilaterally implement an outcome which guaranteed them a utility of zero, then they would be guaranteed a non-negative net utility.
However in our GNK application to electricity networks, the enforcement of the power conservation constraints (per equation \ref{dcopf1}) seems to have disallowed this eventuality.

For instance, under GNK value in the context of our generalised game modelling DC networks, the power conservation constraint causes the incorporation of unrealistic bargaining manoeuvres, such as participants extortionately threatening to oversupply others.
It is very possible that alterations to the GNK value, and similar types of Shapley Value structures over non-cooperative game solutions could be made to amend this issue.

It may be possible, for instance, to reconsider the electrical interaction between participants, perhaps by allowing player's action space to be selection of voltage level at their location, rather than directly their power input/output; however in this context power line limits would be manifest as non-linear constraints on the optimisation problem, subsequently making computation more difficult.
Alternatively, it might also be possible to implement blackout costs to participants to curtail the possibility of unrealistic bargaining manoeuvres in the GNK value logic.

In these cases, it might be possible to give participants actions to guarantee themself a disagreement point which affords them a zero utility, and hence potentially restore individual rationality to the resultant GNK value; however these investigations are a topic for future research.

At a broader level of consideration, it is interesting how perfect competition may or may not coincide with what is equal and ethical.
The question of when and where these coincide, and particularly if they might coincide in the context of electricity networks, was part of the motivation of this research.
Unfortunately our investigations demonstrated that the most direct application of the GNK value would be expected to fail wider social equality considerations.

In the next chapter we delve into the details of the sampling techniques that were developed and tested throughout this research, and we provide a final conclusion to our research in Chapter \ref{cha:conc}.


\chapter{Implicit One-Step Methods and Long-Term Stability}

\begin{intro}
  In the first chapter, we studied methods for the solution of IVP and
  the analysis of their convergence with shrinking step size $h$.  We
  could gain a priori error estimates from consistency and stability
  for sufficient small $h$.

  All of these error estimates are based on Grönwall's
  inequality. Therefore, they contain a term of the form $e^{Lt}$
  which increases fast with increasing length of the time interval
  $[t_0,T]$. Thus, the analysis is unsuitable for the study of
  long-term integration, since the exponential term will eventually
  outweigh any term of the form $h^p$.

  On the other hand, for instance our solar system has been moving on
  stable orbits for several billion years and we do not observe an
  exponential increase of velocities. Thus, there are in fact
  applications for which the simulation of long time periods is
  worthwhile and where exponential growth of the discrete solution
  would be extremely disturbing.

  This chapter first studies conditions on differential equations with
  bounded long term solutions, and then discusses numerical methods
  mimicking this behavior.
\end{intro}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Monotonic initial value problem}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{example}
  \label{ex:linear-model}
  We consider for $\lambda \in \C$ the linear initial value problem
  \begin{equation}
    \begin{split}
      \label{eq:impl:testproblem}
      u' & = \lambda u\\
      u(0) & = 1.
    \end{split}
  \end{equation}
  Splitting $\lambda=\Re(\lambda)+i\Im(\lambda)$ into its real and
  imaginary part, the (complex valued) solution to this problem is
  \begin{gather*}
     u(t) = e^{\lambda t}
     = e^{\Re(\lambda)t}\bigl(
     \cos(\Im(\lambda)t)+i\sin(\Im(\lambda)t)
     \bigr).
  \end{gather*}
  The behavior of $u(t)$ for $t\to\infty$ is determined by the real
  part of $\lambda$:
  \begin{gather}
    \label{eq:impl:cases}
    \begin{aligned}
      \Re(\lambda) < 0 & :\qquad& u(t) &\to 0 \\
      \Re(\lambda) = 0 & :\qquad& |u(t)| &= 1 \\
      \Re(\lambda) > 0 & :\qquad& u(t) &\to \infty
    \end{aligned}
  \end{gather}
  Moreover, the solution is bounded for $\lambda$ with non-positive
  real part for all points in time $t$.
\end{example}

\begin{remark}
  Since we deal in the following again and again with eigenvalues of
  real-valued matrices, we will always consider complex valued IVP
  hereafter, due to the well known fact, that these eigenvalues can be
  complex.
\end{remark}

\begin{remark}
  Due to Grönwall's inequality and the stability
  \slideref{Theorem}{IVP-stability}, the solution to the IVP above
  admits the estimate $|u(t)| \le e^{|\lambda|t} |u(0)|$.  This is
  seen easily by applying the comparison function $v(t) \equiv 0$.  As
  soon as $\lambda \neq 0$ has a non-positive real part, this estimate
  is still correct but very pessimistic and therefore useless for
  large $t$.  Since problems with bounded long-term behavior are quite
  important in applications, we will have to introduce an improved
  notation of stability.
\end{remark}

\input{definitions/lipschitz-one-sided}

\begin{remark}
  The term monotonic from the previous definition is consistent with
  the term \emph{monotonically decreasing}, which we know from real-valued
  functions. We can see this by observing for $y>x$
  \begin{gather*}
    \bigl(f(t,y)-f(t,x)\bigr)(y-x) \le 0
    \quad \Leftrightarrow \quad f(t,y)-f(t,x) < 0.
  \end{gather*}
\end{remark}

\input{theorems/stability-monotonic}

\begin{proof}
  We consider the auxiliary function $m(t) = \abs{v(t)-u(t)}^2$ and
  its derivative
  \begin{align*}
    m'(t) &= 2\Re\scal({v'(t)-u'(t)},{v(t)-u(t)}) \\
    &= 2 \Re\scal({f\bigl(t,v(t)\bigr)-f\bigl(t,u(t)\bigr)},{v(t)-u(t)})
    \\
    &\le 2 \nu \abs{v(t)-u(t)}^2 \\
    &= 2 \nu m(t).
  \end{align*}
  According to Grönwall's inequality in \slideref{Lemma}{gronwall}
  we obtain for $t > t_0$:
  \begin{gather*}
    m(t) \le m(t_0) e^{2\nu(t-t_0)}.
  \end{gather*}
  Taking the square root yields the stability
  estimate~\eqref{eq:implicit:3}.
\end{proof}

\begin{remark}
  Analog to example~\ref{ex:linear-model} we obtain from the
  stability estimate, that for the difference of two solutions
  $u(t)$ and $v(t)$ of the differential equation $u'=f(t,u)$ we obtain
	in the limit $t\to\infty$:
  \begin{gather}
    \label{eq:implicit:4}
    \begin{aligned}
      \nu < 0: & \qquad & \abs{v(t)-u(t)} &\to 0 \\
      \nu = 0: & \qquad & \abs{v(t)-u(t)} &\le \abs{v(t_0)-u(t_0)}
    \end{aligned}
  \end{gather}
\end{remark}

\begin{Lemma}{linear-monotonic}
  The linear differential equation $u'=Au$ with $u(t)\in \C^d$ and a
  diagonalizable matrix function $A(t) \in \C^{d\times d}$ admits the
  one-sided Lipschitz condition~\eqref{eq:impl:Lipschitz} with the
  constant
  \begin{gather*}
    \nu = \max_{\substack{i=1,\dots,d\\t\in \R}} \Re(\lambda_i).
  \end{gather*}
  Accordingly, this ODE is monotonic if and only if for all
  eigenvalues $\lambda_i$ of $A(t)$ there holds
  \begin{gather}
    \label{eq:implicit:10}
    \Re (\lambda_i) \le 0.
  \end{gather}
  This is the vector-valued form of example~\ref{ex:linear-model}.
\end{Lemma}

\begin{proof}
  For the right hand side of the equation we have
  \begin{gather*}
    \Re \scal(A(t)y-A(t)x,y-x)
    \le \Re \frac{\scal(A(t)y-A(t)x,y-x)}{\abs{y-x}}\abs{y-x}
    \le \max_{i=1,\dots,d}\Re (\lambda_i) \abs{y-x}.
  \end{gather*}
  Hence, we obtain already $\nu \le \max_{i=1,\dots,d}
  \Re(\lambda_i)$. If we now insert for $x-y$ an eigenvector of
	eigenvalue	$\lambda$ for which the maximum is accepted,
	then we obtain the equality and therefore $\nu = \max_{i=1,\dots,d}
  \Re(\lambda_i)$.
\end{proof}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Stiff initial value problems}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{example}
  \label{ex:impl:2}
  We consider the IVP
  \begin{gather}
    \label{eq:implicit:5}
    u' =
    \begin{pmatrix}
      -1 & 0 \\ 0 & -100
    \end{pmatrix}u,
    \qquad u(0) =
    \begin{pmatrix}
      1 \\ 1
    \end{pmatrix}.
  \end{gather}
  This has the solution
  \begin{gather*}
    u(t) =
    \begin{pmatrix}
      e^{-t} \\ e^{-100t}
    \end{pmatrix}.
  \end{gather*}
  We see, that the solution has a component which decreases slowly in
  time with $e^{-t}$ and a second one, which decreases fast with
  $e^{-100t}$. If we apply the Euler method with step size $h$ to this
  equation, then we obtain the method step
  \begin{gather*}
    y^{(n)} = y^{(n-1)} + h
    \begin{pmatrix}
      -1 & 0 \\ 0 & -100
    \end{pmatrix} y^{(n-1)}
    = \begin{pmatrix}
      1-h & 0 \\ 0 & 1-100h
    \end{pmatrix}y^{(n-1)}
  \end{gather*}
  with the solution
  \begin{gather*}
    y^{(n)} =
    \begin{pmatrix}
      (1-h)^n \\ (1-100h)^n
    \end{pmatrix}
  \end{gather*}
  If we are interested in the second solution component, the one which
  decreases fast, we choose $h$ to be small, say $h<1/100$. Thus, for
  $n\to\infty$ we have $y_n\to 0$, slowly in the first component, fast
  in the second one, just like the solution $u(t)$ of the continuous
  solution.  (Recall that for fixed chosen step size $h$ the limits
  $t\to\infty$ and $n\to\infty$ are equal.)

  If we are just interested in the first, the slow component, at a
  time where it has changed significantly.  then a considerably larger
  step size is appropriate, say $h=1/10$. For this step size the
  first solution component is still converging to zero with $y^{(n)}_1
  = 0.9^n$.  For the second one we have however $|y^{(n)}_2| =
  |(-9)^n| \to \infty$. Therefore the approximate solution for this
  step size diverges for $n\to\infty$, very much in contrast to the
  behavior of the exact solution $u(t)\to 0$ for $t\to\infty$.
\end{example}

\begin{remark}
  Of course, it would have been possible to ignore the second
  component in the previous example. But this is not a simple task in
  general, due to the fact that most solution components are
  coupled through the equation.  In such cases the step size of the
  Euler method must be chosen to accommodate the ``fast
  components''. This can lead to significant computational overhead.
  Therefore, we define in the following characteristic
  properties of such problems and develop to that specially adapted
  solution methods.
\end{remark}

\input{definitions/stiff}

\begin{remark}
  Even though we used the term definition, the notion of ``stiffness
  of an IVP'' has something vague or even inaccurate about it. In fact
  that is due to the very nature of the problems and cannot be fixed.
  Instead we are forced to sharpen our understanding by means of a few
  examples.
\end{remark}

\begin{remark}
  The third condition in the definition of stiffness is rather rare to
  find in the literature, but it is in general implicitly assumed by
  the discussion for time step methods for stiff IVP.  It is important
  though to realize, that the methods of the previous chapter do not
  cause problems computing a good resolution for the fastest time
  scales. In this case, $e^{Lt}$ will be not too much greater than
  $e^{\nu t}$.
\end{remark}

\begin{example}
  \label{ex:impl:2a}
  First of all we will have a look at equation~\eqref{eq:implicit:5}
  in example~\ref{ex:impl:2}. The first condition of the stiffness
  definition is fulfilled. The decrease to $1/e$ happens at $t=1$ and
  at $t=1/100$ for the first and second component, respectively.
  Thus, the second condition holds as well.

  According to the discussion of example~\ref{ex:impl:2}, the third
  condition depends on the purpose of the computation.  If we want
  to compute the solution at time $t=1/100$, we would not denote the
  problem as stiff. As one is interested on the solution at time
  $t=1$, on which the second component with $e^{-100}$ is already
  below typical machine accuracy, the problem is stiff indeed. Here
  we have seen that Euler's method requires disproportionately small
  time steps.
\end{example}

\begin{remark}
  \label{remark:impl:1}
  The definition of stiffness and the discussion of the examples
  reveal that numerical methods are needed, which are not just
  convergent for time steps $h\to 0$ but also for fixed step size $h$,
  even in the presence of time scales clearly below $h$. In this case,
  methods still have to produce solutions with correct limit behavior
  for $t\to\infty$.
\end{remark}

\begin{example}
  The \define{implicit Euler method} is defined by the one-step
  formula
  \begin{gather}
    \label{eq:implicit:17}
    y_1 = y_0 + hf(t_1, y_1)
    \quad \Leftrightarrow \quad
    y_1 - h f(t_1, y_1) = y_0.
  \end{gather}
  Applied to our example~\eqref{eq:implicit:5}, we observe
  \begin{gather*}
    y^{(n)} =
    \begin{pmatrix}
      1 + h & 0 \\ 0 & 1+100h
    \end{pmatrix}^{-1}
    y^{(n-1)}.
  \end{gather*}
  This yields the solution
  \begin{gather*}
    y^{(n)} =
    \begin{pmatrix}
      \frac1{(1+h)^n} \\
      \frac1{(1+100h)^n}
    \end{pmatrix}
  \end{gather*}
  which converges to zero for $n\to \infty$, independent of $h$. Thus,
  although the implicit Euler method requires in general the solution
  of a nonlinear system in each step, it allows for much larger time
  steps than the explicit Euler method,when applied to a stiff problem.

  For a visualization of the implicit and explicit Euler method see the
  appendix.
\end{example}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{A- and B-stability}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{intro}
  In this section, we will investigate desirable properties of
  one-step methods for stiff IVP~\eqref{eq:implicit:6}. We will first
  study linear problems of the form
\begin{gather}
  \label{eq:implicit:6}
  u'=Au \qquad u(t_0) = u_0.
\end{gather}
and the related notion of A-stability in
  detail From the conditions for stiffness we derive the following
  problem characteristics:
  \begin{enumerate}
  \item All eigenvalues of the matrix $A$ lie in the left half-plane
    of the complex plane. With~\eqref{eq:impl:cases} all solutions are
    bounded for $t\to\infty$.
  \item There are eigenvalues close to zero and eigenvalues with a
    large negative real part.
  \item We are interested in time spans which make it necessary, that
    the product $h\lambda$ of a time step and an arbitrary eigenvalue,
    is allowed to be large.
  \end{enumerate}
  For this case we now want to derive criteria for the boundedness of
  the discrete solution for $t\to\infty$. The important part is not to
  derive an estimate holding for $h\to 0$, but one that holds for any
  value of $h\lambda$ in the left half-plane of the complex numbers.
\end{intro}

\begin{Definition}{stability-function}
  The \define{stability function} $R(z) = R(h \lambda)$ is the
  function generated by applying the one-step method
  \begin{gather*}
    y_1 = y_0 + h \verfahren_h(t_0, y_0)
  \end{gather*}
  to the linear test problem $u'(t)=\lambda u(t)$. Therefore,
  \begin{equation}
    y_1 = R(h \lambda) u_0,
  \end{equation}
  and
  \begin{equation}
    y^{(n)} = R(h \lambda)^n u_0.
  \end{equation}
  The \define{stability region} of a one-step method is the set
  \begin{gather}
    S = \bigl\{ z \in \C \big| \abs{R(z)} \le 1 \bigr\}.
  \end{gather}
\end{Definition}

\begin{example}
  The stability function of the explicit Euler method is derived as follows:
  \begin{equation}\begin{split}
      \label{eq:impl:stabil:expleuler}
      y_1 &= y_0 + h \lambda y_0 = (1 + h \lambda) y_0 \\
      \Rightarrow R(h \lambda) &= 1 + h \lambda \\
      R(z) &= 1 + z \\
    \end{split}\end{equation}
  The stability region for the explicit Euler is a circle with radius
  1 around the point (-1,0) in the complex plane (see Figure~\ref{fig:implicit:stability-euler})
\end{example}

\begin{example}
  The stability function of the implicit Euler method is derived as follows:
  \begin{equation}\begin{split}
  \label{eq:impl:stabil:impleuler}
    y_1 &= y_0 + h f(t_1, y_1) \\
    y_1 &= y_0 + h \lambda y_1 \\
    (1 - h \lambda) y_1 &= y_0 \\
    \Rightarrow R(h \lambda) &= \frac{1}{1- h \lambda} \\
    R(z) &= \frac{1}{1-z} \\
  \end{split}\end{equation}
The stability region for the implicit Euler is the complement of a
circle with radius 1 around the point (1,0) in the complex plane (see
Figure~\ref{fig:implicit:stability-euler}). %% anschauliches Bild?
\end{example}
\begin{figure}[tp]
  \centering
  \includegraphics[width=.47\textwidth]{fig/stability-euler}
  \hfill
  \includegraphics[width=.47\textwidth]{fig/stability-euler2}
  \caption{Stability regions of the explicit and implicit Euler
    methods (blue stable, red unstable)}
  \label{fig:implicit:stability-euler}
\end{figure}

\begin{Definition*}{a-stability}{A-stability}
  A method is called \define{A-stable}, if its stability region contains
  the left half-plane of $\C$, hence
  \begin{gather}
    \{ z \in \C | \Re(z) \le 0 \} \subset S
  \end{gather}
\end{Definition*}

\begin{Theorem}{a-stability}
  Let $\left\{y^{(k)}\right\}$ be the sequence generated by an
  A-stable one-step method of step size $h$ for the linear, autonomous IVP
  \begin{gather*}
    u'=Au, \qquad u(t_0) = u_0
  \end{gather*}
  with a diagonalizable matrix $A$ an initial value $y^{(0)} =
  u_0$. If additionally all eigenvalues of $A$ have a non-positive
  real part, then the sequence members $y^{(k)}$ are uniformly bounded
  for all $h$.
\end{Theorem}

 \begin{remark}
  The notion of A-stability was deliberately chosen neutrally by Dahlquist.
  In particaluar, note that A-stability does not stand for asymptotic stability.
 \end{remark}

\begin{proof}
  Let $\{v_\ell\}_{\ell=1,\dots,d}$ be a basis of $\C^d$ consisting of
  eigenvectors of $A$. Such a basis exists since $A$ is
  diagonalizable. Let $y_0 = \sum_{\ell=1}^d \alpha^\ell v_\ell$ be
  the representation of $y_0$ in this basis. Furthermore, we introduce
  the representations $g_i = \sum_{\ell=1}^d \gamma_i^\ell
  v_\ell$. Then, we see that equations of the form
  \begin{gather*}
    g_i = y_0 + h \sum_{j=1}^s a_{ij} A g_i
  \end{gather*}
  decouple into
  \begin{gather*}
    \gamma_{i}^\ell = \alpha^\ell + h \sum_{j=1}^s a_{ij} \lambda_\ell \gamma_j^\ell.
  \end{gather*}
  Similarly, if $y_1 = \sum_{\ell=1}^d \eta^\ell v_\ell$ we have the separation
  \begin{gather*}
    y_1 = y_0 + h\sum_{i=1}^s b_i g_i
    \quad\longrightarrow\quad
    \eta^\ell = \alpha^\ell  + h\sum_{i=1}^s b_i \gamma_i^\ell.
  \end{gather*}
  Thus, instead of a vector valued problem, the method solves $d$
  decoupled scalar problems, inside and across time steps. But for each of the scalar problems, the definition of A-stability implies boundedness of the solution, if $\Re(\lambda_\ell) \le 0$.
\end{proof}

\begin{Theorem}{implicit:a-stability-erk}
  No explicit Runge-Kutta method is A-stable.
\end{Theorem}

\begin{proof}
  We show that for such methods $R(z)$ is a polynomial.  Then, the
  theorem follows immediately, it is known for polynomials, that the
  absolute value of its values goes to infinity, if the absolute value
  of the argument goes to infinity.

  From the equation~\eqref{eq:explicit:1b} follows $k_i = \lambda
  g_i$. If we insert that into the equation~\eqref{eq:explicit:1a},
  we obtain
  \begin{gather*}
    g_i = y_0 + h \sum_{j=1}^{i-1} \rka_{ij}k_j = y_0 + h\lambda
    \sum_{j=1}^{i-1} \rka_{ij} g_j.
  \end{gather*}
  With $g_1 = y_0$ and $z=h\lambda$ one has
  \begin{align*}
    g_2 &= y_0+ \rka_{21} z y_0 = (1+\rka_{21}z) y_0\\
    g_3 &= y_0 +  \rka_{32} z g_2 =  y_0 +  \rka_{32} z (1+\rka_{21} z) y_0 =
    (1+\rka_{32} z(1+\rka_{21} z)) y_0.
  \end{align*}
	Therefore one shows easily per induction that $k_j$ results as
	multiplication of a polynomial of order $j-1$ with $y_0$.
  With formula~\eqref{eq:explicit:1c} we have that $R(z)$ is a
  polynomial of order $\rks-1$.
\end{proof}

\begin{remark}
  The notion of A-stability is only applicable to linear problems with
  diagonalizable matrices. Now we are considering its extension to
  nonlinear problems with monotonic right hand sides.
\end{remark}

\input{definitions/b-stability}

\begin{Theorem}{b-stability}
  Let be $\left\{y^{(k)}\right\}$ the sequence generated by a
  B-stable method for the IVP
  \begin{gather*}
    u'=f(u), \qquad u(t_0) = u_0
  \end{gather*}
  with initial values $y^{(0)} = u_0$. If the right hand side $f$ is
  monotonic, then the values $y^{(k)}$ of the sequence are uniformly
  bounded for for $k\to\infty$ independent of the time step size $h$.
\end{Theorem}

\begin{proof}
  The theorem follows immediately by iterating over the definition of
  B-stability.
\end{proof}

\begin{Corollary}{b-stable-a-stable}
  A B-stable method is A-stable.
\end{Corollary}

\begin{proof}
  Apply the method to the linear differential model equation, which is
  monotonic for $\Re(\lambda) \le 0$. Now, the definition of B-stability
  implies $\abs{R(z)} \le 1$, and thus, the method is A-stable.
\end{proof}

\subsection{L-stability}

An undesirable feature of complex differentiable functions in the
context of stability of Runge-Kutta methods is the fact, that the
limit $\lim_{z\to \infty} R(z)$ is well-defined on the \putindex{Riemann sphere},
independent of the
path chosen to approach this limit in the complex plane. Thus, for any
real number $x$, we have
\begin{gather}
  \lim_{x\to\infty} R(x) = \lim_{x\to\infty} R(ix).
\end{gather}

Thus, a method, which has exactly the left half-plane of $\C$ as its
stability domain, seemingly a desirable property, has the undesirable
property that components in eigenspaces corresponding to very large
negative eigenvalues, and thus decaying very fast in the continuous
problem, are decaying very slowly if such a method is applied.

This gave raise to the following notion of L-stability. We
nevertheless point out, that the L-stable methods are not always to be
considered better than A-stable ones, like it is not always necessary
to require A-stability. Judgment must be applied according to the
problem being solved.

\begin{Definition}{l-stability}
  An A-stable one-step method is called \define{L-stable}, if for its
  \putindex{stability function} there holds
  \begin{gather}
    \lim_{\Re (z) \to -\infty} \abs{R(z)} = 0.
  \end{gather}
  Some authors refer to L-stable methods as \define{strongly A-stable}.
\end{Definition}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{General Runge-Kutta methods}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\begin{intro}
  According to theorem~\ref{Theorem:implicit:a-stability-erk}, ERK
  cannot be A- or B-stable. Thus, they are not suitable for long term
  integration of stiff IVP. The goal of this chapter is the study of
  methods not suffering from this limitation. The cure will be
  implicit methods, where stages may not only depend on
  known values from the past, but also on the value to be computed.

  We point out at the beginning of this chapter, that the main
  drawback of these methods is the fact that they require the solution
  of a typically nonlinear system of equations and thus involve much
  higher computational effort. Therefore, careful judgment should
  always be applied to determine whether a problem is really stiff or
  an implicit method is needed for other reasons.
\end{intro}

\input{definitions/rk}

\begin{example}[Two-stage SDIRK]
  Both SDIRK methods in table~\ref{tab:SDIRK2} are of
  order three
  \begin{table}[htp]
    \centering
  \begin{gather}
    \def\arraystretch{1.5}
    \begin{array}{c|cc}
      \frac12 - \frac{\sqrt{3}}{6} & \frac12 - \frac{\sqrt{3}}{6} & 0 \\
      \frac12 + \frac{\sqrt{3}}{6} & \frac{\sqrt{3}}{3} & \frac12 - \frac{\sqrt{3}}{6} \\
      \hline
      & \frac12 & \frac12
    \end{array}
    \qquad
    \begin{array}{c|cc}
      \frac12 + \frac{\sqrt{3}}{6} & \frac12 + \frac{\sqrt{3}}{6} & 0 \\
      \frac12 - \frac{\sqrt{3}}{6} & -\frac{\sqrt{3}}{3} & \frac12 + \frac{\sqrt{3}}{6} \\
      \hline
      & \frac12 & \frac12
    \end{array}
  \end{gather}
    \caption{Two-stage SDIRK method of order 3}
    \label{tab:SDIRK2}
  \end{table}
\end{example}

\input{theorems/stability-rk}

\begin{proof}
  Entering $f(u) = \lambda u$ into the definition of the stages $g_i$,
  we obtain the linear system
  \begin{gather*}
    g_i = y_0 + h \sum_{j=1}^s a_{ij}\lambda g_j, \quad i=1,\dots,s.
  \end{gather*}
  In matrix notation with $z=h\lambda$, we obtain
  $(\identity-z A) g = (y_0,\dots,y_0)^T$, where $g$ is the vector
  $(g_1,\dots,g_s)^T$. Equally, we obtain
  \begin{align*}
    R(z) y_0 = y_1 &= y_0 + h \sum_{i=1}^s b_i \lambda g_i = y_0 + z
    b^T g\\
    &= y_0 + z b^T(\identity-z A)^{-1}
    \begin{pmatrix}
      y_0\\\vdots\\y_0
    \end{pmatrix}
    \\
    &= \left(1+z b^T(\identity-z A)^{-1}
      \begin{pmatrix}
        1\\\vdots\\1
      \end{pmatrix}
    \right) y_0.
  \end{align*}
  In order to prove the second representation, we write the whole Runge-Kutta method
  as a single system of equations of dimension $s+1$:
  \begin{gather*}
    \begin{pmatrix}
      I-z A & 0 \\
      -z b^T & 1
    \end{pmatrix}
    \begin{pmatrix}
      g \\ y_1
    \end{pmatrix}
    = y_0
    \begin{pmatrix}
      1\\\vdots\\1
    \end{pmatrix}.
  \end{gather*}
  Applying Cramer's rule yields the result.
\end{proof}

\input{theorems/stability-examples-rk}
\begin{figure}[tp]
  \centering
  \includegraphics[width=.3\textwidth]{fig/stability-RK2}
  \hfill
  \includegraphics[width=.3\textwidth]{fig/stability-RK4}
  \hfill
  \includegraphics[width=.3\textwidth]{fig/stability-DOPRI5}
  \caption{Stability regions of the modified Euler method, the
    classical Runge-Kutta method of order 4 and the Dormand/Prince
    method of order 5 (blue stable, red unstable)}
  \label{fig:implicit:stability-explicit-rk}
\end{figure}

\input{definitions/theta}

\begin{Theorem}{theta-a-stability}
  The $\theta$-scheme is A-stable for $\theta\ge 1/2$. Furthermore, if
  there exists a constant $c$ such that $\theta-1/2 \le ch$, the
  method is consistent of second order.
\end{Theorem}

\begin{proof}
  Left as a homework question. Additionally, we show stability regions
  for different $\theta$ in figure ~\ref{fig:implicit:theta}.
  \begin{figure}[tp]
    \centering
    \includegraphics[width=.45\textwidth]{fig/stability-CR}
    \hfill
    \includegraphics[width=.45\textwidth]{fig/stability-theta6}

    \includegraphics[width=.45\textwidth]{fig/stability-theta7}
    \hfill
    \includegraphics[width=.45\textwidth]{fig/stability-euler2}
    \caption{Stability regions of the $\theta$-scheme with
      $\theta=0.5$ (Crank-Nicolson), $\theta=0.6$, $\theta=0.7$, and
      $\theta=1$ (implicit Euler).}
    \label{fig:implicit:theta}
  \end{figure}
\end{proof}

\subsection{Existence and uniqueness of discrete solutions}

While it was clear that the steps of an explicit Runge-Kutta method
can always be executed, implicit methods require the solution of a
possibly nonlinear system of equations. The solvability of such a
system is not always clear. We will investigate several cases here:
First, Lemma~\ref{Lemma:existence-implicit-1} based on a Lipschitz
condition on the right hand side. Since this result suffers from a
severe step size constraint, we add
Lemma~\ref{Lemma:existence-implicit-2} for DIRK methods based on right
hand sides with a one-sided Lipschitz condition. Finally, we present
Theorem~\ref{Theorem:existence-implicit-3} for general Runge-Kutta
methods with one-sided Lipschitz condition.

\input{theorems/existence-implicit-1}

\begin{proof}
  We prove existence and uniqueness by a fixed-point argument. To this
  end, define the vectors
  $k^{(m)}=(k_1^{(m)},\dots,k_s^{(m)})^T\in \R^{s d}$ for $m=1,\dots$
  and the iteration $k^{(m)} = F(k^{(m-1)}$ by
  \begin{gather*}
    k_i^{(m)} = F_i(k^{(m-1)}) = f\left(
      t_0+c_i h, y_0+ h \sum_{j=1}^s a_{ij} k_j^{(m-1)}\right),
    \quad i=1,\dots,s.
  \end{gather*}
  Clearly the vector $k=(k_1,\dots,k_s)^T\in \R^{s d}$ of the
  Runge-Kutta method is a fixed-point of this iteration. Using on
  $\R^{s d}$ the norm $\norm{k} = \max_{i=1,\dots,s} \abs{k_i}$, where
  $\abs{.}$ is the regular Euclidean norm on $\R^d$, we obtain the
  estimate
  \begin{gather*}
    \norm{F(k_1)-F(k_2)}
    \le  \left(h L \max_{i=1,\dots,s} \sum_{j=1}^s \abs{a_{ij}}\right)
    \norm{k_1-k_2}.
  \end{gather*}
  Under assumption~\eqref{eq:existence-implicit-1:1}, the term in
  parentheses is strictly less than unity and thus, the mapping $F$ is
  a contraction. The \putindex{Banach fixed-point theorem} yields the
  unique existence.
\end{proof}

\input{theorems/existence-implicit-2}

\begin{proof}
  The proof simplifies compared to the general case of an IRK, since
  each stage depends explicitly on the previous stages and implicitly
  only on itself.  Thus, we can write
  \begin{gather}
    \label{eq:implicit:18}
    g_i = y_0 + v_i + h a_{ii} f(g_i),
    \quad\text{with}\quad
    v_i = h \sum_{j=1}^{i-1} a_{ij} f(g_j).
  \end{gather}
  For linear IVP with diagonalizable matrix $A$, we have
  \begin{gather*}
    \left(I-ha_{ii} A\right) g_i = y_0 + v_i,
  \end{gather*}
  and assumption~\eqref{eq:existence-implicit-2:1} implies that all
  eigenvalues of $\left(I-ha_{ii} A\right)$ are positive, thus, the
  inverse exists and we obtain a unique solution.

  In the nonlinear case, we use a homotopy argument. To this end, we
  introduce the parameter $\tau\in [0,1]$ and set up the family of
  equations
  \begin{gather*}
    g(\tau) = y_0 + \tau v_i + h a_{ii} f(g(\tau)) + (\tau-1) h a_{ii} f(y_0).
  \end{gather*}
  For $\tau=0$ this equation has the solution $g(0) = y_0$, and for
  $\tau=1$ the solution $g(1) = g_i$. By showing, that
  $\frac{d}{d\tau}g$ is bounded, we conclude that a solution exists,
  since
  \begin{gather}
    \label{eq:implicit:19}
    g(1) = g(0) + \int_0^1 g'(s)\ds
  \end{gather}
  There holds
  \begin{gather*}
    g'(\tau) = v_i + h a_{ii} \partial_y f g'(\tau) + h a_{ii} f(y_0).
  \end{gather*}
  Thus
  \begin{align*}
    \abs{g'}^2 &= \scal(g',{v_i + h a_{ii}f(y_0)})
    + h a_{ii} \scal(g', \partial_y f g')\\
    & \le \abs{g'}\abs{v_i + h a_{ii} f(y_0)}
    + h a_{ii} \nu \abs{g'}^2.
  \end{align*}
  Here, we used that by the mean value theorem, there holds
  \begin{gather*}
    \scal(\partial_y f u,u) \le \nu \abs{u}^2,\quad\forall u\in \C^d.
  \end{gather*}
  We continue by stating that by assumption $1-h a_{ii} \nu > 0$ and thus
  \begin{gather*}
    \abs{g'} \le  \frac{\abs{v_i + h a_{ii} f(y_0)}}{1-h a_{ii} \nu}.
  \end{gather*}
  Thus, we have proved existence of the stages $g_i$. On the other
  hand $y_1$ is just a fixed linear combination of these values, such
  that it exists as well.
  Uniqueness follows immediately from A- or B-stability of the method.
\end{proof}

\input{theorems/existence-implicit-3}

\begin{proof}
  We omit the proof here and refer to~\cite[Theorem IV.14.2]{HairerWanner10}
\end{proof}

% \begin{todo}
% In the following we want to discuss under which conditions a solution
% for our IRK exists and is unique.
% For that we will need the help of some definitions:

% \begin{definition}
% 	We consider the inner product $\scal(u,v)_D = u^T Dv$,
%   where $D$ is a diagonal matrix with the entries $d_i > 0$.
% 	We now denote with $\alpha_D(A^{-1})$ the largest number
%   $\alpha$ for which holds
%   \begin{gather}
%     \scal(u,A^{-1}u)_D \geq \alpha \scal(u,u)_D \qquad \forall u \in \R^\rks
%   \end{gather}

%   Futhermore we set
%   \begin{gather}
%     \alpha_0 (A^{-1}) = \sup_{D>0} \alpha_D (A^{-1})
%   \end{gather}
% \end{definition}

% \begin{theorem}[Existence of a solution for IRKs]
%   Let be f continuously differentiable and it satisfies the
% 	one-sided Lipschitz condition with the Lipschitz constant $L$. If the
%   Runge-Kutta matrix $A$ is invertible and satisfies the condition
%   \begin{gather}
%     \label{eq:implicit:cond}
%     hL < \alpha_0 (A^{-1}),
%   \end{gather}
%   then the nonlinear system~\ref{eq:implicit:1a} has a solution
%   $(\rkg_1,...,\rkg_\rks)$.
% \end{theorem}

% \begin{remark}
% 	Now we examine the perturbation susceptibility of the solution of the IRK.
% 	For this purpose we use as before the notation
%   \begin{gather}
%     \begin{array}{lll}
%       || u ||_D & = \sqrt{u^TDu} = \sqrt{\langle u,u \rangle_D} & \ \ \ \forall u \in \R^\rks \\
%       || g ||_D & = \sqrt{g^T(D\otimes I)g} & \ \ \ \forall g \in \R^{\rks n}
%     \end{array}
%   \end{gather}
% \end{remark}

% \begin{theorem}
%   \label{theorem:implicit:perturb}
%   Given $\rkg_i$ and $y_1$ as in ~\ref{eq:implicit:1a} and ~\ref{eq:implicit:1c} and the perturbed values $\hat{\rkg}_i$ and $\hat{y}_1$ are defined as follows
%   \begin{gather}
%     \begin{split}
%       \hat{\rkg}_i & = y_0 + h \sum\limits_{j=1}^\rks \rka_{ij} f(x_0 + \rkc_j h, \hat{\rkg}_j) + \delta_i \\
%       \hat{y}_1 & = y_0 + h \sum\limits_{j=1}^\rks \rkb_j f(x_0 + \rkc_j h, \hat{\rkg}_j) \\
%     \end{split}
%   \end{gather}

%   If the Runge-Kutta matrix A is invertible, the differential equation
% 	satisfies the one-sided Lipschitz condition and for a positive diagonal
% 	matrix D holds $hL<\alpha_D(A^{-1})$, then we have the estimates
%   \begin{gather}
%     \begin{split}
%       ||\hat{\rkg}-\rkg||_D & \le \frac{||A^{-1}||_D}{\alpha_D(A^{-1})-hL}||\delta||_D \\
%       ||\hat{y}_1 - y_1|| & \le ||b^TA^{-1}||_D \left( 1+\frac{||A^{-1}||_D}{\alpha_D(A^{-1})-hL} \right) ||\delta||_D \\
%     \end{split}
%   \end{gather}

%   where $\rkg = (\rkg_1,...,\rkg_\rks)^T$, $\hat{\rkg}=(\hat{\rkg}_1,...,\hat{\rkg}_\rks)^T$ and $\delta=( \delta_1,...,\delta_\rks)^T$.
% \end{theorem}

%  \begin{proof}
% % \begin{todo}
% %     \cite[IV.14.3]{HairerWanner10}
% % \end{todo}
%  \end{proof}

% \begin{theorem}[Uniqueness of the solution for IRKs]
%   For a differential equation, which satisfies the Lipschitz conditions,
%   the following holds true: a Runge-Kutta method has maximal one solution
% 	if its matrix $A$ is invertible
%   and the condition $~\ref{eq:implicit:cond}$ is satisfied.
% \end{theorem}

% \begin{proof}
%   Set $\delta = 0$ in $~\ref{Theorem:implicit:perturb}$.
% \end{proof}
% \end{todo}

% See also Lemma IV 13.16 in HW10
\input{definitions/order-conditions}
\input{theorems/order-conditions}

\begin{proof}
  For the proof, we refer to~\cite[Ch. II, Theorem
  7.4]{HairerNorsettWanner93}. Here, we only observe, that
  \begin{gather*}
    \int_0^1 t^{q-1} \dt = \frac1{q}, \qquad
    \int_0^{c_i} t^{q-1} \dt = \frac{c_i^{q}}{q}.
  \end{gather*}
  If we now insert the function $x$ at the places $c_i$ into the
  quadrature formula with the quadrature weights $\rkb_i$, then we
  obtain~\eqref{eq:implicit:8}. Similarly we get~\eqref{eq:implicit:9}, if we insert the value $t^{q}/q$ at
  the places $c_i$ from the quadrature formula with weights
  $\rka_{ij}$ for $j=1,\dots,s$. In both cases we carry this out for
  all monomials until the desired degree is reached.  Due to linearity
  of the formulas the exactness holds for all polynomials until this
  degree.
\end{proof}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Methods based on quadrature and B-stability}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\subsection{Gauss-, Radau-, and Lobatto-quadrature}

\begin{intro}
  In this subsection, we review some of the basic facts of quadrature
  formulas based on orthogonal polynomials.
\end{intro}

\begin{Definition}{gauss-quadrature}
  Let $L_n(t)$ be the Legendre polynomial of degree $n$ on $[0,1]$, up
  to scaling,
  \begin{gather*}
    L_n(t) = \frac{d^{n}}{dt^{n}}(t^2-1)^{n}.
  \end{gather*}
  A quadrature formula, which uses the $n$ roots of $L_n$ as its
  quadrature points and the integrals of the Lagrange interpolating
  polynomials as its weights is called \define{Gauß quadrature}, more
  precisely, Gauß-Legendre quadrature.
\end{Definition}

\begin{Definition}{radau-lobatto-quadrature}
  The \define{Radau quadrature} formulas use one end point of the interval
  $[0,1]$ and the roots of orthogonal polynomials of degree $n-1$ as
  their abscissas. We distinguish left and right Radau quadrature
  formulas, depending on which end is included. \define{Lobatto quadrature}
  formulas use both end points and the roots of a polynomial of degree
  $n-2$. The polynomials are
  \begin{xalignat}2
    &\text{Radau left}
    & p_n(t) &= \frac{d^{n-1}}{dt^{n-1}}\bigl(t^n(t-1)^{n-1}\bigr), \\
    &\text{Radau right}
    & p_n(t) &= \frac{d^{n-1}}{dt^{n-1}}\bigl(t^{n-1}(t-1)^{n}\bigr), \\
    &\text{Lobatto}
    & p_n(t) &= \frac{d^{n-2}}{dt^{n-2}}\bigl(t^{n-1}(t-1)^{n-1}\bigr).
  \end{xalignat}
\end{Definition}

\begin{Lemma}{gauss-quadrature}
  A Gauß quadrature formula with $n$ points is exact for polynomials
  of degree $2n-1$. A Radau quadrature formula with $n$ points is
  exact for polynomials of degree $2n-2$. A Lobatto quadrature formula
  with $n$ points is exact for polynomials of degree $2n-3$. The
  quadrature weights of these formulas are positive.
\end{Lemma}

\subsection{Collocation methods}
\begin{intro}
  An alternative to solving IVP in individual points in time, is to
  develop methods, which first approximate the solution function
  through a simpler function. For example this could be a polynomial.

  Polynomials are especially suitable for the computation with
  computers. They are not suited though for high-order interpolation
  of large intervals.  Therefore, we apply them not on the entire
  interval but rather on subintervals. The subintervals correspond
  to the time steps, which we used until now.
\end{intro}

\input{definitions/collocation}
\input{theorems/collocation}

\begin{proof}
  The polynomial $y'(t)$ is of degree $\rks-1$ and therefore uniquely
  defined through $\rks$ interpolation conditions in
  equation~\eqref{eq:implicit:12}.  We set $y'(x_0 + \rkc_i h) =
  f\bigl(t_0+\rkc_i h, y(t_0+\rkc_i h)\bigr) = k_i$, such that we have
  \begin{gather}
    y'(x_0+t h) = \sum\limits_{j=1}^\rks k_j \cdot L_j(t)
  \end{gather}
  with the Lagrange interpolation polynomial $L_j(t)$.
  By integration we obtain:
  \begin{gather}
    g_i = y(x_0 + \rkc_i h)
    = y_0 + h \int\limits_0^{\rkc_i} y'(x_0 + t h) \dt
    = y_0+h \sum_{j=1}^s k_j \int_0^{\rkc_i} L_j(t)\dt,
  \end{gather}
  which defines the coefficients $\rka_{ij}$ by comparison with
  ~\eqref{eq:implicit:1a}.  If we integrate to one instead of until
  $c_i$, then we obtain the coefficients $\rkb_j$ by comparison
  with~\eqref{eq:implicit:1c}.
\end{proof}

\input{theorems/collocation-equivalence}

\begin{proof}
  Condition $C(s)$ from~\eqref{eq:implicit:9} results in
  $s^2$ interpolation conditions for $s^2$ coefficients $a_{ij}$.
  Therefore these coefficients are defined uniquely. On the other
  hand~\eqref{eq:implicit:9} yields for $q<s$:
  \begin{gather*}
    \sum_{j=1}^s \rka_{ij} \rkc_j^q = \frac{\rkc_i^{q+1}}{q+1} =
    \int_{0}^{\rkc_i} t^q \dt.
  \end{gather*}
  As a consequence of linearity we have
  \begin{gather*}
    \sum_{j=1}^s \rka_{ij} p(\rkc_i) = \int_{0}^{\rkc_i} p(t)
    \dt,\qquad\forall p\in \Pol_{s-1}.
  \end{gather*}
  Applying this to Lagrange interpolation polynomials $L_j(t)$, we
  obtain the coefficients of equation~\eqref{eq:implicit:kolkoef},
  which were in turn computed from the collocation polynomial.
\end{proof}

\input{theorems/collocation-order}

\begin{proof}
  The condition on $\pi$ implies that the quadrature rule is exact for
  polynomials of degree $s+r-1$, thus $B(s+r)$ holds.
  We have already shown in the proof of
  Lemma~\ref{Lemma:collocation-equivalence}, that $C(s)$
  holds. Therefore, it remains to show $D(r)$.

  First, we observe that first by $C(s)$ and then by $B(s+r)$ for
  any $p<s$ and $q\le r$ there holds
  \begin{gather*}
    \sum_{i=1}^s\sum_{j=1}^s b_i c_i^{q-1} a_{ij} c_j^{p-1}
    = \frac1p \sum_{i=1}^s b_i c_i^{p+q-1}
    = \frac1{p(p+q)}.
  \end{gather*}
  Furthermore, since $B(s+r)$ we have for the same $p$ and $q$:
  \begin{gather*}
    \frac1q\sum_{j=1}^s\bigl(b_j c_j^{p-1} - b_j c_j^{p+q-1}\bigr)
    = \frac1q\left(\frac1p - \frac1{p+q}\right)
    = \frac1{p(p+q)}.
  \end{gather*}
  Subtracting these two and integrating the last result yields
  \begin{gather*}
    0 = \frac{1}{p(p+q)}-\frac{1}{p(p+q)} =
    \sum_{j} c_j^{p-1} \underbrace{\left(
      \sum_i b_i c_i^{q-1}a_{ij} - \frac1q b_j \bigl(1-c_j^q\bigr)
    \right)}_{:= \xi_i}.
  \end{gather*}
  This holds for $p=1,\dots,s-1$ and thus amounts to a homogeneous,
  linear system in the variables $\xi_i$. Thus, $\xi_i=0$ and the
  theorem holds. % \marginpar{Oops!}
%%% Todo!
\end{proof}

\begin{corollary}
  An $\rks$-stage collocation method is at least of order
  $\rks$ and at most of order $2\rks$.
\end{corollary}

\begin{proof}
  The polynomial $\pi(t)$ in~\eqref{eq:implicit:14} is of degree
  $\rks$. As a result it can be orthogonal on all polynomials of
  degree $\rks-1$ in the best case.  Otherwise it would be orthogonal
  to itself.  The transformed Legendre polynomial of degree $\rks$ on
  the interval $[0,1]$ satisfies this condition and
  theorem~\ref{Theorem:collocation-order} holds true with
  $r=s$. On the other hand $\pi(t)$ is not orthogonal on the
  constants. In this case the theorem holds true with $r=0$.
\end{proof}

\input{theorems/collocation-continuous}

\begin{proof}
  $y'$ is the interpolation polynomial of degree $s-1$ in the interpolation
	points $c_1, \dots, c_s$. There holds
	\begin{gather*}
	\max_{t \in [t_0, t_1]} |y'(t) - u'(t)| \le c \max_{t \in [t_0, t_1]}
		|u^{(s+1)} (t)| \cdot h^s.
	\end{gather*}
	We now write
	\begin{gather*}
	y(t) - u(t) = \int_0 ^t y'(\tau) - u'(\tau) \diffd \tau \le \int_0 ^t h^s
		\cdot c \max_{t \in [t_0, t_1]} |u^{(s+1)} (t)| \diffd \tau = ch^s t
		\le ch^{h+1}.
	\end{gather*}
	Since by taking the derivative one loses one order, we obtain
	\begin{gather*}
	\max_{t \in [t_0, t_1]} |y^{(k)}(t) - u^{(k)}(t)| \le ch^{s-k+1}
		\max_{t \in [t_0, t_1]} |u^{(s+1)} (t)|.
	\end{gather*}
	Defining $C = c \max_{t \in [t_0, t_1]} |u^{(s+1)} (t)|$ yields the desired result.
\end{proof}

\input{definitions/gauss-collocation}

\begin{Example*}{Gauss-collocation-2-3}{2- and 3-stage Gauss collocation methods}
  \mbox{}
  \begin{minipage}[t]{.4\linewidth}
    \input{tables/gauss2}
  \end{minipage}
  \begin{minipage}[t]{.59\linewidth}
    \input{tables/gauss3}
  \end{minipage}
  % The two- and three-stage Gauß-Collocation methods were described by
%  Hammer and Hollingsworth or\ Kuntzmann and Butcher.
	see~\cite[Tables
  7.3, 7.4]{HairerNorsettWanner93}
\end{Example*}

\input{theorems/gauss-consistency}

\begin{proof}
  Gauß quadrature is exact for polynomials of degree $2s-1$ and we
  have that $\pi$ in Theorem~\ref{Theorem:collocation-order} is of
  order $s$. Therefore, the same theorem concludes that the method is
  of order $2s$. The same proof applies to Radau- and Lobatto-
  quadrature rules with their reduced orders.
\end{proof}

\begin{Theorem}{gauss-stability}
% HW p. 181 Example 12.3
  Collocation methods with Gauß-, right Radau- and Lobatto-quadrature are
  \putindex{B-stable}. The stability region of Gauß-collocation is
  exactly the left half-plane of $\C$.
\end{Theorem}

\begin{proof}
  We only prove the theorem for Gauß-collocation, where the proof is
  simple and instructive. The proof for Radau- and Lobatto-collocation
  can be found in~\cite{HairerWanner10}.

  Let be $y(t)$ and $z(t)$ the collocation polynomials according
  to~\eqref{eq:implicit:13} with respect to initial values $y_0$ or
  $z_0$. Analogous to the proof of
  theorem~\ref{Theorem:stability-monotonic} we introduce the auxiliary
  function $m(t) = | z(t)-y(t)|^2$. In the collocation points
  $\xi_i = t_0 + \rkc_i h$, there holds
  \begin{multline}
    \label{eq:implicit:20}
    m'(\xi_i) = 2 \Re\left<
      z'(t)-y'(t),z(t)-y(t)\right>
    \\
    = 2 \Re\left<f(\xi_i,z(\xi_i)) -
      f(\xi_i,y(\xi_i)),z(t)-y(t)\right> \le 0.
  \end{multline}
  Since Gauß quadrature is exact for polynomials of degree $2s-1$, we have:
  \begin{multline*}
    |z_1-y_1|^2 = m(t_0+h) = m(t_0) + \int_{t_0}^{t_0+h} m'(t)\dt
    \\
    = m_0+h \sum_{i=1}^s b_i m'(\xi_i) \le m(t_0) =  |z_0-y_0|^2.
  \end{multline*}
  Applying this argument to $f(t,u) = \lambda u$, we see
  $A$, we see from~\eqref{eq:implicit:20} that
  \begin{gather*}
    m'(t) = 2\Re(\lambda) \abs{z(t)-y(t)}^2,
  \end{gather*}
  which proves the statement about the stability domain.
\end{proof}

\begin{Example*}{Radau-right-2-3}{2- and 3-stage right Radau collocation methods}
  \mbox{}
  \begin{minipage}[t]{.25\linewidth}
    \input{tables/radau2}
  \end{minipage}
  \begin{minipage}[t]{.58\linewidth}
    \input{tables/radau3}
  \end{minipage}
\end{Example*}

\begin{remark}
  The Radau-collocation methods with right end point of the interval
  $[0,1]$ included in the quadrature set are L-stable. The stability
  regions of the first three are shown in
  Figure~\ref{fig:radau-collocation}.

  Observe that the stability domains are growing with order of the
  method. Also, observe that the computation of $y_1$ coincides with
  that of $g_s$, such that we can save a few operations.
\end{remark}

\begin{figure}[tp]
  \centering
  \includegraphics[width=.3\textwidth]{fig/stability-radau1}
  \hfill
  \includegraphics[width=.3\textwidth]{fig/stability-radau2}
  \hfill
  \includegraphics[width=.3\textwidth]{fig/stability-radau3}
  \caption{Stability domains of right Radau-collocation methods with
    one (implicit Euler), two, and three collocation points (left to
    right). Note the different scaling of coordinate axes in
    comparison with previous figures.}
  \label{fig:radau-collocation}
\end{figure}
%\endinput
\section{Considerations on implementation}

\begin{intro}
  Implicit Runge-Kutta methods require the solution of a nonlinear
  system of size $\rks\cdot d$, where $\rks$ is the number of stages
  and $d$ the dimension of the system of ODE. DIRK methods are simpler
  and only require the solution of a system of dimension $d$. Thus, we
  should prefer this class of methods, weren't it for
\end{intro}

\begin{Theorem}{DIRK-order-barrier}
  A B-stable DIRK method has at most order 4
\end{Theorem}

\begin{proof}
  See~\cite[Theorem 13.13]{HairerWanner10}.
\end{proof}

\begin{remark}
  In each step of an IRK, we have to solve a (non-)linear system for
  the quantities $g_i$. First, we note that in order to reduce
  round-off errors, it is advantageous to solve for $z_i = g_i-y_0$,
  since, especially for small time steps, $z_i$ is expected to be much
  smaller than $g_i$. Thus, we have to solve the system
  \begin{gather}
    \label{eq:implicit:21}
    z_i = h \sum_{j=1}^{\rks} a_{ij} f(t_0+c_jh, y_0+z_j),
    \quad
    i=1,\dots,\rks.
  \end{gather}
  Using the Runge-Kutta matrix $A$, we rewrite this as
  \begin{gather}
    \label{eq:implicit:22}
    \begin{pmatrix}
      z_1\\\vdots\\z_\rks
    \end{pmatrix}
    = A
    \begin{pmatrix}
      hf(t_0+c_1h, y_0+z_1)
      \\\vdots\\
      hf(t_0+c_\rks h, y_0+z_\rks)
    \end{pmatrix}.
  \end{gather}
  The latter can be used to avoid additional function evaluations by
  computing
  \begin{gather}
    \label{eq:implicit:23}
    y_1 = y_0 + b^TA^{-1}z,
  \end{gather}
  which again is numerically much more stable than evaluating $f$ with
  a possibly large Lipschitz constant.
\end{remark}

%\begin{remark}
%  Each step of the Newton iteration requires the inversion of a
%\end{remark}

%%% Local Variables:
%%% mode: latex
%%% TeX-master: "notes"
%%% End:


\documentclass[11pt,reqno]{amsart}
\usepackage{amsfonts,amssymb,amscd,amsmath,mathrsfs,amsthm}
\usepackage[pdftex]{graphicx}

\setlength{\oddsidemargin}{0.25in}  %please do not change
\setlength{\evensidemargin}{0.25in} %please do not change
\setlength{\marginparwidth}{0in} %please do not change
\setlength{\marginparsep}{0in} %please do not change
\setlength{\marginparpush}{0in} %please do not change
\setlength{\topmargin}{0in} %please do not change

\setlength{\footskip}{.3in} %please do not change
\setlength{\textheight}{8.75in} %please do not change
\setlength{\textwidth}{6in} %please do not change
\setlength{\parskip}{4pt} %please do not change

\begin{document}

\section{Typesetting mathematical formulas}

\begin{enumerate}

\item  Typeset
\[
\delta(x) = \int_1^{15} \cos x\frac{ x\sin x + 4}{x^2-90} dx
\]

\item Typeset
\begin{align}
\Gamma(x) = \sum_{j=0}^\infty \binom{j}{2} x^j
\end{align}

\item Typeset $x\in \mathbb{R}$, $y\in \mathbb{C}$.

\item Typeset
\begin{align*}
f(x) &= x^n e^{-x^2 \sin x}\\
g(x) &= x^{2n} + f\left( \frac{\sqrt{x} }{\cos x} \right)
\end{align*}

\item Typeset (matrix with [] or bmatrix if amsmath is used)
\[
A = \left[ \begin{matrix} a & b \\ c & d \end{matrix} \right]
\]

\item Typeset
\[
f(x) = \begin{cases}
\displaystyle \int_1^{30} e^{-xt} \sinh(t) dt\text{ if }x>0,\\[1.5ex]
\displaystyle \int_{-30}^0 e^{-xt} \cosh(t) \text{ if }x\le 0.
\end{cases}
\]

\item Typeset (note the position of the equation number)
\begin{align}
\begin{split}
k_n & = \sum_{\substack{k\in\mathbb{Z} \\ k\neq 0}} k^{-n} (k+n!)\\
\ell_n & = \sum_{\substack{ k\in\mathbb{N} \\ k>20}} k^{-n} (k+n!)
\end{split}
\end{align}

\item Typeset: Define the set $A_n = \{ 1, \dots ,n\}$ and the matrix
\[
B = \begin{pmatrix}
 a_{1,1} & \dots & a_{1,n} \\
 a_{2,1} & \dots & a_{2,n} \\
 \vdots & \ddots & \vdots \\
 a_{n1,} & \dots & a_{n,n}
\end{pmatrix}
\]

\item Typeset
\[
 C = \overset{\circ}{\bigcup_{j\ge 1}} A_j
\]

\end{enumerate}
\end{document}


\documentclass[a4paper]{article}

%% Language and font encodings
\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{amsmath}

%% Sets page size and margins
\usepackage[a4paper,top=3cm,bottom=2cm,left=3cm,right=3cm,marginparwidth=1.75cm]{geometry}

%% Useful packages
\usepackage{amsmath}
\usepackage{amsthm}
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage[colorinlistoftodos]{todonotes}
\usepackage[colorlinks=true, allcolors=blue]{hyperref}
\usepackage{neuralnetwork}
\usepackage{sidecap}
\usepackage{hyperref}

\newtheorem{theorem}{Theorem}[section]
\newtheorem{corollary}{Corollary}[theorem]
\newtheorem{lemma}[theorem]{Lemma}
\theoremstyle{definition}
\newtheorem{definition}{Definition}[section]

\newcommand{\setreal}{\mathbb{R}}

\title{Idea: Adaption of theory to networks with biases}
\author{Leander Kurscheidt}

\begin{document}
\maketitle
\begin{abstract}
\end{abstract}
\section{Affine Transformations}

Since our NN "somewhat" resembles linear functions following from the rule $\sigma(x)=\sigma'(x)x$ (i see this rule as separating the linear from the non-linear part),
i am curious to see whether ideas of affine spaces and how to integrate them into linear-spaces translate.

I think they do.

\section{Idea}

Affine subspaces are usually define via $A=v + U_V$, where $U_V$ is a subspace of $V$ and $v$ is a vector of $V$ (see \href{https://de.wikipedia.org/wiki/Affiner_Unterraum}{(german) wikipedia}). But you can work in them using \href{https://en.wikipedia.org/wiki/Homogeneous_coordinates}{homogenous coordinates}.

Every affine transformation can be turned into a linear transformation with a constant 1-input using the \href{https://en.wikipedia.org/wiki/Augmented_matrix}{Augmented Matrix} trick.\\
For example the intercept in linear models in statistics is often modeled this way. \\
We can adapt this trick with only one additional constant input using this matrix:
\[
W'=\begin{bmatrix}
    w_{11}       & w_{12} & w_{13} & \dots & w_{1n} & bias_1\\
    w_{21}       & w_{22} & w_{23} & \dots & w_{2n} & bias_2\\
    \hdotsfor{6} \\
	w_{d1}       & w_{d2} & w_{d3} & \dots & w_{dn} & bias_n\\
	0 & \hdotsfor{3} & 0 & 1
\end{bmatrix}
=
\begin{bmatrix}
    W & & & b\\
	0 & \dots & 0 & 1
\end{bmatrix}
\]\\
the last line is the difference to the usual construction and should be ommitted in the last layer.\\

This results in:$f'_\theta(x)=f_{\theta'}(\begin{bmatrix}x\\1\end{bmatrix})$\footnote{slight abuse of notation. We define $\begin{bmatrix}x\\1\end{bmatrix}:=\begin{bmatrix}x_1\\\vdots\\x_n\\1\end{bmatrix}$},\\
where $\theta'$ is the parameter adpated in the above schema.\\

Usually proofs get a lot simple when you just have to wory about keeping one input constant. I think a lot of the proofs from the paper should still hold, for example Leamma 2.1 doesn't change. I hope the other proofs only need some tweaking here and there.\\

For this to work, we need to add the additional requirement that $\sigma(1)=1$
\pagebreak
\section{Proof}

In the following section, neural networks with biases as defined as:\\
\begin{align*}
f_{\theta}(x)=\sigma_{L+1}(\sigma_L(\dots \sigma_2(\sigma_1(xW^0 + b^0)W^1 + b^1)W^2 +b^2)\dots W^L + b^L)
\end{align*}
We call $b^i$ bias and additionally need the follwing constraint on the activation function $\sigma_i$:\\
\begin{align*}
\forall i \in {L+1, \dots, 1}. \exists c_i. \sigma_i(c_i)=1
\end{align*}
For $f_{\theta}$ with $\theta = (W^0, W^1,\dots, W^L)$ we define:
\begin{flalign*}
            &f_{0,\theta^0}(z) := \sigma_1(zW'^0)\text{, where } \theta^0 := (W'^0) \\
            &f_{L,\theta^L} \text{, so that}\\
            &f_{\theta} = f_{L,\theta^L} \circ f_{0,\theta^0}
            \text{ and }\\
            &\theta^L := (W^1,\dots, W^L)
\end{flalign*}

\theoremstyle{definition}
\begin{definition}{Homogenous Coordinates for Neural Networks}\\
Let $f_{\theta}$ be a neural network with biases. Then augumented neural network $f_{\theta'}$ is a neural network as defined in Definition 1, (2.1) in [TODO: ref], where:\\
$\theta' = (W'^0, \dots, W'^L)$\\
and\\
\[
W'^i=\begin{bmatrix}
    w_{11}^i       & w_{12}^i & w_{13}^i & \dots & w_{1k_i}^i & b_1^i\\
    w_{21}^i       & w_{22}^i & w_{23}^i & \dots & w_{2k_i}^i & b_2^i\\
    \hdotsfor{6} \\
	w_{k_{i+1}1}^i       & w_{k_{i+1}2}^i & w_{k_{i+1}3}^i & \dots & w_{k_{i+1}k_i}^i & b_{k_{i+1}}^i\\
	0 & \hdotsfor{3} & 0 & c_i
\end{bmatrix}
=:
\begin{bmatrix}
    W^i & & & b^i\\
	0 & \dots & 0 & c_i
\end{bmatrix} \in \mathbb{R}^{k_i + 1, k_{i + 1} + 1}  \forall i \in \{0, \dots (L-1)\}
\]\\
\[
W'^i=\begin{bmatrix}
    w_{11}^i       & w_{12}^i & w_{13}^i & \dots & w_{1k_i}^i & b_1^i\\
    w_{21}^i       & w_{22}^i & w_{23}^i & \dots & w_{2k_i}^i & b_2^i\\
    \hdotsfor{6} \\
	w_{k_{i+1}1}^i       & w_{k_{i+1}2}^i & w_{k_{i+1}3}^i & \dots & w_{k_{i+1}k_i}^i & b_{k_{i+1}}^i\\
\end{bmatrix}
=:
\begin{bmatrix}
    W^i & b^i
\end{bmatrix} \in \mathbb{R}^{k_i + 1, K}  \text{ with } i = L
\]
\end{definition}

\begin{theorem}{Equality of the augumented NN}\\
For all NN with Bias $f_{\theta}$:\\
$f'_{\theta}(x)=f_{\theta'}(\begin{bmatrix}x\\1\end{bmatrix})$
\end{theorem}

\begin{proof}
    via Induction over L.\\
    \begin{enumerate}
        \item For $L=0$:\\
        \begin{align}
            \Theta'&=(W'^0) \\
            W'^0 &= \begin{bmatrix}
                W^0 & b^0
            \end{bmatrix} \text{ because } 0=L \\
            f_{\theta'}(\begin{bmatrix}x\\1\end{bmatrix}) &= \sigma_1(\begin{bmatrix}x\\1\end{bmatrix}W'^0) \\
            &= \sigma_1(\begin{bmatrix}x\\1\end{bmatrix}\begin{bmatrix}
                W^0 & b^0
            \end{bmatrix})\\
            &= \sigma_1(xW^0 + b^0)\\
            &= f_{\theta}(x)
        \end{align}
        \item For $L \rightarrow (L+1)$ with $f_{L,\theta'} \rightarrow f_{L,\theta'} \circ f_{0,\theta'^0}$ holds Theorem 3.1
        \item $L \rightarrow (L+1)$:\\
        \begin{align}
            \theta' &= (W'^0, W'^1,\dots, W'^{(L+1)}) \\
            W'^{0} &= \begin{bmatrix}
                W^0 & & & b^0\\
                0 & \dots & 0 & c_0
            \end{bmatrix} \text{ because } 0\neq(L+1)\\
            \\
            f_{0,\theta'^0}(\begin{bmatrix}x\\1\end{bmatrix}) &= \sigma_1(\begin{bmatrix}x\\1\end{bmatrix}W'^0)\\
            &=\begin{bmatrix}
                \sigma_1(\begin{bmatrix}x\\1\end{bmatrix}\begin{bmatrix}
                    W^0 & & & b^0\\
                \end{bmatrix})\\
                \sigma_1(1*c_0)
            \end{bmatrix}\\
            &=\begin{bmatrix}
                f_{0,\theta^0}(x)\\
                1
            \end{bmatrix}\\
            f_{\theta'}(\begin{bmatrix}x\\1\end{bmatrix}) &=(f_{L,\theta'^L} \circ f_{0,\theta'^0}) (\begin{bmatrix}x\\1\end{bmatrix})\\
            &= f_{\theta'^L}(\begin{bmatrix}
                f_{\theta^0}(x)\\
                1
            \end{bmatrix}) & \text{using (12)}\\
            &= (f_{L,\theta^L} \circ f_{0,\theta^0})(x) & \text{induction hypothesis}\\
            &= f_{\theta}(x)
        \end{align}
    \end{enumerate}
\end{proof}

Let $g(x) := \begin{bmatrix}x\\1\end{bmatrix}$ (which is continous). Then, following Theorem 3.1:\\
For all NN with Bias $f_{\theta}$:$f'_{\theta}=f_{\theta'} \circ g$

\begin{theorem}{Transformation of the Data-Distribution}\\
Let $P_{data}$ be Data-Distribution, then $P'_{data} = g(P_{data})$ is the transformed Data-Distribution.\footnote{The question now arises: What is the distribution of $P'_{data}$? For this innocent and trivial transformation, the answer is harder than it might seem. It is a degenerate distribution and $P'_{data} \sim \begin{bmatrix}P_{data}\\Dirac\end{bmatrix}$, where $Dirac$ is the Dirac-Distribution.}
Then:\\
$E_{x \sim P_{data}}f_{\theta}(x) = E_{x \sim P'_{data}}f_{\theta'}(x)$
\end{theorem}

\begin{proof}
    \begin{align}
        E_{x \sim P_{data}}f_{\theta}(x) &= E_{x \sim P_{data}} (f_{\theta'} \circ g)(x)\\
        &= E_{x \sim P'_{data}}f_{\theta'}(x)
    \end{align}
    (18) follows from \textit{Law of the unconscious statistician}.
\end{proof}

The Rest of the proofs from the "Fisher-Rao Metric, Geometry ..."-paper should now follow from Theorem 3.1 and 3.2.

\bibliographystyle{alpha}
\bibliography{sample}

\end{document}