\documentclass[a4paper]{scrartcl}
\pdfinfoomitdate=1
\pdftrailerid{}
\author{Angewandte Informatik}
\title{Grundlagen der Informatik}
%Language specific formating options
\usepackage[utf8]{inputenc} % use utf8 file encoding for TeX sources
\usepackage[T1]{fontenc}    % avoid garbled Unicode text in pdf
\usepackage[english]{babel} % english grammar so words can go in two lines
\usepackage{graphicx}
\graphicspath{ {./images/} }
\usepackage{multirow}
\usepackage{longtable}
\usepackage{amsmath}
\usepackage{circuitikz}
\date{WS 20/21}
\begin{document}
    \maketitle
    \newpage
    \tableofcontents
    \newpage
    \section{Intro}
        \subsection{Representation of numbers characters}
        Numbers and characters are saved in the memory and need a binary representation. There are different ways how one can represent numbers and characters,
        depending on the needs the program has. Having a program which needs a counter, only needs positive integers so there is no need for saving decimals.  Also the
        range is important. Is the program counting to 100 or 100 million. Different datatypes need less bytes to store data, but then the range or precision (Genauigkeit) suffers.

        \subsubsection*{Integers}
        Unsigned (only positive) integers only differ in how many bits they use. Typical sizes are  8-bit (short), 16-bit (half word), 32-bit (word) and 64-bit (double word). \\
        Signed integers need to save the minus symbol somewhere. There are several options to "save the minus". One is just saying if the MSB is 1, the number is negative. The problem
        is that \(0000\) and \(1000\) are both \(0\), but one is a positive and one is a negative \(0\) which isn't very effective. \\
        Another implementation is the \textbf{one-complement}.
        Here you just invert every bit to get the "negative version" of the number. Again the \(\pm 0\) is possible, but the one-complement
        creates a symmetry with negative and positive numbers and is needed for the two-complement. \\
        The \textbf{two-complement} takes the result from the one complement and adds \(+1\) to it. The symmetry is gone but the \(\pm 0\) is gone (only positive 0) and an extra negative number is won. It can be calculated like this:
        $$Y_z = -z_{N-1}*2^{N-1} + \sum_{i=0}^{N-2}z_i * 1^i, z_i \in \{0, 1\}$$
        One other way to create negative number is by using a \textbf{bias/offset}. One needs to define the offset first. Now every number in the memory will be read nad the offset will be
        subtracted from it. An offset of 128 means that the positive numbers will start at \(1000.0000_b \)\footnote{\(1000.0000_b \equiv 0\)}. The offset is used in floating point numbers for the exponent.

        \subsection*{Decimal numbers}
        Decimal numbers also have different possible representation. An easy with a fixed point. The number is treated as an integer but at a specific bit, the point is
        set. The position of the point needs to be defined first. If there are 8-bit to save the number and the point is defined at bit 3, there will be 5 bits for the integer
        and 3 bits for the mantissa\footnote{Nachkommastellen}. The problem is, that very big numbers or very small numbers aren't possible. \\
        Floating point numbers fix this by introducing an exponent to the number. The exponent has an offset, so it can be negative. A negative exponent makes very small numbers
        possible, but because the exponent can be positive as well big numbers are possible too. The formula for calculating a normalized float is:
        \begin{equation*}
            f = (-1)^{\text{sign}} \cdot 1.\text{mantissa} \cdot 2^{\text{exponent}}
        \end{equation*}
        Depending on how many bits the float uses, different values need to be inserted into the formula.
        \begin{center}\begin{tabular}{|c|c|c|c|c|}
            \hline
             & sign & exponent & mantissa & offset  \\
            \hline
            32-bit & 1-bit & 8-bit & 23-bit & 127-bit \\
            \hline
            64-bit & 1-bit & 11-bit & 52-bit & 1023-bit \\
            \hline
        \end{tabular}\end{center}
        $$Y = (-1)^s\cdot \left(1+\sum_{i=-1}^{-F}z_i\cdot 2^i\right)\cdot 2^{e-bias}$$
        Besondere Werte wenn der Exponent nur aus 1 besteht.
        $\pm \infty$ oder NaN (Wenn Mantissa Werte enthält)

    \section{Boolean Algebra}
        Boolean Algebra takes (binary) parameters and return a  binary result. There are different operands in boolean algebra:
        \begin{itemize}
            \item NOT \(\overline{A}\): takes a single bit and toggles the value (1 -> 0, or 0 -> 1).
            \item AND \(A * B\) or \(A \land B\): looks if all operands (here just A and B) are set to one and if os returns 1, else returns 0
            \item OR \(A + B\)  or \(A \lor B\): returns 1 if any operand is set to 1, else all operands need to be 0
            \item XOR \(A \otimes B\): returns one if only 1 parameter is one
            \item NAND/NOR \(\overline{A * B}/A + B\): inverse to AND and OR
        \end{itemize}
        With boolean algebra there are some rules:
        \begin{itemize}
            \item DeMorgan's law: \(\overline{x} + \overline{y} = \overline{x + y}\) and \(\overline{x} + \overline{y} = \overline{x + y}\)
            \item Absorption: \(x * (\overline{x} + y) = x * y\) \\ \(x + (\overline{x} * y) = x +y\) \\
                \(x * (x + y) = x\) \\ \(x + (x * y) = x\)
            \item Neighborhood:  \((x * y) + (\overline{x} + y)\) and \((x + y) * (\overline{x} + y)\)
        \end{itemize}

        \begin{center}

            \begin{circuitikz}
                \draw
                (0,0) node[not port] () {}
                (5,0) node[and port] () {}
                (10,0) node[or port] () {};
            \end{circuitikz}
        \end{center}

    \section{Instruction Set Architecture (ISA)}
        The ISA contains definitions how a processor can be programed. It defines...
        \begin{itemize}
            \item ..the description of instructions (semantics etc)
            \item ..how the data behaves (how and were the data will be stored and processed)
            \item ..operation modi (user mode, supervisor mode etc)
            \item ..and the handling of traps, errors and interrupts
        \end{itemize}
    %
    %   Definition of numbers are above, may need to go in to more detail though TODO
    %
        \subsection{addresses}
            addresses are used to store/load data and can be the target of a (un)conditional jump. %addressing mode TODO
            It is typically divided into 3 categories:
            \begin{itemize}
                \item register storage space: fast, but small and often only a limited number of registers are available
                \item data storage space: bigger but slower
                \item instruction storage space: stores the instruction of the ISA
            \end{itemize}

            % May be wrong
            \subsection*{Addressing Modes}
            \begin{itemize}
                \item immediate addressing: The instruction receives the data directly (adding a constant to the Accumulator)
                \item direct addressing: The address for the instruction is hard coded
                \item register direct addressing: The instruction addresses the register directly (address is constant)
                \item indirect addressing: first the address is loaded from a specific register and then the memory address is used to processed (register only stores address instead of value)
                \item indexed indirect addressing: two registers are used to get the address from the value. One is contains the address and one is a counter. Adding both together results in the actual address (arrays)
                \item program counter relative addressing: like indexed indirect access but the program counter functions as the address counter
                %TODO pre/post modifying indirect addressing TODO 134 has some information which may be useful
            \end{itemize}

        \subsection{Instructions}
            Processors work after the control flow principle. The basic idea is that an operation takes operands and generates a result. In order for an processor to run
            algorithms, the processor needs to be able to process different kinds of operations:
            \begin{itemize}
                \item algorithmic operations: add, subtract, ...
                \item comparisons: if (greater, lower, equal, ...)
                \item logic operations: boolean algebra
                \item shift operations: rotate the byte left/right (multiply/divide by 2)
                \item control the control-flow: jumps
            \end{itemize}
            Instruction can be classified into different types, looking at how many addresses are used. Monadic operations only use one address (NOT for example), dyadic
            operations use 2 addresses. ADD A,B for example adds A and B together and saves the result in A. Some operations use 0 addresses by addressing implicitly for example the
            registers or program counter.
        \subsection{States}
            An processor need to be able to handle exceptions. Exceptions are differentiated in two groups:
            \begin{itemize}
                \item traps: synchronous events, occur when something in the program happens which shouldn't happen (division by zero)
                \item interrupts: asynchronous event, occurs when something external from the program needs to be executed (button press, timer)
            \end{itemize}
            Operation modes disconnect sensitive areas and none sensitive areas from a computer. A program for viewing pictures doesn't need full access to the whole
            computer and it's resources. The most basic case is implementing a user-mode which has access to its own files and supervisor mode which can access everything when needed.

    \section{ISA-ARM}
        The ARM architecture uses the stored-program concept. Instructions and data are both stored in memory (as numbers). This results into a great bit of flexibility and
        leads to the stored-program computer.
        %Table of ARM instructions TODO:maybe
        \subsection{Operations}
            Creating a program typically involves using a programing language instead of assembly languages for convenience reasons. Processors only understand compiled code and
            depending on the processor the compiled code will look different. Let's say f = (a + b) - (c + d) is c code we want to compile for ARM. ARM only allows
            arithmetic operations using registers, so first all values of the variables need to be loaded into register. ARM also only allows 2 addresses for adding and subtracting at once
            so the results need to split into pieces and then stitched together at the end.

        \subsection{Operands}
            Operands are in short word (32 bit) and double word (64 bit, size of ARMv8 register size). Registers and variables (from programing languages) are different
            because registers are limited in size. Too many registers would increase the clock cycle time. %lookup clock cycle time TODO%
            Therefore if too many variables were created register values need to be moved into the memory (and vice versa). Those operations are called data transfer instructions.
            %aligned data, data is aligned in memory for example in 64 bit iterations

        \subsection{Instructions}
            ARMv8 uses its own assembly language. It's pretty close to the machine code but it still needs to be converted to proper machine code. 'ADD x9, x20, x21'
            would be translated into '1112 21 0 20 9'. It is divided as following tabling shows (a R Format instruction):
            \begin{center}
            \begin{tabular}{|p{5cm}|c|c|c|c|}
                \hline
                opcode & Rm & shamt & Rn & Rd \\
                \hline
                11 bits & 5 bits & 6 bits & 5 bits & 5 bits \\
                \hline
            \end{tabular}
            \end{center}
            \begin{center}
                \begin{tabular}{|p{5cm}|c|c|}
                    \hline
                    Instruction & Format & opcode  \\
                    \hline
                    ADD & R & $1112_{10}$ \\
                    \hline
                    SUB & R & $1624_{10}$ \\
                    \hline
                    ADDI & I & $580_{10}$ \\
                    \hline
                    SUBI & I & $836_{10}$ \\
                    \hline
                    LDUR & D & $1986_{10}$ \\
                    \hline
                    STUR & D & $1984_{10}$ \\
                    \hline
                \end{tabular}
            \end{center}
            Example: 1986 240 0 10 9 -> Load the address in register 10 with an offset of 30 bytes (240/8=30) into register 9
            \begin{itemize}
                \item opcode: the numeric representation of the instruction
                \item Rm: second register
                \item shamt: shift amount
                \item Rn: first register
                \item Rd: destination register
            \end{itemize}

            \begin{center}
                \begin{tabular}{|c|c|c|c|c|}
                    \hline
                    \multicolumn{5}{|c|}{D-Format} \\
                    \hline
                    opcode & address & op2 & Rn & Rt \\
                    \hline
                    11 bits & 9 bits & 2 bits & 5 bits & 5 bits \\
                    \hline \hline
                    \multicolumn{5}{|c|}{I-Format} \\
                    \hline
                    opcode & \multicolumn{2}{|c|}{immediate}  & Rn & Rd \\
                    \hline
                    10 bits & \multicolumn{2}{|c|}{12 bits} & 5 bits & 5 bits \\
                    \hline
                \end{tabular}
            \end{center}
            R-Format is often used for arithmetic instructions using addresses or for shifting a register (and saving it to another),
            while I-Format is often used for immediate instructions. D-Format is used for loading and storing values from/to register to/from memory.

            \begin{center}
                \begin{tabular}{|l|c|c|}
                    \hline
                    Instruction & ARM code & description \\
                    \hline
                    LSL & LSL X11, X19, \#4 & shifts x19 4 times left and stores result in X11 \\
                    \hline
                    AND & AND X9, X10, X11 & X9 = X10 * X11 (binary AND) \\
                    \hline
                    OR & OR X9, X10, X11 & X9 = X10 + X11 (binary OR) \\
                    \hline
                    EOR & EOR X9, X10, X11 & X9 = X10 \(\otimes \) X11 (binary EOR/XOR) \\
                    \hline
                    NOT & EOR X9, X10, X11(=1111...111) & Not isn't implemented so EOR is used \\
                    \hline
                \end{tabular}
            \end{center}
            ANDI, ORRI, EORI are the immediate variations of the above instructions.

            \subsubsection*{Branches}
            \begin{itemize}
                \item if: uses 'CBZ Register, label' (jump to 'label' if Register is zero) and CBZN (jump if not zero)
                \item loop: uses a decreasing counter and CBZ and jumps to the beginning of the loop as long as the counter isn't zero
            \end{itemize}
            There are more conditions like less, less or equal, etc.:
            % \begin{center}
            %     \begin{tabular}{|c|c|c|c|c|}
            %         \hline
            %         & \multicolumn{2}{|c|}{Signed numbers} & \multicolumn{2}{|c|}{Unsigned numbers} \\
            %         \hline \hline
            %         Comap. & Instr. & CC test & Instr. &  CC test \\
            %         \hline
            %         \(=\) & B.EQ & Z = 1 & B.EQ & Z = 1 \\
            %         \hline
            %         \(\neq\) & B.NE & Z = 0 & B.NE & Z = 0 \\
            %         \hline
            %         \(<\) & B.LT & N = V & B.LO & C = 1 \\
            %         \hline
            %         \(\le\) & B.LE & ~ (Z = 0)&(N = V) & B.LS & ~(Z = 0)(C = 1) \\
            %         \hline
            %         \(>\) & B.GT & (Z = 0)&(N = V) & B.HI & (Z = 0)&(C = 1) \\
            %         \hline
            %         \(\ge\) & B.GE & N = V & B.HS & C = 1 \\
            %         \hline
            %     \end{tabular}
            % \end{center}\footnote{Copied from slides, probably need to rework}
            To check if a branch went out of bounds signed numbers could be treated as unsigned numbers and compared to a negative number (MSB = 1) so out of bounds can be identified.
            %look this up again TODO

            \subsection{Procedures}
                Procedures are subroutines of a program and are good for implementing abstraction in the program. For a procedures to work the hardware needs to be able to perform the following steps:
                \begin{enumerate}
                    \item save parameters in a place where the procedures can access them (X0 - X7)
                    \item give procedure the control
                    \item acquire storage resources for the procedure
                    \item do the task
                    \item put the result in a place where the main program can access it (X0 - X7)
                    \item return control the previous procedure (return address saved in LR(X30) )
                \end{enumerate}
                ARMv8 supports the branch-and-link instruction (BL). It branches to the procedure address and writes the return address in X30. This is needed because if a procedure
                is called by different parts of the program so the return doesn't have to be hardcoded.%not 100% sure
                The caller calculates the return address by adding 4 to the program counter. The current program counter points to the branch so it need to go one step further.
                The registers should hold the same value after the branch back so registers are saved into a stack before the program branches off. This is called spilling.
                Here the stack pointer (SP) is needed.
                Its a register which saves the last spilled address. The operations push and pop, adds or retrieves elements to/from the stack. The stack grows from high to low, so if an
                element is pushed to the stack, the value in the stack pointer needs to be decreased. \\
                X9 - X17 are registers which aren't preserved by a procedure call, X19 - x28 will be restored if necessary.\\
                C classifies variables into automatic and static, static variables are those declared outside a procedure. In ARMv8
                a so called global pointer points to the static area. A lot of ARMv8 compilers reserve X27 as the GP (global pointer).\\
                The stack can be also used to store variables which don't have space in the registers. It a segment in the stack called procedure frame or activation record.
                % STACK POINTER FRAME POINTER TODO
        \subsection{addresses}
            Basically nothing to compared to the the already done addressing section.
        \subsection{program}
            A programing language like c compiles it's code into assembly code. Assembly code is a symbolic language which will be translated into machine code. It uses the
            symbol table which matches the names of labels to the corresponding addresses in memory. It creates an object file which typically exists in 6 pieces:
            \begin{enumerate}
                \item file header which describes size and position of the other pieces in the object file
                \item text segment which contains the machine language code
                \item static data segment (UNIX allows static data which is allocated throughout the program and dynamic data which can grow and shrink as needed)
                \item relocation information identifies instructions and data which rely on absolute addresses (and probably adjusts them accordingly)
                \item symbol table contains not defined labels like external references
                \item debugging information contains descriptions on how a the modules were compiled
            \end{enumerate}
            The linker or link editor that links the independent compiled modules and resolves the undefined labels to create one executable file. The executable is loaded
            by the loader (at least in UNIX). It reads the file header to determine the size of the text and data segments. Then creates an address space large enough to fit all and
            copies the data into memory. If there are parameters they will also be copied into memory. The registers of the processor are initialized and the stack pointer is set to the
            first free location. At last it branches to a start up routine which initializes the argument registers with parameters and then calls the main routine. If the main routine is done
            the program terminates with an 'exit' system call.
    \section{Peripherals}
        A computer system needs to be able to support different kinds of peripherals.
            \subsection{Bus System}
            Bus-Systems in computers are there so different components are able to talk to each other. Typically the critical components (cpu, ram,...) have
            their own bus system (AHP or ASP bus system) and peripherals or none critical components (keyboard, interrupt time,...) share another bus system (APB bus system).
            They are connected via so called 'bridge', which connects (bridges) both bus systems if needed (if the keyboard input needs to be processed by the cpu).
            \subsection{Interrupt, Exceptions, Trap}
            Sometimes the flow of a program needs to be interrupted, for example when a button is pressed. A program could see in a while loop if a button was pressed and react
            accordingly, but this wastes resources and as long as the while loop is active, the program can't do anything else. Because a program is deterministic (if everything stays the
            same, the program will always behave in the same way), it can be interrupted. If the interrupt is done (interrupt function is done) the pre interrupt state needs to be
            restored so the program can work as it did before. \\
            What an interrupt is and what an exceptions is defined by the processor manufacturer. Interrupts and exceptions can occur internal or external though.
            Internal means that an invalid instructions is requested (user wants to access admin resources, div by zero) and external typically means that an
            the program is influenced from outside of the computer (usb stick is plugged in, power button is pressed). \\
            If an exception occurs, a branch will happen. The address of the branch target can either be fixed (division by zero interrupt has an own constant address, and other
            exceptions also have constant addresses ) or the branch address is saved in a exception table where the processor first reads the address (depending on the exception)
            and then jumps to the given address.
            %vector and register based exception handlers
            Context switches (when registers need to be saved so another (part of a) program can be executed):
            \begin{enumerate}
                \item process change
                \item exceptions
                \item sub-program calls
            \end{enumerate}

    \section{Bus Systems}
        There are many different kinds off bus systems and they can be classified in multiple categories:
        \begin{itemize}
            \item on/off chip busses
            \item parallel/serial busses
            \item synchronous/asynchronous busses
            \item automotive busses
            \item (and more)
        \end{itemize}
        In general, busses consists of a couple of wires and are designed to allow computer components to communicate to each other. Parallel busses couple at least two
        wires to transfer information in (nearly) the same direction (example: cpu uses 5 wires to write to a disc). There are often multiple data sources (sender)
        and data sink (receiver) on a bus (a sink can also be a source and vice versa). Other problems are that it needs to be defined how data can travel from source to
        sink so there also must be a well defined timing behavior.
        \subsection{General Definitions}
            See slide in moodle, too compact to summarize here.
        \subsection{UART, RS232}
            Before the data can be send, sender and receiver need to agree on some parameters of the transfer:
            \begin{itemize}
                \item full or half duplex
                \item number of bts per character
                \item band rate speed
                \item use parity or not
                \item if parity is used, how many bits
                \item number of stop bits (at least the number receiver needs)
                \item mark and space symbols %look this up
            \end{itemize}
            8N1 was a common implementation. 8 data bits, one stop bit and no parity bit (+ 1 start bit)%I think
            . If the band rate is set to then, then one just needs
            divide the signal rate by ten to get how many character were sent per second.
        \subsection{I\(^2\)C}
            I\(^2\)C(also called 'I two C') is a multi master bus and is commonly used for low speed peripherals. Multi master means that any component on the bus can be a
            master and communicate with the other components on the bus (they are called slaves). Masters can also switch as soon the current master is finished with its
            operations. SMBbus is a subset of I\(^2\)C and defines stricter protocols and electrical conventions to promote robustness and interoperability. A master is
            the component who issues the clock for the timing while the slave receives the clock. \\
            A master is initially in master transmit mode. It sends a start bit, followed by the salve address the master wants to communicate followed by one bit which
            represents if the master wants to read(1) or write (0). If the slave exists it sends an ACK (acknowledge) bit and as soon the master receives it, the master changes
            it mode to transfer/receive (depending what the master wants to do) and the slave changes to its complementary mode. \\
            I\(^2\)C defines three basic messaging types, while each begin with START and end with a STPO:
            \begin{enumerate}
                \item single message where master writes to slave
                \item single message where master reads from slave
                \item 'combined' message where master reads or writes at least 2 times to one or more slaves
            \end{enumerate}
\end{document}


% Created 2018-11-20 Tue 17:34
% Intended LaTeX compiler: pdflatex
\documentclass[11pt]{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{graphicx}
\usepackage{grffile}
\usepackage{longtable}
\usepackage{wrapfig}
\usepackage{rotating}
\usepackage[normalem]{ulem}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{textcomp}
\usepackage{amssymb}
\usepackage{capt-of}
\usepackage{hyperref}
\author{dustin}
\date{\today}
\title{}
\hypersetup{
 pdfauthor={dustin},
 pdftitle={},
 pdfkeywords={},
 pdfsubject={},
 pdfcreator={Emacs 25.3.2 (Org mode 9.1.7)},
 pdflang={English}}
\begin{document}

\section*{Budgeter}
This program is designed to provide the user with an optimized budget for the
items given. The user should provide a list of items in the form
\begin{center}
\begin{tabular}{ccccc}
Item Name & Importance & Min cost & Max cost & Needed\\
"String" & 1-10 & >0 & > Min cost & True/False\\
\end{tabular}
\end{center}
As well as a the number of year until retirement and the total saved and or
owed.
The algorithm will find the budget allocation that will maximize total value
\begin{align*}
  V_t = \sum_i V_i
\end{align*}
where each value \(V\) is determined by the function.
% \begin{align*}
%   V = -\frac{I}{10} \left( \frac{1}{k(x-m)+0.1} + 10 \right)
% \end{align*}
\begin{align*}
  V &= -I((k(x-max))^2-1) \\
  V &= -I(k(x-max))^2-I \\
  V &= -Ik^2(x-max)^2-I \\
  V &= -Ik^2(x^2-2(max)x+(max)^2)-I \\
  V &= -Ik^2x^2+2Ik^2(max)x-Ik^2(max)^2-I \\
\end{align*}
Where x is the price, I is the importance of the item given by the user,
\(k=\frac{1}{max-min}\), and \(m\) is the maximum price. The derivative in
regard to price is therefore
\begin{align*}
  \frac{dV}{dx} &= -2Ik^2x^2 + 2Ik^2(max)
\end{align*}
The restriction would be
\begin{align*}
  Income &= \sum_i x_i
\end{align*}
The solution of which can be found using the method of langrange multipliers
\begin{align*}
  \mathcal{L} &= f(\vec{x}) - \lambda(g(\vec{x} - b)
\end{align*}
Assuming I only have two costs
\begin{align*}
  Income &= x_1 + x_2 = g(x_1,x_2) = b
\end{align*}
and
\begin{align*}
  V_t &= V_1 + V_2
\end{align*}
At the critical points
\begin{align*}
  \nabla \mathcal{L} &= 0
\end{align*}
Which would lead to a set of linear equations
\begin{align*}
  -2Ik_1^2x_1 + 2Ik_1^2(max_1) &= \lambda \\
  -2Ik_2^2x_2 + 2Ik_2^2(min_2) &= \lambda \\
  x_1 + x_2 &= X_{total}
\end{align*}
which can be simplified to
\begin{align*}
  -2I_1k_1^2x_1 - \lambda &=  -2I_1k_1^2(max_1)\\
  -2I_2k_2^2x_2 - \lambda &=  -2I_2k_2^2(max_2) \\
  x_1 + x_2 &= X_{total}
\end{align*}
Which corresponds to the linear equation
\begin{equation}
\begin{bmatrix}
  -2I_1k_1^2 & 0 & -1\\
  0 & -2I_2k_2^2 & -1\\
  1 & 1 & 0\\
\end{bmatrix}
\begin{bmatrix}
  x_1 \\ x_2 \\ \lambda
\end{bmatrix}
=
\begin{bmatrix}
  -2I_1k_1^2(max_1) \\ -2I_2k_2^2(max_2) \\ X_{total}
\end{bmatrix}
\end{equation}
The solution for this equation is easily found and which can easily be
generalized.
\\
Note that an item may be such a bad deal, that the algorithm gives its value a negative
number. Of course you can't have a negative number so the item(s) will be
temporarily removed and the algorithm ran again.  The items will later be added
to final result but with their value over needed set to 0.
\newpage
\subsection*{Test Case Without Savings}
Lets assume we have two items
\begin{itemize}
\item ``efficient'' 10 0 1000 1
\item ``average 5 0 3000 1
\item ``inefficient'' 1 0 5000 1
\end{itemize}
We have a total cost of 4000.
Lets calculate our $k's$
\begin{align*}
  k_1 &= 0.001 \\
  k_2 &= 3.33e-4 \\
  k_3 &= 2e-4
\end{align*}
Our equation should then look like
\begin{equation}
\begin{bmatrix}
  -2(10)(0.001)^2 & 0 & 0 & -1\\
  0 & -2(5)(3.33e-4)^2 & 0 & -1\\
  0 & 0 & -2(10)(2e-4)^2 & -1\\
  1 & 1 & 1 & 0\\
\end{bmatrix}
\begin{bmatrix}
  x_1 \\ x_2 \\ \lambda
\end{bmatrix}
=
\begin{bmatrix}
  -2(10)(0.001)^2(1000) \\ -2(5)(3.33e-4)^2(3000) \\ -2(1)(2e-4)^2(5000) \\4000
\end{bmatrix}
\end{equation}

\begin{equation}
\begin{bmatrix}
  -2e-5 & 0 & 0 & -1\\
  0 & -1.1e-6 & 0 & -1\\
  0 & 0 & -8e-7 & -1\\
  1 & 1 & 1 & 0\\
\end{bmatrix}
\begin{bmatrix}
  x_1 \\ x_2 \\ x_3 \\ \lambda
\end{bmatrix}
=
\begin{bmatrix}
  -0.02 \\ -3.32e-3 \\ -4e^-4 \\ 4000
\end{bmatrix}
\end{equation}
Answer = [989,2804,206], Yay!

\subsection*{Special treatment for needed}
If an object is ``needed'' then the minimum value of the budget item should be
removed for income, and the min and max should be set to
\begin{align*}
  min &= 0 \\
  max &= max-min
\end{align*}
The $min$s should be added the vector outputted by calculatebudget

\newpage
\subsection*{Determining Savings}
The primary goal of this calculation is to assure that at retirement that you
can survive while paying your current costs. This will happen when you are able
to live off the interest of your investment. For this to workout, the future
value of your savings/debt + savings/year need to be equal to the (costs/year)/apr
The future value can calculated for each future year taking the

\begin{align*}
  V_1 = V_0(1+r) + S
\end{align*}
Where $V_1$ is the current value of savings in year 1, $V_0$ is the value of the savings
year 0 and $S$ is the savings over the year, and $r$ is the annual interest rate.

\subsection*{Choosing importance}
Importance should be scored either in range from 1-10 or from 1-100. A good rule
of thumb is that the score should roughly equal to the percentage of your time
that the effects of the item \textbf{linger} throughout your life. Lets do a few
examples, I will use a scale from 1-10
\subsubsection*{Food}
You may spend anywhere from $30\,min$ to $2\,hr$ eating food throughout the day
which means would mean that you would score it a 1 if I didn't include the word
linger. Though you don't spend a lot of time eating, the nutritional effects of
the food you eat will stay with you the entire day, $100\%$ of your life. While
the ``feel good'' effect of a delicious meal will last a few hours after you
eat. We can divide food into two subsections with their importance values: taste=5, nutrition=10.
Nutrition can be treated as a needed primary price, while taste can be treated
as a premium. A simpler approach would be to average the two numbers to get
food=8, and set a price range appropriate for delicious healthy food.
\subsubsection*{Transportation}
Similar to food, transportation can be separated into a few categories:
Reliability, Convenience, and Status. Reliability affects my entire life and I
would thus score it a 10. Convenience features only affects me while I drive, or
about and hour a day, so I'll score it a 1. Status, for me, will only affect me
in extremely rare occasions, so it'll also give me a 1. Note that I'm married.
For those looking for a partner, the status may possibly get more weight since
the likelihood of the affects could linger increases. Since the importance of
reliability so dramatically surpasses the other considerations, I'll only
consider it during my purchase. Transport:10, min-max based on reliability.
\subsubsection*{Phone}
I'll split the phone into parts: Coverage, Features, and Status.
I give coverage a score of 5, since it's usually not a big deal if I loose
signal while traveling, since I'm usually with others. Features gets a 10, since
I need access to email, text messages, video phone calls, and navigation, for
work. Status receives a 1 for reasons already discussed. Nearly all phones have
good coverage, so my focus will only be on the available features and status.
Status is often cheap with phones on a per/month basis compared to your other costs.
\subsubsection*{Housing}
This has three major parts: Location, size, and status. Location will get a 10,
since it will affect your entire life: education for your kids, commute to work,
neighbors, nearby restaurants etc. The size of the house would get a 5, since I
spend roughly half my time inside my house, while status once again would get a 1.

My input would thus look something like this
\begin{center}
\begin{tabular}{lcccc}
  After-Tax Monthly Income= 1750 &&&&\\
  Saved/Owed= 5000 &&&&\\
  Years till retirement= 25 &&&&\\
  APR = 0.05 &&&&\\
  Item Name & Importance & Min cost & Max cost & Needed\\
  Food(Nutrition) & 10 & 200 & 500 & 1 \\
  Food(Taste) & 5 & 0 & 200 & 0 \\
  Transportation(Reliability) & 10 & 250 & 400 & 1 \\
  Transportation(Status) & 1 & 0 & 500 & 1 \\
  Phone(Features) & 10 & 90 & 120 & 1 \\
  Phone(Status) & 1 & 0 & 30 & 0 \\
  Housing(Location) & 10 & 400 & 1200 & 1 \\
  Housing(Size) & 5 & 0 & 4000 & 0 \\
  Housing(Status) & 1 & 0 & 2000 & 0 \\
\end{tabular}
\end{center}
\end{document}


% This is part of the TFTB Reference Manual.
% Copyright (C) 1996 CNRS (France) and Rice University (US).
% See the file refguide.tex for copying conditions.

\markright{tfrgabor}
\section*{\hspace*{-1.6cm} tfrgabor}

\vspace*{-.4cm}
\hspace*{-1.6cm}\rule[0in]{16.5cm}{.02cm}
\vspace*{.2cm}

{\bf \large \sf Purpose}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
Gabor representation of a signal.
\end{minipage}
\vspace*{.5cm}

{\bf \large \sf Synopsis}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
\begin{verbatim}
[tfr,dgr,gam] = tfrgabor(x)
[tfr,dgr,gam] = tfrgabor(x,N)
[tfr,dgr,gam] = tfrgabor(x,N,Q)
[tfr,dgr,gam] = tfrgabor(x,N,Q,h)
[tfr,dgr,gam] = tfrgabor(x,N,Q,h,trace)
\end{verbatim}
\end{minipage}
\vspace*{.5cm}

{\bf \large \sf Description}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
	{\ty tfrgabor} computes the Gabor representation of signal {\ty x},
        for a given synthesis window {\ty h}, on a rectangular grid of size
        {\ty (N,M)} in the time-frequency plane. {\ty M} and {\ty N} must
        be such that {\ty N1 = M * N / Q} where {\ty N1=length(x)} and {\ty
        Q} is an integer corresponding to the degree of oversampling. The
        expression of the Gabor representation is the following :
\begin{eqnarray*}
G_x[n,m;h] &=& \sum_k x[k]\ h^*[k-n]\ \exp{[-j2\pi m k]}
\end{eqnarray*}

\hspace*{-.5cm}\begin{tabular*}{14cm}{p{1.5cm} p{8.5cm} c}
Name & Description & Default value\\
\hline
        {\ty x}   & signal to be analyzed ({\ty length(x)=N1})\\
        {\ty N}   & number of Gabor coefficients in time ({\ty N1} must be a multiple
              of {\ty N})                 & {\ty divider(N1)}\\
        {\ty Q}   & degree of oversampling ; must be a divider of {\ty N} &
		{\ty Q=divider(N)}\\
        {\ty h}   & synthesis window, which was originally chosen & {\ty
			window(odd(N),}\\
		& as a Gaussian window by Gabor. {\ty Length(h)} should be
		as closed as possible from {\ty N}, and must be $\geq${\ty N}.
              {\ty h} must be of unit energy, and centered & {\ty
			'gauss')}\\
        {\ty trace} & if nonzero, the progression of the algorithm is shown
                                         & {\ty 0}\\
     \hline {\ty tfr} & square modulus of the Gabor coefficients\\
        {\ty dgr} & Gabor coefficients (complex values)\\
        {\ty gam} & biorthogonal (dual frame) window associated to {\ty h}\\

\hline
\end{tabular*}
\vspace*{.2cm}

When called without output arguments, {\ty tfrgabor} runs {\ty tfrqview}.\\
\end{minipage}

\newpage

\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
If {\ty Q=1}, the time-frequency plane (TFP) is critically sampled, so
there is no redundancy.\\
If {\ty Q>1}, the TFP is oversampled, allowing a greater numerical
stability of the algorithm. \\

\end{minipage}
\vspace*{1cm}

{\bf \large \sf Example}
\begin{verbatim}
         sig=fmlin(128);
         tfrgabor(sig,64,32);
\end{verbatim}
\vspace*{.5cm}

{\bf \large \sf See Also}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
all the {\ty tfr*} functions.
\end{minipage}
\vspace*{.5cm}

{\bf \large \sf References}\\
\hspace*{1.5cm}
\begin{minipage}[t]{13.5cm}
[1] Zibulski, Zeevi "Oversampling in the Gabor Scheme" IEEE Trans. on
	    Signal Processing, Vol. 41, No. 8, pp. 2679-87, August 1993.\\

[2] Wexler, Raz "Discrete Gabor Expansions" Signal Processing, Vol. 21, No.
	    3, pp. 207-221, Nov 1990.

\end{minipage}


%%%%%%%%%%%%%%%%%%%%%definitions%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\input{../../doc/related_pages/header.tex}
\input{../../doc/related_pages/newcommands.tex}
\newcommand{\PP}{H^+}
\newcommand{\QQ}{H^-}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%DOCUMENT%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}

\title{The heat diffusion project}
\author{ M.~Wiesenberger and M.~Held}
\maketitle

\begin{abstract}
    Test parallel advection and diffusion schemes on the 1d Navier-Stokes equations
    along a parallel magnetic field.
\end{abstract}
\tableofcontents

\section{Parallel advection schemes}
In this section we want to focus on the discretization of the Navier-Stokes equation
along a magnetic field-line:
\begin{align}
    \frac{\partial}{\partial t} n &= - \nc\left( nu\bhat\right) \\
    \frac{\partial}{\partial t} u &= - \npar\left(\frac{u^2}{2}\right) - \frac{\tau}{\mu} \frac{\npar n}{n} + \nu_u \frac{\Delta_\parallel u}{n}
\end{align}
with
\begin{align}
\Delta_\parallel u &= \nabla\cdot ( \bhat\bhat\cdot\nabla u)
\end{align}
$\bhat = \bhat(\vec x)$ is the prescribed magnetic field unit vector,
and $\nu_u$ is the viscosity coefficient parallel to this field.

The first equation is a linear advection type equation while the second one includes
a quadratic non-linearity (the Burger term $u^2/2$) which tends to generate
shocks (at least in the dissipationless case). This is because crests of a wave
travel faster than the valley of waves.
The continuity equation can be written as
\begin{align}
    \frac{\partial}{\partial t} n = - \npar (nu) - nu \nc\bhat = -u\npar n - n \npar u - nu\nc \bhat
\end{align}
The first term on the right hand side is the advection term (of density with velocity u). The second term is the wave term, because together with $\npar n$ in the velocity
equation it couples the two equations to make waves. This can be seen by linearizing
the system around $n= 1 + \tilde n$, $u = \tilde u$
\begin{align}
    \frac{\partial^2}{\partial t^2} \tilde n = \frac{\tau}{\mu} \npar^2 \tilde n + \frac{\tau}{\mu} \npar \tilde n \nc\bhat
\end{align}
This equation describes sound waves with speed $\sqrt{\tau/\mu}$ and a
dispersive term.

The above equations have the following conservative formulation:
\begin{align}
    \frac{\partial}{\partial t} n  + \nc\left( nu \bhat\right) &= 0 \\
    \frac{\partial}{\partial t} (\mu nu)  + \nc\left(\mu nu^2 \bhat\right) &=
    -\tau \npar n + \mu \nu_u \Delta_\parallel u \\
    \frac{\partial}{\partial t} \left(\tau n \ln n + \frac{1}{2}\mu nu^2\right)
    + \nc\left( \left( \tau n ( 1 + \ln n)  +\mu \frac{1}{2} nu^2\right) u\bhat \right) &= \mu \nu_u u \Delta_\parallel u
\end{align}
A good scheme:
\begin{itemize}
    \item respects the above written conservation equations, in particular when
        integrated over a flux-aligned volume the flux terms must vanish
    \item works with the FCI discretization developed in previous sections
\end{itemize}
Note that the latter point excludes discontinuous Galerkin methods, which we have not
yet developed for the FCI discretization.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Staggered grids in 1d}
When $\bhat = \xhat$ the system reduces to the one-dimensional Navier-Stokes equations for which there is a large amount of literature on numerical schemes available
( see for example \cite{LeVeque} and references therein).
\begin{align}
    \
    \frac{\partial}{\partial t} n &= - \frac{\partial}{\partial x}\left( nu\right) \\
    \frac{\partial}{\partial t} u &= - \frac{\partial}{\partial x}\left(\frac{u^2}{2} + \frac{\tau}{\mu} \ln n\right) + \nu_u \frac{1}{n}\frac{\partial^2}{\partial x^2} u
\end{align}

We here choose a class of schemes known as staggered grids. The main is to discretize
the variables $n$ and $u$ on two grids that are adjoint to each other.
If $n$ is discretized on the cell-centers $x_k$ then $u$ is discretized on the cell
boundaries $x_{k+1/2}$. We will indicate this by writing $u^\dagger_k$ that is $u^\dagger_k = u_{k+1/2}$.
We base our discretization on the finite volume scheme presented by
\cite{Gunawan2015,Herbin2013}:
\begin{align}
    \frac{n_k^{n+1} - n_k^n}{\Delta t} = - \frac{1}{\Delta x}(\hat q^\dagger_k  - \hat q^\dagger_{k-1})
\end{align}
where
\begin{align}
    \hat q^\dagger_k := u^\dagger_k \begin{cases}
        n_k     + \frac{1}{2}\Lambda( n_{k+1}-n_k    , n_k - n_{k-1})&\text{ if } u^\dagger_k \geq 0 \\
        n_{k+1} - \frac{1}{2}\Lambda( n_{k+2}-n_{k+1}, n_{k+1} - n_k)&\text{ if } u^\dagger_k < 0
    \end{cases}
\end{align}
and
\begin{align}
    \frac{(n_k^\dagger u_k^\dagger)^{n+1}-(n_k^\dagger u_k^\dagger)^{n}}{\Delta
    t} = - \frac{1}{\Delta x} \left[ \hat f_{k+1} - \hat f_k -
    \frac{\tau}{\mu} \left(n_{k+1}^{n+1} - n_k^{n+1}\right)  \right]
\end{align}
where $n^\dagger_k = \left(n_{k+1}+n_k\right)/2$ and $u_k =
\left(u^\dagger_k + u^\dagger_{k-1}\right)/2$ (local shock speed in Burger's equation)
and analogous for $q_k$ and
\begin{align}
    \hat f_k := q_k \begin{cases}
        u^\dagger_{k-1} + \frac{1}{2}\Lambda( u^\dagger_{k}-u^\dagger_{k-1}, u^\dagger_{k-1} - u^\dagger_{k-2})&\text{ if } q_k \geq 0 \\
        u^\dagger_k     - \frac{1}{2}\Lambda( u^\dagger_{k+1}-u^\dagger_k    , u^\dagger_k - u^\dagger_{k-1})&\text{ if } q_k < 0
    \end{cases}
\end{align}
where we have the flux limiter
\begin{align}
\Lambda_{\text{minmod}}(x,y) &:=
    \begin{cases}
        \min(x,y) & \text{ if } x,y \geq 0 \\
        \max(x,y) & \text{ if } x,y \leq 0 \\
        0         & \text{ else }
    \end{cases}
    \\
\Lambda_{\text{vanLeer}}(x,y) &:=
    \begin{cases}
        0               & \text{ if } xy \leq 0 \\
        \frac{2xy}{x+y} & \text{ else }
    \end{cases}
\end{align}

We note that this is a semi-implicit scheme (the pressure term in the momentum equation is implicit). The implicit equation can however be solved trivially because
the density equation decouples and is completely explicit.

We implement this scheme and investigate several variations in this notebook
\url{https://mwiesenberger.github.io/advection/OneDimensional.html}.

The authors of the scheme claim that if the pressure term is treated explicitly the
scheme loses its shock conserving properties. We cannot reproduce this behaviour
in our experiments. Furthermore, we were looking into discretizing the
velocity equation over the momentum formulation because the Feltor equations
prefer that formulation. In this formulation the equations are not shock-capturing
but are very close to the original scheme once viscosity is added to the system.

The best scheme that we found in this formulation is
\begin{align}
    \frac{n_k^{n+1} - n_k^n}{\Delta t} = &- \frac{1}{\Delta x}(q^\dagger_k  - q^\dagger_{k-1}) \\
    \frac{(u_k^\dagger)^{n+1}-(u_k^\dagger)^{n}}{\Delta t} = &- \frac{1}{\Delta
    x} \left( \hat f_{k+1} -\hat f_k\right) - \frac{1}{\Delta x}\left[
        \frac{\tau}{\mu} \left(n_{k+1} - n_k\right) \frac{1}{2}\left(\frac{1}{n_{k+1}}
    +\frac{1}{n_k}\right)\right]
    \nonumber\\
    &+ \frac{\nu_u}{n^\dagger_k (\Delta x)^2}
    \left(u^\dagger_{k+1} - 2 u^\dagger_k + u^\dagger_{k-1}\right)
\end{align}
with
\begin{align}
    \hat q^\dagger_k &:= u^\dagger_k \begin{cases}
        n_k     + \frac{1}{2}\Lambda( n_{k+1}-n_k    , n_k - n_{k-1})&\text{ if } u^\dagger_k \geq 0 \\
        n_{k+1} - \frac{1}{2}\Lambda( n_{k+2}-n_{k+1}, n_{k+1} - n_k)&\text{ if } u^\dagger_k < 0
    \end{cases}
    \\
    \hat f_k &:= \frac{1}{2}u_k \begin{cases}
        u^\dagger_{k-1} + \frac{1}{2}\Lambda( u^\dagger_{k}-u^\dagger_{k-1}, u^\dagger_{k-1} - u^\dagger_{k-2})&\text{ if } u_k \geq 0 \\
        u^\dagger_k     - \frac{1}{2}\Lambda( u^\dagger_{k+1}-u^\dagger_k    , u^\dagger_k - u^\dagger_{k-1})&\text{ if } u_k < 0
    \end{cases}
\end{align}
\begin{tcolorbox}[title=Note]
    We divide by the harmonic mean in the pressure gradient term, which from empirical tests
    yields better convergence than other types like the arithmetic
    or geometric mean
\end{tcolorbox}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{Elevating the scheme to three dimensions}
Our idea to generalize the scheme to the FCI formulation is to use the
locally fieldaligned coordinate system.
Care must be taken as we need to integrate the fieldlines twice, once to the
next plane and once to the adjoint plane halfway between the original planes.

We denote the interpolation between the original planes with $I^+$ and $I^-$ and
the interpolation between the half planes with $\PP$ and $\QQ$.
We will use $\varphi$ as the fieldline-following coordinate as the resulting
grid is then equidistant in the transformed coordinates.

First, we denote quantities transformed into the coordinate system centered on $k$ as
\begin{align}
    \bar n_{k-2} = (I^-)^2n_{k-2} \quad \bar n_{k-1} = I^- n_{k-1}
    \quad \bar n_k = n_k \quad
    \bar n_{k+1} = I^+n_{k+1} \quad \bar n_{k+2} = (I^+)^2 n_k \nonumber\\
\bar u_{k-1/2} = \QQ u_{k-1}^\dagger \quad \bar u_{k+1/2} = \PP u_k^\dagger \qquad\qquad\qquad\qquad
\end{align}
With these definitions we can discretize the continuity equation by
\begin{align}
    \frac{n_k^{n+1} - n_k^n}{\Delta t} = &-\frac{1}{\Delta\varphi \sqrt{G^k}}
    \left( \sqrt{G^{k+1/2}}b^\varphi_{k+1/2}  \bar q_{k+1/2}
    -\sqrt{G^{k-1/2}}b^\varphi_{k-1/2}  \bar q_{k-1/2} \right)
\end{align}
where $\sqrt{G^{k\pm 1/2}}$ and $b^\varphi_{k\pm 1/2}$ are the volume form and
the contravariant $\varphi$ component of $\bhat$ on the half planes and
\begin{align}
    \bar q_{k+1/2} &:= \bar u_{k+1/2} \begin{cases}
        \bar n_k     + \frac{1}{2}\Lambda( \bar n_{k+1}-\bar n_k    , \bar n_k - \bar n_{k-1})&\text{ if } \bar u_{k+1/2} \geq 0 \\
        \bar n_{k+1} - \frac{1}{2}\Lambda( \bar n_{k+2}-\bar n_{k+1}, \bar n_{k+1} - \bar n_k)&\text{ if } \bar u_{k+1/2} < 0
    \end{cases}
\end{align}
Next, we denote quantities transformed into the coordinate system centered on the adjoint plane $k$ as
\begin{align}
    \bar u^\dagger_{k-2} = (I^-)^2u^\dagger_{k-2} \quad \bar u^\dagger_{k-1} = I^- u^\dagger_{k-1}
    \quad \bar u^\dagger_k = u^\dagger_k \quad
    \bar u^\dagger_{k+1} = I^+u^\dagger_{k+1} \quad \bar u^\dagger_{k+2} = (I^+)^2 u^\dagger_k \nonumber\\
    \bar n^\dagger_{k-1/2} = \QQ n_{k} \quad \bar n^\dagger_{k+1/2} = \PP n_{k+1} \qquad\qquad\qquad\qquad
\end{align}
Such that for the velocity equation we have
\begin{align}
    \frac{(u_k^\dagger)^{n+1}-(u_k^\dagger)^{n}}{\Delta t} = &- \frac{b^\varphi_k}{\Delta
    \varphi} \left( \bar f^\dagger_{k+1/2} -\bar f^\dagger_{k-1/2}\right) -\frac{b^\varphi_k}{\Delta\varphi}
        \left[\frac{\tau}{\mu} \left(\bar n^\dagger_{k+1/2} - \bar n^\dagger_{k-1/2}\right) \frac{1}{2}\left(\frac{1}{\bar n^\dagger_{k+1/2}}
    +\frac{1}{\bar n^\dagger_{k-1/2}}\right)\right]
    \nonumber\\
    &+ \frac{2\nu_u
        \left(\sqrt{G^{k+1/2}}b^\varphi_{k+1/2}b^\varphi_{k+1/2} (\bar u^\dagger_{k+1} - \bar u^\dagger_k)
        - \sqrt{G^{k-1/2}}b^\varphi_{k-1/2}b^\varphi_{k-1/2} (\bar u^\dagger_k - \bar u^\dagger_{k-1})\right)
    }{\sqrt{G^k}(\bar n^\dagger_{k+1/2} + \bar n^\dagger_{k-1/2}) (\Delta \varphi)^2}
\end{align}
The flux is discretized according to
\begin{align}
    \bar f^\dagger_{k+1/2} &:= \frac{1}{4}(\bar u^\dagger_{k+1}+\bar u^\dagger_k)
    \begin{cases}
        \bar u^\dagger_{k} + \frac{1}{2}\Lambda\left( \bar u^\dagger_{k+1} - \bar u^\dagger_k, \bar u^\dagger_k - \bar u^\dagger_{k-1}\right)&
        \text{ if } \frac{1}{2}(\bar u^\dagger_{k+1}+\bar u^\dagger_k) \geq 0 \\
        \bar u^\dagger_{k+1}   - \frac{1}{2}\Lambda\left( \bar u^\dagger_{k+2} - \bar u^\dagger_{k+1}, \bar u^\dagger_{k+1} -\bar u^\dagger_k \right) &
        \text{ if } \frac{1}{2}(\bar u^\dagger_{k+1}+\bar u^\dagger_k) < 0
    \end{cases}
\end{align}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\subsection{2nd Elevating the scheme to three-dimensions}
A second possibility is to first center the coordinate system on the fluxes on the staggered grid and then
in a second step center on the time-derivative:
\begin{align}
n_k^\dagger =& \frac{1}{2}\left( \QQ n_k + \PP n_{k+1}\right) \\
u_k =& \frac{1}{2}\left( \QQ u^\dagger_{k-1} + \PP u_{k}^\dagger\right)\\
\left[\partial_\parallel u^\dagger \right]_k =& \left( \PP u^\dagger_{k} - \QQ
u^\dagger_{k-1}\right) \\
\left[\partial_\parallel n\right]^\dagger_k =& \left( \PP n_{k+1} - \QQ
n_k\right)
\end{align}
Such that in total we have
\begin{align}
    \frac{n_k^{n+1} - n_k^n}{\Delta t} = &-\frac{1}{\Delta\varphi \sqrt{G^k}}
    \left( \sqrt{G^{k+1/2}}b^\varphi_{k+1/2}  \PP \hat q_{k}^\dagger
    -\sqrt{G^{k-1/2}}b^\varphi_{k-1/2}  \QQ \hat q_{k-1}^\dagger \right)
     \\
    \frac{(u_k^\dagger)^{n+1}-(u_k^\dagger)^{n}}{\Delta t} = &-
 \frac{b^\varphi_k}{\Delta\varphi}\left( \PP\hat f_{k+1} - \QQ\hat f_k\right)
 -\frac{\tau}{\mu}\frac{b^\varphi_k}{\Delta\varphi} \frac{\PP n_{k+1} - \QQ n_k}{2}\left( \frac{1}{\PP n_{k+1}} + \frac{1}{\QQ n_k} \right)
    \nonumber\\
    &+ \frac{\nu_u
        \left(\sqrt{G^{k+1/2}}b^\varphi_{k+1/2}b^\varphi_{k+1/2} (I^+ u^\dagger_{k+1} - u^\dagger_k)
        - \sqrt{G^{k-1/2}}b^\varphi_{k-1/2}b^\varphi_{k-1/2} ( u^\dagger_k - I^- u^\dagger_{k-1})\right)
    }{n^\dagger_{k} \sqrt{G^k}(\Delta \varphi)^2}
\end{align}
The fluxes are discretized according to
\begin{align}
    \hat q^\dagger_k &:= u^\dagger_k \begin{cases}
        \QQ n_k     + \frac{1}{2}\Lambda\left( (\partial_\parallel n)^\dagger_k, I^- (\partial_\parallel n)^\dagger_{k-1}\right)&\text{ if } u^\dagger_k \geq 0 \\
        \PP n_{k+1} - \frac{1}{2}\Lambda\left( I^+(\partial_\parallel n)^\dagger_{k+1}, (\partial_\parallel n)^\dagger_k\right) &\text{ if } u^\dagger_k < 0
    \end{cases}
    \\
    \hat f_k &:= \frac{1}{2}u_k\begin{cases}
        \QQ u^\dagger_{k-1} + \frac{1}{2}\Lambda\left( (\partial_\parallel u^\dagger )_k, I^- (\partial_\parallel u^\dagger)_{k-1}\right)&\text{ if } u_k \geq 0 \\
        \PP u^\dagger_{k}   - \frac{1}{2}\Lambda\left( I^+(\partial_\parallel u^\dagger)_{k+1}, (\partial_\parallel u^\dagger)_k\right) &\text{ if } u_k < 0 \\
    \end{cases}
\end{align}

\end{document}


\documentclass[10pt,a4paper]{article}

\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{a4wide}
\usepackage[table]{xcolor}
\usepackage[numbers]{natbib}

\begin{document}
\title{Feasibility-prone Differential Dynamic Programming \\
Is DDP a Multiple Shooting Algorithm?}
\author{N. Mansard -- LAAS-CNRS}

\maketitle

%% \begin{abstract}
%% \end{abstract}

\newcommand{\xtraj}{\underline{x}}
\newcommand{\utraj}{\underline{u}}
\newcommand{\lambdatraj}{\underline{\lambda}}
\newcommand{\dxtraj}{\underline{\Delta x}}
\newcommand{\dutraj}{\underline{\Delta u}}
\newcommand{\dxtrajguess}{\underline{\Delta \bar x}}
\newcommand{\dutrajguess}{\underline{\Delta \bar u}}
\newcommand{\dx}{\Delta x}
\newcommand{\du}{\Delta u}
\newcommand{\Treal}{\mathbb{T}}
\newcommand{\bmat}{\begin{bmatrix}}
\newcommand{\emat}{\end{bmatrix}}
\newcommand{\qed}{\hfill$\square$}
\definecolor{mygray}{gray}{0.9}

\section{Introduction}

\subsection{Problem definition}
We are interested to find an approximate solution to the following optimal control problem OCP:
$$\min_{\xtraj,\utraj} \int_0^\Treal \ell(x(t),u(t),t) dt + \ell_\Treal(x(\Treal))$$
$$s.t. \quad x(0) = f_0$$
$$\quad \forall t \in [0,\Treal], \quad \dot{x}(t) = f(x(t),u(t),t))$$
where $\xtraj: t \rightarrow x(t)$ is the state trajectory, $\utraj: t \rightarrow u(t)$ is the control trajectory, $\ell$ is the integral --running-- cost, $\ell_T$ is the terminal cost, $f_0$ is the initial state value, $f$ is the robot dynamics and $T$, the time interval, is fixed.

The decision variables are $\xtraj,\utraj$, both of infinite dimension.
We approximate this problem using a discrete version of it, by following the so-called direct --discretize first, solve second -- approach.

\subsection{Discretize first}
The time interval $[0,\Treal]$ is divided into $T$ sub-intervals (evenly distributed or not).
In each sub-interval $t$, the control trajectory $\utraj_t$ is constrained to be in the span of a given trajectory finite basis, and we represent the trajectory by its coefficient in the function basis (i.e as a vector of finite dimension). We typically write $\utraj_t$ as a polynomial, and it is often taken in practice constant on the interval.

The values of $\xtraj_t$ on the interval $t$ are obtained by integrating the dynamics from the value $x_t$ at the beginning of the sub-interval.
As closed-form integrals of $f$ are often not available, $\xtraj_t$ is approximated by any numerical integration scheme, e.g. Runge-Kutta-4. We then represent $\xtraj$ by its values at each interval ends, i.e. as a list of $T+1$ elements.

In summary, the control variable $\utraj$ is represented by $T$ basis coefficients of the chosen trajectory basis --which often boils to $T$ constant controls-- and $\xtraj$ is represented by $T+1$ states.
In the following, we will often abusively use the same symbols for the true object (e.g. the trajectory) and its representation (e.g. the coefficients of its discretization), in the aim of keeping the notations simple. With this choice, the discretized problem can be written as:
$$\min_{\xtraj,\utraj} \sum_{t=0}^{T-1} \ell(x_t,u_t) + \ell_T(x_T)$$
$$s.t. \quad x_0 = f_0$$
$$\quad \forall t=0..T-1, \quad x_{t+1} = f(x_t,u_t)$$
As announce, both $\ell$ and $f$ now represent the discretization of their respective objects in the original problem. They both typically depend on time (i.e. $\ell_t,f_t$) but we omit this dependency in the notation for readability.

\subsection{Solve second}
This new problem is now a static optimization problem under constraints, typically nonlinear and non-convex (NLP). We will solve it with a sequential-quadratic-programming (SQP) strategy, i.e. by iteratively solving the linear-quadratic-regulator (LQR) problem obtained by computing the linearization of the dynamics $f$ and the quadratic model of the cost $\ell$ at the current candidate values of $\xtraj,\utraj$.

We denote the derivatives of $f$ by $F_x, F_u$, and the gradient and the Hessian of $\ell$ by $(L_x, L_u$) and $(L_{xx}, L_{xu}, L_{ux}, L_{uu})$, respectively. When possible, we will omit the time indexes for all these quantities. For the LQR case, due to it is a finite-horizon problem, we can consider without loss of generality that the $F$ and $L$ are constant matrices (for general case, you just need to add the evident indices $_t$ into each quantity).
We also denote $f_t$ by the drift of $f$ (i.e. change in $x$ when $u$ is zero), whose role is clear for the LQR and whose role will become clear later for solving the NLP. The LQR is then formulated as:
$$\min_{\dxtraj,\dutraj} \Big( \sum_{t=0}^{T-1}  \frac{1}{2} \bmat \dx^T,\du^T \emat \bmat L_{xx} & L_{xu} \\ L_{ux} & L_{uu} \emat \bmat \dx \\ \du \emat + \bmat L_x & L_u \emat \bmat \dx \\ \du \emat $$
$$ + \frac{1}{2} \dx_T L_{xx} \dx_T + L_x \dx_T \Big)$$
$$s.t. \quad \dx_0 = f_0 $$
$$\quad \forall t=0\cdots T-1, \quad \dx_{t+1} = F_x \dx_t + F_u \du_t + f_{t+1} $$

This problem is a quadratic program (under linear equality constraints -- QP).
Various solutions can be chosen to solve the QP.
It is obvious to recall that all of them will lead to the same solution, at least neglecting numerical effects related to noise and numerical stability. We will favor two solutions.
For understanding the nature of the problem, we will write the solution to this QP by forming the KKT matrix. For solving it in practice, we will use the Ricatti recursion typical in differential dynamic programming (DDP).

\subsection{The Russian way: using the Karush-Kuhn-Tucker matrix}

\subsubsection{Optimality principle}
The Lagrangian of the LQR QP is:
$$\mathcal{L}(\dxtraj,\dutraj,\lambdatraj) = \sum_{t=0}^{T-1} \Big( \frac{1}{2} \bmat \dx_t^T,\du_t^T \emat \bmat L_{xx} & L_{xu} \\ L_{ux} & L_{uu} \emat \bmat \dx_t \\ \du_t \emat + \bmat L_x & L_u \emat \bmat \dx_t \\ \du_t \emat $$
$$- \lambda_{t+1} (\dx_{t+1} - F_x \dx_t - F_u \du_t - f_{t+1} ) \Big)
%$$ $$
+ \frac{1}{2} \dx_T L_{xx} \dx_T + L_x \dx_T - \lambda_0(\dx_0 - f_0) $$

\subsubsection{Solving the LQR QP}
The optimum of the QP is reached for the zero of the gradient of $\mathcal{L}$ with respect to $\xtraj$, $\utraj$ and $\lambdatraj$, i.e. when:
$$
\left[ \begin{array}{cccc!{\color{mygray}\vrule}ccc!{\color{mygray}\vrule}cccccc}
L_{xx}        & & & & L_{xu} & & & -I & F_x^T \\
& \ddots     & & & & \ddots& & & \ddots & \ddots \\
& & L_{xx}    & & & & L_{xu}& & & -I & F_x^T\\
& & & L_{xx}  & & & & & & & -I\\
\arrayrulecolor{mygray}\hline
L_{ux} & & &       & L_{uu} & & & & F_u^T\\
& \ddots & &       &       & \ddots & & & & \ddots\\
& & L_{ux} &       &       &       & L_{uu} & && & F_u^T \\
\arrayrulecolor{mygray}\hline
-I &&&&&&&&&&\\
F_x & -I &  & & F_u &&&&&& \\
&\ddots & \ddots & & & \ddots &&&&&&\\
&    &  F_x & -I & & & F_u
\end{array} \right]
%
\bmat
\dx_0 \\
\vdots \\
\dx_{T-1} \\
\dx_T \\
\arrayrulecolor{mygray}\hline
\du_0 \\
\vdots \\
\du_{T-1} \\
\arrayrulecolor{mygray}\hline
\lambda_0 \\
\lambda_1 \\
\vdots \\
\lambda_{T}
\emat
%
=
%
-\bmat
L_x \\
\vdots \\
L_x \\
L_x \\
\arrayrulecolor{mygray}\hline
L_u \\
\vdots \\
L_u \\
\arrayrulecolor{mygray}\hline
f_0 \\
f_1 \\
\vdots \\
f_{T}
\emat
$$

Solving this linear equation (by inverting the full-rank KKT matrix) provides both the optimal state and control trajectories $\dxtraj$, $\dutraj$ and the Lagrange multipliers corresponding to the robot dynamics.
These multipliers indeed represent the trajectory of the co-state at the shooting points.
Solving the LQR QP by searching the zeros of the Lagrangian indeed corresponds to applying the Pontryagin's Minimum Principle (PMP) on the discretize LQR system.

\subsubsection{Intuition of the results}
Solving the QP provides at the same time the satisfaction of the constraints (i.e. the resulting $\xtraj$ is a continuous state trajectory that corresponds to the continuous control trajectory $\utraj$, following the underlying linear integrator); and the resulting trajectory pair is of minimal (quadratic) cost.

Indeed, solving the QP can be seen as computing a step to modify an initial-guess $\dxtrajguess$, $\dutrajguess$. Such an observation is somehow trivial, as for a QP, any initial guess will lead to the exact same optimum in a single step.
However, understanding this observation is important before going to the more complex SQP case.
As any initial guess works the same, we typically consider $\dxtrajguess=0$ and $\dutrajguess=0$.
Then this initial guess is not feasible, i.e. it does not satisfy the constraints (except in the particular case where $f_0 = \cdots = f_{T-1} = 0$). These $f_t$ can then be seen as the gaps in the state trajectory, i.e. the state trajectory is piece-wise feasible inside each shooting interval, but is discontinuous at each shooting node $t$ with a gap between the previous piece of trajectory $t$ and the next one $t+1$ of $f_t$ ($f_0$ being a gap with respect to the initial guess state $\Delta \bar x_0= 0$).

Then solving the QP corresponds to both making a step that nullifies the gaps $f_t$ and a step that optimizes the trajectory cost.

\subsection{The American way: computing the optimal flow}

\subsubsection{Optimality principle}
DDP is generally not associated with the KKT matrix but with the backward Riccati recursion.
We denote the Value function (running cost) at time $t$ by $V_t: \dx \rightarrow \mathbb{R}$, and it represents the minimal cost we can obtained with the state being $\dx$ at time $t$.
We denote the Hamiltonian of the system (Q-value) by $Q_t: \dx,\du \rightarrow Q_t(\dx,\du) = \ell_t(\dx,\du) + V \circ \Delta f(\dx,\du)$, where the linear dynamics is denoted by $\Delta f(\dx,\du) = F_x \dx + F_u \du + f_t$.

As the problem is LQR, the Value and Hamiltonian functions have quadratic form; they can be represented by their gradient and their Hessian.
It is important to note that the gradient of a quadratic function is not constant but varies according to the point where it is computed. However, the gradient at any point can be easily computed using the gradient at a given point plus the Hessian times the difference between the two points $\nabla (a) = \nabla(b) + \nabla^2\cdot(a-b)$. And often, we conveniently compute the gradient at the origin.

\subsubsection{Solving the backward recursion}
From the Bellman principle, we know that $V_t(\dx) = \min_{\du} Q_t(\dx,\du)$.
A backward recursion can then be set up, starting from the final observation that $V_t = \ell_T$.
The backward recursion can be computed along any given guess trajectory $\dxtrajguess, \dutrajguess$, to compute $Q_t$ at each shooting node $t$ from $V_{t+1}$, and then $V_t$ by optimizing $Q_t(.,\du)$.
It is important to remember that any trajectory can be chosen, as the problem is LQR, hence the optimal flow $V$ can  be equivalently recovered from any trajectory $\dxtrajguess$.
In particular, the trajectory does not have to be optimal, feasible, or even continuous.

When computing backward the optimal flow, we only compute the $V$ values at the shooting times.
However, the flow exists at any time and it is implicitly handled by the recursion through the integrated dynamics $F_x,F_u$.
Then care has to be taken for discontinuous trajectories.
In such a case the flow $V_t$ would be typically computed at $\dx_t$, while the Jacobians $F_x$, $F_u$ are computed at the state reached at the end of interval $t-1$, i.e. $\dx_{t-1}^+ = F_x \dx + F_u \du + f_{t-1}$.
In particular, when $\dxtrajguess =0$ and $\dutrajguess=0$, then $\dx_{t-1}^+ = f_{t-1}$.
The gradient of $V$ in $\dx_{t-1}^+$ is obtained from the gradient of $V$ at $\dx_t=0$ with:
$$V_x^+ = V_x + V_{xx} f_t$$
and, off course, if $f_t=0$, then they are both equals.

The backward recursion is then twofold. First it propagates the Q function from $Q=\ell+V\circ \Delta f$:
$$Q_{xx} = L_{xx} + F_x^T V_{xx} F_x$$
$$Q_{xu} = L_{xu} + F_x^T V_{xx} F_u$$
$$Q_{ux} = L_{ux} + F_u^T V_{xx} F_x$$
$$Q_{uu} = L_{uu} + F_u^T V_{xx} F_u$$
$$Q_{x}  = L_x + F_x^T V_x^+$$
$$Q_{u}  = L_u + F_u^T V_x^+$$
The Value function is then obtained by solving the minimum of $Q(.,\du)$
$$\text{arg}\min_{\du} Q(\dx,\du) = -k - K \dx$$
with $k=Q_{uu}^{-1} Q_u$ and $K = Q_{uu}^{-1} Q_{ux}$. The Value is then:
$$V_{xx} = Q_{xx} - Q_{xu} K$$
$$V_x = Q_x - Q_{xu} k + V_{xx} f$$
where the gradient $V_x$ is computed at the end of the previous interval $x^+$ and not at the shooting state $x_t$.

To obtain the complete solution $\dxtraj,\dutraj$, a forward pass must then be performed.
We discuss it later.

\subsubsection{Intuition of the result}

While the KKT approach computes the solution using the dual $\lambda$ variable, the DDP approach computes it using the $V, Q$ auxiliary variables.
$\lambda$ presents the co-state trajectory, while $V$ is the value functions.
Both are connected, as the PMP principle writes the optimal control in term of the co-state, while HJB express the optimal policy in term of the Value space gradient.

The optimal flow is evaluated along the candidate trajectory $\dxtrajguess,\dutrajguess$.
As the problem is LQR, any initial guess produces the same backward pass, the same $V$ and the same solution.
For this reason, choosing $\dxtrajguess=0, \dutrajguess=0$ is very relevant.
In that case, if $f_t$ is nonzero, then the initial guess is not feasible and the term $V_{xx} f$ in the Value gradient back-propagation is important.
This term is a direct application of LQR equation with drift, but it is rarely mentioned in DDP works \cite{GiftthalerCoRR2017,LaineCoRR2018}.
Its role will become very important in the following.

The solution of the LQR has once more two effects: it generates a feasible trajectory where the gap between $x_{t-1}^+$ and $x_t$ is zero; and this trajectory is optimal.

\subsection{Equivalence}

Both solutions are equivalent.

Proof: the KKT solution is obtained by applying PMP on the LQR system. The Riccati solution comes from integrating HJB equation on the LQR. In the LQR case, both PMP and HJB are sufficient optimality conditions. Then both solutions are equal.

\subsection{Partial step}

In many algorithm, the QP step is only partly integrated, i.e. the initial guess is only modified by $\alpha \Delta$ where $\alpha \in [0,1[$ and $\Delta = \Delta \dxtraj,\Delta \dutraj$ the solution to the QP (for example if considering inequality constraints inside an active-set algorithm, or using the QP as the inner solver inside a SQP).
What is the effect of taking a partial step?

For the KKT formulation, the effect is pretty clear. As $\Delta \dxtraj$ was a step from a zero initial guess $\dxtrajguess = 0$, making a partial step $\alpha \Delta$ will bridge only a part of the gap. Denoting by $\dxtraj^\alpha$ the solution obtained after making a step of length $\alpha<1$, we have:
$$\dx_0^\alpha = \alpha f_0$$
$$\dx_{t+1}^\alpha - F_x \dx_t^\alpha - Fu \du_t^\alpha = \alpha f_t$$
The new solution  $\dxtraj^\alpha,\dutraj^\alpha$ is then infeasible and the gaps at each shooting points have only be reduced of $(1-\alpha)$.

For the DDP formulation, this is less clear as it is not described in the literature.
As we started to explain, the complete solution should be obtained by rolling out the quadratic policy $k,K$ from the initial state $f_0$.
But this is only when making a full step.
When making a partial step, the KKT partial solution can be obtained if (i) applying only a part of $k$ and (ii) keeping a part of the gap at each shooting node.

\subsection{Solving the DDP forward pass with a partial step}

The forward pass for a partial step $\alpha \le 1$ then becomes:
$$\dx_0 = \alpha f_0$$
$$\du_t = -\alpha k_t - K_t \dx_t$$
$$\dx_{t+1} = F_x \dx_t + F_u \du_t + \alpha f_t$$

Proof: by recurrence. We denote the $\dx^*,\du^*$ the optimal solution given by the KKT and by the DDP for a full step. We show the the proposed forward pass produced the partial KKT step.
$$\dx_0 = \alpha f_0 = \alpha \dx_0^*$$
Now, assuming the $\dx_t = \alpha \dx_t^*$, we have:
\begin{align*}
  \forall t=0..T\!\!-\!\!1,\quad\quad\quad \du_t &= -\alpha k_t - K_t \alpha \dx_t^* = \alpha \du_t^*\\
 \dx_{t+1} &= F_x \alpha  \dx_t^* + F_u \alpha \du_t + \alpha f_t = \alpha \dx_{t+1}
\end{align*}
\qed

\section{A feasibility-prone OCP solver using DDP}

So far we have detailed a method to solve a LQR program. Let's now look at the more generic case where cost and dynamics are any smooth functions.
The transcription of the OCP is a NLP with nonlinear equality constraints representing the robot dynamics.
We solve it with a SQP approach, i.e. at each iteration we compute the LQR corresponding to the tangent (differential) of the OCP at the current candidates of the decision variables; we solve the SQP and obtain a descent direction; and we search along the descent direction for a step of adequate length.

\subsection{LQR and descent direction}
The LQR is uniquely defined by the gradients of $f$ and $\ell$ and the Hessian of $\ell$ at the current guess $\xtraj,\utraj$.
The solution of the LQR is also uniquely defined and can be computed by any adequate method, in particular by inverting the KKT or by solving the backward-forward Riccati recursions (at least if not considering the numerical and complexity issues).
Both methods will give exactly the same descent directions (neglecting the rounding errors).

\subsection{Line search and integration}
Once the direction has been computed, any line search algorithm can be implemented.
Basically, the idea is to try several directions and to take the longer step which gives a reward that corresponds to what the LQR model predicts.
When considering a SQP, two contradictory objectives have to be considered: (i) the cost should decrease similarly to what the quadratic model predicts and (ii) the constraints residual should not increase.
The trade-off between these two objectives is typically decided following a merit function, chosen by the user.

It is important to better understand why the constraint residual may increase.
First, the current guess may, or may not, be feasible, i.e the state at the end of each shooting interval may, or may not correspond to the value of the state at the beginning of the next interval.
Following the names chosen in the LQR case, we name gap the discontinuity at the shooting nodes: the current guess is feasible if and only if all the $T$ gaps are zero.

If all the gaps are zero, the descent direction may make them nonzero because the descent direction is only computed from a linear model of the dynamics $F_x,F_u$.
The longer the step, the more incorrect the linear model, and the larger the gaps will grow.
The merit function then adjust the step length to all some gap growth (it is impossible with the linear model to prevent some growth) but forbid to large gaps to appear.

If some gaps are nonzero, then the corresponding $f_t$ in the LQR corresponds to these gaps.
In that case, the LQR direction will bridge the gap thanks to the linear prediction $f_t$, but it will simultaneously increase the gap because the linear prediction is inaccurate.
More precisely, the gap at time $t$ after a step $\dx,\du$ will be:
$$
f(x_t+\dx_t,u_t+\du_t) - (x_{t+1}+\dx_{t+1})= f(x_t,u_t)-x_{t+1} + F_x \dx_t + F_u \du_t  -  \dx_{t+1} + \circ(\alpha^2)
$$
where $\dx_{t+1} - F_x \dx_t - F_u \du_t = \alpha f_t $ by construction of the LQR, and the step length $\alpha$ is the magnitude of the step $\dxtraj,\dutraj$ that leads to quadratic $\circ(\alpha^2)$ errors of the linear model.
$$
f(x_t+\dx_t,u_t+\du_t) - (x_{t+1}+\dx_{t+1}) = (1-\alpha) (f(x_t,u_t) - x_{t+1}) + \circ(\alpha^2)
$$
The gap evolution is composed of two terms: the first one decreases when $\alpha$ growth, and collapses for full step $\alpha=1$; the second one growths with $\alpha$, and vanishes with $\alpha$ small.
Only the second one exists when the gap of the current guess is null.

\subsection{Nonlinear roll-out}

Once more, it is important to recall that the descent direction is the same for KKT and DDP.
However, the DDP is typically associated with a particular line search variant.
First, recall that the exact classical line search can be implemented with the DDP: for that, the roll-out should be performed on the linear model, and a merit function should be considered.

Yet, the DDP classically observes that a feasible solution is directly obtained by integrating the nonlinear dynamics around a candidate solution $\utraj$ from the initial state $x_0$.
As the nonlinear dynamics $f(x,u)$ is not exactly the same as the linear dynamics $F_x,F_u$, the feedback term $K$ must be used during the integration to avoid the divergence.
With such a roll-out, the resulting candidate decision variable is a feasible trajectory, where all the gaps are zero.

This behavior is very different from the one observed with classical line search.
We rather suggest that the same gaps than for the linear line search should be used for the nonlinear roll-out.
We then impose the gaps at the next candidate solution to be:
$$
f(x_t+\dx_t,u_t+\du_t) - (x_{t+1}+\dx_{t+1}) = (1-\alpha) (f(x_t,u_t) - x_{t+1})
$$
No need here to consider the second-order disturbance term $\circ(\alpha^2)$ as we are considering the exact nonlinear dynamics.

\subsection{Discussion}

\subsubsection{Bridging the gap}
It has been observed in multiple shooting that keeping the gaps closed during the entire search might be counterproductive as it makes the NLP search very sensitive to the instabilities of the dynamics.
We agree with this observation, which it is correlated to the fact that the DDP tends to be an algorithm with poor exploration (globalization) capabilities.
We have demonstrated in our experiment that the DDP is much more prone to face feasibility problem, and discover good solutions despite poor initial guesses, when the gaps are kept during the first part of the search.

\subsubsection{Using the true dynamics or its approximation}
Using the nonlinear dynamics has some advantages.
First, despite intuition, the nonlinear dynamics might be faster to compute, as in robotics $F_x$ and $F_u$ are large matrices with little structure (sparsity) while very efficient nonlinear routines exists to compute $f(x,u)$. On the other hand, taking a linear step provides an exact Newton step with strong convergence guarantees, at least when the NLP is convex (and often this is not the case).

We claim that the choice should be taken by considering the effect on the gaps.
With the nonlinear step, the gaps strictly decreases with nonzero step.
With the linear step, it does not strictly decreases and we have to relying on the merit function to accept reasonable growth.
While we agree that maintaining the gaps open during the search is interesting, and that it might even be interesting to enlarge them for globalization purpose, it is doubtful that the $\circ(\alpha^2)$ term might be an interesting growth direction.
Indeed, this term corresponds to some growth direction that are not predicted in the linear model.
The LQR is then not informed to choose an interesting value for that perturbation.

We have tried to experiment gap growth coming from the linear prediction error versus other perturbation terms, in particular random, and of course zero (with the nonlinear roll-out).
While it seems clear that the term $(1-\alpha)(f-x)$ is interesting, the interest or noxiousness of $\circ(\alpha^2)$ has not been observed.

In conclusion, we advice to take a nonlinear step while maintaining the gaps open.
If it is desirable to effectively relax the continuity constraints, we then advice to really relax the dynamic constraint (e.g. putting it in the cost as a penalty) and not to rely on the unpredicted disturbance $\circ(\alpha^2)$.

\section{Conclusion}

We have proposed two modifications to the classical DDP algorithm to solve OCP written as NLP problems.
First, we modified the backward pass to accept infeasible trajectories, i.e. trajectories were a discontinuity exists at each shooting interval.
Second, we modified the line search algorithm to avoid bridging the gap after the first step is taken.

In consequence, the DDP algorithm accepts any initial guess and has a much better globalization capability.
We can observe that the behavior is nearly the same as a multiple-shooting solver.
Indeed, it is exactly the same if the classical line search is implemented.
We did not demonstrate that the feasibility-prone DDP or the multiple shooting is better: the performance where equivalent on all the problems we have considered.
We discussed that the DDP might be easier to implement and make efficient, and we advice to choose it.

\section{Expectation model of the FDDP}

At each Newton step of a NLP solver, a line-search is performed.
One of the simple conditions of acceptance of the step is that the actual improvement of the cost should be similar to the expected improvement provided by the quadratic model.
For example, if considering a function $f$ and its model $m$ with $f(x+\Delta x) = f(x) + m(\delta x) + o(\Delta x)$, then a step $\delta x$ would be accepted if $f(x+\Delta x)-f(x)$ and $m(x)$ are close enough (in practice, the ratio between the two quantities is close to 1).

When considering the DDP solver, computing the expected improvement is more difficult, as the $\xtraj,\utraj$ of the LQR are never explicitly computed: the backward pass only provides $k$, from which the actual $\xtraj,\utraj$ are obtained using a nonlinear rollout.
This section provides efficient evaluation of the expectation model.

\subsection{Expectation model in $\xtraj,\utraj$}
Let's first write the expectation model in term of the increments $\xtraj,\utraj$ (let's recall that, to keep the notations concise, we use $x$ and $u$ for the LQR variables, while they should be intepreted as ``deltas'' in the nonlinear optimization algorithm).
If making a step of length $\alpha$ (typically in $]0,1]$) in the direction $\xtraj,\utraj$, then the improvement of the cost should have the following form:
$$\Delta = \Delta_1 \alpha + \frac{1}{2} \Delta_2 \alpha^2$$
$\Delta_1$ is the sum at each shooting node of the cost gradient times the change in $x$ and $u$:
\begin{equation}
  \label{eq:d1}
  \Delta_1 = \sum_{t=0}^T L_{xt}^T x_t + L_{ut}^T u_t
\end{equation}
(to keep the sum simpler, we treat $T$ similarly to the other nodes, by introducing $L_{uT} = 0$).

\subsubsection{Linear rollout}
The states and controls are obtained from a linear roll-out as:
$$ x_{t+1} = F_{xt} x_t + F_{ut} u_t + f_{t+1}$$
$$ u_{t} = K_t x_t + k_t$$
Propagating these two equations, we get:
$$ x_{t+1} = (F_{xt} + F_{ut} K_t) x_t + F_{ut} k_t + f_{t+1} =  F_{t-1} x_{t-1} + c_{t+1}$$
with $F_{t} = F_{xt} + F_{ut} K_t$ and $c_{t+1} = F_{ut} k_{t} + f_{t+1}$ (with $c_0 = f_0$).
And finally:
\begin{align}
  x_t &= F_{t-1} ... F_0 c_0 + F_{t-1} ... F_1 c_1 + ... + F_{t-1} c_{t-1} + c_t \\
  &= \sum_{i=0}^t F_{t-1} ... F_i c_i \label{eq:lroll}
\end{align}

\subsubsection{First-order model $\Delta_1$}
Replacing $u_t$ by $k_t + K_t x_t$, the first-order term is:
\begin{equation}
  \label{eq:d1}
  \Delta_1 = \sum_{t=0}^T (L_{xt} + K_t^T L_{ut}) ^T x_t + \sum_{t=0}^T L_{ut}^T k_t
\end{equation}
where we denote $l_t = L_{xt} + K_t^T L_{ut}$ to simplify the notation.
Putting \eqref{eq:lroll} in \eqref{eq:d1}, we get:
\begin{align}
  \Delta_1 &= \sum_{t=0}^{T} l_t \sum_{i=0}^{t} F_{t-1} ... F_i c_i + L_{ut}^T k_t \\
  & =  \sum_{i=0}^{T} c_i^T  \sum_{t=i}^{T} F_t^T ... F_T^T l_t + k_i^T L_{ui}
\end{align}
Each term of the sum is composed of a product of $f_i$ and a product of $k_i$, and can then be evaluated from the result of the backward pass.
Let's exhibit these 2 terms.
The term in $f_i$ is:
$$\Delta_{ft} = F_i^T ... F_T^T l_i = L_{xi} + F_{xi}^T \Delta_{fi+1} + K_i^T (L_{ui} + F_{ui} \Delta_{fi+1})$$
The term in $k_i$ is:
$$\Delta_{ft} = l_{ui} + F_{ui}^T ... F_T^T l_i = L_{ui} + F_{ui}^T \Delta_{fi+1}$$
In the case $f_i$ are all zeros, we can recognize that $\Delta_f$ is the value gradient and $\Delta_k$ is the Hamiltonian control gradient:
$\Delta_f = V_x$ and $\Delta_k = Q_u$.
In that case, we simply have:
$$\Delta_1 = \sum_{t=0}^{T} Q_{ut}^T k_t$$

In the general case where the LQR is not drift-free, then $\Delta_f$ and $\Delta_k$ must be collected during the backward pass while propagating $V_x$ and $Q_u$.
The cost is similar, and an order of magnitude less than propagating the Value Hessians.

\subsubsection{Second-order term $\Delta_2$}

This section is empty, work remains to be done here.

\subsubsection{The simple case where $T=1$}
It is disapointing that the expectation model is so simple in the drift-free case and only depends on backward-computed quantities, while it is so complex and requires to compute additional quantities in the general case.
Let's investigate that.
The intuition is that the expectation model should only depends on the gradient and hessians of the Value and Hamiltonian functions.

In the case where we only consider one control $u_0$, the expectation model is:
\begin{align*}
  \Delta_1 &= L_{x0}^T x_0 + L_{u0}^T u_0 + L_{x1}^T x1 \\
  &= L_{0}^T f_0 + L_{u0} k_0 + L_{x1} F_{0} f_0  + L_{x1} F_{u0} k_0  + L_{x1} f_1 \\
  &= (L_0 + F_0^T L_{x1})^T f_0 + (L_{u0} + F_{u0}^T L_{x1})^T k_0 + L_{x1} f_1
\end{align*}
We nearly recognize the gradients $V_{x0}, Q_{u0}, V_{x1}$ respectively in factor of $f_0,k_0,f_1$, but some terms are missing:
$$V_{x0} = L_0 + F_0^T (L_{x1} + L_{xx1} f_1) + L_{xx0} f_0$$
$$Q_{u0} = L_{u0} + F_{u0}^T (L_{x1} + L_{xx1} f_1)$$
$$V_{x1} = L_{x1} + L_{xx1} f_1$$
Basically, the missing terms correspond to the re-linearization of the gradient at the $f_t$ points at the end of the intervals.
Then, we get:
\begin{align*}
  \Delta_1 &= V_{x0}^T f_0 + Q_{u0}^T k_0 + V_{x1}^T f_1 - \left( f_0^T V_{xx0} f_0  + f_0^T F_0^T L_{xx1} f_1 + k_0^T F_{u0} L_{xx1} f_1 + f_1^T V_{xx1} f_1\right) \\
  &= V_{x0}^T f_0 + Q_{u0}^T k_0 + V_{x1}^T f_1 - \left( f_0^T V_{xx0} x_0 + f_1^T V_{xx1} x_1 \right)
\end{align*}

The second-order term is:
\begin{align*}
  \Delta_2 &= f_0^T V_{xx0} f_0 + k_0^T Q_{uu0} k_0 + f_1^T V_{xx1} f_1 + 2(f_0^T F_0^T L_{xx1} f_1 + k_0^T F_{u0} L_{xx1} f_1) \\
  &= f_0^T V_{xx0} f_0 + k_0^T Q_{uu0} k_0 + f_1^T V_{xx1} f_1 + 2\big(f_1^T V_{xx1} (x_1-f_1) \big) \\
  &= -f_0^T V_{xx0} f_0 + k_0^T Q_{uu0} k_0 - f_1^T V_{xx1} f_1 + 2\big(f_0^T V_{xx0} x_0 + f_1^T V_{xx1} x_1 \big)
\end{align*}
We can recognize in the additional terms (the 2 last ones) the same terms as in $\Delta_1$.
Nicely, they will cancel out in the case we make a full step $\alpha=1$:
$$\Delta(\alpha) = \alpha( \Delta_1+\frac{\alpha}{2} \Delta_2)$$
$$\Delta(1)= V_{x0}^T f_0 + Q_{u0}^T k_0 + V_{x1}^T f_1
- \frac{1}{2} f_0^T V_{xx0} f_0 + \frac{1}{2} k_0^T Q_{uu0}^T k_0 - \frac{1}{2} f_1^T V_{xx1} f_1 $$

But they do not cancel out in the general case:
\begin{align*}
  \Delta(\alpha) = \alpha \Big( V_{x0}^T f_0 + Q_{u0}^T k_0 + V_{x1}^T f_1
+ \frac{\alpha}{2} ( - f_0^T V_{xx0} f_0 - f_1^T V_{xx1} f_1 + k_0^T Q_{uu0}^T k_0 ) \\
+ (\alpha-1) ( f_0^T V_{xx0} x_0 + f_1^T V_{xx1} x_1 ) \Big)
\end{align*}

\subsection{Extending to $T>1$ by recurence}
We can now work by recurence to extend the exact same shape to $T>1$.

On the first order term, addind a new time step will add two terms in $k_1$ and $f_2$ where respectively $L_{x2}$ and $(L_{u1} + F_{u1}^T L_{x2})$ are in factor, and also extends the previous factors.
The new factors have the same form as the previous ones and can be handled similarly.
The extension of the previous factors simply corresponds to the extension of the preview horizon when writing $V_x$ and $Q_u$.
As previously, we are missing the $L_{xx} f$ terms (corresponding to the relinearization), that can be collected.
Each of this additional term is a product term involving two $f$ or one $f$ and one $k$.
Regrouping them by decreasing order of the $f$ index, this finally boils to the sum of the $f^T V_{xx} x$:
$$\Delta_1 = \sum_{t=0}^T V_{xt}^T f_t + Q_{ut}^T k_t - f_t^T V_{xxt} x_t $$
(with again the simplification of treating symmetrically the last time step with $k_T=0$).

Similar observations can be made on the second-order term, and lead to:
$$\Delta_2 = \sum_{t=0}^T k_t^T Q_{uut}^T k_t-f_t^T V_{xxt} f_t +2 f_t^T V_{xxt} x_t $$

The expectation model is finally:
$$\Delta(\alpha) = \alpha \sum_{t=0}^T V_{xt}^T f_t + Q_{ut}^T k_t
+ \frac{\alpha}{2} \Big( k_t^T Q_{uut}^T k_t-f_t^T V_{xxt} f_t \Big)
+ (\alpha-1) f_t^T V_{xxt} x_t$$

\subsection{Line-search algorithm}

First, let us note that if all the gaps $f_t$ are null, it is simply:
\begin{align*}
  \Delta(\alpha) &= \alpha \big( \sum Q_u^T k + \frac{\alpha}{2} k^T Q_{uu} k \big) \\
  &= \alpha(\frac{\alpha}{2} - 1) \sum  \ Q_u^T\ Q_{uu}^{-1} \ Q_u
\end{align*}
This is always negative.

\subsubsection{Merit function ... or not}
However, $\Delta$ can be positive (i.e. corresponds to an increase of the cost function) when some gap $f_t$ are nonzero.
This corresponds to the expected behavior of an SQP algorithm: a step is used to reduce the error in the constraints, which can makes the cost function increases.
The point is to monitor both the decrease or the increase of the cost function when reducing the gaps in the trajectory.
One objective is to find a line-search strategy that holds (at least is consistent) whether the gaps are nonzero or zero.
Let us first consider the case where some of the gaps are nonzero.

We introduce the following merit function:
$$\phi(\xtraj,\utraj) = \ell(\xtraj,\utraj) + \mu \sum_{t=0}^T \| c_t(\xtraj,\utraj) \|_1$$
where $\ell$ is the total cost function (integral plus terminal) and the constraints $c_t$ are:
$$ c_{t+1} = x_{t+1} - f(x_t,u_t) = f_{f+1} $$
$$ c_0 = x_0 - x_0^* = f_0$$
where the $f_t$ have already been introduced as the trajectory gaps (defects).
We consider how $\phi$ changes when changing $\xtraj,\utraj$ in the direction $\dxtraj,\dutraj$:
$$\xtraj'=\xtraj + \alpha \dxtraj$$
$$\utraj'= \utraj + \alpha \dutraj$$
We abusively denote by $\phi$ the merit changes along the line search:
$$\phi(\alpha) := \phi(\xtraj + \alpha \dxtraj,\utraj + \alpha \dutraj) - \phi(\xtraj,\utraj)$$
We have:
$$\phi(\alpha) = \ell'-\ell - \alpha \mu \sum_{t=0}^T  \| f_t \|_1$$
As the $f_t$ does not depend on $\alpha$ (thanks to the nonlinear rollout used in the forward pass, as explained in the first part of this document), we can always find a penalization $\mu$ that makes this function decreases.
This means that, if $\mu$ is large enough, any step from an infeasible guess would be accepted.
The drawback is that the step might induces very large increase of the cost function.
In particular, the cost increase might be much larger than predicted by the LQR model, in particular when the initial guess is far from being feasible (for unstable dynamics system, when the initial control guess is very far from stabilizing the initial state trajectory).

As we now that $\sum \| f_t \|_1$ will decrease with nonzero $\alpha$, we rather suggest to only consider the first term $\ell'-\ell$.
This term exactly corresponds to the expectation model that we described above.
$$\ell'-\ell = \Delta(\alpha) + \mathcal O(\alpha^3)$$

\subsubsection{Goldstein condition}
We cannot use it a second order version of the Wolfe (Armijo) conditions, first because $\Delta$ might be positive (not a descent direction), and second because strong Wolfe conditions uses the gradient at the next candidate point, which are very expensive to compute in our case.
We rather suggest to take a second-order version of the Goldstein conditions, i.e. accept a step if the actual cost change is similar to the expected cost change:
$$ b_1 \le \frac{\ell'-\ell}{\Delta(\alpha)} \le b_2$$
with $b_1,b_2$ are two adjustable parameters.
More precisely, if $\Delta$ is negative (the direction is descending), this imposes that:
$$\ell'-\ell \le b_1 \Delta(\alpha)$$
i.e. that the cost decrease at least a fraction of the expectation model.
A contrario, if $\Delta$ is positive (the direction is ascending), this imposes that:
$$\ell'-\ell \le b_2 \Delta(\alpha)$$
i.e. the cost does not increase more than a multiple of the expectation model.
In practice, we suggest to use $b_1=0.1$ and $b_2=2$.
This might be better replaced by a switch to avoid the quotient.
The condition finally is to accept the step if:
$$\ell'-\ell \le
\begin{cases}
  b_1 \Delta(\alpha) & \textrm{if }\Delta(\alpha)\le 0 \\
  b_2 \Delta(\alpha) & \textrm{otherwise}
\end{cases}
$$

\subsubsection{Approximating $\Delta$ with a nonlinear rollout}

The expectation model exhibited above implies the explicit values of the changes in the state trajectory in the term $f_t^T V_{xx} \dx_t$.
However, the DDP algorithm never explicitly computes the $\dx_t$, but rather directly computes the next $x_t'$ in the rollout, using the nonlinear dynamics.
We do not have the linear direction $\dx_t$, however, we can easily computes the change in the state trajectory by $x_t'(\alpha)-x_t$ where $x_t'(\alpha)$ is the state reached at time $t$ when applying the changes in the control trajectory and at the trajectory gaps.
We then set:
$$\dx_t = x_t'-x_t$$
And we modify the expectation model accordingly:
$$\Delta(\alpha) = \alpha \sum_{t=0}^T V_{xt}^T f_t + Q_{ut}^T k_t
+ \frac{\alpha}{2} \Big( k_t^T Q_{uut}^T k_t-f_t^T V_{xxt} f_t \Big)
+ (\alpha-1) f_t^T V_{xxt} (x_t'(\alpha) - x_t)$$
Again, this boils down to the sum of the $Q.k$ when the gaps are all zero.

\bibliographystyle{plainnat}
{
\small
\bibliography{references}
}

\end{document}