\section{Logic Gates}

The basic building blocks of computers are called \emph{gates}. Their only function is to report, with some yes-or-no electrical output, the result of their circuit following some input from somewhere else in the circuit. The way to understand how gates make decisions is called ``Boolean Algebra", but really it's a fairly simple set of decisions that get strung together in useful ways. The circuits are called gates because they either let electricity pass or prevent electricity from passing through, like a gate allows physical movement or prevents it.

\subsection*{Boolean Logic}

Boolean logic is the sort of decision making and answers that computers use. You can use it too, of course, and it will help you learn to program computers. The answer to every question in Boolean logic is either ``yes'' or ``no'' -- ``true'' or ``false''. In computers, that's a one or a zero, of course.   Every operation (decision) is composed of a set of operations: \emph{and}, \emph{or}, and \emph{not}. A British man named George Boole introduced these ideas to the world in 1847\footnote{
\emph{The Mathematical Analysis of Logic} (1847), and \emph{An Investigation of the Laws of Thought} (1854). For more, see his {\color{webblue}\href{https://en.wikipedia.org/wiki/George_Boole}{Wikipedia entry}}.}. He did not realize he was laying the foundations for computing, but 90 years later another very important mathematician, Claude Shannon, understood that Boole's system for understanding decisions could be used to design electronic circuits\footnote{Shannon, Claude E. 1938. ``A Symbolic Analysis of Relay and Switching Circuits". \emph{Trans. AIEE.} 57 (12): 713--723.}.

It may be helpful to see some of the logic and operations (decisions) made in a graphical form, looking at the overlap between circles, or between a circle and its surrounding area. Here, the shaded area shows the part of the relationship that is ``true'' for the given operation. That is, the region that is {\sc{not}} A, in the third diagram, is shaded.

% Draw Venn diagrams to graphically demonstrate OR, AND, and NOT.
\begin{figure}[!ht]
\begin{center}
\begin{minipage}{6in}
\input{./include/vennand.tex}
\caption{A graphic representing {\sc{and}}, {\sc{or}}, and {\sc{not}} with overlapping circles. The shaded area represents the location where the Boolean operation is ``true''.}
\label{fig:vennlogic}
\end{minipage}
\end{center}
\end{figure}

Also note: more complex decisions can be made by combining these operations: NOT $+$ AND, for instance).

\subsection*{The OR Gate}

OR gates take two inputs and provide a ``yes'' if line 1 OR line 2 is equal to 1 -- that is, if either line has a voltage level above zero.

\medskip
\begin{center}

\begin{tabular}{p{2.3in} p{3.7in} }
\hline\\[\negsep]
The symbol for an OR gate is:

\vspace{0.25in}

\begin{circuitikz}
	\draw(0,0)
	node[american or port](orgate){}
	(orgate.in 1) node [left](){{\color{red}$INPUT~A$}}
	(orgate.in 2) node [left](){{\color{red}$INPUT~B$}}
	(orgate.out) node [right](){{\color{red}$OUT$}}
;
\end{circuitikz}

&

\centering

The ``Truth Table" (how they behave) is:
\vspace{0.15in}

\begin{tabular}{ll | c}
\multicolumn{3}{c}{\textbf{OR Gate }}\\
\multicolumn{3}{c}{\textbf{Truth Table}}\\
\hline\\[\negsep]
\textbf{A} & \textbf{B} & \textbf{OUT}\\
\hline
0 & 0 & 0  \\
1 & 0 & 1  \\
0 & 1 & 1  \\
1 & 1 & 1  \\
\hline
\end{tabular}
\\
\tabularnewline
\hline\\[\negsep]
\end{tabular}

\end{center}
\bigskip

\noindent OR gates are very simple. They are so simple that transistors are not even needed, though they certainly can be used.

\begin{figure}[!ht]
\begin{center}
\input{./include/diodeorgate.tex}
\caption{A very simple OR gate schematic. Diodes conduct electricity only in the ``forward'' direction. So Input A cannot affect Input B. But either way, if there is a signal on Input A or on Input B, OUT will carry a voltage (and therefore, a ``1''). Diode \emph{propagation delay} (how much time the signal takes to cross through the diode) is slower than transistors, meaning transistorized circuits run faster.}
\label{fig:diodeorgate}
\end{center}
\end{figure}

\stbox{6.0in}{\emph{Experiment:} Wire up an OR gate on a breadboard, like the one seen in the cover image. Use LEDs as indicators for Input A, Input B, and OUT. Feel free to use diodes, but using transistors is pretty easy. It's best to use N-type transistors as switches for each LED and for the OR-output LED.}

\begin{figure}[!ht]
\begin{center}
\input{./include/npnorgate.tex}
\caption{A simple transistor-based OR gate. Transistors are better than diodes in this application because transistors amplify (or at least do not degrade) signals, while diodes lose half a volt or more.  In this schematic, each transistor acts as an independent switch for the output -- either switch pushes OUT to high.}
\label{fig:npnorgate}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=1.25]{orgateschembetter.png}
\caption{A schematic from a real circuit design program showing a practical, real implementation of the idealized schematic in Figure \ref{fig:npnorgate}. The wires connecting components are the green lines. Mostly there are more resistors to help keep the transistors behaving well. For instance, the 100K resistors make sure the transistors are always at a low logic level (zero volts) when the inputs are `off'.}
\label{fig:orgateeagleschematic}
\end{center}
\end{figure}

\begin{figure}[!hb]
\begin{center}
\includegraphics[scale=0.25]{trimmedorgate.png}
\caption{A circuit board designed using the above OR gate schematic. You can solder these without much trouble. You can see the top and bottom of the board here, but all components go on the top. However, both top and bottom have \emph{traces}, that carry electricity instead of using loose/floppy wires. The larger resistors are the lowest value (they're just there to make the transistors behave a little better): 100R means ``100 Ohms''. The smaller resistors are either 100,000 Ohms (that is, 100K Ohms) or 1,000 Ohms (1K Ohms). The OR gate takes two \emph{inputs}, A and B, and produces one \emph{result}, R.}
\label{fig:orgatepcbs}
\end{center}
\end{figure}

\clearpage
\newpage

\subsection*{The AND Gate}

AND gates take two inputs and provide a ``yes'' (a one, or a ``logic high" signal) only if both line 1 AND line 2 are equal to 1.

\medskip
\begin{center}

\begin{tabular}{p{2.3in} p{3.7in} }
\hline\\[\negsep]

The symbol for an AND gate is:

\vspace{0.25in}

\begin{circuitikz}
	\draw(0,0)
	node[american and port](andgate){}
	(andgate.in 1) node [left](){{\color{red}$INPUT~A$}}
	(andgate.in 2) node [left](){{\color{red}$INPUT~B$}}
	(andgate.out) node [right](){{\color{red}$OUT$}};

\end{circuitikz}

&

\centering

The ``Truth Table" (how they behave) is:
\vspace{0.15in}

\begin{tabular}{ll | c}
\multicolumn{3}{c}{\textbf{AND Gate }}\\
\multicolumn{3}{c}{\textbf{Truth Table}}\\
\hline\\[\negsep]
\textbf{A} & \textbf{B} & \textbf{OUT}\\
\hline
0 & 0 & 0  \\
1 & 0 & 0  \\
0 & 1 & 0  \\
1 & 1 & 1  \\
\hline
\end{tabular}
\\
\tabularnewline

\hline\\[\negsep]

\end{tabular}
\end{center}

\bigskip

There are much more complicated AND gates, but see below for an easy-to-understand example (and it totally works). Remember that transistors act like switches: the \emph{gate} acts like the switch that opens up the flow of electricity from the \emph{source} to the \emph{drain}. Look at the schematic in Figure \ref{fig:simpleandgate} -- note that the drain of the top transistor ($Q_1$) flows into the source of the other transistor ($Q_2$), meaning that even if $Q_2$ is turned on, it may not necessarily start carrying electricity (because $Q_1$ may not allow the electricity from $V_{cc}$ to pass through). Thus, both gate 1 AND gate 2 have to be turned on in order to see voltage at the output.

\bigskip

\begin{figure}[!ht]
\begin{center}
\input{./include/simpleand.tex}

\caption{A simple AND gate schematic. When Input A is a zero, no current flows into the ``collector" (voltage input wire) of Transistor 2. When Input B is zero, no current can flow to OUT, regardless of whether Input A is 0 or 1. Only when Input A is 1 (current can flow past Q1) and when Input B is 1 (current is available to Q2, and Q2 is turned ``on'', allowing current to pass) does OUT go ``high". $R_3$ prevents too much current from flowing to ground, and is not strictly necessary in all gates, but is included here because it is common to see them. $R_1$ and $R_2$ should be about 100 ohms. $R_3$ should be about 100K ohms. This design is just for demonstration.}
\label{fig:simpleandgate}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
\fbox{
\includegraphics[scale=0.14]{andgatebreadboard.jpg}
}
\medskip

A single AND gate on a solderless breadboard. The transistor on the left, controlled by the gray button, makes electricity available to the input of the second transistor, controlled by the green button. Only if both transistors are ``on'' (that is, A \emph{and} B are passing electricity) will the LED light up. Using this picture, wire up an AND gate on a breadboard with N-type transistors. Use LEDs as indicators for Input A, Input B, and the result.
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.25]{andgate.png}
\caption{A circuit board designed using the above AND gate schematic, though with two 100K pull-down resistors also implemented (these guarantee the transistor's gate are reliably held at 0 volts instead of allowed to \emph{float}, until either of the inputs goes high (that is, pushes some voltage into the transistor gate). The pull-down resistors are important practically, but they do not appear in this circuit diagram for simplicity. This board looks almost exactly like the OR gate, but the wiring is different! Also, you can see both the front and back sides of the circuit board in this image. The AND gate takes two \emph{inputs}, A and B, and produces one \emph{result}, R.}
\end{center}
\label{fig:andgate}
\end{figure}

\clearpage
\newpage

\subsection*{The NOT Gate}

NOT gates, sometimes called \emph{inverters}, take one input and reverse the value of that input. So a 1 becomes a 0, or vice versa. Put another way, this gate creates the \emph{complement} of the input value.

\medskip
\begin{center}

\begin{tabular}{p{2.3in} p{3.7in} }
\hline\\[\negsep]

The symbol for a NOT gate is:
\vspace{0.25in}

\begin{circuitikz}
	\draw(0,-0.5)
	node[american not port](notgate){}
	(notgate.in) node [left](){{\color{red}$INPUT$}}
	(notgate.out) node [right](){{\color{red}$OUT$}}
;
\end{circuitikz}

&

\centering

The ``Truth Table" (how they behave) is:
\vspace{0.15in}

\begin{tabular}{l | c}
\multicolumn{2}{c}{\textbf{NOT Gate }}\\
\multicolumn{2}{c}{\textbf{Truth Table}}\\
\hline\\[\negsep]
\textbf{IN} & \textbf{OUT}\\
\hline
0 & 1 \\
1 & 0  \\
\hline
\end{tabular}
\\
\tabularnewline

\hline\\[\negsep]

\end{tabular}
\end{center}

\bigskip

A simple NOT circuit, using a very old resistor-transistor logic (RTL) design. It is easy to understand, though this example is not how NOT gates are actually implemented any more, because of the poor efficiency---wasted energy turns into heat, which then has to be cooled somehow. It is better to not make the heat in the first place.

\begin{figure}[!ht]
\begin{center}
\input{./include/simplenot.tex}
\caption{A very simple NOT gate schematic. When IN is logic low (off), electricity cannot pass through to the ground, and so it is available at the OUT terminal---OUT is a 1. When IN is carrying current, the transistor opens up and electricity starts flowing to ground -- the ``path of least resistance''. As electricity flows to ground instead of OUT, OUT has a very low voltage available -- making it a zero.}
\label{fig:simplenot}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
\input{./include/CMOSNOT.tex}

\caption{A practical inverter (NOT) circuit. While more complicated than a simple NOT circuit, it is more efficient, wasting much less energy. It uses one P-type and one N-type \emph{field effect transistor} (FET). P-type transistors clamp off electricity when their gate \emph{has} voltage on it. N-type transistors clamp off when their gate \emph{does not have} voltage on it. So, here, when the input is high, the top transistor clamps off voltage input, and the bottom transistor opens up, allowing any current to flow to ground--but there's no current available! The situation reverses when the input is low: the top transistor opens up, allowing current to flow, and the bottom transistor clamps off, preventing current from going anywhere -- but putting a high signal at the output.}
\label{fig:cmosnot}
\end{center}
\end{figure}

\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.15]{trimmednotgate.png} % convert NOTgate.png -alpha off -trim  trimmednotgate.png
\caption{A circuit board designed using the more elegant NOT gate schematic seen in Figure \ref{fig:cmosnot}. The NOT gate takes exactly one \emph{input}, A, and produces a \emph{result}, R.}
\label{fig:notgateboard}
\end{center}
\end{figure}

\begin{table}
\input{./include/NOTgateBOM.tex}
\caption{The list of components needed to make the NOT gate seen in Figures \ref{fig:cmosnot} and \ref{fig:notgateboard}.}

\end{table}

\clearpage
\newpage

\section{Complex Logic Gates}

Combining gates is possible to produce more complex circuits that answer different questions, like outputs that tell us if two inputs are ``NOT AND" or ``NOT OR" or ``Exclusively OR". Each of these gates allows new comparisons between two inputs, but each is composed of the fundamental three gates, seen above. There is even an "Exclusive NOT OR" gate, which is how computers determine if two numbers are equal to each other.

\subsection*{The NAND Gate}

A ``NOT-AND", or ``NAND", gate just reverses the value obtained from an AND gate -- so it has the exact opposite truth table. Since a NAND gate just reverses the output of AND, one way to create a NAND gate is to put a NOT gate on the result (output) line of an AND gate and make a NAND gate, just like you were using Legos. The AND gate goes through the NOT gate, and becomes NOT-AND, or ``NAND".

\medskip
\begin{center}

\begin{tabular}{p{2.3in} p{3.7in} }
\hline\\[\negsep]

The NAND gate symbol is:

\vspace{0.25in}

\begin{circuitikz}
	\draw(0,0)
	node[american nand port](nandgate){}
	(andgate.in 1) node [left](){{\color{red}$INPUT~A$}}
	(andgate.in 2) node [left](){{\color{red}$INPUT~B$}}
	(andgate.out) node [right](){{\color{red}$OUT$}};

\end{circuitikz}

\vspace{0.15in}

See the circle on the output? That means ``\emph{not}".

&

\centering

The ``Truth Table" (how they behave) is:
\vspace{0.15in}

\begin{tabular}{ll | c}
\multicolumn{3}{c}{\textbf{NAND Gate }}\\
\multicolumn{3}{c}{\textbf{Truth Table}}\\
\hline\\[\negsep]
\textbf{A} & \textbf{B} & \textbf{OUT}\\
\hline
0 & 0 & 1  \\
1 & 0 & 1  \\
0 & 1 & 1  \\
1 & 1 & 0  \\
\hline
\end{tabular}
\\
\tabularnewline

\hline\\[\negsep]

\end{tabular}
\end{center}

You can think of a NAND gate as two separate gates (AND and NOT):
\begin{center}

\begin{circuitikz}
	\draw(0,0)
	node[american and port](andgate){}
	(andgate.in 1) node [left](){{\color{red}$INPUT~A$}}
	(andgate.in 2) node [left](){{\color{red}$INPUT~B$}}
	;

	\draw(1.5,0)
	node[american not port](notgate){}
	(andgate.out) |- (notgate.in)
	;

	\draw(3,0)
	(notgate.out) node [right](){{\color{red}$OUT$}}
	;

\end{circuitikz}

\end{center}

\bigskip

\begin{figure}[!ht]
\begin{center}
\includegraphics[scale=0.16]{trimmednandgate.png}
\caption{A circuit board designed to make a NAND gate using four transistors.}
\label{fig:nandgate}
\end{center}
\end{figure}

\clearpage
\newpage

\subsection*{The NOR Gate}

Not surprisingly, there is also a NOR gate, that has the reverse truth table to an OR gate. NOR gates are easy to implement, though as we have said elsewhere, the more complicated NOR gates save power and run cooler.

\medskip
\begin{center}

\begin{tabular}{p{2.3in} p{3.7in} }
\hline\\[\negsep]

The symbol for a NOR gate is:

\vspace{0.25in}

\begin{circuitikz}
	\draw(0,0)
	node[american nor port](norgate){}
	(andgate.in 1) node [left](){{\color{red}$INPUT~A$}}
	(andgate.in 2) node [left](){{\color{red}$INPUT~B$}}
	(andgate.out) node [right](){{\color{red}$OUT$}};

\end{circuitikz}

\vspace{0.10in}

See the circle on the output? That means ``\emph{not}".
\vspace{0.15in}

&

\centering

The ``Truth Table" (how they behave) is:
\vspace{0.15in}

\begin{tabular}{ll | c}
\multicolumn{3}{c}{\textbf{NOR Gate }}\\
\multicolumn{3}{c}{\textbf{Truth Table}}\\
\hline\\[\negsep]
\textbf{A} & \textbf{B} & \textbf{OUT}\\
\hline
0 & 0 & 1  \\
1 & 0 & 0  \\
0 & 1 & 0  \\
1 & 1 & 0  \\
\hline
\end{tabular}
\\

\tabularnewline

\hline\\[\negsep]

\end{tabular}
\end{center}

You can think of a NOR gate as two separate gates (OR and NOT):
\begin{center}
\begin{circuitikz}
	\draw(0,0)
	node[american or port](orgate){}
	(orgate.in 1) node [left](){{\color{red}$INPUT~A$}}
	(orgate.in 2) node [left](){{\color{red}$INPUT~B$}}
	;

	\draw(1.5,0)
	node[american not port](notgate){}
	(orgate.out) |- (notgate.in)
	;

	\draw(3,0)
	(notgate.out) node [right](){{\color{red}$OUT$}}
	;

\end{circuitikz}
\end{center}

\bigskip

\begin{figure}[!hb]
\begin{center}
\includegraphics[scale=0.4]{Agc_nor2.jpg}
\caption{An early integrated circuit NOR gate. It has six transistors, eight resistors, and three pairs of A/B inputs (thus, it has three separate yes/no outputs). This chip was part of the guidance computer that took Americans to the moon in 1969. \emph{Photo credit: Wikimedia Foundation}.}
\end{center}
\label{fig:apollonorgate}
\end{figure}

\clearpage
\newpage

\subsection*{The Exclusive-OR Gate}
There is also the ``eXclusive-OR", or ``XOR", gate. Here, ``exclusive'' means ``one, but not both.'' The XOR gate output (result) is 1 when \emph{the inputs are not the same}. It returns a 1 if, and \emph{only} if, both inputs are different. That is, it provides a one ``exclusively if'' there is a single ``one'' (sometimes called ``logic high", for ``voltage above zero") in the two inputs. XOR gates take a lot of transistors to implement. The most common version requires 12, though it is possible to make an XOR gate with 8 transistors. See Figures \ref{fig:xorcomposite}, \ref{fig:xorbreadboard}, and \ref{fig:xorgate}.

\medskip
\begin{center}

\begin{tabular}{p{2.3in} p{3.7in} }
\hline\\[\negsep]

The symbol for an XOR gate is:

\vspace{0.25in}

\begin{circuitikz}
	\draw(0,0)
	node[american xor port](xorgate){}
	(andgate.in 1) node [left](){{\color{red}$INPUT~A$}}
	(andgate.in 2) node [left](){{\color{red}$INPUT~B$}}
	(andgate.out) node [right](){{\color{red}$OUT$}};

\end{circuitikz}

\vspace{0.15in}

(the curved line behind the OR gate means that it is ``exclusive")

&

\centering

The ``Truth Table" (how they behave) is:
\vspace{0.15in}

\begin{tabular}{ll | c}
\multicolumn{3}{c}{\textbf{XOR Gate }}\\
\multicolumn{3}{c}{\textbf{Truth Table}}\\
\hline\\[\negsep]
\textbf{A} & \textbf{B} & \textbf{OUT}\\
\hline
0 & 0 & 0  \\
1 & 0 & 1  \\
0 & 1 & 1  \\
1 & 1 & 0  \\
\hline
\end{tabular}
\\
\tabularnewline

\hline\\[\negsep]

\end{tabular}
\end{center}

\bigskip

% XOR gate, composited:
\begin{figure}[h!]
\begin{center}
\input{include/xorcomposited.tex}
\caption{XOR gate, as a composite of the simple gates.}
\label{fig:xorcomposite}
\end{center}
\end{figure}

XOR gates return ``true'' if---and \emph{only} if---the inputs are different. Put another way, the inputs ``must sum to 1'', or the inputs ``must not be equal''. OR gates return ``true'' if the inputs are different, but \emph{also} if both inputs are 1. So, it isn't the same. There are two cases where XOR can be ``true''--where A is 1 and B is 0, or where A is 0 and B is 1. The logic of the XOR circuit must handle either case, and list the result of 1 (``true'') for only those two cases.

Take a look at Figure \ref{fig:xorcomposite}: You can see that there are two AND gates. Recall that AND gates only return ``true'' if both inputs are 1. For XOR, we require that both inputs are different. So to correctly identify ``A is 1 and B is 0'', we use an inverter (a NOT gate) on the B input -- so if the B is a zero, the NOT gate converts the B to a 1, and then the AND gate gets both inputs as 1. The lower AND gate ($AND_2$) captures the opposite case: if A is 0 and B is 1, then the A input gets inverted by the NOT gate into a 1, and for that case, also, the AND gate correctly reports that the inputs are ``true''. Since the OR gate will return ``true'' if either input is 1, then both opposite cases where XOR is true can be passed along to the result.

It's also worth knowing that XOR gates can be used like a NOT gate, by holding one of the two inputs at 1. Then the other input will always be negated. If one input of an XOR gate is always held at zero, then the XOR gate always passes the same value that comes in on the other input (a gate that passes on its input without changing it is called a ``buffer").

\bigskip

\begin{figure}[!hb]
\begin{center}
\fbox{
\includegraphics[scale=0.15, clip=true, trim=0 0 100 0]{xorgatebreadboard.jpg}
}
\caption{An XOR gate wired up on a breadboard with discrete transistors and resistors. This is the same schematic (and the same color-coded signal wires) seen in Figure \ref{fig:xorgate}. The red switch is signal A and the blue switch is signal B. On the LED line, 0 and 1 are A and B, and 2 and 3 (lit up) are NOT A and NOT B.}
\label{fig:xorbreadboard}
\end{center}
\end{figure}

\begin{figure}[ht!]
\begin{center}
\input{include/xorgate.tex}
\caption{An exclusive-or (XOR) gate, composed of 12 transistors and some supporting resistors. You don't need to understand how it works, but it's not that hard if you break it down into the colored signals, two NOT gates, two AND gates with N-type FETs, and two linked AND gates with P-type FETs. A ``0'' (ground) signal opens a P-type FET, and a ``1'' signal opens an N-type FET.}
\label{fig:xorgate}
\end{center}
\end{figure}

\clearpage


\chapter{Interpreting a Linear Regression Model \label{chapter:linreg}}

This chapter is devoted to understanding the structure of linear regression models. We first encountered them in Chapter~\ref{chapter:regression} as ``just one example'' of a regression model. However, linear regression's overwhelming popularity in the clinical domain means that one cannot do clinical data science without fully understanding these models' structure and how to interpret the output produced by model fitting software.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Biomarker Example from Chapter~\ref{chapter:regression}}

In Chapter~\ref{chapter:regression}, we saw an example where information about two predictors -- a disease severity score ($x_1$) and a social determinants score ($x_2$) -- was used to predict the level of a disease recurrence biomarker. One of the three supervised learning algorithms we tried was a \textbf{linear regression} model (Section~\ref{ssect:linreg}). The output from that model is repeated below.
\begin{center}
\includegraphics[width=0.35\textwidth]{img/esl-reg-linear.png}
\includegraphics[width=0.64\textwidth]{img/linear-regression-model-output.png}
\end{center}

\begin{question}{}
What are the number of samples, $n$, and the number of predictors, $p$, for this dataset?
\end{question}

\noindent But what do all these numbers \emph{mean}?

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Understanding the Model Summary}

A linear regression model looks like this (see also Chapter~\ref{chapter:regression}, Section~\ref{ssect:linreg}):
$$ y = \beta_0 + \beta_1 x_1 + \dots + \beta_p x_p + \varepsilon $$
where we assume that the error, $\varepsilon$, is normally distributed, $\mathcal{N}(0, \sigma)$. Another way of saying this is that we are assuming the outcome, $y$, is normally distributed with mean $\beta_0 + \beta_1 x_1 + \dots + \beta_p x_p$ and standard deviation $\sigma$.

\subsection{The Call}

The first line of the output is just repeating the call you made to the \verb|lm| function in R to fit the model. The \texttt{lm} package fits linear regression models using ordinary least squares. However, linear regression models are also a type of generalized linear model (see Chapter~\ref{chapter:glms}) and can be fit using maximum likelihood. In R, if you use the \texttt{glm} package with \texttt{family = "gaussian"} you should get identical coefficients and error estimates to what you get using the \texttt{lm} package.

\subsection{The Residuals}

A \textbf{residual} is a measure of how much the true outcome value ($y$) of one datapoint differs from the model's prediction. For a linear regression model, the residual of training point $i$ is:
$$ \text{residual}^{(i)} = y^{(i)} - \hat{y}^{(i)} $$
where $\hat{y}^{(i)}$ is the model's prediction:
$$ \hat{y}^{(i)} = \beta_0 + \beta_1 x_1^{(i)} + \dots + \beta_p x_p^{(i)}. $$
Here is a histogram of the residuals for this model:
\begin{center}
\includegraphics[width=0.7\textwidth]{img/linreg-example-residuals.png}
\end{center}

\begin{question}{}
Estimate the maximum, minimum, and mean residuals from this graph. Do they match what is in the model output?
\end{question}

The residuals play an important role in linear regression models because they are what allow us to estimate $\sigma$. They also play an important role in model diagnostics because they enable us to check one of the core model assumptions: the assumption that $\sigma$ is constant. We can check this assumption by making a plot of the residuals vs. $\hat{y}$:
\begin{center}
\includegraphics[width=0.7\textwidth]{img/linreg-example-residual-vs-fitted.png}
\end{center}
We can also check the normality of the residuals by making a plot of their values vs. what we would expect if the residuals were perfectly normally distributed with the same mean and standard deviation:
\begin{center}
\includegraphics[width=0.7\textwidth]{img/linreg-example-almost-qq-plot.png}
\end{center}
This is not quite a \textbf{QQ-plot}, one of the standard diagnostic plots of linear regression, because it's plotting the actual values of the residuals instead of their quantiles. But aside from the axis scales, it's exactly the same. We'll see formal QQ-plots later.

The fact that the points in the second plot lie close to a line is a good indication that the residuals are normally distributed. Another thing to notice from this plot is that the colors of the points (their $y$ values) are unrelated to their residuals.
\vspace{5mm}

\begin{question}{}
The four plots below show a famous dataset called \textbf{Anscombe's quartet}. The regression lines produced by fitting a linear regression model to each dataset are identical, but only one dataset actually fulfills the assumptions of a linear regression model.
\begin{center}
\includegraphics[width=0.7\textwidth]{img/anscombe.png}
\end{center}
We can check these assumptions by examining plots of the residuals vs. fitted values of the model (here the ``fitted value'' of point $i$ means $\hat{y}^{(i)}$).
\begin{center}
\includegraphics[width=0.7\textwidth]{img/anscombe-diagnostics-1.png}
\end{center}
Which of the four datasets fulfills the assumption of a linear regression model that the error has constant variance? How can you tell?
\end{question}

\subsection{Coefficients and Standard Errors}

Although residuals are important, the parts of the model output that will be scrutinized, reported in papers, etc. are the coefficients, along with their standard errors and hypothesis tests. The coefficients are what create the regression surface, which captures the model's prediction for every point in the feature space (see Section~\ref{ssect:linreg}).

There are actually a few different ways to derive the coefficients in a linear regression model. The most common way uses \textbf{least squares}, which adjusts the coefficients $\beta_0, \beta_1, \dots, \beta_p$ until the sum of the squared residuals is minimized:
\begin{align*} \text{SSE} &= \sum_{i=1}^n (y^{(i)} - \hat{y}^{(i)})^2 \\
&= \sum_{i=1}^n (y^{(i)} - \beta_0 - \beta_1 x_1^{(i)} - \beta_2 x_2^{(i)} - \dots - \beta_p x_p^{(i)})^2 \end{align*}
It turns out that you can find the optimal values of the $\beta$s analytically by taking derivatives of this thing and setting them equal to zero. Alas, this requires some matrix multiplication and the taking of matrix inverses, so we will save it for a later chapter. Suffice it to say that the $\beta$s are adjusted to minimize the SSE, and the values in the model output are the optimal values.
\vspace{5mm}

\begin{question}{}
Looking at the form of the linear regression model
$$ y = \beta_0 + \beta_1 x_1 + \dots + \beta_p x_p + \varepsilon $$
what does the value of each of the $\beta$s mean? What is $\beta_j$ telling us about how $y$ varies with the predictor $j$, all else being equal?
\end{question}

The model's overall estimate of $\sigma$, the standard deviation of the error term, is obtained very naturally by averaging over the residuals, taking into account the number of predictors, $p$:
$$ \hat{\sigma}^2 = \frac{1}{n-p-1} \sum_{i=1}^n (y^{(i)} - \hat{y}^{(i)})^2.$$

\begin{question}{}
The sum of the squared residuals for our model is $4480.678$. There are $n=200$ datapoints, and the number of predictors, $p$, is 2. Calculate $\hat{\sigma}$ for this model. Do you see this number anywhere in the model output? What is it called?
\end{question}

The standard errors of the model coefficients likewise require matrix multiplication to fully understand, but they are related to three factors: (1) the value of $\hat{\sigma}$, (2) the spread of the values of the corresponding covariate about its mean (more spread will decrease the standard error), and (3) correlations between that covariate and other covariates in the model (tighter correlations will increase the standard error).
\vspace{5mm}

\begin{question}{}
The standard errors attempt to capture how much we expect our estimates of the model coefficients to vary if we were to refit the model using a different dataset, provided that the new dataset is similar to (i.e., sampled from the same population distribution as) the one used to fit the model. On average, approximately how much would we expect $\beta_0$ (the intercept) to deviate from its fitted value of 49.8600? How much would we expect $\beta_1$ and $\beta_2$ to deviate from their fitted values?
\end{question}

\subsection{Hypothesis Tests of Coefficients}

Armed with our coefficients and standard errors, we can perform a hypothesis test on each regression coefficient. Our null hypothesis in each case will be that the true value of that coefficient is zero: that is, it has no effect on the outcome. Under the null hypothesis that $\beta_j = 0$ and assuming $n$, our number of samples, is large enough, the quantity $\hat{\beta_j}/\text{se}(\hat{\beta}_j)$ will be distributed according to a Student's T distribution (see Chapter~\ref{chapter:probabilitydistributions}, Section~\ref{sect:tdist}) with $n-p-1$ degrees of freedom.
\vspace{5mm}

\begin{question}{}
Sketch the null distributions for the hypothesis tests of our three regression coefficients, $\beta_0$, $\beta_1$, and $\beta_2$. Do you see why the $p$-values for these tests are so low?
\end{question}

\subsection{Other Model Output}

The software also provides some other output. The quantity \textbf{R-squared} is defined as:
\begin{align*} R^2 &= 1 - \frac{\sum_{i=1}^n (y^{(i)} - \hat{y}^{(i)})^2}{\sum_{i=1}^n (y^{(i)} - \overline{y})^2} \\[3mm]
&=  1 - \frac{\sum_{i=1}^n (y^{(i)} - \hat{\beta}^T x^{(i)})^2}{\sum_{i=1}^n (y^{(i)} - \overline{y})^2} \end{align*}
or rather, the proportion of total variance in $y$ explained by the model. The \textbf{adjusted R-squared} is almost exactly the same except it fixes a source of bias in $R^2$, namely that $R^2$ will favor models with more parameters. Adjusted $R^2$ penalizes models with more parameters. It is defined as:
\begin{align*}
 R^2_{\text{adj}} &= 1 - \frac{n-1}{n-p-1} \frac{\sum_{i=1}^n (y^{(i)} - \hat{y}^{(i)})^2}{\sum_{i=1}^n (y^{(i)} - \overline{y})^2} \\[3mm]
 &= 1 - (1 - R^2) \frac{n-1}{n-p-1}
\end{align*}
The \textbf{F-statistic} is the ratio of two variances: the variance in the outcome, $y$, that is explained by the model parameters (``sum of squares of regression'', or SSR) and the residual, or unexplained variance (``sum of squares of error'', or SSE). The corresponding \textbf{F-test} tests the null hypothesis that a model with no independent variables (that is, an intercept-only model with $\beta_1 = 0$ and $\beta_2 = 0$) fits the data as well as our model. The F-statistic follows an $F$ distribution with $p$ and $n-p-1$ degrees of freedom (see Chapter~\ref{chapter:probabilitydistributions}, Section~\ref{sect:fdist}). The $p$-value reported in the model output is the $p$-value for this hypothesis test.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Example: Small Cities Pollution Dataset}

The following data come from an early study that examined the possible link between air pollution and mortality. The authors examined 60 cities throughout the United States and recorded the following data:
\begin{center}
\texttt{ \small \begin{tabular}{ll}
\toprule
MORT & Total age-adjusted mortality from all causes, \\
& in deaths per 100,000 population \\
PRECIP & Mean annual precipitation (in inches) \\
EDUC & Median number of school years completed \\
& for persons of age 25 years or older \\
NONWHITE & Percentage of the 1960 population that is nonwhite \\
NOX & Relative pollution potential of oxides of nitrogen \\
SO2 & Relative pollution potential of sulfur dioxide \\
\bottomrule
\end{tabular}
}
\end{center}
Note: ``Relative pollution potential'' refers to the product of the tons emitted per day per square kilometer and a factor correcting the SMSA dimensions and exposure.

We want to predict the value of \texttt{MORT} ($y$) using the predictors \texttt{PRECIP, EDUC, NONWHITE, NOX,} and \verb|SO2| ($x_1, x_2, x_3, x_4$ and $x_5$). Here is the output for a fitted linear regression model:
\begin{center}
\includegraphics[width=0.8\textwidth]{img/linear-small-cities-model.png}
\end{center}

\begin{question}{}
Interpret the values of each of these coefficients. Based on the coefficient values and their standard errors, which predictor(s) do you think have the greatest impact on mortality?
\end{question}

\begin{question}{}
In this model, is the effect of one predictor (say, \verb|PRECIP|) impacted by the value(s) of any of the other predictor(s)? How does this differ from the other regression algorithms we've seen (KNN and decision trees)? What are the advantages and disadvantages of this choice?
\end{question}

\begin{question}{}
Is a normal distribution the right distribution to model an outcome of age-adjusted mortality (MORT)? Why or why not? Look back at our discussion of the normal distribution in Chapter~\ref{chapter:probabilitydistributions} if you need a refresher.
\end{question}


\documentclass{subfile}

\begin{document}
	\section{Majorization and Symmetric Sums}\label{sec:mazorization}

	Let $\mathbf{x}$ and $\mathbf{y}$ be two vectors of $n$ real numbers. We say that $\mathbf{x}$ \textit{dominates} or \index{majorization}\textit{majorizes} $y$ if
		\begin{align*}
			x_{1}
				& \geq x_{2}\geq \ldots\geq x_{n}\\
			y_{1}
				& \geq y_{2}\geq \ldots\geq y_{n}\\
			x_{1}+\ldots+x_{n}
				& = y_{1}+\ldots+y_{n}\\
			x_{1}+\ldots+x_{k}
				& \geq y_{1}+\ldots+y_{k}
		\end{align*}
	for $1\leq k\leq n-1$. If $\mathbf{x}$ dominates $\mathbf{y}$ (resp. $\mathbf{y}$ is \textit{dominated by} $\mathbf{x}$), then we write $\mathbf{x}\succ\mathbf{y}$ (resp. $\mathbf{y}\prec\mathbf{x}$). For example, $(4,0,0)\succ(3,1,0)\succ(2,2,0)$. The vectors $\mathbf{x}$ and $\mathbf{y}$ need not be monotonic because we can just sort them into monotonic vectors.

	We will also introduce the cyclic and symmetric polynomials and notations juxtaposed with them. The expression $x^{2}+y^{2}+z^{2}$ is \textit{symmetric} whereas $x^{2}y+y^{2}z+z^{2}x$ is \textit{cyclic} but not symmetric because $y^{2}x+z^{2}y+x^{2}z\neq x^{2}y+y^{2}z+z^{2}x$. A symmetric polynomial in the variables $x_{1},\ldots,x_{n}$ should remain same regardless of the order in which the variables are used. So $f(x_{1},\ldots,x_{n})$ is symmetric if $f$ remains \textit{invariant} for all permutations of $x_{1},\ldots,x_{n}$ in the expression unlike the cyclic example we just saw. For example, $xy+yz+zx$ is symmetric and so is $xyz$. And $\frac{a}{b}+\frac{b}{c}+\frac{c}{a}$ is cyclic but not symmetric. We can use the cyclic and symmetric notations to represent the expressions in a short form. Here are some demonstrations.
		\begin{align*}
			a^{2}+b^{2}+c^{2}
				& = \sum_{cyc}a^{2}\\
			a^{2}b+b^{2}c+c^{2}a
				& = \sum_{cyc}a^{2}b\\
			xy+yz+zx
				& = \sum_{cyc}xy
		\end{align*}
	Note that even though the expression $xy+yz+zx$ and $a^{2}+b^{2}+c^{2}$ are symmetric, we do not consider them symmetric polynomial sums in this notation. A symmetric polynomial sum should have all $n!$ terms in the sum since it is symmetric on all $n!$ permutations of the variables in it. Even if there can be duplicates, the total number of terms should still remain $n!$. For this reason, this sum is often denoted by
		\begin{align*}
			\sum{!}
				&\mbox{ or }\sum_{\sigma}
		\end{align*}
	where $\sigma$ indicates that the sum is taken over all possible permutations. Here are some examples.
		\begin{align*}
			\sum_{sym}a^{2}
				& = 2(a^{2}+b^{2}+c^{2})\\
			\sum{!} x^{2}y
				& = x^{2}y+y^{2}x+y^{2}z+z^{2}y+z^{2}x+x^{2}z\\
			\sum_{\sigma}x^{3}y
				& = x^{3}y+x^{3}z+y^{3}z+y^{3}x+z^{3}x+z^{3}y
		\end{align*}
	Let $\mathbf{x}$ and $\mathbf{a}$ be vectors with $n$ elements. We write
		\begin{align*}
			F(\mathbf{x}, \mathbf{a})
				& = x_{1}^{a_{1}}\cdots x_{n}^{a_{n}}\\
			T(\mathbf{x},\mathbf{a})
				& = \sum{!}F(\mathbf{x},\mathbf{a})\\
				& = \sum_{sym} F(\mathbf{x},\mathbf{a})\\
				& = \sum{!}x_{1}^{a_{1}}\cdots x_{n}^{a_{n}}
		\end{align*}
	We will explain it a little bit more. Say, we want to find the symmetric polynomials of the type $x^{3}y$ for the variables $x,y,z$. Write it as $x^{3}y^{1}z^{0}$. Now, we fix the powers $3,1,0$ in their respective places and let $x,y,z$ run through the permutations. Then we get
		\begin{align*}
			x^{3}y^{1}z^{0}+x^{3}z^{1}y^{0}+y^{3}z^{1}x^{0}+y^{3}x^{1}z^{0}+z^{3}x^{1}y^{0}+z^{3}y^{1}x^{0}
		\end{align*}
	See that this is exactly what we got in the last line of
		\begin{align*}
			\sum_{\sigma}x^{3}y
		\end{align*}
	Some authors write this notation as
		\begin{align*}
			T[\mathbf{a}](\mathbf{x})
				& = T[a_{1},\ldots,a_{n}](x_{1},\ldots,x_{n})
		\end{align*}
	Simply $T[\mathbf{a}]=T[a_{1},\ldots,a_{n}]$ is used if it is clear what $\mathbf{x}$ is.
		\begin{align*}
			T[1,0,\ldots,0]
				& = (x_{1}+\ldots+x_{n})(n-1)!\\
			T\left[\dfrac{1}{n},\ldots,\dfrac{1}{n}\right]
				& = n!\sqrt[n]{x_{1}\cdots x_{n}}
		\end{align*}
	Then for a vector of positive real numbers $\mathbf{x}$, we can write the arithmetic-geometric mean inequality as
		\begin{align*}
			T[1,0,\ldots,0]
				& \geq T\left[\dfrac{1}{n},\ldots,\dfrac{1}{n}\right]
		\end{align*}
	We can also define mean values based on the symmetric polynomials. We call
		\begin{align*}
			\mathfrak{M}[\mathbf{a}](\mathbf{x})
				& = \dfrac{T[\mathbf{a}](\mathbf{x})}{n!}
		\end{align*}
	the \index{symmetrical mean}\textit{symmetrical mean}. For example,
		\begin{align*}
			\mathfrak{M}[1,0\ldots,0]
				& = \dfrac{(n-1)!}{n!}(a_{1}+\ldots+a_{n})\\
				& = \mathfrak{A}(\mathbf{a})\\
			\mathfrak{M}\left[\dfrac{1}{n},\ldots,\dfrac{1}{n}\right]
				& = \dfrac{n!}{n!}a_{1}^{\frac{1}{n}}\cdots a_{n}^{\frac{1}{n}}\\
				& = \mathfrak{G}(\mathbf{a})
		\end{align*}
	So, the symmetrical mean is a generalization of $\mathfrak{A}$ and $\mathfrak{G}$. While we are talking about symmetric sums, we should consider the next identity.
		\begin{align*}
			(x+a_{1})\cdots(x+a_{n})
				& = x^{n}+\binom{n}{1}x^{n-1}D_{1}+\binom{n}{2}x^{n-2}d_{2}+\ldots+\binom{n}{n}D_{n}
		\end{align*}
	where $D_{k}$ is the sum of products of $a_{1},\ldots,a_{n}$ taken $k$ at the same time. If $S_{k}$ is the coefficient of $x^{n-k}$ in the expansion, then
		\begin{align*}
			S_{k}
				& = \binom{n}{i}D_{k}
		\end{align*}

		\begin{theorem}[\itshape Newton's inequality]\label{thm:newton}
			For $1\leq i\leq n-1$, we have
				\begin{align*}
					D_{i}^{2}
						& \geq D_{i-1}D_{i+1}
				\end{align*}
		\end{theorem}

		\begin{theorem}[\itshape Maclaurin's inequality]\label{thm:maclaurin}
			With the same notation as \nameref{thm:newton},
				\begin{align*}
					D_{1}
						& \geq \sqrt{D_{2}}\geq\ldots\geq\sqrt[n]{D_{n}}
				\end{align*}
		\end{theorem}
\end{document}


% !TEX TS-program = pdflatexmk
\documentclass{article} % For LaTeX2e
\usepackage{nips15submit_e,times}
\usepackage{hyperref}
\usepackage{url}
%\documentstyle[nips14submit_09,times,art10]{article} % For LaTeX 2.09

\usepackage{amsthm}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{mathtools}
\usepackage{thmtools}

\usepackage{algpseudocode,algorithm,algorithmicx}

%\newtheorem{theorem}{Theorem}
%\newtheorem{corollary}{Corollary}[theorem]
%\newtheorem{lemma}[theorem]{Lemma}
%\newtheorem{example}{Example}[theorem]

%\theoremstyle{definition}
\declaretheorem[style=definition,qed=$\blacksquare$]{definition}
\declaretheorem[style=definition,qed=$\blacktriangle$,sibling=definition]{example}

\declaretheorem[style=plain,qed=$\blacktriangle$,sibling=definition]{theorem}
\declaretheorem[style=plain,qed=$\lhd$,sibling=definition]{lemma}

%\newtheorem{definition}{Definition}
\newtheorem{assumption}{A}
\newtheorem{question}{Q}

\newcommand\myeq{\stackrel{\mathclap{\tiny\mbox{d}}}{=}}

\newcommand\mc{\mathcal} %calligraphic
\newcommand\ts{\mathcal} %tensor
\newcommand\mt{} %matrix
\newcommand\vt{\mathbf} %vector
\newcommand\fn{} %function
\newcommand\triple[3]{(#1 \stackrel{#2}\rightarrow #3)}
%\newcommand\triple[3]{(#1, #2, #3)}

\title{Collaborative Matrix Completion}

\author{
Dongwoo Kim
}

% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to \LaTeX{} to determine where to break
% the lines. Using \AND forces a linebreak at that point. So, if \LaTeX{}
% puts 3 of 4 authors names on the first line, and the last on the second
% line, try using \AND instead of \And before the third author name.

\newcommand{\fix}{\marginpar{FIX}}
\newcommand{\new}{\marginpar{NEW}}

\nipsfinalcopy % Uncomment for camera-ready version

\begin{document}

\maketitle

\begin{abstract}
Low-rank or approximately low-rank matrix approximation is a popular method to complete a matrix given partial observation of its entry. As the seminar work by \cite{candes2009exact} pointed out, one can recover the original matrix with high probability given a certain number of samples. In practice, when the number of observations is not satisfying this criteria, one widely used heuristic to overcome the shortage of samples is to jointly completing multiple matrices that shares some common dimensions. For example, to get a reliable estimator of user-movie rating matrix, one may use movie-actor matrix as additional information to get more information about movies. In this work, we would like to uncover when and how this joint completion would work and why.
\end{abstract}

\section{Preliminary}
Assume that unknown rank-$r$ matrix $M$ is $m_1 \times m_2$. The goal of single matrix completion is to recover original matrix $M$ from partially observed entries. The singular value decomposition (SVD) is
\begin{equation}
M = \sum_{i=1}^{r}\sigma_i u_i v_i^\top,
\end{equation}
where $\sigma_i,...,\sigma_r \geq 0$ are the singular values, and $u_1,...,u_r$ and $v_1,...,v_r$ are two sets of orthonormal singular vectors. The degree of freedom of this matrix is $(m_1+m_2)r - r^2$, which means we cannot recover the original matrix if less than $(m_1+m_2)r - r^2$ entries are available.

Let $\Omega_M$ be a set of indices of observed entries, i.e. $(i,j) \in \Omega_M$ if $M_{ij}$ is observed, and $\mc{P}_{\Omega}: \mathbb{R}^{m_1\times m_2} \rightarrow \mathbb{R}^{m_1\times m_2}$ be the orthogonal projection onto index set $\Omega$ which vanishes outside of $\Omega$; that is $\bar{M} = \mc{P}_{\Omega}(M)$ is defined as
\begin{equation}
\bar{M}_{ij} = \left\{
  \begin{array}{lr}
    M_{ij}, & (i,j) \in \Omega\\
    0, & \text{otherwise}.
  \end{array}
\right.
\end{equation}

The spectral norm of matrix $M$ is denoted by
\begin{equation}
||M|| := \sup_{u\in \mathbb{R}^n:||u||=1}||Mu|| = \sup_{j\in[n]}\sigma_j(M),
\end{equation}
which corresponds to the largest singular value of matrix $M$. The nuclear norm of matrix $M$ is denoted by
\begin{equation}
||M||_* := \sum_{j\in[n]}\sigma_j(M),
\end{equation}
which corresponds to the sum of singular values of matrix $M$. Additionally, we let $||M||_\infty := \max_{i,j}|M_{ij}|$.

The Euclidean inner product between two matrices is defined by
\begin{align}
\langle M, N \rangle := \text{tr}(M^\top N),
\end{align}
and corresponding Frobenius norm is denoted as
\begin{align}
||M||_F
\end{align}

By the duality between nuclear and spectral norm, we can compute an upper bound on the Euclidean inner product between matricies
\begin{align}
\langle M, N \rangle & = ||M|| \langle \frac{M}{||M||}, N \rangle \\
& = ||M|| \text{tr}(\frac{M}{||M||}^\top N)\\
& \leq ||M|| \sup_{||X|| \leq 1} \text{tr}(X^\top A)\\
& = ||M||\,||A||_*.
\end{align}

\subsection{Sampling scheme}
Let $\pi_{ij}$ be the probability to observe the $(i,j)$ entiry of $M$. Let $C_j = \sum_{i=1}^{m_1} \pi_{ij}$ be the probability of observing an entry from column $j$ and $R_i = \sum_{j=1}^{m_2} \pi_{ij}$ be the probability of observing an entry from row $i$.
If there is a noise in observation, then noisy observation of entry
\begin{align}
\bar{M}_{ij} = M_{ij} + \sigma\xi_{ij}, \quad (i,j) \in \Omega
\end{align}
where noise variable $\xi_{ij}$ satisfies $\mathbb{E}[\xi_{ij}] = 0$ and $\mathbb{E}[\xi_{ij}^2] = 1$, and $\sigma$ is the variance of the noise.

In terms of the trace regression model, the observation also can be defined as
\begin{align}
\bar{M}_{i} = \text{tr}(E_i^\top M) + \sigma\xi_i, \quad i=1,...,n
\end{align}
where $E_i$ are random matrices of dimension $m_1 \times m_2$, and tr($X$) denotes the trace of matrix $X$. Each $E_i$ is i.i.d copy from a random matrix $E$ having distribution $\Pi$ on the set $\mathcal{E} = \{e_j(m_1)e_k^\top(m_2), 1\leq j \leq m_1, 1\leq k \leq m_2 \}$, where $e_j(m)$ is $j$th canonical basis vector in $\mathbb{R}^m$, i.e., zero vector except $i$th entry equal to 1. Following two stochastic terms play an important role in the case of noisy observation analysis. For an Rademacher sequence $\{\epsilon_i\}_{i=1}^{n}$, we define
\begin{align}
\Sigma_R = \frac{1}{n}\sum_{i=1}^{n}\epsilon_i E_i \quad \text{and} \quad\Sigma = \frac{\sigma}{n}\sum_{i=1}^{n}\xi_i E_i.
\end{align}
Let $||M||_{L_2(\Pi)}^2 = \mathbb{E}[\langle E,M \rangle)]^2$, where the expectation is taken over ${E}$. If $\pi_{ij} \geq pm_1m_2$ for all entries, then this implies
\begin{align}
\label{eqn:fro_ineq}
||M||_{L_2(\Pi)}^2 \geq \frac{||M||_F^2}{pm_1m_2}.
\end{align}

\subsection{Orthogonal projection}

Set $U := \text{span}(u_1,...,u_r)$ and $V := \text{span}(v_1,...,v_r)$. It is convenient to introduce the orthogonal decomposition $\mathbb{R}^{m_1 \times m_2} = T \oplus T^\perp$ where $T$ is the linear space spanned by elements of the form $u_ky^\top$ and $xv_k^\top$, $1\leq k \leq r$, where $x$ and $y$ are arbitrary vectors, and $T^\perp$ is its orthogonal complement. The orthogonal projection $\mc{P}_T:\mathbb{R}^{m_1 \times m_2} \rightarrow T$ onto $T$ is given by
\begin{equation}
\mc{P}_T(Z) = P_U(Z) + ZP_V - P_UZP_V,
\end{equation}
where $P_U$ and $P_V$ are the orthogonal projections onto $U$ and $V$ respectively. The orthogonal projection onto $T^\perp$ is given by
\begin{equation}
\mc{P}_{T^\perp}(Z) = (I_{m_1} - P_U)Z(I_{m_2} - P_V),
\end{equation}
where $I_m$ denotes the $m \times m$ identity matrix.

\section{Single Matrix Completion}
The noiseless matrix completion was first studied by \cite{candes2009exact}, where the number of samples needed to recover a matrix of rank $r$ exactly is provided under some incoherent assumptions on the singular vectors of the matrix.
\begin{definition}[Coherence \cite{candes2009exact}]
Let $U$ be a subspace of $\mathbb{R}^n$ of dimension $r$, i.e., $U := \text{span}(u_1, ..., u_r)$, and $P_U$ be the orthogonal projection onto $U$. Then the coherence of $U$ is defined to be
\begin{equation}
\mu(U) := \frac{m_1}{r}\max_{1\leq i \leq m_1}||P_U e_i||^2,
\end{equation}
where $e_i$ is the $i$th canonical basis vector.
\end{definition}

\begin{assumption}
The coherences obey $\max(\mu(U),\mu(V)) \leq \mu_0$ for some positive $\mu_0$.
\end{assumption}
\begin{assumption}
The matrix $E=\sum_{1\leq k \leq r} u_k v_k^\top$ has a maximum entry bounded by $\mu_1 \sqrt{r/(m_1m_2)}$ in absolute value for some positive $\mu_1$.
\end{assumption}

\begin{theorem}[\cite{candes2009exact}]
Under the assumption 1 and 2, a nuclear norm minimisation
\begin{align}
\text{minimise}&\quad ||X||_* \\
\text{subject to}&\quad \mc{P}_{\Omega}(X) = \mc{P}_{\Omega}(M)
\end{align}
perfectly recovers the original matrix with high probability if the number of uniformly sampled entires $n$ obey $n\geq \mathcal{O}(\max(\mu_1^2,\mu_0^{1/2},\mu_1,\mu_0n^{1/4}) m_2 r \beta \log m_2)$ for some $\beta > 2$.
\end{theorem}
Later, a tighter analysis of the same convex relaxation was developed in \cite{candes2010power,recht2011simpler}. More practical settings, where the few observed entries are corrupted by noise, has been extensively studied recently \cite{candes2010matrix,keshavan2010matrix,negahban2012restricted,klopp2014noisy,lafond2015low}.
%These studies show that when the distribution on noise is additive and sub-exponential, then the prediction error with the nuclear norm minimiser $\hat{X}$ satisfies with high probability
%\begin{equation}
%\frac{||\hat{X} - X||^2_{F}}{m_1 m_2} = \mathcal{O}\bigg(
%\frac{(m_1+m_2) \text{rank}(X) \log (m_1 + m_2)}{m}
%\bigg),
%\end{equation}
%where $||\cdot||_F$ denotes the Frobenius norm, and $X\in\mathbb{R}^{m_1 \times m_2}$.
To prevent an overfitting of the noisy matrix observation, a penalised nuclear norm estimator of $M$ has been introduced
\begin{align}\label{eqn:nn_est}
\hat{X} = {\arg\min}_{||{A}||_\infty \leq \gamma} \bigg\{\frac{1}{n} \sum_{(i,j) \in \Omega}(\bar{M}_{ij} - {A}_{ij})^2 + \lambda ||{A}||_* \bigg\},
\end{align}
where $\bar{M}_{ij}$ is a noisy observation of $M_{ij}$, $\lambda > 0$ is a regularisation parameter, and $\gamma$ is an upper on the absolute value of each entry. The optimiser consists of the sum of a data fitting terms and a nuclear norm penalisation term.

\begin{theorem}[Noisy matrix completion \cite{klopp2014noisy}]
Let noisy observation of $M$ be sampled i.i.d. from distribution $\Pi$ with known variance of the noise $\sigma$. If $\lambda \geq 3||\Sigma||$ and following two assumptions are satisfied:

\begin{assumption}
\label{assume:row_column}
There exists a positive constant $L \geq 1 $ such that
$\max_{i,j}(C_i, R_j) \leq L / \min(m_1, m_2)$.
\end{assumption}

\begin{assumption}
\label{assume:min_p}
There exists a positive constant $p \geq 1 $ such that
$\pi_{jk} \geq (p m_1 m_2)^{-1}$.
\end{assumption}

Then, there exist numerical constants $c_1$, and $c_2$ such that
\begin{equation}
\frac{||\hat{X} - M||_F^2}{m_1m_2} \leq \max\bigg\{ c_1 p^2 m_1 m_2 r (\lambda^2 + \gamma^2(\mathbb{E}||\Sigma_R||)^2), c_2\gamma^2p\sqrt{\frac{\log(m_1 + m_2)}{n}} \bigg\},
\end{equation}
with probability at least $1 - 2/(m_1 + m_2)$.
\end{theorem}

An upper bound of the stochastic term $\mathbb{E}||\Sigma_R||$ could be also obtained under certain assumptions (e.g., sub-exponential noise). See more details in \cite{klopp2014noisy}.

%\textbf{Sketch of Proof}. From the definition of estimator in Equation $\ref{eqn:nn_est}$, we know that
%\begin{equation}
%\frac{1}{n} \sum_{(i,j) \in \Omega}(\bar{M}_{ij} - \hat{X}_{ij})^2 + \lambda||\hat{X}||_*
%\leq \frac{1}{n} \sum_{(i,j) \in \Omega}(\bar{M}_{ij} - {M}_{ij})^2 + \lambda||M||_*,
%\end{equation}
%which implies
%\begin{equation}
%\frac{1}{n}||\mathcal{P}_\Omega(M - \hat{X}) ||_F^2 \leq 2||\Sigma|| ||M - \hat{X}||_* + \lambda(||M||_* - ||\hat{X}||_*).
%\end{equation}

\section{Joint Matrix Completion}
Let $A_0 \in \mathbb{R}^{m_1 \times m_2}$ and $B_0 \in \mathbb{R}^{m_1 \times m_3}$ be two matrices where the first dimension represents the same object, i.e. user-rating matrix $A$ and user-attribute matrix $B$. When the number of observed entries $n_A$ of matrix $A_0$ is insufficient to obtain a stable estimator of the unobserved entries, one widely used heuristic method is a joint factorisation of $A_0$ and $B_0$ with the hope that the both matrices share some common low-rank structure in its first dimension. The collaborative matrix factorisation has been widely used in practice, but there is not theoretical guarantee so far.

One of our main question is how many samples are required to perfectly recover the original matrices $A_0$ and $B_0$ under what assumptions. More precisely, an interesting question might be the number of observation $n_A$ of matrix $A_0$ needed to recover the matrix $A_0$ given the number of observation $n_B$ of matrix $B_0$ or vice versa. Often, it is more easier to obtain the entries of $B_0$ instead of those in $A_0$, therefore, understanding the nature of joint completion will provide some guidelines to the joint completion approach. We start from an assumption to make the analysis feasible.

\begin{assumption}\label{assume:share}
Let SVD of $A_0 = \sum_{i=1}^{r_A} \sigma_i^{(A_0)} u_i^{(A_0)} v_i^\top$ and $B_0 = \sum_{i=1}^{r_B} \sigma_i^{(B_0)} u_i^{(B_0)} w_i^\top$. We assume that the rank of two matrices are the same, i.e., $r_M = r_N = r$, and the singular vector $u_i^{(A_0)}$ and $u_i^{(B_0)}$ are the same in order to share the common low-rank structure between two matrices\footnote{Remark: most of existing knowledge graph factorisation models such as RESCAL follow this assumption if we consider each relation as a matrix (a knowledge graph is a collection of matrices).}.
\end{assumption}

Let $Z_0 = [A_0, B_0] \in \mathbb{R}^{m_1 \times (m_2+m_3)}$ be the horizontally combined matrix of $A_0$ and $B_0$. The SVD of $Z$ is $\sum_{i=1}^{r}\sigma_i u_i v_i^\top$ where $v_i^\top = [v_i^{(A_0)\top}, v_i^{(B_0)\top}]$ is the stacked right singular vectors of $A_0$ and $B_0$. If both singular vectors satisfy the incoherence assumptions, the joint matrix completion problem is reduced to the single matrix completion problem with different sampling rates between the first $m_2$ columns and the rightmost $m_3$ columns of combined matrix. Some of the previous studies focus on a non-uniform sampling distribution of matrix \cite{foygel2011learning,lounici2011optimal,negahban2012restricted,klopp2014noisy}. For example, popular movies are more likely to be rated by many users in a collaborative filtering. This will lead to different sampling distributions between different columns (rows).

Our problem is a special case where we sample $n_M$ samples from leftmost $m_2$ columns and $n_N$ from rightmost $m_3$ columns. More formally, let $\Pi_A$ be a distribution on set $\mc{E}^{A} = \{e_j(m_1)e_k^\top(m_2+m_3), 1\leq j \leq m_1, 1\leq k \leq m_2\}$, and $\Pi_{B}$ a distribution on set $\mc{E}^{B} = \{e_j(m_1)e_k^\top(m_2+m_3), 1\leq j \leq m_1, 1+m_2\leq k \leq m_2+m_3\}$. And define $\Pi_\rho = \rho \Pi^{A} + (1-\rho)\Pi^{B}$ where $0 < \rho < 1/2$. If $\Phi_A$ and $\Phi_B$ satisfy Assumption \ref{assume:min_p} with parameter $p_A$ and $p_B$, Equation \ref{eqn:fro_ineq} implies
\begin{align}
||Z_0||_{L_2(\Pi_\rho)}^2 \geq \min(\frac{\rho}{p_Am_1m_2},\frac{1-\rho}{p_Bm_1m_3})||Z_0||_F^2 \geq \frac{\rho}{pm_1(m_2+m_3)} ||Z_0||_F^2,
\end{align}
where $p = \min(p_A, p_B)$.

%And define $\Pi_\rho = \rho \Pi^{(M)} + (1-\rho) \Pi^{(N)}$ where $0 \leq \rho \leq 1$. %If we set $\rho = \frac{n_M}{n_M+n_N}$, and
%There are $p^{(M)}$ and $p^{(N)}$ which satisfies Assumption \ref{assume:min_p} for each distribution, respectively, then the inequality in Equation \ref{eqn:fro_ineq} also holds as well
%\begin{align}
%||Z||_{\Pi_\rho}^2 \geq \frac{||Z||_F^2}{p'm_1(m_2+m_3)},
%\end{align}
%where $p' = \min(\rho p^{(M)}m_1m_2, (1-\rho) p^{(N)}m_1m_3)$.

\begin{question}[$n_A$ given $n_B$]
Given $n_B$ uniformly sampled entries of matrix $B_0$, how many samples do we need from $A_0$ to perfectly recover matrix $A_0$? More practically, what will be the relation between the reconstruction error on $A_0$ and the sample size $n_A$ and $n_B$.
\end{question}
Let the penalised estimator of joint matrix $Z$
\begin{align}
\label{eqn:z_estimator}
\hat{Z} = {\arg\min}_{||{A}||_\infty \leq \gamma} \bigg\{\frac{1}{n_M + n_N} \sum_{(i,j) \in \Omega}(\bar{Z}_{ij} - {A}_{ij})^2 + \lambda ||{A}||_* \bigg\},
\end{align}
where $\bar{Z}_{ij}$ denotes the noisy observation of $(i,j)$ entry of $Z_0$. Let $\hat{A}$ (\textit{resp.} $\hat{B}$) be the corresponding part of $A_0$ (\textit{resp.} $B_0$) in $\hat{Z}$, then the first question corresponds to identify an upper bound on
$||\hat{A} - A_0||_F^2$
with respect to the other quantities such as $n_A$ and $n_B$.

The Frobenius norm error of the combined matrix can be decomposed into two different parts:
\begin{equation}
\label{eqn:decomp}
\frac{||\hat{Z} - Z||_F^2}{m_1(m_2+m_3)} = \frac{||\hat{A} - A_0||_F^2}{m_1(m_2+m_3)} + \frac{||\hat{B} - B_0||_F^2}{m_1(m_2+m_3)}.
\end{equation}

%From the previous work, it is possible to obtain an upper bound on Frobenius error of combined matrix. However, this does not give any information gain obtained from the shared low rank structure. Therefore, a lower bound on the second term in RHS of equation \ref{eqn:decomp} is needed to induce an upper bound on equation \ref{eqn:upper_m}.

From the definition of estimator in Equation \ref{eqn:z_estimator}, the following inequality holds (the objective function  w.r.t optimiser $\hat{Z}$ is always less or equal than the objective function w.r.t unknown true matrix $Z_0$).
\begin{align}
\frac{1}{n} \sum_{(i,j) \in \Omega}(\bar{Z}_{ij} - \hat{Z}_{ij})^2 + \lambda ||\hat{Z}||_*
\leq \frac{1}{n} \sum_{(i,j) \in \Omega}(\bar{Z}_{ij} - {Z}_{0ij})^2 + \lambda ||Z_0||_*,
\end{align}
where $n = n_M + n_N$. This implies
\begin{equation}
\frac{1}{n}||\mathcal{P}_\Omega(Z_0 - \hat{Z}) ||_F^2 \leq 2||\Sigma||\,||Z_0 - \hat{Z}||_* + \lambda(||Z_0||_* - ||\hat{Z}||_*). \\
\end{equation}
Applying Lemma \ref{lmm:normdiff} and using triangular inequality $||Z_0 - \hat{Z}||_* \leq ||\mc{P}_{Z_0}(Z_0 - \hat{Z})||_* + ||\mc{P}_{Z_0}^\perp(Z_0 - \hat{Z})||_*$, we obtain
\begin{equation}
\frac{1}{n}||\mathcal{P}_\Omega(Z_0 - \hat{Z}) ||_F^2 = \frac{1}{n}||\mc{P}_{\Omega_A}(A_0-\hat{A})||_F^2 + \frac{1}{n}||\mc{P}_{\Omega_B}(B_0-\hat{B})||_F^2
\leq \frac{5}{3}\lambda ||\mc{P}_{Z_0}(Z_0-\hat{Z})||_*.
\label{eqn:tri}
\end{equation}

Let $\zeta = 3168p\gamma^2\text{rank}(Z_0)m_1(m_2+m_3)\rho^{-1}(\mathbb{E}||\Sigma_R||)^2$. Since $||\hat{Z} - Z_0||_* \leq \sqrt{72\text{rank}(Z_0)}||\hat{Z} - Z_0||_F$, if $||\hat{Z}-Z_0||_\infty^{-1}(\hat{Z}-Z_0) \in \mc{C}(72\text{rank}(Z_0))$, then Lemma \ref{lmm:constraintset} and \ref{lmm:prj_rankbound} leads an upper bound on the squared error of entire matrix
\begin{align}
\frac{1}{2}||\hat{Z}-Z_0||_{L_2(\Pi_\rho)}^2 &\leq \frac{5}{3}\lambda\sqrt{2\text{rank}(Z_0)}||\hat{Z} - Z_0||_F + \zeta \\
\frac{\rho}{2pm_1(m_2+m_3)} ||\hat{Z} - Z_0||_F^2 &\leq \frac{5}{3}\lambda\sqrt{2\text{rank}(Z_0)}||\hat{Z} - Z_0||_F + \zeta \label{eqn:bound1}
\end{align}
with probability at least $1-2/(m_1+m_2+m_3)$. By using inequality $ab \leq a^2 + b^2$, the first term on the RHS is bounded
\begin{align}
\frac{5}{3}\lambda\sqrt{2\text{rank}(Z_0)}||\hat{Z} - Z_0||_F & \leq \bigg(\frac{5}{3}\lambda\sqrt{\frac{8\text{rank}(Z_0)pm_1(m_2+m_3)}{\rho}}\bigg)^2
+ \bigg(\sqrt{\frac{\rho}{4pm_1(m_2+m_3)}}||\hat{Z} - Z_0||_F\bigg)^2 \\
& = \frac{320\lambda^2 \text{rank}(Z_0)pm_1(m_2+m_3)}{9\rho} + \frac{\rho}{4pm_1(m_2+m_3)}||\hat{Z}-Z_0||_F^2.
\end{align}
Applying this inequality to Equation \ref{eqn:bound1} yields
\begin{align}
\frac{\rho}{4pm_1(m_2+m_3)} ||\hat{Z} - Z_0||_F^2 \leq  36\lambda^2 \text{rank}(Z_0)pm_1(m_2+m_3)\rho^{-1} + \zeta
\end{align}
Therefore, there exist numerical constants $c_1$ such that
\begin{align}
||\hat{Z} - Z_0||_F^2 \leq  c_1 \text{rank}(Z_0)\bigg(\frac{pm_1(m_2+m_3)}{\rho}\bigg)^2(\lambda^2 + \gamma^2(\mathbb{E}||\Sigma_R||)^2).
\end{align}
It is trivial to see that the bound on matrix $A_0$
\begin{align}
\frac{||\hat{A}-A_0||_F^2}{m_1m_2} \leq  c_1 \text{rank}(Z_0)\frac{m_1}{m_2}\bigg(\frac{p(m_2+m_3)}{\rho}\bigg)^2(\lambda^2 + \gamma^2(\mathbb{E}||\Sigma_R||)^2).
\end{align}

%Let $\alpha = ||M - \hat{M}||_\infty (\leq 2\gamma)$. If $\alpha^{-1}(M-\hat{M}) \in \mc{C}(t)$, then by Lemma \ref{lmm:constraintset}
%\begin{equation}
%\frac{n_M}{2n\alpha^2}||M-\hat{M}||_{\Pi^{(M)}}^2  - 44 \frac{ptm_1m_2n_M}{n}(\mathbb{E}||\Sigma_R^{(M)}||)^2
%\leq \frac{n_M}{n}\frac{1}{\alpha^2n_M}||\mc{P}_{\Omega_M}(M-\hat{M})||_F^2 \label{eqn:apl_lmm}
%\end{equation}
%Same argument can be made on the lower bound of $||\mc{P}_{\Omega_N}(N-\hat{N})||_F^2$ if $\beta^{-1}(N-\hat{N}) \in \mc{C}(t)$
%\begin{equation}
%\label{eqn:apl_lmm2}
%\frac{n_N}{2n\beta^2}||N-\hat{N}||_{\Pi^{(N)}}^2  - 44 \frac{ptm_1m_3n_N}{n}(\mathbb{E}||\Sigma_R^{(N)}||)^2
%\leq \frac{n_N}{n}\frac{1}{\beta^2n_N}||\mc{P}_{\Omega_N}(N-\hat{N})||_F^2
%\end{equation}
%Combining \ref{eqn:tri}, \ref{eqn:apl_lmm} and \ref{eqn:apl_lmm2} yields
%\begin{align}
%&\frac{n_M}{2n}||M-\hat{M}||_{\Pi^{(M)}}^2 + \frac{n_N}{2n}||N-\hat{N}||_{\Pi^{(N)}}^2&  \notag\\
%&\quad \leq \frac{5}{3}\lambda ||\mc{P}_Z(Z-\hat{Z})||_* + 176 \frac{ptm_1\gamma^2}{n}\bigg(m_2n_M(\mathbb{E}||\Sigma_R^{(M)}||)^2 + m_3n_N(\mathbb{E}||\Sigma_R^{(N)}||)^2\bigg).&
%\end{align}
%Equation \ref{eqn:fro_ineq} implies
%\begin{align}
%&\frac{n_M}{2nm_1m_2p}||M-\hat{M}||_F^2 + \frac{n_N}{2nm_1m_3p}||N-\hat{N}||_F^2& \notag\\
%&\quad \leq \frac{5}{3}\lambda ||\mc{P}_Z(Z-\hat{Z})||_* + 176 \frac{ptm_1\gamma^2}{n}\bigg(m_2n_M(\mathbb{E}||\Sigma_R^{(M)}||)^2 + m_3n_N(\mathbb{E}||\Sigma_R^{(N)}||)^2\bigg).&
%\end{align}
%The upper bound on $||\mc{P}_Z(Z-\hat{Z})||_*$ can be decomposed into Frobenius error on two sub matrices:
%\begin{align}
%||\mc{P}_Z(Z-\hat{Z})||_* & \leq \sqrt{2r}||Z-\hat{Z}||_F \\
%& =  \sqrt{2r} \sqrt{||M-\hat{M}||_F^2 + ||N-\hat{N}||_F^2}\\
%& \leq 2r + \frac{1}{4}\bigg(||M-\hat{M}||_F^2 + ||N-\hat{N}||_F^2\bigg),
%\end{align}
%where the first inequality comes from $||A||_* \leq \sqrt{\text{rank}(A)}||A||_F$ and $\text{rank}(\mc{P}_Z(A)) \leq \text{rank}(P_{U^\perp}AP_V) + \text{rank}(P_{U}A)$, and we use the inequality $ab \leq a^2 + b^2/4$ in the third line.

\begin{lemma} \label{lmm:constraintset} Given constraint set
\begin{equation}
\mc{C}(t) = \bigg\{ A \in \mathbb{R}^{m_1 \times m_2}: ||A||_\infty = 1, ||A||_{L_2(\Pi)}^2 \geq \sqrt{\frac{64\log(m_1+m_2)}{\log(6/5)n}}, ||A||_* \leq \sqrt{t}||A||_F \bigg\},
\end{equation}
%where $A$ is a submatrix of $B$. Note that the condition $||A||_* \leq \sqrt{t}||B||_F$ always satisfied if $t$ = rank($B$) since $||A||_* \leq ||B||_*$.
Let $E_i$ be i.i.d. from distribution $\Pi$ on $\mc{E}$ which satisfies Assumptions \ref{assume:row_column} and \ref{assume:min_p}, and $\Sigma_R = n^{-1}\sum_{i=1}^{n}\epsilon_i E_i$, where $\{\epsilon_i\}_{i=1}^{n}$ is Rademacher sequence. Then, for all $A\in \mc{C}(t)$
\begin{align}
\frac{1}{2}||A||_{L_2(\Pi)}^2  - 44 ptm_1m_2(\mathbb{E}||\Sigma_R||)^2 \leq \frac{1}{n}||\mc{P}_{\Omega}(A)||_F^2
\end{align}
with probability at least $1-2(m_1+m_2)^{-1}$.
\end{lemma}

\begin{lemma}\label{lmm:normdiff}For any pair of matrices $A, B \in \mathbb{R}^{m_1 \times m_2}$, we have
\begin{equation}
||A||_* - ||B||_* \leq ||\mc{P}_A(A-B)||_* - ||\mc{P}_A^\perp(A-B)||_*
\end{equation}
\end{lemma}
\textbf{Proof} The definition on $\mc{P}_A^\perp(B)$ for any matrix $B$ implies $||A + \mc{P}_A^\perp(B-A)||_* = ||A||_* + ||\mc{P}_A^\perp(B-A)||_*$. Then the nuclear norm on $B$
\begin{align}
||B||_* = ||B + A - A||_* &= ||A + \mc{P}^\perp_A(B-A) + \mc{P}_A(B-A)||_*\\
&\geq ||A + \mc{P}^\perp_A(B-A)||_* + ||\mc{P}_A(B-A)||_*\\
&= ||A||_* + ||\mc{P}^\perp_A(B-A)||_* + ||\mc{P}_A(B-A)||_*,
\end{align}
where we use the triangular inequality on the projected space.

\begin{lemma}\label{lmm:bregman} If $\lambda > 3||\Sigma||$, then
$||\mc{P}_{Z_0}^\perp(\hat{Z}-Z_0)||_* \leq 5 ||\mc{P}_{Z_0}(\hat{Z}-Z_0)||_*$
\end{lemma}
\textbf{Proof} Let $Q^2(Z) = \frac{1}{n}\sum_{i=1}^{n}(\bar{Z}_i - \langle E_i, Z \rangle)^2$ which is convex in $Z$. From the Bregman divergence between $\hat{Z}$ and $Z_0$ with generating function $Q^2$, the following inequality holds
\begin{align}
Q^2(\hat{Z}) - Q^2(Z_0) &\geq - \frac{2}{n}\sum_{i=1}^n(\bar{Z}_i - \langle E_i, Z \rangle)\langle E_i, \hat{Z} - Z_0 \rangle = -2 \langle \Sigma, \hat{Z} - Z_0 \rangle \\
&\geq -2 ||\Sigma||\,||\hat{Z} - Z_0||_* \\
&\geq - \frac{2}{3} \lambda ||\hat{Z} - Z_0||_*
\end{align}
The definition of estimator $\hat{Z}$ implies
\begin{align}
\lambda||\hat{Z}||_* - \lambda||Z_0||_* \leq Q^2(Z_0) - Q^2(\hat{Z}) \leq \frac{2}{3} \lambda ||\hat{Z} - Z_0||_*
\end{align}
Using Lemma \ref{lmm:normdiff}, we compute
\begin{align}
\frac{2}{3}||\hat{Z} - Z_0||_* \leq ||\mc{P}_{Z_0}(\hat{Z} - Z_0)||_* - ||\mc{P}_{Z_0}^\perp(\hat{Z} - Z_0)||_*,
\end{align}
which together with the triangular inequality on $||\hat{Z} - Z_0||_*$ leads to Lemma \ref{lmm:bregman}.

\begin{lemma}\label{lmm:prj_rankbound} For any pair of matrices $A, B \in \mathbb{R}^{m_1 \times m_2}$, $\text{rank}(\mc{P}_{A}(B)) \leq 2 \text{rank}(A)$
\end{lemma}

\subsection{Lower Bound on $||B_0-\hat{B}||_F^2$}

It is crucial to derive a lower bound on $||N-\hat{N}||_F^2$. We can use the same argument used in \cite{lafond2015low} to show the lower bound.

\begin{question}[Approximate singular vectors]
What if the left singular vectors of $M$ and those of $N$ are not equal, but approximate one another? (i.e., $u_i^{(M)} \approx u_i^{(N)}$)
\end{question}

\begin{question}[Lower bound on $n_M$]
Assume that we have fully observed matrix $N$. How many samples do we need from $M$ to perfectly recover matrix $M$?
\end{question}

\begin{question}[Active learning on $N$ to recover $M$]
In many cases, it is more easy to obtain entries of $N$ instead of $M$. The question is which entry of $N$ would be actively sampled to improve the reconstruction performance of $M$.
\end{question}

%If the low-rank representations are the same, then we can apply the standard theories on the combined matrix $[X, Y] \in \mathbb{R}^{m_1 \times (m_2 + m_3)}$. However, this is not the case, under which conditions we can apply joint optimisation for reconstructing $X$ and what would be the error bound of the estimate? Following questions might be of interest to solve this problem.
%\begin{itemize}
%\item What are the corresponding optimiser of the nuclear norm optimiser for the joint factorisation?
%\item What structural assumptions should be made to link two matrices? for example, a certain similarity assumption between two singular value vectors or two low-rank representations?
%\item Noise / noiseless assumptions. which noise model?
%\item Sampling distribution. uniform? non-uniform?
%\end{itemize}

\bibliographystyle{apalike}
\bibliography{ref}

\end{document}