% !TEX root = ../main.tex

The protocols we will discuss work by recursively computing and comparing fingerprints of sets.
This chapter defines and motivates a specific fingerprinting scheme that admits efficient computation with small overhead for the storage and maintenance of auxiliary data structures. \Cref{initial-considerations} outlines the solution space and theoretic bounds. \Cref{group-fingerprints} characterizes a family of functions that admit efficient incremental computation, \Cref{collisions} proposes members of this family that can be used for fingerprinting, and \cref{crypto} examines security concerns in the face of malicious parties trying to find fingerprint collisions. We examine randomized solutions which compute with high probability fingerprints in logarithmic time in \cref{randomization}.

\section{Initial Considerations}
\label{initial-considerations}

Our protocols work by recursively testing fingerprints for equality. For our purposes, we can define a fingerprint or hash function as follows:

\begin{definition}
A \defined{hash function} is a function $\fun{\h}{U}{D}$ with a finite codomain such that for randomly chosen $u \in U$ and $d \in D$ the probability that $\h(u) = d$ is roughly\footnote{To keep the focus on data structure synchronization rather than being sidetracked by cryptography, we will for the most part keep arguments about probabilities qualitative rather than quantitative.} $\frac{1}{\abs{D}}$. $\h(u)$ is called the \defined{hash of $u$}, \defined{fingerprint of $u$} or \defined{digest of $u$}.
\end{definition}

Given a universe $U$ of items, a function $\enc : U \rightarrow \{0, 1\}^{*}$ for encoding items as binary strings, a linear order $\preceq$ on $U$, a hash function $\h: \{0, 1\}^{*} \rightarrow \{0, 1\}^{k}$ mapping binary strings to binary strings of length $k$, and some finite $S \subseteq U$, a natural starting point for defining a fingerprint of the set $S$ is to sort the items according to $\preceq$, concatenate the encodings, and hash the resulting string.

While this is straightforward to specify and implement, it does not suffice for our purposes. To allow for efficient set reconciliation, we need to be able to efficiently compute the new fingerprint after a small modification to the set such as insertion or deletion of a single item. Furthermore, we want to be able to efficiently compute the fingerprints of all subsets defined by a range of the original set.

The fingerprint based on concatenating encodings does not allow for efficient incremental reevaluation. When an item is added to $S$ which is less than any item previously in $S$, the hash function needs to be run over the whole string of length $\complexity{\abs{S} + 1}$ again. Furthermore, for any subrange of the set, a full fingerprint computation needs to be performed as well. Precomputing the fingerprints of all subranges requires a prohibitive amount of space. Every subrange corresponds to a substring of the string consisting of all items in $S$ in ascending order, so there are $\frac{\abs{S} \cdot (\abs{S} + 1)}{2} + 1 \in \complexity{n^2}$ many subranges in total.

The go-to approach for efficiently handling small changes to a set of totally ordered items is using a (balanced) search tree; we briefly state some definitions.

\begin{definition}
Let $U$ be a set and $\preceq$ a binary relation on $U$.
We call $\preceq$ a \defined{linear order on $U$} if it satisfies three properties:

  \begin{description}
    \item[anti-symmetry:] for all $x, y \in U$: if $x \preceq y$ and $y \preceq x$ then $x = y$
    \item[transitivity:] for all $x, y, z \in U$: if $x \preceq y$ and $y \preceq z$ then $x \preceq z$
    \item[linearity:] for all $x, y \in U$: $x \preceq y$ or $y \preceq x$
  \end{description}

If $\preceq$ is a linear order, we write $x \prec y$ to denote that $x \preceq y$ and $x \neq y$.
\end{definition}

\begin{definition}
Let $U$ be a set, $\preceq$ a linear order on $U$, and $V \subseteq U$. Let $T$ be a rooted directed tree with vertex set $V$.

Let $v \in V$, then $T_v$ denotes the subtree of $T$ with root $v$.

$T$ is a \defined{binary search tree on V} if for all inner vertices $v$ with left child $a$ and right child $b$: $a' \prec v$ for all $a' \in T_a$ and $v \prec b'$ for all $b' \in T_b$.

\end{definition}

\begin{definition}
Let $T = (V, E)$ be a binary search tree and $\epsilon \in \mathbb{R}_{> 0}$.
We call $T$ \defined{$\epsilon$-balanced} if $\textit{height}(T) \leq \ceil*{\epsilon \cdot log_2(|V|)}$.
Since the precise choice of $\epsilon$ will not matter for our complexity analyses, we will usually simply talk about \defined{balanced} trees.
\end{definition}

In the context of fingerprinting, balanced trees often take the form of Merkle trees~\cite{merkle1989certified} --- binary trees storing items in their leaves, in which each leaf vertex is labeled with the hash of the associated item, and inner vertices are labeled with the hash of the concatenation of the child labels. The root label serves as a fingerprint for the set of items stored in the leaves.

When inserting or removing an item, the number of labels that need updating is proportional to the length of of the path from the root to the item, so in a balanced tree of $n$ items $\complexity{\mathit{log}(n)}$. The problem with this approach however is that fingerprints are not unambiguous: there are different balanced search trees storing the same items set, and different tree arrangements result in different root labels.

Unfortunately it does not suffice to specify a particular balancing scheme, since different insertion orders of the same overall set of items can result in different trees, even when using the same balancing scheme. While this is sufficient for a setting in which only a single machine updates the set and all other machines apply updates in the same order, as assumed e.g. in~\cite{naor2000certificate}, we aim for a less restrictive setting in which the evolution of the local set does not influence synchronization.

An alternative would be to define exactly one valid tree shape for any set of items, but this precludes logarithmic time complexity of updates, as \cite{uniquerepresentation} shows that search, insertion and deletion in such trees take at least $\complexity{\sqrt{n}}$ time in the worst case.

We will inspect two options for circumventing this lower bound and to achieve logarithmic complexity: letting the fingerprint function abstract over the tree shape, downgrading it to an implementation detail, which we examine in \cref{group-fingerprints} to \cref{crypto}, or utilizing randomization to define a unique tree layout which allows logarithmic operations with high probability, which we examine in \cref{randomization}.

\section{Incremental Computations}
\label{group-fingerprints}

We now study a family of fingerprinting functions for sets that admit efficient incremental computation based on an auxiliary tree structure, but whose output does not depend on the exact shape of that tree. We first examine a general class of such functions, which reduce a finite set to a single value according to a monoid.

\begin{definition}
Let $M$ be a set, $\groupaddsym: M \times M \rightarrow U$, and $\neutraladd \in M$.

We call $(U, \groupaddsym, \neutraladd)$ a \defined{monoid} if it satisfies two properties:

  \begin{description}
    \item[associativity:] for all $x, y, z \in M$: $\groupadd{(\groupadd{x}{y})}{z} = \groupadd{x}{(\groupadd{y}{z})}$
    \item[neutral element:] for all $x \in M$: $\groupadd{\neutraladd}{x} = x = \groupadd{x}{\neutraladd}$.
  \end{description}
\end{definition}

\begin{definition}
\label{def-lift}
Let $U$ be a set, $\preceq$ a linear order on $U$, $\mathcal{M} \defeq (M, \groupaddsym, \neutraladd)$ a monoid, and $\fun{\f}{U}{M}$.

We \defined{lift $\f$ to finite sets via $\mathcal{M}$} to obtain $\partialfun{\lift{\f}{\mathcal{M}}}{\powerset{U}}{M}$ with:

\begin{align*}
\lift{\f}{\mathcal{M}}(\emptyset) &\defeq \mymathbb{0}\\
\lift{\f}{\mathcal{M}}(S) &\defeq \f(\set{\min(S)}) \oplus \lift{\f}{\mathcal{U}}(S \setminus \set{\min(S)})\\
\end{align*}

In other words, if $S = \set{s_0, s_1, \ldots, s_{\abs{S} - 1}}$ with $s_0 \prec s_1 \prec \ldots \prec s_{\abs{S} - 1}$, then $\lift{\f}{\mathcal{M}}(S) = \groupadd{\f(s_0)}{\groupadd{\f(s_1)}{\groupadd{\ldots}{\f(s_{\abs{S} - 1})}}}$.
\end{definition}

Functions of the form $\lift{\f}{\mathcal{M}}$ can be incrementally computed by using labeled binary search trees:

\begin{definition}
Let $U$ be a set, $S \subset U$ a finite set, $\preceq$ a linear order on $U$, $\mathcal{M} \defeq (M, \groupaddsym, \neutraladd)$ a monoid, $\fun{\f}{U}{M}$, and let $T$ be a binary search tree on $S$.

We define a \defined{labeling function} $\fun{\liftlabel{\f}{\mathcal{M}}}{S}{M}$:

  \[
   \liftlabel{\f}{\mathcal{M}}(v) \defeq \begin{cases}
\f(v), &  \text{for leaf $v$} \\
\groupadd{\liftlabel{\f}{\mathcal{M}}(c_{<})}{\f(v)} & \, \parbox[t]{.6\textwidth}{$v$ internal vertex with left child $c_{<}$\\and no right child}\\
\groupadd{\f(v)}{\liftlabel{\f}{\mathcal{M}}(c_{<})} & \, \parbox[t]{.6\textwidth}{$v$ internal vertex with right child $c_{>}$\\and no left child}\\
\groupadd{\liftlabel{\f}{\mathcal{M}}(c_{<})}{\groupadd{\f(v)}{\liftlabel{\f}{\mathcal{M}}(c_{>})}} & \, \parbox[t]{.6\textwidth}{$v$ internal vertex with left child $c_{<}$\\and right child $c_{>}$}
\end{cases}
  \]

See \cref{fig:fp-tree} for an example.
\end{definition}

\begin{figure*}
\begin{scaletikzpicturetowidth}{\textwidth}
\begin{tikzpicture}[scale=\tikzscale]
	\pgfdeclarelayer{background}
	\pgfdeclarelayer{foreground}
	\pgfsetlayers{background,main,foreground}

	\begin{pgfonlayer}{main}
		%vertices
		\node (vroot) at (0, 1) [aux] {\aux{\exampled}{
\groupadd{
    (\groupadd{\hexamplea}{\groupadd{\hexampleb}{\hexamplec)}}}
{\groupadd{\hexampled}
{(\groupadd{\hexamplee}{\groupadd{\hexamplef}{\hexampleg}})}
}
}};

		\node (v00) at (-4, -1) [aux] {\aux{\exampleb}{\groupadd{\hexamplea}{\groupadd{\hexampleb}{\hexamplec}}}};
		\node (v01) at (4, -1) [aux] {\aux{\examplef}{\groupadd{\hexamplee}{\groupadd{\hexamplef}{\hexampleg}}}};

                \node (v10) at (-6, -3) [aux] {\aux{\examplea}{\hexamplea}};
                \node (v11) at (-2, -3) [aux] {\aux{\examplec}{\hexamplec}};
                \node (v12) at (2, -3) [aux] {\aux{\examplee}{\hexamplee}};
                \node (v13) at (6, -3) [aux] {\aux{\exampleg}{\hexampleg}};
		%edges
                \draw (vroot) edge[edge] (v00);
                \draw (vroot) edge[edge] (v01);

		\draw (v00) edge[edge] (v10);
		\draw (v00) edge[edge] (v11);
		\draw (v01) edge[edge] (v12);
		\draw (v01) edge[edge] (v13);
	\end{pgfonlayer}
\end{tikzpicture}
\end{scaletikzpicturetowidth}

\caption{
A balanced search tree labeled by $\liftlabel{\h}{(M, \groupaddsym, \neutraladd)}$. For fingerprinting, $\h$ could be a hash function and $\groupaddsym$ the xor operation on fixed-width bitstrings.
}

\label{fig:fp-tree}
\end{figure*}

\begin{proposition}
Let $U$ be a set, $S \subset U$ a finite set, $\preceq$ a linear order on $U$, $\mathcal{M} \defeq (M, \groupaddsym, \neutraladd)$ a monoid, $\fun{\f}{U}{M}$, and let $T$ be a binary search tree on $S$ with root $r \in S$.

Then $\liftlabel{\f}{\mathcal{M}}(r) = \lift{\f}{\mathcal{M}}(S)$.

\begin{proof}
By induction on the number of vertices of $T$.

\textbf{IB:} If $r$ is a leaf, then $\abs{\V(T)} = 1$ and thus $\liftlabel{\f}{\mathcal{M}}(r) \overset{\text{def}}= \f(r) \overset{\text{def}}= \lift{\f}{\mathcal{M}}(\V(T)) = \lift{\f}{\mathcal{M}}(S)$.

\textbf{IH:} Let  $c_{<}$ and $c_{>}$ be vertices for which $\liftlabel{\f}{\mathcal{M}}(c_{<}) = \lift{\f}{\mathcal{M}}(\V(T_{c_{<}}))$ and $\liftlabel{\f}{\mathcal{M}}(c_{>}) = \lift{\f}{\mathcal{M}}(\V(T_{c_{>}}))$.

\textbf{IS:} If $r$ is an internal vertex with left child $c_{<}$ and right child $c_{>}$, then:

\begin{align*}
 \liftlabel{\f}{\mathcal{M}}(r) &\overset{\text{def}}= \groupadd{\liftlabel{\f}{\mathcal{M}}(c_{<})}{\groupadd{\f(r)}{\liftlabel{\f}{\mathcal{M}}(c_{>})}}\\
&\overset{\text{IH}}= \groupadd{\lift{\f}{\mathcal{M}}(\V(T_{c_{<}}))}{\groupadd{\f(r)}{\lift{\f}{\mathcal{M}}(\V(T_{c_{>}}))}}\\
& \overset{\text{def}}= \lift{\f}{\mathcal{M}}(\V(T))\\
&= \lift{\f}{\mathcal{M}}(S)\\
\end{align*}

The cases for internal vertices with exactly one child follow analogously.
\end{proof}
\end{proposition}

This correspondence can be used to incrementally compute $\lift{\f}{\mathcal{M}}(S)$: Initially, a labeled search tree storing the items in $S$ is constructed. $\lift{\f}{\mathcal{M}}(S)$ is the root label. When an item is inserted or removed, only the labels on the path from the root to the point of modification require recomputation, so only a logarithmic number of operations is performed if a self-balancing tree is used.

Note that the exact shape of the tree determines the grouping of how to apply $\groupaddsym$, but by associativity all groupings yield the same result. All trees storing the same set have the same root label.

If $U$ is small enough that space usage of $\complexity{\abs{U}}$ is acceptable, an implicit tree representation such as a binary indexed tree (Fenwick tree)~\cite{fenwick1994new} can be used. In that case, array positions that correspond to some $u \in U \setminus S$ are simply filled with a dummy value whose hash is defined to be $\neutraladd$.

\subsection{Subsets}
\label{subsets-associative}

In addition to incremental computation of the fingerprint of a given set, the reconciliation protocol also requires the efficient computation of the fingerprints of arbitrary ranges of the given set. We first fix some terminology and notation:

\begin{definition}
\label{def-range}
Let $S \subseteq U$, $\preceq$ a linear order over $U$, and $x, y \in U$.

The \defined{range from $x$ to $y$ in $S$}, denoted by $\range{x}{y}{S}$, is the set $\set{s \in S \mid x \preceq s \prec y}$ if $x \prec y$, and $S \setminus \range{y}{x}{S}$ if $y \preceq x$. We call $x$ the \defined{lower boundary} and $y$ the \defined{upper boundary} of the range (even if $y \preceq x$).

Note that the upper boundary is excluded from the range, so in particular $\range{x}{x}{S} = S$.
\end{definition}

In the remainder of this section, we assume that for all ranges $\range{x}{y}{S}$ we have $x \prec y$. If that is not the case, all computations can be performed for the sets $\set{s \in S \mid x \preceq s}$ and $\set{s \in S \mid s \prec y}$ which partition $\range{x}{y}{S}$ if $y \preceq x$, and the resulting values can be combined via $\groupaddsym$ to obtain the desired result.

Given a balanced search tree $T$ with root $r$ for a set $S$ that is labeled by $\liftlabel{\f}{\mathcal{M}}$, we can compute $\lift{\f}{\mathcal{M}}(\range{x}{y}{S})$ in logarithmic time. Intuitively, one traces paths in $T$ to both $x$ and $y$, and then the result is the sum over all vertices ``in the area between'' these paths. For every vertex on the traced paths, the label of the ``inner'' child vertex summarizes multiple vertices within the area. Summing over all these children yields the value corresponding to the whole inner area. Since the length of the delimiting paths is logarithmic, overall only a logarithmic number of labels needs to be added up.

We now give a precise definition of this algorithm, defining it via Haskell~98~\cite{jones2003haskell} code, not necessarily because it is executable, but mostly to have a typed mathematic notation. For clarity of presentation, only complete binary trees are considered. Arbitrary binary trees can also have inner nodes with exactly one child. These can be handled with almost the same algorithm by acting as if these nodes had a second child which contributes \texttt{emptyFingerprint} to any application of \texttt{combine}.

We start by specifying the types involved in the computation, being slightly more general than currently necessary so that the algorithm can later be reused for certain non-monoidal fingerprinting schemes:

\begin{minted}[mathescape]{haskell}
-- $\mathtt{U}$ is the type of items, $\mathtt{D}$ the codomain of the hash function.
hash :: U -> D

-- For now, the set of fingerprints is identical to the set of hashes.
type Fingerprint = D

-- The fingerprint of the empty set, for now the neutral element.
emptyFp :: Fingerprint
emptyFp = 0

-- Combine three fingerprints into a new fingerprint.
combine :: Fingerprint -> Fingerprint -> Fingerprint -> Fingerprint
combine a b c = a + b + c

-- Convert a hash to a fingerprint, for now the identity function.
liftFp :: D -> Fingerprint
liftFp d = d

-- A node is either a leaf or an inner vertex.
data Node = Leaf U D | Inner Node U Node D

-- Extract the label of a node.
label :: Node -> D
label Leaf _ fp      = fp
label Inner _ _ _ fp = fp
\end{minted}

The algorithm proceeds by first finding the vertex $v$ with the smallest distance to the root such that $x \preceq v \prec y$. This might be $r$ itself.

\begin{minted}[mathescape]{haskell}
-- Find the node within $\range{x}{y}{S}$ that is closest to the root,
-- assuming $x \prec y$.
findInitial :: Node -> U -> U -> Maybe Node
findInitial (Leaf v _) x y
    | x <= v && v < y = Just v
    | otherwise       = Nothing
findInitial (Inner l v r _) x y
    | v < x           = findInitial r x y
    | v >= y          = findInitial l x y
    | otherwise       = Just v
\end{minted}

If there is no such $v$, then $\range{x}{y}{S} = \emptyset$ and thus $\lift{\f}{\mathcal{M}}(\range{x}{y}{S}) = \neutraladd$. If however there is such a $v$, then all vertices of $T$ which are not vertices in $T_v$ are either greater than or equal to $y$ if $v \prec x$, or strictly less than $x$ if $x \prec v$; in either case they do not influence $\lift{\f}{\mathcal{M}}(\range{x}{y}{S})$.

If $v$ is a leaf, $\range{x}{y}{S} = \set{v}$ and thus $\lift{\f}{\mathcal{M}}(\range{x}{y}{S}) = \f(v)$. Otherwise, let $c_{<}$ be the left child of $v$ and let $c_{>}$ be the right child. Since all vertices in $T_{c_{>}}$ are greater than $v$, they are in particular greater than $x$. Analogously all vertices in $T_{c_{<}}$ are less than $y$.

Keeping in mind that $\range{x}{\max(\V(T_{c_{<}}))}{\V(T_{c_{<}})}$ simply denotes the set of vertices in $T_{c_{<}}$ that are strictly greater than $x$, and analogously $\range{\min(\V(T_{c_{>}}))}{y}{\V(T_{c_{>}})}$ denotes the vertices of $T_{c_{>}}$ that are less than or equal to $y$, we have:

\begin{align*}
\range{x}{y}{S} &= \disjointunion{\range{x}{\max(\V(T_{c_{<}}))}{\V(T_{c_{<}})}}{\disjointunion{\set{v}}{\range{\min(\V(T_{c_{>}}))}{y}{\V(T_{c_{>}})}}}\\
&\mathrm{implying}\\
\lift{\f}{\mathcal{M}}(\range{x}{y}{S}) &= \groupadd{\lift{\f}{\mathcal{M}}(\range{x}{\max(\V(T_{c_{<}}))}{\V(T_{c_{<}})})}{\groupadd{\f(v)}{\lift{\f}{\mathcal{M}}(\range{\min(\V(T_{c_{>}}))}{y}{\V(T_{c_{>}})})}}\\
\end{align*}

\begin{minted}[mathescape]{haskell}
-- Compute the fingerprint over all items stored in $v$
-- within the range $\range{x}{y}{S}$, assuming $x \prec y$.
rangeFingerprint :: Node -> U -> U -> Fingerprint
rangeFingerprint v x y = case findInitial v x y of
    Nothing               -> emptyFp
    Just (Leaf _ fp)      -> liftFp 'sfp
    Just (Inner l v' r _) -> combine
                                (accGeq l x) -- $\lift{\f}{\mathcal{M}}({u \in \V(T_l) \mid x \preceq u})$
                                (liftFp (hash v')) -- $\f(v')$
                                (accLt r y)  -- $\lift{\f}{\mathcal{M}}({u \in \V(T_l) \mid u \prec y})$
\end{minted}

$\lift{\f}{\mathcal{M}}(\range{x}{\max(\V(T_{c_{<}}))}{\V(T_{c_{<}})})$ can be computed by traversing from $c_{<}$ to $x$, summing over those vertices along the path that are greater or equal to $x$, as well as the right children of all vertices along the path. Correctness follows from a rather technical but straightforward induction we omit.

\begin{minted}[mathescape]{haskell}
-- Sum up the fingerprints of all items in the given tree
-- which are greater than or equal to $x$.
accGeq :: Node -> U -> Fingerprint
accGeq (Leaf v fp) x
    | v < x     = emptyFp
    | otherwise = liftFp fp
accGeq (Inner l v r _) x
    | v < x     = accGeq r x
    | otherwise = combine (accGeq l x) (liftFp (hash v)) (label r)
\end{minted}

Analogously $\lift{\f}{\mathcal{M}}(\range{\min(\V(T_{c_{>}}))}{y}{\V(T_{c_{>}})})$ can be computed by traversing from $c_{>}$ to $y$, summing over those vertices along the path that are strictly less than $y$, as well as the left children of all vertices along the path.

\begin{minted}[mathescape]{haskell}
-- Sum up the fingerprints of all items in the given tree
-- which are strictly less than $y$.
accLt :: Node -> U -> Fingerprint
accLt (Leaf v fp) y
    | v >= y    = emptyFp
    | otherwise = liftFp fp
accLt (Inner l v r _) y
    | v >= y    = accLt l y
    | otherwise = combine (label l) + (liftFp (hash v)) + (accLt r y)
\end{minted}

A simplified approach to these computations can be taken if $M$ has efficiently computable inverses with respect to $\groupaddsym$, i.e. if $(M, \groupaddsym, \neutraladd)$ is a group.

\begin{definition}
Let $(M, \groupaddsym, \neutraladd)$ be a monoid.
We call it a \defined{group} if for all $x \in M$ there exists $y \in M$ such that $\groupadd{x}{y} = \neutraladd$.
This $y$ is necessarily unique and denoted by $\inverseadd{x}$.
For $x, y \in M$ we write $\groupsubtract{x}{y}$ as a shorthand for $\groupadd{x}{\inverseadd{y}}$.
\end{definition}

Observe that $\range{x}{y}{S} = \range{\min(S)}{y}{S} \setminus \range{\min(S)}{x}{S}$, and thus also $\lift{\f}{\mathcal{M}}(\range{x}{y}{S}) = \groupadd{\inverseadd{\lift{\f}{\mathcal{M}}(\range{\min(S)}{x}{S})}}{\lift{\f}{\mathcal{M}}(\range{\min(S)}{y}{S})}$. We can thus simplify \texttt{rangeFingerprint}, requiring neither \texttt{findInitial} nor \texttt{accGeq}:

\begin{minted}{haskell}
rangeFingerprintGroup :: Node -> U -> U -> Fingerprint
rangeFingerprintGroup v x y = -(accLt v x) + (accLt v y)
\end{minted}

\subsubsection{Complexity}

The worst-case running time occurs when the vertex $v$ with the smallest distance to $r$ such that $x \preceq v \prec y$ is $r$ itself, since then both \texttt{sumGeq} and \texttt{sumLt} perform a traversal to a leaf of maximal length. Since $T$ is balanced, only a logarithmic number of recursive calls is executed. Assuming $\f$ can be computed in $\complexity{1}$, the resulting time complexity is in $\complexity{\log(\abs{S})}$.

Neither \texttt{accLt} nor \texttt{accGeq} are tail-recursive, but they can be rewritten into a tail-recursive form via the standard technique of adding an accumulator argument:

\begin{minted}[mathescape]{haskell}
accGeq :: Node -> U -> Fingerprint
accGeq v x = accGeq' emptyFp v x

accGeq' :: Fingerprint -> Node -> U -> Fingerprint
accGeq' acc (Leaf v fp) x
    | v < x     = acc
    | otherwise = combine emptyFp (liftFp fp) acc
accGeq' acc (Inner l v r _) x
    | v < x     = accGeq' acc r x
    | otherwise = accGeq' (
                      combine (liftFp (hash v)) (label r) acc
                  ) l x

accLt :: Node -> U -> Fingerprint
accLt v y = accLt' emptyFp v y

accLt' :: Fingerprint -> Node -> U -> Fingerprint
accLt' acc (Leaf v fp) y
    | v >= y    = acc
    | otherwise = combine acc (liftFp fp) emptyFp
accLt' acc (Inner l v r _) y
    | v >= y    = accLt' acc l y
    | otherwise = accLt' (
                      combine acc (label l) + (liftFp (hash v))
                  ) r y
\end{minted}

The space complexity of computing the fingerprint for a range is thus $\complexity{1}$. Note however that the tail-recursive forms require associativity to compute the same fingerprints as the original versions.

As a closing remark on these computations, we want to point out that associativity not only allows us to abstract over differently shaped binary search trees, but also results in trees of higher degree leading to the same fingerprints. Binary trees simplify the presentation and are theoretically optimal, but on actual hardware traversing pointers is expensive whereas loading sequential memory is cheap, which becomes even more pronounced when the tree resides on secondary storage.

Optimized tree implementations thus have higher branching and store multiple items per vertex, a typical example being B-trees~\cite{bayer2002organization}. Our fingerprinting scheme admits such optimized implementations, in particular, different nodes that want to synchronize data can choose their implementation techniques fully independently and yet maintain compatibility.

\subsection{Bulk Fingerprint Computation}
\label{bulk}

The synchronization protocol partitions item sets into disjoint ranges. When computing the fingerprints of $k$ such ranges using the previous algorithms, the overall time complexity is in $\complexity{k \cdot \log(\abs{S})}$. The reason why these algorithms lead to a worse complexity than the overall $\complexity{\abs{S}}$ vertices and edges in the tree, is that there is some amount of redundancy in these independent computations. For example, every single computation touches the root node, roughly half of them touch each of the root's children, etc.

We can bound the complexity of multiple fingerprint computation for disjoint ranges by $\complexity{\abs{S}}$ by performing a more efficient tree traversal which visits any given vertex at most twice. This is achieved by starting at the lower boundary of the first range, then tracing the (unique) path to the upper boundary of the first range, which is also the lower boundary of the second range, so from here the path to the upper boundary of the second range can be traced, and so on.

Thus we consider a traversal algorithm that traces the shortest path from some vertex $x$ to another vertex $y$ with $x \preceq y$. We express the traversal as a function that takes as arguments some vertex $y$ to search for and a starting vertex $x$, and that then returns either the next vertex on the shortest path from $x$ to the least vertex $z$ such that $y \preceq z$, or $\nil$ if no such $z$ exists or the traversal has finished, i.e. $x = z$.

Since we use the traversal for the computation of fingerprints over successive ranges, the tree is always traversed from a lesser to a strictly greater node. During a traversal, the function can be called with $y \preceq x$, but only if $x$ lies on the shortest path from some $w \prec y$ to $z$. The definition we give does not correctly handle inputs for which this does not hold, it cannot perform backward traversals.

If the function is to trace of the shortest path in $\complexity{1}$ space, the decision which vertex to move to must be based on local information only. A vertex $x$ however does not store enough information to determine whether the target vertex $y$ can be reached by a downward traversal starting at $x$ or whether one first needs to find a common ancestor of $x$ and $y$. To enable this decision, we store in each vertex a reference to its greatest descendent.

To allow upward traversal in the tree in $\complexity{1}$ space, we store in each vertex a reference to its parent, if one exists. As usual, we restrict the presentation of the algorithm to complete binary trees.

\begin{minted}[mathescape]{haskell}
-- Second-to-last member of inner nodes is a pointer to the greatest
-- descendent, last member of all nodes is the parent pointer.
data Node = Leaf U D (Maybe Node) | Inner Node U Node D Node (Maybe Node)

value :: Node -> U
value Leaf v _ _ = v
value Inner _ v _ _ _ _ _ = v

greatestDescendent :: Node -> Node
greatestDescendent v@(Leaf _ _ _) = v
greatestDescendent Inner _ _ _ _ _ g _ = g

-- Return the next vertex on the shortest path
-- from $\mathtt{x}$ to the least vertex $\mathtt{z}$ with $\mathtt{y} \preceq \mathtt{z}$,
-- assuming there exists $\mathtt{w} \prec \mathtt{y}$ such that $\mathtt{x}$ lies
-- on the shortest path from $\mathtt{w}$ to $\mathtt{z}$.
next :: Node -> U -> Maybe Node
next (Leaf _ _ p) y
    | x < y  = p
    | x >= y = Nothing
next (Inner l x r _ g p) y
    | (value g) < y = p
    | y < x && ((value (greatestDescendent l)) >= y)  = Just l
    | y > x  = Just r
    | otherwise = Nothing -- traversal done
\end{minted}

Since paths in trees are unique, this traversal touches the same vertices as \texttt{rangeFingerprint}, except those visited during the execution of \texttt{findInitial}. The fingerprint can thus also be computed during the traversal, by maintaining a single bit of state indicating whether the traversal is in the upward or downward phase:

\begin{minted}[mathescape]{haskell}
data Phase = Up | Down

isPeak :: Node -> U -> Bool
isPeak src@(Leaf _ _ p) y = (next src y) == p
isPeak src@(Inner _ _ _ _ _ p) y = (next src y) == p

-- Compute the fingerprint for the range $\range{x}{y}{S}$,
-- and also return the last vertex of the traversal.
-- The accumulator must initially be $\mathtt{emptyFp}$, the phase $\mathtt{Up}$,
-- and the starting vertex of the traversal $\mathtt{x}$.
computeFingerprint :: D -> Phase -> Node -> U -> U -> (D, Node)
-- Phase $\mathtt{Up}$ correspondends to computing $\mathtt{accGeq}$.
computeFingerprint acc Up src@(Leaf v fp _) x y =
     case (next src y) of
        Nothing
            | v < x = (acc, src)
            | otherwise = (combine acc (liftFp fp) emptyFp, src)
        Just n
            | v < x = computeFingerprint acc Up n x y
            | otherwise = computeFingerprint
                (combine acc (liftFp fp) emptyFp) Up n x y
computeFingerprint acc Up src@(Inner _ v r _ _ _) x y =
    case (next src y) of
        Nothing
            | v < x = (acc, src)
            | otherwise = (combine acc (liftFp (hash v)) emptyFp, src)
        Just n
            | isPeak src y = computeFingerprint
                (combine acc (liftFp (hash v)) emptyFp) Down n x y
            | otherwise && v < x = computeFingerprint acc Up n x y
            | otherwise = computeFingerprint
                (combine acc (liftFp (hash v)) (label r)) Up n x y
-- Phase $\mathtt{Down}$ correspondends to computing $\mathtt{accLt}$.
computeFingerprint acc Down src@(Leaf v fp _) x y =
    | v >= y = (acc, src)
    | otherwise = (combine acc (liftFp fp) emptyFp, src)
computeFingerprint acc Down src@(Inner l v _ _ _ _) x y =
    case (next src y) of
        Nothing
            | v >= y = (acc, src)
            | otherwise = ((combine
                    acc
                    (liftFp (label l))
                    (liftFp (hash v))
                ), src)
        Just n
            | v >= y -> computeFingerprint acc Down n x y
            | otherwise -> computeFingerprint
                (combine
                    acc
                    (liftFp (label l))
                    (liftFp (hash v))
                ) Down n x y
\end{minted}

By computing the fingerprints of successive ranges via this function, using the output vertex as the starting point for the next traversal, the fingerprints are computed while traversing any edge at most two times (once during a downward traversal, and once during an upward traversal). The overall time complexity for $k$ ranges is thus in $\complexity{\min(k \cdot \log(\abs{S}), \abs{S})}$. Since the function is tail recursive, the space overhead is in $\complexity{1}$.

\subsection{Differences to Homomorphic Hashing}

The construction of lifting a function to sets resembles that of multiset homomorphic hash functions, introduced in \cite{clarke2003incremental}, with further instantiations given in \cite{cathalo2009comparing} and \cite{maitin2017elliptic}. We briefly define multiset homomorphic hashing in order to point out salient differences to our construction.

\begin{definition}
Let $\mathcal{U}_0 \defeq (U_0, \groupaddsym_0, \neutraladd_0)$ and $\mathcal{U}_1 \defeq (U_1, \groupaddsym_1, \neutraladd_1)$ be monoids and let $\fun{f}{U_0}{U_1}$.

We call $\f$ a \defined{monoid homomorphism from $\mathcal{U}_0$ to $\mathcal{U}_1$} if it satisfies two properties:

\begin{description}
  \item[preserves operation:] for all $x, y \in U_0$: $\f(x \groupaddsym_0 y) = \f(x) \groupaddsym_1 \f(y)$
  \item[preserves neutral element:] $f(\neutraladd_0) = \neutraladd_1$
\end{description}

The second property directly follows from the first one by considering the cases $x \defeq \neutraladd_0$ and $y \defeq \neutraladd_0$.
\end{definition}

\begin{definition}
Let $\mathcal{S} \defeq (\N^U, \cup, \emptyset)$ be the monoid of multisets over the universe $U$ under union, $\mathcal{M} \defeq (M, \groupaddsym, \neutraladd)$ a monoid, and $\fun{\f}{\N^U}{M}$.

We call $\f$ a \defined{multiset homomorphic hash function} if $\f$ is a hash function and a monoid homomorphism from $\mathcal{S}$ to $\mathcal{M}$.
\end{definition}

A multiset homomorphic hash function allows to incrementally maintain the hash of a multiset under insertions using only $\complexity{1}$ storage space, by storing the hash of the multiset and combining it with the hash of any inserted item. This can be generalized to support deletions as well: by allowing negative multiplicities for multiset items, multisets form a group under union. If the hash function is a group homomorphism, the hash of the multiset can be updated on deletion by subtracting the hash of the deleted item.

This is more efficient than our approach, which requires $\complexity{n}$ space for the tree structure. This efficiency comes however at the cost of a loss in flexibility. As multiset union is commutative, so must be the homomorphic monoid/group of fingerprints. Since our protocols require the ability to compute fingerprints over arbitrary ranges and we need the tree of size $\complexity{n}$ to do so in any case, homomorphic hashing would not yield any space usage benefits, and we thus choose the approach that does not require commutativity.

We additionally want to explicitly work with regular sets, not multisets. The reason why homomorphic hashes operate on multisets is that they admit more natural homomorphisms under union. For example, taking the size of a multiset is a homomorphism into the additive monoid on the natural numbers, whereas this is not true for regular sets, as e.g. $\abs{\set{x, y} \cup \set{y, z}} \neq \abs{\set{x, y}} + \abs{\set{y, z}}$. Set union is however well behaved with respect to taking the size of the sets if it is a disjoint union, i.e. no item occurs in both sets.

For an algebraic-flavored characterization of our fingerprint functions, we thus need to account for items occurring on both sides of the union. This alone still would not change that the resulting concept would necessarily require a commutative monoid, so we must also restrict possible inputs to be sorted. This leads to the following definition:

\begin{definition}
Let $U$ be a set, $\preceq$ a linear order on $U$, $\mathcal{M} \defeq (M, \groupaddsym, \neutraladd)$ a monoid, and $\partialfun{\f}{\powerset{U}}{M}$ a partial function mapping all finite subsets of $U$ into $M$.

We call $\f$ a \defined{\somewhatmorphism} if for all finite sets $S_0, S_1 \in \powerset{U}$ such that $\max(S_0) \prec \min(S_1)$: $\f(S_0 \cup S_1) = \groupadd{\f(S_0)}{\f(S_1)}$.
\end{definition}

This definition captures exactly the functions of form $\lift{\f}{\mathcal{M}}$, as shown in the following propositions:

\begin{proposition}
Let $U$ be a set, $\preceq$ a linear order on $U$, $\mathcal{M} \defeq (M, \groupaddsym, \neutraladd)$ a monoid, and $\fun{\f}{U}{M}$.

Then $\lift{\f}{\mathcal{M}}$ is a \somewhatmorphism.

\begin{proof}
Let $S_0, S_1 \in \powerset{U}$ be finite sets such that $\max(S_0) \prec \min(S_1)$. Then:

\begin{align*}
\lift{\f}{\mathcal{M}}(S_0 \cup S_1) &= \biggroupadd_{\substack{s_i \in S_0 \cup S_1,\\ \text{ascending}}} \f(s_i)\\
&= \biggroupadd_{\substack{s_i \in S_0,\\ \text{ascending}}} \f(s_i) \groupaddsym \biggroupadd_{\substack{s_i \in S_1,\\ \text{ascending}}} \f(s_i)\\
&= \lift{\f}{\mathcal{M}}(S_0) \groupaddsym \lift{\f}{\mathcal{M}}(S_1)\\
\end{align*}
\end{proof}
\end{proposition}

\begin{proposition}
Let $U$ be a set, $\preceq$ a linear order on $U$, $\mathcal{M} \defeq (M, \groupaddsym, \neutraladd)$ a monoid, and $\partialfun{\g}{\powerset{U}}{M}$ a \somewhatmorphism.

Then there exists $\fun{\f}{U}{M}$ such that $\g = \lift{\f}{\mathcal{M}}$.

\begin{proof}
Define $\fun{\f}{U}{M}$ as $\f(u) \defeq \g(\set{u})$. We show by induction on the size of $S \subseteq U$ that $\g(S) = \lift{f}{\mathcal{M}}(S)$.

\vspace{10pt}

\textbf{IB:} If $S = \emptyset$, then $\g(S) = \neutraladd = \lift{\f}{\mathcal{M}}(S)$. Suppose that $\g(\emptyset) \neq \neutraladd$, this would contradict the fact that for all $x \in U$ we have $\g(\set{x}) = \g(\set{x}) \groupaddsym \g(\emptyset) = \g(\emptyset) \groupaddsym \g(\set{x})$, which holds because $\set{x} = \set{x} \cup \emptyset = \emptyset \cup \set{x}$ and $\g$ is a \somewhatmorphism.

If $S = \set{x}$, then $\g(S) = \f(x) = \lift{\f}{\mathcal{M}}(S)$.

\textbf{IH:} For all sets $T$ with $\abs{T} = n$ it holds that $\g(T) = \lift{\f}{\mathcal{M}}(T)$.

\textbf{IS:} Let $S \subseteq U$ with $\abs{S} = n + 1$, then:

\begin{align*}
\g(S) &= \g(\set{\min(S)}) \groupaddsym \g(S \setminus \set{\min(S)})\\
&\overset{\text{IH}}= \g(\set{\min(S)}) \groupaddsym \lift{\f}{\mathcal{M}}(S \setminus \set{\min(S)})\\
&= \f(\min(S)) \groupaddsym \lift{\f}{\mathcal{M}}(S \setminus \set{\min(S)})\\
&= \lift{\f}{\mathcal{M}}(S)\\
\end{align*}

\vspace{10pt}
As $\g$ is only defined over finite inputs, we thus have $\g = \lift{\f}{\mathcal{M}}$.
\end{proof}
\end{proposition}

\section{Monoidal Fingerprints}
\label{collisions}

Now that we have characterized a family of functions that admit efficient recomputation in response to changes to the underlying set as well as efficient computation for ranges within the set, the remaining task is to find such functions which are also suitable fingerprints. This consists of deciding on the monoid of fingerprints, and choosing the mapping from items to monoid elements.

As the fingerprint of a singleton set $\lift{\f}{\mathcal{M}}(\set{u})$ is equal to $\f(u)$, $\f$ must itself already be a hash function. Typical hash functions map values to bit strings of a certain length, i.e. the codomain is $\set{0, 1}^k$ for some $k \in \N$. We will thus consider monoids whose elements can be represented by such bit strings.

A natural choice of the monoid universe is then $\range{0}{2^k}{\N}$, some simple monoidical operations on this universe include bitwise xor, addition modulo $2^k$, and multiplication modulo $2^k$. In the following, addition and multiplication will always be implicitly taken modulo $2^k$. Note that $\xor$ and addition also admit inverses, so the slightly simplified computation of fingerprints can be used.

Of these three options, multiplication is the least suitable, because multiplying any number by $0$ yields $0$. Consequently for every set containing an item $u$ with $\f(u) = 0$ the fingerprint of the set would also be $0$, which clearly violates the criterion that all possible values for fingerprints occur with equal probability.

Addition and xor however are particularly well-behaved in that regard, as they form finite commutative groups:

\begin{proposition}
Let $\mathcal{G} \defeq (G, \groupaddsym, \neutraladd)$ be a finite commutative group, i.e. a group with a finite universe such that for all $x, y \in G$: $\groupadd{x}{y} = \groupadd{y}{x}$. Let $U$ be a set and let $\fun{f}{U}{G}$ be a hash function.

Then $\lift{\f}{\mathcal{G}}$ is a hash function as well.

\begin{proof}
We first show that for randomly chosen $x, y, z \in G$ the probability that $\groupadd{x}{y} = z$ is $\frac{1}{\abs{G}}$.

For $x, z \in G$ there is $y \in G$ such that $\groupadd{x}{y} = z$, namely $y \defeq \groupsubtract{z}{x}$ (because $\groupadd{x}{\groupsubtract{z}{x}} \overset{\text{commutativity}}= \groupadd{\groupadd{x}{\inverseadd{x}}}{z} = z$). As $G$ is finite, this $y$ has to be unique, since otherwise there would not be enough elements left that can be added to $x$ to result in all of the $\abs{G} - 1$ possible remaining $z'$. Thus for any fixed $x, z$ the probability that a randomly chosen $y$ satisfies $\groupadd{x}{y} = z$ is $\frac{1}{\abs{G}}$.

Computing $\lift{\f}{\mathcal{G}}(S)$ consists of repeatedly adding group elements which by themselves are distributed uniformly at random if $S$ was chosen randomly and $\f$ is a high quality hash function. Thus the accumulated value after every step is any given $z \in G$ with probability $\frac{1}{\abs{G}}$, as is in particular the probability for the final result being equal to $z$.
\end{proof}

A more formal proof for xor specifically is given in section 6.2 of \cite{maziarz2021hashing}.
\end{proposition}

By the same argument, knowing the fingerprint for some set does not provide any information about the fingerprints for sets that differ by even only a single value. In conclusion, high quality fingerprints can be achieved by choosing any transitive and commutative operation for the monoid, for example xor or addition modulo $2^k$, as long as values are mapped into the monoid with a high quality hash function.

While multiplication does not form a group when performed on the numbers in $\range{0}{2^k}{\N}$, there still are groups based on multiplication modulo some number, e.g. $\Z_n^\ast$, the group yielded by multiplication modulo $n$ on the set $\set{x \in \range{0}{n}{\N} \mid \text{$x$ is coprime to $n$}}$. In the following, when talking about multiplication, we will assume that the universe is chosen such that multiplication forms a group.

\section{Cryptographically Secure Fingerprints}
\label{crypto}

In the protocols for synchronizing data structures, fingerprints of sets are used for probabilistic equality checking: sets with equal fingerprints are assumed to be equal. Synchronization can thus become faulty if it involves unequal sets with equal fingerprints. If the universe of possible fingerprints is chosen large enough, and the distribution of fingerprints of randomly chosen sets is random within that universe, the probability for this to occur becomes negligible.

Random distribution of input sets is however a very strong assumption. What if a malicious party can influence the sets to be fingerprinted, with the goal of causing fingerprint collisions and consequently triggering faulty behavior of the system? Cryptographically secure fingerprints are an answer to this problem, being chosen such that it is computationally infeasible for an adversary to find inputs that lead to faulty synchronization.

\subsection{General Considerations}
\label{crypto-general}

A typical definition of cryptographically secure hash functions is the following~\cite{menezes2018handbook}:

\begin{definition}
A \defined{secure hash function} is a hash function $\fun{\h}{U}{D}$ that satisfies three additional properties:

\begin{description}
  \item[pre-image resistance:] Given $d \in D$, it is computationally infeasible to find a $u \in U$ such that $\h(u) = d$.
  \item[second pre-image resistance:] Given $u \in U$, it is computationally infeasible to find a $u' \in U, u' \neq u$ such that $\h(u) = \h(u')$.
  \item[collision resistance:] It is computationally infeasible to find $u, v \in U, u ~= v$ such that $\h(u) = \h(v)$.
\end{description}
\end{definition}

Pre-image resistance has no influence on the vulnerability of the protocol to malicious actors, so all of the following discussion will focus on collision resistance only.

Since $\lift{\f}{\mathcal{M}}(\set{u}) = \f(u)$, $\f$ must necessarily be collision resistant if $\lift{\f}{\mathcal{M}}$ is to be collision resistant. This alone is unfortunately not sufficient, we will see a specific counterexamples in \cref{{specific-monoids}}. Choosing a secure hash function $\f$ always comes with a performance cost, insecure hash functions usually take less time and less space to compute. If the synchronization protocol is only being run in a trusted environment, an insecure hash function might be preferable.

Whether a hash function is secure is not a binary dichotomy, but depends on what is considered ``feasible'' for an adversary. Greater security can usually be obtained at the cost of longer digests and longer computation times. Before presenting options for secure hash functions, we thus examine the impact of hash collisions first.

We can generally distinguish between malicious actors in two different positions: those who can actively impact the contents of the data structure to be synchronized, and those who passively relay updates and need to search for a collision within the available data. As a set of size $n$ has $2^n$ subsets, if fingerprints are bit strings of length $k$, then by the pigeonhole principle a fingerprint collision can be found within any set of size at least $k + 1$.

An attack against the fingerprinting scheme by an active adversary can involve computing many fingerprints and adding the required items to the set once a collision has been found. Such an attack is not usable by the passive adversary. We will primarily focus on discussing active adversaries, as they are strictly more powerful than passive ones. Yet it should be kept in mind that passive adversaries can be more common in certain settings, particularly in peer-to-peer systems: if a node is interested in synchronizing a data structure, it probably trusts the source of the data, otherwise it would have little reason for expending resources on synchronization. The data may however be synchronized not with the original source but with completely untrusted nodes. Additionally, an active adversary that does not want to risk detection by adding suspicious items to the data structure is restricted to the operations of a passive adversary.

Fingerprint collisions result in parts of the data structure not being synchronized, so information is being withheld from one or both of the synchronizing nodes. When a malicious node synchronizes with an honest one, the malicious node can withhold arbitrary information by simply pretending not to have certain data, which does not require finding collisions at all.

So the cases in which a malicious node can do actual damage by finding a collision are those where it supplies data to two different, honest nodes such that these two nodes perform faulty synchronization amongst each other. Specifically: let $\mathcal{M}$ be a malicious node, $\mathcal{A}$ and $\mathcal{B}$ be honest nodes, then a successful attack consists of $\mathcal{M}$ crafting sets $X_A, X_B$ and sending these to $\mathcal{A}$ and $\mathcal{B}$ respectively, so that when $\mathcal{A}$ and $\mathcal{B}$ then run the synchronization protocol, they end up with different data structures. A passive adversary does not craft $X_A, X_B$ but must find them as subsets of some set $X$ supplied by an honest node. As we assume the underlying hash function $\f$ to be secure, at least one non-singleton set has to be involved in a collision.

There are some qualitative arguments that even if an adversary finds a fingerprint collision, the impact is rather low. Let $S_A \subseteq X_A$ and $S_B \subseteq X_B$ be nonequal sets with the same fingerprint. To have any impact on the correctness of a particular protocol run, their two fingerprints need to actually be compared during that run. For that to happen, they need to be of the form $\range{x_A}{y_A}{X_A}$ and $\range{x_B}{y_B}{X_B}$ respectively. The fingerprints of these ranges can then be compared if one of the nodes sends the fingerprint for the range $\range{\min(x_A, x_B)}{\max(y_A, y_B)}{X_i}$. That alone is still not sufficient, as any item within that range that is not part of the sets would change the fingerprint. So the two ranges actually need to be of the form $\range{\min(x_A, x_B)}{\max(y_A, y_B)}{X_A}$ and $\range{\min(x_A, x_B)}{\max(y_A, y_B)}{X_B}$, or simplified: there have to be $x, y \in U$ such that $S_A = \range{x}{y}{X_A}$ and $S_B = \range{x}{y}{X_B}$.

\label{why-random-boundaries}
If the adversary has found such sets, this provides still no guarantee that the range $\range{x}{y}{X_i}$ is being compared during the synchronization session of $\mathcal{A}$ and $\mathcal{B}$. For a set containing $n$ items, there are $n^2 - (n - 1) \in \complexity{n^2}$ distinct ranges, but only $\complexity{n}$ are compared in a given protocol run, since the worst-case message complexity is $\complexity{n}$ (see \cref{communication-complexity}). These numbers should only be considered as rough guidelines, they gloss over details such as the fact that there are more ranges containing roughly $\frac{n}{2}$ items than there are ranges containing almost all or almost no items. Yet they demonstrate that finding suitable colliding ranges still does not guarantee that a particular pair of nodes will be affected. In particular, there is no need for $\mathcal{A}$ and $\mathcal{B}$ to choose the range boundaries that occur in a protocol run deterministically (see \cref{random-boundaries}). They can even perform multiple randomized protocol runs in parallel, while keeping track of item transmissions and sending every item at most once across all these protocol runs.

Another factor mitigating the impact of an adversary finding fingerprint collisions is the communication with other, non-colluding nodes. A fourth party could send some $u \in U, x \preceq u \prec y$ to $\mathcal{A}$ or $\mathcal{B}$ before $\mathcal{A}$ and $\mathcal{B}$ synchronize, disrupting the collision.

Finally, in systems where nodes repeatedly synchronize with different other nodes, a single fingerprint collision in a single synchronization session would merely delay propagation of information rather than stop it completely (note that this does not hold for collisions of two singleton sets). Since the attack model requires both $\mathcal{A}$ and $\mathcal{B}$ to synchronize with more than one node in total, this might apply to many affected settings. Peer-to-peer systems communicating on a random overlay network in particular fall into this category. A malicious actor with enough control over the communication of other nodes to guarantee a tangible benefit from fingerprint collisions can likely disrupt operation of the network more effectively by exercising that control than by sabotaging synchronization.

All of these arguments are however purely qualitative and should as such be taken into account with caution, they are not a substitute for quantitative cryptographic analysis. A strong attacker might be able to find many pairs of sets of colliding fingerprints, or many sets that all share the same fingerprint, and none of the above arguments consider these cases.

In a system with a consensus mechanism among all participating nodes, the choice of $\f$ can periodically be changed, with the frequency being some function of the time it takes to find a viable collision, the cost of rebuilding all auxiliary data structures, and the general level of paranoia among the participating nodes. The $\complexity{n}$ cost of rebuilding the data structures is then being amortized over the number of synchronization sessions occurring between rebuilds, which is still an improvement over protocols that need to perform $\complexity{n}$ computations per synchronization session.

\subsection{Specific Monoids}
\label{specific-monoids}

Multiplication, addition and xor as ways of combining hashes have been studied in \cite{bellare1997new}, in the context of hashing sequences, of which our ordered sets are a special case. Their setting requires not only associativity but also commutativity. They thoroughly break xor by reducing the problem of finding a collision to that of solving a system of linear equations. We will not restate the full attack, but we will describe a weaker but simpler mechanism based on similar ideas that allows finding subsets whose fingerprint is a specific target value. This can be used as an optimization in a trusted setting (see \cref{subset-checks}).

The main observation is that xor is the additive operation in $\mathds{F}_2$, the finite field on two elements $\set{0, 1}$. A fingerprint can be interpreted as a vector $\begin{bmatrix}b_1 & b_2 & \ldots & b_k\end{bmatrix} \in \set{0, 1}^{1 \times k}$. The fingerprint of a set $S = \set{s_1, s_2, \ldots, s_n}$ can thus be computed as the sum (within $\mathds{F}_2$, i.e. as the xor) of the vectors corresponding to $s_1, s_2, \ldots, s_n$. The fingerprint of some $S' \subseteq S$ can also be regarded as the sum over all vectors, but each vector being first multiplied with a coefficient of $1$ if $s_i \in S'$ and coefficient $0$ if $s_i \notin S'$. In other words, the fingerprint of $S'$ is a linear combination of the hashes of the items in $S$.

This enables us to efficiently find subsets whose fingerprint are a particular vector. Let $S = \set{s_1, s_2, \ldots, s_n}$ be a set of items, and let $b = \begin{bmatrix}b_1 & b_2 & \ldots & b_k\end{bmatrix} \in \set{0, 1}^{1 \times k}$ be the target fingerprint. For $0 < i \leq n$, define $a_i = \begin{bmatrix}a_{i, 1} & a_{i, 2} & \ldots & a_{i, k}\end{bmatrix} \in \set{0, 1}^{1 \times k}$ to be $\f(s_i)$ interpreted as a vector over $\mathds{F}_2$. The coefficients for linear combinations of the $a_i$ equal to $b$ are the solutions to the following system of linear equations over $\mathds{F}_2$:

\[
\begin{bmatrix}
a_{1, 1} & a_{1, 2} & \ldots & a_{1, k}\\
a_{2, 1} & a_{2, 2} & \ldots & a_{2, k}\\
\vdots & \vdots & \ddots & \vdots \\
a_{n, 1} & a_{n, 2} & \ldots & a_{n, k}\\
\end{bmatrix} \cdot \begin{bmatrix}
x_1\\
x_2\\
\vdots\\
x_n
\end{bmatrix} = \begin{bmatrix}
b_1\\
b_2\\
\vdots\\
b_n
\end{bmatrix}
\]

Solutions can be found by gaussian elimination in $\complexity{n^3}$ time, an exponential improvement over brute forcing by computing the fingerprints of all subsets.

As xor admits polynomial-time attacks by solving linear equations, the authors of \cite{bellare1997new} next consider addition and multiplication. They unify parts of their discussion by relating the hardness of finding collisions to solving the balance problem: in a commutative group $(G, \groupaddsym, \neutraladd)$, given a set of group elements $S = \set{s_1, s_2, \ldots, s_n}$, find disjoint, nonempty subsets $S_0 = \set{s_{0, 0}, s_{0, 1}, \ldots, s_{0, k}} \subseteq S, S_1 = \set{s_{1, 0}, s_{1, 1}, \ldots, s_{1, l}} \subseteq S$ such that $s_{0, 0} \groupaddsym s_{0, 1}  \groupaddsym \ldots  \groupaddsym s_{0, k} = s_{1, 0}  \groupaddsym s_{1, 1}  \groupaddsym \ldots  \groupaddsym s_{1, l}$. They then reduce the hardness of the balance problem to other problems.

For addition, the balance problem is as hard as subset sum, which was at the time of publication conjectured to be sufficiently hard. Wagner showed however in \cite{wagner2002generalized} how to solve the balance problem in subexponential time for addition. To give an impression for the impact of this attack, consider \cite{mihajloska2015reviving} which uses addition for combining SHA-3~\cite{dworkin2015sha} digests, producing fingerprints of length between $2688$ and $4160$ or $6528$ to $16512$ bits to achieve security levels of $128$ or $256$ bit respectively against Wagner's attack. \cite{lyubashevsky2005parity} gives an improvement over Wagner's attack finding collisions in $\complexity{2^{n^\epsilon}}$ for arbitrary $\epsilon < 1$, further weakening addition as a choice of monoid operation.

For multiplication, the balance problem is as hard as the discrete logarithm problem in the group. This is a more ``traditional'' hardness assumption than subset sum, there are groups for which no efficient algorithm is known. The main drawback is that multiplication is less efficient to compute than addition. \cite{stanton2010fastad} includes a comparison between the performance of addition and multiplication for incremental hashing, the additive hash outperforms the multiplicative one by two orders of magnitude, even though it uses longer digests to account for Wagner's attack.

For our context in which fingerprints are frequently sent over the network, longer computation time might still be preferable over longer hashes. Fingerprints based on multiplication nevertheless need larger digests than traditional, non-incremental hash functions. \cite{maitin2017elliptic} suggests fingerprints of $3200$ bit to achieve $128$ bit security, and motivated by this uses binary elliptic curves as the underlying group to achieve more compact fingerprints - fingerprints of length $2k$ bits give $\complexity{2^k}$ security.

\cite{bellare1997new} also proposes a fourth monoid based on lattices. \cite{lewi2019securing} give a specific instantiation providing 200 bits of security with fingerprints of size $16 \cdot 1024 = 16384$ bit.

All of the preceeding monoid operations are commutative, which our approach to fingerprint computation does not require. A typical associative but not commutative operation is matrix multiplication. Study of a family of hash functions based on multiplication of invertible matrices was initiated in \cite{zemor1991hash}, whose security is related to solving hard graph problems on the Cayley graph of the matrix multiplication group. \cite{petit2011rubik} gives a good overview about the general principle and the security aspects of Cayley hash functions.

While \cite{tillich1994hashing}, an improvement over the originally proposed scheme, has been successfully attacked in \cite{grassl2011cryptanalysis}\cite{petit2010preimages}, there are several modifications~\cite{petit2009graph}\cite{bromberg2017navigating}\cite{sosnovski2016cayley} for which no attacks are currently known, and \cite{mullan2016text} showed random self-reducibility for Cayley hash functions.

Whereas the schemes based on commutative groups operate on two bitstrings at a time, the Cayley hash functions operate on two individual bits at a time. Attacks thus assume that the manipulated bits can be freely chosen, using this to e.g. craft palindromic inputs. This may mean that such attacks are hard to apply to our setting where the input bits are produced in non-reorderable batches by another, cryptographically secure hash function. After finding a bit sequence that yields a Cayley hash collision, an attacker still needs to find items for which the concatenation of their hashes is the desired bit sequence.

Alternatively, one can forgo the additional hash function and apply the Cayley hash function directly to the encodings of items. This simplifies the overall scheme and removes reliance on a second hash function. On the other hand, if one expects the additional hash function to be harder to attack than the Cayley hash, then making the attacks against the Cayley hash function less flexible might overall make attacks more difficult. Furthermore, Cayley hash functions are usually slower than typical choices for the hash function from items to bitstrings, so using both can produce significant speedups if items have long encodings.

Aside from Cayley hashes, we are not aware of any non-commutative monoids used for hashing. We would further like to note that any hashes based on groups still have more structure than required for our designs, as we don't need existence of inverse elements. Strictly speaking our designs do not even require a neutral element: the fingerprint of the empty set never needs to be exchanged in a protocol run, so we could have just as well defined fingerprints for all nonempty sets without relying on the neutral element to do so. The presentation we chose is merely more elegant. And finally, we do not require pre-image resistance for our fingerprints. Suitable hash functions could thus be located in a more general design space than studied in any literature we know of.

\section{Pseudorandom Data Structures}
\label{randomization}

Monoidal fingerprints allow efficient deterministic computation of fingerprints, but at the price of having to rely on some non-standard cryptographic primitives. A more established cryptographic construction is that of Merkle trees~\cite{merkle1989certified} and related data structures, all based on hashing the concatenation of child hashes. There is a significant body of work in the context of authenticated data structures based on these constructions, from rather simple balanced binary trees~\cite{naor2000certificate} to arbitrary acyclic directed graphs~\cite{martel2004general}, and we will be able to refer to such work for proofs of collision resistance.

The Merkle construction is however not associative when using a cryptographically secure hash function $\h$, as $\h(\h(\h(a) \concatenate \h(b)) \concatenate \h(c)) = \h(\h(a) \concatenate \h(\h(b) \concatenate \h(c)))$ would constitute a violation of collision resistance. Different such tree representations of the same set thus lead to completely different fingerprints. So in order to use this construction, there must be a unique search tree representation defined for every set.

\cite{uniquerepresentation} shows that deterministic search trees with unique shapes require $\complexity{\sqrt{n}}$ time for insertions and deletions in the worst case, making them unsuitable for our use case. A natural extension is to look at randomized data structures that define a unique presentation which allows modification and search in $\complexity{\log(n)}$ with high probability.

This has been thoroughly researched in the field of \defined{history-independent data structures}, intuitively speaking data structures whose bitlevel representations do not leak information about their construction sequence. This requirement has been shown in~\cite{hartline2005characterizing} to be equivalent to having a unique representation. For more background, we refer to the excellent introduction of~\cite{bender2016anti}.

In the following, we present two such data structures for representing sets - a randomized binary tree, and the skip list - and define Merkle-like fingerprinting schemes for them such that the fingerprints of arbitrary ranges can be computed in logarithmic time with high probability. As all nodes need to arrive at the same representation, no true randomness can be involved. Instead, all random decisions are based on pseudorandom bits derived from the data itself.

These schemes allow cryptographically secure fingerprints whose collision resistance does not depend on uncommon cryptographic building blocks. The price to pay is that computing fingerprints can take linear time in the worst case. Note however that the communication complexity of any synchronization protocol using these fingerprints is completely unaffected by the randomized computation time.

Unfortunately, adversaries can efficiently create data sets for which the randomized solutions degrade to linear performance, thus enabling denial-of-service attacks as the cost of computing the fingerprints during a protocol run can be made to dominate the communication cost. Passive adversaries however cannot effectively attack these fingerprints: even though they can technically only forward those parts of the data structure that have degraded performance relative to the number of items in those parts, they would still increase the absolute resource usage of their victims if they simply forwarded all data.

Proponents of randomized data structures also claim that they are significantly easier to implement than balanced search trees (see e.g.~\cite{seidel1996randomized} or~\cite{pugh1990skip}), making it worthwhile to consider them even in trusted environments, in which their expected logarithmic update complexities suffice.

\subsection{Pseudorandom Treaps}

A \defined{treap}~\cite{seidel1996randomized} is a search tree in which every vertex has an associated \defined{priority} from a totally ordered set, and in which the priorities of the vertices form a heap:

\begin{definition}
Let $U$ be a set, $\prec$ a linear order on $U$, $P$ a set, $\leq$ a linear order on $P$, $\fun{\priority}{U}{P}$, $V \subseteq U$, and $T$ a binary search tree on $V$.

Then we call $T$ a treap if for all $v \in V$ with parent $p$ it holds that $\priority(v) \leq \priority(p)$.
\end{definition}

From the properties of treaps shown in~\cite{seidel1996randomized} we will rely on the following:

\begin{itemize}
  \item If the priorities of all vertices are pairwise unequal, then there is exactly one treap on the vertex set, which is equal to the search tree obtained by successively inserting items in decreasing priority without any balancing.
  \item If the priorities are distributed uniformly at random, the height of the treap is expected to be logarithmic in the number of vertices.
  \item Inserting or deleting an item while maintaining the treap properties can be done in time proportional to the height of the treap.
\end{itemize}

In the following, we will fix some cryptographically secure hash function $\fun{\p}{U}{\set{0, 1}^k}$, and define $\priority(u) \defeq \p(u)$, using the numeric order on $\set{0, 1}^k$ for comparing priorities. Since $\p$ is collision resistant, we can assume the resulting treaps to be unique, and since the output of a secure hash function is indistinguishable from a random mapping, we can assume the resulting treaps to have expected logarithmic height.

Treaps store items in every vertex, whereas Merkle trees only store items in their leaves. \cite{buldas2002eliminating} proposes the following natural generalization and proves that the labels are collision free if the underlying hash function $\h$ is collision free:

  \[
   \tlabel{\h}(v) \defeq \begin{cases}
\h(v), &  \text{for leaf $v$} \\
\tlabel{\h}(\tlabel{\h}(c_{<}) \concatenate \tlabel{\h}(v)) & \, \parbox[t]{.6\textwidth}{v internal vertex with left child $c_{<}$\\and no right child}\\
\tlabel{\h}(\tlabel{\h}(v) \concatenate \tlabel{\h}(c_{>})) & \, \parbox[t]{.6\textwidth}{v internal vertex with right child $c_{>}$\\and no left child}\\
\tlabel{\h}(\tlabel{\h}(c_{<}) \concatenate \h(v) \concatenate \tlabel{\h}(c_{>})) & \, \parbox[t]{.6\textwidth}{v internal vertex with left child $c_{<}$\\and right child $c_{>}$}
\end{cases}
  \]

We will call a treap labeled according to this scheme a \defined{Merkle treap}. Since Merkle treaps are unique, we can define the fingerprint of some set as the root label of the Merkle treap on that set if it is nonempty, or $\nil$ otherwise.

\subsubsection{Subsets}

Now that we have defined the fingerprint of a set, we need a way of efficiently computing the fingerprint of any range $\range{x}{y}{S}$ given the Merkle treap of the set $S$. The algorithm can be expressed in exactly the same framework as that of \cref{subsets-associative}, changing merely some type definitions and the \texttt{combine} function:

\begin{minted}[mathescape]{haskell}
-- A fingerprint is now either a hash or $\mathtt{Nothing}$.
type Fingerprint = Maybe D

emptyFp :: Fingerprint
emptyFp = Nothing

liftFp :: D -> Fingerprint
liftFp d = Just d

-- Combine three fingerprints into a new fingerprint.
combine :: Fingerprint -> Fingerprint -> Fingerprint -> Fingerprint
combine Nothing Nothing Nothing = Nothing
combine (Just a) Nothing Nothing = Just a
combine Nothing (Just b) Nothing = Just b
combine Nothing Nothing (Just c) = Just c
combine (Just a) (Just b) Nothing = Just (hash (concat a b))
combine (Just a) Nothing (Just c) = Just (hash (concat a c))
combine Nothing (Just b) (Just c) = Just (hash (concat b c))
combine (Just a) (Just b) (Just c) = Just (hash (concat (concat a b) c))
\end{minted}

While the tree traversal and accumulation of data otherwise stays the same, the nature of the correctness argument changes quite a bit, since we cannot rely on the associativity of \texttt{combine} anymore. Instead, the core idea of the argument is that the computed fingerprint depends only on the relative positioning between the tree vertices within the range, which stays identical no matter how many vertices outside the range are added.

Recall that the shape of a treap is always that of the tree obtained by inserting items sorted by decreasing priority without rebalancing. Now consider the vertex $v$ with the smallest distance to the root $r$ such that $x \preceq v \prec y$, i.e. the vertex computed by \texttt{findInitial}. All items within $\range{x}{y}{S}$ are contained in $\V(T_v)$. The fingerprint computation for the range is clearly local to $T_v$, and while insertion of items outside that tree might change the path from $r$ to $v$, they do not change the shape of $T_v$.

It thus remains to show that \texttt{accGeq} and \texttt{accLt} compute the same result no matter how many items outside the range are contained in the tree they process. We give the full argument for \texttt{accGeq}, the case for \texttt{accLt} follows analogously.

\begin{proof}
Let $v$ be the root of a tree $T_v$ and $x \in U$. We show by induction on $\abs{\V(T_v)}$ that \texttt{accGeq} computes a value equal to the root label of the Merkle treap on $S_{\succeq} \defeq \set{u \in \V(T_v) \mid x \preceq u}$ (or $\nil$ if the set is empty).

\textbf{IB:} If $v$ is a leaf with $v \prec x$, then $\mathtt{(accGeq\ v\ x)} = \nil$ and indeed $S_{t} = \emptyset$. If $v$ is a leaf with $x \preceq v$, then $\mathtt{(accGeq\ v\ x)} =  \mathtt{Just \h(v)} = \mathtt{Just \tlabel{\h}(v)}$.

\textbf{IH:} Let  $n \in \N$ and $T_r$ be a tree rooted at $r$ with $\abs{\V(T_r)} = n$. Then $\mathtt{(accGeq\ v\ x)} = \tlabel{\h}(v)$.

\textbf{IS:} Let $T_v$ be a tree rooted at $v$ with $\abs{\V(T_v)} = n + 1$, let $c_{<}$ be the right child of $v$  and let $c_
{>}$ be the left child of $v$.

\begin{caselist}
\case If $v \prec x$, then:

\begin{align*}
\mathtt{(accGeq\ v\ x)} &\overset{def}= \mathtt{(accGeq\ c_{>}\ x)}\\
&\overset{IH}= \tlabel{\h}(c_{>})\\
&\overset{v \prec x}= tlabel{\h}(v).\\
\end{align*}

\case Otherwise, $x \preceq v$. Neither $v$ nor any vertex in $T_{c_{<}}$ has any influence on the shape of $T_{c_{>}}$. We thus have:

\begin{align*}
\mathtt{(accGeq\ v\ x)} &\overset{def}= \mathtt{(combine\ (accGeq\ c_{<}\ x)\ \h(v) \tlabel{\h}(c_{>}))}\\
&\overset{IH}= \mathtt{(combine\ \tlabel{\h}(c_{<})\ \h(v) \tlabel{\h}(c_{>}))}\\
&\overset{def}= \tlabel{\h}(\tlabel{\h}(c_{<}) \concatenate \h(v) \concatenate \tlabel{\h}(c_{>}))\\
&\overset{def}= \tlabel{\h}(v).\\
\end{align*}
\end{caselist}
\end{proof}

The computation time is proportional to the height of the treap, so expected $\complexity{\log(n)}$. But as the hash function is not associative, the tail-recursive versions of \texttt{accGeq} and \texttt{accLt} do not compute the correct result. Intuitively, the problem is that the computation traverses the tree from top to bottom, but the fingerprints are defined through a computation from the bottom to the top. Accordingly, tail-recursive formulations can be achieved by adding parent-pointers to all vertices and performing the computation by first traversing to the range boundary and then accumulating fingerprints while moving back up through the tree. The space complexity thus remains in $\complexity{1}$.

Just as with balanced search trees, a binary treap might be the most natural formulation, but not the most efficient one on real hardware. \cite{golovin2009b} describes B-treaps, which are treaps of higher degree. If such an optimized realization is chosen, the need to map between the actual data layout and the shape of the treap which defines the fingerprints complicates the implementation, in particular compared to monoidal fingerprints which naturally abstract over tree implementation details.

Beyond treaps there are other randomized search trees. \cite{pugh1989incremental} suggests hash tries for fingerprinting sets, and our mechanism for computing fingerprints of ranges is compatible with their approach. The reason we opted for treaps is that the pseudorandom selection of the tree shape is completely decoupled from the ordering over the items, whereas in a hash trie the ordering follows from the tree shape (or vice versa, depending on the viewpoint). As our protocols can benefit from a non-arbitrary ordering (compare \cref{observations}), treaps are the more flexible solution.

There exists also some work on deterministic unique tree representations that stay efficient if the number of items is either a particularly small or a particularly large fraction of the size of the universe, see~\cite{sundar1994unique} for both a specific approach and more related work. These approaches could also be adapted for our purposes, but since they pose restrictions on the number of items that can be stored, they are less flexible than our other approaches.

\subsubsection{Adversarial Treap Construction}

The main motivation for looking at pseudorandom schemes was to be able to rely on well-known, conventional, non-associative hash functions, implying that resilience against malicious peers is desired. While Merkle-treaps are collision-resistant, they open up another angle of attack: a malicious party can try to construct unbalanced treaps to make the cost of computing fingerprints super-logarithmic in the number of items, leading to a denial-of-service attack as the cost of computing the fingerprints during a protocol around dominates the communication cost.

There is a fairly straightforward way of creating treaps on $n$ vertices of height $n$ in expected $\complexity{n^2}$ time and $\complexity{1}$ space, making treaps unsuitable for an adversarial environment. A sequence $(u_0, u_1, \ldots, u_{n - 1})$ of items such that the treap on these items has height $n$ can be computed by successively choosing arbitrary but always increasing with respect to $\preceq$ items $p_j$ and letting $u_i \defeq p_j$ if $\floor{i \cdot \frac{2^k}{n}} \leq \priority(p_j) < \floor{(i + 1) \cdot \frac{2^k}{n}}$ (recall that $k$ is the number of bits of a hash digest). The probability for the priority of an arbitrary item to fall within the desired range is $\frac{1}{n}$, so finding one takes expected $\complexity{n}$ time, finding a sequence of $n$ of these accordingly takes expected $\complexity{n^2}$ time.

\subsection{Pseudorandom Skip Lists}

The skip list~\cite{pugh1990skip}\cite{pugh1998skip} is a probabilistic data structure for storing sets that does not use a tree representation, but rather a hierarchy of progressively sparser linked lists containing set items in sorted order. The list at layer $0$ contains every item, the list at layer $i + 1$ contains every item from layer $i$ with a probability of $\frac{1}{2}$. Intuitively, when inserting an item into the skip list, one performs coin flips until one loses, the item is then added at the correct position into as many layers as one performed coin flips. \Cref{fig:skip-list-example} gives an example.

\newcommand{\skiplistnode}[3]{
\node (n#1#2) at (#1, #2) [skiplistnode] {#3};
}

\newcommand{\skiplisttower}[3]{
\foreach \y in {0,...,#2}
{
    \skiplistnode{#1}{\y}{#3}
}
}

\newcommand{\toweredges}[2]{
\foreach \y in {1,...,#2}
{
    \pgfmathtruncatemacro{\myPrev}{\y-1}
    \draw (n#1\y) edge[edge] (n#1\myPrev);
}
}

\newcommand{\listedges}[2]{
\foreach \y in {0,...,#2}
{
    \pgfmathtruncatemacro{\myPrev}{#1-1}
    \draw (n#1\y) edge[edge] (n\myPrev\y);
}
}

\begin{figure*}
\begin{scaletikzpicturetowidth}{\textwidth}
\begin{tikzpicture}[scale=\tikzscale,font=\footnotesize]
	\pgfdeclarelayer{background}
	\pgfdeclarelayer{foreground}
	\pgfsetlayers{background,main,foreground}

	\begin{pgfonlayer}{main}
\skiplisttower{0}{2}{$\bot$}
\skiplisttower{1}{2}{$\examplea$}
\skiplisttower{2}{0}{$\exampleb$}
\skiplisttower{3}{1}{$\examplec$}
\skiplisttower{4}{0}{$\exampled$}
\skiplisttower{5}{1}{$\examplee$}
\skiplisttower{6}{0}{$\examplef$}
\skiplisttower{7}{0}{$\exampleg$}
\skiplisttower{8}{1}{$\exampleh$}
\skiplisttower{9}{2}{$\top$}

\toweredges{0}{2}
\toweredges{1}{2}
\toweredges{3}{1}
\toweredges{5}{1}
\toweredges{8}{1}
\toweredges{9}{2}

\listedges{1}{2}
\listedges{2}{0}
\listedges{3}{0}
\listedges{4}{0}
\listedges{5}{0}
\listedges{6}{0}
\listedges{7}{0}
\listedges{8}{0}
\listedges{9}{1}

\draw (n31) edge[edge] (n11);
\draw (n51) edge[edge] (n31);
\draw (n81) edge[edge] (n51);
\draw (n92) edge[edge] (n12);
	\end{pgfonlayer}
\end{tikzpicture}
\end{scaletikzpicturetowidth}

\caption{
An example skip list, demonstrating the layers of sorted linked lists. The $\bot$ and $\top$ vertices are the start and end points for all lists.
}

\label{fig:skip-list-example}
\end{figure*}

To define a pseudorandom skip list, we again fix some cryptographically secure hash function $\fun{\p}{U}{\set{0, 1}^k}$, and determine the layers of an item $u$ by counting the number of leading zero bits of $\p(u)$ and adding one.

Toward a fingerprinting scheme, we define a labeling over all items in all layers. Let $u_{l, i}$ denote the $i$-th item in layer $l$, recall that $\h$ is some cryptographically secure hash function.
Then $\sllabel{\h}(u_{0, i}) \defeq \h(u_{0, i})$ and $\sllabel{\h}(u_{l + 1, i}) \defeq \h(\sllabel{\h}(u_{l, j}) \cdot \h( \sllabel{\h}(u_{l, j + 1}) \cdot \h(\ldots \cdot \sllabel{\h}(u_{l, j + m}))))$ where $(u_{l, j} \ldots u_{l, j + m})$ is the longest subsequence of layer $l$ such that $u_{l, j} = u_{l + 1, i}$ and $u_{l, j + m} \prec u_{l + 1, i + 1}$. \Cref{fig:skip-list-flow} visualizes an example.

\begin{figure*}
\begin{scaletikzpicturetowidth}{\textwidth}
\begin{tikzpicture}[scale=\tikzscale,font=\footnotesize]
	\pgfdeclarelayer{background}
	\pgfdeclarelayer{foreground}
	\pgfsetlayers{background,main,foreground}

	\begin{pgfonlayer}{main}
\skiplisttower{0}{2}{$\bot$}
\skiplisttower{1}{2}{$\examplea$}
\skiplisttower{2}{0}{$\exampleb$}
\skiplisttower{3}{1}{$\examplec$}
\skiplisttower{4}{0}{$\exampled$}
\skiplisttower{5}{1}{$\examplee$}
\skiplisttower{6}{0}{$\examplef$}
\skiplisttower{7}{0}{$\exampleg$}
\skiplisttower{8}{1}{$\exampleh$}
\skiplisttower{9}{2}{$\top$}

\toweredges{0}{2}
\toweredges{1}{2}
\toweredges{3}{1}
\toweredges{5}{1}
\toweredges{8}{1}
\toweredges{9}{2}

\listedges{1}{2}
\listedges{2}{0}
\listedges{3}{0}
\listedges{4}{0}
\listedges{5}{0}
\listedges{6}{0}
\listedges{7}{0}
\listedges{8}{0}
\listedges{9}{1}

\draw (n31) edge[edge] (n11);
\draw (n51) edge[edge] (n31);
\draw (n81) edge[edge] (n51);
\draw (n92) edge[edge] (n12);

\draw[->] (n10) edge[dep] (n11);
\draw[->] (n20) edge[dep] (n11);

\draw[->] (n30) edge[dep] (n31);
\draw[->] (n40) edge[dep] (n31);

\draw[->] (n50) edge[dep] (n51);
\draw[->] (n60) edge[dep] (n51);
\draw[->] (n70) edge[dep] (n51);

\draw[->] (n80) edge[dep] (n81);

\draw[->] (n11) edge[dep] (n12);
\draw[->] (n31) edge[dep] (n12);
\draw[->] (n51) edge[dep] (n12);
\draw[->] (n81) edge[dep] (n12);
	\end{pgfonlayer}
\end{tikzpicture}
\end{scaletikzpicturetowidth}

\caption{
A visualization of which labels influence which other labels. One can see that these dependencies form a tree.
}

\label{fig:skip-list-flow}
\end{figure*}

Every label contributes to the computation of at most one other label, which is located on the next layer. Maintaining labels when inserting or deleting an item thus takes a number of hash computations bounded by the height of the skip list, which is expected $\complexity{\log(n)}$. The number of label concatenations that need to be performed at each layer is also randomized, but on average there are two, as the probability for an item to also occur at the next layer is $\frac{1}{2}$. The expected time thus stays in $\complexity{\log(n)}$.

Denote by $\skipdomain(u_{l, i})$ the set of items $u$ such that $u_{l, i} \preceq u \prec u_{l, i + 1}$, i.e. the set of all items whose hash contributes to the label of $u_{l, i}$. Now let $S \subseteq U$ be a finite set. The fingerprint of $S$ is defined as $\nil$ if $S = \emptyset$, and otherwise as $\h(u_{l_0, i_0} \cdot \h(u_{l_1, i_1} \cdot \h(\ldots \cdot u_{l_m, i_m})))$ where $(u_{l_0, i_0}, u_{l_1, i_1}, \ldots, u_{l_m, i_m})$ is the unique shortest sequence of vertices in the pseudorandom skip list on $S$ such that its items are strictly increasing and their $\skipdomain$s partition $S$, choosing the vertex of the highest possible layer if multiple vertices have equal domains. See \cref{fig:skip-list-fingerprint} for an example.

\begin{figure*}
\begin{scaletikzpicturetowidth}{\textwidth}
\begin{tikzpicture}[scale=\tikzscale,font=\footnotesize]
	\pgfdeclarelayer{background}
	\pgfdeclarelayer{foreground}
	\pgfsetlayers{background,main,foreground}

	\begin{pgfonlayer}{main}
\skiplisttower{0}{2}{$\bot$}
\skiplisttower{1}{2}{$\examplea$}
\node (n20) at (2, 0) [fingerprintnode] {$\exampleb$};
\skiplisttower{3}{0}{$\examplec$}
\node (n31) at (3, 1) [fingerprintnode] {$\examplec$};
\skiplisttower{4}{0}{$\exampled$}
\skiplisttower{5}{0}{$\examplee$}
\node (n51) at (5, 1) [fingerprintnode] {$\examplee$};
\node (n60) at (6, 0) [fingerprintnode] {$\examplef$};
\skiplisttower{7}{0}{$\exampleg$}
\skiplisttower{8}{1}{$\exampleh$}
\skiplisttower{9}{2}{$\top$}

\toweredges{0}{2}
\toweredges{1}{2}
\toweredges{3}{1}
\toweredges{5}{1}
\toweredges{8}{1}
\toweredges{9}{2}

\listedges{1}{2}
\listedges{2}{0}
\listedges{3}{0}
\listedges{4}{0}
\listedges{5}{0}
\listedges{6}{0}
\listedges{7}{0}
\listedges{8}{0}
\listedges{9}{1}

\draw (n31) edge[edge] (n11);
\draw (n51) edge[edge] (n31);
\draw (n81) edge[edge] (n51);
\draw (n92) edge[edge] (n12);
	\end{pgfonlayer}
\end{tikzpicture}
\end{scaletikzpicturetowidth}

\caption{
The skip list nodes contributing to the fingerprint of the range $\range{\exampleb}{\exampleg}{S}$. If the range had been $\range{\exampleb}{\examplef}{S}$, it would have again been the topmost $\examplee$ vertex whose label would have been hashed.
}

\label{fig:skip-list-fingerprint}
\end{figure*}

We next sketch an algorithmic way of determining this sequence for any range $\range{x}{y}{S}$ given the deterministic skip list for $S$, but refrain from giving correctness proofs as they are rather verbose and convey no particular insight that is not better conveyed in the accompanying figures.

Let $\bar{u} = (u_0, u_1, \ldots, u_m)$ be the shortest path in the skip list from its beginning to the vertex $y$ excluding $y$ itself, i.e. the sequence of vertices traversed when looking up $y$ in the skip list. This sequence has expected logarithmic length and can be computed in expected logarithmic time. Obtain a shorter sequence $\bar{u'} = (u'_0, u'_1, \ldots, u'_{m'})$ by keeping from all vertices in $\bar{u}$ that correspond to the same item the one in the greatest layer such that its $\skipdomain$ does not contain any item greater than or equal to $y$. \Cref{fig:skip-list-fingerprint-traversal} provides an example.

\begin{figure*}
\begin{scaletikzpicturetowidth}{\textwidth}
\begin{tikzpicture}[scale=\tikzscale,font=\footnotesize]
	\pgfdeclarelayer{background}
	\pgfdeclarelayer{foreground}
	\pgfsetlayers{background,main,foreground}

	\begin{pgfonlayer}{main}
\skiplisttower{0}{2}{$\bot$}
\skiplisttower{1}{2}{$\examplea$}
\node (n20) at (2, 0) [fingerprintnode] {$\exampleb$};
\skiplisttower{3}{0}{$\examplec$}
\node (n31) at (3, 1) [fingerprintnode] {$\examplec$};
\skiplisttower{4}{0}{$\exampled$}
\skiplisttower{5}{0}{$\examplee$}
\node (n51) at (5, 1) [fingerprintnode] {$\examplee$};
\node (n60) at (6, 0) [fingerprintnode] {$\examplef$};
\skiplisttower{7}{0}{$\exampleg$}
\skiplisttower{8}{1}{$\exampleh$}
\skiplisttower{9}{2}{$\top$}

\toweredges{0}{2}
\toweredges{1}{2}
\toweredges{3}{1}
\toweredges{5}{1}
\toweredges{8}{1}
\toweredges{9}{2}

\listedges{1}{2}
\listedges{2}{0}
\listedges{3}{0}
\listedges{4}{0}
\listedges{5}{0}
\listedges{6}{0}
\listedges{7}{0}
\listedges{8}{0}
\listedges{9}{1}

\draw (n31) edge[edge] (n11);
\draw (n51) edge[edge] (n31);
\draw (n81) edge[edge] (n51);
\draw (n92) edge[edge] (n12);

\begin{scope}[transparency group, opacity=0.3]
			\draw[red,line width=14pt,rounded corners=0.1em,line cap=round] (n02.center) -- (n12.center) -- (n11.center) -- (n31.center) -- (n51.center) -- (n50.center) -- (n60.center) -- (n70.center);
			\draw[red,line width=14pt,rounded corners=1em,line cap=round] (n02.center) -- (n12.center) -- (n11.center) -- (n31.center) -- (n51.center) -- (n50.center) -- (n60.center) -- (n70.center);
		\end{scope}

\begin{scope}[transparency group, opacity=0.3]
			\draw[blue,line width=14pt,rounded corners=0.1em,line cap=round] (n92.center) -- (n91.center) -- (n81.center) -- (n51.center) -- (n31.center) -- (n30.center) -- (n20.center);
			\draw[blue,line width=14pt,rounded corners=1em,line cap=round] (n92.center) -- (n91.center) -- (n81.center) -- (n51.center) -- (n31.center) -- (n30.center) -- (n20.center);
		\end{scope}
	\end{pgfonlayer}
\end{tikzpicture}
\end{scaletikzpicturetowidth}

\caption{
The forward search path to $\exampleg$ (starting at the top-left corner) and the backward search path to $\exampleb$ (starting at the top-right corner) allow to read off which vertices contribute to the fingerprint of $\range{\exampleb}{\exampleg}{S}$.
}

\label{fig:skip-list-fingerprint-traversal}
\end{figure*}

Now let $\bar{v} = (v_0, v_1, \ldots, v'_o)$ be the shortest path in the skip list obtained by inverting the order on items from its beginning to the vertex $x$. Intuitively, this corresponds to looking up $x$ ``from the back'' in a doubly-linked skip list. Obtain from $\bar{v}$ the corresponding shorter sequence $\bar{v'} = (v'_0, v'_1, \ldots, v'_{o'})$ by keeping from all vertices that correspond to the same item the one of the greatest layer such that its $\skipdomain$ does not contain any item strictly less than $x$.

We claim that $\bar{v'}$ and $\bar{u'}$ always intersect (if $\range{x}{y}{S} \neq \emptyset$) in at least one vertex, i.e. there are unique $v'_i \in \bar{v'}$ and $u'_j \in \bar{u'}$ with $v'_i = u'_j$. Then the sequence $(v'_0, v'_1, \ldots, v'_i = u'_j, u'_{j + 1}, \ldots, u'_{m})$ is the unique shortest sequence of vertices of strictly increasing items whose $\skipdomain$s partition $\range{x}{y}{S}$. Since $\bar{v'}$ and $\bar{u'}$ can be computed in expected logarithmic time, so can this sequence, and thus fingerprints can be computed in expected logarithmic time.

\subsubsection{Adversarial Skip List Construction}

Pseudorandom skip lists are even less suited for adversarial environments then pseudorandom treaps. Performance degrades if most items have the same maximum layer. A randomly chosen item has a hash beginning with a $1$ bit with probability $\frac{1}{2}$, these items only reside in layer $0$. An adversary can thus create a linear list of exclusively layer $0$ items of length $n$ in expected $2n \in \complexity{n}$ time.