code:
=encoding utf-8

=head1 NAME

XRCU is a library that provides efficient lock-less synchronization for read-mostly tasks and structures

=head1 ABOUT THIS DOCUMENT

This file is meant to document the XRCU library, including internal details
and the rationale behind the decisions taken.

This document assumes that the reader is at least somewhat familiar with such
concepts as multi-threading, lock-free algorithms and data structures, and
read-copy-update (RCU).

=head1 ABOUT LOCK FREE ALGORITHMS AND STRUCTURES

The term I<lock free> is used to refer to several different things depending
on the author, but the basic idea is that an algorithm or structure is lock
free if multiple threads can operate on it without the end result being
dependent on said threads' scheduling. That is, it doesn't matter if the
involved threads are suspended or preempted; progress is guaranteed.

The usually mentioned advantage of lock freedom is performance. When you are
using lock free algorithms, you naturally avoid using things like mutexes,
which can have a non-negligible performance impact. On the other hand, other
people point out that lock free algorithms are typically harder to understand,
that debugging is notoriously difficult, and that the benefits generally do
not compensate for these complexities.

The reason why this library was developed is to give other programmers another
option when designing high performant, low latency systems. We do not claim
that lock freedom is a silver bullet when it comes to making programs faster.
On the other hand, it's also hard to argue that lock contention tends to be a
real bottleneck, and several projects have moved towards lock free structures
given the performance benefits they bring (see for example, many kernels).

=head1 ABOUT RCU

There is a big problem with using lock free algorithms in programming languages
that feature manual memory management, and it has to do with memory reclamation.
Since lock freedom implies that multiple threads may operate on a given
structure concurrently, it is possible to find ourselves in a situation in which
a thread is reading some data, while another one is deleting that same piece of
data, potentially freeing the memory associated to it.

As such, lock freedom needs an additional twist in languages like C and C++.
We need an additional subsystem that allows us to synchronize reclamation
without using heavyweight methods like mutexes (we are developing a lock-free
library, after all).

This is where RCU comes in. RCU, short for read-copy-update, is a mechanism
that allows multiple threads to I<read> from memory, while postponing
reclamations until it is safe to do so (i.e: until no readers are operating
on said memory). RCU typically assumes that updates are performed using atomic
instructions, and isn't concerned with them (At least, that's the case in this
library).

There are several different ways to implement RCU, and the one chosen in this
library will be detailed later. For now, it's important to point out that RCU
typically makes reading from shared memory a very cheap operation, while
imposing more overhead on updates. Thus, it's more suited for tasks in which
reads are more frequent than modifications.

It should be noted that RCU is not the only way to manage memory reclamations
in a multi-threaded environment, but as its name implies, XRCU chose it because
it was deemed the best option. Since our lock-free structures rely on RCU to
work properly, we'll describe the RCU interface that this library exposes first.

=head2 API design

All public interfaces (functions, types, methods, etc) reside in the namespace
C<xrcu>. It's the explicit intent of the authors to mantain compatibility with
every interface that was ever exposed. Any breaking changes should be done in
different interfaces (either by adding new functions, or by overloading).

The implementation of some template types require additional, internal details,
and those usually reside in a namespace appropriately named C<detail> (Said
namespace is inside C<xrcu>). It goes without saying that users should I<not>
rely on those internal details, and that the authors may freely break
compatibility with them.

XRCU uses some thread-specific data in order to function properly. There may
be some systems that have issues when dynamically loading libraries that use
thread-specific data (I believe Windows Vista was among them). Consult with
your working environment's manual to verify that if you notice any problem.

Finally, it should be mentioned that XRCU requires no initializatio or cleanup
at a library level to function properly. Once the library is installed, you
can freely use its API without any further action needed.

Note that XRCU may expose more functions or types that are specified in this
document. This is usually done to avoid code bloat. Anything that isn't
included in this file should be treated as private and subject to change.

=head2 RCU API

  #include <xrcu/xrcu.hpp>

The following section documents the RCU API that is exposed in XRCU. It's used
internally quite a bit by other parts of the library, yet it's very useful by
itself when implementing other concurrent algorithms or data structures.

=head3 RCU critical section management

RCU is based on the concept of I<critical sections>, code fragments that are
executed under the guarantee that no reclamation can take place. This allows
the user to safely read from shared memory, knowing that it will be valid
during the critical section. Writes have to be synchronized independently
of RCU, however.

Naturally, RCU critical sections don't prevent I<all> memory reclamation,
they only affect the destruction of objects that are derived from a type
defined in XRCU, called C<finalizable>. If you wish to use the RCU API, all
you need to do is make your types derive from C<finalizable> and use its API,
and you'll be set. Any other memory management function, like C<malloc> and
friends is unaffected by RCU as implemented in this library.

Critical sections are managed through the following API:

=over 4

=item void enter_cs (void);

Makes the calling thread enter a critical section. Critical sections can be
safely nested without problem. As long as we are in one, C<finalizable> objects
cannot be reclaimed.

=item void exit_cs (void);

Exits a critical section. This is generally called after the calling thread
is done reading from shared memory, and wants to signal that reclamations
should be re-allowed. The effects of calling C<exit_cs> when the calling thread
is not in a critical section are undefined.

=item bool in_cs (void);

Returns true if the calling thread is in a critical section; false otherwise.

=item bool sync (void);

Waits until all threads are outside pre-existing critical sections, and returns
true afterwards. If a deadlock is detected (because the calling thread is in a
critical section, for example), this function returns false immediately
without waiting.

=item struct cs_guard

This type is defined such that its constructor calls C<enter_cs>, and its
destructor calls C<exit_cs>; it has no internal state. As its name implies,
it's useful as a guard to manage entering and exiting a critical section in
a way that is exception safe. A few types in this library are derived from
C<cs_guard>, such as iterators, since they need to examine potentially many
elements from a container without their memory being reclaimed.

=back

=head3 RCU finalizable objects

As it was mentioned before, XRCU defines a type called C<finalizable> that
is specifically designed to make its destruction safe (i.e: Only once all
threads are outside a critical section). This type defines the following
interface:

  struct finalizable
    {
      virtual void safe_destroy ();
      virtual ~finalizable ();
    };

Under most circumstances, it's enough for a user-defined type to derive from
C<finalizable> and leave it at that. However, the above 2 methods are provided
as virtual for customization's sake. When a C<finalizable> object is reclaimed
by the RCU subsystem, it will call the C<safe_destroy> method. The default
implementation simply calls the object's destructor and frees the memory
associated to it. If, for whatever reason, a user wants to override this
behaviour, they may do so by extending either of those methods.

In addition, the following API is available when dealing with finalizables:

=over 4

=item void finalize (finalizable *F);

Adds the object C<F> to the calling thread's list of pending finalizable objects.
Each thread has a limit on the number of pending C<finalizables>. Once that
limit is reached, they are scheduled for reclamation, and will be collected
once it's safe to do so.

Only a single call to C<finalize> is allowed on a particular object. If this
function is called more than once on the same object (Either by the same
thread, or another), the behaviour is undefined.

=item bool flush_finalizers (void);

Schedule all the calling thread's accumulated C<finalizable> objects for
reclamation immediately. Returns true if successful, false if a deadlock was
detected. Note that this call may block until all threads are outside a
critical section (Much as a call to C<sync> would).

Once a thread exits, an implicit call to this function is made.

=back

=head3 Miscellaneous functions

These functions don't really belong anywhere else, but they are included in
this file for convenience's sake:

    struct atfork
      {
        void (*prepare) (void);
        void (*parent) (void);
        void (*child) (void);
      };

=over 4

=item atfork atfork_data ();

Returns a structure that implements the needed callbacks for XRCU to mantain a
consistent state across calls to C<fork>. The returned callbacks should be
passed to C<pthread_atfork> (or similar). XRCU itself does not install these
callbacks, because most calls to C<fork> are followed immediately after C<exec>,
and because multi-threaded programs that call C<fork> are rare.

=item void library_version (int& major, int& minor);

Returns the library version as a pair of C<major>, C<minor>. Useful to assert
that a program is using the correct version (at runtime).

=back

=head3 Implementation details

There are many ways to implement RCU. In this library, it was fundamental that
RCU be implemented in a portable, simple way, even if it meant slightly more
overhead. As such, things like signals and OS-specific syscalls were out.

In the end, for XRCU, we decided to use a global registry that uses a stamp
to monitor which threads are in critical sections. By using C++ C<thread_local>
objects, we avoid the need for explicit registration, so that when a thread
first uses the RCU API, it is added automatically to the global registry.

In order to enter a critical section, all a thread has to do is read a global
counter, usually bump it by some small value, and then store that value with
release semantics in its thread-specific data. Exiting a critical section is
almost entirely symmetric (We decrement the thread-specific value), but with a
small caveat that will be explained below.

When an object is finalized, it's prepended to a singly-linked list that is
also kept in thread-specific storage. Once a certain number of them have been
accumulated (specified by the constant C<XRCU_MAX_FINS>), they are scheduled to
be reclaimed. However, if the calling thread is inside a critical section at
that point, a special flag is set instead, that tells the thread to immediately
flush its C<finalizable> objects once it's outside the critical section.

In this implementation, the most expensive operation is undoubtedly C<sync>.
It works by locking the global registry, then checking if any thread is in a
critical section, and sleeping for short periods of time in case there are.
The overhead associated to C<sync> is the main reason why critical sections
should be short, and also why C<finalizable> objects are accumulated instead of
being reclaimed right away.

=head2 Interlude: optionals

    #include <xrcu/optional.hpp>

Before moving on to the topic of containers, we need to take a look at an
auxiliary structure implemented to make things easier and more convenient
for users. This is the C<optional> type.

An optional is a template type that can hold either an object of type I<T>, or
be in an uninitialized state. In other words, the value may or may not be
present. Unlike the usage of a pointer which may be null, an C<optional> never
performs dynamic memory allocation, as the space required to hold the value is
always there, even if it's not being used.

Optional types have been standardized in C++17, and are present in many other
libraries as well. However, since XRCU aims to work with the base minimum of
C++11 compliant compilers, and because it's designed not to depend on other
libraries or frameworks, it contains its own (lightweight) implementation.

=head3 Optional API

An optional is defined as such:

    template <class T>
    struct optional
      {
      };

And its public interface is the following:

=over 4

=item optional ();

Default constructor. Initializes the optional without a value.

=item optional (const T& value);

Initializes the optional to contain C<value>.

=item optional (const optional<T>& other);

Copy constructor. If C<other> has a value, then it initializes the optional to
contain that same value. Otherwise, the optional will contain no value.

=item optional (T&& value);

Move constructor. Initializes the optional by taking ownership of C<value>.

=item optional (optional<T>&& other);

Move constructor. If C<other> has a value, then the optional is initialized
by taken ownership of it. Otherwise, the optional will have no value.

=item T& operator* ();

=item const T& operator* () const;

Returns a (possibly const) reference to the optional's value. The results are
undefined if the optional does not contain one.

=item T* operator-> ();

=item const T* operator-> () const;

Returns a (possibly const) pointer to the optional's value. The results are
undefined if the optional does not contain one.

=item bool has_value () const;

Returns true if the optional has a value.

=item void reset () const;

If the optional contains a value, call its destructor and make the optional
contain no value afterwards. Otherwise, there are no effects.

=item optional& operator= (const optional<T>& other);

If C<other> has a value, assigns it to the optional. Otherwise, calls C<reset>
on the optional and leaves it without a value. Returns C<*this>.

=item optional& operator= (const T& value);

Assigns C<value> to the optional. Returns C<*this>.

=item optional& operator= (optional<T>&& other);

If C<other> has a value, assigns it to the optional by moving it. Otherwise,
calls C<reset> on the optional and leaves it without a value. Returns C<*this>.

=item optional& operator= (T&& value);

Assigns C<value> to the optional by moving it. Returns C<*this>.

=item ~optional ();

Destroys the value associated to the optional, if it had any.

=back

=head3 Implementation details

Optionals are rather simple: They are implemented by using a flat buffer
on which placement new is called. Once an object has been constructed,
a pointer is cached for the buffer. This pointer will be used when fetching
the value, or destroying the object (it's null for empty optionals).

Optionals are very useful when performing lookups in mapped containers, since
they allow us to bypass the need of additional output parameters to determine
if a search was successful in an atomic way.

Another reason as to why optional values are used so extensively in XRCU is
because we consider them to be the best way to signal both a valid object,
as well as an error. When using lock-less data structures, it's impossible to
determine beforehand whether an error will occur when calling a method (because
multithreading makes things non deterministic). Throwing an exception was
considered a bit too "punishing" - it's not precisely a programming error,
after all.

=head2 Stacks

  #include <xrcu/stack.hpp>

Stacks are the simplest of the lock-free data structures: They represent a
basic LIFO stack on which you can push and pop items, one at a time. Although
their functionality is a bit more limited than other structures, they are still
very useful to implement things like atomic free lists.

In XRCU, stacks meet the requirements for the C++ concept of I<Container>.

=head3 Stack API

Stacks are templated types, that can be instantiated with a type T:

  template <class T>
  struct stack
    {
      typedef T value_type;
      typedef T& reference;
      typedef const T& const_reference;
      typedef T* pointer;
      typedef const T* const_pointer;
      typedef ptrdiff_t difference_type;
      typedef size_t size_type;

      struct iterator;
      struct const_iterator;
    };

In C++'s parlance, a stack iterator is a I<forward> iterator, and it may be
used in any function that accepts them.

A stack iterator is a C<cs_guard>, which means that an implicit critical
section is entered when one is created, and is extended until its destruction.

The following describes the public interface for C<stack<TE<gt>>

=over 4

=item stack ();

Default constructor. Initializes a stack to be empty.

=item template <class Integer> stack (Integer n, T value);

Initializes a stack to contain C<n> times C<value>.

=item template <class Iter> stack (Iter first, Iter last);

Initializes a stack with the values in the range [C<first>, C<last>)

=item stack (std::initializer_list<T> lst);

Initializes a stack with the values in C<lst>.

=item stack (const stack<T>& other);

Copy constructor. Initializes a stack with the values in C<other>.

=item stack (stack<T>&& other);

Move constructor. Takes ownership of the values in C<other>.

=item void push (const T& value);

Pushes C<value> to the top of the stack.

=item template <class Iter> void push (Iter first, Iter last)

Pushes the values in the range [C<first>, C<last>) to the stack.

=item template <class Integer> void push (Integer n, T value)

Pushes C<n> times C<value> to the stack.

=item template<class ...Args> void emplace (Args&& ...args)

Same as C<push>, only it constructs a new item by calling the move
constructor with C<args...>.

=item optional<T> pop ();

Removes the item at the top of the stack, and returns an optional with that
value. If the stack is empty, the method returns an optional with no value.

=item optional<T> top ();

Fetches the current item at the top of the stack, and returns an optional
with it as its value. If the stack is empty, returns an optional with no value.

=item size_type size () const;

Returns the current size of the stack.

=item size_type max_size () const;

Returns the maximum allowed size for the stack.

=item bool empty () const;

Returns true if the stack is empty (equivalent to C<size () == 0>).

=item void swap (stack<T>& other);

Swaps the contents of the stack with C<other>, but only if C<other> is not the
same object.

=item stack<T>& operator= (const stack<T>& other);

Assigns the contents of C<other> to the stack and returns C<*this>.

=item stack<T>& operator= (stack<T>&& other);

Move assignment. Takes ownership of the elements in C<other>. Returns C<*this>.

=item bool operator== (const stack<T>& other) const;

Compares the elements of the stack with the ones in C<other>. Returns true
if they are all equal.

=item bool operator!= (const stack<T>& other) const;

Compares the elements of the stack with the ones in C<other>. Returns true
if any two of them are not equal.

=item bool operator<  (const stack<T>& other) const;

=item bool operator>  (const stack<T>& other) const;

=item bool operator<= (const stack<T>& other) const;

=item bool operator>= (const stack<T>& other) const;

Lexicographically compares the elements of the stack with the ones in C<other>,
in a way that is equivalent to calling C<std::lexicographical_compare>.

=item void clear ();

Removes every element from the stack.

=item template <class T1, class T2> void assign (T1 x, T2 y);

Assigns to the stack the elements described by (C<x>, C<y>). They could be an
iterator range, or a pair of (integer, value), as is the case with the stack's
constructor.

=item void assign (std::initializer_list<T> lst);

Assigns to the stack the items in C<lst>.

=item iterator begin ();

=item const_iterator cbegin () const;

Returns an iterator to the first element of the stack.

=item iterator end ();

=item const_iterator cend () const;

Returns an iterator one past the last element of the stack.

=item iterator::iterator ();

Default constructor for stack iterators. Leaves the object in an invalid state.
Dereferencing or incrementing such an iterator has undefined behavior.

=item iterator::iterator (const iterator& other);

Copy constructor. Initializes the iterator to be equal to C<other>.

=item T& iterator::operator* ();

=item const T& const_iterator::operator* () const;

Returns a (possibly const) reference to the iterator's underlying object.

=item T* iterator::operator-> ();

=item const T* const_iterator::operator-> () const;

Returns a (possibly const) pointer to the iterator's underlying object.

=item iterator& iterator::operator++ ();

Pre-increment operator. Moves the iterator forward, and returns it. If the
iterator was at the end of the stack, the results are undefined.

=item iterator iterator::operator++ (int);

Post-increment operator. Moves the iterator forward, and returns an iterator
equal to what it was before incrementing. If the iterator was at the end of
the stack, the results are undefined.

=item bool iterator::operator== (const iterator& other);

=item bool iterator::operator!= (const iterator& other);

Tests for iterator (in)equality.

=back

=head3 Implementation details

There really isn't much to say about the internal details of the stack. It's
probably the easiest lock free structure to implement. As with most other
implementations, the one in XRCU simply consists of an atomic pointer to the
top node. The use of RCU prevents the biggest issue with this design, that is,
the ABA problem.

The only noteworthy thing to point out is that the C<swap> method is safe to
call from multiple threads as well. In order to achive this, the library uses
a special I<sentinel> bit, that is temporarily set as the head node when a
swap is undergoing. During a swap, C<push> and C<pop> cannot proceed, since
they check against that the special bit is not set before modifying the stack.

Note that because stack iterators are implicit critical sections, and because
of stacks' implementation, iterating a stack will always be safe, even in the
presence of operations like C<push> and C<pop>. The only way for an iterator
to be invalidated is it's at the beggining of the stack, and a call to C<pop>
is made. Even then, dereferencing the iterator is valid, but advancing it will
end prematurely, since the object was unlinked from the stack.

=head2 Queues

    #include <xrcu/queue.hpp>

This is a multi-producer, multi-consumer, FIFO queue. Much like the stack,
elements can be pushed or popped from it, but the order is different: They
will be retrieved in the same order they were inserted.

In XRCU, queues meet the requirements for the C++ concept of I<Container>.

=head3 Queue API

Queues are templated types, and they may be instantiated with a type T:

  template <class T>
  struct queue
    {
      typedef T value_type;
      typedef T& reference;
      typedef const T& const_reference;
      typedef T* pointer;
      typedef const T* const_pointer;
      typedef ptrdiff_t difference_type;
      typedef size_t size_type;

      struct iterator;
      struct const_iterator;
    };

As with the stack, a queue's iterator can be considered a I<forward> iterator,
and it's also a C<cs_guard>, so that a queue may be examined by an iterator,
at any time.

The following describes the public interface for C<queue<TE<gt>>

=over 4

=item queue ();

Default constructor. Initializes a queue to be empty.

=item template <class Integer> queue (Integer n, T value);

Initializes a queue to contain C<n> times C<value>.

=item template <class Iter> queue (Iter first, Iter last);

Initializes a queue with the values in the range [C<first>, C<last>).

=item queue (std::initializer_list<T> lst);

Initializes a queue with the values in C<lst>.

=item queue (const queue<T>& other);

Copy constructor. Initializes a queue with the values in C<other>.

=item queue (queue<T>&& other);

Move constructor. Takes ownership of the values in C<other>.

=item void push (const T& value);

Pushes C<value> to the top of the queue.

=item template <class ...Args> void emplace (Args&& ...args)

Same as C<push>, only the new item is constructed by calling the move
constructor with C<args...>.

=item optional<T> pop ();

Removes the first item from the queue, and returns an optional with that value.
If the queue is empty, the method returns an optional with no value.

=item optional<T> front () const;

Returns an optional with the first value from the queue. If the queue is empty,
an optional with no value is returned instead.

=item optional<T> back () const;

Returns an optional with the last value from the queue. If the queue is empty,
an optional with no value is returned instead.

=item size_type size () const;

Returns the size of the queue.

=item size_type max_size () const;

Returns the maximum allowed size for the queue.

=item void swap (queue<T>& other);

Swaps the contents of the queue with C<other>, but only if C<other> is not
the same object.

=item queue<T>& operator= (const queue<T>& other);

Assigns the contents of C<other> to the queue and returns C<*this>.

=item queue<T>& operator= (queue<T>&& other);

Move assignment. Takes ownership of the elements in C<other>. Returns C<*this>.

=item bool operator== (const queue<T>& other);

Compares the elements of the queue with the ones in C<other>. Returns true
if they are all equal.

=item bool operator!= (const queue<T>& other);

Compares the elements of the queue with the ones in C<other>. Returns true
if any two of them are not equal.

=item bool operator<  (const queue<T>& other) const;

=item bool operator>  (const queue<T>& other) const;

=item bool operator<= (const queue<T>& other) const;

=item bool operator>= (const queue<T>& other) const;

Lexicographically compares the elements of the queue with the ones in C<other>,
in a way that is equivalent to calling C<std::lexicographical_compare>.

=item void clear ();

Removes every element from the queue.

=item template <class T1, class T2> void assign (T1 x, T2 y);

Assigns to the queue the elements described by (C<x>, C<y>). They can be an
iterator range or a pair of (integer, value), as is the case with the queue's
constructor.

=item void assign (std::initializer_list<T> lst);

Assigns to the queue the elements in C<lst>.

=item iterator begin ();

=item const_iterator cbegin () const;

Returns an iterator to the first element of the queue.

=item iterator end ();

=item const_iterator cend () const;

Returns an iterator one past the last element of the queue.

=item iterator::iterator ();

Default constructor for queue iterators. Leaves the object in an invalid state.
Dereferencing or incrementing such an iterator has undefined behavior.

=item iterator::iterator (const iterator& other);

Copy constructor. Initializes the iterator to be equal to C<other>.

=item const T& iterator::operator* ();

=item const T& const_iterator::operator* () const;

Returns a const reference to the iterator's underlying object.

=item iterator& iterator::operator++ ();

Pre-increment operator. Moves the iterator forward, and returns it. If the
iterator was at the end of the stack, the results are undefined.

=item iterator iterator::operator++ (int);

Post-increment operator. Moves the iterator forward, and returns an iterator
equal to what it was before incrementing. If the iterator was at the end of
the stack, the results are undefined.

=item bool iterator::operator== (const iterator& other);

=item bool iterator::operator!= (const iterator& other);

Tests for iterator (in)equality.

=back

=head3 Implementation details

The design of the multi-producer, multi-consumer queue in XRCU is rather
simple and different from other implementations. It consists of a single array
that holds either the elements themselvers, or pointers to them, a decision
taken at compile time via type traits.

In addition to the vector, the queue is composed of three additional counters:
the capacity, a read cursor and a write cursor. The capacity is static, and
simply stores the raw number of elements that the queue can hold before it's
forced to reallocate. The read and write cursors and indices that indicate
at which positions the C<pop> and C<push> or C<emplace> operations may take
place.

When an element is pushed into a queue, the write cursor is checked against
the capacity. If the queue is full, then it must be reallocated. This is done
by marking each element with a special bit, allocating a new queue and then
copying over the elements. If there's room in the queue, the element is
atomically set via compare-and-swap, and if successful, the write cursor is
also updated atomically.

Popping elements proceeds in a similar way. The read cursor is compared against
the write cursor, and returns prematurely with failure if there are no more
elements to pop. Otherwise, it's atomically replaced with a special marker,
after which the read cursor is updated.

=head2 Skip lists

    #include <xrcu/skip_list.hpp>

A skip list is an associative container that holds a sorted set of unique
objects of a particular type. In addition to the I<Key> type, skip lists are
instantiated with a comparator type that allows them to determine ordering.

In XRCU, skip lists meet the requirements for the C++ concept of I<Container>,
and I<AssociativeContainer>.

=head3 Skip list API

Skip lists are templated types, defined in the following way:

    template <class T, class Cmp = std::less<T> >
    struct skip_list
      {
        typedef T value_type;
        typedef T key_type;
        typedef Cmp key_compare;
        typedef Cmp value_compare;
        typedef T& reference;
        typedef const T& const_reference;
        typedef T* pointer;
        typedef const T* const_pointer;
        typedef ptrdiff_t difference_type;
        typedef size_t size_type;

        struct iterator;
        struct const_iterator;
      };

The template parameter C<T> refers to the key type, whereas C<Cmp> is the
comparator type, which defaults to C<std::less>. Under most circumstances,
that is usually enough, but users may instantiate with any other type that
defines the C<operator()> which returns a boolean.

As shown above, skip list has iterators, but they are non-const in all cases,
for practicality reasons. Much like with stacks, skip list iterators are
I<forward> iterators, and an implicit C<cs_guard>.

The following describes the public interface for C<skip_list<T, CmpE<gt>>

=over 4

=item skip_list (Cmp c = Cmp (), unsigned int depth = ...);

Initializes the skip list with comparator C<c>, which defaults to a default
constructed value. Also takes a parameter indicating the maximum depth a skip
list node may have, which defaults to an implementation-specified value. Users
shouldn't need to change the latter, but it can be useful when tuning the
application for performance. As a general rule, a higher value implies more
memory usage, but better performance.

=item template <class Iter> skip_list (Iter first, Iter last, Cmp c = Cmp (), unsigned int depth = ...);

Initializes the skip list to the values in [C<first>, C<last>). The comparator
and depth parameters are the same as explained above.

=item skip_list (std::initializer_list<T> lst, Cmp c = Cmp (), unsigned int depth = ...);

Initializes the skip list to the values in C<lst>. The comparator and depth
parameters are the same as explained above.

=item skip_list (const skip_list<T, Cmp>& other);

Copy constructor. Initializes the skip list to hold the values in C<other>.

=item skip_list (skip_list<T, Cmp>&& other);

Move constructor. Takes ownership of the values in C<other>.

=item optional<T> find (const T& key) const;

Searches for an element equivalent to C<key> in the skip list, and returns an
optional with it as its value. If the key couldn't be found, the returned
optional has no value.

=item bool contains (const T& key) const;

Returns true if C<key> is present in the skip list.

=item bool insert (const T& key) const;

Inserts C<key> in the skip list. Returns true if the key wasn't present
previous to this call.

=item bool erase (const T& key) const;

Erases C<key> from the skip list. Returns true if the key was present previous
to this call.

=item optional<T> remove (const T& key);

Erases C<key> from the skip list and returns an optional with that element as
its value, if the key was present. Otherwise, the returned optional has no
value.

=item iterator begin ();

=item const_iterator cbegin () const;

Returns an iterator to the first element of the skip list.

=item iterator end ();

=item const_iterator cend () const;

Returns an iterator one past the last element of the skip list.

=item size_type size () const;

Returns the current size of the skip list.

=item size_type max_size () const;

Returns the maximum allowed size for a skip list.

=item bool empty () const;

Returns true if the skip list is empty (i.e: its size is 0).

=item template <class Iter> void assign (Iter first, Iter last);

Assigns to the skip list the elements in [C<first>, C<last>).

=item void assign (std::initializer_list<T> lst);

Assigns to the skip list the elements in C<lst>.

=item skip_list& operator= (const skip_list<T, Cmp>& other);

Assigns the elements in C<other> to the skip list. Returns C<*this>.

=item skip_list& operator= (skip_list<T, Cmp>&& other);

Move assignment. Takes ownership of the elements in C<other>. Returns C<*this>.

=item void swap (skip_list<T, Cmp>& other);

Swaps the contents of the skip list with C<other>, but only if C<other> is a
different object.

=item void clear ();

Removes every element from the skip list.

=back

=head3 Implementation details

In XRCU, skip lists are implemented as described in any piece of literature
that talks about them. Basically, every skip list of depth I<D> has a head
node that can be linked with up to other I<D> nodes. When performing a lookup
for an element, we start at the head node, and move horizontally until the
current element is equal or greater. If it's equal, we the lookup succeeded.
Otherwise, we move vertically to the next node, until we either find the
element, or we exhausted every node.

Insertions first perform a lookup, and if the search failed, we create a new
node with the specified value as its key. The number of linked nodes it will
have is determined in a semi-random way, but it must never exceed the maximum
depth of the skip list. After the node is created, we try to atomically link it
with its computed predecesors and successors (Returned in the same lookup call
we did before). If we succeed, the node is considered part of the skip list;
otherwise, we free it and retry.

Erasing an element proceeds in a similar fashion, only we check that the lookup
I<did not> fail, and then atomically relink the node's predecessors with the
node's own successors, thereby removing the key from the skip list.

Skip lists mantain an atomic word to store the length. However, this is used
slightly differently as one would think, because the lowest bit is used as an
ad-hoc lock bit in order to serialize concurrent calls to C<swap>.

=head2 Hash tables

    #include <xrcu/hash_table.hpp>

Hash tables are associative containers that map unique keys to values. Ordering
is unspecified for both keys and values. In addition to the key and value types,
hash tables are instantiated with a hashing type and an equality type;
callables that compute a I<hash value> for a given key, and one that tests for
equality, given two keys, respectively.

Additionally, hash tables mantain a I<load factor> that determines when a full
rehash is performed. A rehash implies moving all the key-value pairs into a new
location to reduce the overhead of most operations.

In XRCU, hash tables meet the requirements for the C++ concept of I<Container>
and I<UnorderedAssociativeContainer>.

=head3 Hash table API

As mentioned above, hash tables are template types, defined like this:

    template <class Key, class Val,
              class Equal = std::equal<Key>,
              class Hash = std::hash<Key> >
    struct hash_table
      {
        typedef Val mapped_type;
        typedef Key key_type;
        typedef std::pair<Key, Val> value_type;
        typedef Equal key_equal;
        typedef Hash hasher;
        typedef value_type& reference;
        typedef const value_type& const_reference;
        typedef value_type* pointer;
        typedef const value_type* const_pointer;
        typedef ptrdiff_t difference_type;
        typedef size_t size_type;

        struct iterator;
        struct const_iterator;
      };

The template parameters should be pretty self explanatory: They refer to the
key, value, equality and hashing types, in that order. The C<Equal> type has to
operate on two keys and return a boolean value that determines their equality,
whereas the C<Hash> type has to operate on keys and return unsigned integers.
Both are allowed to throw exceptions, although it is not really wise to do so.

Much like with skip lists, hash table iterators are always constant, and also
an implicit C<cs_guard>.

The following describes public interface for hash tables:

=over 4

=item hash_table (size_t size = 0, float lf = 0.85, Equal e = Equal (), Hash h = Hash ())

Initializes the hash table to hold C<size> elements, with a load factor of
C<lf>, using the equality predicate C<e> and the hasher C<h>. The load factor
must be in the range [0.4, 0.9]; if it's not, then it will silently be set to
the default of 0.85.

=item template <class It> hash_table (It first, It last, float lf = 0.85, Equal e = Equal (), Hash h = Hash ())

Initializes the hash table with the values between (C<first>, C<last>]. The
elements in that range must have two public members defined: C<first> and
C<second>, referring to the key and value of each element, respectively, and
in a similar fashion to what C<std::pair> does. The other parameters work as
with the previous constructor.

=item hash_table (std::initializer_list<std::pair<Key, Val> >, float lf = 0.85, Equal e = Equal (), Hash h = Hash ())

Initializes the hash table with the values in C<lst>. The rest of the parameters
work as with the previous constructors.

=item hash_table (const hash_table<Key, Val, Hash, Equal>& other);

Copy constructor. Initializes the hash table with the values in C<other>.

=item hash_table (hash_table<Key, Val, Hash, Equal>&& other);

Move constructor. Takes ownership of the values in C<other>.

=item size_t size () const;

Returns the hash table size.

=item size_t max_size () const;

Returns the maximum allowed size for a hash table.

=item bool empty () const;

Returns true if the hash table is empty.

=item optional<Val> find (const Key& key) const;

=item Val find (const Key& key, const Val& defl) const;

Look for C<key> in the hash table. If it's found, returns the value associated
to it. If it's not found, returns an empty optional in the first version, or
C<defl> in the second one.

=item bool contains (const Key& key) const;

Returns true if the key is present in the hash table.

=item bool insert (const Key& key, const Val& val);

Associates C<key> with C<val> in the hash table. Returns true if the value was
not present. Otherwise, the former value is replaced.

=item template <class Fn, class ...Args> bool update (const Key& key, Fn f, Args... args);

Updates the value associated to C<key> by calling C<f> with it and the rest of
the arguments. In other words, if C<val> is the value associated to C<key>,
calls C<f (val, args...)> and updates the value with the result. If no value was
present for C<key>, the function is called with a default constructed value
instead (as if by calling C<Val ()>).

Note that the function takes a I<reference> to the value, and so it's possible
to modify it in place. Additionally, if the function returns the very same
object that was passed (i.e: the same reference that it received), there won't
be any updates made. The function can also return a freshly made object, and
C<update> will work the same. The function may be called more than once if the
atomic updates fail.

Returns true if C<key> was present in the hash table before the call.

=item bool erase (const Key& key);

Erases C<key> from the hash table. Returns true if C<key> was present before
the call.

=item optional<Val> remove (const Key& key);

Removes C<key> from the hash table and returns the value that was associated
to if, if it was present.

=item void clear ();

Removes every element from the hash table.

=item template <class Iter> void assign (Iter first, Iter last);

Assigns the elements in [C<first>, C<last>) to the hash table. The same
restrictions as the range constructor apply here.

=item void assign (std::initializer_list<std::pair<Key, Val>> lst);

Assigns the elements in C<lst> to the hash table.

=item hash_table<Key, Val>& operator= (const hash_table<Key, Val>& other);

Assigns the elements in C<other> to the hash table. Returns C<*this>.

=item hash_table<Key, Val>& operator= (hash_table<Key, Val>&& other);

Move assignment. Takes ownership of the elements in C<other>. Returns C<*this>.

=item void swap (hash_table<Key, Val>& other);

Swaps the contents of the hash table with C<other>, but only if C<other> is a
different object.

=item iterator begin ();

=item const_iterator cbegin () const;

Returns an iterator to the first element of the hash table.

=item iterator end ();

=item const_iterator cend () const;

Returns an iterator one past the last element of the hash table.

=item iterator::iterator ();

Default constructor for hash table iterators. Leaves the object in an invalid
state. Dereferencing or incrementing such an iterator has undefined behavior.

=item iterator::iterator (const iterator& other);

Copy constructor. Initializes the iterator to be equal to C<other>.

=item Key iterator::key () const;

Returns the key for the iterator.

=item Val iterator::value () const;

Returns the value for the iterator.

=item std::pair<Key, Val> iterator::operator* () const;

Returns an C<std::pair> constructed with the key and value of the iterator.

=item iterator& iterator::operator++ ();

Pre-increment operator. Moves the iterator forward and returns it. If the
iterator was at the end, the results are undefined.

=item iterator iterator::operator++ (int);

Post-increment operator. Moves the iterator forward, and returns an iterator
equal to what it was before incrementing. If the iterator was at the end, the
results are undefined.

=item bool iterator::operator== (const iterator& other);

=item bool iterator::operator!= (const iterator& other);

Test for iterator (in)equality.

=back

=head3 Implementation details

Hash tables are somewhat complex, because the atomicity requirements force us
to do some rather convoluted things. To start off, a hash table is essentially
a vector of consecutive C<key> and C<value> pairs, with some special values
indicating C<free> and C<deleted> entries. However, since we can only operate
atomically on integers, we wrap any other type that is not integral into a
dynamically allocated pointer. This is done based on the template instantiation
and is figured out at compile time.

When a lookup is performed, the passed C<key> is hashed with the table's own
internal hasher and a hash code is computed. Using this code, we produce an
index into the vector and then we iterate over it, looking for a key that
matches, or until we find a free entry, in which case the lookup fails.

Insertions are a bit more complicated. We start by performing a lookup, as
described above, and then determine what to do based on the result: If the
lookup came up empty, then we need to insert both the key I<and> the value;
otherwise, we only need to update the value.

If only the value needs to be updated (because the key is already present),
then the insertion simply consists of atomically swapping the old value with
the new one, and, if succesful, finalizing the old value as well.

If we need to insert the key as well, first we decrement an internal counter
that keeps track of how many additional elements we can insert before a rehash
is triggered. If the counter has already reached zero, then we have to rehash
before inserting the key and value. If no rehash is needed, or after it's done,
both the key and value are updated atomically (Some platforms allow us to do it
in a single step, otherwise the operations are done sequentially).

At this point, we have to pause to make a sort of confession: Hash tables as
implemented in XRCU are not I<entirely> lock free, because rehashing actually
takes an internal lock (I know, you're crushed). This not because of any
intrinsical limitation, but rather, to make things easier. Since rehashes are
probably the least common of the operations, we figured it wasn't that big of
a deal anyways.

Going back to rehashes, these are the most expensive operations. As mentioned,
they start off by acquiring an internal lock, and then they iterate over the
full vector, marking the C<value> indexes with a special bit that signals that
the table is being rehashed. When this bit is set, insertions and erasures are
forbidden (They both check for this bit before proceeding).

Erasures are pretty simple in comparison. They obviously perform a lookup on
the key, and if it didn't come up empty, they atomically swap out the value
for the special C<empty> constant. Afterwards, they can mutate the key entry
without atomicity (Because erased entries cannot be reused).

=head1 BUGS

All implemented containers use standard operators C<new> and C<delete> to
perform memory (de)allocations. There's no way to specify custom allocators
yet, although it's planned in the future.

=head1 AUTHORS

Agustina Arzille (avarzille@riseup.net) and Luciano Lo Giudice (lmlogiudice@gmail.com)


code:
#include "hash_table.hpp"
#include <new>
#include <thread>
#include <cstdint>

namespace xrcu
{

namespace detail
{

inline ht_vector* ht_vector_make (size_t n)
{
  void *p = ::operator new (sizeof (ht_vector) + n * sizeof (uintptr_t));
  return (new (p) ht_vector ((uintptr_t *)((char *)p + sizeof (ht_vector))));
}

void ht_vector::safe_destroy ()
{
  ::operator delete (this);
}

static const size_t PRIMES[] =
{
  0xb, 0x25, 0x71, 0x15b, 0x419, 0xc4d, 0x24f5, 0x6ee3, 0x14cb3, 0x3e61d,
  0xbb259, 0x23170f, 0x694531, 0x13bcf95, 0x3b36ec3, 0xb1a4c4b, 0x214ee4e3,
  0x63ecaead
#if SIZE_MAX > 0xffffffffu
  , 0x12bc60c09ull, 0x38352241dull, 0xa89f66c5bull, 0x1f9de34513ull,
  0x5ed9a9cf3bull, 0x11c8cfd6db5ull, 0x355a6f84921ull, 0xa00f4e8db65ull,
  0x1e02deba9233ull, 0x5a089c2fb69bull, 0x10e19d48f23d3ull, 0x32a4d7dad6b7dull,
  0x97ee879084279ull, 0x1c7cb96b18c76dull, 0x55762c414a564bull,
  0x1006284c3df02e3ull, 0x301278e4b9d08abull, 0x90376aae2d71a05ull,
  0x1b0a6400a8854e11ull, 0x511f2c01f98fea35ull
#endif
};

size_t find_hsize (size_t size, float mvr, size_t& pidx)
{
  intptr_t i1 = 0, i2 = sizeof (PRIMES) / sizeof (PRIMES[0]);
  while (i1 < i2)
    {
      intptr_t step = (i1 + i2) >> 1;
      if (PRIMES[step] < size)
        i1 = step + 1;
      else
        i2 = step;
    }

  pidx = i1;
  return ((size_t)(PRIMES[i1] * mvr));
}

ht_vector* make_htvec (size_t pidx, uintptr_t key, uintptr_t val)
{
  size_t entries = PRIMES[pidx], tsize = table_idx (entries);
#ifdef XRCU_HAVE_XATOMIC_DCAS
  auto ret = ht_vector_make (tsize + 1);

  // Ensure correct alignment for double-width CAS.
  if ((uintptr_t)ret->data % (2 * sizeof (uintptr_t)) != 0)
    ++ret->data;
#else
  auto ret = ht_vector_make (tsize);
#endif

  for (size_t i = 0; i < tsize; i += 2)
    ret->data[i] = key, ret->data[i + 1] = val;

  ret->entries = entries;
  ret->pidx = pidx;

  return (ret);
}

} // namespace detail

} // namespace xrcu

repo_name: avarzille/xrcu

license: mit


code:
#ifndef __XRCU_HASHTABLE_HPP__
#define __XRCU_HASHTABLE_HPP__   1

#include "xrcu.hpp"
#include "xatomic.hpp"
#include "optional.hpp"
#include "lwlock.hpp"
#include "utils.hpp"
#include <atomic>
#include <functional>
#include <type_traits>
#include <initializer_list>

namespace std
{
  struct forward_iterator_tag;
}

namespace xrcu
{

namespace detail
{

inline constexpr size_t table_idx (size_t idx)
{
  return (idx * 2);
}

struct alignas (uintptr_t) ht_vector : public finalizable
{
  uintptr_t *data;
  size_t entries;
  size_t pidx;
  std::atomic<size_t> nelems { 0 };

  ht_vector (uintptr_t *ep) : data (ep) {}

  void safe_destroy ();

  size_t size () const
    {
      return (table_idx (this->entries));
    }
};

inline size_t
secondary_hash (size_t hval)
{
  static const size_t keys[] = { 2, 3, 5, 7 };
  return (keys[hval % (sizeof (keys) / sizeof (keys[0]))]);
}

extern size_t find_hsize (size_t size, float ldf, size_t& pidx);

extern ht_vector* make_htvec (size_t pidx, uintptr_t key, uintptr_t val);

struct ht_sentry
{
  lwlock *lock;
  uintptr_t xbit;
  ht_vector *vector = nullptr;

  ht_sentry (lwlock *lp, uintptr_t xb) : lock (lp), xbit (~xb)
    {
      this->lock->acquire ();
    }

  ~ht_sentry ()
    {
      if (this->vector)
        for (size_t i = table_idx (0) + 1; i < this->vector->size (); i += 2)
          xatomic_and (&this->vector->data[i], this->xbit);

      this->lock->release ();
    }
};

template <class Ktraits, class Vtraits>
struct ht_iter : public cs_guard
{
  const ht_vector *vec = nullptr;
  size_t idx = 0;
  uintptr_t c_key;
  uintptr_t c_val;
  bool valid = false;

  typedef std::forward_iterator_tag iterator_category;

  void _Init (const ht_vector *vp)
    {
      this->vec = vp;
      this->_Adv ();
    }

  ht_iter ()
    {
    }

  ht_iter (const ht_iter<Ktraits, Vtraits>& right) :
      vec (right.vec), idx (right.idx), c_key (right.c_key),
      c_val (right.c_val), valid (right.valid)
    {
    }

  ht_iter (ht_iter<Ktraits, Vtraits>&& right) :
      vec (right.vec), idx (right.idx), c_key (right.c_key),
      c_val (right.c_val), valid (right.valid)
    {
      right.vec = nullptr;
      right.idx = 0;
      right.valid = false;
    }

  void _Adv ()
    {
      this->valid = false;
      for (; this->idx < this->vec->size (); )
        {
          this->c_key = this->vec->data[idx + 0];
          this->c_val = this->vec->data[idx + 1] & ~Vtraits::XBIT;

          this->idx += 2;
          if ((this->c_key & ~Ktraits::XBIT) != Ktraits::FREE &&
              this->c_val != Vtraits::FREE && this->c_val != Vtraits::DELT)
            {
              this->valid = true;
              break;
            }
        }
    }

  bool operator== (const ht_iter<Ktraits, Vtraits>& right) const
    {
      return ((!this->valid && !right.valid) ||
        (this->vec == right.vec && this->idx == right.idx));
    }

  bool operator!= (const ht_iter<Ktraits, Vtraits>& right)
    {
      return (!(*this == right));
    }
};

struct ht_inserter
{
  uintptr_t value;
  ht_inserter (uintptr_t v) : value (v) {}

  uintptr_t call0 () const noexcept
    {
      return (this->value);
    }

  uintptr_t call1 (uintptr_t) const noexcept
    {
      return (this->value);
    }

  void free (uintptr_t) {}
};

} // namespace detail

template <class KeyT, class ValT,
  class EqFn = std::equal_to<KeyT>,
  class HashFn = std::hash<KeyT> >
struct hash_table
{
  typedef detail::wrapped_traits<(sizeof (KeyT) < sizeof (uintptr_t) &&
      std::is_integral<KeyT>::value) || (std::is_pointer<KeyT>::value &&
      alignof (KeyT) >= 8), KeyT> key_traits;

  typedef detail::wrapped_traits<(sizeof (ValT) < sizeof (uintptr_t) &&
      std::is_integral<ValT>::value) || (std::is_pointer<ValT>::value &&
      alignof (ValT) >= 8), ValT> val_traits;

  typedef hash_table<KeyT, ValT, EqFn, HashFn> self_type;
  typedef KeyT key_type;
  typedef ValT mapped_type;
  typedef std::pair<KeyT, ValT> value_type;
  typedef EqFn key_equal;
  typedef HashFn hasher;
  typedef value_type& reference;
  typedef const value_type& const_reference;
  typedef value_type* pointer;
  typedef const value_type* const_pointer;
  typedef ptrdiff_t difference_type;
  typedef size_t size_type;

  detail::ht_vector *vec;
  EqFn eqfn;
  HashFn hashfn;
  float loadf = 0.85f;
  std::atomic<intptr_t> grow_limit;
  lwlock lock;

  void _Set_loadf (float ldf)
    {
      if (ldf >= 0.4f && ldf <= 0.9f)
        this->loadf = ldf;
    }

  float load_factor (float ldf)
    {
      this->lock.acquire ();
      float ret = this->loadf;
      this->_Set_loadf (ldf);
      this->lock.release ();
      return (ret);
    }

  float load_factor () const
    {
      return (this->loadf);
    }

  void _Init (size_t size, float ldf, EqFn e, HashFn h)
    {
      this->_Set_loadf (ldf);
      size_t pidx, gt = detail::find_hsize (size, this->loadf, pidx);

      this->vec = detail::make_htvec (pidx,
        key_traits::FREE, val_traits::FREE);
      this->eqfn = e;
      this->hashfn = h;
      this->grow_limit.store (gt, std::memory_order_relaxed);
      this->vec->nelems.store (0, std::memory_order_relaxed);
    }

  hash_table (size_t size = 0, float ldf = 0.85f,
      EqFn e = EqFn (), HashFn h = HashFn ())
    {
      this->_Init (size, ldf, e, h);
    }

  template <class Iter>
  hash_table (Iter first, Iter last, float ldf = 0.85f,
      EqFn e = EqFn (), HashFn h = HashFn ())
    {
      this->_Init (0, ldf, e, h);
      for (; first != last; ++first)
        this->insert ((*first).first, (*first).second);
    }

  hash_table (std::initializer_list<std::pair<KeyT, ValT> > lst,
      float ldf = 0.85f, EqFn e = EqFn (), HashFn h = HashFn ()) :
      hash_table (lst.begin (), lst.end (), ldf, e, h)
    {
    }

  hash_table (const self_type& right) :
      hash_table (right.begin (), right.end ())
    {
    }

  hash_table (self_type&& right)
    {
      this->vec = right.vec;
      this->eqfn = right.eqfn;
      this->hashfn = right.hashfn;
      this->loadf = right.loadf;
      this->grow_limit.store ((intptr_t)(this->loadf * this->size ()),
                              std::memory_order_relaxed);

      right.vec = nullptr;
    }

  size_t size () const
    {
      cs_guard g;
      return (this->vec->nelems.load (std::memory_order_relaxed));
    }

  size_t max_size () const
    {
      size_t out;
      return (detail::find_hsize (~(size_t)0, 0.85f, out));
    }

  bool empty () const
    {
      return (this->size () == 0);
    }

  size_t _Probe (const KeyT& key, const detail::ht_vector *vp,
      bool put_p, bool& found) const
    {
      size_t code = this->hashfn (key);
      size_t entries = vp->entries;
      size_t idx = code % entries;
      size_t vidx = detail::table_idx (idx);

      found = false;
      uintptr_t k = vp->data[vidx];

      if (k == key_traits::FREE)
        return (put_p ? (found = true, vidx) : (size_t)-1);
      else if (k != key_traits::DELT &&
          this->eqfn (key_traits::get (k), key))
        return (vidx);

      for (size_t initial = idx, sec = detail::secondary_hash (code) ; ; )
        {
          if ((idx += sec) >= entries)
            idx -= entries;

          if (idx == initial)
            return ((size_t)-1);

          vidx = detail::table_idx (idx);
          k = vp->data[vidx];

          if (k == key_traits::FREE)
            return (put_p ? (found = true, vidx) : (size_t)-1);
          else if (k != key_traits::DELT &&
              this->eqfn (key_traits::get (k), key))
            return (vidx);
        }
    }

  size_t _Probe (const KeyT& key, const detail::ht_vector *vp, bool put_p) const
    {
      bool unused;
      return (this->_Probe (key, vp, put_p, unused));
    }

  size_t _Gprobe (uintptr_t key, detail::ht_vector *vp)
    {
      size_t code = this->hashfn (key_traits::get (key));
      size_t entries = vp->entries;
      size_t idx = code % entries;
      size_t vidx = detail::table_idx (idx);

      if (vp->data[vidx] == key_traits::FREE)
        return (vidx);

      for (size_t sec = detail::secondary_hash (code) ; ; )
        {
          if ((idx += sec) >= entries)
            idx -= entries;

          vidx = detail::table_idx (idx);
          if (vp->data[vidx] == key_traits::FREE)
            return (vidx);
        }
    }

  void _Rehash ()
    {
      detail::ht_sentry s (&this->lock, val_traits::XBIT);

      if (this->grow_limit.load (std::memory_order_relaxed) <= 0)
        {
          auto old = this->vec;
          auto np = detail::make_htvec (old->pidx + 1,
              key_traits::FREE, val_traits::FREE);
          size_t nelem = 0;

          s.vector = old;

          for (size_t i = detail::table_idx (0); i < old->size (); i += 2)
            {
              uintptr_t key = old->data[i];
              uintptr_t val = xatomic_or (&old->data[i + 1], val_traits::XBIT);

              if (key != key_traits::FREE && key != key_traits::DELT &&
                  val != val_traits::FREE && val != val_traits::DELT)
                {
                  size_t nidx = this->_Gprobe (key, np);
                  np->data[nidx + 0] = key;
                  np->data[nidx + 1] = val;
                  ++nelem;
                }
            }

          s.vector = nullptr;

          np->nelems.store (nelem, std::memory_order_relaxed);
          this->grow_limit.store ((intptr_t)(np->entries * this->loadf) -
                                  nelem, std::memory_order_relaxed);
          std::atomic_thread_fence (std::memory_order_release);

          /* At this point, another thread may decrement the growth limit
           * from the wrong vector. That's fine, it just means we'll move
           * the table sooner than necessary. */
          this->vec = np;
          finalize (old);
        }
    }

  uintptr_t _Find (const KeyT& key) const
    {
      auto vp = this->vec;
      size_t idx = this->_Probe (key, vp, false);
      return (idx == (size_t)-1 ? val_traits::DELT :
        vp->data[idx + 1] & ~val_traits::XBIT);
    }

  optional<ValT> find (const KeyT& key) const
    {
      cs_guard g;
      uintptr_t val = this->_Find (key);
      return (val == val_traits::DELT ? optional<ValT> () :
        optional<ValT> (val_traits::get (val)));
    }

  ValT find (const KeyT& key, const ValT& dfl) const
    {
      cs_guard g;
      uintptr_t val = this->_Find (key);
      return (val == val_traits::DELT ? dfl : val_traits::get (val));
    }

  bool contains (const KeyT& key) const
    {
      cs_guard g;
      return (this->_Find (key) != val_traits::DELT);
    }

  bool _Decr_limit ()
    {
      if (this->grow_limit.load (std::memory_order_relaxed) <= 0)
        return (false);

      this->grow_limit.fetch_sub (1, std::memory_order_acq_rel);
      return (true);
    }

  template <class Fn, class ...Args>
  bool _Upsert (uintptr_t k, const KeyT& key, Fn f, Args... args)
    {
      cs_guard g;

      while (true)
        {
          auto vp = this->vec;
          uintptr_t *ep = vp->data;
          bool found;
          size_t idx = this->_Probe (key, vp, true, found);

          if (!found)
            {
              uintptr_t tmp = ep[idx + 1];
              if (tmp != val_traits::DELT && tmp != val_traits::FREE &&
                  (tmp & val_traits::XBIT) == 0)
                {
                  uintptr_t v = f.call1 (tmp, args...);
                  if (v == tmp ||
                      xatomic_cas_bool (ep + idx + 1, tmp, v))
                    {
                      key_traits::free (k);
                      if (v != tmp)
                        val_traits::destroy (tmp);
                      return (found);
                    }

                  f.free (v);
                  continue;
                }
            }
          else if (this->_Decr_limit ())
            {
              /* NOTE: If we fail here, then the growth threshold will end up
               * too small. This simply means that we may have to rehash sooner
               * than absolutely necessary, which is harmless. On the other
               * hand, we must NOT try to reincrement the limit back, because
               * it risks ending up too big, which can be harmful if, for
               * example, a rehash is triggered before the increment. */

              uintptr_t v = f.call0 (args...);
#ifdef XRCU_HAVE_XATOMIC_DCAS
              if (xatomic_dcas_bool (&ep[idx], key_traits::FREE,
                                     val_traits::FREE, k, v))
#else
              if (xatomic_cas_bool (ep + idx + 0, key_traits::FREE, k) &&
                  xatomic_cas_bool (ep + idx + 1, val_traits::FREE, v))
#endif
                {
                  this->vec->nelems.fetch_add (1, std::memory_order_acq_rel);
                  return (found);
                }

              f.free (v);
              continue;
            }

          // The table was being rehashed - retry.
          this->_Rehash ();
        }
    }

  bool insert (const KeyT& key, const ValT& val)
    {
      uintptr_t k = key_traits::make (key), v = val_traits::XBIT;

      try
        {
          v = val_traits::make (val);
          return (this->_Upsert (k, key, detail::ht_inserter (v)));
        }
      catch (...)
        {
          key_traits::free (k);
          if (v != val_traits::XBIT)
            val_traits::free (v);

          throw;
        }
    }

  template <class Fn, class Vtraits>
  struct _Updater
    {
      Fn fct;

      _Updater (Fn f) : fct (f) {}

      template <class ...Args>
      uintptr_t call0 (Args ...args)
        { // Call function with default-constructed value and arguments.
          auto tmp = (typename Vtraits::value_type ());
          auto&& rv = this->fct (tmp, args...);
          return (Vtraits::make (rv));
        }

      template <class ...Args>
      uintptr_t call1 (uintptr_t x, Args ...args)
        { // Call function with stored value and arguments.
          auto&& tmp = Vtraits::get (x);
          auto&& rv = this->fct (tmp, args...);
          return (Vtraits::XBIT == 1 &&
            &rv == &tmp ? x : Vtraits::make (rv));
        }

      void free (uintptr_t x)
        {
          Vtraits::free (x);
        }
    };

  template <class Fn, class ...Args>
  bool update (const KeyT& key, Fn f, Args... args)
    {
      uintptr_t k = key_traits::make (key);

      try
        {
          return (this->_Upsert (k, key,
                                 _Updater<Fn, val_traits> (f), args...));
        }
      catch (...)
        {
          key_traits::free (k);
          throw;
        }
    }

  bool _Erase (const KeyT& key, optional<ValT> *outp = nullptr)
    {
      cs_guard g;

      while (true)
        {
          auto vp = this->vec;
          uintptr_t *ep = vp->data;
          size_t idx = this->_Probe (key, vp, false);

          if (idx == (size_t)-1)
            return (false);

          uintptr_t oldk = ep[idx], oldv = ep[idx + 1];

          if ((oldv & val_traits::XBIT) == 0)
            {
              if (oldk == key_traits::DELT || oldk == key_traits::FREE ||
                  oldv == val_traits::DELT)
                return (false);
              else if (!xatomic_cas_bool (ep + idx + 1,
                                          oldv, val_traits::DELT))
                continue;

              this->vec->nelems.fetch_sub (1, std::memory_order_acq_rel);
              // Safe to set the key without atomic ops.
              ep[idx] = key_traits::DELT;
              key_traits::destroy (oldk);
              val_traits::destroy (oldv);

              if (outp)
                *outp = val_traits::get (oldv);

              return (true);
            }

          // The table was being rehashed - retry.
          this->_Rehash ();
        }
    }

  bool erase (const KeyT& key)
    {
      return (this->_Erase (key));
    }

  optional<ValT> remove (const KeyT& key)
    {
      optional<ValT> ret;
      this->_Erase (key, &ret);
      return (ret);
    }

  struct iterator : public detail::ht_iter<key_traits, val_traits>
    {
      typedef detail::ht_iter<key_traits, val_traits> base_type;

      iterator () : base_type ()
        {
        }

      iterator (const self_type& self) : base_type ()
        {
          this->_Init (self.vec);
        }

      iterator (const iterator& right) : base_type (right)
        {
        }

      iterator (iterator&& right) : base_type (std::move (right))
        {
        }

      KeyT key () const
        {
          return (key_traits::get (this->c_key));
        }

      ValT value () const
        {
          return (val_traits::get (this->c_val));
        }

      iterator& operator++ ()
        {
          this->_Adv ();
          return (*this);
        }

      iterator operator++ (int)
        {
          iterator ret = *this;
          this->_Adv ();
          return (ret);
        }

      std::pair<KeyT, ValT> operator* () const
        {
          return (std::pair<KeyT, ValT> (this->key (), this->value ()));
        }
    };

  typedef iterator const_iterator;

  iterator begin () const
    {
      return (iterator (*this));
    }

  iterator end () const
    {
      return (iterator ());
    }

  iterator cbegin () const
    {
      return (this->begin ());
    }

  iterator cend () const
    {
      return (this->end ());
    }

  void _Assign_vector (detail::ht_vector *nv, intptr_t gt)
    {
      // First step: Lock the table.
      this->lock.acquire ();
      auto prev = this->vec;

      // Second step: Finalize every valid key/value pair.
      for (size_t i = detail::table_idx (0) + 1; i < prev->size (); i += 2)
        {
          uintptr_t v = xatomic_or (&prev->data[i], val_traits::XBIT);
          if (v != val_traits::FREE && v != val_traits::DELT)
            {
              key_traits::destroy (prev->data[i - 1]);
              val_traits::destroy (v);
            }
        }

      // Third step: Set up the new vector and parameters.
      this->grow_limit.store (gt, std::memory_order_relaxed);
      std::atomic_thread_fence (std::memory_order_release);

      this->vec = nv;
      this->lock.release ();
      finalize (prev);
    }

  void clear ()
    {
      size_t pidx, gt = detail::find_hsize (0, this->loadf, pidx);
      auto nv = detail::make_htvec (pidx, key_traits::FREE, val_traits::FREE);
      this->_Assign_vector (nv, gt);
    }

  template <class Iter>
  void assign (Iter first, Iter last)
    {
      self_type tmp (first, last, 0, this->loadf);
      this->_Assign_vector (tmp.vec,
        tmp.grow_limit.load (std::memory_order_relaxed));
      tmp.vec = nullptr;
    }

  void assign (std::initializer_list<std::pair<KeyT, ValT> > lst)
    {
      this->assign (lst.begin (), lst.end ());
    }

  self_type& operator= (const self_type& right)
    {
      this->assign (right.begin (), right.end ());
      return (*this);
    }

  self_type& operator= (self_type&& right)
    {
      this->_Assign_vector (right.vec,
        right.grow_limit.load (std::memory_order_relaxed));
      this->loadf = right.loadf;
      right.vec = nullptr;
      return (*this);
    }

  void swap (self_type& right)
    {
      if (this == &right)
        return;

      detail::ht_sentry s1 (&this->lock, ~val_traits::XBIT);
      detail::ht_sentry s2 (&right.lock, ~val_traits::XBIT);

      // Prevent further insertions (still allows deletions).
      this->grow_limit.store (0, std::memory_order_release);
      right.grow_limit.store (0, std::memory_order_release);

      std::swap (this->vec, right.vec);
      std::swap (this->eqfn, right.eqfn);
      std::swap (this->hashfn, right.hashfn);
      std::swap (this->loadf, right.loadf);

      this->grow_limit.store ((intptr_t)(this->loadf *
        this->vec->entries) - this->size (), std::memory_order_release);
      right.grow_limit.store ((intptr_t)(right.loadf *
        right.vec->entries) - right.size (), std::memory_order_release);
    }

  ~hash_table ()
    {
      if (!this->vec)
        return;

      for (size_t i = detail::table_idx (0); i < this->vec->size (); i += 2)
        {
          uintptr_t k = this->vec->data[i] & ~key_traits::XBIT;
          if (k == key_traits::FREE || k == key_traits::DELT)
            continue;

          key_traits::free (k);
          val_traits::free (this->vec->data[i + 1] & ~val_traits::XBIT);
        }

      this->vec->safe_destroy ();
      this->vec = nullptr;
    }
};

} // namespace xrcu

namespace std
{

template <class KeyT, class ValT, class EqFn, class HashFn>
void swap (xrcu::hash_table<KeyT, ValT, EqFn, HashFn>& left,
           xrcu::hash_table<KeyT, ValT, EqFn, HashFn>& right)
{
  left.swap (right);
}

} // namespace std

#endif


code:
#include "lwlock.hpp"
#include "xatomic.hpp"

#if (defined (linux) || defined (__linux) || defined (__linux__)) &&   \
    defined (__BYTE_ORDER__)

#include <linux/futex.h>
#include <unistd.h>
#include <syscall.h>

#ifndef FUTEX_PRIVATE_FLAG
#  define FUTEX_PRIVATE_FLAG   0
#endif

template <bool Wide>
void* futex_ptr (void *ptr)
{
  return (ptr);
}

template <>
void* futex_ptr<true> (void *ptr)
{
#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
  return (ptr);
#else
  return ((int *)ptr + 1);
#endif
}

static inline void
lwlock_acquire (uintptr_t *ptr)
{
  const int MAX_SPINS = 1000;

  if (xrcu::xatomic_cas_bool (ptr, 0, 1))
    return;

  while (true)
    {
      for (int i = 0; i < MAX_SPINS && *ptr != 0; ++i)
        xrcu::xatomic_spin_nop ();

      if (xrcu::xatomic_swap (ptr, 2) == 0)
        return;

      syscall (SYS_futex, futex_ptr<sizeof (int) < sizeof (void *)> (ptr),
        (long)(FUTEX_WAIT | FUTEX_PRIVATE_FLAG), 2l, 0ul);
    }
}

static inline void
lwlock_release (uintptr_t *ptr)
{
  if (xrcu::xatomic_swap (ptr, 0) != 1)
    syscall (SYS_futex, futex_ptr<sizeof (int) < sizeof (void *)> (ptr),
      (long)(FUTEX_WAKE | FUTEX_PRIVATE_FLAG), 1ul);
}

#else

#include <thread>
#include <atomic>

static inline void
lwlock_acquire (uintptr_t *ptr)
{
  const int MAX_SPINS = 1000;

  while (true)
    {
      if (xrcu::xatomic_cas_bool (ptr, 0, 1))
        return;

      for (int i = 0; i < MAX_SPINS && *ptr != 0; ++i)
        xrcu::xatomic_spin_nop ();

      if (xrcu::xatomic_swap (ptr, 1) == 0)
        break;

      std::this_thread::sleep_for (std::chrono::milliseconds (1));
    }
}

static inline void
lwlock_release (uintptr_t *ptr)
{
  *ptr = 0;
  std::atomic_thread_fence (std::memory_order_release);
}

#endif

namespace xrcu
{

void lwlock::acquire ()
{
  lwlock_acquire (&this->lock);
}

void lwlock::release ()
{
  lwlock_release (&this->lock);
}

} // namespace xrcu


code:
#ifndef __XRCU_LWLOCK_HPP__
#define __XRCU_LWLOCK_HPP__   1

#include <cstdint>

namespace xrcu
{

/* Lightweight lock used when including <mutex> is overkill. Should be just as
 * performant in certain platforms (i.e: Those that support something akin to
 * Linux's futexes), but much lighter space-wise. */
struct lwlock
{
  uintptr_t lock = 0;

  void acquire ();
  void release ();

  lwlock () = default;

  lwlock (const lwlock&) = delete;
  void operator= (const lwlock&) = delete;
};

}

#endif


code:
#ifndef __XRCU_OPTIONAL_HPP__
#define __XRCU_OPTIONAL_HPP__   1

#include <utility>

namespace xrcu
{

template <class T>
struct optional
{
  T *xptr;
  alignas (alignof (T)) char buf[sizeof (T)];

  T* _Ptr ()
    {
      return ((T *)&this->buf[0]);
    }

  const T* _Ptr () const
    {
      return ((const T *)&this->buf[0]);
    }

  void _Init (const T& value)
    {
      new (this->_Ptr ()) T (value);
      this->xptr = this->_Ptr ();
    }

  void _Init (T&& value)
    {
      new (this->_Ptr ()) T (std::forward<T&&> (value));
      this->xptr = this->_Ptr ();
    }

  optional () : xptr (nullptr)
    {
    }

  optional (const T& value)
    {
      this->_Init (value);
    }

  optional (const optional<T>& right) : xptr (nullptr)
    {
      if (right.xptr)
        this->_Init (*right);
    }

  optional (T&& value)
    {
      this->_Init (std::forward<T&&> (value));
    }

  optional (optional<T>&& right) : xptr (nullptr)
    {
      if (right.xptr)
        {
          this->_Init (std::forward<T&&> (*right));
          right.xptr = nullptr;
        }
    }

  T& operator* ()
    {
      return (*this->xptr);
    }

  const T& operator* () const
    {
      return (*this->xptr);
    }

  T* operator-> ()
    {
      return (this->xptr);
    }

  const T* operator-> () const
    {
      return (this->xptr);
    }

  bool has_value () const
    {
      return (this->xptr != nullptr);
    }

  void reset ()
    {
      if (!this->xptr)
        return;

      this->xptr->~T ();
      this->xptr = nullptr;
    }

  optional<T>& operator= (const optional<T>& right)
    {
      if (&right == this)
        ;
      else if (!right.xptr)
        this->reset ();
      else if (!this->xptr)
        this->_Init (*right);
      else
        **this = *right;

      return (*this);
    }

  optional<T>& operator= (optional<T>&& right)
    {
      if (&right == this)
        return (*this);

      if (!right.xptr)
        this->reset ();
      else if (!this->xptr)
        this->_Init (std::forward<T&&> (*right));
      else
        **this = std::forward<T&&> (*right);

      right.xptr = nullptr;
      return (*this);
    }

  optional<T>& operator= (const T& value)
    {
      if (!this->xptr)
        this->_Init (value);
      else
        **this = value;

      return (*this);
    }

  optional<T>& operator= (T&& value)
    {
      if (!this->xptr)
        this->_Init (std::forward<T&&> (value));
      else
        **this = std::forward<T&&> (value);

      return (*this);
    }

  ~optional ()
    {
      this->reset ();
    }
};

}   // namespace xrcu

#endif


code:
#include "queue.hpp"
#include <new>

namespace xrcu
{

namespace detail
{

q_data* q_data::make (size_t cnt, uintptr_t empty)
{
  size_t ns = cnt + 1;
  q_data *ret = (q_data *)::operator new (sizeof (*ret) +
                                          ns * sizeof (uintptr_t));

  new (ret) q_data ();
  ret->ptrs = (uintptr_t *)(ret + 1);
  ret->cap = cnt;

  for (uintptr_t *p = ret->ptrs; p != ret->ptrs + ns; )
    *p++ = empty;

  ret->wr_idx.store (0, std::memory_order_relaxed);
  ret->rd_idx.store (0, std::memory_order_relaxed);
  return (ret);
}

void q_data::safe_destroy ()
{
  ::operator delete (this);
}

} // namespace detail

} // namespace xrcu


code:
#ifndef __XRCU_QUEUE_HPP__
#define __XRCU_QUEUE_HPP__   1

#include "xrcu.hpp"
#include "optional.hpp"
#include "xatomic.hpp"
#include "utils.hpp"
#include <cstddef>
#include <atomic>
#include <type_traits>
#include <initializer_list>

namespace std
{
  struct forward_iterator_tag;
}

namespace xrcu
{

namespace detail
{

static inline uint32_t upsize (uint32_t x)
{
  x |= x >> 1;
  x |= x >> 2;
  x |= x >> 4;
  x |= x >> 8;
  x |= x >> 16;
  return (x + 1);
}

static inline uint64_t upsize (uint64_t x)
{
  x |= x >> 1;
  x |= x >> 2;
  x |= x >> 4;
  x |= x >> 8;
  x |= x >> 16;
  x |= x >> 32;
  return (x + 1);
}

struct alignas (uintptr_t) q_data : public finalizable
{
  uintptr_t *ptrs;
  size_t cap;
  std::atomic<size_t> wr_idx;
  std::atomic<size_t> rd_idx;

  static q_data* make (size_t cnt, uintptr_t empty);

  size_t _Wridx () const
    {
      return (this->wr_idx.load (std::memory_order_relaxed));
    }

  size_t _Rdidx () const
    {
      return (this->rd_idx.load (std::memory_order_relaxed));
    }

  bool push (uintptr_t val, uintptr_t xbit, uintptr_t empty)
    {
      while (true)
        {
          size_t curr = this->_Wridx ();
          if (curr >= this->cap)
            return (false);

          uintptr_t xv = this->ptrs[curr];
          if ((xv & xbit) != 0)
            return (false);
          else if (xv == empty &&
              xatomic_cas_bool (&this->ptrs[curr], xv, val))
            {
              this->wr_idx.fetch_add (1, std::memory_order_acq_rel);
              return (true);
            }

          xatomic_spin_nop ();
        }
    }

  uintptr_t pop (uintptr_t xbit, uintptr_t dfl)
    {
      while (true)
        {
          size_t curr = this->_Rdidx ();
          if (curr >= this->_Wridx ())
            return (dfl);

          uintptr_t rv = this->ptrs[curr];
          if ((rv & xbit) != 0 || rv == dfl)
            return (xbit);
          else if (xatomic_cas_bool (&this->ptrs[curr], rv, dfl))
            {
              this->rd_idx.fetch_add (1, std::memory_order_relaxed);
              return (rv);
            }

          xatomic_spin_nop ();
        }
    }

  uintptr_t front () const
    {
      return (this->ptrs[this->_Rdidx ()]);
    }

  uintptr_t back () const
    {
      size_t idx = this->_Wridx ();
      if (idx == 0)
        return (this->ptrs[this->cap]);

      return (this->ptrs[idx - 1]);
    }

  size_t size () const
    {
      return (this->_Wridx () - this->_Rdidx ());
    }

  void safe_destroy ();
};

inline void
q_replace_cb (std::atomic<q_data *>& ptr, q_data *old, q_data *nq, uintptr_t)
{
  finalize (old);
  ptr.store (nq, std::memory_order_relaxed);
}

inline void
q_clear_cb (std::atomic<q_data *>&, q_data *old, q_data *, uintptr_t empty)
{
  old->wr_idx.store (old->cap, std::memory_order_relaxed);
  old->rd_idx.store (old->cap, std::memory_order_relaxed);

  for (size_t i = 0; i < old->cap; ++i)
    old->ptrs[i] = empty;

  old->rd_idx.store (0, std::memory_order_release);
  old->wr_idx.store (0, std::memory_order_release);
}

} // namespace detail

template <class T>
struct queue
{
  typedef detail::wrapped_traits<(sizeof (T) < sizeof (uintptr_t) &&
      std::is_integral<T>::value) || (std::is_pointer<T>::value &&
      alignof (T) >= 8), T> val_traits;

  std::atomic<detail::q_data *> impl;

  typedef T value_type;
  typedef T& reference;
  typedef const T& const_reference;
  typedef T* pointer;
  typedef const T* const_pointer;
  typedef ptrdiff_t difference_type;
  typedef size_t size_type;

  void _Init (size_t size)
    {
      this->impl = detail::q_data::make (size, val_traits::FREE);
    }

  detail::q_data* _Data ()
    {
      return (this->impl.load (std::memory_order_relaxed));
    }

  const detail::q_data* _Data () const
    {
      return (this->impl.load (std::memory_order_relaxed));
    }

  void _Set_data (detail::q_data *qdp)
    {
      this->impl.store (qdp, std::memory_order_relaxed);
    }

  struct iterator : public cs_guard
    {
      const detail::q_data *qdp;
      size_t idx;
      uintptr_t c_val;

      typedef std::forward_iterator_tag iterator_category;

      iterator (const detail::q_data *q, size_t s) : qdp (q), idx (s)
        {
          if (this->qdp)
            this->_Adv ();
        }

      iterator (const iterator& it) :
          qdp (it.qdp), idx (it.idx), c_val (it.c_val)
        {
        }

      iterator (iterator&& it) : qdp (it.qdp), idx (it.idx), c_val (it.c_val)
        {
          it.qdp = nullptr;
        }

      void _Adv ()
        {
          for (; this->idx < this->qdp->cap; ++this->idx)
            {
              this->c_val = this->qdp->ptrs[this->idx] & ~val_traits::XBIT;
              if (this->c_val != val_traits::DELT &&
                  this->c_val != val_traits::FREE)
                return;
            }

          this->qdp = nullptr;
          this->idx = 0;
        }

      T operator* ()
        {
          return (val_traits::get (this->c_val));
        }

      iterator& operator++ ()
        {
          ++this->idx;
          this->_Adv ();
          return (*this);
        }

      iterator operator++ (int)
        {
          iterator rv { *this };
          ++*this;
          return (rv);
        }

      bool operator== (const iterator& it) const
        {
          return (this->qdp == it.qdp && this->idx == it.idx);
        }

      bool operator!= (const iterator& it) const
        {
          return (!(*this == it));
        }
    };

  typedef iterator const_iterator;

  queue ()
    {
      this->_Init (8);
    }

  template <class T1, class T2>
  void _Init (T1 n, const T2& val, std::true_type)
    {
      size_t ns = (size_t)n;
      auto qdp = detail::q_data::make (detail::upsize (ns), val_traits::FREE);

      try
        {
          for (size_t i = 0; i < ns; )
            {
              qdp->ptrs[i] = val_traits::make (val);
              qdp->wr_idx.store (++i, std::memory_order_relaxed);
            }
        }
      catch (...)
        {
          _Destroy (qdp);
          throw;
        }

      this->_Set_data (qdp);
    }

  template <class It>
  void _Init (It first, It last, std::false_type)
    {
      auto qdp = detail::q_data::make (8, val_traits::FREE);
      try
        {
          for (size_t i = 0; first != last; ++first)
            {
              qdp->ptrs[i] = val_traits::make (*first);
              qdp->wr_idx.store (++i, std::memory_order_relaxed);

              if (i == qdp->cap)
                {
                  auto q2 = detail::q_data::make (i * 2, val_traits::FREE);
                  for (size_t j = 0; j < i; ++j)
                    q2->ptrs[j] = qdp->ptrs[j];

                  qdp->safe_destroy ();
                  qdp = q2;
                }
            }
        }
      catch (...)
        {
          _Destroy (qdp);
          throw;
        }

      this->_Set_data (qdp);
    }

  template <class T1, class T2>
  queue (T1 first, T2 last)
    {
      this->_Init (first, last, typename std::is_integral<T1>::type ());
    }

  queue (const queue<T>& right) : queue (right.begin (), right.end ())
    {
    }

  queue (queue<T>&& right)
    {
      this->_Set_data (right._Data ());
      right._Set_data (nullptr);
    }

  queue (std::initializer_list<T> lst) : queue (lst.begin (), lst.end ())
    {
    }

  bool _Rearm (uintptr_t elem, detail::q_data *qdp)
    {
      size_t ix;
      uintptr_t prev;

      while (true)
        {
          ix = qdp->_Rdidx ();
          prev = xatomic_or (&qdp->ptrs[ix], val_traits::XBIT);

          if (prev == val_traits::DELT)
            {  // Another thread deleted this entry - Retry.
              xatomic_spin_nop ();
              continue;
            }
          else if ((prev & val_traits::XBIT) == 0)
            break;

          while (true)
            {
              if (qdp != this->_Data ())
                // Impl pointer has been installed - Return.
                return (false);
              else if ((qdp->ptrs[ix] & val_traits::XBIT) == 0)
                /* The thread rearming the queue raised an exception.
                 * Recurse and see if we can pick up the slack. */
                return (this->_Rearm (elem, qdp));

              xatomic_spin_nop ();
            }
        }

      detail::q_data *nq = nullptr;
      try
        {
          nq = detail::q_data::make (qdp->cap * 2, val_traits::FREE);
        }
      catch (...)
        {
          xatomic_and (&qdp->ptrs[ix], ~val_traits::XBIT);
          val_traits::free (elem);
          throw;
        }

      uintptr_t *outp = nq->ptrs;
      for (*outp++ = prev; ++ix < qdp->cap; )
        *outp++ = xatomic_or (&qdp->ptrs[ix], val_traits::XBIT);

      *outp++ = elem;
      nq->wr_idx.store (outp - nq->ptrs, std::memory_order_relaxed);
      finalize (qdp);

      this->_Set_data (nq);
      return (true);
    }

  void _Push (uintptr_t val)
    {
      while (true)
        {
          auto qdp = this->_Data ();
          if (qdp->push (val, val_traits::XBIT, val_traits::FREE) ||
              this->_Rearm (val, qdp))
            break;
        }
    }

  void push (const T& elem)
    {
      cs_guard g;
      this->_Push (val_traits::make (elem));
    }

  template <class ...Args>
  void emplace (Args&& ...args)
    {
      cs_guard g;
      this->_Push (val_traits::make (std::forward<Args>(args)...));
    }

  optional<T> pop ()
    {
      cs_guard g;

      while (true)
        {
          auto qdp = this->_Data ();
          uintptr_t val = qdp->pop (val_traits::XBIT, val_traits::DELT);

          if (val == val_traits::DELT)
            // Queue is empty.
            return (optional<T> ());
          else if (val != val_traits::XBIT)
            {
              val_traits::destroy (val);
              optional<T> rv (val_traits::get (val));
              return (rv);
            }

          while (qdp == this->_Data ())
            xatomic_spin_nop ();
        }
    }

  optional<T> front () const
    {
      cs_guard g;
      uintptr_t rv = this->_Data()->front () & ~val_traits::XBIT;

      if (rv == val_traits::FREE)
        return (optional<T> ());

      return (optional<T> (val_traits::get (rv)));
    }

  optional<T> back () const
    {
      cs_guard g;
      uintptr_t rv = this->_Data()->back () & ~val_traits::XBIT;

      if (rv == val_traits::FREE)
        return (optional<T> ());

      return (optional<T> (val_traits::get (rv)));
    }

  size_t size () const
    {
      cs_guard g;
      return (this->_Data()->size ());
    }

  bool empty () const
    {
      return (this->size () == 0);
    }

  size_t max_size () const
    {
      return (~(size_t)0);
    }

  iterator begin () const
    {
      cs_guard g;
      auto qdp = this->_Data ();
      return (iterator (qdp, qdp->_Rdidx ()));
    }

  iterator end () const
    {
      return (iterator (nullptr, 0));
    }

  const_iterator cbegin () const
    {
      return (this->begin ());
    }

  const_iterator cend () const
    {
      return (this->end ());
    }

  void _Call_cb (detail::q_data *nq, uintptr_t xv,
    void (*f) (std::atomic<detail::q_data *>&,
               detail::q_data *, detail::q_data *, uintptr_t))
    {
      while (true)
        {
          auto qdp = this->_Data ();
          size_t ix = qdp->_Rdidx ();
          uintptr_t prev = xatomic_or (&qdp->ptrs[ix], val_traits::XBIT);

          if (prev == val_traits::DELT)
            continue;
          else if ((prev & val_traits::XBIT) == 0)
            {
              if (prev != val_traits::FREE)
                val_traits::destroy (prev);

              while (++ix < qdp->cap)
                {
                  prev = xatomic_or (&qdp->ptrs[ix], val_traits::XBIT);
                  if (prev != val_traits::FREE && prev != val_traits::DELT)
                    val_traits::destroy (prev);
                }

              f (this->impl, qdp, nq, xv);
              return;
            }

          while (true)
            {
              if (qdp != this->_Data () ||
                  (qdp->ptrs[ix] & val_traits::XBIT) == 0)
                break;

              xatomic_spin_nop ();
            }
        }
    }

  void _Assign (detail::q_data *nq)
    {
      this->_Call_cb (nq, 0, detail::q_replace_cb);
    }

  template <class T1, class T2>
  void assign (T1 first, T2 last)
    {
      cs_guard g;
      auto tmp = queue<T> (first, last);
      this->_Assign (tmp._Data ());
      tmp._Set_data (nullptr);
    }

  void assign (std::initializer_list<T> lst)
    {
      this->assign (lst.begin (), lst.end ());
    }

  queue<T>& operator= (const queue<T>& right)
    {
      if (this == &right)
        return (*this);

      this->assign (right.begin (), right.end ());
      return (*this);
    }

  bool operator== (const queue<T>& right) const
    {
      auto x1 = this->cbegin (), x2 = this->cend ();
      auto y1 = right.cbegin (), y2 = right.cend ();

      for (; x1 != x2 && y1 != y2; ++x1, ++y1)
        if (*x1 != *y1)
          return (false);

      return (x1 == x2 && y1 == y2);
    }

  bool operator!= (const queue<T>& right) const
    {
      return (!(*this == right));
    }

  bool operator< (const queue<T>& right) const
    {
      auto x1 = this->cbegin (), x2 = this->cend ();
      auto y1 = right.cbegin (), y2 = right.cend ();

      for (; x1 != x2; ++x1, ++y1)
        {
          if (y1 == y2 || *y1 < *x1)
            return (false);
          else if (*x1 < *y1)
            return (true);
        }

      return (y1 != y2);
    }

  bool operator> (const queue<T>& right) const
    {
      return (right < *this);
    }

  bool operator<= (const queue<T>& right) const
    {
      return (!(right < *this));
    }

  bool operator>= (const queue<T>& right) const
    {
      return (!(*this < right));
    }

  queue<T>& operator= (queue<T>&& right)
    {
      auto prev = this->impl.exchange (right._Data (),
                                       std::memory_order_acq_rel);
      finalize (prev);
      right._Set_data (nullptr);
      return (*this);
    }

  static void _Destroy (detail::q_data *qdp)
    {
      for (size_t i = qdp->_Rdidx (); i < qdp->cap; ++i)
        {
          uintptr_t val = qdp->ptrs[i] & ~val_traits::XBIT;
          if (val != val_traits::FREE && val != val_traits::DELT)
            val_traits::free (val);
        }

      qdp->safe_destroy ();
    }

  void clear ()
    {
      cs_guard g;
      this->_Call_cb (nullptr, val_traits::FREE, detail::q_clear_cb);
    }

  size_t _Lock ()
    {
      while (true)
        {
          auto qdp = this->_Data ();
          size_t ix = qdp->_Rdidx ();
          uintptr_t prev = xatomic_or (&qdp->ptrs[ix], val_traits::XBIT);

          if ((prev & val_traits::XBIT) == 0)
            return (qdp->wr_idx.exchange (qdp->cap,
                                          std::memory_order_acq_rel));

          while (qdp == this->_Data () &&
              (qdp->ptrs[ix] & val_traits::XBIT) != 0)
            xatomic_spin_nop ();
        }
    }

  void swap (queue<T>& right)
    {
      if (this == &right)
        return;

      size_t s1 = this->_Lock ();
      size_t s2 = right._Lock ();

      auto tmp = this->_Data ();
      this->_Set_data (right._Data ());
      right._Set_data (tmp);

      auto d1 = this->_Data();
      auto d2 = right._Data();

      d1->wr_idx.store (s2, std::memory_order_relaxed);
      d2->wr_idx.store (s1, std::memory_order_relaxed);

      d1->ptrs[d1->_Rdidx ()] &= ~val_traits::XBIT;
      d2->ptrs[d2->_Rdidx ()] &= ~val_traits::XBIT;
    }

  ~queue ()
    {
      auto qdp = this->_Data ();
      if (!qdp)
        return;

      _Destroy (qdp);
      this->_Set_data (nullptr);
    }
};

} // namespace xrcu

#endif


code:
#include "skip_list.hpp"
#include <cstring>
#include <new>

namespace xrcu
{

namespace detail
{

void* sl_alloc_node (unsigned int lvl, size_t size, uintptr_t **outpp)
{
  void *p = ::operator new (size + lvl * sizeof (uintptr_t));
  *outpp = (uintptr_t *)((char *)p + size);
  memset (*outpp, 0, lvl * sizeof (uintptr_t));
  return (p);
}

void sl_dealloc_node (void *ptr)
{
  ::operator delete (ptr);
}

} // namespace detail

} // namespace xrcu