The COPY INTO command for SQL is heavily CPU bound, which means that ideally we would like to exploit the multi-cores to do that work in parallel. Complicating factors are the initial record offset, the possible variable length of the input, and the original sort order that should preferrable be maintained.
The COPY INTO command for SQL is heavily CPU bound, which means
that ideally we would like to exploit the multi-cores to do that
work in parallel.
Complicating factors are the initial record offset, the
possible variable length of the input, and the original sort order
that should preferrable be maintained.

The code below consists of a file reader, which breaks up the
file into distinct lines/fields. Then multiple parallel threads
can grab them, convert the value and update the underlying tables.

The threads get a reference to a private copy of the READERtask.
It includes a list of columns they should handle. This is a basis
to distributed cheap and expensive columns over threads.

A double buffering scheme might improve CPU and IO concurrent work (todo).
Also the buffer size of the block stream might be a little small for
this task (1MB).

Further improvements: use semaphores to wakeup the reader thread
instead of spinlock.

(final checks being made with developers)

zyzyis/monetdb,zyzyis/monetdb,zyzyis/monetdb,zyzyis/monetdb,zyzyis/monetdb,zyzyis/monetdb,zyzyis/monetdb,zyzyis/monetdb,zyzyis/monetdb,zyzyis/monetdb