Reduce concurrency of migrate_index task
Reduce concurrency of migrate_index task

This job is run every night so that we can recreate our indices
to include popularity measures from google analytics.
Currently it takes about 30 minutes.

How it works:

- We create a number of consumer threads that pull documents off a single queue.
- This queue is populated from the original index and the consumers
reindex the documents in parallel

What changed:

We changed indexing to call publishing api to fetch links
data every time a document indexed via a call to get_expanded_links.

Since then, the nightly job has been failing due to the
get_expanded_links requests timing out.

This is a workaround until we can improve the performance of the publishing api.

It's not ideal, because the longer this job takes to run, the longer
the index is locked for, which impacts publishing (rummager sidekiq jobs will fail until the index becomes available).

We think this is acceptable in this case because it's an overnight job.

alphagov/rummager,alphagov/rummager