Use actual size to decide sparse vs dense matrix form
Use actual size to decide sparse vs dense matrix form

As the density of a matrix increases the sparse form becomes less
efficient to the point where it's better to use the dense form i.e. a
standard numpy array. (Efficiency here means both memory usage and
performance when summing.)

Previously we used an cutoff of 50% which turns out to be much too high
(20% would be more reasonable). However the exact threshold depends on
the dtypes of the stored values, so a better solution is to work out the
actual memory usage and decide based on that.

This makes the resulting data file slightly smaller (4.3GB vs 4.4GB)
and, more importantly, makes summing matrices slightly faster as well.
The exact increase depends on which matrices are being summed, but
larger, chapter-wide queries can see an improvements of about 20%.

ebmdatalab/openprescribing,annapowellsmith/openpresc,ebmdatalab/openprescribing,ebmdatalab/openprescribing,annapowellsmith/openpresc,annapowellsmith/openpresc,annapowellsmith/openpresc,ebmdatalab/openprescribing