benchmark store test for 1 billion 20 byte keys, 0-sized values
benchmark store test for 1 billion 20 byte keys, 0-sized values

Note, the batch size is rather large (100000) in this test...

On my macbookpro, 16GB ram, SSD...

15 MB/sec write throughput to populate database with sequential 1B
items, and wait for persistence to complete.

56 MB/sec read throughput to sequentially scan through all 1B items
after a file reopen (and, OS cache is warm).

The benchmark test driver is single threaded.

$ go test -timeout=10h -bench=BenchmarkStoreBig -tags=benchmark_store_big
PASS
BenchmarkStoreBig_numItems1B_keySize20_valSize0_batchSize100000-8
  spec: {numItems:1000000000 keySize:20 valSize:0 batchSize:100000 accesses:[]}
     open time:        0 (ms), phase       0 wop/s,      0 wkb/s,       0 rop/s,      0 rkb/s, cum       0 wop/s,      0 wkb/s,       0 rop/s,      0 rkb/s
     load time:  1178848 (ms), phase  848285 wop/s,  16568 wkb/s,       0 rop/s,      0 rkb/s, cum  848285 wop/s,  16568 wkb/s,       0 rop/s,      0 rkb/s
    drain time:    54730 (ms), phase       0 wop/s,      0 wkb/s,       0 rop/s,      0 rkb/s, cum  810649 wop/s,  15833 wkb/s,       0 rop/s,      0 rkb/s
    close time:     1112 (ms), phase       0 wop/s,      0 wkb/s,       0 rop/s,      0 rkb/s, cum  810649 wop/s,  15833 wkb/s,       0 rop/s,      0 rkb/s
   reopen time:        8 (ms), phase       0 wop/s,      0 wkb/s,       0 rop/s,      0 rkb/s, cum  810649 wop/s,  15833 wkb/s,       0 rop/s,      0 rkb/s
     iter time:   347233 (ms), phase       0 wop/s,      0 wkb/s, 2879622 rop/s,  56242 rkb/s, cum  810649 wop/s,  15833 wkb/s, 2879622 rop/s,  56242 rkb/s
    close time:      174 (ms), phase       0 wop/s,      0 wkb/s,       0 rop/s,      0 rkb/s, cum  810649 wop/s,  15833 wkb/s, 2879622 rop/s,  56242 rkb/s
    total time:  1582105 (ms)
  file size: 35996 (mb), amplification: 1.800
       1     1582834143442 ns/op
ok     github.com/couchbase/moss        1585.631s

Change-Id: Ib985b114d8598ee208ba09abd2858f93b5bde33f
Reviewed-on: http://review.couchbase.org/66139
Reviewed-by: Marty Schoch <ba9951d7d66412e2a57aea395f01176a793cf5eb@gmail.com>
Tested-by: Steve Yen <ef850f73f7832052d3c6d9b5ddd3a1716d9a57f1@gmail.com>

steveyen/moss,couchbaselabs/moss,steveyen/moss