Ran in to spark partitions exceeding integer MAX_VALUE size. Upped partitions to 3k (target 128 MB/partition) from default 200 based on an estimate of 200*2GB of data being shuffled.
Ran in to spark partitions exceeding integer MAX_VALUE size. Upped partitions to 3k (target 128 MB/partition) from default 200 based on an estimate of 200*2GB of data being shuffled.

bio-guoda/guoda-datasets,bio-guoda/guoda-datasets