Add Pysam setup retries to make COST_OPTIMIZED dataflow more robust. (#657)
Add Pysam setup retries to make COST_OPTIMIZED dataflow more robust. (#657)

* Add Pysam setup retries to make COST_OPTIMIZED dataflow more robust.

When running `vcf_to_bq` under with Dataflow cost optimizations
(https://cloud.google.com/dataflow/docs/guides/flexrs):

```
   --flexrs_goal=COST_OPTIMIZED \
   --runner DataflowRunner
```

It was observed (running at moderate scale) that new workers would fail with:

```
   File "/usr/local/lib/python2.7/site-packages/pysam/__init__.py", line 5, in <module>
     from pysam.libchtslib import *
   ImportError: No module named libchtslib
```

If they retried and failed 4 times, then the Dataflow job would
terminate. With enough workers (and perhaps because of restarts from
worker preemption), this was observed to cause a single job of 1000 VCFs
to fail consistently.

The root cause for the Pysam setup failure has not been determined, but with the
simple retry logic added here (10 attempts with a 10 second sleep
interval), the problem went away.

* Change "import *" to avoid SyntaxWarning (setup.py)

* Restore comments to setup.py.

* Add pylint directives, per PR feedback.

Co-authored-by: Saman Vaisipour <8a058ff676a84145be647c0c7d8f2022fdc436e5@gmail.com>
googlegenomics/gcp-variant-transforms,googlegenomics/gcp-variant-transforms