pkg/destroy/aws: 10-second lower bound for deletion period
pkg/destroy/aws: 10-second lower bound for deletion period

Currently this is a somewhat-warm loop while we wait for Amazon to
reap dependencies that have already had successful "deleted" calls.
For example:

  $ curl -s  https://storage.googleapis.com/origin-ci-test/pr-logs/pull/openshift_installer/1346/pull-ci-openshift-installer-master-e2e-aws/4324/artifacts/e2e-aws/installer/.openshift_install.log| grep '[Dd]elet.*vpc-08dc4aea3c1c2db8f' | tail -n4
  time="2019-03-01T15:21:47Z" level=debug msg="deleting arn:aws:ec2:us-east-1:460538899914:vpc/vpc-08dc4aea3c1c2db8f: DependencyViolation: The vpc 'vpc-08dc4aea3c1c2db8f' has dependencies and cannot be deleted.\n\tstatus code: 400, request id: 0c437bd8-ed5f-4989-abda-987aefdac6ce"
  time="2019-03-01T15:21:50Z" level=debug msg="deleting arn:aws:ec2:us-east-1:460538899914:vpc/vpc-08dc4aea3c1c2db8f: DependencyViolation: The vpc 'vpc-08dc4aea3c1c2db8f' has dependencies and cannot be deleted.\n\tstatus code: 400, request id: 04085023-f9ca-442b-85a7-ab4a242604e4"
  time="2019-03-01T15:21:52Z" level=debug msg="deleting arn:aws:ec2:us-east-1:460538899914:vpc/vpc-08dc4aea3c1c2db8f: DependencyViolation: The vpc 'vpc-08dc4aea3c1c2db8f' has dependencies and cannot be deleted.\n\tstatus code: 400, request id: 1d3f92e3-1e7e-432f-ae4d-be47b5fab722"
  time="2019-03-01T15:21:54Z" level=info msg=Deleted arn="arn:aws:ec2:us-east-1:460538899914:vpc/vpc-08dc4aea3c1c2db8f" id=vpc-08dc4aea3c1c2db8f

which shows an ~2 second period during that time.  So this should be
an ~5x reduction for that phase of deletion.

In a quiet account, a few DependencyViolation failures aren't a big
deal.  But in our CI account we can create 600+ clusters a day, and
Amazon starts giving us RequestLimitExceeded responses as we pound
away on multiple deletions in parallel.

The downsides to limiting this loop are:

* For resources that are cleaned up immediately, we are slowing our
  walk through the dependency graph.  But the graph isn't that deep,
  so I'm not worried about this at the moment.

* Waiting gives the cluster time to self-heal.  For example, after we
  terminate an instance, the remaining instances may notice and create
  a replacement.  And while we'll attempt to terminate that full
  generation in the current round, if we wait too long between rounds
  the replacement machines may be alive enough to be able to launch
  grandchildren.  But 10 seconds seems short enough that we don't have
  to worry about machines going from boot to child-spawning in that
  window.

From [1]:

  Some intervals may be missed if the condition takes too long or the
  time window is too short.

which is what we want, because we're only attempting each resource
once per cycle.  If the first round takes 40 seconds, and the second
takes 5, we still want to sleep 5 seconds instead of executing
immediately.

[1]: https://godoc.org/k8s.io/apimachinery/pkg/util/wait#PollImmediateInfinite

derekhiggins/installer,derekhiggins/installer