orchestrator: Avoid restarting a task that has already been restarted
orchestrator: Avoid restarting a task that has already been restarted

When I was debugging integration test failures a week ago, I dug deep
into the recent restart supervisor changes, and noticed a possible race
which is almost impossible in practice.

This is the sequence of events that has to happen:

1) Restart task A -> new task B.

2) B fails before the restart delay has elapsed (pull failure):
Restart(B) starts waiter goroutine.

3) B's node fails at exact moment after the restart delay finishes. The
orchestrator calls Restart(B) again, this time there's no delay in
progress so Restart proceeds to set B.DesiredState = SHUTDOWN and create
a new task.

4) The waiter goroutine from step 2 is scheduled. It calls Restart(B) to
resume the first restart attempt. This sets B.DesiredState = SHUTDOWN
(which means no change) and creates a new task.

We'd end up with an extra task.

Again, this would be almost impossible to trigger, but I'm fixing it for
the sake of correctness. The general principle here is that Restart
should never been called on a task that already has DesiredState >
RUNNING, since what Restart does is "shut down this task and replace it
with a new one".

I also added a sanity check to Restart. I'm not sure this is really
helpful because returning an error probably does more harm than good in
this case. But at least it would cause an error message to be logged.

Signed-off-by: Aaron Lehmann <8ecfc6017a87905413dcd7d63696a2a4c351b604@docker.com>

cezarsa/swarmkit,docker/swarmkit,aaronlehmann/swarmkit,sungwonh/swarmkit,fate-grand-order/swarmkit,elifa/swarmkit,dperny/swarmkit-1,deljuven/swarmkit,vasil-yordanov/swarmkit,RenaudWasTaken/swarmkit,yongtang/swarmkit,alyyousuf7/swarmkit,AliyunContainerService/swarmkit,cheyang/swarmkit,fate-grand-order/swarmkit,jlhawn/swarmkit,aaronlehmann/swarmkit,elifa/swarmkit,Noyan759/swarmkit,cezarsa/swarmkit,yp-engineering/swarmkit,yongtang/swarmkit,allencloud/swarmkit,allencloud/swarmkit,yp-engineering/swarmkit,cheyang/swarmkit,docker/swarmkit,vasil-yordanov/swarmkit,deljuven/swarmkit,Noyan759/swarmkit,jlhawn/swarmkit,alyyousuf7/swarmkit,diogomonica/swarmkit,diogomonica/swarmkit,RenaudWasTaken/swarmkit,AliyunContainerService/swarmkit,sungwonh/swarmkit,dperny/swarmkit-1