storage: Fix CommandQueue command dependency leak on error
storage: Fix CommandQueue command dependency leak on error

Fixes #4810.

The bug reported was seeing OOM crashes after running for long periods
of time due to an apparent goroutine leak. Looking into this under
a blocking or goroutine profiler pointed out that these leaked
goroutines were primarily in `Replica.beginCmds`. This method
purpose is to determine a commands dependencies and wait for them
to clear up before returning the command queue insertion keys.
It made sense that the `WaitGroup`s used in this method to gate a
command from proceeding would see a good amount of blocking, but
they seemed to get stuck, indicating `WaitGroup.Done` was never
being called on them. The commands` WaitGroups were supposed to be
freed up when dependee commands were evicted/removed from the
`CommandQueue`, but clearly this wasn't working everywhere. When looking
at `Replica.addWriteCmd`, it became clear what the issue was.

When the `Context.Done` handling was added to the method, it did not
remove the commands from the queue. This meant that if a deadline for
a request was ever reached (which eventually would happen under
contention), we would leave all of it's commands waiting. The fix
was to defer the `CommandQueue` removal to guarantee that we remove
the commands from the queue no success or error.

vaibhavarora/cockroach,cockroachdb/cockroach,glycerine/cockroach,vivekmenezes/cockroach,vivekmenezes/cockroach,petermattis/cockroach,glycerine/cockroach,cuongdo/cockroach,vivekmenezes/cockroach,paperstreet/cockroach,vivekmenezes/cockroach,vivekmenezes/cockroach,paperstreet/cockroach,glycerine/cockroach,sunya123/cockroach,vaibhavarora/cockroach,arjun4084346/cockroach,maxlang/cockroach,cuongdo/cockroach,cockroachdb/cockroach,sunya123/cockroach,cuongdo/cockroach,maxlang/cockroach,veteranlu/cockroach,veteranlu/cockroach,glycerine/cockroach,maxlang/cockroach,arjun4084346/cockroach,cuongdo/cockroach,veteranlu/cockroach,paperstreet/cockroach,paperstreet/cockroach,cockroachdb/cockroach,petermattis/cockroach,sunya123/cockroach,bg451/cockroach,cockroachdb/cockroach,glycerine/cockroach,nieyy/cockroach,arjun4084346/cockroach,cuongdo/cockroach,nieyy/cockroach,bg451/cockroach,bg451/cockroach,veteranlu/cockroach,vaibhavarora/cockroach,nieyy/cockroach,glycerine/cockroach,petermattis/cockroach,petermattis/cockroach,maxlang/cockroach,veteranlu/cockroach,petermattis/cockroach,sunya123/cockroach,cuongdo/cockroach,cuongdo/cockroach,vaibhavarora/cockroach,paperstreet/cockroach,vaibhavarora/cockroach,paperstreet/cockroach,nieyy/cockroach,bg451/cockroach,petermattis/cockroach,sunya123/cockroach,veteranlu/cockroach,vaibhavarora/cockroach,sunya123/cockroach,veteranlu/cockroach,vaibhavarora/cockroach,veteranlu/cockroach,petermattis/cockroach,bg451/cockroach,glycerine/cockroach,nieyy/cockroach,bg451/cockroach,maxlang/cockroach,sunya123/cockroach,petermattis/cockroach,arjun4084346/cockroach,vivekmenezes/cockroach,bg451/cockroach,cuongdo/cockroach,sunya123/cockroach,nieyy/cockroach,cockroachdb/cockroach,cockroachdb/cockroach,vaibhavarora/cockroach,paperstreet/cockroach,cockroachdb/cockroach,vivekmenezes/cockroach,nieyy/cockroach,maxlang/cockroach,glycerine/cockroach,nieyy/cockroach,maxlang/cockroach,arjun4084346/cockroach,arjun4084346/cockroach,maxlang/cockroach,arjun4084346/cockroach,arjun4084346/cockroach,paperstreet/cockroach,cockroachdb/cockroach,vivekmenezes/cockroach