daemon: get list of frontends from ServiceCache before acquiring BPFMapMu
daemon: get list of frontends from ServiceCache before acquiring BPFMapMu

The `ServiceCache` mutex should always be acquired before acquiring the
`BPFMapMu` of the `LoadBalancer`.

The following deadlock was observed in a cilium instance:

We are trying to acquire the `BPFMapMu` in the daemon's `LoadBalancer` to add a
Kubernetes service as observed by the Kubernetes watcher:

```
goroutine 150 [semacquire, 225 minutes]:
sync.runtime_SemacquireMutex(0xc000401d44, 0x0)
        /usr/local/go/src/runtime/sema.go:71 +0x3d
sync.(*Mutex).Lock(0xc000401d40)
        /usr/local/go/src/sync/mutex.go:134 +0x109
sync.(*RWMutex).Lock(0xc000401d40)
        /usr/local/go/src/sync/rwmutex.go:93 +0x2d
main.(*Daemon).svcAdd(0xc0001b83c0, 0xc002d96eb0, 0x10, 0x10, 0xc001afa069, 0x3, 0x2474, 0x38f, 0x4d8bf28, 0x0, ...)
        /go/src/github.com/cilium/cilium/daemon/loadbalancer.go:132 +0x630
main.(*Daemon).addK8sSVCs(0xc0001b83c0, 0xc001badec0, 0x2d, 0xc001bbb000, 0xb, 0x0, 0xc000e73130, 0xc003e5e8a0, 0x0, 0x0)
        /go/src/github.com/cilium/cilium/daemon/k8s_watcher.go:1288 +0xf37
main.(*Daemon).k8sServiceHandler(0xc0001b83c0)
        /go/src/github.com/cilium/cilium/daemon/k8s_watcher.go:1064 +0xa29
created by main.(*Daemon).runK8sServiceHandler
        /go/src/github.com/cilium/cilium/daemon/k8s_watcher.go:1118 +0x3f
```

The controller which makes sure that the daemon's LoadbLanacer maps are in sync
with Kubernetes has possession of the aforementioned `BPFMapMu` mutex, but is
trying to acquire the mutex for the `ServiceCache` after having acquired the
`BPFMapMu`:

```
goroutine 401 [semacquire, 225 minutes]:
sync.runtime_SemacquireMutex(0xc0001b8514, 0x0)
        /usr/local/go/src/runtime/sema.go:71 +0x3d
sync.(*RWMutex).RLock(...)
        /usr/local/go/src/sync/rwmutex.go:50
github.com/cilium/cilium/pkg/k8s.(*ServiceCache).UniqueServiceFrontends(0xc0001b8500, 0x0)
        /go/src/github.com/cilium/cilium/pkg/k8s/service_cache.go:427 +0x37d
main.(*Daemon).syncLBMapsWithK8s(0xc0001b83c0, 0x0, 0x0)
        /go/src/github.com/cilium/cilium/daemon/loadbalancer.go:643 +0x1a4
main.(*Daemon).initRestore.func2.1(0x35013c0, 0xc00590cb80, 0x4d6d3e0, 0x0)
        /go/src/github.com/cilium/cilium/daemon/state.go:483 +0x2a
github.com/cilium/cilium/pkg/controller.(*Controller).runController(0xc005abe000)
        /go/src/github.com/cilium/cilium/pkg/controller/controller.go:203 +0x9e1
created by github.com/cilium/cilium/pkg/controller.(*Manager).UpdateController
        /go/src/github.com/cilium/cilium/pkg/controller/manager.go:112 +0xae0
```

The deadlock is due to the following channel send being stuck:

```
goroutine 356 [chan send, 225 minutes]:
github.com/cilium/cilium/pkg/k8s.(*ServiceCache).UpdateEndpoints(0xc0001b8500, 0xc0042a87c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/cilium/cilium/pkg/k8s/service_cache.go:297 +0x454
main.(*Daemon).addK8sEndpointV1(...)
        /go/src/github.com/cilium/cilium/daemon/k8s_watcher.go:1136
main.(*Daemon).EnableK8sWatcher.func7.2(0x0, 0xc000e3ca20)
        /go/src/github.com/cilium/cilium/daemon/k8s_watcher.go:530 +0x46
github.com/cilium/cilium/pkg/serializer.(*FunctionQueue).run(0xc000e36930)
        /go/src/github.com/cilium/cilium/pkg/serializer/func_queue.go:70 +0x6e
created by github.com/cilium/cilium/pkg/serializer.NewFunctionQueue
        /go/src/github.com/cilium/cilium/pkg/serializer/func_queue.go:50 +0xb9
```

The function `UpdateEndpoints` is holding the mutex for the `ServiceCache`. The
reason that the channel send above is stuck, is because the goroutine which
receives off of the channel (the first one shown above) is trying to acquire the
`BPFMapMu`.  But, since the goroutine holding the `ServiceCache` mutex is
blocking on sending to the channel, it will never release the mutex, which the
goroutine holding the `BPFMapMu` is trying to acquire before it can release the
`BPFMapMu`, which in turn is blocking the channel receive goroutine from
progressing further!

The simplest fix for now is to call `UniqueServiceFrontends` before acquiring
the `BPFMapMu` in the controller which synchronizes state in the datapath. This
ensures that the controller will eventually release the `BPFMapMu`, which will
allow the goroutine receiving events in the Kubernetes watcher to acquire it,
and read from the channel of service-related events.

Signed-off by: Ian Vernon <ian@cilium.io>

michi-covalent/cilium,tgraf/cilium,tgraf/cilium,cilium-team/cilium,tklauser/cilium,michi-covalent/cilium,tklauser/cilium,cilium/cilium,tklauser/cilium,michi-covalent/cilium,tgraf/cilium,tgraf/cilium,tklauser/cilium,cilium/cilium,michi-covalent/cilium,tgraf/cilium,tgraf/cilium,cilium-team/cilium,cilium/cilium,tklauser/cilium,cilium/cilium,michi-covalent/cilium,cilium/cilium