MB-34003: Fix AddWatcher workflow in case of old listener
MB-34003: Fix AddWatcher workflow in case of old listener

In case of old listener with same fid being present, if a
new listener is added to list of watchers before removing
the old listener, "new" listener will get wrongly removed
from the watchers' list (instead of "old" listener).
This happens beacuse: on termination, listener gets removed
from the list of watchers asynchronously, based on its fid.
Since fid is same for old and new listener, the new listener
gets wrongly removed.

This leads to skipped responses.

Queryport metadata provider maintains a list of indexers. When new
indexer node is added to the cluster, IP address of the existing
indexer node in cluster info, changes (in case of co-located index
and query nodes). IP address changes from localhost to actual IP.
So, queryport metadata provider detects this as a change
(as indexer list is maintained based on their IP)
and starts a new watcher for new IP. Here new watcher is started
with same follower id as before, even though the IP address has
changed. Before registering this new watcher, metadata provider
makes a ServiceMap request to the new indexer node, and waits for
the response.

As explained above, indexer skips repsonding to the request as the
new watcher has been removed from the list of watchers on indexer
side.

So, queryport metadata provider keeps on retrying ServiceMap
request, but never receives the response. So, the queryport
metadata provider has one less watcher registered with it.

If create index requires this existing node in the cluster to satisfy
replica requirement, create index will fail as it will see one less
node/watcher registerd in the queryport metadata provider.

The Fix: Introduce "listener id" to identify old and new listeners
separately. So on cleanup, new lister won't get removed if the
id of the listener being removed won't match with id of the listener
in the list.

Change-Id: Icb53e576e9b3a2970ef2d9bea413f61ceacfa344

couchbase/gometa