Fix double remove from surface destroy_listener_list
Fix double remove from surface destroy_listener_list

We remove the listener when a device loses its pointer focus, but
doesn't insert it in another destroy_listener list if surface is NULL.
When we set another pointer focus surface, we remove it again.
Tracked down and tested by Iskren Chernev.

mchalupa/wayland,abooij/wayland,sir-murray/wayland,tulcod/wayland,sir-murray/wayland,abooij/wayland,eyolfson/wayland,bitfactor/wayland,abooij/wayland,bitfactor/wayland,mchalupa/wayland,bitfactor/wayland,sir-murray/wayland,mchalupa/wayland,eyolfson/wayland,tulcod/wayland,eyolfson/wayland,tulcod/wayland