Only drop listeners if last surface.
Only drop listeners if last surface.

Whoops that was a bad bug I looked over. Still, it has been fixed.

This should fix wl_shell_test.

way-cooler/wlroots-rs,way-cooler/wlroots-rs