Fix removing component listener
Fix removing component listener

instead of adding it again.

Fixes https://bugzilla.gnome.org/show_bug.cgi?id=791970

GNOME/java-atk-wrapper,GNOME/java-atk-wrapper,GNOME/java-atk-wrapper