>

People can use anything to avoid responsibility or gain things, the thing they're using to their advantage isnt the problem (ie. labels) but the tactic...which is called manipulation. So, I don't think that's a solid argument.
But I like that you mentioned the "us vs them" phenomenon. Labels can and do fuel that mindset, which isn't ...good. But I also think that the 'us vs them' has been a thing long before people started having "official" labels. People do the us vs them based on differences, its fueled by competition and to be better. So, point is, getting rid of labels won't directly get rid of this mindset either. 
Main point is that yes, labels can separate people and has done so even when you look back at history, but to a larger extent I'd argue they unite people.
I mean, it's a pretty easy concept, right?
People hide behind labels to avoid responsibility. People use labels to gain benefits. People use labels to belittle and diminish value.
Labels can be used to come together, but of course they can be used to divide. It's "US" vs "THEM".