Why are there "unique" characters in many sets? When I was young, about 30 years ago, I'd had no problem to have two policemen, two yellow astronauts or knights. Then I had some lack of Lego sets and didn't know what was happening on this field.

Now, when my son begins (or rather "has begun") his Lego adventure, I realized, that there are theme sets based on movies, stories etc. But there are sets containing the same character.

When he was about 3-4, he was an admirer of Disney/Pixar Cars movie. Lego had a collection, firstly four sets 5813, 5814, 5815 and 5816.

As the Lightning McQueen is in three sets, there is no reason (from my point of view) to repeat Sheriff in 5814 and 5816 sets. So my son has two Sheriffs.

Some years later he changed from Cars to Lego Ninjago. On every occasion (birthday, Christmas, ...) he was receiving Lego Ninjago sets. Now he has eight Zanes, three Coles, three Kais, two Jays.

The same is now, but in smaller scale, in Lego Chima.

As I was looking at the Lego StarWars theme this is even worse. Ok, I can agree to have 8 clone troopers, but for what reasons have three Darth Vaders?

In my opinion this is not a good marketing practice. If for someone the sets are expensive, he would buy one set and (perhaps) a Luke Skywalker minifigure. He can re-think whether to buy a new set with a character he already has.

If each set had a unique minifigure, I would be more likely to buy more sets, just to make a collection.

It's also not ok (for me) to have the same characters for example in Lego Heroica.

I know that sets are larger and smaller, and there are people who buy cheaper ones, and they also should have something interesting, but why then copy the same characters to larger ones?

Does it has some rational explanation?
It seems like you understand why there are unique minifigs, but I'll point it out explicitly anyway. Unique minifigs add character to sets, and add to playability. For example, I always wanted this guy as a kid:



Without him, my pirates were just  a leaderless band, but once they have a fearless leader with a peg leg and hook, things become a lot more exciting. As you pointed out, unique figs are also a necessary part of the licensed themes. What would Star Wars be without Darth Vader?

The core of your questions seems to be why these unique figs end up in so many sets such that you have duplicates and triplicates of them. I'm sure this isn't TLG's goal, but there are some marketing reasons which cause it to happen.

First, TLG ideally wants each set to be a whole product. This generally means that the sets should be able to stand on their own, and upon seeing a set, a customer would actually want to buy it. Keep in mind that your rationality as a parent isn't always the same as others who may be buying Lego for your child, or children themselves. Here's an example of set to help me explain what I mean:



That's Gandalf Arrives (set 9469). As you can see, it is a pretty small set with Gandalf and Frodo. These are both core characters, and they each appear in several sets, but this set wouldn't be able to stand on its own without them. While this set may have less appeal to you or your child because you might already have these minifigs, the set has greater appeal to children who don't already have these minifigs. It also has greater appeal to your sister-in-law who needs to pick up a Lego set for your child's birthday party. She won't know what minifigs your child has already, and she's just looking for a nice reasonably cheap set with good playability.

Second, TLG uses unique minifigs to create desire to collect the whole product line. While there is duplication among the major characters, if you want to have all of the minor characters available, you pretty much have to buy every major set in the theme. This makes a lot of sense from a marketing point of view. For example, if you already own Helm's Deep (9474), does a small add-on really have much appeal to you? Uruk-Hai Army (9471) is a small set which attaches to the side of Helm's Deep:



This set may not have much appeal. It's just a basic castle wall with some men fighting some bad guys. However, this is the only set which contains the guy (Eomer) who brings in the cavalry and ultimately makes the battle a victorious one. This adds to this otherwise somewhat boring set, and makes it more appealing to consumers.

As an aside, many Lego fans want to get a complete set of minifigs, but can't afford all the sets, so your extra Vader's have good resale value on sites like Bricklink where they sell for about $10 used.