What was done with animals WITH defects? A quick search reveals that the phrase "without defect" appears at least 46 times in the Old Testament. What happened to the animals that had defects? What is a defect, and were they fit for consumption by people?
I think Malachi offers a little more insight into this:


  When you offer blind animals in sacrifice, is that not evil? And when you offer those that are lame or sick, is that not evil? Present that to your governor; will he accept you or show you favor? says the LORD of hosts. - Malachi 1:8 ESV


These are animals that would not be eaten or bred due to illness or other problems.  The idea is that it's not really a sacrifice if it has little value to begin with.