Is 不错 used only to mean "good" and never mean "not bad"? 不错 is a composite word of 不 and 错, the former means "NOT" and the latter means "wrong", if I understand it correctly.

However, when I look up 不错 in my dictionary (macOS's built-in dictionary), it only shows "right" and "pretty good", and not "not bad".

Is 不错 never used to express "not bad"? And do you have nothing to do but just look up in the dictionary and memorize them one by one in case of these composite words? I happened to look it up today, but without that I would keep misunderstanding its meaning...
In addition to the other answers:

The exact phrase 不錯 is almost always used to mean "pretty good". But there are variations on the phrase that mean "not wrong", "not mistaken" without having the same positive implication:



沒錯

This one is still somewhat positive, but it's more of an agreement than an endorsement:


  他說得沒錯 - What he said is correct.
  他說得不錯 - He said it well / what he said is good.




不是錯的

Use this phrase if you just want to say "not incorrect" without implying anything else about the quality:


  這個答案很不錯 - This answer is pretty good.
  這個答案不是錯的 - This answer is not wrong.




In response to your general question, most composite words do mean exactly what you'd think from looking up each individual character. But when it comes to this kind of nuance in their usage, you'll have to learn each compound word individually.
Occasionally you'll find a compound word where the meaning is quite far removed from the component characters, but this will normally be apparent from context.