A Fine-grained Safety Landscape of Large Language Models