Fix database customization of robots.txt, add crawl-delay, block comment permalink pages
Fix database customization of robots.txt, add crawl-delay, block comment permalink pages

Discordius/Lesswrong2,Discordius/Telescope,Discordius/Telescope,Discordius/Lesswrong2,Discordius/Telescope,Discordius/Lesswrong2,Discordius/Telescope,Discordius/Lesswrong2