fixing crawlable rules and content-type header in case of redirect
fixing crawlable rules and content-type header in case of redirect

codeguy/arachnid,codeguy/arachnid,zrashwani/arachnid,zrashwani/arachnid