fix-issue-24
fix-issue-24

Sometimes, when the robots.txt file cannot be parsed properly, we get problematic results. This may happen on bad robots.txt files, and doesn't always mean there is a issue with the code.

For example:
isAllowed('/') = (bool) false 
and 
isDisallowed('/') = (bool) false

With this fix, its now returning a NULL value in case of parsing errors.
t1gor/Robots.txt-Parser-Class