Better user agents, MIME type handling
Better user agents, MIME type handling

Ok, we need to start making tricky http aware of more than just HTML,
but that needs some upfront redesign of the MIME type checkers.  I
should be able to easily add support for image types soon so we can
get decoding of file system, height and width, image type, and perhaps
interesting info from exif where supported (bitmaps won't have that).

Also, we're emitting more modern user agent strings, specifically the
current stable and beta of google chrome and the last 2 released
versions for firefox.  This should deal well with any servers configured
to ignore robots (we were using 'Ruby' before) but may end up giving
a user agent string of 'Trickbot/WebScraper' in the long run.  We'll
see.

chuckination/trickbot