Script to detect and show invalid UTF-8 sequences in documents.
Script to detect and show invalid UTF-8 sequences in documents.

Summary:
'arc' is now less forgiving about playing it fast-and-loose with encodings, so
make it easier to find and fix these problems in a codebase with preexisting
issues. See D431.

This script takes a list of files and either does a simple binary test for utf8
validity with the "-t" flag, or shows you exactly what problems exist without
it. The idea is that if you have an existing codebase with a bunch of code and
you want to fix all the UTF-8 problems, you can do something like this:

  find . -type f -name '*.php' -print0 | xargs -0 -n256 ./utf8.php -t

That will give you a list of all the files with problems. Now you can inspect
them individually or in groups and fix the issues:

  ./utf8.php file1 file2 ...

You can also find problems in a diff, or some other piece of command output:

  git diff -U99999 HEAD | ./utf8.php -

Test Plan:
Ran this script on various valid and invalid files.

Reviewed By: jungejason
Reviewers: edward, aran, jungejason, tuomaspelkonen
CC: aran, jungejason
Differential Revision: 433

yangshiqi/libphutil,Khan/libphutil,newbeess/libphutil,uhd-urz/libphutil,PageUpPeopleOrg/libphutil-patches,uber/libphutil,zhihu/libphutil,wxstars/libphutil,Symplicity/libphutil,r4nt/libphutil,wikimedia/phabricator-libphutil,julkiewicz/libphutil,eSpark/libphutil,christopher-johnson/libphutil,wxstars/libphutil,julkiewicz/libphutil,memsql/libphutil,Automattic/libphutil,librewiki/libphutil,devurandom/libphutil,Symplicity/libphutil,zhihu/libphutil,memsql/libphutil,schlaile/libphutil,r4nt/libphutil,eSpark/libphutil,yangshiqi/libphutil,schlaile/libphutil,memsql/libphutil,zhihu/libphutil,newbeess/libphutil,Symplicity/libphutil,librewiki/libphutil,codelabs-ch/libphutil,yangshiqi/libphutil,fivestars/libphutil,julkiewicz/libphutil,fivestars/libphutil,eSpark/libphutil,Automattic/libphutil,devurandom/libphutil,wikimedia/phabricator-libphutil,Khan/libphutil,wxstars/libphutil,aik099/libphutil,newbeess/libphutil,Automattic/libphutil,r4nt/libphutil,codelabs-ch/libphutil,codelabs-ch/libphutil,aik099/libphutil,PageUpPeopleOrg/libphutil-patches,fivestars/libphutil,schlaile/libphutil,Khan/libphutil,PageUpPeopleOrg/libphutil-patches,devurandom/libphutil,christopher-johnson/libphutil,uber/libphutil,uhd-urz/libphutil,christopher-johnson/libphutil,librewiki/libphutil,aik099/libphutil,uber/libphutil,uhd-urz/libphutil,wikimedia/phabricator-libphutil