dvdnav: clean up, fix error message and probing encoding (fix #3816)
dvdnav: clean up, fix error message and probing encoding (fix #3816)

Paths for error message must be UTF-8; they were locale.
Paths for probing must be UTF-8 also, as vlc_fopen() does the
conversion to locale internally.

shyamalschandra/vlc,vlc-mirror/vlc-2.1,shyamalschandra/vlc,krichter722/vlc,shyamalschandra/vlc,jomanmuk/vlc-2.2,xkfz007/vlc,krichter722/vlc,vlc-mirror/vlc,jomanmuk/vlc-2.2,jomanmuk/vlc-2.2,vlc-mirror/vlc,vlc-mirror/vlc,xkfz007/vlc,krichter722/vlc,vlc-mirror/vlc-2.1,jomanmuk/vlc-2.1,jomanmuk/vlc-2.1,shyamalschandra/vlc,vlc-mirror/vlc,xkfz007/vlc,vlc-mirror/vlc,jomanmuk/vlc-2.2,jomanmuk/vlc-2.2,krichter722/vlc,krichter722/vlc,vlc-mirror/vlc-2.1,shyamalschandra/vlc,vlc-mirror/vlc-2.1,jomanmuk/vlc-2.1,jomanmuk/vlc-2.1,jomanmuk/vlc-2.2,vlc-mirror/vlc,jomanmuk/vlc-2.1,xkfz007/vlc,jomanmuk/vlc-2.1,xkfz007/vlc,krichter722/vlc,jomanmuk/vlc-2.2,vlc-mirror/vlc-2.1,shyamalschandra/vlc,krichter722/vlc,jomanmuk/vlc-2.1,vlc-mirror/vlc-2.1,vlc-mirror/vlc,xkfz007/vlc,xkfz007/vlc,vlc-mirror/vlc-2.1,shyamalschandra/vlc