avformat: check memory allocations
avformat: check memory allocations

(cherry picked from commit 0361af77c96214ce04257fe843e6bf31a05f4ad4)

jomanmuk/vlc-2.2,jomanmuk/vlc-2.2,jomanmuk/vlc-2.2,jomanmuk/vlc-2.2,jomanmuk/vlc-2.2,jomanmuk/vlc-2.2,jomanmuk/vlc-2.2