fail early on oversize frames
fail early on oversize frames

Even though frame_size is calculated in uint64_t, it winds up in an int
size value.

This was exposed with the msan test because the memset is called with
(int)frame_size, leading to a segfault.

Change-Id: I7fd930360dca274adb8f3e43e5e6785204808861

ShiftMediaProject/libvpx,webmproject/libvpx,webmproject/libvpx,ShiftMediaProject/libvpx,ShiftMediaProject/libvpx,ShiftMediaProject/libvpx,ShiftMediaProject/libvpx,webmproject/libvpx,webmproject/libvpx,webmproject/libvpx,webmproject/libvpx