Fix ubsan warnings: vp9/encoder/vp9_pickmode.c
Fix ubsan warnings: vp9/encoder/vp9_pickmode.c

This commit fixes a number of integer out of range issue in HBD build.

BUG=webm:1219

Change-Id: Ib4192dc74a500e1b86c37a399114c7f6d4ed5185

mwgoldsmith/libvpx,webmproject/libvpx,webmproject/libvpx,Topopiccione/libvpx,webmproject/libvpx,mwgoldsmith/libvpx,mwgoldsmith/vpx,mwgoldsmith/vpx,mwgoldsmith/vpx,webmproject/libvpx,mwgoldsmith/libvpx,ShiftMediaProject/libvpx,Topopiccione/libvpx,mwgoldsmith/libvpx,ShiftMediaProject/libvpx,webmproject/libvpx,mwgoldsmith/vpx,mwgoldsmith/vpx,ShiftMediaProject/libvpx,ShiftMediaProject/libvpx,mwgoldsmith/vpx,Topopiccione/libvpx,Topopiccione/libvpx,webmproject/libvpx,Topopiccione/libvpx,ShiftMediaProject/libvpx,mwgoldsmith/libvpx,mwgoldsmith/libvpx,Topopiccione/libvpx