added checks to prevent rate/distortion overflow
added checks to prevent rate/distortion overflow

At speed 2, due to the threshold scheme used, it is possible the rate
and distortion assigned with INT_MAX value. The patch added checking
to prevent the INT_MAX value is used in further calculation of RD
scores. The patch also changed the assertion in rd_use_partition() to
be mirror similar assertion in rd_pick_partition().

Change-Id: Idb52c543cc1e10abdf6e6a5d6e9cb535a42214dc

openpeer/libvpx_new,sanyaade-teachings/libvpx,Suvarna1488/webm.libvpx,goodleixiao/vpx,iniwf/webm.libvpx,kleopatra999/webm.libvpx,Acidburn0zzz/webm.libvpx,Acidburn0zzz/webm.libvpx,jdm/libvpx,turbulenz/libvpx,thdav/aom,matanbs/vp982,kleopatra999/webm.libvpx,ittiamvpx/libvpx-1,jdm/libvpx,reimaginemedia/webm.libvpx,charup/https---github.com-webmproject-libvpx-,WebRTC-Labs/libvpx,Distrotech/libvpx,cinema6/libvpx,Suvarna1488/webm.libvpx,felipebetancur/libvpx,smarter/aom,felipebetancur/libvpx,iniwf/webm.libvpx,Topopiccione/libvpx,Acidburn0zzz/webm.libvpx,running770/libvpx,GrokImageCompression/aom,running770/libvpx,GrokImageCompression/aom,abwiz0086/webm.libvpx,Suvarna1488/webm.libvpx,Suvarna1488/webm.libvpx,thdav/aom,mwgoldsmith/libvpx,pcwalton/libvpx,turbulenz/libvpx,Acidburn0zzz/webm.libvpx,turbulenz/libvpx,Topopiccione/libvpx,ittiamvpx/libvpx,sanyaade-teachings/libvpx,thdav/aom,ittiamvpx/libvpx-1,zofuthan/libvpx,pcwalton/libvpx,Distrotech/libvpx,running770/libvpx,reimaginemedia/webm.libvpx,gshORTON/webm.libvpx,charup/https---github.com-webmproject-libvpx-,mbebenita/aom,reimaginemedia/webm.libvpx,ShiftMediaProject/libvpx,smarter/aom,reimaginemedia/webm.libvpx,liqianggao/libvpx,abwiz0086/webm.libvpx,charup/https---github.com-webmproject-libvpx-,goodleixiao/vpx,lyx2014/libvpx_c,charup/https---github.com-webmproject-libvpx-,webmproject/libvpx,WebRTC-Labs/libvpx,cinema6/libvpx,VTCSecureLLC/libvpx,shyamalschandra/libvpx,Suvarna1488/webm.libvpx,turbulenz/libvpx,kalli123/webm.libvpx,pcwalton/libvpx,Topopiccione/libvpx,kalli123/webm.libvpx,ittiamvpx/libvpx,Suvarna1488/webm.libvpx,lyx2014/libvpx_c,cinema6/libvpx,zofuthan/libvpx,shyamalschandra/libvpx,mwgoldsmith/vpx,pcwalton/libvpx,kim42083/webm.libvpx,GrokImageCompression/aom,Laknot/libvpx,mbebenita/aom,matanbs/webm.libvpx,altogother/webm.libvpx,gshORTON/webm.libvpx,kleopatra999/webm.libvpx,zofuthan/libvpx,smarter/aom,jacklicn/webm.libvpx,shacklettbp/aom,webmproject/libvpx,Topopiccione/libvpx,webmproject/libvpx,jmvalin/aom,mwgoldsmith/vpx,sanyaade-teachings/libvpx,jdm/libvpx,turbulenz/libvpx,abwiz0086/webm.libvpx,altogother/webm.libvpx,smarter/aom,mbebenita/aom,shacklettbp/aom,mwgoldsmith/libvpx,altogother/webm.libvpx,thdav/aom,kalli123/webm.libvpx,sanyaade-teachings/libvpx,running770/libvpx,shyamalschandra/libvpx,thdav/aom,cinema6/libvpx,kim42083/webm.libvpx,liqianggao/libvpx,felipebetancur/libvpx,openpeer/libvpx_new,jdm/libvpx,goodleixiao/vpx,mbebenita/aom,Maria1099/webm.libvpx,Laknot/libvpx,felipebetancur/libvpx,shyamalschandra/libvpx,ittiamvpx/libvpx,liqianggao/libvpx,jmvalin/aom,n4t/libvpx,matanbs/webm.libvpx,turbulenz/libvpx,jacklicn/webm.libvpx,reimaginemedia/webm.libvpx,turbulenz/libvpx,matanbs/vp982,gshORTON/webm.libvpx,Laknot/libvpx,kim42083/webm.libvpx,kim42083/webm.libvpx,VTCSecureLLC/libvpx,felipebetancur/libvpx,matanbs/webm.libvpx,mbebenita/aom,shareefalis/libvpx,Topopiccione/libvpx,n4t/libvpx,Distrotech/libvpx,ittiamvpx/libvpx-1,hsueceumd/test_hui,jmvalin/aom,mwgoldsmith/vpx,goodleixiao/vpx,Distrotech/libvpx,liqianggao/libvpx,kalli123/webm.libvpx,felipebetancur/libvpx,shyamalschandra/libvpx,iniwf/webm.libvpx,mwgoldsmith/libvpx,Distrotech/libvpx,liqianggao/libvpx,turbulenz/libvpx,running770/libvpx,n4t/libvpx,jacklicn/webm.libvpx,shareefalis/libvpx,openpeer/libvpx_new,VTCSecureLLC/libvpx,abwiz0086/webm.libvpx,mbebenita/aom,Maria1099/webm.libvpx,ShiftMediaProject/libvpx,openpeer/libvpx_new,stewnorriss/libvpx,webmproject/libvpx,smarter/aom,jmvalin/aom,WebRTC-Labs/libvpx,GrokImageCompression/aom,jdm/libvpx,Laknot/libvpx,luctrudeau/aom,openpeer/libvpx_new,lyx2014/libvpx_c,mbebenita/aom,Acidburn0zzz/webm.libvpx,stewnorriss/libvpx,zofuthan/libvpx,hsueceumd/test_hui,jacklicn/webm.libvpx,shyamalschandra/libvpx,running770/libvpx,liqianggao/libvpx,Maria1099/webm.libvpx,mwgoldsmith/vpx,Maria1099/webm.libvpx,luctrudeau/aom,kleopatra999/webm.libvpx,cinema6/libvpx,shacklettbp/aom,abwiz0086/webm.libvpx,matanbs/vp982,pcwalton/libvpx,kim42083/webm.libvpx,stewnorriss/libvpx,matanbs/vp982,matanbs/vp982,shareefalis/libvpx,smarter/aom,matanbs/webm.libvpx,ittiamvpx/libvpx-1,stewnorriss/libvpx,webmproject/libvpx,kleopatra999/webm.libvpx,mwgoldsmith/libvpx,kalli123/webm.libvpx,zofuthan/libvpx,n4t/libvpx,altogother/webm.libvpx,ittiamvpx/libvpx,ittiamvpx/libvpx-1,altogother/webm.libvpx,charup/https---github.com-webmproject-libvpx-,mbebenita/aom,matanbs/vp982,GrokImageCompression/aom,sanyaade-teachings/libvpx,zofuthan/libvpx,ShiftMediaProject/libvpx,GrokImageCompression/aom,jacklicn/webm.libvpx,cinema6/libvpx,Distrotech/libvpx,openpeer/libvpx_new,mwgoldsmith/libvpx,luctrudeau/aom,kleopatra999/webm.libvpx,shacklettbp/aom,gshORTON/webm.libvpx,hsueceumd/test_hui,WebRTC-Labs/libvpx,ShiftMediaProject/libvpx,cinema6/libvpx,Laknot/libvpx,goodleixiao/vpx,ittiamvpx/libvpx,matanbs/vp982,matanbs/webm.libvpx,jmvalin/aom,Maria1099/webm.libvpx,mwgoldsmith/vpx,VTCSecureLLC/libvpx,Laknot/libvpx,reimaginemedia/webm.libvpx,thdav/aom,iniwf/webm.libvpx,webmproject/libvpx,stewnorriss/libvpx,shacklettbp/aom,shareefalis/libvpx,altogother/webm.libvpx,Maria1099/webm.libvpx,hsueceumd/test_hui,mbebenita/aom,lyx2014/libvpx_c,luctrudeau/aom,Acidburn0zzz/webm.libvpx,shacklettbp/aom,lyx2014/libvpx_c,ShiftMediaProject/libvpx,hsueceumd/test_hui,jmvalin/aom,luctrudeau/aom,gshORTON/webm.libvpx,lyx2014/libvpx_c,mwgoldsmith/libvpx,turbulenz/libvpx,pcwalton/libvpx,shareefalis/libvpx,stewnorriss/libvpx,mwgoldsmith/vpx,ittiamvpx/libvpx-1,VTCSecureLLC/libvpx,kalli123/webm.libvpx,VTCSecureLLC/libvpx,ittiamvpx/libvpx,jdm/libvpx,jacklicn/webm.libvpx,Topopiccione/libvpx,matanbs/webm.libvpx,iniwf/webm.libvpx,goodleixiao/vpx,WebRTC-Labs/libvpx,abwiz0086/webm.libvpx,hsueceumd/test_hui,n4t/libvpx,kim42083/webm.libvpx,charup/https---github.com-webmproject-libvpx-,shareefalis/libvpx,iniwf/webm.libvpx,luctrudeau/aom,gshORTON/webm.libvpx