Fix ioc in trellis optimization with hbd
Fix ioc in trellis optimization with hbd

Use int64_t type for distortion. This avoids integer overflow
issues in the trellis optimization function in high bit-depth
settings.

Change-Id: I550c3ca9f11a3191ef8638a152887018cd476141

mbebenita/aom,mbebenita/aom,luctrudeau/aom,GrokImageCompression/aom,mbebenita/aom,luctrudeau/aom,luctrudeau/aom,mbebenita/aom,luctrudeau/aom,smarter/aom,smarter/aom,mbebenita/aom,luctrudeau/aom,smarter/aom,mbebenita/aom,luctrudeau/aom,smarter/aom,mbebenita/aom,mbebenita/aom,GrokImageCompression/aom,GrokImageCompression/aom,mbebenita/aom,smarter/aom,smarter/aom,GrokImageCompression/aom,GrokImageCompression/aom,GrokImageCompression/aom