avcodec/motion_est: Fix score squaring overflow

Fixes: CID1604552 Overflowed constant

Sponsored-by: Sovereign Tech Fund
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
(cherry picked from commit f18b442370)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>

libavcodec/motion_est.c

@@ -1446,7 +1446,7 @@ static inline int direct_search(MpegEncContext * s, int mb_x, int mb_y)
         s->b_direct_mv_table[mot_xy][0]= 0;
         s->b_direct_mv_table[mot_xy][1]= 0;
 
-        return 256*256*256*64;
+        return 256*256*256*64-1;
     }
 
     c->xmin= xmin;