avcodec/videotoolboxenc: fix vtctx reset condition

In vtenc_populate_extradata, the cleanup function vtenc_reset should not
be used when no error occurs, otherwise some color information is lost
(#11036).

This patch checks the status code and conducts the correct cleanup.

Signed-off-by: Hao Guan <hguandl@gmail.com>
Signed-off-by: Zhao Zhili <zhilizhao@tencent.com>

libavcodec/videotoolboxenc.c

@@ -2767,7 +2767,14 @@ static int vtenc_populate_extradata(AVCodecContext   *avctx,
 
 pe_cleanup:
     CVPixelBufferRelease(pix_buf);
-    vtenc_reset(vtctx);
+
+    if (status) {
+        vtenc_reset(vtctx);
+    } else if (vtctx->session) {
+        CFRelease(vtctx->session);
+        vtctx->session = NULL;
+    }
+
     vtctx->frame_ct_out = 0;
 
     av_assert0(status != 0 || (avctx->extradata && avctx->extradata_size > 0));