avcodec/pcm-blurayenc: Don't presume every channel layout to be native

The pcm_bluray encoder has AVCodec.ch_layouts set, so that
ff_encode_preinit() checks that the channel layout in use
is equivalent to one of the layouts from AVCodec.ch_layouts.
Yet equivalent is not the same as identical; in particular,
custom layouts equivalent to native layouts are possible
(and necessary if one wants to use the name/opaque fields
with an ordinary channel layout), so one must not simply
use AVChannelLayout.u.mask. Use av_channel_layout_subset()
instead.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>

libavcodec/pcm-blurayenc.c

@@ -63,7 +63,7 @@ static av_cold int pcm_bluray_encode_init(AVCodecContext *avctx)
         return AVERROR_BUG;
     }
 
-    switch (avctx->ch_layout.u.mask) {
+    switch (av_channel_layout_subset(&avctx->ch_layout, ~(uint64_t)0)) {
     case AV_CH_LAYOUT_MONO:
         ch_layout = 1;
         break;