avformat/mpegts: fix max_packet_size in mpegts payload parsing

The maximum allowed useful PES payload data was set to PES_packet_length, but
it is in fact smaller by the length of the PES header.

This changes how corrupt streams are packetized:
- If PES header length is bigger than PES_packet_length then the PES packet
  payload will be handled as an unbound packet
- PES packets with payload across multiple MPEGTS packets will always be
  splitted if with the next chunk of data the payload should exceed
  PES_packet_length, previously a PES_header_length amount of excess was
  allowed.

Fixes ticket #9355.

Signed-off-by: Marton Balint <cus@passwd.hu>

libavformat/mpegts.c

@@ -1358,8 +1358,8 @@ skip:
         case MPEGTS_PAYLOAD:
             do {
                 int max_packet_size = MAX_PES_PAYLOAD;
-                if (pes->PES_packet_length)
-                    max_packet_size = pes->PES_packet_length;
+                if (pes->PES_packet_length && pes->PES_packet_length + PES_START_SIZE > pes->pes_header_size)
+                    max_packet_size = pes->PES_packet_length + PES_START_SIZE - pes->pes_header_size;
 
                 if (pes->data_index > 0 &&
                     pes->data_index + buf_size > max_packet_size) {