Commit 8c8e5d5286 added a way to reduce
seek time by waiting for the windowed tcp packets instead of creating a
new socket connection. It implemented this by overwriting
s->short_seek_threshold in avio_seek(). However,
s->short_seek_threshold could already be set and be higher than the
threshold set by the protocol (i.e. s->short_seek_threshold is set in
ff_configure_buffers_for_index()).
This new feature was only enabled for tls connections in
70d8077b79. As in Ticket #9148 it reduced
performance because instead of waiting to refill the AVIOContext buffers
with an existing connections, a new HTTP request was often made instead.
Fixes Ticket #9148.
Reviewed-by: Martin Storsjö <martin@martin.st>
Signed-off-by: Andriy Gelman <andriy.gelman@gmail.com>
The previous threshold, 4 KB, maybe was reasonable when it was set
(in 2010), but in today's settings and with typical network speeds
and data sizes, it's pretty small. 32 KB probably is a more reasonable
default now, regardless of input.
This changes the test references for two seek tests.
When using the normal seek function, which boils down to the lseek(2)
function, a seek to an out of bounds position doesn't return an error,
but that condition is only reported when doing the subsequent read
(which returns EOF). When doing more seeks by fast forwarding, the
fact that the seeked to destination is out of bounds is noticed and
reported sooner in these cases.
Signed-off-by: Martin Storsjö <martin@martin.st>
This should increase the effectiveness of ffio_ensure_seekback by reducing the
number of buffer reallocations and memmoves/memcpys because even a small
seekback window requires max_buffer_size+window_size buffer space.
Signed-off-by: Marton Balint <cus@passwd.hu>
Previously ffio_ensure_seekback never flushed the buffer, so successive
ffio_ensure_seekback calls were all respected. This could eventually cause
unlimited memory and CPU usage if a demuxer called ffio_ensure_seekback on all
it's read data.
Most demuxers however only rely on being able to seek back till the position of
the last ffio_ensure_seekback call, therefore we change the semantics of
ffio_ensure_seekback so that a new call can invalidate seek guarantees of the
old. In order to support some level of "nested" ffio_ensure_seekback calls, we
document that the function only invalidates the old window (and potentially
discards the already read data from the IO buffer), if the newly requested
window does not fit into the old one.
This way we limit the memory usage for ffio_ensure_seekback calls requesting
consecutive data windows.
Signed-off-by: Marton Balint <cus@passwd.hu>
The new buf_size was detemined too conservatively, maybe because of the
off-by-one issue which was fixed recently in fill_buffer. We can safely
substract 1 more from the new buffer size, because max_buffer_size space must
only be guaranteed when we are reading the last byte of the requested window.
Comparing the new buf_size against filled did not make a lot of sense, what
makes sense is that we want to reallocate the buffer if the new buf_size is
bigger than the old, therefore the change in the check.
Signed-off-by: Marton Balint <cus@passwd.hu>
Existing code did not check if the requested seekback buffer is
already read entirely. In this case, nothing has to be done to guarantee
seekback.
Signed-off-by: Marton Balint <cus@passwd.hu>
There was an off-by-one error when checking if the IO buffer still has enough
space till the end. One more byte can be safely written.
Signed-off-by: Marton Balint <cus@passwd.hu>
Two kinds of errors can happen when working with dynamic buffers:
(Re)allocation errors or truncation errors (one has to truncate the
buffer to a size of INT_MAX because avio_close_dyn_buf() and
avio_get_dyn_buf() both return an int). Right now, avio_get_dyn_buf()
returns an empty buffer in either case. But given that
avio_get_dyn_buf() does not destroy the dynamic buffer, one can return
the buffer in case of truncation and let the user check the error flags
and decide for himself instead of hardcoding a single way to proceed
in case of truncation.
(This actually restores the behaviour from before commit
163bb9ac0af495a5cb95441bdb5c02170440d28c.)
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
This has originally been done in 568e18b15e
as a precaution against integer overflows, but it is actually easy to
support the full range of int without overflows.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
If adding two ints overflows, it doesn't matter whether the result will
be stored in an unsigned or not; and checking afterwards does not make it
retroactively defined.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
and make it static again.
These functions have been moved from nutenc to aviobuf and internal.h
in f8280ff4c0 in order to use them in a
forthcoming patch in utils.c. Said patch never happened, so this commit
moves them back and makes them static, effectively reverting said
commit as well as f8280ff4c0 (which added
the ff-prefix to these functions).
Reviewed-by: Michael Niedermayer <michael@niedermayer.cc>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Resetting a dynamic buffer means to keep the AVIOContext and the
internal buffer used by the dynamic buffer. This is done in order to
save (re)allocations when one has a workflow where one opens and closes
dynamic buffers in sequence.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
In the Libav commit cae448cf, the opaque of every AVIOContext opened
by ffio_fdopen() (which is used internally by avio_open() and avio_open2())
changed: It was a simple pointer to an URLContext before, but now it was
a structure (namely AVIOInternal) containing a pointer to an URLContext
as its only member. The next commits (namely 8c0ceafb and ec4c4839) added
members to AVIOInternal to allow white-/blacklisting of protocols.
But these two commits were never merged into FFmpeg (they were only
merged as no-ops in 510046c2 and 063b26d3), because FFmpeg chose
a different way to implement this (in 93629735); and so our AVIOInternal
still has exactly one member.
This of course means that it is unnecessary to use AVIOInternal as
opaque as it is just adding a level of indirection (not only pointer
dereference, but also wrapper functions). Therefore this commit
removes AVIOInternal entirely and essentially reverts cae448cf.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Reviewed-by: Paul B Mahol <onemda@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
The documentation of both avio_open() as well as avio_open2() states
that on failure, the pointer to an AVIOContext given to this function
(via a pointer to a pointer to an AVIOContext) will be set to NULL. Yet
it didn't happen upon failure of ffurl_open_whitelist() or when allocating
the internal buffer failed. This commit changes this.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Up until now, using a dynamic buffer entailed at least three
allocations: One for the AVIOContext, one for the AVIOContext's opaque
(which, among other things, contains the small write buffer), and one
for the big buffer that is independently allocated that is returned when
calling avio_close_dyn_buf().
It is possible to avoid the third allocation if one doesn't use a
packetized dynamic buffer, if all the data written so far fit into the
write buffer and if one does not require the actual (big) buffer to have
an indefinite lifetime. This is done by making avio_get_dyn_buf() return
a pointer to the data in the write buffer if nothing has been written to
the main buffer yet. The dynamic buffer will then be freed using
ffio_free_dynamic_buffer (which needed to be modified not to call
avio_close_dyn_buf() internally).
So a typical use-case like:
size = avio_close_dyn_buf(dyn_pb, &buf);
do something with buf
av_free(buf);
can be converted to:
size = avio_get_dyn_buf(dyn_pb, &buf);
do something with buf
ffio_free_dynamic_buffer(&dyn_pb);
In more complex scenarios this can simplify freeing as well, because it
is now clear that freeing always has to be performed via
ffio_free_dynamic_buffer().
Of course, in case this saves an allocation it also saves a memcpy.
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@gmail.com>
These functions can be used to print a variable number of strings consecutively
to the IO context. Unlike av_bprintf, no temporary buffer is necessary.
Signed-off-by: Marton Balint <cus@passwd.hu>
This makes sure no field is ever used uninitialized.
Reviewed-by: Carl Eugen Hoyos <ceffmpeg@gmail.com>
Reviewed-by: wm4 <nfxjfg@googlemail.com>
Signed-off-by: James Almer <jamrial@gmail.com>
Print a warning to let applicatios fix their use.
After a deprecation period, check with a low-level assert.
Also make the constraint explicit in the doxygen comment.
Signed-off-by: Nicolas George <george@nsup.org>
transfer_func variable passed to retry_transfer_wrapper
are h->prot->url_read and h->prot->url_write functions.
These need to return EOF or other error properly.
In case of returning >= 0, url_read/url_write is retried
until error is returned.
Signed-off-by: Daniel Kucera <daniel.kucera@gmail.com>
Main use-case is proxying avio through a foreign I/O layer and a custom
AVIO context, without losing latency and performance characteristics.
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Merged from Libav commit 173b56218f.
Before this commit, AVIOContext is to be freed with a plain av_free(),
which prevents us from adding any deeper structure to it.
(cherry picked from commit 99684f3ae7)
Signed-off-by: James Almer <jamrial@gmail.com>
Main use-case is proxying avio through a foreign I/O layer and a custom
AVIO context, without losing latency and performance characteristics.
Signed-off-by: Luca Barbato <lu_zero@gentoo.org>
Signed integer overflow is undefined behavior.
Detected with clang and -fsanitize=signed-integer-overflow
Signed-off-by: Vitaly Buka <vitalybuka@google.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
This patch makes aviobuf work more like traditinal file IO, which is how people
think about it.
For example, in the past, aviobuf only flushed buffers until the current buffer
position, even if more data was written to it previously, and a backward seek
was used to reposition the IO context.
From now, aviobuf will keep track of the written data, so no explicit seek will
be required till the end of the buffer, or till the end of file before flushing.
This fixes at least one regression, fate-vsynth3-flv was broken if
flush_packets option was set to false, an explicit seek was removed in
4e3cc4bdd8.
Also from now on, if a forward seek in the write buffer were to cause a gap
between the already written data and the new file position, a flush will
happen.
The must_flush varable is also removed, which might have caused needless
flushes with multiple seeks whithin the write buffer. Since we know the amount
of data written to it, we will know when to flush.
Signed-off-by: Marton Balint <cus@passwd.hu>
This commit optimizes HTTP performance by reducing forward seeks, instead
favoring a read-ahead and discard on the current connection (referred to
as a short seek) for seeks that are within a TCP window's worth of data.
This improves performance because with TCP flow control, a window's worth
of data will be in the local socket buffer already or in-flight from the
sender once congestion control on the sender is fully utilizing the window.
Note: this approach doesn't attempt to differentiate from a newly opened
connection which may not be fully utilizing the window due to congestion
control vs one that is. The receiver can't get at this information, so we
assume worst case; that full window is in use (we did advertise it after all)
and that data could be in-flight
The previous behavior of closing the connection, then opening a new
with a new HTTP range value results in a massive amounts of discarded
and re-sent data when large TCP windows are used. This has been observed
on MacOS/iOS which starts with an initial window of 256KB and grows up to
1MB depending on the bandwidth-product delay.
When seeking within a window's worth of data and we close the connection,
then open a new one within the same window's worth of data, we discard
from the current offset till the end of the window. Then on the new
connection the server ends up re-sending the previous data from new
offset till the end of old window.
Example (assumes full window utilization):
TCP window size: 64KB
Position: 32KB
Forward seek position: 40KB
* (Next window)
32KB |--------------| 96KB |---------------| 160KB
*
40KB |---------------| 104KB
Re-sent amount: 96KB - 40KB = 56KB
For a real world test example, I have MP4 file of ~25MB, which ffplay
only reads ~16MB and performs 177 seeks. With current ffmpeg, this results
in 177 HTTP GETs and ~73MB worth of TCP data communication. With this
patch, ffmpeg issues 4 HTTP GETs and 3 seeks for a total of ~22MB of TCP data
communication.
To support this feature, the short seek logic in avio_seek() has been
extended to call a function to get the short seek threshold value. This
callback has been plumbed to the URLProtocol structure, which now has
infrastructure in HTTP and TCP to get the underlying receiver window size
via SO_RCVBUF. If the underlying URL and protocol don't support returning
a short seek threshold, the default s->short_seek_threshold is used
This feature has been tested on Windows 7 and MacOS/iOS. Windows support
is slightly complicated by the fact that when TCP window auto-tuning is
enabled, SO_RCVBUF doesn't report the real window size, but it does if
SO_RCVBUF was manually set (disabling auto-tuning). So we can only use
this optimization on Windows in the later case
Signed-off-by: Joel Cunningham <joel.cunningham@me.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Andreas Rheinhardt
James Almer
Andreas Rheinhardt
Andriy Gelman
Martin Storsjö
Michael Niedermayer
Marton Balint
Anton Khirnov
Paul B Mahol
tomajsjiang
Nicolas George
Jun Zhao
Carl Eugen Hoyos
Jun Zhao
Jeyapal, Karthick
Daniel Kucera
wm4
Vitaly Buka
Joel Cunningham