Some real-world sites use an authorization header with a bearer token; when
combined with lengthy request parameters to identify the video segment,
it's rather trivial these days to have a request body of more than 4k bytes.
MAX_URL_SIZE is hard-coded to 4k bytes in libavformat/internal.h, and
HTTP_HEADERS_SIZE is 4k as well in libavformat/http.h, so this patch increases
the buffer size to 8k, as that is the default request body limit in Apache, and
most other httpds seem to support at least as much, if not more.
Signed-off-by: Marton Balint <cus@passwd.hu>
v2: Use s->buffer for creating request (as the old code did) instead of
the AVBPrint internal buffer. Some minor cosmetics.
Signed-off-by: Marton Balint <cus@passwd.hu>
It is explicitly required by the HTTP RFC. Without this patch URLs like
http://example.com?query will not work.
Fixes ticket #8466.
Signed-off-by: Marton Balint <cus@passwd.hu>
RFC 3986 states that the fragment identifier is separated from the rest of the
URI prior to a dereference, and thus the identifying information within the
fragment itself is dereferenced solely by the user agent.
Signed-off-by: Marton Balint <cus@passwd.hu>
add ff_http_do_new_request2() which supports options to be applied to
HTTPContext after initialisation with the new uri
Signed-off-by: Steven Liu <lq@onvideo.cn>
Signed-off-by: vectronic <hello.vectronic@gmail.com>
this function is used to get the previous shutdown status
when reusing the old connection in block mode.
Signed-off-by: Steven Liu <lq@chinaffmpeg.org>
Fix ticket #7297
The current setting for send-expect-100 option is either
enabled if applicable or forced enabled, no option to force
disable the header. This change is to expand the option setting
to provide more flexibility, which is useful for rstp case.
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
This avoids making invalid HTTP Range requests for a byte range past the
known end of the file during a seek. Those requests generally return a HTTP
response of 416 Range Not Satisfiable, which causes an error response.
Reference: https://tools.ietf.org/html/rfc7233
Signed-off-by: Vittorio Giovara <vittorio.giovara@gmail.com>
Previously, AVERROR(EIO) was returned on failure of
http_open_cnx_internal(). Now the value is passed to upper level, thus
it is possible to distinguish ECONNREFUSED, ETIMEDOUT, ENETUNREACH etc.
Signed-off-by: Martin Storsjö <martin@martin.st>
In write only mode, the TCP receive buffer's data keeps growing with
http response messages and the buffer eventually becomes full.
This results in zero tcp window size, which in turn causes unwanted
issues, like, terminated tcp connection. The issue is apparent when
http persistent connection is enabled in hls/dash live streaming use
cases. To overcome this issue, the logic here reads the buffer data
when a file transfer is completed, so that any accumulated data in
the recieve buffer gets flushed out.
This refactors get_cookies to simplify some code paths, specifically for
skipping logic in the while loop or exiting it. It also simplifies the logic
for appending additional values to *cookies by replacing strlen/malloc/snprintf
with one call av_asnprintf.
This refactor fixes a bug where the cookie_params AVDictionary would get leaked
if we failed to allocate a new buffer for writing to *cookies.
If the string consists entirely of whitespace, this could in theory
continue to write '\0' before the start of the memory allocation. In
practice, it didn't really happen: the generic HTTP header parsing code
already skips leading whitespaces, so the string is either empty, or
consists a non-whitespace. (The generic code and the cookie code
actually have different ideas about what bytes are whitespace: the
former uses av_isspace(), the latter uses WHITESPACES. Fortunately,
av_isspace() is a super set of the http.c specific WHITESPACES, so
there's probably no case where the above assumption could have been
broken.)
It's trivial to craft a HTTP response that will make the code for
skipping trailing whitespace access and possibly overwrite bytes outside
of the memory allocation. Why this can happen is blindingly obvious: it
accesses cstr[strlen(cstr)-1] without checking whether the string is
empty.
libavformat prints a warning that the cookie couldn't be parsed (see
callers of parse_cookie()). This is obviously not true - it could be
parsed, but was simply ignored. Don't return an error to avoid the
warning.
The condition was a bit too long, and most editors will break the line
and turn it into an unreadable mess. Move out some of the conditions.
This should not change the behavior.
If the stream was aborted using the libavformat interrupt callback, we
don't want it to log the reconnect warning. (Exiting after logging this
warning worked well, so this is only for avoiding the ugly warning.)
It makes no sense to return an error after the first reconnect, and then
somehow resume the next time it's called. Usually this will lead to
demuxer errors. Make reconnecting block instead, until it has either
successfully reconnected, or given up.
Also make the wait reasonably interruptible. Since there is no mechanism
for this in the API, polling is the best we can do. This behaves roughly
the same as other interruptible network functions in libavformat.
(The original code would work if it returned AVERROR(EAGAIN) or so,
which would make retry_transfer_wrapper() repeat the read call. But I
think having an explicit loop for this is better anyway.)
I also snuck in a fix for reconnect_at_eof. It has to check for
AVERROR_EOF, not 0.
Can be used by the api user to figure out what http features the server supports based on the response received.
Signed-off-by: Aman Gupta <aman@tmm1.net>
This makes do_new_request fail early when dealing with a http/1.0 server, avoiding unnecessary "reconnecting" warnings shown to the user.
Signed-off-by: Aman Gupta <aman@tmm1.net>
This fixes a deadlock when using the hls demuxer's new http_persistent feature
to stream a youtube live stream over HTTPS. The youtube servers are http/1.1
compliant, but return a "Connecton: close". Before this commit, the demuxer
would attempt to send a new request on the partially shutdown connection and
cause a deadlock in the tls protocol.
Signed-off-by: Aman Gupta <aman@tmm1.net>
This mimics logging that was added in 53e0d5d724 for security
purposes.
Signed-off-by: Aman Gupta <aman@tmm1.net>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
This will prevent improper use of ff_http_do_new_request() if the user
tries to send a request for a different host to a previously connected
persistent http/1.1 connection.
Signed-off-by: Aman Gupta <aman@tmm1.net>
Signed-off-by: Karthick J <kjeyapal@akamai.com>
Currently if you use the multiple_requests=1 option and try to
receive a chunked-encoded response, http_buf_read() will hang forever.
After this patch, EOF is emulated once a 0-byte final chunk is
received by setting a new flag. This flag is reset in ff_http_do_new_request(),
which is used to make additional requests on the open socket.
Reviewed-by: Ronald S. Bultje <rsbultje@gmail.com>
Signed-off-by: Aman Gupta <aman@tmm1.net>
transfer_func variable passed to retry_transfer_wrapper
are h->prot->url_read and h->prot->url_write functions.
These need to return EOF or other error properly.
In case of returning >= 0, url_read/url_write is retried
until error is returned.
Signed-off-by: Daniel Kucera <daniel.kucera@gmail.com>
This fixes redirects, where the original redirect response indicated
support for compression, while the actual redirected content didn't.
Signed-off-by: Martin Storsjö <martin@martin.st>
This commit optimizes HTTP performance by reducing forward seeks, instead
favoring a read-ahead and discard on the current connection (referred to
as a short seek) for seeks that are within a TCP window's worth of data.
This improves performance because with TCP flow control, a window's worth
of data will be in the local socket buffer already or in-flight from the
sender once congestion control on the sender is fully utilizing the window.
Note: this approach doesn't attempt to differentiate from a newly opened
connection which may not be fully utilizing the window due to congestion
control vs one that is. The receiver can't get at this information, so we
assume worst case; that full window is in use (we did advertise it after all)
and that data could be in-flight
The previous behavior of closing the connection, then opening a new
with a new HTTP range value results in a massive amounts of discarded
and re-sent data when large TCP windows are used. This has been observed
on MacOS/iOS which starts with an initial window of 256KB and grows up to
1MB depending on the bandwidth-product delay.
When seeking within a window's worth of data and we close the connection,
then open a new one within the same window's worth of data, we discard
from the current offset till the end of the window. Then on the new
connection the server ends up re-sending the previous data from new
offset till the end of old window.
Example (assumes full window utilization):
TCP window size: 64KB
Position: 32KB
Forward seek position: 40KB
* (Next window)
32KB |--------------| 96KB |---------------| 160KB
*
40KB |---------------| 104KB
Re-sent amount: 96KB - 40KB = 56KB
For a real world test example, I have MP4 file of ~25MB, which ffplay
only reads ~16MB and performs 177 seeks. With current ffmpeg, this results
in 177 HTTP GETs and ~73MB worth of TCP data communication. With this
patch, ffmpeg issues 4 HTTP GETs and 3 seeks for a total of ~22MB of TCP data
communication.
To support this feature, the short seek logic in avio_seek() has been
extended to call a function to get the short seek threshold value. This
callback has been plumbed to the URLProtocol structure, which now has
infrastructure in HTTP and TCP to get the underlying receiver window size
via SO_RCVBUF. If the underlying URL and protocol don't support returning
a short seek threshold, the default s->short_seek_threshold is used
This feature has been tested on Windows 7 and MacOS/iOS. Windows support
is slightly complicated by the fact that when TCP window auto-tuning is
enabled, SO_RCVBUF doesn't report the real window size, but it does if
SO_RCVBUF was manually set (disabling auto-tuning). So we can only use
this optimization on Windows in the later case
Signed-off-by: Joel Cunningham <joel.cunningham@me.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
Reported-by: SleepProgger <security@gnutp.com>
Reviewed-by: Steven Liu <lingjiujianke@gmail.com>
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>
A negative chunk size is illegal and would end up used as
length for memcpy, where it would lead to memory accesses
out of bounds.
Found-by: Paul Cher <paulcher@icloud.com>
CC: libav-stable@libav.org
Signed-off-by: Martin Storsjö <martin@martin.st>
Joey Smith
Limin Wang
Marton Balint
Steven Liu
vectronic
Jun Li
Carl Eugen Hoyos
Justin Ruggles
Wenxiang Qian
Gyan Doshi
kjeyapal@akamai.com
Andrey Utkin
Clément Bœsch
Vishwanath Dixit
Stephan Holljes
Richard Shaffer
wm4
Aman Gupta
Aman Gupta
Daniel Kucera
Jacek Jendrzej
Clément Bœsch
raymondzheng
Micah Galizia
Joel Cunningham
Michael Niedermayer
Martin Storsjö