Some external users swap out the entire IP stack, such as
[Seastar](https://github.com/scylladb/seastar) that uses DPDK. So we
need to allow them to override all network stack functions used by
c-ares. Since we don't know what network functions may be needed in the
future, we need to make a versioned structure that can be expanded.
We'll need to disable any features in the future if the versioned
interface is less than is required to support any new features.
We recently had a regression when we added DNS Cookie support as we
relied on `getsockname()` to get the local source ip address, but we
were calling the native version since there was no registered callback
available. We will mark this as optional and just skip this step in DNS
Cookie invalidation if its not available for the legacy api users.
Supersedes #893
Authored-By: Brad House (@bradh352)
The c-ares dig utility outputs a similar format to the bind dig utility,
we should also make it take arguments that mimic theirs for
compatibility.
This PR starts down that path supporting some of the major options.
Fixes#885
Authored-By: Brad House (@bradh352)
The DNS server format is insufficient for future configurations, such as
supporting DNS over TLS (DoT) and DNS over HTTPS (DoH), as well as
additional functionality such as domain-specific servers. Already, in
the case where different UDP and TCP ports are used, it is impossible to
represent in the current format.
In order to try to use some standardized format, we are going to define
our own URI schemes that should be parse-able by any URI parser. The new
scheme will only be used when the configuration cannot otherwise be
expressed using the current `ipaddr%iface:port` format, which is the
format used as the nameserver configuration in `/etc/resolv.conf`.
However, the parser `ares_set_servers_csv()` shall accept the new URI
scheme format even when it is not necessary.
This PR implements a URI parser and writer and hooks the basic usage
into `ares_set_servers_csv()` and `ares_get_servers_csv()` as well as
provides updated documentation in the relevant manpages.
We will define these URI schemes:
* `dns://` - Normal DNS server (UDP + TCP). We need to be careful not to
conflict with query params defined in
https://datatracker.ietf.org/doc/html/rfc4501 since we'd technically be
extending this URI scheme. Port defaults to `53`.
* `dns+tls://` - DNS over TLS. Port defaults to `853`.
* `dns+https://` - DNS over HTTPS. Port defaults to `443`.
We initially will define these query parameters (additional arguments
may be required in the future to specify options such as TLS certificate
validation rules):
* `tcpport` - TCP port to use, only for `dns://` scheme. The `port`
specified as part of the `authority` component of the URI will be used
for both UDP and TCP by default, this option will override the TCP port.
* `ipaddr` - Only for `dns+tls://` and `dns+https://`. If the
`authority` component of the URI contains a hostname, this is used to
specify the ip address of the hostname. If not specified, will need to
use a non-secure server to perform a DNS lookup to retrieve this
information. It is always recommended to have both the ip address and
fully qualified domain name specified.
* `hostname` - Only for `dns+tls://` and `dns+https://`. If the
`authority` component of the URI contains an ip address, this is used to
specify the fully qualified domain name of the server. If not specified,
will need to use a non-secure server to perform a DNS reverse lookup to
retrieve this information. It is always recommended to have both the ip
address and fully qualified domain name specified.
* `domain` - If specified, this server is a domain-specific server. Any
queries for this domain will be routed to this server. Multiple servers
may be tagged with the same domain.
Examples:
```
dns://8.8.8.8
dns://[2001:4860:4860::8888]
dns://[fe80::b542:84df:1719:65e3%en0]
dns://192.168.1.1:55
dns://192.168.1.1?tcpport=1153
dns://10.0.1.1?domain=myvpn.com
dns+tls://8.8.8.8?hostname=dns.google
dns+tls://one.one.one.one?ipaddr=1.1.1.1
```
NOTE: While we are defining the scheme for things like domain-specific
servers, DNS over TLS and DNS over HTTPS, the underlying implementations
for those features do not yet exist and therefore will result in errors
if they are attempted to be used.
### Non-compliance in implementation
All these could be easily implemented/fixed if desired, however any such
changes would be of no use to the current c-ares usage of URIs:
* Does not currently support relative references
* Requires use of the authority section, blank is not allowed
* The query string is interpreted to be in
[application/x-www-form-urlencoded](https://en.wikipedia.org/wiki/Application/x-www-form-urlencoded)
format only and will result in parse errors if it is not. This is the
most common format used, however technically not valid to mandate this
format is used. We could add flags in the future to treat the query
string as opaque and leave it to the user to process. Or we could
internally have a list of schemes that use this format.
* [IDNA](https://en.wikipedia.org/wiki/Internationalized_domain_name) is
not supported.
* Does not support hex-encoded IPv4 addresses (this is compliant with RFC3986, but not WHATWG)
Authored-By: Brad House (@bradh352)
The previous implementation would redirect a query to a failed server
based on a timeout and random chance per query. This could lead to
issues of having to deal with server timeout scenarios when the server
isn't back online yet causing latency issues. Instead, we should
continue to use the known good servers for the query itself, but spawn a
second query with the same question to a different downed server. That
query will be able to be processed in the background and potentially
bring the server back online.
Also, when using the `rotate` option, servers were previously chosen at
random from the complete list. This PR changes that to choose only from
the servers that share the same highest priority.
Authored-By: Brad House (@bradh352)
Split socket from connection functions for a more clear delineation in
functionality. This is needed to support future DNS over TLS.
Also makes the new ares_process_fds() return a status code so we can
detect out of memory conditions, this too is going to be needed for
handling TLS which is more likely to run into such a situation.
Finally, add in a new ares_htable_vpstr_t type and functions which will
be needed as part of TLS session lookups.
Authored-By: Brad House (@bradh352)
Add a new public function of `ares_process_fds()` to work around the
shortcomings of `ares_process_fd()` and `ares_process()`.
This new function allows integrators to specify more than one file
descriptor at a time in an array, with an event mask per file
descriptor. It also adds a flags parameter which can be used to skip
extra expensive processing if they may be calling the function multiple
times before waiting on more events.
This new function is used by the event thread, and all the other
functions internally call this new function simplifying the code
structure.
Authored-By: Brad House (@bradh352)
The new notify pending write feature name should be reduced as ISO C
specifies compilers can support unique identifiers of as few as 31
characters.
Since this feature has not yet been made public in a release we
should go ahead and fix this issue so we aren't stuck with it.
Authored-By: Brad House (@bradh352)
This allows for default arguments to be provided through a user
configuration file, useful for avoiding repetitive passing of flags on
the command line when using adig frequently. The syntax within
$XDG_CONFIG_HOME/adigrc resembles the one from dig's ~/.digrc, meaning
any command line arguments can be provided, separated either by line
breaks or whitespaces.
---------
Authored-By: Georg Pfuetzenreuter <mail@georg-pfuetzenreuter.net>
We don't want people using ares_process() anymore due to issues/limitations
in select(). Lets also remove the example and reword the manpage.
Authored-By: Brad House (@bradh352)
The main purpose of this PR is to modularize the communication library,
and streamline the code paths for TCP and UDP into a single flow. This
will help us add additional flows such as TLS, and also make sure these
communication functions return a known set of error codes so that
additional error codes can be added for new flows in the future.
It also adds a new optional callback of
`ares_set_notify_pending_write_callback()` that will assist in
aggregating data from multiple queries into a single write operation.
This doesn't apply to UDP, but on TCP and especially TLS in the future
this can be a significant win. This is automatically applied for the
Event Thread.
It also fixes a long standing issue if UDP connection became saturated
and started returning `EWOULDBLOCK` or `EAGAIN` it was treated as a
failure. Since this is more inline with the TCP code now, it will wait
on a write event to retry.
Finally there are additional cleanups due to not needing to be able to
retrieve socket errnos all over the place.
Authored-By: Brad House (@bradh352)
DNS cookies are a simple form of learned mutual authentication supported
by most DNS server implementations these days and can help prevent DNS
Cache Poisoning attacks for clients and DNS amplification attacks for
servers.
Fixes#620
Fix By: Brad House (@bradh352)
As per #738, there are usecases where the DNS TXT record strings should
not be concatenated like RFC 7208 indicates. We cannot break ABI with
those using the new API, so we need to support retrieving the
concatenated version as well as a new API to retrieve the individual
strings which will be used by `ares_parse_text_reply_ext()` to restore
the old behavior prior to c-ares 1.20.
Fixes Issue: #738
Fix By: Brad House (@bradh352)
This PR enables DNS 0x20 as per
https://datatracker.ietf.org/doc/html/draft-vixie-dnsext-dns0x20-00 .
DNS 0x20 adds additional entropy to the request by randomly altering the
case of the DNS question to help prevent cache poisoning attacks.
Google DNS has implemented this support as of 2023, even though this is
a proposed and expired standard from 2008:
https://groups.google.com/g/public-dns-discuss/c/KxIDPOydA5M
There have been documented cases of name server and caching server
non-conformance, though it is expected to become more rare, especially
since Google has started using this.
This can be enabled via the `ARES_FLAG_DNS0x20` flag, which is currently
disabled by default. The test cases do however enable this flag to
validate this feature.
Implementors using this flag will notice that responses will retain the
mixed case, but since DNS names are case-insensitive, any proper
implementation should not be impacted.
There is currently no fallback mechanism implemented as it isn't
immediately clear how this may affect a stub resolver like c-ares where
we aren't querying the authoritative name server, but instead an
intermediate recursive resolver where some domains may return invalid
results while others return valid results, all while querying the same
nameserver. Likely using DNS cookies as suggested by #620 is a better
mechanism to fight cache poisoning attacks for stub resolvers.
TCP queries do not use this feature even if the `ARES_FLAG_DNS0x20` flag
is specified since they are not subject to cache poisoning attacks.
Fixes Issue: #795
Fix By: Brad House (@bradh352)
With very little effort we should be able to determine fairly proper
timeouts we can use based on prior query history. We track in order to
be able to auto-scale when network conditions change (e.g. maybe there
is a provider failover and timings change due to that). Apple appears to
do this within their system resolver in MacOS. Obviously we should have
a minimum, maximum, and initial value to make sure the algorithm doesn't
somehow go off the rails.
Values:
- Minimum Timeout: 250ms (approximate RTT half-way around the globe)
- Maximum Timeout: 5000ms (Recommended timeout in RFC 1123), can be
reduced by ARES_OPT_MAXTIMEOUTMS, but otherwise the bound specified by
the option caps the retry timeout.
- Initial Timeout: User-specified via configuration or
ARES_OPT_TIMEOUTMS
- Average latency multiplier: 5x (a local DNS server returning a cached
value will be quicker than if it needs to recurse so we need to account
for this)
- Minimum Count for Average: 3. This is the minimum number of queries we
need to form an average for the bucket.
Per-server buckets for tracking latency over time (these are ephemeral
meaning they don't persist once a channel is destroyed). We record both
the current timespan for the bucket and the immediate preceding timespan
in case of roll-overs we can still maintain recent metrics for
calculations:
- 1 minute
- 15 minutes
- 1 hr
- 1 day
- since inception
Each bucket contains:
- timestamp (divided by interval)
- minimum latency
- maximum latency
- total time
- count
NOTE: average latency is (total time / count), we will calculate this
dynamically when needed
Basic algorithm for calculating timeout to use would be:
- Scan from most recent bucket to least recent
- Check timestamp of bucket, if doesn't match current time, continue to
next bucket
- Check count of bucket, if its not at least the "Minimum Count for
Average", check the previous bucket, otherwise continue to next bucket
- If we reached the end with no bucket match, use "Initial Timeout"
- If bucket is selected, take ("total time" / count) as Average latency,
multiply by "Average Latency Multiplier", bound by "Minimum Timeout" and
"Maximum Timeout"
NOTE: The timeout calculated may not be the timeout used. If we are
retrying
the query on the same server another time, then it will use a larger
value
On each query reply where the response is legitimate (proper response or
NXDOMAIN) and not something like a server error:
- Cycle through each bucket in order
- Check timestamp of bucket against current timestamp, if out of date
overwrite previous entry with values, clear current values
- Compare current minimum and maximum recorded latency against query
time and adjust if necessary
- Increment "count" by 1 and "total time" by the query time
Other Notes:
- This is always-on, the only user-configurable value is the initial
timeout which will simply re-uses the current option.
- Minimum and Maximum latencies for a bucket are currently unused but
are there in case we find a need for them in the future.
Fixes Issue: #736
Fix By: Brad House (@bradh352)
The query cache should be enabled by default. This will help with
determining proper timeouts for #736. It can still be disabled by
setting the ttl to 0. There should be no negative consequences of this
in real-world scenarios since DNS is based on the TTL concept and
upstream servers will cache results and not recurse based on this
information anyhow. DNS queries and responses are very small, this
should have negligible impact on memory consumption.
Fix By: Brad House (@bradh352)
With the current c-ares parser, as per PR #765 parsing was broken due to
validation that didn't understand the `SIG` record class. This PR adds
basic, non validating, and incomplete support for the `SIG` record type.
The additional `KEY` and `NXT` which would be required for additional
verification of the records is not implemented. It also does not store
the raw unprocessed RR data that would be required for the validation.
The primary purpose of this PR is to be able to recognize the record and
handle some periphery aspects such as validation of the class associated
with the RR and to not honor the TTL in the RR in the c-ares query cache
since it will always be 0.
Fixes#765
Fix By: Brad House (@bradh352)
Automatically detect configuration changes and reload. On systems which
provide notification mechanisms, use those, otherwise fallback to
polling. When a system configuration change is detected, it
asynchronously applies the configuration in order to ensure it is a
non-blocking operation for any queries which may still be being
processed.
On Windows, however, changes aren't detected if a user manually
sets/changes the DNS servers on an interface, it doesn't appear there is
any mechanism capable of this. We are relying on
`NotifyIpInterfaceChange()` for notifications.
Fixes Issue: #613
Fix By: Brad House (@bradh352)
at https://github.com/c-ares/c-ares/pull/601#issuecomment-1801935063 you
chose not to scatter `const` on the public interface because of the plan
- now realised - to add threading to c-ares, and in the expectation that
even read operations would need to lock the mutex.
But the threading implementation has a _pointer_ to a mutex inside the
ares channel and as I understand it, that means that it is just fine to
mark `ares__channel_lock` (and `ares__channel_unlock`) as taking a
`const` channel. It is the pointed-to mutex that is not constant, but C
does not propagate `const`-ness through pointers.
This PR sprinkles const where appropriate on public interfaces.
Fix By: David Hotham (@dimbleby)
**Summary**
This PR adds a server state callback that is invoked whenever a query to
a DNS server finishes.
The callback is invoked with the server details (as a string), a boolean
indicating whether the query succeeded or failed, flags describing the
query (currently just indicating whether TCP or UDP was used), and
custom userdata.
This can be used by user applications to gain observability into DNS
server health and usage. For example, alerts when a DNS server
fails/recovers or metrics to track how often a DNS server is used and
responds successfully.
**Testing**
Three new regression tests `MockChannelTest.ServStateCallback*` have
been added to test the new callback in different success/failure
scenarios.
Fix By: Oliver Welsh (@oliverwelsh)
**Summary**
By default c-ares will select the server with the least number of
consecutive failures when sending a query. However, this means that if a
server temporarily goes down and hits failures (e.g. a transient network
issue), then that server will never be retried until all other servers
hit the same number of failures.
This is an issue if the failed server is preferred to other servers in
the list. For example if a primary server and a backup server are
configured.
This PR adds new server failover retry behavior, where failed servers
are retried with small probability after a minimum delay has passed. The
probability and minimum delay are configurable via the
`ARES_OPT_SERVER_FAILOVER` option. By default c-ares will use a
probability of 10% and a minimum delay of 5 seconds.
In addition, this PR includes a small change to always close out
connections to servers which have hit failures, even with
`ARES_FLAG_STAYOPEN`. It's possible that resetting the connection can
resolve some server issues (e.g. by resetting the source port).
**Testing**
A new set of regression tests have been added to test the new server
failover retry behavior.
Fixes Issue: #717
Fix By: Oliver Welsh (@oliverwelsh)
As per Issue #734 some people use `ndots:0` in their configuration which
is allowed by the system resolver but not by c-ares. Add support for
`ndots:0` and add a test case to validate this behavior.
Fixes Issue: #734
Fix By: Brad House (@bradh352)
c-ares has historically passed around raw dns packets in binary form.
Now that we have a new parser, and messages are already parsed
internally, lets pass around that parsed message rather than requiring
multiple parse attempts on the same message. Also add a new
`ares_send_dnsrec()` and `ares_query_dnsrec()` similar to
`ares_search_dnsrec()` added with PR #719 that can return the pointer to
the `ares_dns_record_t` to the caller enqueuing queries and rework
`ares_search_dnsrec()` to use `ares_send_dnsrec()` internally.
Fix By: Brad House (@bradh352)
This PR adds a new function `ares_search_dnsrec()` to search for records
using the new DNS record parser.
The function takes an arbitrary DNS record object to search (that must
represent a query for a single name). The function takes a new callback
type, `ares_callback_dnsrec`, that is invoked with a parsed DNS record
object rather than the raw buffer(+length).
The original motivation for this change is to provide support for
[draft-kaplan-enum-sip-routing-04](https://datatracker.ietf.org/doc/html/draft-kaplan-enum-sip-routing-04);
when routing phone calls using an ENUM server, it can be useful to
include identifying source information in an OPT RR options value, to
help select the appropriate route for the call. The new function allows
for more customisable searches like this.
**Summary of code changes**
A new function `ares_search_dnsrec()` has been added and exposed.
Moreover, the entire `ares_search_int()` internal code flow has been
refactored to use parsed DNS record objects and the new DNS record
parser. The DNS record object is passed through the `search_query`
structure by encoding/decoding to/from a buffer (if multiple search
domains are used). A helper function `ares_dns_write_query_altname()` is
used to re-write the DNS record object with a new query name (used to
append search domains).
`ares_search()` is now a wrapper around the new internal code, where the
DNS record object is created based on the name, class and type
parameters.
The new function uses a new callback type, `ares_callback_dnsrec`. This
is invoked with a parsed DNS record object. For now, we convert from
`ares_callback` to this new type using `ares__dnsrec_convert_cb()`.
Some functions that are common to both `ares_query()` and
`ares_search()` have been refactored using the new DNS record parser.
See `ares_dns_record_create_query()` and
`ares_dns_query_reply_tostatus()`.
**Testing**
A new FV has been added to test the new function, which searches for a
DNS record containing an OPT RR with custom options value.
As part of this, I needed to enhance the mock DNS server to expect
request text (and assert that it matches actual request text). This is
because the FV needs to check that the request contains the correct OPT
RR.
**Documentation**
The man page docs have been updated to describe the new feature.
**Futures**
In the future, a new variant of `ares_send()` could be introduced in the
same vein (`ares_send_dnsrec()`). This could be used by
`ares_search_dnsrec()`. Moreover, we could migrate internal code to use
`ares_callback_dnsrec` as the default callback.
This will help to make the new DNS record parser the norm in C-Ares.
---------
Co-authored-by: Oliver Welsh (@oliverwelsh)
Since acountry cannot be restored due to nerd.dk being decommissioned,
we should completely remove the manpage and source. This also
will resolve issue #718.
Fixes Issue: #718
Fix By: Brad House (@bradh352)
Hello, I work on an application for Microsoft which uses c-ares to
perform DNS lookups. We have made some minor changes to the library over
time, and would like to contribute these back to the project in case
they are useful more widely. This PR adds a new channel init flag,
described below.
Please let me know if I can include any more information to make this PR
better/easier for you to review. Thanks!
**Summary**
When initializing a channel with `ares_init_options()`, if there are no
nameservers available (because `ARES_OPT_SERVERS` is not used and
`/etc/resolv.conf` is either empty or not available) then a default
local named server will be added to the channel.
However in some applications a local named server will never be
available. In this case, all subsequent queries on the channel will
fail.
If we know this ahead of time, then it may be preferred to fail channel
initialization directly rather than wait for the queries to fail. This
gives better visibility, since we know that the failure is due to
missing servers rather than something going wrong with the queries.
This PR adds a new flag `ARES_FLAG_NO_DFLT_SVR`, to indicate that a
default local named server should not be added to a channel in this
scenario. Instead, a new error `ARES_EINITNOSERVER` is returned and
initialization fails.
**Testing**
I have added 2 new FV tests:
- `ContainerNoDfltSvrEmptyInit` to test that initialization fails when
no nameservers are available and the flag is set.
- `ContainerNoDfltSvrFullInit` to test that initialization still
succeeds when the flag is set but other nameservers are available.
Existing FVs are all passing.
**Documentation**
I have had a go at manually updating the docs to describe the new
flag/error, but couldn't see any contributing guidance about testing
this. Please let me know if you'd like anything more here.
---------
Fix By: Oliver Welsh (@oliverwelsh)
Add a function to request the number of active queries from an ares
channel. This will return the number of inflight requests to dns
servers. Some functions like `ares_getaddrinfo()` when using `AF_UNSPEC`
may enqueue multiple queries which will be reflected in this count.
In the future, if we implement support for queuing (e.g. for throttling
purposes), and/or implement support for tracking user-requested queries
(e.g. for cancelation), we can provide additional functions for
inspecting those queues.
Fix By: Brad House (@bradh352)
It may be useful to wait for the queue to be empty under certain conditions (mainly test cases), expose a function to efficiently do this and rework test cases to use it.
Fix By: Brad House (@bradh352)
Brad House
Cristian Rodríguez
Georg
Gregor Jasny
David Hotham
Oliver Welsh
Andriy Utkin