The following attributes were completely unused:
- kOffset
- kIndex
- kSize
- kFilename
- kKey
- kValue
The following attributes were added but never programmatically accessed, and I've moved them into the status messages themselves, which is another step toward #22883:
- kErrorNo
- kTsiCode
- kWsaError
- kHttpStatus
- kOsError
- kSyscall
- kTargetAddress
- kRawBytes
- kTsiError
Closes#36523
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36523 from markdroth:grpc_error_attribute_cleanup b289c399fe
PiperOrigin-RevId: 639147583
[grpc][Gpr_To_Absl_Logging] Migrating from gpr to absl logging - gpr_log
In this CL we are migrating from gRPCs own gpr logging mechanism to absl logging mechanism. The intention is to deprecate gpr_log in the future.
We have the following mapping
1. gpr_log(GPR_INFO,...) -> LOG(INFO)
2. gpr_log(GPR_ERROR,...) -> LOG(ERROR)
3. gpr_log(GPR_DEBUG,...) -> VLOG(2)
Reviewers need to check :
1. If the above mapping is correct.
2. The content of the log is as before.
gpr_log format strings did not use string_view or std::string . absl LOG accepts these. So there will be some elimination of string_view and std::string related conversions. This is expected.
Closes#36703
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36703 from tanvi-jagtap:test_core_gpr_log_01 26c4307b08
PiperOrigin-RevId: 636801504
In a follow-up PR, we will use these helper APIs to add validation checks when creating TLS credentials objects.
Closes#36357
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36357 from matthewstevenson88:pem-util dca4f2e6c5
PiperOrigin-RevId: 636162540
[grpc][Gpr_To_Absl_Logging] Migrating from gpr to absl logging - gpr_log
In this CL we are migrating from gRPCs own gpr logging mechanism to absl logging mechanism. The intention is to deprecate gpr_log in the future.
We have the following mapping
1. gpr_log(GPR_INFO,...) -> LOG(INFO)
2. gpr_log(GPR_ERROR,...) -> LOG(ERROR)
3. gpr_log(GPR_DEBUG,...) -> VLOG(2)
Reviewers need to check :
1. If the above mapping is correct.
2. The content of the log is as before.
gpr_log format strings did not use string_view or std::string . absl LOG accepts these. So there will be some elimination of string_view and std::string related conversions. This is expected.
Closes#36608
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36608 from tanvi-jagtap:regex_test_core_misc e6b9481dac
PiperOrigin-RevId: 634246311
[grpc][Gpr_To_Absl_Logging] Migrating from gpr to absl logging - BUILD
In this CL we are just editing the build and bzl files to add dependencies.
This is done to prevent merge conflict and constantly having to re-make the make files using generate_projects.sh for each set of changes.
Closes#36606
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36606 from tanvi-jagtap:build_test_core_tsi_and_misc 708a724c46
PiperOrigin-RevId: 633518709
[grpc][Gpr_To_Absl_Logging] Migrating from gpr to absl logging GPR_ASSERT
Replacing GPR_ASSERT with absl CHECK
These changes have been made using string replacement and regex.
Will not be replacing all instances of CHECK with CHECK_EQ , CHECK_NE etc because there are too many callsites. Only ones which are doable using very simple regex with least chance of failure will be replaced.
Given that we have 5000+ instances of GPR_ASSERT to edit, Doing it manually is too much work for both the author and reviewer.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36442
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36442 from tanvi-jagtap:tjagtap_xds_tsi_address_utils 5c10771ac1
PiperOrigin-RevId: 628081981
grpc][Gpr_To_Absl_Logging] Migrating from gpr to absl logging GPR_ASSERT
Replacing GPR_ASSERT with absl CHECK
These changes have been made using string replacement
Will not be replacing all instances of CHECK with CHECK_EQ , CHECK_NE etc because there are too many callsites. Only ones which are doable using very simple regex with least chance of failure will be replaced.
Given that we have 5000+ instances of GPR_ASSERT to edit, Doing it manually is too much work for both the author and reviewer.
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#36368
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36368 from tanvi-jagtap:tjagtap_test_02 0072bd8854
PiperOrigin-RevId: 626114257
There were some failures in the Crl Directory related tests after https://github.com/grpc/grpc/pull/36031
This came down to https://github.com/grpc/grpc/pull/36031 adding some CRLs with bad qualities (invalid content/signatures, overriding issuer names) to the `test_creds/crl_data/crls` directory, which is used in the directory reloading tests. The tests began failing on some platforms because they were picking up these bad crls which were failing various checks, but the test was designed to assume that `test_creds/crl_data/crls` was a valid and good directory.
This PR moves the bad CRLs to their own directory to prevent this accidental mash-up of test data. It also adds debug logging to our custom verification stack.
Closes#36122
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36122 from gtcooke94:crl_fix 508dd1370d
PiperOrigin-RevId: 616280898
This PR changes how CRLs are handled purely internally. After discussing with davidben@, there are various problems with the `X509_STORE_set_get_crl` API and we shouldn't use it. This change keeps the behavior and external API the same, but instead of bulk pushing CRL information into OpenSSL, we instead iterate through the built chain and check each certificate for revocation, as well as doing the CRL validation ourselves.
Closes#36031
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36031 from gtcooke94:CrlInternalRefactor 5f4c816648
PiperOrigin-RevId: 615139682
This PR adds the Authority Key Identifier to CertificateInfo. This value _can be_ important in finding the right CRLs to use if there are Issuer name overlaps or a more complicated CA setup with multiple signing keys.
We should observe no behavior change in our `CrlProvider` implementations, this is just adding an important field for users who implement it themselves.
Closes#35931
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35931 from gtcooke94:AkidCheck dd048a53b6
PiperOrigin-RevId: 611143198
…p and add some verification helpers (#35641)"
This reverts commit 310770d61d.
It breaks all portability tests.
```
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from CrlUtils
[ RUN ] CrlUtils.HasCrlSignBitExists
/[var/local/git/grpc/test/core/tsi/ssl_transport_security_utils_test.cc:566](https://cs.corp.google.com/piper///depot/google3/var/local/git/grpc/test/core/tsi/ssl_transport_security_utils_test.cc?l=566): Failure
Value of: HasCrlSignBit(root_ca_)
Actual: false
Expected: true
[ FAILED ] CrlUtils.HasCrlSignBitExists (0 ms)
[----------] 1 test from CrlUtils (0 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (0 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] CrlUtils.HasCrlSignBitExists
```
Closes#35962
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35962 from drfloob:revert-310770d61d3b25d358d877a074bebf6ae85549d0 e6a2f03309
PiperOrigin-RevId: 609075479
This PR does 2 distinct things, I can unbundle them if desired
1) Add functions in `ssl_transport_security_utils` and associated tests that will eventually be used for additional Crl validation (the logic of actually doing this will be in a future PR), so other than the tests these fns are currently unused.
2) Remove the use of `X509_NAME_oneline` - it is not a guaranteed stable way to get the issuer name for lookups. Instead, use the DER encoding via `i2d_X509_NAME` - the results in a non-human readable string that is stable for lookup, and necessitated some change to the CrlProvider test code that previously used a human readable string for this value.
Neither should result in behavior changes.
Closes#35641
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35641 from gtcooke94:CrlRefactor 2b6f63717c
PiperOrigin-RevId: 607701254
There are a select few tests that are failing when building with OpenSSL102 - disable them until we can fix.
Closes#35354
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35354 from gtcooke94:fix_ossl_102 8708d6ce86
PiperOrigin-RevId: 595761932
Enable OpenSSL 1.0.2 tests and add a container for 1.1.1 so that it is tested during portability testing as well.
Closes#35236
PiperOrigin-RevId: 590345568
This PR fixes a bug identified in #29667, where the TLS channel
credentials still require a trust bundle even if the user has explicitly
opted to not verify the server certificate. This PR is based on #29810.
As such, `alts_zero_copy_grpc_protector_create` will take a
`GsecKeyFactoryInterface` to create `GsecKeyInterface` objects for the
underlying crypter to use.
This enables the caller to control how all the key related buffers are
prepared and protected.
`gsec_aes_gcm_aead_crypter` holds the raw pointer to `GsecKeyInterface`
instead of a `unique_ptr` possibly because somewhere in the test (and
maybe production code as well), the structure is getting copied. A SEGV
error would be caused with `unique_ptr` which doesn't support copy
operations.
This reverts commit 601aaf80b7, which
results in rolling forward #34408.
The 601aaf reversion happened because of a deadlock found in Python. The
root cause ended up being an issue with the Python wrapper and was fixed
in #34712 , so this can be rolled forward again
The basic APIs for the CRL Reloading features.
This adds external types to represent CRL Providers, CRLs, and
CertificateInfo.
Internally we will use `CrlImpl` - this layer is needed to hide OpenSSL
details from the user.
GRFC - https://github.com/grpc/proposal/pull/382
Things Done
* Add external API for `CrlProvider`, `Crl`, `CertInfo` (`CertInfo` is
used during CRL lookup rather than passing the entire certificate).
* Add code paths in `ssl_transport_security` to utilize CRL providers
* Add `StaticCrlProvider`
* Refactor `crl_ssl_transport_security_test.cc` so it is more extensible
and can be used with providers
This behavior is dangerous because we will crash when the cache is
created, which is not necessarily on application startup and is likely
when you first try to establish an SSL connection. Instead, we log an
error. If the SSL library attempts to put a session ticket in the cache
it will fail to do so, but everything else will continue as normal. In
particular, we will always seamlessly fall back to a full SSL handshake.
Along the way, we also ensure that you cannot put a null `SSL_SESSION`
into the cache, which would lead to a segfault when it is fetched from
the cache.
Revert the reversion of the SSL_CTX_new change (#34355 reverted #34180 )
with a fix.
There was an issue with using `strcpy` on a `new[] string` in the
constructor of `ssl_credentials`. An ASAN test caught this in some CI
down the line - `ERROR: AddressSanitizer: alloc-dealloc-mismatch
(operator new [] vs free)`
That `strcpy` call was changed to `grp_strdup` which duplicates a string
in a way that can be freed by `gpr_free` and should resolve the ASAN
failure.
Move the SSL_CTX to the level of the credentials rather than the
subchannel.
The SSL_CTX should only get created once per credential rather than once
per subchannel.
We should observe no behavior change with this PR, only efficiency
gains.
To fix the following build error with the head of abseil
```
/var/local/git/grpc/test/core/tsi/ssl_transport_security_utils_test.cc:231:42: error: no member named 'StrCat' in namespace 'absl'
return absl::InternalError(absl::StrCat("Client error:", client_err));
~~~~~~^
/var/local/git/grpc/test/core/tsi/ssl_transport_security_utils_test.cc:238:42: error: no member named 'StrCat' in namespace 'absl'
return absl::InternalError(absl::StrCat("Server error:", server_err));
~~~~~~^
```
Update from gtcooke94:
This PR adds support to build gRPC and it's tests with OpenSSL3. There were some
hiccups with tests as the tests with openssl haven't been built or exercised in a
few months, so they needed some work to fix.
Right now I expect all test files to pass except the following:
- h2_ssl_cert_test
- ssl_transport_security_utils_test
I confirmed locally that these tests fail with OpenSSL 1.1.1 as well,
thus we are at least not introducing regressions. Thus, I've added compiler directives around these tests so they only build when using BoringSSL.
---------
Co-authored-by: Gregory Cooke <gregorycooke@google.com>
Co-authored-by: Esun Kim <veblush@google.com>
This PR is expected to fix the flakes of
`//test/core/tsi:ssl_transport_security_test` when built under UBSAN.
Why is this needed? There are several tests in
`ssl_transport_security_test.cc` that involve doing many expensive
operations and PR #33638 recently added one more (namely, repeatedly
signing with an ECDSA key). The slow tests are already altered for MSAN
and TSAN, and now we need to do the same for UBSAN.
There is a bug in the SSL stack that was only partially fixed in #29176:
if more than 17kb is written to the BIO buffer, then everything over
17kb will be discarded, and the SSL handshake will fail with a bad
record mac error or hang if not enough bytes have arrived yet.
It's relatively uncommon to hit this bug, because the TLS handshake
messages need to be much larger than normal for you to have a chance of
hitting this bug. However, there was a separate bug in the SSL stack
(recently fixed in #33558) that causes the ServerHello produced by a
gRPC-C++ TLS server to grow linearly in size with the size of the trust
bundle; these 2 bugs combined to cause a large number of TLS handshake
failures for gRPC-C++ clients talking to gRPC-C++ servers when the
server had a large trust bundle.
This PR fixes the bug by ensuring that all bytes are successfully
written to the BIO buffer. An initial quick fix for this bug was planned
in #33611, but abandoned because we were worried about temporarily
doubling the memory footprint of all SSL channels.
The complexity in this PR is mostly in the test: it is fairly tricky to
force gRPC-C++'s SSL stack to generate a sufficiently large ServerHello
to trigger this bug.
This PR does the following: for the TLS server credentials, stops
calling `SSL_CTX_set_client_CA_list` by default in
`ssl_transport_security.cc`, and gives users a knob to re-enable calling
this API.
## What does the `SSL_CTX_set_client_CA_list` API do?
When this API is called, a gRPC TLS server sends the following data in
the ServerHello: for each certificate in the server's trust bundle, the
CA name in the certificate.
This API does not change the set of certificates trusted by the server
in any way. Rather, it is just providing a hint to the client about what
client certificate should be sent to the server.
## Why are we removing the use of `SSL_CTX_set_client_CA_list` by
default for the TLS server credentials?
Removing the use of this API by default has 2 benefits:
1. Calling this API makes gRPC TLS unusable for servers with a
sufficiently large trust bundle. Indeed, if the server trust bundle is
too large, then the server will always fail to build the ServerHello.
2. Calling this API is introducing a huge amount of overhead (1000s of
bytes) to each ServerHello, so removing this feature will improve
connection establishment latency for all users of the TLS server
credentials.
I'm fixing the ALTS/Envoy transport socket extension (which is currently
broken). Along the way, I'm trying to remove as many uses of gRPC
internals as possible (with the eventual goal of only relying on public
gRPC APIs and the alts_zero_copy_grpc_protector). To this end, I need to
remove the ExecCtx check in the alts_zero_copy_grpc_protector_create
function, so that Envoy can call into this function without needing to
create an ExecCtx.
The logic is straightforward: attempt to read the
`GRPC_ALTS_MAX_CONCURRENT_HANDSHAKES` environment variable and, if it
set to an integer, instantiate the handshake queues based on this
integer.
Based on go/grpc-alts-concurrent-handshake-cap.
This PR is a small code change with a lot of new test data.
[In OpenSSL, there are two flags that configure CRL checks. Coping
relevant
section:](https://www.openssl.org/docs/man1.0.2/man3/X509_VERIFY_PARAM_get_depth.html)
> - X509_V_FLAG_CRL_CHECK enables CRL checking for the certificate chain
leaf certificate. An error occurs if a suitable CRL cannot be found.
> - X509_V_FLAG_CRL_CHECK_ALL enables CRL checking for the entire
certificate chain.
We currently only set `X509_V_FLAG_CRL_CHECK`, so we will only ever
check if the leaf certificate is revoked. We should check the whole
chain. I am open to making this a user configuration if we want to do it
that way, but we certainly need to be able to check the whole chain.
So, this PR contains the small code change in
`ssl_transport_security.cc` to use the `X509_V_FLAG_CRL_CHECK_ALL` flag.
Then the rest of the changes are in tests. I've added all the necessary
files to have a chain built that looks as follows
`Root CA -> Revoked Intermediate CA -> Leaf Certificate`, and added a
test for this case as well.
You can verify that on master this new test will fail (i.e. the
handshake will succeed even though the intermediate CA is revoked) by
checking out this branch, running `git checkout master --
./src/core/tsi/ssl_transport_security.cc`, then running the test.
I also slightly reorganized test/core/tsi/test_creds/ so that the CRLs
are in their own directory, which is the way our API intends to accept
CRLs.
This check only works if all handshake RPCs have an OK status, and it's
racey e.g. if the client is cancelling handshake RPCs (being when an RPC
is cancelled, termination of the RPC at the client is asynchronous from
termination at the server, so the client can resume the queue before the
server RPC completes).
* Add info about ca cert used to verify chain.
The tsi_peer object will now contain the subject of the root/ca cert
that was used to verify the peer's chain during a handshake.
* temp investigation
* Fix issues relating to overlapping CRL callback
* formatting on ssl_transport_security.cc
* Swap ca_cert naming
* Use preverify_ok instead of numbers
* Continue some renaming, addressing pr comments
* Removed early return if peer property setting fails
* Continue renaming
* clang-tidy
* Fix clang problem
* clang fixes
* Add null check in tests
* More PR changes. Behavior change to include root cert extract when TSI_REQUEST_CLIENT_CERTIFICATE_AND_VERIFY
* Add intermediate ca, leaf cert, and test with them
* clang-tidy
* Basic formatting
* Add new keys to build for export
* Add new cert files to test BUILD
* build file style fix
* changes for chain test
* clang-format
* build clean
* Add $ to lines of code in README
* Add directive about X509_STORE_CTX_get0_chain
* formatting
Mark D. Roth
Tanvi Jagtap
Matthew Stevenson
Luwei Ge
AJ Heller
Craig Tiller
Gregory Cooke
ericsalo
Matthew Stevenson
apolcyn
nanahpang
Esun Kim
jrandolf
Gregory Cooke
Yash Tibrewal