This is a big rewrite of global config.
It does a few things, all somewhat intertwined:
1. centralize the list of configuration we have to a yaml file that can
be parsed, and code generated from it
2. add an initialization and a reset stage so that config vars can be
centrally accessed very quickly without the need for caching them
3. makes the syntax more C++ like (less macros!)
4. (optionally) adds absl flags to the OSS build
This first round of changes is intended to keep the system where it is
without major changes. We pick up absl flags to match internal code and
remove one point of deviation - but importantly continue to read from
the environment variables. In doing so we don't force absl flags on our
customers - it's possible to configure grpc without the flags - but
instead allow users that do use absl flags to configure grpc using that
mechanism. Importantly this lets internal customers configure grpc the
same everywhere.
Future changes along this path will be two-fold:
1. Move documentation generation into the code generation step, so that
within the source of truth yaml file we can find all documentation and
data about a configuration knob - eliminating the chance of forgetting
to document something in all the right places.
2. Provide fuzzing over configurations. Currently most config variables
get stashed in static constants across the codebase. To fuzz over these
we'd need a way to reset those cached values between fuzzing rounds,
something that is terrifically difficult right now, but with these
changes should simply be a reset on `ConfigVars`.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
---------
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Alongside https://github.com/grpc/grpc/pull/32496, this makes this test
behave the same on all platforms.
FWIW, I verified this causes us to see the previous lock cycle problem
in https://github.com/grpc/grpc/pull/32491 on linux - originally that
lock cycle was only on mac, because of environmental differences between
mac and linux in CI.
* Revert "Revert "xds_override_host LB: add support for draining state (#31985)" (#32189)"
This reverts commit 5222eafdc1.
* fixup: Add missing dependency to grpc_xds_client targetZ
* WRR: port StaticStrideScheduler to OSS
* WIP
* Automated change: Fix sanity tests
* fix build
* remove unused aliases
* fix another type mismatch
* remove unnecessary include
* move benchmarks to their own file, and don't run it on windows
* Automated change: Fix sanity tests
* add OOB reporting
* generate_projects
* clang-format
* add config parser test
* clang-tidy and minimize lock contention
* add config defaults
* add oob_reporting_period config field and add basic test
* Automated change: Fix sanity tests
* fix test
* change test to use basic RR
* WIP: started exposing peer address to LB policy API
* first WRR test passing!
* small cleanup
* port RR fix to WRR
* test helper refactoring
* more test helper refactoring
* WIP: trying to fix test to have the right weights
* more WIP -- need to make pickers DualRefCounted
* fix timer ref handling and get tests working
* clang-format
* iwyu and generate_projects
* fix build
* add test for OOB reporting
* keep only READY subchannels in the picker
* add file missed in a previous commit
* fix sanity
* iwyu
* add weight expiration period
* add tests for weight update period and OOB reporting period
* Automated change: Fix sanity tests
* lower bound for timer interval
* consistently apply grpc_test_slowdown_factor()
* cache time in test
* add blackout_period tests
* avoid some unnecessary copies
* clang-format
* add field to config test
* simplify orca watcher tracking
* attempt to fix build
* iwyu
* generate_projects
* add "_experimental" suffix to policy name
* WRR: update tests to cover qps plumbing
* WIP
* more WIP
* basic WRR e2e test working
* add OOB test
* fix sanity
* ignore duplicate addresses
* Automated change: Fix sanity tests
* add new tracer to doc/environment_variables.md
* retain scheduler state across pickers
* Automated change: Fix sanity tests
* use separate mutexes for scheduler and timer
* sort addresses to avoid index churn
* remove fetch_sub for wrap around in RR case
Co-authored-by: markdroth <markdroth@users.noreply.github.com>
* WRR: port StaticStrideScheduler to OSS
* Automated change: Fix sanity tests
* fix build
* remove unused aliases
* fix another type mismatch
* remove unnecessary include
* move benchmarks to their own file, and don't run it on windows
* Automated change: Fix sanity tests
Co-authored-by: markdroth <markdroth@users.noreply.github.com>
* LB policy test framework: add helper methods and improve debuggability
* add ExpectPickFail() method
* fix build
* STL magic
* clang-tidy
* more clang-tidy
* add WaitForConnectionFailed() and log unexpected pick results
* add check for proper status for the connectivity state
* Declarative JSON parser
* Automated change: Fix sanity tests
* fix
* shrinking stuff a little
* static vtables
* separate fns
* simpler?
* make maps work
* windows fixes
* Automated change: Fix sanity tests
* simplify code
* Automated change: Fix sanity tests
* vtable-test
* dont always create vec/map impls for every type
* comments
* make error consistent
* move method private
* progress
* durations!
* Automated change: Fix sanity tests
* fix
* fix
* fix
* Automated change: Fix sanity tests
* post-load
* Automated change: Fix sanity tests
* document JsonPostLoad() and add static_assert
* don't copy field names, to avoid length limitations
* use absl::Status
* accept either string or number for numeric values
* add test for direct data member of another struct type
* remove unused method
* add support for retaining part of the JSON wirthout processing
* update test for changes in Json::Parse() API
* add absl::optional support
* Automated change: Fix sanity tests
* fix tests, improve error messages, and add overload to parse to existing object
* remove overload of LoadFromJson()
* change special case for Json to instead use Json::Object
* rename resolver_result_parsing to client_channel_service_config
* split up service_config_test into a separate test for each component
* convert client channel service config parser to new API
* fix build
* converted retry global params
* convert retry method-level parsing, but still need to find a way to control parsing via a channel arg
* improve error structure, add missing types, and improve tests
* clang-format
* Automated change: Fix sanity tests
* fix build
* add LoadJsonObjectField(), add LoadFromJson() overload that takes an ErrorList parameter, and add tests for parsing bare top-level types
* fix msan
* Automated change: Fix sanity tests
* fix error message
* Automated change: Fix sanity tests
* fix test
* add ability to disable fields
* plumb in channel args to disable parsing
* Automated change: Fix sanity tests
* use const char* instead of absl::string_view for enable_key
* fix resolver_component_test
* Automated change: Fix sanity tests
* work around mac build problem
* Automated change: Fix sanity tests
* work around gcc6 problem
* Automated change: Fix sanity tests
* fix build
* fix build
* don't use alternative builder in tests
* convert message size service config parser
* convert fault injection service config parser
* rename files
* add specialization for unique_ptr
* avoid moves in client channel service config parser
* avoid moves in retry service config parser, and do some cleanup
* avoid moves in message_size service config parser
* avoid moves for fault injection service config parser, and use internal channel arg
* convert rbac service config parser
* clang-format
* WIP
* convert top-level service config parser
* clang-format
* fix build
* fix rbac service config parser test
* fix signed-ness problem and reversed-conditional bug
* fix unused param
* fix json_string method
* fix max message length defaults
* add copy ctors to appease windows compiler
* fix RLS LB config parser test
* fix name resolution test
* fix build
* work around gmock portability bug
* fix sanity
* add missing build dep
* make RBAC principal and permissions movable, not copyable
* Revert "make RBAC principal and permissions movable, not copyable"
This reverts commit 53315bccc9.
* attempt to simplify HeaderMatcher and StringMatcher parsing
* more bloat reduction in RBAC service config parser
* fix sanity
* add missing build dep
* attempt work-around for MSVC bug
* Revert "attempt work-around for MSVC bug"
This reverts commit e54c89e1e4.
* attempt work-around for Windows build problem
* try another work-around
* fix sanity
* appease clang-tidy
* generate_projects
* attempt to fix Windows build
* more windows fixes
* more windows fix
* yet more windows fixes
* try without noexcept
* remove unnecessary boilerplate
* code review changes
* fix breakage
Co-authored-by: Craig Tiller <craig.tiller@gmail.com>
Co-authored-by: ctiller <ctiller@users.noreply.github.com>
Co-authored-by: Craig Tiller <ctiller@google.com>
Co-authored-by: markdroth <markdroth@users.noreply.github.com>
* Precondition ChannelArgs with EventEngines
If an EventEngine is not explicitly provided to ChannelArgs, the default
EventEngine will be set when ChannelArgs are preconditioned.
* channel_idle_filter: EE from channel_args
* grpclb: EE from channel_args
* weighted_target: ee from channel_args
* sanitize
* xds cluster manager
* posix native resolver: own an EE ref from iomgr initialization
* reviewer feedback
* reviewer feedback
* iwyu
* iwyu
* change ownership and remove unneeded methods
* clang_format and use consistent engine naming
* store EE ref in channel_stack and use it in channel idle filter
* don't store a separate shared_ptr in NativeDNSResolver
* add GetEventEngine() method to LB policy helper interface
* stop holding refs to the EE instance in LB policies
* clang-format
* change channel stack to get EE instance from channel args
* update XdsWrrLocalityLb
* fix lb_policy_test
* precondition channel_args in ServerBuilder and microbenchmark fixtures
* add required engine to channel_stack test
* sanitize
* dep fix
* add EE to filter fuzzer
* precondition BM_IsolatedFilter channelargs
* fix
* remove unused using statement
* iwyu again??
* remove preconditioning from C++ surface API
* fix bm_call_create
* Automated change: Fix sanity tests
* iwyu
* rm this->
* rm unused deps
* add internal EE arg macro
* precondition filter_fuzzer
* Automated change: Fix sanity tests
* iwyu
* ChannelStackBuilder requires preconditioned ChannelArgs
* iwyu
* iwyu again?
* rm build.SetChannelArgs; rm unused declaration
* fix nullptr string creation
Co-authored-by: Mark D. Roth <roth@google.com>
Co-authored-by: drfloob <drfloob@users.noreply.github.com>
* XdsBootstrap: move two more methods out of the interface
* Automated change: Fix sanity tests
* XdsClient: add unit test
* Automated change: Fix sanity tests
* fix memory leaks
* add helper method
* add unsubscription
* add test for multiple subscriptions
* clang-format
* fix build
* fix flakiness
* add checking for other node fields
* add v2 test
* add response builder
* add test for update from server
* add test for update containing only changed resources
* clang-format
* fix build
* add test for resource not existing upon subscription
* add test for stream closed by server
* add test for multiple watchers for the same resource
* add test for connection failure
* clang-format
* add test for resources wrapped in Resource wrapper message
* add test for resource validation failure
* add test for multiple invalid resources, and fix a case in XdsClient
* add test for validation failure for already-cached resource
* add test for server not resending resources after stream disconnect
* clang-format
* fix XdsClient to report channel errors to newly started watchers
* fix XdsClient to send cached errors/does-not-exists to newly started watchers
* fix watcher to ensure events arrive in the expected order
* fix tests
* clang-format
* add test for multiple resource types
* fix xds_cluster_e2e_test
* Automated change: Fix sanity tests
* cleanup
* add federation tests
* clang-format
* remove now-unnecessary XdsCertificateProviderPluginMapInterface
* code review comments
* simplify XdsResourceType::Decode() API
* XdsClient: add unit tests for XdsClusterResourceType
* add XdsClient with gRPC bootstrap config
* add LB policy tests
* started adding CertificateProvider tests
* update for recent API changes
* fix merge bugs
* xDS resource validation: identify extensions by type_url instead of name
* fix build
* migrate to ValidationErrors
* add xds_common_types_test
* finish TLS tests and add LRS tests
* move ScopedExperimentalEnvVar to its own library and remove redundant e2e tests
* add circuit breaking and outlier detection tests
* add validation to outlier detection LB policy parsing
* clang-format
* Automated change: Fix sanity tests
* fix signedness
* fix sanity
* xDS: implement xds_wrr_locality LB policy and return xDS LB config from XdsClient
* fix unused parameter
* fix sanity
* fix test
* Automated change: Fix sanity tests
* fix aggregate cluster bug
* Automated change: Fix sanity tests
* absl::make_unique -> std::make_unique
* fix sanity
* fix sanity
* iwyu
* iwyu
* update code for XdsResourceTypeImpl changes
Co-authored-by: markdroth <markdroth@users.noreply.github.com>
Jan Tattermusch
Craig Tiller
Yash Tibrewal
apolcyn
Mark D. Roth
Esun Kim
Eugene Ostroukhov
AJ Heller
Yijie Ma
Cheng-Yu Chung