PDB files are consuming a significant amount of storage space on our Windows test machines. With each PDB file being around 200MB and every target having its own corresponding file, the disk usage adds up quickly.
Fortunately, we can address this issue without sacrificing debugging functionality. [Shrink my Program Database (PDB) file](https://devblogs.microsoft.com/cppblog/shrink-my-program-database-pdb-file/) outlines several techniques to reduce PDB file size. We can implement these methods to achieve our goal. (From my experiment, the compression option only can reduce a 200MB PDB file by 60%)
This change only picks options which don't reduce debuggability.
Closes#36259
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36259 from veblush:pdbdiet 5781a789b6
PiperOrigin-RevId: 622232176
Earlier, the tests just had bazel support. With CMake support added in #36063, we can also add CI CMake support for the tests. A major benefit of this is that we also get coverage for the various platforms that we test from our portability test suite.
Closes#36087
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36087 from yashykt:OTelCISupport b28fbe02e5
PiperOrigin-RevId: 615543685
Change was created by the release automation script. See go/grpc-release.
Additional Changes:
* Boring SSL started to [Require SSE2 when targetting 32-bit x86](56d3ad9d23), thus added `-msse2` to fix some build failures.
Closes#36089
PiperOrigin-RevId: 614822548
Changes -
* Add CMake build support to `grpcpp_otel_plugin`. Currently, we are only supporting the `find_package CONFIG` method for depending on `opentelemetry-cpp`.
* Since, `grpcpp_otel_plugin` is an extension of gRPC, it will not be built by default. To enable building of this target, a new CMake option `gRPC_BUILD_GRPCPP_OTEL_PLUGIN` is being added.
* Also add `CMakeLists.txt` to the otel example.
The `otel_plugin_test` can also be built through cmake but, for the CI to work, there are some additional changes. Those will be made in an upcoming PR.
Closes#36063
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/36063 from yashykt:OTelCMakeSupport 3bc783823b
PiperOrigin-RevId: 613734473
Fixes#30385 for C++/cmake.
Add a `gRPC_DOWNLOAD_ARCHIVES` switch that is on by default but can be disabled for environments building without Internet connectivity.
Passes build tests:
```
$ python tools/run_tests/run_tests.py -l c++ --build_only
PASSED: tools/run_tests/helper_scripts/build_cxx.sh [time=7116.3sec, retries=0:0SUCCESS: All tests passed
=== run_tests.py DEBUG INFO ===
command: "tools/run_tests/run_tests.py -l c++ --build_only"
===============================
$
```
And tested in a Docker container with `eth0` down to ensure we can now build when `-DgRPC_DOWNLOAD_ARCHIVES="OFF"` is passed to cmake.
Closes#34587
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/34587 from matthewjmiller1:no-download-option 9c07bdce87
PiperOrigin-RevId: 599954208
`GPR_BACKWARDS_COMPATIBILITY_MODE` was devised to support old Linux kernel with old glibc but gRPC is now expected to support glibc 2.17 and later (2.17 is derived from the oldest but supported Linux distro, CentOS 7). Effectively, gRPC doesn't need `GPR_BACKWARDS_COMPATIBILITY_MODE` anymore. Hence, let's remove it.
gRPC Python and Ruby that distribute binary artifacts already began to use `manylinux` environment dealing with this issue in a better way so they don't need this flag anymore.
Closes#35602
PiperOrigin-RevId: 599895270
We've been trying to upgrade Cmake to 3.13 or later as OSS policy bumped it. But we couldn't as Android has a weird linker error with Cmake 3.18 (you can see the error from https://github.com/grpc/grpc/pull/34331) This PR instead upgrades it to use 3.22 for Android test.
Closes#35572
PiperOrigin-RevId: 599317285
<!--
If you know who should review your pull request, please assign it to that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the appropriate
lang label.
-->
Closes#35484
COPYBARA_INTEGRATE_REVIEW=https://github.com/grpc/grpc/pull/35484 from dawidcha:win_shared_gs_api f8f244da99
PiperOrigin-RevId: 598863993
This commit upgrades gRPC to protobuf v25.0 and makes some fixes to
account for upb changes. One major change is that upb has been merged
into the protobuf repo, so we can now drop the separate `@upb`
dependency. Another is that `.upb.c` files no longer exist and there are
new `.upb_minitable.h` and `.upb_minitable.c` files. The longer
filenames exceeded a Windows restriction, so to work around that I
renamed the `upb-generated` directory to just `upb-gen`, and likewise
for `upbdefs-generated`.
Not all POSIX platforms have rt as a separate library. QNX has rt as
part of libc (same as pthreads).
Add condition to check if the library can be found and link with it only
in positive scenario.
A solution for: https://github.com/grpc/grpc/issues/34254
This is a follow up to https://github.com/grpc/grpc/pull/34103
That pull request explicitly aimed to introduce shared library builds
for Windows (DLLs) while effecting zero material change to the existing
build pipelines. That aspiration meant that the grpc++_unsecure library
had to be effectively excluded from the build (because including it
would have also included a dependency on openssl, which makes no sense
given its purpose)
This PR addresses that by:
* Extracting the single function in grpc_tls_certificate_provider with a
dependency on openssl into a separate compilation unit
* Including that new .cc file into the grpc library
* Including grpc_tls_certificate_provider and one other source file into
grpc_unsecure for the Windows DLL build only.
* Reinstating the grpc++_unsecure library which is a prerequisite for
many tests.
* Regenerating all files affected by the changes in Bazel BUILD that
introduce the new source file.
This change does affect the operation of other build pipelines - I have
confirmed that it does not break the Linux Bazel build.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
Towards https://github.com/grpc/grpc/issues/33032,
Reopen after botched force-push in #33175 that then got "merged" and
cannot be reopened anymore.
More context in that PR.
---------
Co-authored-by: Isuru Fernando <isuruf@gmail.com>
Co-authored-by: David Chamberlin <david.chamberlin@ln.email.gs.com>
Some versions of MacOS (as well as some `glibc`-based platforms) require
`__STDC_FORMAT_MACROS` to be defined prior to including `<cinttypes>` or
`<inttypes.h>`, otherwise build fails with undeclared `PRIdMAX`,
`PRIdPTR` etc.
See note here: https://en.cppreference.com/w/cpp/types/integer
label: (release notes: no)
- Extract build metadata for some external dependencies from bazel
build. This is achieved by letting extract_metadata_from_bazel_xml.py
analyze some external libraries and sources. The logic is basically the
same as for internal libraries, I only needed to teach
extract_metadata_from_bazel_xml.py which external libraries it is
allowed to analyze.
* currently, the list of source files is automatically determined for
`z`, `upb`, `re2` and `gtest` dependencies (at least for the case where
we're building in "embedded" mode - e.g. mostly native extensions for
python, php, ruby etc. - cmake has the ability to replace some of these
dependencies by actual cmake dependency.)
- Eliminate the need for manually written gen_build_yaml.py for some
dependencies.
- Make the info on target dependencies in build_autogenerated.yaml more
accurate and complete. Until now, there were some depdendencies that
were allowed to show up in build_autogenerated.yaml and some that were
being skipped. This made generating the CMakeLists.txt and Makefile
quite confusing (since some dependencies are being explicitly mentioned
and some had to be assumed by the build system).
- Overhaul the Makefile
* the Makefile is currently only used internally (e.g. for ruby and PHP
builds)
* until now, the makefile wasn't really using the info about which
targets depend on what libraries, but it was effectively hardcoding the
depedendency data (by magically "knowing" what is the list of all the
stuff that e.g. "grpc" depends on).
* After the overhaul, the Makefile.template now actually looks at the
library dependencies and uses them when generating the makefile. This
gives a more correct and easier to maintain makefile.
* since csharp is no longer on the master branch, remove all mentions of
"csharp" targets in the Makefile.
Other notable changes:
- make extract_metadata_from_bazel_xml.py capable of resolving workspace
bind() rules (so that it knows the real name of the target that is
referred to as e.g. `//external:xyz`)
TODO:
- [DONE] ~~pkgconfig C++ distribtest~~
- [DONE} ~~update third_party/README to reflect changes in how some deps
get updated now.~~
Planned followups:
- cleanup naming of some targets in build metadata and buildsystem
templates: libssl vs boringssl, ares vs cares etc.
- further cleanup of Makefile
- further cleanup of CMakeLists.txt
- remote the need from manually hardcoding extra metadata for targets in
build_autogenerated.yaml. Either add logic that determines the
properties of targets automatically, or use metadata from bazel BUILD.
This PR implements a c-ares based DNS resolver for EventEngine with the
reference from the original
[grpc_ares_wrapper.h](../blob/master/src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.h).
The PosixEventEngine DNSResolver is implemented on top of that. Tests
which use the client channel resolver API
([resolver.h](../blob/master/src/core/lib/resolver/resolver.h#L54)) are
ported, namely the
[resolver_component_test.cc](../blob/master/test/cpp/naming/resolver_component_test.cc)
and the
[cancel_ares_query_test.cc](../blob/master/test/cpp/naming/cancel_ares_query_test.cc).
The WindowsEventEngine DNSResolver will use the same EventEngine's
grpc_ares_wrapper and will be worked on next.
The
[resolve_address_test.cc](https://github.com/grpc/grpc/blob/master/test/core/iomgr/resolve_address_test.cc)
which uses the iomgr
[DNSResolver](../blob/master/src/core/lib/iomgr/resolve_address.h#L44)
API has been ported to EventEngine's dns_test.cc. That leaves only 2
tests which use iomgr's API, notably the
[dns_resolver_cooldown_test.cc](../blob/master/test/core/client_channel/resolvers/dns_resolver_cooldown_test.cc)
and the
[goaway_server_test.cc](../blob/master/test/core/end2end/goaway_server_test.cc)
which probably need to be restructured to use EventEngine DNSResolver
(for one thing they override the original grpc_ares_wrapper's free
functions). I will try to tackle these in the next step.
<!--
If you know who should review your pull request, please assign it to
that
person, otherwise the pull request would get assigned randomly.
If your pull request is for a specific language, please add the
appropriate
lang label.
-->
* Revert "Revert "Add support for systemd socket activation (#30485)" (#31617)"
This reverts commit 867dc6cae2.
* Add checks to unix tests
* Ran generate_projects.sh and fixed styling in test
* Fix variable in unit test
* Use reinterpret_cast in test
* Rebase and fix sanity failures
* cmake: add separate export for plugin targets
Add a separate CMake export for the plugin targets to separate binaries
and libraries into their own CMake target export files. Skip the cross
compiled binary plugin targets during cross compile because they are not
usable and not always available.
The Yocto build system doesn't install cross compiled binaries into the
target sysroot. This makes the CMake gRPC config useless as it checks
the existent of binaries and fails without the binaries.
This is a strip down version of #22498 and related to #26148 and #26857.
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Clément Péron <peron.clem@gmail.com>
* regenerate cmake file
Signed-off-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Signed-off-by: Clément Péron <peron.clem@gmail.com>
Co-authored-by: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
* Use gRPC_INSTALL_LIBDIR for pkgconfig files
Fixes#25635.
If grpc libraries are installed in <prefix>/lib64, then .pc files should
be installed in <prefix>/lib64/pkgconfig. Before this commit, they were
always installed in <prefix>/lib/pkgconfig.
* Re-generate projects
* [xDS Proto] Enhence gRPC buildgen for 3rd party proto compilation
* Rebase from master to update envoy-api version
* Address reviewer's comments
* Address reviewer's comment
* Regenerate project
* Rename external_library
* Address reviewer's comments
* Add comments for the internals of generate C++ proto code
* Add proto file as a dependency to the custom command
* add cmake support for ccache
* cleanup: use --env-file for docker run invocations
* make python build compatible with using ccache
* enable building using ccache in selected kokoro jobs
* print ccache stats and the end of run_tests.py
* Add bundle destination for cmake install commands
Fixes the problem:
"no BUNDLE DESTINATION for MACOSX_BUNDLE" when building for iOS
* Automated change: Fix sanity tests
Co-authored-by: anton-danielsson <anton-danielsson@users.noreply.github.com>
Fix regression introduced by previous commit. Original problem that
commit was trying to fix was solved by modification of grpc recipe in
meta-openembedded project repository
Fixes#26857
Signed-off-by: Vyacheslav Yurkov <uvv.mail@gmail.com>
* cmake: fix cross-compilation with gRPC_BUILD_GRPC_CPP_PLUGIN=OFF
When cross-compiling gRPC, a _native_ version of 'grpc_cpp_plugin' is
searched in the environment. For most use cases, a _cross_ version of this
file is not needed and gRPC_BUILD_GRPC_CPP_PLUGIN can be set to OFF.
However, when cross-building with -DgRPC_BUILD_GRPC_CPP_PLUGIN=OFF, there
are some build errors, for example:
make[3]: *** No rule to make target 'grpc_cpp_plugin', needed by 'gens/src/proto/grpc/reflection/v1alpha/reflection.grpc.pb.cc'. Stop.
This is because there is still a hard dependency on 'grpc_cpp_plugin' for
these targets, not taking into account the cross-compilation case.
Fix by depending on the variable gRPC_CPP_PLUGIN, which is set correctly for
either cross or native case.
* regenerate projects
Esun Kim
Yash Tibrewal
Xuan Wang
Matt Miller
David Chamberlin
Adam Cozzette
Roman-Byshliaha-Bose
David Chamberlin
h-vetinari
Sergey Fedorov
Jan Tattermusch
Yijie Ma
Andres Beltran
Clément Péron
Valentin
Craig Tiller
Michael Nosthoff
Ben Beasley
hongnod
Jérôme Duval
Lidi Zheng
Jan Tattermusch
Anton Danielsson
hexchain
Vyacheslav Yurkov
Pablo Busse
Thomas De Schampheleire