This pulls in updates to system_python, which allows for better handling of our non-hermetic setup. Specifically, our python tests are marked incompatible on systems without a python installation. This will cause them to get skipped by wildcard expansions (which is why we add 1 explicit target to the kokoro builds).
PiperOrigin-RevId: 492286518
When homebrew automatically updates packages we have less control over what version of packages we are running, this will make the script more deterministic. In addition, it will fix errors relating to updating irrelevant packages.
* Add remote cache to linux builds
* Remove BES flags
* Remove BES flags from the right file
* Migrate all Bazel kokoro builds to use remote caching
* Remove BES logic
* Fix mac ruby tests
* Give mac/windows builds GCP access
* Adding quotes to prevent issues with common flags
* Adding command echoing in windows builds
* Try enabling command echoing again
* Adding invocation id for windows bazel build
* Third try
* Adding credentials to windows build
* Attempt to dedupe Bazel test
* Intentionally break a test to make sure presubmits still run
* Moving some shared configs to higher directories
* Adding root common.cfg
* Adding action cfgs at root directory
* Adding empty presubmit/continuous
* Going back to leaf directories
* Adding empty presubmit/continuous
* Unify continuous and presubmit configs
* Consolidate all presubmit/continuous into a common config file
* Revert "Intentionally break a test to make sure presubmits still run"
This reverts commit 978eb19f80ea46f4a0e07789f86e791c5f9790ac.
* Fixing POC config files
* Manually run brew cleanup and initialize php
* Add debug loggin for visibility
* Allow underscores in php folder name
* Add logging to php tests
* Add tracing
* Only use the latest installation of php
* Disabling broken mac php tests
* Fix ruby permissions after Monterey upgrade
* Install m4 via Homebrew
* Adding ruby/python fixes to presubmits
* Adding homebrew prefix command
* More fixes for objc, python, and benchmark build
* Properly disabling C++ benchmark warning
* Use python 2 for testing
* Splitting venv and python 2
* Setup tox-pyenv
* Silencing more warnings
* Cleanup
* Upgrade python tests to 3.7 on mac
* Switch to python 2 by default (googletest requires it)
* Remove venv for python tests, use python 3.7
* Disable all compiler warnings for benchmark builds
* Enable benchmark LTO to silence warnings
* Fix locale issues in cocoapods
* Remove benchmark build from C++ tests
* Removing deprecated use_bazel command
Removing python 3.6 mac build, since Monterey doesn't support it
* Bazelfying conformance tests
Adding infrastructure to "Bazelify" languages other than Java and C++
* Delete benchmarks for languages supported by other repositories
* Bazelfying benchmark tests
* Bazelfying python
Use upb's system python rule instead of branching tensorflow
* Bazelfying Ruby
* Bazelfying C#
* Bazelfying Objective-c
* Bazelfying Kokoro mac builds
* Bazelfying Kokoro linux builds
* Deleting all deprecated files from autotools cleanup
This boils down to Makefile.am and tests.sh and all of their remaining references
* Cleanup after PR reorganizing
- Enable 32 bit tests
- Move conformance tests back
- Use select statements to select alternate runtimes
- Add internal prefixes to proto library macros
* Updating READMEs to use bazel instead of autotools.
* Bazelfying Kokoro release builds
* First round of review fixes
* Second round of review fixes
* Third round of review fixes
* Filtering out conformance tests from Bazel on Windows (b/241484899)
* Add version metadata that was previously scraped from configure.ac
* fixing typo from previous fix
* Adding ruby version tests
* Bumping pinned upb version, and adding tests to python CI
These tests verify that various things can be built/installed/tested
using our published distribution archives, but starting with 22.0 we
will no longer be publishing those.
* Migrating macos php builds from 7.0/7.3 to 7.4/8.0
* Removing local comments
* Fix kokoro issues
* Further cleanup
* Fix permissions in Ruby release test
* Allow pre-compiled binaries for ruby 3.1.0 (#9566)
* Allow pre-compiled binaries for ruby 3.1.1
* add comment
* fix build and use ruby 3.1.0
* add ruby31 to build CI for tests and release
* trying to fix ci
* install ruby 3.1.0 in ruby_build_environment.sh
* use head for rvm to install 3.1.0
* just install master version of rvm in prepare_build_macos_rc
* force install of master rvm in ruby_build_environment.sh
* Use coroutine=universal when compiling ruby31
* use ucontext
* fix filename
* fix coroutine name
* use git head for rake-compiler-dock
* use newest rake-compiler-dock version
* Updated CHANGES.txt for Ruby changes.
* Fixed Ruby 3.1 tests by marking intersect? as unimplemented. (#9645)
* Fixed Ruby 3.1 tests by marking intersect? as unimplemented.
* Updated compatibility tests.
Co-authored-by: Marco Concetto Rudilosso <marcoconcettorudilosso@gmail.com>
* Allow pre-compiled binaries for ruby 3.1.1
* add comment
* fix build and use ruby 3.1.0
* add ruby31 to build CI for tests and release
* trying to fix ci
* install ruby 3.1.0 in ruby_build_environment.sh
* use head for rvm to install 3.1.0
* just install master version of rvm in prepare_build_macos_rc
* force install of master rvm in ruby_build_environment.sh
* Use coroutine=universal when compiling ruby31
* use ucontext
* fix filename
* fix coroutine name
* use git head for rake-compiler-dock
* use newest rake-compiler-dock version
* Dropped Ruby 2.4 support for CI and releases.
* Removed Kokoro configs for old Ruby versions.
* Update Dockerfile to allow it to be rebuilt
Co-authored-by: Joshua Haberman <jhaberman@gmail.com>
Our Mac test runs recently started failing to download RVM. The issue
appears to be a combination of an SSL certificate expiring and old
OpenSSL versions having a bug preventing them from validating the
replacement certificate: https://github.com/rvm/rvm/issues/5133
This commit works around the problem by downloading RVM from GitHub as
suggested in one of the comments on the issue above.
Installing and upgrading Homebrew packages is taking quite a lot of time
(around 20-30 minutes) for each run. This commit removes all Homebrew
usage from the test runs. Homebrew may have been necessary at some point
in the past, but now it appears that everything works without it. The
preinstalled build tools seem to be sufficient for building protoc, and
Ruby is something we already get from rvm.
* WIP.
* Added build config for all of the tests.
* Use ../src/protoc if it is available, for cases where Bazel isn't available.
* Added test_php.sh.
* Fix for the broken macOS tests.
* Move all jobs to use php80 instead of lots of separate jobs.
* Only pass -t flag if we are running in a terminal.
* Updated php_all job to use new Docker stuff.
* Add the kotlin and kotlin-lite directories to the main pom.xml
* Updated Kotlin versions to 3.17.0-rc1
* Fix Mac CI tests by resolving brew package conflict
* WIP.
* WIP.
* WIP.
* WIP.
* WIP.
* WIP.
* Added some missing files.
* WIP.
* WIP.
* Updated upb.
* Extension loads, but crashes immediately.
* Gets through the test suite without SEGV!
Still a lot of bugs to fix, but it is a major step!
214 tests, 378 assertions, 37 failures, 147 errors, 0 pendings, 0 omissions, 0 notifications
14.0187% passed
* Test and build for Ruby 3.0
* Fixed a few more bugs, efficient #inspect is almost done.
214 tests, 134243 assertions, 30 failures, 144 errors, 0 pendings, 0 omissions, 0 notifications
18.6916% passed
* Fixed message hash initialization and encode depth checking.
214 tests, 124651 assertions, 53 failures, 70 errors, 0 pendings, 0 omissions, 0 notifications
42.5234% passed
* A bunch of fixes to failing tests, now 70% passing.
214 tests, 202091 assertions, 41 failures, 23 errors, 0 pendings, 0 omissions, 0 notifications
70.0935% passed
* More than 80% of tests are passing now.
214 tests, 322331 assertions, 30 failures, 9 errors, 0 pendings, 0 omissions, 0 notifications
81.7757% passed
Unfortunately there is also a sporadic bug/segfault hanging around
that appears to be GC-related.
* Add linux/ruby30 and macos/ruby30
* Use rvm master for 3.0.0-preview2
* Over 90% of tests are passing!
214 tests, 349898 assertions, 15 failures, 1 errors, 0 pendings, 0 omissions, 0 notifications
92.5234% passed
* Passes all tests!
214 tests, 369388 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed
* A bunch of cleanup.
1. Removed a bunch of internal-only symbols from headers.
2. Required a frozen check to get a non-const pointer to a map or array.
3. De-duplicated the code to get a type argument for Map/RepeatedField.
* Removed a bunch more stuff from protobuf.h. There is an intermittent assert failure.
Intermittent failure:
ruby: ../../../../ext/google/protobuf_c/protobuf.c:263: ObjectCache_Add: Assertion `rb_funcall(obj_cache2, (__builtin_constant_p("[]") ? __extension__ ({ static ID rb_intern_id_cache; if (!rb_intern_id_cache) rb_intern_id_cache = rb_intern2((("[]")
), (long)strlen(("[]"))); (ID) rb_intern_id_cache; }) : rb_intern("[]")), 1, key_rb) == val' failed
* Removed a few more things from protobuf.h.
* Ruby 3.0.0-preview2 to 3.0.0
* Require rake-compiler-dock >= 1.1.0
* More progress, fighting with the object cache.
* Passes on all Ruby versions!
* Updated and clarified comment regarding WeakMap.
* Fixed the wyhash compile.
* Fixed conformance tests for Ruby.
Conformance results now look like:
RUBYLIB=../ruby/lib:. ./conformance-test-runner --enforce_recommended --failure_list failure_list_ruby.txt --text_format_failure_list text_format_failure_list_ruby.txt ./conformance_ruby.rb
CONFORMANCE TEST BEGIN ====================================
CONFORMANCE SUITE PASSED: 1955 successes, 0 skipped, 58 expected failures, 0 unexpected failures.
CONFORMANCE TEST BEGIN ====================================
CONFORMANCE SUITE PASSED: 0 successes, 111 skipped, 8 expected failures, 0 unexpected failures.
Fixes include:
- Changed Ruby compiler to no longer reject proto2 maps.
- Changed Ruby compiler to emit a warning when proto2 extensions are
present instead of rejecting the .proto file completely.
- Fixed conformance tests to allow proto2 and look up message by name
instead of hardcoding a specific list of messages.
- Fixed conformance test to support the "ignore unknown" option for
JSON.
- Fixed conformance test to properly report serialization errors.
* Removed debug printf and fixed #inspect for floats.
* Fixed compatibility test to have proper semantics for #to_json.
* Updated Makefile.am with new file list.
* Don't try to copy wyhash when inside Docker.
* Fixed bug where we would forget that a sub-object is frozen in Ruby >=2.7.
* Avoid exporting unneeded symbols and refactored a bit of code.
* Some more refactoring.
* Simplified and added more comments.
* Some more comments and simplification. Added a missing license block.
Co-authored-by: Masaki Hara <hara@wantedly.com>
It seems that updating shallow Git clones is expensive, and as a result
Homebrew recently started refusing to update shallow clones (see
https://github.com/Homebrew/discussions/discussions/226). This commit
tries to fix the problem by making these repos into full clones before
running "brew update".
I also came across another error about there being a conflicting version
of pip3 in /usr/local. I suspect that is related to the other Python
binaries that the script has to delete, so I added pip3 to the list and
that seemed to solve the problem.
This commit cleans up our environment setup for macOS builds.
- Always use `python -m {pip | virtualenv}` form of commands.
- Only upgrade in-place, and never use `--ignore-installed`.
- Use latest tox on macos, and test 3.6 on macos presubmit.
These changes ensure that multibuilder won't break the system-installed Python on the Kokoro macOS builder, and thus won't resolve the ancient system version. For Tox, make sure we test with Python 3.6 (the version of Python in current macOS).
- Remove the `brew install`; the kokoro image already has it, just
update/upgrade instead.
- Remove `prune`, logs had:
"""
Error: Unknown command: prune
"""
- Remove `uninstall`, logs had:
"""
Error: Refusing to uninstall /usr/local/Cellar/cmake/3.16.4 and /usr/local/Cellar/icu4c/64.2
because they are required by ceres-solver, ffmpeg, harfbuzz, libass and opencv, which are currently installed.
You can override this and force removal with:
brew uninstall --ignore-dependencies node icu4c cmake wget
"""
- Skip installing some things since they are already in the base image, logs
had:
"""
Warning: gflags 2.2.2 is already installed and up-to-date
To reinstall 2.2.2, run `brew reinstall gflags`
Warning: openssl@1.1 1.1.1g is already installed and up-to-date
To reinstall 1.1.1g, run `brew reinstall openssl@1.1`
Warning: pcre 8.44 is already installed and up-to-date
To reinstall 8.44, run `brew reinstall pcre`
"""
- Don't install gpg gpg2 as gnupg is already installed, also use gpg instead of
gpg2 for commands (and update the commands), logs had:
"""
kokoro/macos/prepare_build_macos_rc: line 44: gpg2: command not found
kokoro/macos/prepare_build_macos_rc: line 45: gpg2: command not found
"""
- Add env guards to control all the option installs and only request them be
installed in the cases that need it. This avoids having to install/update
the things like ruby when some other tool only needed in some configs is
install differently and could have conflicts.
- Switch to brew for cocoapods to avoid compat issues on the supporting
libraries.
Currently all of our MacOS tests are failing with the error: "autom4te: need GNU m4 1.4 or later: /usr/bin/m4". This is likely to be happening because Kokoro has updated their Xcode version to 11.3.
Currently all of our MacOS tests are failing with the error: "autom4te: need GNU m4 1.4 or later: /usr/bin/m4". This is likely to be happening because Kokoro has updated their Xcode version to 11.3.
Mike Kruskal
Mike Kruskal
Eric Salo
Deanna Garcia
Protobuf Team Bot
deannagarcia
Adam Cozzette
Sandy Zhang
theodorerose
David L. Jones
Thomas Van Lenten
Joshua Haberman
Marco Concetto Rudilosso
Paul Yang
listonjesse
Masaki Hara
Rafi Kamal