Protobuf uses utf8_range library for utf8 string validation.
Currently, only SSE implementation is integrated.
This patch adapts utf8_range Neon implementation to protobuf.
Closes#18126
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/18126 from cyb70289:utf8-neon 5edbcc2692
PiperOrigin-RevId: 680711032
This PR implements lookup of service descriptor and method descriptor objects in Ruby as described in issue https://github.com/protocolbuffers/protobuf/issues/14891.
It contains three implementations - one for the CRuby extension API, one for JRuby, and one for FFI.
With this patch,
* `DescriptorPool#lookup('fully.qualified.service.name')` works and returns a `Google::Protobuf::ServiceDescriptor` object
* You can call `#options` on that to get the service options
* You can call `#methods` on that to get the services' methods as `Google::Protobuf::MethodDescriptor` objects,
* You can call `MethodDescriptor#options` to get method options
* You can also get the streaming flags & input/output types of the method with `#input_type`, `#output_type`, `#client_streaming`, and `#server_streaming`.
In order to make the FFI implementation work, I had to mark some more methods in the UPB header as exported - I guess that's something which will have to be done on the UPB side, like this 01fed1cc1b
CC @dazuma & @haberman from the original issue, and @JasonLunn (since you work on protobuf it seems - small world!)
I apologies for the large volume of copy-pasta'd code from the existing descriptor class implementations into the new ones - I felt this was probably better than designing new abstractions to reduce it off the bat though; this feels like it "fits in" with the existing implementation.
Closes#15817
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/15817 from KJTsanaktsidis:ktsanaktsidis/add_service_method_descriptors 54d7218431
PiperOrigin-RevId: 618221016
Three of these runtimes are based on upb, and the fourth is based on the Java runtime. Both of these already have editions support, so this was mostly just a matter of:
- Advertising support to allow editions codegen
- Stripping features from the runtime options
- Hooking up conformance tests
- Adding some lightweight editions tests
There are also a few minor orthogonal fixes included here:
- Ruby's upb hack for treating all enums as open enums needed tweaking
- The `enable_editions` flag is no longer needed in our internal proto rules
PiperOrigin-RevId: 616256211
We have a rake based extension `ext/google/protobuf_c/Rakefile` (added since 3.25.0), which depends on `rake`. However, when gem is installed by `bundle` command, `rake` is not guaranteed to be activated by `bundle` (e.g. when `BUNDLE_PATH` is not default and rake is not already installed in `BUNDLE_PATH`). Therefore, we have to explicit declare a runtime dependency on `rake` for the extension, or otherwise the installation will fail with `can't find gem rake (>= 0.a) with executable rake (Gem::GemNotFoundException)`.
Here is a reproduction:
```
root@4d9cb6ba41b1:/build# cat Gemfile
source 'https://rubygems.org'
gem 'google-protobuf'
root@4d9cb6ba41b1:/build# bundle config --local path vendor/bundle
root@4d9cb6ba41b1:/build# bundle
Fetching gem metadata from https://rubygems.org/...........
Resolving dependencies...
Installing google-protobuf 3.25.1 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
current directory: /build/vendor/bundle/ruby/3.3.0/gems/google-protobuf-3.25.1/ext/google/protobuf_c
rake RUBYARCHDIR\=/build/vendor/bundle/ruby/3.3.0/extensions/aarch64-linux/3.3.0/google-protobuf-3.25.1 RUBYLIBDIR\=/build/vendor/bundle/ruby/3.3.0/extensions/aarch64-linux/3.3.0/google-protobuf-3.25.1
/usr/local/lib/ruby/3.3.0/rubygems.rb:259:in `find_spec_for_exe': can't find gem rake (>= 0.a) with executable rake (Gem::GemNotFoundException)
from /usr/local/lib/ruby/3.3.0/rubygems.rb:278:in `activate_bin_path'
from /usr/local/bin/rake:25:in `<main>'
rake failed, exit code 1
Gem files will remain installed in /build/vendor/bundle/ruby/3.3.0/gems/google-protobuf-3.25.1 for inspection.
Results logged to /build/vendor/bundle/ruby/3.3.0/extensions/aarch64-linux/3.3.0/google-protobuf-3.25.1/gem_make.out
/usr/local/lib/ruby/3.3.0/rubygems/ext/builder.rb:125:in `run'
/usr/local/lib/ruby/3.3.0/rubygems/ext/rake_builder.rb:30:in `build'
/usr/local/lib/ruby/3.3.0/rubygems/ext/builder.rb:193:in `build_extension'
/usr/local/lib/ruby/3.3.0/rubygems/ext/builder.rb:227:in `block in build_extensions'
/usr/local/lib/ruby/3.3.0/rubygems/ext/builder.rb:224:in `each'
/usr/local/lib/ruby/3.3.0/rubygems/ext/builder.rb:224:in `build_extensions'
/usr/local/lib/ruby/3.3.0/rubygems/installer.rb:852:in `build_extensions'
/usr/local/lib/ruby/3.3.0/bundler/rubygems_gem_installer.rb:76:in `build_extensions'
/usr/local/lib/ruby/3.3.0/bundler/rubygems_gem_installer.rb:28:in `install'
/usr/local/lib/ruby/3.3.0/bundler/source/rubygems.rb:205:in `install'
/usr/local/lib/ruby/3.3.0/bundler/installer/gem_installer.rb:54:in `install'
/usr/local/lib/ruby/3.3.0/bundler/installer/gem_installer.rb:16:in `install_from_spec'
/usr/local/lib/ruby/3.3.0/bundler/installer/parallel_installer.rb:132:in `do_install'
/usr/local/lib/ruby/3.3.0/bundler/installer/parallel_installer.rb:123:in `block in worker_pool'
/usr/local/lib/ruby/3.3.0/bundler/worker.rb:62:in `apply_func'
/usr/local/lib/ruby/3.3.0/bundler/worker.rb:57:in `block in process_queue'
<internal:kernel>:187:in `loop'
/usr/local/lib/ruby/3.3.0/bundler/worker.rb:54:in `process_queue'
/usr/local/lib/ruby/3.3.0/bundler/worker.rb:90:in `block (2 levels) in create_threads'
An error occurred while installing google-protobuf (3.25.1), and Bundler cannot continue.
In Gemfile:
google-protobuf
```
Closes#15203
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/15203 from ntkme:ruby-rake-dependency 1318863022
PiperOrigin-RevId: 600916613
This makes third_party/utf8_range no longer a Git subtree, but instead the
permanent location and source of truth for utf8_range. It is also now
incorporated into the @com_google_protobuf Bazel repo. Utf8_range still has its
own separate CMake build for now, though.
PiperOrigin-RevId: 580682733
* Bazelfying conformance tests
Adding infrastructure to "Bazelify" languages other than Java and C++
* Delete benchmarks for languages supported by other repositories
* Bazelfying benchmark tests
* Bazelfying python
Use upb's system python rule instead of branching tensorflow
* Bazelfying Ruby
* Bazelfying C#
* Bazelfying Objective-c
* Bazelfying Kokoro mac builds
* Bazelfying Kokoro linux builds
* Deleting all deprecated files from autotools cleanup
This boils down to Makefile.am and tests.sh and all of their remaining references
* Cleanup after PR reorganizing
- Enable 32 bit tests
- Move conformance tests back
- Use select statements to select alternate runtimes
- Add internal prefixes to proto library macros
* Updating READMEs to use bazel instead of autotools.
* Bazelfying Kokoro release builds
* First round of review fixes
* Second round of review fixes
* Third round of review fixes
* Filtering out conformance tests from Bazel on Windows (b/241484899)
* Add version metadata that was previously scraped from configure.ac
* fixing typo from previous fix
* Adding ruby version tests
* Bumping pinned upb version, and adding tests to python CI
Start reducing our reliance on autotools
This commit updates a few of our CI scripts to build protoc using Bazel
instead of autotools. I also updated the setuptools and Rake build
configs to look for protoc under bazel-bin/ so that they will
preferentially use a Bazel-built protoc binary. The goal is to start
chipping away at our use of autotools in preparation for eventually
dropping autotools support.
* Allow pre-compiled binaries for ruby 3.1.0 (#9566)
* Allow pre-compiled binaries for ruby 3.1.1
* add comment
* fix build and use ruby 3.1.0
* add ruby31 to build CI for tests and release
* trying to fix ci
* install ruby 3.1.0 in ruby_build_environment.sh
* use head for rvm to install 3.1.0
* just install master version of rvm in prepare_build_macos_rc
* force install of master rvm in ruby_build_environment.sh
* Use coroutine=universal when compiling ruby31
* use ucontext
* fix filename
* fix coroutine name
* use git head for rake-compiler-dock
* use newest rake-compiler-dock version
* Updated CHANGES.txt for Ruby changes.
* Fixed Ruby 3.1 tests by marking intersect? as unimplemented. (#9645)
* Fixed Ruby 3.1 tests by marking intersect? as unimplemented.
* Updated compatibility tests.
Co-authored-by: Marco Concetto Rudilosso <marcoconcettorudilosso@gmail.com>
* Allow pre-compiled binaries for ruby 3.1.1
* add comment
* fix build and use ruby 3.1.0
* add ruby31 to build CI for tests and release
* trying to fix ci
* install ruby 3.1.0 in ruby_build_environment.sh
* use head for rvm to install 3.1.0
* just install master version of rvm in prepare_build_macos_rc
* force install of master rvm in ruby_build_environment.sh
* Use coroutine=universal when compiling ruby31
* use ucontext
* fix filename
* fix coroutine name
* use git head for rake-compiler-dock
* use newest rake-compiler-dock version
* Updated PHP to the new version of upb.
This is a large change, as the upb API surface has been
renamed to follow Google style more closely.
* Fixed utf8_range.
* Updated Ruby for new utf8_range.
* Picked up new upb for PHP, with spelling fixes.
* Fixed the 32-bit build.
* Dropped Ruby 2.4 support for CI and releases.
* Removed Kokoro configs for old Ruby versions.
* Update Dockerfile to allow it to be rebuilt
Co-authored-by: Joshua Haberman <jhaberman@gmail.com>
* Update ruby_generator.cc to allow proto2 imports in proto3, with updated unit tests
* Update Makefile.am with new ruby_generated_code_proto2_import.proto
* Fix ruby_generator unit test to use temporary test directory for imported protos
* Add test for imported proto2 to ruby/tests
* Fix proto_path, restore to ../src/protoc, and fix/cleanup unit test.
* Rename Proto2TestMessage to TestImportedMessage for consistency, for ruby compiler tests
* Some more updates to PHP testing infrastructure (#8576)
* WIP.
* Added build config for all of the tests.
* Use ../src/protoc if it is available, for cases where Bazel isn't available.
* Added test_php.sh.
* Fix for the broken macOS tests.
* Move all jobs to use php80 instead of lots of separate jobs.
* Only pass -t flag if we are running in a terminal.
* Updated php_all job to use new Docker stuff.
* Fixed PHP memory leaks and arginfo errors (#8614)
* Fixed a bunch of incorrect arginfo and a few incorrect error messages.
* Passes mem check test with no leaks!
* WIP.
* Fix build warning that was causing Bazel build to fail.
* Added compatibility code for PHP <8.0.
* Added test_valgrind target and made tests Valgrind-clean.
* Updated Valgrind test to fail if memory leaks are detected.
* Removed intermediate shell script so commands are easier to cut, paste, and modify.
* Passing all Valgrind tests!
* Hoist addref into ObjCache_Get().
* Removed special case of map descriptors by keying object map on upb_msgdef.
* Removed all remaining RETURN_ZVAL() macros.
* Removed all explicit reference add/del operations.
* Added REFCOUNTING.md to Makefile.am.
* Updated upb version and fixed PHP to not get unset message field. (#8621)
* Updated upb version and fixed PHP to not get unset message field.
* Updated changelog.
* Fixed preproc test to handle old versions of Clang withot __has_attribute().
* A second try at fixing __has_attribute().
* Copy __has_attribute() fix to cc file also.
* Updated failure list for PHP for fixed test.
* Updated version of upb for Ruby (#8624)
* Updated upb.
* Preserve legacy behavior for unset messages.
* Updated failure list.
* Updated CHANGES.txt.
* Added erroneously-deleted test file.
* Fixed condition on compatibility code.
* Re-introduced deleted file again, and fixed Rakefile to not delete it.
* Fix generation of test protos.
* Select appropriate protoc binary for gem build
This should allow gem installation when
`bundle config force_ruby_platform true` has been set.
It has the unfortunate side effect of printing the protoc version during
build, but a quiet cross-platform solution is more complicated.
* Add path to wyhash header for source gem compilation
* WIP.
* WIP.
* WIP.
* WIP.
* WIP.
* WIP.
* Added some missing files.
* WIP.
* WIP.
* Updated upb.
* Extension loads, but crashes immediately.
* Gets through the test suite without SEGV!
Still a lot of bugs to fix, but it is a major step!
214 tests, 378 assertions, 37 failures, 147 errors, 0 pendings, 0 omissions, 0 notifications
14.0187% passed
* Test and build for Ruby 3.0
* Fixed a few more bugs, efficient #inspect is almost done.
214 tests, 134243 assertions, 30 failures, 144 errors, 0 pendings, 0 omissions, 0 notifications
18.6916% passed
* Fixed message hash initialization and encode depth checking.
214 tests, 124651 assertions, 53 failures, 70 errors, 0 pendings, 0 omissions, 0 notifications
42.5234% passed
* A bunch of fixes to failing tests, now 70% passing.
214 tests, 202091 assertions, 41 failures, 23 errors, 0 pendings, 0 omissions, 0 notifications
70.0935% passed
* More than 80% of tests are passing now.
214 tests, 322331 assertions, 30 failures, 9 errors, 0 pendings, 0 omissions, 0 notifications
81.7757% passed
Unfortunately there is also a sporadic bug/segfault hanging around
that appears to be GC-related.
* Add linux/ruby30 and macos/ruby30
* Use rvm master for 3.0.0-preview2
* Over 90% of tests are passing!
214 tests, 349898 assertions, 15 failures, 1 errors, 0 pendings, 0 omissions, 0 notifications
92.5234% passed
* Passes all tests!
214 tests, 369388 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed
* A bunch of cleanup.
1. Removed a bunch of internal-only symbols from headers.
2. Required a frozen check to get a non-const pointer to a map or array.
3. De-duplicated the code to get a type argument for Map/RepeatedField.
* Removed a bunch more stuff from protobuf.h. There is an intermittent assert failure.
Intermittent failure:
ruby: ../../../../ext/google/protobuf_c/protobuf.c:263: ObjectCache_Add: Assertion `rb_funcall(obj_cache2, (__builtin_constant_p("[]") ? __extension__ ({ static ID rb_intern_id_cache; if (!rb_intern_id_cache) rb_intern_id_cache = rb_intern2((("[]")
), (long)strlen(("[]"))); (ID) rb_intern_id_cache; }) : rb_intern("[]")), 1, key_rb) == val' failed
* Removed a few more things from protobuf.h.
* Ruby 3.0.0-preview2 to 3.0.0
* Require rake-compiler-dock >= 1.1.0
* More progress, fighting with the object cache.
* Passes on all Ruby versions!
* Updated and clarified comment regarding WeakMap.
* Fixed the wyhash compile.
* Fixed conformance tests for Ruby.
Conformance results now look like:
RUBYLIB=../ruby/lib:. ./conformance-test-runner --enforce_recommended --failure_list failure_list_ruby.txt --text_format_failure_list text_format_failure_list_ruby.txt ./conformance_ruby.rb
CONFORMANCE TEST BEGIN ====================================
CONFORMANCE SUITE PASSED: 1955 successes, 0 skipped, 58 expected failures, 0 unexpected failures.
CONFORMANCE TEST BEGIN ====================================
CONFORMANCE SUITE PASSED: 0 successes, 111 skipped, 8 expected failures, 0 unexpected failures.
Fixes include:
- Changed Ruby compiler to no longer reject proto2 maps.
- Changed Ruby compiler to emit a warning when proto2 extensions are
present instead of rejecting the .proto file completely.
- Fixed conformance tests to allow proto2 and look up message by name
instead of hardcoding a specific list of messages.
- Fixed conformance test to support the "ignore unknown" option for
JSON.
- Fixed conformance test to properly report serialization errors.
* Removed debug printf and fixed #inspect for floats.
* Fixed compatibility test to have proper semantics for #to_json.
* Updated Makefile.am with new file list.
* Don't try to copy wyhash when inside Docker.
* Fixed bug where we would forget that a sub-object is frozen in Ruby >=2.7.
* Avoid exporting unneeded symbols and refactored a bit of code.
* Some more refactoring.
* Simplified and added more comments.
* Some more comments and simplification. Added a missing license block.
Co-authored-by: Masaki Hara <hara@wantedly.com>
It is no longer possible to "bundle install" these versions:
$ bundle install
Fetching gem metadata from https://rubygems.org/.................
reline-0.1.3 requires ruby version >= 2.5, which is incompatible with the current version, ruby 2.4.9p362
* WIP.
* WIP.
* Builds and runs. Tests need to be updated to test presence.
* Ruby: proto3 presence is passing all tests.
* Fixed a bug where empty messages has the wrong oneof count.
* Build extensions for Ruby 2.7
* Try installing bundler 2.x
* Try bumping rake-compiler-dock
* Use standard RCD images
* Avoid 'rake cross native' with rake-compiler-dock
* Use Ruby 2.5 for building Ruby <= 2.6
* Use rake-compiler 1.1.0
* Specify target
* Don't update Ruby test image for now
Yibo Cai
Joshua Haberman
KJ Tsanaktsidis
Mike Kruskal
なつき
Protobuf Team Bot
Adam Cozzette
Jason Lunn
Deanna Garcia
Eric Salo
Mike Kruskal
johnnyshields
Joshua Haberman
Marco Concetto Rudilosso
zhangskz
Bo Yang
James D
Masaki Hara
Rob Widmer
Daniel Azuma
listonjesse
Masaki Hara
Joe Bolinger
bmoyles0117