Add some missing endian conversions so that the conformance tests can be run on big endian platforms.
The message length value created by the conformance test runner is little endian according to the comments in the file but actually was sent in the native endianness of the host. I was able to run the java, python, ruby, php and csharp test executables and they all expect little endian length values so those tests would hang on big endian machines. Only the cpp test executable was using native endian so it has been changed to expect little endian too.
Also change the fixed32 and fixed64 functions in binary_json_conformance_test_suite.cc to send the data as little endian which fixes some failures in the python conformance tests on big endian platforms.
Closes#13443
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/13443 from linux-on-ibm-z:conformance-runner-little-endian-fix 4ef7948997
PiperOrigin-RevId: 553958649
Going forward we will be forbidding `null` in `json_name`, but we will continue to support embedded `null` in JSON value. This conformance test will help clarify that this is a supported use case.
PiperOrigin-RevId: 527918031
This PR removes the DSL from the code generator, in anticipation of splitting the DSL out into a separate package.
Given a .proto file like:
```proto
syntax = "proto3";
package pkg;
message TestMessage {
optional int32 i32 = 1;
optional TestMessage msg = 2;
}
```
Generated code before:
```ruby
# Generated by the protocol buffer compiler. DO NOT EDIT!
# source: protoc_explorer/main.proto
require 'google/protobuf'
Google::Protobuf::DescriptorPool.generated_pool.build do
add_file("test.proto", :syntax => :proto3) do
add_message "pkg.TestMessage" do
proto3_optional :i32, :int32, 1
proto3_optional :msg, :message, 2, "pkg.TestMessage"
end
end
end
module Pkg
TestMessage = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("pkg.TestMessage").msgclass
end
```
Generated code after:
```ruby
# frozen_string_literal: true
# Generated by the protocol buffer compiler. DO NOT EDIT!
# source: test.proto
require 'google/protobuf'
descriptor_data = "\n\ntest.proto\x12\x03pkg\"S\n\x0bTestMessage\x12\x10\n\x03i32\x18\x01 \x01(\x05H\x00\x88\x01\x01\x12\"\n\x03msg\x18\x02 \x01(\x0b\x32\x10.pkg.TestMessageH\x01\x88\x01\x01\x42\x06\n\x04_i32B\x06\n\x04_msgb\x06proto3"
begin
Google::Protobuf::DescriptorPool.generated_pool.add_serialized_file(descriptor_data)
rescue TypeError => e
# <compatibility code, see below>
end
module Pkg
TestMessage = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("pkg.TestMessage").msgclass
end
```
This change fixes nearly all remaining conformance problems that existed previously. This is a side effect of moving from the DSL (which is lossy) to a serialized descriptor (which preserves all information).
## Backward Compatibility
This change should be 100% compatible with Ruby Protobuf >= 3.18.0, released in Sept 2021. Additionally, it should be compatible with all existing users and deployments. However there is some special compatibility code I inserted to achieve this level of backward compatibility.
Without the compatibility code, there is an edge case that could break backward compatibility. The existing code is lax in a way that the new code would be more strict.
When we use a full serialized descriptor, it will contain a list of all `.proto` files imported by this file (whereas the DSL never added dependencies properly): dfb71558a2/src/google/protobuf/descriptor.proto (L65-L66)
`add_serialized_file` will verify that all dependencies listed in the descriptor were previously added with `add_serialized_file`. Generally that should be fine, because the generated code will contain Ruby `require` statements for all dependencies, and the descriptor will fail to load anyway if the types we depend on were not previously defined in the DescriptorPool.
But there is a potential for problems if there are ambiguities around file paths. For example, consider the following scenario:
```proto
// foo/bar.proto
syntax = "proto2";
message Bar {}
```
```proto
// foo/baz.proto
syntax = "proto2";
import "bar.proto";
message Baz {
optional Bar bar = 1;
}
```
If you invoke `protoc` like so, it will work correctly:
```
$ protoc --ruby_out=. -Ifoo foo/bar.proto foo/baz.proto
$ RUBYLIB=. ruby baz_pb.rb
```
However if you invoke `protoc` like so, and didn't have any compatibility code, it would fail to load:
```
$ protoc --ruby_out=. -I. -Ifoo foo/baz.proto
$ protoc --ruby_out=. -I. -Ifoo foo/bar.proto
$ RUBYLIB=foo ruby foo/baz_pb.rb
foo/baz_pb.rb:10:in `add_serialized_file': Unable to build file to DescriptorPool: Depends on file 'bar.proto', but it has not been loaded (Google::Protobuf::TypeError)
from foo/baz_pb.rb:10:in `<main>'
```
The problem is that `bar.proto` is being referred to by two different canonical names: `bar.proto` and `foo/bar.proto`. This is a user error: each import should always be referred to by a consistent full path. Hopefully user errors of this sort are rare, but it is hard to know without trying.
The code in this PR prints a warning using `warn` if we detect that this edge case has occurred. We will plan to remove this compatibility code in the next major version.
Closes#12319
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/12319 from haberman:ruby-gencode-binary 5c0e8f20b1
PiperOrigin-RevId: 524129023
- General naming up to "Objective-C" that some linters want, so just
use it every where for consistence.
- Add a bazel macOS config
- Includes the conformance test, so drop it from the Xcode flows.
- Tweak the naming on the Xcode flows so they are a little more clear.
- CocoaPods flows:
- Give them (hopefully) clear naming
- Use the same `image` as the Xcode tests
- Pin the Xcode version to match the Xcode tests
PiperOrigin-RevId: 508156130
This deletes our old branches of internal Abseil code in favor of their newly open-sourced library. Notably, this removes the ability to turn CHECK crashes into exceptions.
PiperOrigin-RevId: 504460562
Implemented in java, c++, python and upb. Also added conformance test.
https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Value
where it says:
attempting to serialize NaN or Infinity results in error. (We can't serialize these as string "NaN" or "Infinity" values like we do for regular fields, because they would parse as string_value, not number_value).
PiperOrigin-RevId: 500828964
Implemented in java, c++, python and upb. Also added conformance test.
https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#google.protobuf.Value
where it says:
attempting to serialize NaN or Infinity results in error. (We can't serialize these as string "NaN" or "Infinity" values like we do for regular fields, because they would parse as string_value, not number_value).
PiperOrigin-RevId: 500139380
This uses https://github.com/protocolbuffers/rules_ruby to fully Bazelify our ruby runtime code. The Rakefile is left in place for now and is still used by our aarch64 tests. With the current implementation ruby behaves similarly to our python wrapper, which selects whatever version is installed in the system. Future enhancements will allow for more hermetic builds via Bazel flags to pin a specific version
Closes#10525
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/10525 from mkruskal-google:rules_ruby 97fa1f70ab
PiperOrigin-RevId: 499283908
reassign sys.stdin and sys.stdout with files that do not match the text API
contract of those. This fixes a stdin pipe IO race condition on input that was
leading to flakiness depending on the test system load.
PiperOrigin-RevId: 489320918
* Adding jsoncpp submodule
* Adding bazel dependency
* Hook up jsoncpp in Bazel builds
* Hook up jsoncpp dependency in CMake
* Fix conformance binary path
* Move jsoncpp import to the end of the file to avoid confusing add_test
* Added staleness test for ruby-upb.{c,h} and updated.
* Removed file comment markers, too much trouble for too little benefit.
* Ran clang-format.
* Updated ruby-upb.{c,h}.
* Added missing table code to amalgamation.
* Updated to latest upb, patch no longer needed.
* Reverted changes to third_party sub-modules.
* Added missing unicode file.
* Removed conformance failures for Ruby.
* Enable warnings as errors by default for test builds
* Fixing C++ warnings
* Adding host flags, and enabling warnings as error for non-C++ too
* Switch to BUILD copts instead of bazelrc to treat Windows as a snowflake
* Disable warnings as errors on Windows, since it doesn't like the c++14 flag
* Adding detailed documentation on how to run conformance tests with bazel
* Fix C# instructions
* Fixing formatting of command snippets
* More formatting fixes
* Capitalization
* Sync from Piper @469587494
PROTOBUF_SYNC_PIPER
* Fixing github SOT protoc builds
* Fixing typos from google
* Remove leaked util/hash reference
* Fixing bad python merge
* Fixing python C++ library order
* Fixing typos
* Revert new files that were deleted by sync script
* Fix CMake breakages
* bump upb version
* Sync from Piper @468772608
PROTOBUF_SYNC_PIPER
* Adding abseil to include path for python C++ extension
* Adding abseil linkage for python C++ extension
* Fixing linkage order
* Update conformance test documentation
* Fix bug to allow custom argument forwarding to conformance tests
* Fix trailing sentence
* Add note about linux CMake support up top
* Add note about CMake being C++-only
* Bazelfying conformance tests
Adding infrastructure to "Bazelify" languages other than Java and C++
* Delete benchmarks for languages supported by other repositories
* Bazelfying benchmark tests
* Bazelfying python
Use upb's system python rule instead of branching tensorflow
* Bazelfying Ruby
* Bazelfying C#
* Bazelfying Objective-c
* Bazelfying Kokoro mac builds
* Bazelfying Kokoro linux builds
* Deleting all deprecated files from autotools cleanup
This boils down to Makefile.am and tests.sh and all of their remaining references
* Cleanup after PR reorganizing
- Enable 32 bit tests
- Move conformance tests back
- Use select statements to select alternate runtimes
- Add internal prefixes to proto library macros
* Updating READMEs to use bazel instead of autotools.
* Bazelfying Kokoro release builds
* First round of review fixes
* Second round of review fixes
* Third round of review fixes
* Filtering out conformance tests from Bazel on Windows (b/241484899)
* Add version metadata that was previously scraped from configure.ac
* fixing typo from previous fix
* Adding ruby version tests
* Bumping pinned upb version, and adding tests to python CI
Protobuf Team Bot
Jonathan Albrecht
Jason Lunn
Joshua Haberman
Joshua Haberman
Mike Kruskal
Jie Luo
Gregory P. Smith
Anton Grbin
Mike Kruskal
Thomas Van Lenten
deannagarcia
zhangskz
Sandy Zhang
Adam Cozzette
Zouhair Mahieddine