This represents the future direction of protobuf, replacing proto2/proto3 syntax with editions. These will enable more incremental evolution of protobuf APIs through features, which are individual behaviors (such as whether field presence is explicit or implicit). For more details see https://protobuf.dev/editions/overview/.
This PR contains a working implementation of editions for the protoc frontend and C++ code generation, along with various infrastructure improvements to support it. It gives early access for anyone who wants to a preview of editions, but has no effect on proto2/proto3 syntax. It is flag-guarded behind the `--experimental_editions` flag, and is an experimental feature with no guarantees.
PiperOrigin-RevId: 544805690
Prior to this CL, asserts have no effect for Ruby 3+, because Ruby unconditionally defines `NDEBUG`: https://bugs.ruby-lang.org/issues/18777
We work around this by doing `#undef NDEBUG` right after including Ruby, if `NDEBUG` was not previously defined.
PiperOrigin-RevId: 535359088
An extension range is either of verification state "UNVERIFIED" or "DECLARATION". If "DECLARATION", all extension fields of the range must be declared, or build error otherwise. The current default is "UNVERIFIED", but we will flip the default later.
Deprecate `is_repeated` in favor of `repeated`.
PiperOrigin-RevId: 526184056
This PR removes the DSL from the code generator, in anticipation of splitting the DSL out into a separate package.
Given a .proto file like:
```proto
syntax = "proto3";
package pkg;
message TestMessage {
optional int32 i32 = 1;
optional TestMessage msg = 2;
}
```
Generated code before:
```ruby
# Generated by the protocol buffer compiler. DO NOT EDIT!
# source: protoc_explorer/main.proto
require 'google/protobuf'
Google::Protobuf::DescriptorPool.generated_pool.build do
add_file("test.proto", :syntax => :proto3) do
add_message "pkg.TestMessage" do
proto3_optional :i32, :int32, 1
proto3_optional :msg, :message, 2, "pkg.TestMessage"
end
end
end
module Pkg
TestMessage = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("pkg.TestMessage").msgclass
end
```
Generated code after:
```ruby
# frozen_string_literal: true
# Generated by the protocol buffer compiler. DO NOT EDIT!
# source: test.proto
require 'google/protobuf'
descriptor_data = "\n\ntest.proto\x12\x03pkg\"S\n\x0bTestMessage\x12\x10\n\x03i32\x18\x01 \x01(\x05H\x00\x88\x01\x01\x12\"\n\x03msg\x18\x02 \x01(\x0b\x32\x10.pkg.TestMessageH\x01\x88\x01\x01\x42\x06\n\x04_i32B\x06\n\x04_msgb\x06proto3"
begin
Google::Protobuf::DescriptorPool.generated_pool.add_serialized_file(descriptor_data)
rescue TypeError => e
# <compatibility code, see below>
end
module Pkg
TestMessage = ::Google::Protobuf::DescriptorPool.generated_pool.lookup("pkg.TestMessage").msgclass
end
```
This change fixes nearly all remaining conformance problems that existed previously. This is a side effect of moving from the DSL (which is lossy) to a serialized descriptor (which preserves all information).
## Backward Compatibility
This change should be 100% compatible with Ruby Protobuf >= 3.18.0, released in Sept 2021. Additionally, it should be compatible with all existing users and deployments. However there is some special compatibility code I inserted to achieve this level of backward compatibility.
Without the compatibility code, there is an edge case that could break backward compatibility. The existing code is lax in a way that the new code would be more strict.
When we use a full serialized descriptor, it will contain a list of all `.proto` files imported by this file (whereas the DSL never added dependencies properly): dfb71558a2/src/google/protobuf/descriptor.proto (L65-L66)
`add_serialized_file` will verify that all dependencies listed in the descriptor were previously added with `add_serialized_file`. Generally that should be fine, because the generated code will contain Ruby `require` statements for all dependencies, and the descriptor will fail to load anyway if the types we depend on were not previously defined in the DescriptorPool.
But there is a potential for problems if there are ambiguities around file paths. For example, consider the following scenario:
```proto
// foo/bar.proto
syntax = "proto2";
message Bar {}
```
```proto
// foo/baz.proto
syntax = "proto2";
import "bar.proto";
message Baz {
optional Bar bar = 1;
}
```
If you invoke `protoc` like so, it will work correctly:
```
$ protoc --ruby_out=. -Ifoo foo/bar.proto foo/baz.proto
$ RUBYLIB=. ruby baz_pb.rb
```
However if you invoke `protoc` like so, and didn't have any compatibility code, it would fail to load:
```
$ protoc --ruby_out=. -I. -Ifoo foo/baz.proto
$ protoc --ruby_out=. -I. -Ifoo foo/bar.proto
$ RUBYLIB=foo ruby foo/baz_pb.rb
foo/baz_pb.rb:10:in `add_serialized_file': Unable to build file to DescriptorPool: Depends on file 'bar.proto', but it has not been loaded (Google::Protobuf::TypeError)
from foo/baz_pb.rb:10:in `<main>'
```
The problem is that `bar.proto` is being referred to by two different canonical names: `bar.proto` and `foo/bar.proto`. This is a user error: each import should always be referred to by a consistent full path. Hopefully user errors of this sort are rare, but it is hard to know without trying.
The code in this PR prints a warning using `warn` if we detect that this edge case has occurred. We will plan to remove this compatibility code in the next major version.
Closes#12319
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/12319 from haberman:ruby-gencode-binary 5c0e8f20b1
PiperOrigin-RevId: 524129023
This can be useful when an extension field/declaration is deleted in schema to avoid data corruption.
It also adds a check for duplicate numbers in the declarations.
PiperOrigin-RevId: 523269837
Syntax will become meaningless once we migrate to editions. In the meantime, we've implemented APIs to expose the differences between proto2 and proto3 in terms of the features we plan to release in edition 2023.
PiperOrigin-RevId: 520433607
Write barrier protected objects are allowed to be promoted to the old generation, which means they only get marked on major GC.
The downside is that the `RB_BJ_WRITE` macro MUST be used to set references, otherwise the referenced object may be garbaged collected.
But the `*Descriptor` classes and `Arena` have very few references and are only set in a few places, so it's relatively easy to implement.
cc @peterzhu2118Closes#11793
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/11793 from casperisfine:descriptor-write-barrier 215e8fad4c
PiperOrigin-RevId: 511875342
This fixes a v22 regression for Ruby.
Windows builds of the Ruby package are broken in 3.22.0.rc.2. The reason is, `ruby-upb.h` no longer provides the full path when `#include`ing `utf8_range.h`. See https://github.com/protocolbuffers/protobuf/blob/v22.0-rc2/ruby/ext/google/protobuf_c/ruby-upb.h#L10479 (22.0-rc-2), cf. https://github.com/protocolbuffers/protobuf/blob/v21.12/ruby/ext/google/protobuf_c/ruby-upb.h#L5365 (21.12). The `extconf.rb` build configuration tries to compensate by adding `third_party/utf8_range` to the include path, but does not do so on Windows (i.e. `mingw`) platforms. See https://github.com/protocolbuffers/protobuf/blob/v22.0-rc2/ruby/ext/google/protobuf_c/extconf.rb#L9-L10 (22.0-rc-2), cf. https://github.com/protocolbuffers/protobuf/blob/v21.12/ruby/ext/google/protobuf_c/extconf.rb#L9-L10 (21.12).
We could have fixed this by adding another clause to the if statement for the case `RUBY_PLATFORM =~ /mingw/` and adding the appropriate `-I` flag to `CFLAGS`. However, that `CFLAGS` hack is present in the first place due to a related problem: the usage of `$INCFLAGS` is incorrect.
The `$INCFLAGS` constant in Ruby's `mkmf` is a string in a similar format to `CFLAGS`. It's simply appended to compiler invocations. So when you append new flags to it, you have to (1) provide the flag itself, and (2) precede it with a space to delimit it from the previous entry. In 22.0-rc-2 (and indeed in all earlier versions) the usage is incorrect: it's appending a path to the string without the `-I` and without a space. See https://github.com/protocolbuffers/protobuf/blob/v22.0-rc2/ruby/ext/google/protobuf_c/extconf.rb#L22. Hence, not only does the intended include path not get appended correctly, it also clobbers the previous path in the string. Luckily, the previous path is only `-I$(srcdir)` which happens not to matter for our library. But it does mean that the apparent intent of that line, adding `$(srcdir)/third_party/utf8_range` to the include path, isn't working; hence the code that adds it to `CFLAGS` instead.
(Note that the previous line, adding the path to `$VPATH`, _is_ correct as is, because `$VPATH` is an array.)
So what this PR actually does is fix the `$INCFLAGS` usage so `$(srcdir)/third_party/utf8_range` gets added properly to compiler include paths, for all platforms including Windows. Since that should now be working as intended, we also remove the extra `-I` from CFLAGS. Builds for all platforms should now be able to handle the change to `ruby-upb.h`. This has been tested by running the prototype Ruby build Kokoro job against a patched 22.0-rc-2.
This also needs to be backported to the 22.x branch.
/attn @deannagarciaCloses#11882
COPYBARA_INTEGRATE_REVIEW=https://github.com/protocolbuffers/protobuf/pull/11882 from dazuma:pr/ruby-builds-fix ebb18e0004
PiperOrigin-RevId: 508550039
Mike Kruskal
Jason Lunn
Jie Luo
なつき
Joshua Haberman
Jean byroot Boussier
deannagarcia
Protobuf Team Bot
Protobuf Team Bot
Adam Cozzette
Deanna Garcia
Joshua Haberman
Sandy Zhang
Eric Salo
Jakob Buchgraber
zhangskz
Minh Quy
Daniel Azuma