This CL migrates messages, enums, and primitive types all onto the same blanket
implementation of the `ProxiedInMapValue` trait. This gets us to the point
where messages and enums no longer need to generate any significant amount of
extra code just in case they might be used as a map value.
There are a few big pieces to this:
- I generalized the message-specific FFI endpoints in `rust/cpp_kernel/map.cc`
to be able to additionally handle enums and primitive types as values. This
mostly consisted of replacing `MessageLite*` parameters with a new `MapValue`
tagged union.
- On the Rust side, I added a new blanket implementation of
`ProxiedInMapValue` in rust/cpp.rs. It relies on its value type to implement
a new `CppMapTypeConversions` trait so that it can convert to and from the
`MapValue` tagged union used for FFI.
- In the Rust generated code, I deleted the generated `ProxiedInMapValue`
implementations for messages and enums and replaced them with
implementations of the `CppMapTypeConversions` trait.
PiperOrigin-RevId: 687355817
This change adds delete, clear, serialize, parse, copy_from, and merge_from
operations to the runtime. Since these operations can all be implemented easily
on the `MessageLite` interface, we can use a common implementation in the
runtime instead of generating per-message thunks for all of these.
I suspect this will also make it possible to remove some of our generated trait
implementations and replace them with blanket implementations, but I will leave
that for a future change.
PiperOrigin-RevId: 665910927
With the C++ kernel for Rust, we currently need to generate quite a few C++
thunks for operations on map fields. For each message we generate, we generate
these thunks for all possible map types that could have that message as a
value. These operations are for things such as insertion, removal, clearing,
iterating, etc.
The reason we do this is that templated types don't play well with FFI, so we
effectively need separate FFI endpoints for every possible combination of key
and value types used (or even potentially used) as a map field.
This CL fixes the problem by replacing the generated thunks with functions in
the runtime that can operate on `proto2::MessageLite*` without needing to care
about the specific message type.
The way it works is that we implement the operations using either
`UntypedMapBase` (the base class of all map types, which knows nothing about
the key and value types) or `KeyMapBase`, which knows the key type but not the
value type. I roughly followed the example of the table-driven parser, which
has a similar problem of needing to operate generically on maps without having
access to the concrete types.
I removed 54 thunks per message (that's 6 key types times 9 operations per
key), but had to add two new thunks per message:
- The `size_info` thunk looks up the `MapNodeSizeInfoT`, which is stored in a
small constant table. The important thing here is an offset indicating where
to look for the value in each map entry. This offset can be different for
every pair of key and value types, but we can safely assume that the result
does not depend on the signedness of the key. As a result we only need to
store four entries per message: one each for i32, i64, bool, and string.
- The `placement_new` thunk move-constructs a message in place. We need this
to be able to efficiently implement map insertion.
There are two big things that this CL does not address yet but which I plan to
follow up on:
- Enums still generate many map-related C++ thunks that could be replaced with
a common implementation. This should actually be much easier to handle than
messages, because every enum has the same representation as an i32.
- We still generate six `ProxiedInMapValue` implementations for every message,
but it should be possible to replace these with a blanket implementation that
works for all message types.
PiperOrigin-RevId: 657681421
We recently updated the codebase to comply with the Bazel layering check, which
essentially requires any C++ header inclusion to be matched with a build
dependency on a target providing that header.
As part of that, I removed a handful of dependencies from the `//:protobuf`
target, since these dependencies were not set up in a way that respected the
layering check. However, I realized that this may cause a number of breakages,
especially since we did not provide the correct public targets until very
recently.
This change effectively adds back in the missing dependencies, so that projects
which do not yet adhere to the layering check can continue to depend on them
indirectly. This way, we still adhere to the layering check and make it
possible for projects that depend on us to do so, but in most cases we won't
immediately break anyone.
PiperOrigin-RevId: 607021111
This check enforces that each C++ build target has the correct dependencies for
all headers that it includes. We have many targets that were not correct with
respect to this check, so I fixed them up.
I also cleaned up the C++ targets related to the well-known types. I created a
cc_proto_library() target for each one and removed the :wkt_cc_protos target,
since this was necessary to satisfy the layering check. I deleted the
//src/google/protobuf:protobuf_nowkt target and deprecated :protobuf_nowkt,
because the distinction between the :protobuf and :protobuf_nowkt targets was
not really correct. Neither one exposed the headers for the well-known types in
a way that was valid with respect to the layering check, and the idea of
bundling all the well-known types together is not idiomatic in Bazel anyway.
This is a breaking change, because the //:protobuf target no longer bundles the
well-known types. From now on they should be accessed through the new
//:*_cc_proto aliases in our top-level package.
I renamed the :port_def target to :port, which simplifies things a bit by
matching our internal name.
The original motivation for this change was that to move utf8_range onto our CI
infrastructure, we needed to make its dependency rules_fuzzing compatible with
Bazel 6. The rules_fuzzing project builds with the layering check, and I found
that the process of upgrading it to Bazel 6 made it take a dependency on
protobuf, which caused it to break due to layering violations. I was able to
work around this, but it would still be nice to comply with the layering check
so that we don't have to worry about this kind of thing in the future.
PiperOrigin-RevId: 595516736
To satisfy the layering check, we need to depend on :gtest for the headers, in
addition to :gtest_main which provides the main() function.
There are a bunch of formatting changes as a side effect of this, but they
should be harmless.
PiperOrigin-RevId: 594318263
This change implements maps with keys and values of type string e.g. Map<ProtoStr, i32> and Map<ProtoStr, ProtoStr>.
Implementing the Map type for ProtoStr has been different from scalar types because ProtoStr is an unsized type i.e. its size is not known at compile time. The existing Map implementation assumed sized types in many places. To make unsized types fit into the existing code architecture I have added an associated type 'Value' to the MapWith*KeyOps traits. The associated type needs to be sized and is the type returned by the Map::get(self, key) method e.g. for aProtoStr, the `type Value = &ProtoStr`.
PiperOrigin-RevId: 588783751
This change moves almost everything in the `upb/` directory up one level, so
that for example `upb/upb/generated_code_support.h` becomes just
`upb/generated_code_support.h`. The only exceptions I made to this were that I
left `upb/cmake` and `upb/BUILD` where they are, mostly because that avoids
conflict with other files and the current locations seem reasonable for now.
The `python/` directory is a little bit of a challenge because we had to merge
the existing directory there with `upb/python/`. I made `upb/python/BUILD` into
the BUILD file for the merged directory, and it effectively loads the contents
of the other BUILD file via `python/build_targets.bzl`, but I plan to clean
this up soon.
PiperOrigin-RevId: 568651768
In this CL I'd like to call existing C++ Protobuf API from the V0 Rust API. Since parts of the C++ API are defined inline and using (obviously) C++ name mangling, we need to create a "thunks.cc" file that:
1) Generates code for C++ API function we use from Rust
2) Exposes these functions without any name mangling (meaning using `extern "C"`)
In this CL we add Bazel logic to generate "thunks" file, compile it, and propagate its object to linking. We also add logic to protoc to generate this "thunks" file.
The protoc logic is rather rudimentary still. I hope to focus on protoc code quality in my followup work on V0 Rust API using C++ kernel.
PiperOrigin-RevId: 523479839
In this CL we're adding the barebones infrastructure to generate Rust proto messages using UPB as a backend. The API is what we call a V0, not yet production-quality, not yet rigorously designed, just something to enable parallel work.
The interesting part of switching backend between UPB and C++ will come in a followup.
PiperOrigin-RevId: 517089760
Adam Cozzette
Yamil Morales
Protobuf Team Bot
Jakob Buchgraber
Marcel Hlopko