Chiebot-Mirror
/
protobuf
mirror of https://github.com/protocolbuffers/protobuf.git
8
0
Fork 0
Code Issues Projects Releases Wiki Activity
Protocol Buffers - Google's data interchange format (grpc依赖) https://developers.google.com/protocol-buffers/
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
18556 Commits
201 Branches
346 Tags
254 MiB
Tag: Branch: Tree: 1f472f1db3
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '1f472f1db3'
${ noResults }
protobuf/rust/cpp_kernel/map.h

123 lines
7.7 KiB
Raw Normal View History Unescape

Split up cpp_api.h/.cc into smaller units. PiperOrigin-RevId: 647663342
5 months ago
#ifndef GOOGLE_PROTOBUF_RUST_CPP_KERNEL_MAP_H__
#define GOOGLE_PROTOBUF_RUST_CPP_KERNEL_MAP_H__
Implement bytes field accessors for v0 Rust API PiperOrigin-RevId: 524089369
2 years ago
Rust protobuf: fix memory leaks Our ASAN test runs have not had the heap checker enabled, so this has allowed a few memory leaks to slip in. This CL fixes all of them so that we can turn on the heap checker. The first one takes place whenever we add an entry into a string-valued map using the C++ kernel. The problem is that `InnerProtoString::into_raw()` gives up ownership of the raw `std::string` pointer it holds, but then we never delete that pointer. This CL fixes the problem by deleting the pointer in C++ right after we perform the map insertion. To simplify things, I created a `MakeCleanup()` helper function that we always call in our map insertion thunks, but it's a no-op in the cases where we don't need to free anything. There were a couple similar memory leaks related to repeated field accessors in the C++ kernel, and those were simple to fix just by adding the necessary `delete` call. Finally, there were two benign memory leaks in the upb kernel involving global variables used for empty repeated fields and maps. It turned out that we did not need to use `Box` at all here, so removing that simplified things and fixed the leaks. PiperOrigin-RevId: 652947042
5 months ago
#include <memory>
Protobuf Rust: simplify the MakeCleanup() helper function By changing the return type to `auto`, we can handle `std::string` and other types in a single definition without needing a separate overload. PiperOrigin-RevId: 653272253
5 months ago
#include <type_traits>
Rust protobuf: fix memory leaks Our ASAN test runs have not had the heap checker enabled, so this has allowed a few memory leaks to slip in. This CL fixes all of them so that we can turn on the heap checker. The first one takes place whenever we add an entry into a string-valued map using the C++ kernel. The problem is that `InnerProtoString::into_raw()` gives up ownership of the raw `std::string` pointer it holds, but then we never delete that pointer. This CL fixes the problem by deleting the pointer in C++ right after we perform the map insertion. To simplify things, I created a `MakeCleanup()` helper function that we always call in our map insertion thunks, but it's a no-op in the cases where we don't need to free anything. There were a couple similar memory leaks related to repeated field accessors in the C++ kernel, and those were simple to fix just by adding the necessary `delete` call. Finally, there were two benign memory leaks in the upb kernel involving global variables used for empty repeated fields and maps. It turned out that we did not need to use `Box` at all here, so removing that simplified things and fixed the leaks. PiperOrigin-RevId: 652947042
5 months ago
Rust: cut down on the amount of generated C++ code needed for maps With the C++ kernel for Rust, we currently need to generate quite a few C++ thunks for operations on map fields. For each message we generate, we generate these thunks for all possible map types that could have that message as a value. These operations are for things such as insertion, removal, clearing, iterating, etc. The reason we do this is that templated types don't play well with FFI, so we effectively need separate FFI endpoints for every possible combination of key and value types used (or even potentially used) as a map field. This CL fixes the problem by replacing the generated thunks with functions in the runtime that can operate on `proto2::MessageLite*` without needing to care about the specific message type. The way it works is that we implement the operations using either `UntypedMapBase` (the base class of all map types, which knows nothing about the key and value types) or `KeyMapBase`, which knows the key type but not the value type. I roughly followed the example of the table-driven parser, which has a similar problem of needing to operate generically on maps without having access to the concrete types. I removed 54 thunks per message (that's 6 key types times 9 operations per key), but had to add two new thunks per message: - The `size_info` thunk looks up the `MapNodeSizeInfoT`, which is stored in a small constant table. The important thing here is an offset indicating where to look for the value in each map entry. This offset can be different for every pair of key and value types, but we can safely assume that the result does not depend on the signedness of the key. As a result we only need to store four entries per message: one each for i32, i64, bool, and string. - The `placement_new` thunk move-constructs a message in place. We need this to be able to efficiently implement map insertion. There are two big things that this CL does not address yet but which I plan to follow up on: - Enums still generate many map-related C++ thunks that could be replaced with a common implementation. This should actually be much easier to handle than messages, because every enum has the same representation as an i32. - We still generate six `ProxiedInMapValue` implementations for every message, but it should be possible to replace these with a blanket implementation that works for all message types. PiperOrigin-RevId: 657681421
4 months ago
#include "google/protobuf/map.h"
#include "google/protobuf/message_lite.h"
#include "rust/cpp_kernel/strings.h"
Rust protobuf: fix memory leaks Our ASAN test runs have not had the heap checker enabled, so this has allowed a few memory leaks to slip in. This CL fixes all of them so that we can turn on the heap checker. The first one takes place whenever we add an entry into a string-valued map using the C++ kernel. The problem is that `InnerProtoString::into_raw()` gives up ownership of the raw `std::string` pointer it holds, but then we never delete that pointer. This CL fixes the problem by deleting the pointer in C++ right after we perform the map insertion. To simplify things, I created a `MakeCleanup()` helper function that we always call in our map insertion thunks, but it's a no-op in the cases where we don't need to free anything. There were a couple similar memory leaks related to repeated field accessors in the C++ kernel, and those were simple to fix just by adding the necessary `delete` call. Finally, there were two benign memory leaks in the upb kernel involving global variables used for empty repeated fields and maps. It turned out that we did not need to use `Box` at all here, so removing that simplified things and fixed the leaks. PiperOrigin-RevId: 652947042
5 months ago
namespace google {
namespace protobuf {
namespace rust {
// String and bytes values are passed across the FFI boundary as owned raw
// pointers when we do map insertions. Unlike other types, they have to be
// explicitly deleted. This MakeCleanup() helper does nothing by default, but
// for std::string pointers it returns a std::unique_ptr to take ownership of
// the raw pointer.
template <typename T>
Protobuf Rust: simplify the MakeCleanup() helper function By changing the return type to `auto`, we can handle `std::string` and other types in a single definition without needing a separate overload. PiperOrigin-RevId: 653272253
5 months ago
auto MakeCleanup(T value) {
if constexpr (std::is_same<T, std::string*>::value) {
return std::unique_ptr<std::string>(value);
} else {
return 0;
}
Rust protobuf: fix memory leaks Our ASAN test runs have not had the heap checker enabled, so this has allowed a few memory leaks to slip in. This CL fixes all of them so that we can turn on the heap checker. The first one takes place whenever we add an entry into a string-valued map using the C++ kernel. The problem is that `InnerProtoString::into_raw()` gives up ownership of the raw `std::string` pointer it holds, but then we never delete that pointer. This CL fixes the problem by deleting the pointer in C++ right after we perform the map insertion. To simplify things, I created a `MakeCleanup()` helper function that we always call in our map insertion thunks, but it's a no-op in the cases where we don't need to free anything. There were a couple similar memory leaks related to repeated field accessors in the C++ kernel, and those were simple to fix just by adding the necessary `delete` call. Finally, there were two benign memory leaks in the upb kernel involving global variables used for empty repeated fields and maps. It turned out that we did not need to use `Box` at all here, so removing that simplified things and fixed the leaks. PiperOrigin-RevId: 652947042
5 months ago
}
} // namespace rust
} // namespace protobuf
} // namespace google
Transpose the map thunk macros, relocate to header file PiperOrigin-RevId: 609766889
9 months ago
// Defines concrete thunks to access typed map methods from Rust.
Migrate Repeated::{push, set} and Map::insert to use the IntoProxied trait. * The public Repeated::{push, set} and Map::insert methods now accept any value that implements IntoProxied<T>, allowing us to move owned values instead of copying them. * This change also updates the FFI layer for strings/bytes in the repeated and maps thunks to accept a std::string* that can be moved rather than a PtrAndLen type that needs to be copied. * Tests are updated to no longer .as_view() when setting a message / string on a repeated / map field. The IntoProxied trait makes calling .as_view() obsolete. PiperOrigin-RevId: 650580788
5 months ago
#define __PB_RUST_EXPOSE_SCALAR_MAP_METHODS( \
key_ty, rust_key_ty, ffi_key_ty, to_cpp_key, to_ffi_key, value_ty, \
rust_value_ty, ffi_view_ty, ffi_value_ty, to_cpp_value, to_ffi_value) \
google::protobuf::Map<key_ty, value_ty>* \
proto2_rust_thunk_Map_##rust_key_ty##_##rust_value_ty##_new() { \
return new google::protobuf::Map<key_ty, value_ty>(); \
} \
void proto2_rust_thunk_Map_##rust_key_ty##_##rust_value_ty##_free( \
google::protobuf::Map<key_ty, value_ty>* m) { \
delete m; \
} \
void proto2_rust_thunk_Map_##rust_key_ty##_##rust_value_ty##_clear( \
google::protobuf::Map<key_ty, value_ty>* m) { \
m->clear(); \
} \
size_t proto2_rust_thunk_Map_##rust_key_ty##_##rust_value_ty##_size( \
const google::protobuf::Map<key_ty, value_ty>* m) { \
return m->size(); \
} \
bool proto2_rust_thunk_Map_##rust_key_ty##_##rust_value_ty##_insert( \
google::protobuf::Map<key_ty, value_ty>* m, ffi_key_ty key, ffi_value_ty value) { \
Rust protobuf: fix memory leaks Our ASAN test runs have not had the heap checker enabled, so this has allowed a few memory leaks to slip in. This CL fixes all of them so that we can turn on the heap checker. The first one takes place whenever we add an entry into a string-valued map using the C++ kernel. The problem is that `InnerProtoString::into_raw()` gives up ownership of the raw `std::string` pointer it holds, but then we never delete that pointer. This CL fixes the problem by deleting the pointer in C++ right after we perform the map insertion. To simplify things, I created a `MakeCleanup()` helper function that we always call in our map insertion thunks, but it's a no-op in the cases where we don't need to free anything. There were a couple similar memory leaks related to repeated field accessors in the C++ kernel, and those were simple to fix just by adding the necessary `delete` call. Finally, there were two benign memory leaks in the upb kernel involving global variables used for empty repeated fields and maps. It turned out that we did not need to use `Box` at all here, so removing that simplified things and fixed the leaks. PiperOrigin-RevId: 652947042
5 months ago
auto cleanup = google::protobuf::rust::MakeCleanup(value); \
(void)cleanup; \
Migrate Repeated::{push, set} and Map::insert to use the IntoProxied trait. * The public Repeated::{push, set} and Map::insert methods now accept any value that implements IntoProxied<T>, allowing us to move owned values instead of copying them. * This change also updates the FFI layer for strings/bytes in the repeated and maps thunks to accept a std::string* that can be moved rather than a PtrAndLen type that needs to be copied. * Tests are updated to no longer .as_view() when setting a message / string on a repeated / map field. The IntoProxied trait makes calling .as_view() obsolete. PiperOrigin-RevId: 650580788
5 months ago
auto iter_and_inserted = m->try_emplace(to_cpp_key, to_cpp_value); \
if (!iter_and_inserted.second) { \
iter_and_inserted.first->second = to_cpp_value; \
} \
return iter_and_inserted.second; \
} \
bool proto2_rust_thunk_Map_##rust_key_ty##_##rust_value_ty##_get( \
const google::protobuf::Map<key_ty, value_ty>* m, ffi_key_ty key, \
ffi_view_ty* value) { \
auto cpp_key = to_cpp_key; \
auto it = m->find(cpp_key); \
if (it == m->end()) { \
return false; \
} \
auto& cpp_value = it->second; \
*value = to_ffi_value; \
return true; \
} \
google::protobuf::internal::UntypedMapIterator \
proto2_rust_thunk_Map_##rust_key_ty##_##rust_value_ty##_iter( \
const google::protobuf::Map<key_ty, value_ty>* m) { \
return google::protobuf::internal::UntypedMapIterator::FromTyped(m->cbegin()); \
} \
void proto2_rust_thunk_Map_##rust_key_ty##_##rust_value_ty##_iter_get( \
Rust: cut down on the amount of generated C++ code needed for maps With the C++ kernel for Rust, we currently need to generate quite a few C++ thunks for operations on map fields. For each message we generate, we generate these thunks for all possible map types that could have that message as a value. These operations are for things such as insertion, removal, clearing, iterating, etc. The reason we do this is that templated types don't play well with FFI, so we effectively need separate FFI endpoints for every possible combination of key and value types used (or even potentially used) as a map field. This CL fixes the problem by replacing the generated thunks with functions in the runtime that can operate on `proto2::MessageLite*` without needing to care about the specific message type. The way it works is that we implement the operations using either `UntypedMapBase` (the base class of all map types, which knows nothing about the key and value types) or `KeyMapBase`, which knows the key type but not the value type. I roughly followed the example of the table-driven parser, which has a similar problem of needing to operate generically on maps without having access to the concrete types. I removed 54 thunks per message (that's 6 key types times 9 operations per key), but had to add two new thunks per message: - The `size_info` thunk looks up the `MapNodeSizeInfoT`, which is stored in a small constant table. The important thing here is an offset indicating where to look for the value in each map entry. This offset can be different for every pair of key and value types, but we can safely assume that the result does not depend on the signedness of the key. As a result we only need to store four entries per message: one each for i32, i64, bool, and string. - The `placement_new` thunk move-constructs a message in place. We need this to be able to efficiently implement map insertion. There are two big things that this CL does not address yet but which I plan to follow up on: - Enums still generate many map-related C++ thunks that could be replaced with a common implementation. This should actually be much easier to handle than messages, because every enum has the same representation as an i32. - We still generate six `ProxiedInMapValue` implementations for every message, but it should be possible to replace these with a blanket implementation that works for all message types. PiperOrigin-RevId: 657681421
4 months ago
const google::protobuf::internal::UntypedMapIterator* iter, int32_t, \
ffi_key_ty* key, ffi_view_ty* value) { \
Migrate Repeated::{push, set} and Map::insert to use the IntoProxied trait. * The public Repeated::{push, set} and Map::insert methods now accept any value that implements IntoProxied<T>, allowing us to move owned values instead of copying them. * This change also updates the FFI layer for strings/bytes in the repeated and maps thunks to accept a std::string* that can be moved rather than a PtrAndLen type that needs to be copied. * Tests are updated to no longer .as_view() when setting a message / string on a repeated / map field. The IntoProxied trait makes calling .as_view() obsolete. PiperOrigin-RevId: 650580788
5 months ago
auto typed_iter = \
iter->ToTyped<google::protobuf::Map<key_ty, value_ty>::const_iterator>(); \
const auto& cpp_key = typed_iter->first; \
const auto& cpp_value = typed_iter->second; \
*key = to_ffi_key; \
*value = to_ffi_value; \
} \
bool proto2_rust_thunk_Map_##rust_key_ty##_##rust_value_ty##_remove( \
google::protobuf::Map<key_ty, value_ty>* m, ffi_key_ty key, ffi_view_ty* value) { \
auto cpp_key = to_cpp_key; \
auto num_removed = m->erase(cpp_key); \
return num_removed > 0; \
Transpose the map thunk macros, relocate to header file PiperOrigin-RevId: 609766889
9 months ago
}
// Defines the map thunks for all supported key types for a given value type.
Migrate Repeated::{push, set} and Map::insert to use the IntoProxied trait. * The public Repeated::{push, set} and Map::insert methods now accept any value that implements IntoProxied<T>, allowing us to move owned values instead of copying them. * This change also updates the FFI layer for strings/bytes in the repeated and maps thunks to accept a std::string* that can be moved rather than a PtrAndLen type that needs to be copied. * Tests are updated to no longer .as_view() when setting a message / string on a repeated / map field. The IntoProxied trait makes calling .as_view() obsolete. PiperOrigin-RevId: 650580788
5 months ago
#define __PB_RUST_EXPOSE_SCALAR_MAP_METHODS_FOR_VALUE_TYPE( \
value_ty, rust_value_ty, ffi_view_ty, ffi_value_ty, to_cpp_value, \
to_ffi_value) \
__PB_RUST_EXPOSE_SCALAR_MAP_METHODS( \
int32_t, i32, int32_t, key, cpp_key, value_ty, rust_value_ty, \
ffi_view_ty, ffi_value_ty, to_cpp_value, to_ffi_value); \
__PB_RUST_EXPOSE_SCALAR_MAP_METHODS( \
uint32_t, u32, uint32_t, key, cpp_key, value_ty, rust_value_ty, \
ffi_view_ty, ffi_value_ty, to_cpp_value, to_ffi_value); \
__PB_RUST_EXPOSE_SCALAR_MAP_METHODS( \
bool, bool, bool, key, cpp_key, value_ty, rust_value_ty, ffi_view_ty, \
ffi_value_ty, to_cpp_value, to_ffi_value); \
__PB_RUST_EXPOSE_SCALAR_MAP_METHODS( \
uint64_t, u64, uint64_t, key, cpp_key, value_ty, rust_value_ty, \
ffi_view_ty, ffi_value_ty, to_cpp_value, to_ffi_value); \
__PB_RUST_EXPOSE_SCALAR_MAP_METHODS( \
int64_t, i64, int64_t, key, cpp_key, value_ty, rust_value_ty, \
ffi_view_ty, ffi_value_ty, to_cpp_value, to_ffi_value); \
__PB_RUST_EXPOSE_SCALAR_MAP_METHODS( \
std::string, ProtoString, google::protobuf::rust::PtrAndLen, \
std::string(key.ptr, key.len), \
Move -Werror to our test/dev bazelrc files. Putting it into BUILD files unintentionally forces it on all our downstream users. Instead, we just want to enable this during testing and let them choose for themselves in their builds. Note, that this expands the scope of -Werror to our entire repo for CI, so a bunch of fixes and opt-outs had to be applied to get this change passing. Closed #14714 PiperOrigin-RevId: 666903224
3 months ago
(google::protobuf::rust::PtrAndLen{cpp_key.data(), cpp_key.size()}), value_ty, \
Migrate Repeated::{push, set} and Map::insert to use the IntoProxied trait. * The public Repeated::{push, set} and Map::insert methods now accept any value that implements IntoProxied<T>, allowing us to move owned values instead of copying them. * This change also updates the FFI layer for strings/bytes in the repeated and maps thunks to accept a std::string* that can be moved rather than a PtrAndLen type that needs to be copied. * Tests are updated to no longer .as_view() when setting a message / string on a repeated / map field. The IntoProxied trait makes calling .as_view() obsolete. PiperOrigin-RevId: 650580788
5 months ago
rust_value_ty, ffi_view_ty, ffi_value_ty, to_cpp_value, to_ffi_value);
Transpose the map thunk macros, relocate to header file PiperOrigin-RevId: 609766889
9 months ago
Split up cpp_api.h/.cc into smaller units. PiperOrigin-RevId: 647663342
5 months ago
#endif // GOOGLE_PROTOBUF_RUST_CPP_KERNEL_MAP_H__