This is just part of an effort at locking down (and understand well) how map
entries are serialized into textproto when they have empty fields.
Right now the C++ behaviour for DynamicMessages is that for implicit-presence
(i.e. "proto3") fields, the empty MapEntry fields are not serialized even if
explicitly set. For example, `value: ""` would not show up in textproto
serialization for proto3 MapEntry messages. This contrasts with C++ map
reflection behaviour, because C++ MapEntry messages are always generated with
hasbits (even if they are declared in a proto3 file and have implicit presence
according to their descriptors).
For this Python test, the DynamicMessage fallback path was triggered because
the implementation of the C++ proto was not found. When python is lacking in
implementation, it calls DynamicMessage which respects field presence, even for
map entries. If the corresponding `map_unittest_cc_proto` is linked, this test
actually fails, because the C++ MapEntry implementation, which is always
generated with hasbits even for implicit presence, is used.
PiperOrigin-RevId: 681944020
This should rename the self parameter name to "self_" if any field is named "self", not rename the corresponding keyword parameter name to "self_" if the first field is named "self".
PiperOrigin-RevId: 681872250
This is a very narrow edge case where touching a packed extension via generated APIs first, and then doing so reflectively will trigger a DCHECK. Otherwise, reflective APIs will work but not use packed encoding for the extension. This was likely a pre-existing bug dating back to proto3, where it would only be visible on custom options (the only extensions allowed in proto3).
To help qualify this and uncover similar issues, unittest.proto was migrated to editions. This turned up some other minor issues in DebugString and python.
PiperOrigin-RevId: 675785611
In the worst case, the source object is a constant (eg a global default instance) and modifying them has undefined behavior.
PiperOrigin-RevId: 673402981
This enables enforcement of lifetime specifications on individual enum values for features. It will allow us to add new values to existing features, as well as deprecate/and remove existing values. By default, each value will be scoped to the lifetime spec of its corresponding feature field. However, individual lifetime boundaries can be overridden at the value-level for finer grained control.
In the near-term, this will allow us to deprecate/remove required field presence, and add a stricter utf8 validation feature.
PiperOrigin-RevId: 672710484
These are very old unit-tests that appear to have been written to provide some conformance and fuzzing capabilities before we had more comprehensive solutions. Today, they should be already covered by our conformance tests and fuzz tests and create unnecessary noise.
The goal of this change is to remove the checked-in binary golden data that we no longer know how to regenerate. There is a lot of nearby code that could likely also be cleaned up in a follow-up
PiperOrigin-RevId: 667604310
Pure python and upb do not support it and filtered out the test. This API does
not exists in any other language(except protobuf c++).
GetDebugString() for cpp extension will be removed in Jan 2025
PiperOrigin-RevId: 662640110
-Raise warnings for deprecated google/protobuf/service.py APIs.
service.py APIs are marked as deprecated since 2010. These APIs will be
removed in Jan 2025
PiperOrigin-RevId: 653280370
Python dict is now able to be assigned (by create and copy, not reference) and compared with the Protobuf Struct field.
Python list is now able to be assigned (by create and copy, not reference) and compared with the Protobuf ListValue field.
example usage:
dictionary = {'key1': 5.0, 'key2': {'subkey': 11.0, 'k': False},}
list_value = [6, 'seven', True, False, None, dictionary]
msg = more_messages_pb2.WKTMessage(
optional_struct=dictionary, optional_list_value=list_value
)
self.assertEqual(msg.optional_struct, dictionary)
self.assertEqual(msg.optional_list_value, list_value)
PiperOrigin-RevId: 646099987
This was previously fixed in C++ (https://github.com/protocolbuffers/protobuf/issues/16549), but not ported to other languages. Delimited field encoding can be inherited by fields where it's invalid, such as non-messages and maps. In these cases, the encoding should be ignored and length-prefixed should be used.
PiperOrigin-RevId: 642792988
There is a special case where message factories can be confused: if a module
written in C++ with pybind11 links against a self-recursive message, and that
message is part of another message loaded from Python, then the confusion
will happen.
Example:
# This one is also linked into the C++ module.
message SelfRecursive {
optional SelfRecursive self_recursive = 1;
}
# This one is used only in Python and not linked.
message OnlyUsedInPython {
optional SelfRecursive self_recursive = 2;
}
The caching through message_factory::RegisterMessageClass then happens on one
instance of the factory, but traversal with the lookup in another.
This occurs in the pure Python and upb implementations that have their own
default descriptor pools (and thus message factory).
Fix this by using the already passed message factory to registering the
message class to cache.
A test accounts for this case to avoid regressions.
PiperOrigin-RevId: 642551744
Timestamp and Duration are now have more support with datetime and timedelta:
- Allows assign python datetime to protobuf DateTime field in addition to current FromDatetime/ToDatetime (Note: will throw exceptions for the differences in supported ranges)
- Allows assign python timedelta to protobuf Duration field in addition to current FromTimedelta/ToTimedelta
- Calculation between Timestamp, Duration, datetime and timedelta will also be supported.
example usage:
from datetime import datetime, timedelta
from event_pb2 import Event
e = Event(start_time=datetime(year=2112, month=2, day=3),
duration=timedelta(hours=10))
duration = timedelta(hours=10))
end_time = e.start_time + timedelta(hours=4)
e.duration = end_time - e.start_time
PiperOrigin-RevId: 640639168
Added the following methods:
serialize_length_prefixed(message: Message, output: io.BytesIO) -> None
parse_length_prefixed(message_class: Type[Message], input_bytes: io.BytesIO) -> Message
The output of serialize_length_prefixed should be BytesIO or custom buffered IO that data should be written to. The output stream must be buffered, e.g. using
https://docs.python.org/3/library/io.html#buffered-streams.
PiperOrigin-RevId: 638375900
`DynamicCastMessage`/`DownCastMessage`.
The target does not necessarily need to be a generated type. For example, it
also supports `Message` itself. This makes the API friendlier to generic code and less verbose.
Replace all uses of dynamic_cast/down_cast/**ToGenerated with the new names.
Also, remove checks for RTTI in tests where we only need the casts to work. They don't need RTTI anymore.
PiperOrigin-RevId: 638278948
The second assert in _upb_EncodeRoundTripFloat is raised if val is a nan. This fix just returns the output of first spnprintf.
I am not sure how changes to this repo are made so feel free to ignore this CL.
To test this, you could
1. Define a proto with a float field
message Test {
float val = 1;
}
2. In a python script, import the library and then set the val to nan and try to print it.
proto = Test(val=float('nan'))
print(proto)
This will cause a coredump due to assertion error:
assert.h assertion failed at third_party/upb/upb/lex/round_trip.c:46 in void _upb_EncodeRoundTripFloat(float, char *, size_t): strtof(buf, NULL) == val
Added the corresponding change to double too
PiperOrigin-RevId: 637127851
- add google.protobuf.proto module
- wrap generated SerializeToString and ParseFromString to the new module:
def serialize(message: Message, deterministic: bool=None) -> bytes:
"""Return the serialized proto."""
def parse(message_class: typing.Type[Message], payload: bytes) -> Message:
"""Given a serialized proto, deserialize it into a Message."""
PiperOrigin-RevId: 632223409
(Second attempt. The first attempt missed ListValue)
The “in” operator will be consistent with HasField but a little different with Proto Plus.
The detail behavior of “in” operator in Nextgen
* For WKT Struct (to be consist with old Struct behavior):
-Raise TypeError if not pass a string
-Check if the key is in the struct.fields
* For WKT ListValue (to be consist with old behavior):
-Check if the key is in the list_value.values
* For other messages:
-Raise ValueError if not pass a string
-Raise ValueError if the string is not a field
-For Oneof: Check any field under the oneof is set
-For has-presence field: check if set
-For non-has-presence field (include repeated fields): raise ValueError
PiperOrigin-RevId: 631143378
Jie Luo
Protobuf Team Bot
Tony Liao
Samuel Freilich
Mike Kruskal
Nathan Baulch
Mike Kruskal
zhangskz
jensbjorgensen
Kyle Montemayor