The previous code did not work because timedelta arithmetic does not do timezone conversions. The result of adding a timedelta to a datetime has the same fixed UTC offset as the original datetime. This resulted in the correct timezone not being applied by `Timestamp.ToDatetime(tz)` whenever the UTC offset for the timezone at the represented moment was not the same as the UTC offset for that timezone at the epoch.
Instead, construct the datetime directly from the seconds part of the timestamp. It would be nice to include the nanoseconds as well (truncated to datetime's millisecond precision, but without unnecessary loss of precision). However, that doesn't work, since there isn't a way to construct a datetime from an integer-milliseconds timestamp, just float-seconds, which doesn't allow some datetime values to round-trip correctly. (This does assume that `tzinfo.utcoffset(dt).microseconds` is always zero, and that the value returned by `tzinfo.utcoffset(dt)` doesn't change mid-second. Neither of these is necessarily the case (see https://github.com/python/cpython/issues/49538), though I'd hope they hold in practice.)
This does take some care to still handle non-standard Timestamps where nanos is more than 1,000,000,000 (i.e. more than a second), since previously ToDatetime handled that, as do the other To* conversion methods.
The bug doesn't manifest for UTC (or any fixed-offset timezone), so it can be worked around in a way that will be correct before and after the fix by replacing `ts.ToDatetime(tz)` with `ts.ToDatetime(datetime.timezone.utc).astimezone(tz)`.
PiperOrigin-RevId: 569012931
This change moves almost everything in the `upb/` directory up one level, so
that for example `upb/upb/generated_code_support.h` becomes just
`upb/generated_code_support.h`. The only exceptions I made to this were that I
left `upb/cmake` and `upb/BUILD` where they are, mostly because that avoids
conflict with other files and the current locations seem reasonable for now.
The `python/` directory is a little bit of a challenge because we had to merge
the existing directory there with `upb/python/`. I made `upb/python/BUILD` into
the BUILD file for the merged directory, and it effectively loads the contents
of the other BUILD file via `python/build_targets.bzl`, but I plan to clean
this up soon.
PiperOrigin-RevId: 568651768
This restores the Python wheel CI runs from the old upb repo with only minor
changes. I had to update a path in one of the `py_wheel` rules and also make a
slight tweak to ensure that the `descriptor.upb_minitable.{h,c}` files make it
into the source wheels. The change in text_format_test.py is not strictly
necessary but is a small simplification I made while I was trying to debug an
issue with CRLF newlines.
I had to update test_util.py to use `importlib` to access the golden files from
the installed `protobuftests` package. I suspect the previous incarnation of
thse test runs was somehow reading the goldens from the repo checkout, but I
think the intention is to read them from `protobuftests` instead. This was a
bit tricky to get working because Python versions before 3.9 do not support
`importlib.resources.files()`. I set up the code to fall back on
`importlib.resources.open_binary()` in that case, but that function does not
support subdirectories, so this required putting an `__init__.py` file inside
the `testdata` directory to make sure it is treated as a Python package.
PiperOrigin-RevId: 567366695
I am getting ready to move almost everything under the upb/ directory up one
level to integrate upb better into its new location in the protobuf repo. This
change makes a few tweaks to prepare for that:
- Delete upb's LICENSE and CONTRIBUTING.md files since we already have similar
files at the top level.
- Rename `//python:python_version` so that it won't conflict later with
`//upb/python:python_version`.
- Move the contents of python/BUILD.bazel out to a Bazel macro to facilitate
merging that BUILD.bazel file with upb/python/BUILD.
PiperOrigin-RevId: 567119840
A couple weeks ago we moved upb into the protobuf Git repo, and this change
continues the merger of the two repos by making them into a single Bazel repo.
This was mostly a matter of deleting upb's WORKSPACE file and fixing up a bunch
of references to reflect the new structure.
Most of the changes are pretty mechanical, but one thing that needed more
invasive changes was the Python script for generating CMakeLists.txt,
make_cmakelists.py. The WORKSPACE file it relied on no longer exists with this
change, so I updated it to hardcode the information it needed from that file.
PiperOrigin-RevId: 564810016
The Python comparison protocol requires that if an object doesn't know how to
compare itself to an object of a different type, it returns NotImplemented
rather than False. The interpreter will then try performing the comparison using
the other operand. This translates, for protos, to:
If a proto message doesn't know how to compare itself to an object of
non-message type, it returns NotImplemented. This way, the interpreter will then
try performing the comparison using the comparison methods of the other object,
which may know how to compare itself to a message. If not, then Python will
return the combined result (e.g., if both objects don't know how to perform
__eq__, then the equality operator `==` return false).
This change allows one to compare a proto with custom matchers such as mock.ANY
that the message doesn't know how to compare to, regardless of whether
mock.ANY is on the right-hand side or left-hand side of the equality (prior to
this change, it only worked with mock.ANY on the left-hand side).
Fixes https://github.com/protocolbuffers/protobuf/issues/9173
PiperOrigin-RevId: 561728156
GetOptions on fields (which parse the _serialized_options) will be called for the first time of parse or serialize instead of Build time.
Note: GetOptions on messages are still called in Build time because of message_set_wire_format. If message options are needed in descriptor.proto, a parse error will be raised in GetOptions(). We can check the file to not invoke GetOptions() for descriptor.proto as long as message_set_wire_format not needed in descriptor.proto.
Other options except message options do not invoke GetOptions() in Build time
PiperOrigin-RevId: 560741182
This fixes Python/C++ and upb, and pushes the buggy behavior to pure python. There, it's very difficult to handle options on the bootstrapped proto with the current architecture. Future changes will attempt to address this more isolated issue.
PiperOrigin-RevId: 559450900
The Python comparison protocol requires that if an object doesn't know how to
compare itself to an object of a different type, it returns NotImplemented
rather than False. The interpreter will then try performing the comparison using
the other operand. This translates, for protos, to:
If a proto message doesn't know how to compare itself to an object of
non-message type, it returns NotImplemented. This way, the interpreter will then
try performing the comparison using the comparison methods of the other object,
which may know how to compare itself to a message. If not, then Python will
return the combined result (e.g., if both objects don't know how to perform
__eq__, then the equality operator `==` return false).
This change allows one to compare a proto with custom matchers such as mock.ANY
that the message doesn't know how to compare to, regardless of whether
mock.ANY is on the right-hand side or left-hand side of the equality (prior to
this change, it only worked with mock.ANY on the left-hand side).
See https://github.com/protocolbuffers/protobuf/issues/9173
PiperOrigin-RevId: 559056804
Remove array.h and map.h as hdrs from :collections_internal
Remove alloc.h and arena.h as hdrs from :mem_internal (and add them to :mem)
Remove common.h and decode.h and encode.h as hdrs from :wire_internal
Lock down the visibility of :wire_internal to upb-only
Merge :mini_descriptor_encode_internal into :mini_descriptor_internal
PiperOrigin-RevId: 558235138
This proto is only used in C++ tests, and doesn't need to be built in python. Python doesn't support editions yet, so we should just remove it for now. This also fixes some other issues with setup.py where tests fail.
Closes#13501
PiperOrigin-RevId: 555681295
When build a FileDescriptorProto into pool, FieldDescriptorProto.type may not set if type_name is set. This only add tests. UPB python Fix has been submit in other change
PiperOrigin-RevId: 554516745
The file structure was broken only when invoking from protobuf repo (`bazel build @upb//python/dist:source_tarball`) since filegroup files have a path like `protobuf/external/upb/python/dist/setup.py` rather than `protobuf/setup.py`
This was broken by 79af13abde
PiperOrigin-RevId: 553821217
This flag was added to Protobuf and submitted separately due to version skew between repos.
This attempts to fix the following error from local and release settings both matching.
```
ERROR: /workspace/_build/out/external/upb/python/dist/BUILD.bazel:251:9: Illegal ambiguous match on configurable attribute "platform" in @upb//python/dist:binary_wheel:
@upb//python/dist:linux_x86_64_local
@upb//python/dist:windows_x86_64
Multiple matches are not allowed unless one is unambiguously more specialized or they resolve to the same value. See https://bazel.build/reference/be/functions#select.
```
PiperOrigin-RevId: 553226261
The next in a series of CLs to split upb/BUILD into subdirs.
Create mem/internal/
Delete the deprecated upb/arena.h and upb/alloc.h stub headers
PiperOrigin-RevId: 552864952
Mike Kruskal
Samuel Freilich
Adam Cozzette
Jie Luo
Sandy Zhang
Protobuf Team Bot
Joshua Haberman
Eric Salo
ericsalo
Protobuf Team Bot