Users often encounter a Python import error when trying to build Python
protos if protobuf is installed locally on the machine. In this case,
Python ends up looking in the wrong directory when importing files (see
bazelbuild/bazel#1209 and tensorflow/tensorflow#2021). It seems that the
problem is caused by Python getting confused when there are Python
source files that are meant to be part of the same package but are
in separate directories.
Prior to #1233, the Bazel build setup would copy the Python
runtime sources and all generated sources for the builtin protos into
the root directory (assuming that the protobuf tree is vendored in a
google/protobuf directory).
With #1233, the two sets of sources are kept in their respective
directories but both `src/` and `python/` are added to the `PYTHONPATH`
using the new `imports` attribute of the Bazel Python rules. However,
both the runtime sources and the generated sources are under the same
package: `google.protobuf`, causing Python to become confused when
trying to import modules that are in the other directory.
This patch adds a workaround to the Bazel build to add a modified
version of the original `internal_copied_filegroup` macro to copy the
`.proto` files under `src/` to `python/` before building the
`py_proto_library` targets for the builtin protos. This ensures that the
generated sources for the builtin protos will be in the same directory
as the corresponding runtime sources.
This patch was tested with the following:
* All Python tests in protobuf
* All Python tests in tensorflow
* All tests in [Skydoc](https://github.com/bazelbuild/skydoc)
* Importing protobuf as `//google/protobuf`
* Importing and binding targets under `//external`
* Importing protobuf as `//third_party/protobuf`
proto3.
Added a couple unit test proto3 files, for arena_lite and lite. Cloned
the proto3_arena_unittest to test some of the basics of generated code
(and to ensure that the generated proto3 test files are used by some
test).
This is required to allow Tensorflow to build on Android without hacks. Currently we create a dummy pthread library just to satisfy this dependency for a library that does not exist on Android. See https://github.com/google/protobuf/issues/1373 for more context.
This change makes use of new imports attribute for Bazel's Python rules, which
enable adding directories to the PYTHONPATH. This allows us to remove
the hack for building protobuf's Python support with Bazel and now
allows projects to include protobuf using a Bazel external repository
rather than requiring it to be imported directly into the source tree as
//google/protobuf.
This change also updates the protobuf BUILD file to use a named
repository, @python//, for including Python headers rather than
//util/python. This allows projects to specify their own package for
Python headers when including protobuf with an external repository.
Fixes#1230
--allow_oversize_protos=true is passed to bazel build.
When this macro is set, SetTotalBytesLimit is called to remove
the 64MB limit on binary protos when during ParseFromString.
When trying to compile the protobuf code as a DLL, and then compile other DLLs
with generated .pb.cc/h files that reference
InternalMetadataWithArena::InternalMetadataWithArena(Arena*), MSVC gives an
"unresolved external symbol" error. This seems to be due to the function being
simultaneously exported and inline. Moving it out-of-line fixes things.
There are other functions exported and inline as well but de-inlining them
doesn't seem to be necessary to get the build working, and I'd rather de-inline
as few functions as possible.
David Z. Chen
Jisi Liu
Chad Whipkey
Andrew Harp
Steven Parkes
Pete Warden
Manjunath Kudlur
Ming Zhao
Feng Xiao
Geoffrey Irving
Jon Skeet
Lukacs T. Berki
Peter Kasting
Bo Yang
Jorge Canizales