protobuf

Commit Graph

Author	SHA1	Message	Date
Adam Cozzette	403df1d047	Added a descriptor option number for ocaml-protoc-plugin	5 years ago
Sigurd Meldgaard	f4a3e06101	Add option for dart	5 years ago
Ben Gordon	f5362e11fd	Requesting extension id 1072 This library supports an idiomatic proto3 protobuf generator for kotlin. The library will be open sourced by Toast Inc under the Apache2 license, and is currently used in production at Toast. The following is the readme.md that will be released with the code by the end of Q4 2019. Supports only the Protocol Buffers language version 3. #### Features - Clean data class generation - Oneof types handled as sealed classes - JavaDoc comments on generated code - Deprecation option pass-through to Kotlin's `@Deprecated` annotation - Protokt-specific options: non-null types, wrapper types, interface implementation, and more - Tight integration with Protobuf's Java library: compatibility with its well-known types and usage of CodedInputStream and CodedOutputStream for best performance #### Not yet implemented - Kotlin native support - Kotlin JS support - Support for gRPC service generation - Protobuf JSON support See examples in [protokt-testing](https://github.com/toasttab/protokt/tree/master/protokt-testing). ### Generated Code Generated code is placed in `<buildDir>/generated-sources/main/protokt`. A simple example: ```proto syntax = "proto3"; package com.protokt.sample; message Sample { string sample_field = 1; } ``` will produce: ```kotlin /* * Generated by protokt. Do not modify. / package com.protokt.sample import com.toasttab.protokt.rt. data class Sample( val sampleField: String, val unknown: Map<Int, Unknown> = emptyMap() ) : KtMessage { @Suppress("UNUSED") constructor( sampleField: String = "" ) : this( sampleField, emptyMap() ) override val messageSize by lazy { sizeof() } override fun serialize(serializer: KtMessageSerializer) { if (sampleField.isNotEmpty()) { serializer.write(Tag(10)).write(sampleField) } if (unknown.isNotEmpty()) { serializer.writeUnknown(unknown) } } private fun sizeof(): Int { var res = 0 if (sampleField.isNotEmpty()) { res += sizeof(Tag(1)) + sizeof(sampleField) } res += unknown.entries.sumBy { it.value.sizeof() } return res } companion object Deserializer : KtDeserializer<Sample> { override fun deserialize(deserializer: KtMessageDeserializer): Sample { var sampleField = "" val unknown = mutableMapOf<Int, Unknown>() while (true) { when (deserializer.readTag()) { 0 -> return Sample( sampleField, unknown ) 10 -> sampleField = deserializer.readString() else -> { val unk = deserializer.readUnknown() unknown[unk.fieldNum] = unknown[unk.fieldNum].let { when (it) { null -> unk else -> when (val v = it.value) { is ListVal -> Unknown(unk.fieldNum, ListVal(v.value + unk.value)) else -> Unknown(unk.fieldNum, ListVal(listOf(v, unk.value))) } } } } } } } } } ``` #### Runtime Notes ##### Package The Kotlin package of a generated file can be overridden from protobuf package with the `(protokt).package` option: ```proto syntax = "proto3"; import "protokt.proto"; package com.example; option (protokt).package = "com.package"; ``` ##### Message Each protokt message implements the `KtMessage` interface. `KtMessage` defines the `serialize()` method and its overloads which can serialize to a byte array, a `KtMessageSerializer`, or on the JVM, an `OutputStream`. Each protokt message has a companion object `Deserializer` that implements the `KtDeserializer` interface, which provides the `deserialize()` method and its overloads to construct an instance of the message from a byte array, a Java InputStream, or others. In order to enjoy the full benefits of Kotlin data classes, byte arrays are wrapped in the protokt `Bytes` class, which provides appropriate `equals()` and `hashCode()` implementations. ##### Enums Enum fields are generated as data classes with a single integer field. Kotlin enum classes are closed and cannot retain unknown values, and protobuf requires that unknown enum values are preserved for reserialization. This compromise exposes a constructor taking an integer, but the `from(value: Int)` on an enum's `Deserializer` should be preferred as it avoids instantiation when possible. Other notes: - `optimize_for` is ignored. - `repeated` fields are deserialized to Lists. - `map` fields are deserialized to Maps. - `oneof` fields are represented as data class subtypes of a sealed base class with a single property. ### Extensions See examples of each option in the [protokt-options](https://github.com/toasttab/protokt/tree/master/protokt-testing/protokt-options/src/main/proto) module. All protokt-specific options require importing `protokt.proto` in the protocol file. #### Wrapper Types Sometimes a field on a protobuf message corresponds to a concrete nonprimitive type. In standard protobuf the user would be responsible for this extra transformation, but the protokt wrapper type option allows specification of a converter that will automatically encode and decode custom types to protobuf primitives and well-known types. Some standard types are implemented in [protokt-extensions](https://github.com/toasttab/protokt/tree/master/protokt-extensions/src/main/kotlin/com/toasttab/protokt/ext). Wrap a field by invoking the `(protokt_property).wrap` option: ```proto message DateWrapperMessage { int64 date = 1 [ (protokt_property).wrap = "java.util.Date" ]; } ``` Converters implement the `Converter` interface: ```kotlin interface Converter<S: Any, T: Any> { val wrapper: KClass<S> fun wrap(unwrapped: T): S fun unwrap(wrapped: S): T } ``` and protokt will reference the converter's methods to wrap and unwrap from protobuf primitives: ```kotlin object DateConverter : Converter<Date, Long> { override val wrapper = Date::class override fun wrap(unwrapped: Long) = Date(unwrapped) override fun unwrap(wrapped: Date) = wrapped.time } ``` ```kotlin data class WrapperModel( val date: java.util.Date, ... ) : KtMessage { ... override fun serialize(serializer: KtMessageSerializer) { serializer.write(Tag(10)).write(Int64(DateConverter.unwrap(date))) ... } override fun deserialize(deserializer: KtMessageDeserializer): WrapperModel { var date = 0L while (true) { when (deserializer.readTag()) { 0 -> return WrapperModel( DateConverter.wrap(date), ... ) ... } } } } ``` Converters can also implement the `OptimizedSizeofConverter` interface adding `sizeof()`, which allows them to optimize the calculation of the wrapper's size rather than unwrap the object twice. For example, a UUID is always 16 bytes: ```kotlin object UuidConverter : OptimizedSizeofConverter<UUID, ByteArray> { override val wrapper = UUID::class private val sizeofProxy = ByteArray(16) override fun sizeof(wrapped: UUID) = sizeof(sizeofProxy) override fun wrap(unwrapped: ByteArray): UUID { require(unwrapped.size == 16) { "input must have size 16; had ${unwrapped.size}" } return ByteBuffer.wrap(unwrapped) .run { UUID(long, long) } } override fun unwrap(wrapped: UUID) = ByteBuffer.allocate(16) .putLong(wrapped.mostSignificantBits) .putLong(wrapped.leastSignificantBits) .array() } ``` Rather than convert a UUID to a byte array both for size calculation and for serialization (which is what a naïve implementation would do), UuidConverter always returns the size of a constant 16-byte array. If the wrapper type is in the same package as the generated protobuf message, then it does not need a fully-qualified name. Custom wrapper type converters can be in the same module as protobuf types that reference them. In order to use any wrapper type defined in `protokt-extensions`, the module must be included as a dependency: ```groovy dependencies { implementation 'com.toasttab.protokt:protokt-extensions:0.0.3' } ``` #### Interface implementation To avoid the need to create domain-specific objects from protobuf messages you can declare that a protobuf message implements a custom interface with properties and default methods. ```kotlin package com.protokt.sample interface Model { val id: String } ``` ```proto package com.protokt.sample; message ImplementsSampleMessage { option (protokt_class).implements = "Model"; string id = 1; } ``` If the wrapper interface is in the same package as the generated protobuf message, then it does not need a fully-qualified name. Wrapper interfaces cannot be used by protobuf messages in the same module that defines them; the dependency must be declared with`protoktExtensions` in `build.gradle`: ```groovy dependencies { protoktExtensions project(':api-module') } ``` #### Nonnull fields If there is a message that has no meaning whatsoever when a particular field is missing, you can emulate proto2's `required` key word by using the `(protokt_oneof).non_null` option: ```proto message Sample { } message NonNullSampleMessage { Sample non_null_sample = 1 [ (protokt_property).non_null = true ]; } ``` Generated code will not have a nullable type so the field can be referenced without using Kotlin's `!!`. Oneof fields can also be declared non-null: ```proto message NonNullSampleMessage { oneof non_null_oneof { option (protokt_oneof).non_null = true; string message = 2; } } ``` Note that deserialization of a message with a non-nullable field will fail if the message being decoded does not contain an instance of the required field. #### BytesSlice When reading messages that contain other serialized messages as `bytes` fields, protokt can keep a reference to the originating byte array to prevent a large copy operation on deserialization. This can be desirable when the wrapping message is a thin metadata shim and doesn't include much memory overhead: ```proto message SliceModel { int64 version = 1; bytes encoded_message = 2 [ (protokt_property).bytes_slice = true ]; } ``` ### Usage #### Gradle ```groovy buildscript { dependencies { classpath "com.toasttab.protokt:protokt-gradle-plugin:0.0.3" } } apply plugin: 'com.toasttab.protokt' ``` This will automatically download and install protokt, apply the Google protobuf plugin, and configure all the necessary boilerplate. By default it will also add `protokt-runtime` to the api scope of the project, and `protobuf-java` to the implementation scope. If your project is pure Kotlin you may run into the following error: ``` Execution failed for task ':compileJava'. > error: no source files ``` To work around it, disable all `JavaCompile` tasks in the project: ```groovy tasks.withType(JavaCompile) { enabled = false } ``` or: ```groovy compileJava.enabled = false ``` #### Command line code generation ```bash protokt-codegen$ ./gradlew assemble [OR ./gradlew installDist] protokt-codegen$ ./run-protokt.sh -h protokt-codegen$ ./run-protokt.sh \ -out=../kotlin \ -pkg=com.toasttab.protokt.conformance \ -file=conformance.proto \ -cp=../build/libs/protokt-codegen-0.0.3-SNAPSHOT-all.jar \ -plugin=../bin/protokt.sh ``` ### Contribution To enable rapid development of the code generator, the protobuf conformance tests have been compiled and included in the protokt-testing project. They run on Mac OS 10.14+ and Ubuntu 16.04 x86-64. Publish the plugin to the integration repository: ```bash protokt$ ./gradlew publishToIntegrationRepository ``` Then run the tests from `protokt-testing`: ```bash protokt-testing$ ./gradlew protokt-conformance-tests:test ``` All integration tests can be run with: ``` protokt-testing$ ./gradlew test ```	5 years ago
marwan-at-work	36cd11a2bf	Add twirpql ID to global registry	6 years ago
Alex Konradi	b44a1364ec	Add extension number for protoc-gen-validate This project is used by the [Envoy proxy](https://www.envoyproxy.io/) and several others. The existing code is using the extension value [919191](`54b08a686f/validate/validate.proto (L29)`) but since maintaining binary compatibility is not important, it shouldn't be a problem to change once this allocation is approved.	6 years ago
Xiang Dai	e479410564	delete all duplicate empty blanks (#5758 ) Signed-off-by: Xiang Dai <764524258@qq.com>	6 years ago
James DeFelice	4358133b85	options: reserved extension range for CSI	6 years ago
Toby Schneider	4fd7bdee48	Update options.md Update my projects' (DCCL / Goby) URLs. Propose new extension for CGSN Mooring project.	6 years ago
Gohar Aziz	f3a74befcd	Update options.md \| fix website url for C# port Update options.md \| fix website url for C# port of protocol buffers.. From `http://github.com/jskeet/dotnet-protobufs` to `https://github.com/jskeet/protobuf-csharp-port`	6 years ago
Feng Xiao	afe98de32a	Replace repo links.	6 years ago
Ryland Degnan	5b7d0dc8aa	Update options.md (#4952 ) Added Netifi Proteus to Registered Extensions	6 years ago
Feng Xiao	ef7052b473	Add global extension registry to the repo.	6 years ago

14 Commits (d5e964c8bb39a365b3d958c55cdc405049aacf29)