protobuf/docs/design/editions/java-lite-for-editions.md

Java Lite For Editions

Author: @zhangskz

Approved: 2023-05-26

Background

The "Lite" implementation for Java utilizes a custom format for embedding descriptors motivated by critical code-size and performance requirements for Android.

The code generator for Java Lite encodes an descriptor-like info string which is stored into RawMessageInfo. This is decoded into MessageSchema which serves as the descriptor-like schema for Java lite for parsing and serialization.

The current implementation makes significant use of an is_proto3 bit in the encoding, which is problematic for editions. Note that any parser changes to the format would also need to maintain backwards compatibility, due to our guarantees for parsers to remain backwards compatible within a major version.

Overview

Fortunately, we already have corresponding bits for most Editions Zero Features in the corresponding MessageInfo field entry encoding.

We will move existing remaining syntax usages reading is_proto3 to use these bits. Several other syntax usages need to be made to be editions compatible by merging implementations.

As new editions features are added that must be represented in MessageInfo, we will eventually need to revamp MessageInfo encoding to support these changes. However, this should be avoidable for Editions Zero.

Recommendation

Encoding: Add Is Edition Bit

RawMessageInfo should be augmented with an additional is_edition bit in flags' unused bits.

[0]: flags, flags & 0x1 = is proto2?, flags & 0x2 = is message?, flags & 0x4 = is edition?

The decoded ProtoSyntax should add a corresponding Editions option based on this bit.

public enum ProtoSyntax
  PROTO2;
  PROTO3;
  EDITIONS;

For now, there is no need to explicitly encode the raw editions string or feature options. These resolved features will be encoded directly in their corresponding field entries.

Encoding: Editions Zero Features

Field entries in RawMessageInfo already encode bits corresponding to most resolved Editions Zero features in GetExperimentalJavaFieldType. This is decoded in fieldTypeWithExtraBits by reading the corresponding bits.

Edition Zero Feature	Existing Encoding	Changes
features.field_presence	kHasHasBit (0x1000)	Keep as-is.
java.legacy_closed_enum	kMapWithProto2EnumValue (0x800)	Replace with kLegacyEnumIsClosedBit This will now be set for all enum fields, instead of just enum map values. We will still need to check syntax in the interim in case of gencode.
features.enum_type	EnumLiteGenerator writes UNRECOGNIZED(-1) value for open enums in gencode. This is not encoded in MessageInfo since this is an enum feature.	This is not needed in Editions Zero since enum closedness in Java Lite's runtime is dictated per-field by java.legacy_closed_enum. (Edition Zero Feature: Enum Field Closedness), but should be used when Java non-conformance is fixed. Note, this is implicitly encoded in kLegacyEnumIsClosedBit if java.legacy_closed_enum is unset since the corresponding FieldDescriptor helper should fall back on the EnumDescriptor.
features.repeated_field_encoding	GetExperimentalJavaFieldTypeForPacked	Keep as-is.
features.string_field_validation	kUtf8CheckBit (0x200)	Keep as-is. HINT does not apply to Java and will have the same behavior as MANDATORY or NONE
features.message_encoding	Not present.	Encode as type group. See below.

Several places already use these bits properly, but there are a few syntax usages in the decoding that should be replaced by checking the corresponding feature bit.

There are several unused bits that we could use for future field-level features before breaking the encoding format, but we should not need these for editions zero.

The results of the is_proto3 and feature bits only seem to be used within protobuf, and don't seem to be publicly exposed.

features.message_encoding

In the compiler, message fields with features.message_encoding = DELIMITED should be treated as a group before encoding message info.

This means that GetExperimentalJavaFieldTypeForSingular, should encode the field's type GROUP (17), instead of its actual type MESSAGE (9), e.g.

int GetExperimentalJavaFieldTypeForSingular(const FieldDescriptor* field) {
  int result = field->type();
  if (result == FieldDescriptor::TYPE_MESSAGE) {
    if (field->isDelimited()) {
      return 17; // GROUP
    }
  }
}

ImmutableMessageFieldLiteGenerator::GenerateFieldInfo calls this when generating the message field's field info.

The nested message's MessageInfo encoding does not need to be changed as this is already identical for group and message.

Since each message field will be handled separately, this means that the post-editions proto file below

// foo.proto
edition = "tbd"

message Foo {
  message Bar {
    int32 x = 1;
    repeated int32 y = 2;
  }
  Bar bar = 1 [features.message_encoding = DELIMITED];
  Bar baz = 2; // not DELIMITED

}

will be encoded and treated by MessageSchema like its pre-editions equivalent below.

message Foo {
  group Bar = 1 {
    int32 x = 1;
    repeated int32 y = 2;
  }
  Bar baz = 2; // not DELIMITED
}

We recommended this alternative to minimize changes to the encoding and how groups are treated.

In a future breaking change, we could consider renaming FieldType.GROUP to FieldType.MESSAGE_DELIMITED while preserving the same number and encoding for clarity. For now, we will leave the naming for this enum as-is.

Alternative: Add kIsMessageEncodingDelimitedBit

Alternatively, we could encode features.message_encoding = DELIMITED as-is as type MESSAGE. The MessageInfo encoding would encode these as a normal message field, using an unused (0x1100) bit as kIsMessageEncodingDelimitedBit.

This could be used to indicate that the message should be parsed/serialized from the wire-format as if it were a group. This would need to be passed along to MessageSchema which would then handle treating Messages with this bit set as groups e.g. in case Message.

This is less ideal, since it would require handling this in multiple places.

Unify non-feature syntax usages

There are several places that branch on syntax into separate proto2/proto3 codepaths. These generally duplicate a lot of code and should be unified into a single syntax-agnostic code path branching on the relevant feature bits.

This code tends to be pretty opaque, so we should document this with comments or add helpers (e.g. isEnforceUtf8) to indicate what feature bits are used as we make changes here.

ManifestSchemaFactory.newSchema()	MessageInfo -> Schema	Allow extensions for editions.
MessageSchema.getSerializedSize()	Message -> Serialized Size	Unify getSerializedSizeProto2/3
MessageSchema.writeTo()	Serialize Message	Unify writeFieldsInAscendingOrderProto2/3
MessageSchema.mergeFrom()	Parse Message	Unify parseProto2/3Message
DescriptorMessageInfoFactory.convert()	Descriptor -> MessageInfo	Unify convertProto2/3

There is a lot of dead code in Java Lite so several syntax usages can also be deleted or merged where possible.

Alternatives

Alternative 1: Introduce New Backwards-compatible MessageInfo Encoding

Add a new backwards-compatible MessageInfo encoding for editions.

The is_edition bit could toggle the encoding format being used, where is_edition == true indicates the new encoding format but is_edition == false indicates the old encoding.

This would allow us to encode additional information that the current encoding format does not currently have available bits to support, such as the editions string or additional features.

For example, the current encoding format only has a fixed number of available field entry bits where we could encode new feature bits. We will need to introduce a new encoding format once we exceed these, or if we want to encode features at the message level.

In a future major version bump when support for proto2/3 is officially dropped, we could drop support for the previous encoding format.

The recommendation is to revisit alternative 1 along with alternative 2 post-Editions zero as we need to support additional feature bits.

Pros

• Future-proof for future editions and features

Cons

• Blocks editions zero on more complex encoding changes that won't be used yet.
• Requires more invasive updates to all MessageInfo decodings

Alternative 2: Move to MiniDescriptor encoding

We could switch Java Lite to use the MiniDescriptor encoding specification.

Like Java Lite, this encoding seems to be optimized to be lightweight and with minimal descriptor information.

MiniDescriptors do not encode proto2/proto3 syntax currently, which makes it mostly editions-compatible. MiniDescriptors encode FieldModifier/MessageModifier bits that correspond to some editions zero similarly to the Java Lite field feature bits, and can be augmented to support additional features.

Supposedly, this encoding format should support an arbitrary number of modifier bits, but this needs to be double-checked to verify there isn't a similar hard limit to the number of features.

It is unclear whether this is sufficiently optimized for Android's needs and how compatible this would be with Java Lite's Schemas.

The recommendation is to revisit alternative 2 along with alternative 1 post-Editions zero as we need to support additional feature bits.

Pros

• Unify implementations for lower long-term maintenance cost

• MiniDescriptor encoding will eventually need to be updated for editions anyways.

Cons

• Blocks editions zero on more complex encoding changes that aren't necessary.

• Requires even more invasive updates to all MessageInfo decodings

• Probably requires major version bumps to break compatibility

• Unknown code size /schema compatibility constraints that would need to be explored.

• There are a few possible changes to MiniDescriptors on the table that we should wait to settle before bringing on additional implementations.

Alternative 3: Do Nothing

Doing nothing is always an alternative. Describe the pros and cons of it.

Pros

• No work

Cons

• Editions is blocked since Java Lite protos are stuck in the past