Add internal documentation for Grpc.Tools MSBuild integration (#31784)

* Developer doc for Grpc.Tools msbuild integration

* Fix formatting

* update docs from review comments

* small changes to the documentation

src/csharp/Grpc.Tools/implementation_notes.md

@@ -0,0 +1,222 @@
+# Grpc.Tools MSBuild integration overview
+
+This is an overview for maintainers of Grpc.Tools.
+
+The Grpc.Tools NuGet package provides custom build targets to make it easier to specify `.proto` files
+in a project and for those files to be compiled and their generated files to be included in the project.
+
+# Files in the NuGet package
+
+## .props and .target files
+
+MSBuild properties and targets included from the Grpc.Tools NuGet package are in:
+
+* `build\Grpc.Tools.props`, which imports
+    * `build\_grpc\_Grpc.Tools.props`
+    * `build\_protobuf\Google.Protobuf.Tools.props`
+* `build\Grpc.Tools.targets`, which imports
+    * `build\_grpc\_Grpc.Tools.targets`
+    * `build\_protobuf\Google.Protobuf.Tools.targets`
+
+Details of how NuGet packages can add custom build targets and properties to a project is documented
+here: [MSBuild .props and .targets in a package](https://learn.microsoft.com/en-us/nuget/concepts/msbuild-props-and-targets)
+
+Basically the `.props` and `.targets` files are automatically included in the projects - the `.props` at the top
+of the project and the `.targets` are added to the bottom of the project.
+
+## Visual Studio property pages
+
+For Visual Studio integration - these files provide the properties pages:
+
+* `build\_protobuf\Protobuf.CSharp.xml` (included from `Google.Protobuf.Tools.targets`)
+* `build\_grpc\Grpc.CSharp.xml` (included from `_Grpc.Tools.targets`)
+
+## Custom tasks DLLs
+
+DLLs containing the custom tasks are in:
+
+* `build\_protobuf\netstandard1.3`
+* `build\_protobuf\net45`
+
+## Protobuf compiler and C# gRPC plugin binaries
+
+Native binary executables for the protobuf compiler (_protoc_) and C# gRPC plugin (_grpc_csharp_plugin_) are
+included in the NuGet package. Included are binaries for various OSes (Windows, Linux, macOS) and
+CPU architectures (x86, x64, arm64).
+
+The build determines which executables to use for the particular machine that the it is being run on.
+These can be overridden by specifying MSBuild properties or environment variables to give the paths to custom executables:
+
+* `Protobuf_ProtocFullPath` property or `PROTOBUF_PROTOC` environment variable \
+Full path of protoc executable
+* `gRPC_PluginFullPath` property or `GRPC_PROTOC_PLUGIN` environment variable \
+Full path of gRPC C# plugin
+
+# Grpc.Tools custom build targets
+
+## Hooking the custom targets into the project build
+
+The custom targets hook into various places in a normal MSBuild build by specifying
+before/after targets at the relevant places. See
+[msbuild-targets](https://learn.microsoft.com/en-us/visualstudio/msbuild/msbuild-targets)
+for predefined targets.
+
+* Before `PrepareForBuild`
+    * we add `Protobuf_SanityCheck` that checks this is a supported project type, e.g. a C# project.
+* Before `BeforeCompile`
+    * we add all the targets that compile the `.proto` files and generate the expected `.cs` files. These files are added to those that get compiled by the C# compiler.
+        * The target `_Protobuf_Compile_BeforeCsCompile` is the _glue_ inserting the targets into the build.
+        It may look like it isn't doing anything but by specifying `BeforeTargets` and `DependsOnTargets` it inserts `Protobuf_Compile` into the build - but only doing so if this is a C# project.
+* After `CoreClean`
+    * we add `Protobuf_Clean` that cleans the files generated by the protobuf compiler.
+        * The target `_Protobuf_Clean_AfterCsClean` is the _glue_ inserting `Protobuf_Clean` into the build - but only doing so if this is a C# project.
+
+## Custom tasks
+
+There are a few custom tasks needed by the targets. These are implemented in C# in the Grpc.Tools project
+and packaged in the file `Protobuf.MSBuild.dll` in the NuGet package. See [task writing](https://learn.microsoft.com/en-us/visualstudio/msbuild/task-writing) for information about implementing custom tasks.
+
+* ProtoToolsPlatform
+    * Works out the operating system and CPU architecture
+* ProtoCompilerOutputs
+    * Tries to work out the names and paths of the files that would be generated by the protobuf compiler and returns these as a list of items
+    * Also returns list of items that is the same as the `Protobuf` items passed in with the output directory metadata updated
+* ProtoReadDependencies
+    * Read generated files from previously written dependencies file and return as items.
+* ProtoCompile
+    * Runs the protobuf compiler for a proto file. The executable to run is specified by the property `Protobuf_ProtocFullPath`
+    * To do this it:
+        * first writes out a response file containing the parameters for protobuf compiler 
+        * runs the executable, which generates the `.cs` files and a `.protodep` dependencies file
+        * reads the dependencies file to find the files that were generated and these are returned as a list of _items_ that can then be used in the MSBuild targets
+
+## Build steps
+
+The names of these items and properties are correct at the time of writing this document.
+
+High level builds steps:
+
+* Prepare list of `.proto` files to compile
+    * Makes sure all needed metadata is set for the `<Protobuf>` item, defaulting some values
+    * Removes `<Protobuf>` items that no longer exist or are marked as don't compile
+* Handling incremental builds
+    * Work out files that need to be created or have changed
+* Compile the `.proto` files
+* Add generated files to the list of files for the C# compiler
+
+### Prepare the list of .proto files to compile
+
+At various stages of the build copies of the original `<Protobuf>` items are created
+and/or updated to set metadata and to prune out unwanted items.
+
+Firstly, the build makes sure `ProtoRoot` metadata is set for all `Protobuf` items.
+A new list of items - `Protobuf_Rooted` - is created from the `Protobuf` items with `ProtoRoot` metadata set:
+
+* If `ProtoRoot` already set in the `<Protobuf>` item in the project file then it is left as-is.
+* If the `.proto` file is under the project's directory then set `ProtoRoot="."`.
+* If the `.proto` file is outside of the project's directory then set `ProtoRoot="<relative path to project directory>"`.
+
+Now prune out from `Protobuf_Rooted` the items that the user doesn't want to compile - those don't have
+`ProtoCompile` metadata as `true`.  The pruned list is now called `Protobuf_Compile`.
+
+Set the `Source` metadata on `Protobuf_Compile` items to be the name of the `.proto` file.
+The `Source` metadata is used later as a key to map generated files to `.proto` files.
+
+### Handling incremental builds
+
+#### Gathering files to check for incremental builds
+
+The target `Protobuf_PrepareCompile` tries to work out which files the protobuf compiler will
+generate without actually calling the protobuf compiler. This is a best-effort guess.
+The custom task `ProtoCompilerOutputs` is called to do this. The results are stored in the
+item list `Protobuf_ExpectedOutputs`.
+
+The target `Protobuf_PrepareCompile` also reads previously written `.protodep` files to get
+any actual files previously generated. The custom task `ProtoReadDependencies` is called to
+do this. The results are stored in the item list `Protobuf_Dependencies`.
+This is in case the list of actual files is different from the previous best-effort guess
+from `ProtoCompilerOutputs`.
+
+The expected outputs and previous outputs are needed so that the timestamps of those files
+can be checked later when handling an incremental build.
+
+#### Understanding incremental builds
+
+To avoid unnecessarily recompiling the `.proto` files during an incremental build the
+target `_Protobuf_GatherStaleBatched` tries to work out if any files have changed.
+
+It checks for out of date files using MSBuilds incremental build feature that compares the 
+timestamps on a target's _Input_ files to its _Output_ files.
+See [How to: Build incrementally](https://learn.microsoft.com/en-us/visualstudio/msbuild/how-to-build-incrementally)
+
+The _Inputs_ that are checked are:
+* Timestamps of the `.proto` files
+* Timestamps of previously generated files (list of these files read from `.protodep` files)
+* Timestamps of MSBuild project files
+
+These are checked against the _Outputs_:
+* Timestamps of the expected generated files
+
+[MSBuild target batching](https://learn.microsoft.com/en-us/visualstudio/msbuild/item-metadata-in-target-batching)
+is used to check each `.proto` file against its expected outputs. The batching is done by specifying the `Source` 
+metadata in the _Input_. Items where `Source` metadata matches in both input and output are in each batch.
+
+The target `_Protobuf_GatherStaleBatched` sets the metadata `_Exec=true` on `_Protobuf_OutOfDateProto` 
+items that are out of date.
+
+Later in the target `_Protobuf_GatherStaleFiles`, the items in `_Protobuf_OutOfDateProto` that don't have
+metadata `_Exec==true` are removed from the list of items, leaving only those that need compiling.
+
+### Compile the .proto files
+
+The target `_Protobuf_CoreCompile` is run for each `.proto` file that needs compiling.
+These are in the item list `_Protobuf_OutOfDateProto`. The custom task `ProtoCompile` is called to run the
+protobuf compiler. The files that were generated are returned in the item list `_Protobuf_GeneratedFiles`.
+
+If there are expected files that were not actually generated then the behaviour depends on whether the
+generated files should have been within the project (e.g. in the intermediate directories) or were
+specified to be outside of the project.
+* If within the project - empty files are created to prevent incremental builds doing unnecessary recompiles
+* If outside the project - by default empty files are not created and a warning is output (this behaviour is configurable)
+
+**TODO:** why are files inside and outside the project treated differently?
+
+### Add generated .cs files to the list of files for the C# compiler
+
+The target `_Protobuf_AugmentLanguageCompile` adds to the `Compile` item list
+(the list of files that CSC compiles) the expected generated files.
+
+**Note** - this is the _expected_ files not the _actual_ generated files and this is done
+before the protobuf compiler is called.
+
+**TODO:** why are the _expected_ files not the _actual_ generated files added?
+
+## Handling design time builds
+
+Design-time builds are special builds that Visual Studio uses to gather information about the project.
+They are not user-initiated but may be triggered whenever files are added, removed or saved.
+See [Design-Time Builds](https://github.com/dotnet/project-system/blob/main/docs/design-time-builds.md).
+
+The Grpc.Tools build targets used to try and optimise design time builds by disabling calling the
+protobuf compiler during a design time build.  However this optimisation can lead to errors in
+Visual Studio because the generated files may not exist or be out of date and any code that relies
+on them will then have errors.
+
+Now design time builds behave exactly the same as a normal build.
+The old behaviour can be enabled by setting the setting `DisableProtobufDesignTimeBuild` property
+to `true` in the project file **_if_** it is a design time build, e.g. by adding
+
+```xml
+<PropertyGroup Condition="'$(DesignTimeBuild)' == 'true' ">
+    <DisableProtobufDesignTimeBuild>true</DisableProtobufDesignTimeBuild>
+</PropertyGroup>
+```
+
+## Automatically including .proto files
+
+For SDK projects it is possible to automatically include `.proto` files found in the project
+directory or sub-directories, without having to specify them with a `<Protobuf>` item.
+To do this the property `EnableDefaultProtobufItems` has be set to `true` in the project file.
+
+By default it is not set and `<Protobuf>` items must be included in the project for
+the `.proto` files to be compiled.