Clean up conformance test documentation (#10410)

* Update conformance test documentation

* Fix bug to allow custom argument forwarding to conformance tests

* Fix trailing sentence

* Add note about linux CMake support up top

* Add note about CMake being C++-only

cmake/README.md

@@ -1,7 +1,9 @@
 This directory contains *CMake* files that can be used to build protobuf.
 
 You need to have [CMake](http://www.cmake.org) and
-[Git](http://git-scm.com) installed on your computer before proceeding.
+[Git](http://git-scm.com) installed on your computer before proceeding.  We
+currently support CMake 3.5 and newer on both [Windows](#windows-builds) and
+[Linux](#linux-builds).
 
 Most of the instructions will be given using CMake's command-line interface, but
 the same actions can be performed using appropriate GUI tools.

conformance/BUILD.bazel

@@ -12,7 +12,7 @@ load(
 )
 
 exports_files([
-    "conformance_test_runner.sh",
+    "bazel_conformance_test_runner.sh",
     "failure_list_cpp.txt",
     "failure_list_csharp.txt",
     "failure_list_java.txt",
@@ -348,7 +348,7 @@ pkg_files(
         ["**/*"],
         exclude = [
             # Handled by dist_scripts:
-            "conformance_test_runner.sh",
+            "bazel_conformance_test_runner.sh",
 
             # The following are not in autotools dist:
             "autoload.php",

conformance/README.md

@@ -12,44 +12,42 @@ contains all of the tests themselves.  Then separate programs written
 in whatever language you want to test communicate with the tester
 program over a pipe.
 
-Before running any of these tests, make sure you run `make` in the base
+If you're not using Bazel to run these tests, make sure you build the C++
-directory to build `protoc`, since all the tests depend on it.
+tester code beforehand, e.g. from the base directory:
 
-    $ make
+    $ cmake . -Dprotobuf_BUILD_CONFORMANCE=ON && cmake --build .
+
+This will produce a `conformance_test_runner` binary that can be used to run
+conformance tests on any executable.  Pass it `--help` for more information.
 
 Running the tests for C++
 -------------------------
 
 To run the tests against the C++ implementation, run:
 
-    $ cd conformance && make test_cpp
+    $ bazel test //src:conformance_test
-
-Running the tests for JavaScript (Node.js)
-------------------------------------------
-
-To run the JavaScript tests against Node.js, make sure you have "node"
-on your path and then run:
-
-    $ cd conformance && make test_nodejs
 
-Running the tests for Ruby (MRI)
+Or alternatively with CMake:
---------------------------------
 
-To run the Ruby tests against MRI, first build the C extension:
+    $ ctest -R conformance_cpp_test
 
-    $ cd ruby && rake
+Running the tests for other languages
+-------------------------------------
 
-Then run the tests like so:
+All of the languages in the Protobuf source tree are set up to run conformance
+tests using similar patterns.  You can either use Bazel to run the
+`conformance_test` target defined in the language's root `BUILD.bazel` file,
+or create an executable for a custom test and pass it to
+`conformance_test_runner`.
 
-    $ cd conformance && make test_ruby
+For example, to run the Ruby tests against MRI, you can call:
 
-Running the tests for other languages
+    $ bazel test //ruby:conformance_test
--------------------------------------
 
-Most of the languages in the Protobuf source tree are set up to run
+Note: CMake can be used to build the conformance test runner, but not any of
-conformance tests.  However some of them are more tricky to set up
+the conformance test executables outside C++.  So if you aren't using Bazel
-properly.  See `tests.sh` in the base of the repository to see how
+you'll need to create the executable you pass to `conformance_test_runner` via
-Kokoro runs the tests.
+some alternate build system.
 
 Testing other Protocol Buffer implementations
 ---------------------------------------------

conformance/bazel_conformance_test_runner.sh

@@ -1,4 +1,7 @@
 #!/bin/bash
+# This is an internal file that should only be called from Bazel rules.  For
+# custom conformance tests outside of Bazel use CMAKE with
+# -Dprotobuf_BUILD_CONFORMANCE=ON to build the test runner.
 
 set -x
 echo $@

conformance/conformance_test_runner.cc

@@ -227,7 +227,7 @@ int ForkPipeRunner::Run(
           UsageError();
         }
       } else {
-        program += argv[arg];
+        program += argv[arg++];
         while (arg < argc) {
           program_args.push_back(argv[arg]);
           arg++;
@@ -303,8 +303,10 @@ void ForkPipeRunner::SpawnTestProgram() {
 
     std::vector<const char *> argv;
     argv.push_back(executable.get());
+    GOOGLE_LOG(INFO) << argv[0];
     for (size_t i = 0; i < executable_args_.size(); ++i) {
       argv.push_back(executable_args_[i].c_str());
+      GOOGLE_LOG(INFO) << executable_args_[i];
     }
     argv.push_back(nullptr);
     // Never returns.

conformance/defs.bzl

@@ -30,7 +30,7 @@ def conformance_test(
 
     native.sh_test(
         name = name,
-        srcs = ["//conformance:conformance_test_runner.sh"],
+        srcs = ["//conformance:bazel_conformance_test_runner.sh"],
         data = [testee] + failure_lists + [
             "//conformance:conformance_test_runner",
         ],