Bound STACK_OF(T) sizes by int

Although we've switched STACK_OF(T) to use size_t, OpenSSL used int
pervasively. Much of crypto/x509 and third-party callers use int
indices. As much of that is in the public API now, ensure that
STACK_OF(T) can never exceed INT_MAX elements.

Bug: 516
Change-Id: I26b8fe590655f8c3e449b749b5d0222e28c413f8
Reviewed-on: https://boringssl-review.googlesource.com/c/boringssl/+/60065
Reviewed-by: Adam Langley <agl@google.com>
Commit-Queue: Adam Langley <agl@google.com>
Auto-Submit: David Benjamin <davidben@google.com>

crypto/stack/stack.c

@@ -57,7 +57,9 @@
 #include <openssl/stack.h>
 
 #include <assert.h>
+#include <limits.h>
 
+#include <openssl/err.h>
 #include <openssl/mem.h>
 
 #include "../internal.h"
@@ -161,6 +163,11 @@ size_t sk_insert(_STACK *sk, void *p, size_t where) {
     return 0;
   }
 
+  if (sk->num >= INT_MAX) {
+    OPENSSL_PUT_ERROR(CRYPTO, ERR_R_OVERFLOW);
+    return 0;
+  }
+
   if (sk->num_alloc <= sk->num + 1) {
     // Attempt to double the size of the array.
     size_t new_alloc = sk->num_alloc << 1;

include/openssl/stack.h

@@ -138,7 +138,8 @@ STACK_OF(SAMPLE) *sk_SAMPLE_new(sk_SAMPLE_cmp_func comp);
 // NULL on allocation failure.
 STACK_OF(SAMPLE) *sk_SAMPLE_new_null(void);
 
-// sk_SAMPLE_num returns the number of elements in |sk|.
+// sk_SAMPLE_num returns the number of elements in |sk|. It is safe to cast this
+// value to |int|. |sk| is guaranteed to have at most |INT_MAX| elements.
 size_t sk_SAMPLE_num(const STACK_OF(SAMPLE) *sk);
 
 // sk_SAMPLE_zero resets |sk| to the empty state but does nothing to free the