While I'm here, use a fixed-size uint64_t in RSA_generate_key, rather
than unsigned long. This code also assumes unsigned long fits in
BN_ULONG, which is probably true on all platforms we care about, but
unnecessarily fussy.
The RSA_sign -> RSA_METHOD transition does require a cast. Go ahead and
check length/hash_nid consistency so we know it fits in the cast. This
does mean RSA_METHOD-backed keys are restricted to implementing digests
that we support, but that's probably fine. If anything, I think we
should try to shift away from RSA_METHOD as a story for custom keys.
Bug: 516
Change-Id: I3969da67d1daeff882279a534eb48ca831eb16cd
Reviewed-on: https://boringssl-review.googlesource.com/c/boringssl/+/54465
Commit-Queue: Bob Beck <bbe@google.com>
Auto-Submit: David Benjamin <davidben@google.com>
Commit-Queue: David Benjamin <davidben@google.com>
Reviewed-by: Bob Beck <bbe@google.com>
David Benjamin
58f728eaec