boringssl/crypto/x509/test/pss_sha256_unknown_mgf.pem

Tighten up supported PSS combinations in X.509. Matching Chromium, Go, and TLS 1.3, only allow SHA-256, SHA-384, and SHA-512 RSA-PSS signatures, where MGF-1 and message hash match and salt length is hash length. Sadly, we are stuck tolerating an explicit trailerField for now. See the certificates in cl/362617931. This also fixes an overflow bug in handling the salt length. On platforms with 64-bit long and 32-bit int, we would misinterpret, e.g, 2^62 + 32 as 32. Also clean up the error-handling of maskHash. It was previously handled in a very confusing way; syntax errors in maskHash would succeed and only be noticed later, in rsa_mgf1_decode. I haven't done it in this change, but as a followup, we can, like Chromium, reduce X.509 signature algorithms down to a single enum. Update-Note: Unusual RSA-PSS combinations in X.509 are no longer accepted. This same change (actually a slightly stricter version) has already landed in Chrome. Bug: 489 Change-Id: I85ca3a4e14f76358cac13e66163887f6dade1ace Reviewed-on: https://boringssl-review.googlesource.com/c/boringssl/+/53865 Auto-Submit: David Benjamin <davidben@google.com> Reviewed-by: Adam Langley <agl@google.com> Commit-Queue: Adam Langley <agl@google.com> 3 years ago			`-----BEGIN CERTIFICATE-----`
			`MIIDEzCCAcSgAwIBAgIBADBEBgkqhkiG9w0BAQowN6APMA0GCWCGSAFlAwQCAQUA`
			`oR8wHQYMKoZIhvcSBAGEtwkAMA0GCWCGSAFlAwQCAQUAogMCASAwFDESMBAGA1UE`
			`AwwJQm9yaW5nU1NMMCIYDzAwMDAwMTAxMDAwMDAwWhgPOTk5OTEyMzEyMzU5NTla`
			`MBQxEjAQBgNVBAMMCUJvcmluZ1NTTDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC`
			`AQoCggEBALoL2oQZEgFBdXwuPb29W75T63JfNJKKdYi6YrmKM+EKbcMue/hFrLGQ`
			`XB6a2eQZFn+j3hmexeQF9T8iWxh2S6rzAr1Yj+qXeDBaMf4oBEiEhBxIsaIlws3q`
			`Qa4baeVEEoxw+A+ISrYHTIFcV/i0bcIFt5p7v7wbu686a/w0vIqPfad5amdQJMvm`
			`jZXDI+jGMvFPmBRHr2/1dJUWPaKsJluwR514pJv74urIyEt+dFPM2/5kc2HiLNke`
			`uS1Hbky+dPlDIGrfaHHsCNnb/GjvQ6YfvDXRrYPCxWMk0x3FMSaDK9T0zoJ5hE9f`
			`ViR+D6xcJO2RNUCUENS+Iipj3kIrLbkCAwEAATBEBgkqhkiG9w0BAQowN6APMA0G`
			`CWCGSAFlAwQCAQUAoR8wHQYMKoZIhvcSBAGEtwkAMA0GCWCGSAFlAwQCAQUAogMC`
			`ASADggEBACdP7ToiiM+6TqeAbKRqsPuFX9Q0qJtjX8mGVyzOEy402hEhglMeL9il`
			`FFFKCgo/wux4VIbw7BVLFDfYPcj8E7snj+J0mITIFt5oCwXG+MrJUnfklBEEyelF`
			`kVyr1ZVIwsk/lWSaOcQocbx8/szRRhwwVTT7BySnMV4e/y7Z9beye6tl7lg4373K`
			`E4akVubalnkoTHP5fZ6f0AfOXJEnLw2Mn0zjwrZmasMyXxLH8ApUVruPBQHY7IQB`
			`sMAURzkRtcCyzhQniJFcu7oMmkj+v8FVG02792DOWP+Y/yC0ccmHcvDbdzJW5SDm`
			`rBvNkmnPt6sYs7/RLRaVky7cRMAQrFs=`
			`-----END CERTIFICATE-----`