APPS/pkeyutl: -digest implies -rawin and can only be used with -sign and -verify

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/22910)

3 files changed, 25 insertions, 19 deletions

diff --git a/CHANGES.md b/CHANGES.md
index a32cf79466..77e8fe3b04 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -50,6 +50,13 @@ OpenSSL 3.5
 
    *Małgorzata Olszówka*
 
+ * The `-rawin` option of the `pkeyutl` command is now implied (and thus no
+   longer required) when using `-digest` or when signing or verifying with an
+   Ed25519 or Ed448 key.
+   The `-digest` and `-rawin` option may only be given with `-sign` or `verify`.
+
+   *David von Oheimb*
+
  * Optionally allow the FIPS provider to use the `JITTER` entropy source.
    Note that using this option will require the resulting FIPS provider
    to undergo entropy source validation [ESV] by the [CMVP], without this
@@ -215,11 +222,6 @@ OpenSSL 3.4
 
    *Damian Hobson-Garcia*
 
- * The `-rawin` option of the `pkeyutl` command is now implied (and thus no more
-   required) when signing or verifying with an Ed25519 or Ed448 key.
-
-   *David von Oheimb*
-
  * Added support to build Position Independent Executables (PIE). Configuration
    option `enable-pie` configures the cflag '-fPIE' and ldflag '-pie' to
    support Address Space Layout Randomization (ASLR) in the openssl executable,
diff --git a/apps/pkeyutl.c b/apps/pkeyutl.c
index 961a2f7650..30a4259fe4 100644
--- a/apps/pkeyutl.c
+++ b/apps/pkeyutl.c
@@ -83,7 +83,6 @@ const OPTIONS pkeyutl_options[] = {
 
     OPT_SECTION("Input"),
     {"in", OPT_IN, '<', "Input file - default stdin"},
-    {"rawin", OPT_RAWIN, '-', "Indicate that signature input data is not hashed"},
     {"inkey", OPT_INKEY, 's', "Input key, by default private key"},
     {"pubin", OPT_PUBIN, '-', "Input key is a public key"},
     {"passin", OPT_PASSIN, 's', "Input file pass phrase source"},
@@ -103,8 +102,10 @@ const OPTIONS pkeyutl_options[] = {
      "Verify with public key, recover original data"},
 
     OPT_SECTION("Signing/Derivation/Encapsulation"),
+    {"rawin", OPT_RAWIN, '-',
+     "Indicate that the signature/verification input data is not yet hashed"},
     {"digest", OPT_DIGEST, 's',
-     "Specify the digest algorithm when signing the raw input data"},
+     "The digest algorithm to use for signing/verifying raw input data. Implies -rawin"},
     {"pkeyopt", OPT_PKEYOPT, 's', "Public key options as opt:value"},
     {"pkeyopt_passin", OPT_PKEYOPT_PASSIN, 's',
      "Public key option that is read as a passphrase argument opt:passphrase"},
@@ -288,6 +289,9 @@ int pkeyutl_main(int argc, char **argv)
     if (!app_RAND_load())
         goto end;
 
+    if (digestname != NULL)
+        rawin = 1;
+
     if (kdfalg != NULL) {
         if (kdflen == 0) {
             BIO_printf(bio_err,
@@ -316,15 +320,9 @@ int pkeyutl_main(int argc, char **argv)
             }
             rawin = 1; /* implied for Ed25519(ph) and Ed448(ph) and maybe others in the future */
         }
-    } else if (rawin) {
-        BIO_printf(bio_err,
-                   "%s: -rawin can only be used with -sign or -verify\n", prog);
-        EVP_PKEY_free(pkey);
-        goto opthelp;
-    }
-    if (digestname != NULL && !rawin) {
+    } else if (digestname != NULL || rawin) {
         BIO_printf(bio_err,
-                   "%s: -digest can only be used with -rawin\n", prog);
+                   "%s: -digest and -rawin can only be used with -sign or -verify\n", prog);
         EVP_PKEY_free(pkey);
         goto opthelp;
     }
diff --git a/doc/man1/openssl-pkeyutl.pod.in b/doc/man1/openssl-pkeyutl.pod.in
index 99b2c11aee..22ce8c4edf 100644
--- a/doc/man1/openssl-pkeyutl.pod.in
+++ b/doc/man1/openssl-pkeyutl.pod.in
@@ -70,17 +70,19 @@ and is implied by the Ed25519 and Ed448 algorithms.
 Except with EdDSA,
 the user can specify a digest algorithm by using the B<-digest> option.
 
+The B<-digest> option implies B<-rawin>.
+
 =item B<-digest> I<algorithm>
 
-This specifies the digest algorithm which is used to hash the input data before
+This option can only be used with B<-sign> and B<-verify>.
+It specifies the digest algorithm which is used to hash the input data before
 signing or verifying it with the input key. This option could be omitted if the
 signature algorithm does not require one (for instance, EdDSA). If this option
 is omitted but the signature algorithm requires one, a default value will be
 used. For signature algorithms like RSA, DSA and ECDSA, SHA-256 will be the
-default digest algorithm. For SM2, it will be SM3. If this option is present,
-then the B<-rawin> option must be also specified.
+default digest algorithm. For SM2, it will be SM3.
 At this time, HashEdDSA (the ph or "prehash" variant of EdDSA) is not supported,
-so the B<-digest> option cannot be used with EdDSA.
+so the B<-digest> option cannot be used with EdDSA).
 
 =item B<-out> I<filename>
 
@@ -471,6 +473,10 @@ L<EVP_PKEY_CTX_set_tls1_prf_md(3)>,
 
 =head1 HISTORY
 
+Since OpenSSL 3.5,
+the B<-digest> option implies B<-rawin>, and these two options are
+no longer required when signing or verifying with an Ed25519 or Ed448 key.
+
 The B<-engine> option was deprecated in OpenSSL 3.0.
 
 =head1 COPYRIGHT