wolfsshd: ML-DSA host key cannot complete KEX — wc_MlDsaKey_ExportPubRaw returns BAD_FUNC_ARG in SendKexDhReply

[shaunchokshi]

An ML-DSA-65 host key loads into wolfsshd and ssh-mldsa-65 is negotiated by both peers, but the handshake then fails server-side while building the KEXDH reply. The result is that an ML-DSA host key can be loaded and advertised but cannot sign the KEX, so no client can complete a handshake against it.

I'm reporting rather than patching — the fix looks like it belongs either in wolfCrypt (derive/retain the public key) or in how wolfSSH obtains the host public key, and that seemed like a maintainer call rather than something to guess at in a signing path.

Environment

• wolfSSH master @ 53047bd
• wolfSSL master @ 793608e (5.9.2)
• Linux 6.x (CachyOS), gcc 16.1.1
• wolfSSH configured with --enable-sshd --enable-shell --enable-scp --enable-sftp --enable-fwd

Reproduction

1. Generate an ML-DSA-65 host key with wolfCLU: wolfssl genkey ml-dsa -level 3 -out h -output priv
2. Point a wolfsshd config's HostKey at it and run wolfsshd -D -d -f <config>
3. Connect with the wolfSSH example client: examples/client/client -h 127.0.0.1 -p <port> -u <user> -P <pass>

Observed

[SSHD] Adding name : ssh-mldsa-65
DKI: Server Host Key Algorithms
GNL: name ID ssh-mldsa-65 matches ssh-mldsa-65        # client offered it too
Using ML-DSA Host key
Leaving SendKexDhReply(), ret = -173                  # BAD_FUNC_ARG
[SSHD] Failed to accept WOLFSSH connection ... error -1001

Root cause

The failure is in the ML-DSA block of SendKexDhReply() (src/internal.c, ~L13264-13279). wc_MlDsaKey_PrivateKeyDecode() succeeds, but the immediately following wc_MlDsaKey_ExportPubRaw() returns BAD_FUNC_ARG: the key decoded from a private-key file has no public-key component available to export, and wolfCrypt does not derive it here.

This reproduces standalone against wolfSSL alone, with no wolfSSH involved — Init / SetParams(3) / PrivateKeyDecode / ExportPubRaw on the same key file, mirroring what SendKexDhReply() does:

key file h.der, 4060 bytes, first byte 0x30
  Init             ret=0
  SetParams(3)     ret=0
  PrivateKeyDecode ret=0  scratch=4060
  ExportPubRaw     ret=-173  qSz=1952   <-- BAD_FUNC_ARG

Reproduced with both a -output priv DER key and a -output keypair PKCS#8 PEM key.

Questions

• Should wc_MlDsaKey_ExportPubRaw() derive the public key from a decoded private key, or should wolfSSH load/keep the public half separately for the host key?
• Is a specific key-file encoding (a private key with the embedded public key) expected for ML-DSA host keys? If so, which wolfCLU invocation produces it?

Happy to test a patch against this setup.

---

Found while testing #1118 (loading PKCS#8 PEM host keys in wolfsshd). That PR is independent — it fixes key loading; this is the separate failure that occurs afterward, and it reproduces with DER host keys on unpatched master too.

[stenslae]

Hey @shaunchokshi

Thanks for sending in the report! This is a wolfCrypt gap in how public keys are stored for ML-DSA, ECDSA, and Ed25519.

In the Ed25519 case, wolfSSL has an existing test asserting sign must fail on a priv-only-decoded key. EdDSA hashes the public key into the signature, so it structurally can't sign without it, so no change is needed. ECDSA doesn't have that constraint, its signature doesn't depend on the public point, and there's no test locking in current behavior.

ECDSA and Ed25519 host keys only work because ssh-keygen's OpenSSH format always embeds the pubkey, not because anything derives it.

• Should ExportPubRaw derive, or should wolfSSH keep the public half separately?

Neither, the fix should go in wolfCrypt. ExportPubRaw should stay an accessor. wolfSSH has no way to reconstruct the public key itself, and patching only wolfSSH would leave every other consumer of that decode call exposed. I'm working on a fix for this for ML-DSA and ECDSA in wolfCrypt and will link the PR shortly.

I'll also add a check in IdentifyAsn1Key() after private-key decodes in wolfSSH.

• Is a specific key-file encoding (a private key with the embedded public key) expected for ML-DSA host keys?

There is currently no way to generate an ML-DSA host key with wolfCLU that wolfSSH can load and use for signing. The -output keypair uses the same priv-only DER as -output priv plus a separate .pub file that wolfCLU loads. Neither flag produces the seed or embedded-pubkey encoding that would actually work. wc_MlDsaKey_PrivateKeyToDer() hardcodes NULL, 0 for the pubkey argument, so a wolfCLU-only fix can't work. Will look further into changes in this on the wolfCLU and wolfCrypt end and get back to you.