OpenSearch security plugin certificates expire after 730 days and are never regenerated, breaking chef-server-ctl reconfigure

[slabstone]

Chef Server Version

15.9.38, standalone package install. No additional plugins. Not using Chef Backend / HA.

Platform Details

Ubuntu 22.04.4 LTS, x86_64, 6 vCPU, 16GB RAM, 170GB disk (10% used). Not containerized.

Configuration

Standalone, single-node install. Long-running instance (793 days uptime at time of
failure) — not a fresh install, and this was a routine chef-server-ctl reconfigure
that had not been run in a long time on this host. No non-default OpenSearch/security
settings in /etc/opscode/chef-server.rb.

Scenario

Running chef-server-ctl reconfigure for routine maintenance, after a long period
without running it, causes a hard failure at the OpenSearch security-plugin step —
the whole reconfigure run aborts (Infra Phase failed).

Steps to Reproduce

1. Install Chef Server (fresh bootstrap generates a self-signed root CA + admin +
   node certificate for the bundled OpenSearch security plugin, via
   omnibus/files/server-ctl-cookbooks/infra-server/files/default/opensearch/ssl_script.sh,
   each valid for -days 730 from the opensearch.rb recipe).
2. Let the instance run past that certificate's 730-day validity window without
   ever deleting/regenerating /var/opt/opscode/opensearch/config/admin.pem (or
   root-ca.pem/node.pem).
3. Run chef-server-ctl reconfigure.

Root cause: the execute 'generate ssl certificates' resource in opensearch.rb
is guarded by not_if { ::File.exist?('/var/opt/opscode/opensearch/config/admin.pem') }.
This only checks file existence, never certificate validity, so the one-shot
bootstrap certificate generation never re-runs once the file is present — even
after it expires. There is no other renewal path in chef-server-ctl for this
certificate set. Confirmed this is still present, unchanged, in the current
15.10.126 release (PR #4085 touched the readiness-check/retry logic right below
this same resource but did not add any expiry check or touch the not_if guard).

Expected Result

chef-server-ctl reconfigure completes successfully, or at minimum detects the
expired OpenSearch security-plugin certificate and regenerates it automatically
rather than failing.

Actual Result

Reconfigure aborts with a fatal error at execute[add internal user to opensearch security plugin]:

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '255'
---- Begin output of export JAVA_HOME="/opt/opscode/embedded/open-jre/"; ./securityadmin.sh -f ../securityconfig/internal_users.yml -icl -nhnv -cert /opt/opscode/embedded/opensearch/config/admin.pem -cacert /opt/opscode/embedded/opensearch/config/root-ca.pem -key /opt/opscode/embedded/opensearch/config/admin-key.pem ----
...
javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
...
Caused by: java.security.cert.CertificateExpiredException: NotAfter: Thu Jul 09 15:49:44 UTC 2026
...
ERR: Cannot connect to OpenSearch. Please refer to opensearch logfile for more information
Trace:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{...}{localhost}{127.0.0.1:9300}]]
...
[FATAL] Mixlib::ShellOut::ShellCommandFailed: execute[add internal user to opensearch security plugin] (infra-server::opensearch line 187) had an error

Confirmed via openssl x509 -in admin.pem -noout -dates that root-ca.pem,
admin.pem, and node.pem were all generated on the same date, ~730 days
before failure, and all expired simultaneously.

Workaround: stop OpenSearch, move aside the 7 files in
/var/opt/opscode/opensearch/config/ (root-ca.pem, root-ca-key.pem,
root-ca.srl, admin.pem, admin-key.pem, node.pem, node-key.pem),
then run chef-server-ctl reconfigure again — this trips the not_if guard
and regenerates a fresh certificate set (another 730-day window), after which
reconfigure completes successfully. This is destructive to the existing
internal PKI (a brand-new self-signed CA is generated, not a renewal) but
appears safe since nothing external trusts this CA.