Generalize the boot security upgrade into profile::boot_security_upgrade

[akuzminsky]

Why

The symptom is fleet-wide, not runner-specific. A fresh instance comes up unpatched, unattended-upgrades is on a timer, and AWS Inspector scans the gap and opens a finding. The finding closes on the next unattended-upgrades run — but it has already reopened its vulnerability group by then, and a group old enough to be reopened that way breaks the remediation SLA.

So the fix isn't "patch more often on a timer", it's "don't present an unpatched host to Inspector at all".

What

Lifts the runner-specific script and exec out of profile::github_runner into profile::boot_security_upgrade, an opt-in profile any role can include. Not wired into profile::base — adding it to a role stays a deliberate decision about that role's provisioning time.

Beyond patching early, the script now drops the InspectorEc2Exclusion tag when it's done, so Inspector's first scan sees an already-patched host.

Param	Default	Notes
budget	480	Cumulative wall-clock cap, not per-attempt
fail_on_error	false	Log and continue
exclusion_tag	InspectorEc2Exclusion	Empty string is the off switch

Tag removal is best effort

It needs ec2:DeleteTags on the instance profile and the launch-time tag from Terraform, and neither is in place fleet-wide yet. Every failure path — no permission, no ec2metadata, unreadable instance id — logs a distinct line and returns 0. A host with neither is no worse off than before this PR; it just keeps being scanned on Inspector's own schedule.

The tag is dropped whether or not the upgrade succeeded. Leaving it on a host that failed to patch would hide a genuinely vulnerable instance from Inspector, which inverts the point.

Failure semantics

fail_on_error defaults to false because an unpatched-but-running host beats an ABANDONed one on anything stateful. profile::github_runner passes true to keep today's behavior, which is right for a disposable runner the ASG will just replace. logoutput => true is what makes the "log" half real — the default on_failure prints nothing for a status we just declared a success.

Fixed while generalizing

• Retry loop hot-spun through the last 15s of budget instead of sleeping, hammering the dpkg lock (~27 attempts on a 3s budget). Now sleeps min(remaining, interval).
• cloud-guest-utils and awscli are declared and required rather than probed for with command -v. Without the ordering edge a first-boot exec can evaluate before the packages and silently no-op the tag drop — on exactly the boot it matters most.
• creates replaces unless => "test -f ..." — no shell fork per run, no dependency on test being on path, and absolute-path validation at compile time.

Verification

Live sandbox instance (i-00e8ef380bdd1b0dc, env development):

Exec[boot-security-upgrade]/returns: attempt 1, 480s of budget left
Exec[boot-security-upgrade]/returns: succeeded on attempt 1
Exec[boot-security-upgrade]/returns: removed InspectorEc2Exclusion from i-00e8ef380bdd1b0dc
Exec[boot-security-upgrade]/returns: executed successfully

The second catalog apply in that same run has no Exec[boot-security-upgrade] line at all — the creates marker suppressed it. The tag call returning clean also means ec2:DeleteTags is already granted in that account.

Also: puppet-lint clean on all four targets; puppet parser validate plus catalog compiles across four parameter variants checking the rendered command/returns/timeout/creates and all four require edges; script exercised over ~15 cases in a noble container (success, give-up, mid-budget recovery, permission denied, non-EC2, missing binaries, arg-parsing errors); shellcheck clean.

Scope

Development only. environments/sandbox and modules/ still carry the old runner-specific script — they follow once this settles.

Not covered here

The Terraform side still has to tag instances with InspectorEc2Exclusion at launch. delete-tags ignores a key that isn't there, so the sandbox run above proves the permission exists but not that the tag was ever set. Until that lands, this profile patches early — useful on its own — but doesn't yet close the Inspector window.

🤖 Generated with Claude Code

https://claude.ai/code/session_01KLGAXtoyoakLXUjaJSL4M3