Skip to content

feat: Emit worker pool size as a DEBUG log event during init on Lambda Managed Instances - #219

Open
vip-amzn wants to merge 2 commits into
aws:mainfrom
vip-amzn:feat/lmi-worker-pool-observability
Open

feat: Emit worker pool size as a DEBUG log event during init on Lambda Managed Instances#219
vip-amzn wants to merge 2 commits into
aws:mainfrom
vip-amzn:feat/lmi-worker-pool-observability

Conversation

@vip-amzn

@vip-amzn vip-amzn commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Description

On Lambda Managed Instances,
the runtime sizes and spawns a worker pool during initialization to handle
multiple concurrent invocations per execution environment. Today there is no
way to observe the effective pool size: it can be influenced by configuration,
but nothing confirms what the runtime actually resolved.

This change adds a one-time structured JSON log event, emitted at DEBUG level
during runtime initialization, reporting the resolved worker pool size and the
maximum concurrency the execution environment supports. Both values are
included so the worker count is not mistaken for the supported concurrency.

Example (visible when the function's application log level is DEBUG or lower):

{
  "timestamp": "...",
  "level": "DEBUG",
  "message": {
    "event": "runtime_worker_pool_initializing",
    "workerCount": 17,
    "executionEnvironmentMaxConcurrency": 34
  }
}

Design notes:

  • Emitted exactly once per execution environment, during init — never per invoke.
  • Opt-in by design: routed through the runtime's standard log-level filtering,
    so it only appears when the application log level is set to DEBUG or lower.
    Default configurations see no new log lines and no added CloudWatch cost.
  • No new dependencies and no eager loading: emission reuses logging machinery
    already initialized during bootstrap.
  • No behavior change on the standard (single-concurrency) compute type; that
    path is untouched.
  • The nested message object is queryable in CloudWatch Logs Insights
    (e.g. filter message.event = "runtime_worker_pool_initializing").

See also: Python runtime for Lambda Managed Instances

Testing

  • Full unit test suite passes; new tests cover: emitted exactly once with the
    correct schema and values, suppressed at INFO and above, and not emitted on
    the single-concurrency path.
  • Validated end to end on Lambda Managed Instances: event appears once per
    execution environment at DEBUG, never appears at INFO or above, and repeated
    sequential/concurrent invocations neither re-emit the event nor affect
    normal invocation logs.

@vip-amzn
vip-amzn requested review from maxday and trivenay September 2, 2026 14:41
@maxday

maxday commented Sep 3, 2026

Copy link
Copy Markdown
Member

Nice! My one real concern is the change in JsonFormatter.format: it now passes a raw dict message straight into a JSON encoder that has no fallback, and this applies to every log call, not just the new event.

So if anyone logs a dict that contains a non-serializable value (a datetime, Decimal, bytes, a custom object, etc.), formatting will raise and the log line gets dropped, whereas today getMessage() just stringifies it safely.

Could we add default=str to the encoder (or a try/except that falls back to the old behavior) so we don't regress existing dict logging?

@maxday

maxday commented Sep 3, 2026

Copy link
Copy Markdown
Member

Also, the sink that init_logging() opens is never closed, I think the fd leaks for FramedTelemetryLogSink since handlers.clear() doesn't close it?

Address PR review feedback:
- Add default=str to the JSON log encoder so dict messages (and extra
  attributes) containing non-serializable values are stringified instead
  of raising and dropping the log record.
- Close the log sink opened by init_logging deterministically after the
  parent's handler is removed, instead of relying on GC.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants