Skip to content

fix(telemetry): report hosting type and a single install id - #5676

Merged
adrians5j merged 4 commits into
nextfrom
adrian/standalone-telemetry
Sep 15, 2026
Merged

adrians5j merged 4 commits into
nextfrom
adrian/standalone-telemetry

Conversation

@adrians5j

@adrians5j adrians5j commented Sep 9, 2026

Copy link
Copy Markdown
Member

Why

Before we promote the self-hosted (server) hosting type, its telemetry has to be trustworthy. Right now it isn't:

  • CLI and admin events can't be joined. The CLI sends the per-project install id as installation_id (packages/telemetry/cli.js), while the admin app sends the identical value as project_id (packages/telemetry/react.js). Filtering admin-app-start by installation_id returns nothing, and no funnel crosses the two surfaces.
  • Self-hosted users are reported as new users forever. The newUser flag is cleared by TelemetryNoLongerNewUser, an AdminAfterDeploy hook registered only in project-aws. Self-hosted projects never deploy, so the flag stays true for the life of that machine's ~/.webiny/config and skews every newUser segment.
  • Nothing records the hosting type. hostingType ("aws" or "server") is known at scaffold time and at every CLI invocation, but no event carries it. cli-create-webiny-project-start / -end passed no properties at all, so self-hosted project creations weren't countable.

What changed

Change Where
Admin sends installation_id instead of project_id packages/telemetry/react.js
hostingType on every CLI event, from the existing WEBINY_HOSTING_TYPE marker packages/telemetry/cli.js
hostingType on every admin event, mirrored into the bundle at build/watch time packages/telemetry/react.js, packages/project/src/extensions/Project/SetAdminAppEnvVars*.ts
hostingType on cli-create-webiny-project-end and -error packages/create-webiny-project/src/features/CreateWebinyProject.ts
newUser flips on the self-hosted path, after a successful admin build packages/project-server/src/extensions/ProjectServer/TelemetryNoLongerNewUser.ts
Install handoff to webiny.com runs on any host, not just CloudFront packages/app-admin/src/presentation/installation/components/SystemInstaller/steps/FinishSetup/handleStartUsing.ts

The localStorage key behind the install id stays wts_project_id. Renaming it would mint a fresh id for every admin session that already has one.

Decisions worth reviewing

cli-create-webiny-project-start deliberately has no hostingType. It fires before the hosting-type prompt runs. Sending the placeholder default would inflate "aws", so -start stays the funnel denominator and -end / -error carry the choice. If the run fails before the prompt, -error reports hostingType: "unknown" rather than guessing.

newUser flips on admin build, not serve. A successful admin build is the closest self-hosted analogue of the AWS milestone: same app, point at which the user has something working. It's also the one step every self-hosted path runs, including the container path below.

No CLI-side "it's running" event for self-hosted. An earlier revision of this PR added cli-project-serve-start. Dropped in 5e60851: serve is a convenience for zero-infra self-hosting, not the production path. The deploy artifact ships build/start.mjs, so the common route is to build both apps, copy build/ into a container and run node start.mjs, which never touches the CLI. That event would have measured the dev convenience and read as low adoption whenever people are in Docker. admin-app-start and the install-wizard events cover the same question and do fire in the container path, since hostingType and installation_id are baked into the admin bundle at build time.

If we do want a CLI-side signal later, build is the honest place for it: every self-hosted path runs it. It's hosting-agnostic though, so it would change AWS event volume too and would need care not to fire on every watch rebuild. Worth deciding separately.

Follow-ups, not in this PR

Resolved during review:

  • deploymentType: "standalone" was a false alarm. It doesn't exist anywhere in webiny-js (git log --all -S "deploymentType" returns nothing), nor on wts-server's v3 branch, and self-hosted can't emit cli-project-deploy-* at all, since that decorator lives in cli-aws and webiny-server has no deploy command. Sven traced it: PostHog's AI had picked allow-local-state-files: 1 as the filter for "standalone", so the deploy events under it were never self-hosted events.
  • The project_id to installation_id rename is safe end to end. Checked all three consumers rather than assuming: wts-server (v3) spreads ...event.properties through verbatim in posthog.ts:55, with no allowlist and no property remapping, so the value simply arrives under the new name; the website's /install/finish page reads only the machine_id query param and the wts_did cookie; and per Sven, nothing in PostHog uses either property today. No dashboard migration needed.
  • Joining events per project isn't needed yet, so cli-create-webiny-project-* keeps going out without an installation_id. Worth knowing why it's absent rather than empty: the CLI reads it from <cwd>/package.json, and during scaffolding the cwd is still the parent directory.

Not in scope, and worth stating because the name is misleading: the WCP telemetry client that WcpInjectTelemetryClientAfterBuild injects into the api handler is a different system from the one in this PR. It reports to the WCP API (app.webiny.com) so customers can see their own API call counts. It is usage metering, not product analytics, and it never touches t.webiny.com or PostHog.

Now covered, with a release-order constraint:

  • Self-hosted installs can be aliased to the website session. The "Start using Webiny" CTA routes through webiny.com/install/finish so the site's anonymous wts_did cookie can be aliased to the deployer's machine id, which is what links "read the docs" to "installed Webiny". It only ran when the admin was served from a .cloudfront.net host, so self-hosted installs were never aliased. That check is gone, and the website drops its matching restriction on return_to, so both halves of the handoff work for any host.

    ⚠️ The website change has to be live before this ships. Until it is, a non-CloudFront admin sends the user to /install/finish and the page answers "Invalid request" instead of redirecting back, dead-ending the install wizard. Everything else in this PR is independent of that, so if the website isn't ready, drop commit 3ab9a3a4f1 and merge the rest.

    The website-side restriction exists to stop /install/finish?return_to=https://evil.com turning webiny.com into a phishing redirector, and a self-hosted admin can live on any hostname, so there's no allowlist pattern that covers it. Worth being explicit about what removing it accepts: unvalidated origins can be redirected to, and anyone can POST arbitrary machine_id values and pollute the alias graph. The second was already true for anything CloudFront-shaped.

Test plan

  • yarn build (full monorepo) passes.
  • yarn adio, yarn lint, yarn format, yarn webiny sync-dependencies all clean.
  • No automated tests exist for @webiny/telemetry. Verifying the emitted payloads means creating a project on each hosting type and watching the requests to t.webiny.com, which I haven't done yet.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Installation completion can now hand off to the marketing site from any host when the required browser information is available.
    • Hosting type is now carried through setup, build, watch, and telemetry workflows.
    • Admin telemetry now records installation identifiers and hosting type when available.
    • Successful admin builds update installation status so subsequent builds do not repeat the transition.
  • Bug Fixes

    • Improved analytics accuracy by avoiding placeholder hosting values during setup and reporting unknown only when the hosting type cannot be determined.

@github-actions

github-actions Bot commented Sep 9, 2026

Copy link
Copy Markdown

🚓 Slop Cop

✅ Nothing worth flagging. The diff looks consistent with the PR's stated intent and the code-style rules.

The diff matches the stated intent (telemetry hosting-type/install-id fixes plus the disclosed install-handoff change), is small and coherent, and shows no integrity red flags or code-style rule violations in the added lines.

Automated, non-blocking heads-up from an LLM. It can be wrong — use your judgment. Regenerates on every push.

t and others added 3 commits September 11, 2026 10:36
Self-hosted (server) projects were effectively untrackable, and CLI events
could not be joined to admin events.

- Admin events now send `installation_id`, the name the CLI already uses. They
  previously sent the same value as `project_id`, so nothing joined the two
  surfaces. PostHog queries on `project_id` need updating.
- Every CLI and admin event carries `hostingType` ("aws" or "server"), read
  from the marker both CLI bins already set.
- `cli-create-webiny-project-end` and `-error` carry `hostingType` too. The
  `-start` event does not, because it fires before the user picks one.
- New `cli-project-serve-start` (plus error variants) for self-hosted projects,
  which have no deploy command to report.
- The `newUser` flag now flips for self-hosted projects, after a successful
  admin build. It only hung off `AdminAfterDeploy`, which self-hosted never
  reaches, so those users were reported as new users forever.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`serve` is a convenience for zero-infra self-hosting, not the production path.
The self-hosted deploy artifact ships `build/start.mjs`, so the common route is
to build the api and admin apps, copy `build/` into a container and run
`node start.mjs`, which never touches the CLI. Counting `serve` would measure
the dev convenience and read as low adoption whenever people are in Docker.

What covers the funnel instead: `admin-app-start` and the install-wizard events
now carry `hostingType` and `installation_id`, both baked into the admin bundle
at `webiny build admin` time, so they report in the container path too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…eads

Matches `cli.js`, which already declares each optional property group and fills
it with a plain `if`. Per ai-context/code-style/no-inline-conditional-spreads.md.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@adrians5j
adrians5j force-pushed the adrian/standalone-telemetry branch from 525ecd4 to 078d850 Compare September 11, 2026 08:38
The "Start using Webiny" CTA routes through webiny.com/install/finish so the
site's anonymous wts_did cookie can be aliased to the deployer's machine id.
It only did that when the admin was served from a `.cloudfront.net` host, so
self-hosted installs were never aliased and install-to-marketing attribution
was AWS-only.

The website drops its matching CloudFront check on `return_to`, so both halves
of the handoff work for any host.

Sequencing: the website change has to be live first. Until it is, a non
CloudFront admin sends the user to /install/finish and the page answers
"Invalid request" instead of redirecting back, which dead-ends the install
wizard.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Sep 15, 2026

Copy link
Copy Markdown

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: e3ce1e6f-252d-4633-b6da-82f4df54c6ca

📥 Commits

Reviewing files that changed from the base of the PR and between a219f4e and 3ab9a3a.

⛔ Files ignored due to path filters (1)
  • yarn.lock is excluded by !**/yarn.lock, !**/*.lock
📒 Files selected for processing (9)
  • packages/app-admin/src/presentation/installation/components/SystemInstaller/steps/FinishSetup/handleStartUsing.ts
  • packages/create-webiny-project/src/features/CreateWebinyProject.ts
  • packages/project-server/package.json
  • packages/project-server/src/extensions/ProjectServer.tsx
  • packages/project-server/src/extensions/ProjectServer/TelemetryNoLongerNewUser.ts
  • packages/project/src/extensions/Project/SetAdminAppEnvVarsBeforeBuild.ts
  • packages/project/src/extensions/Project/SetAdminAppEnvVarsBeforeWatch.ts
  • packages/telemetry/cli.js
  • packages/telemetry/react.js

Included review availability: Your plan provides up to 10 included reviews per hour; 8 remain after this review.


📝 Walkthrough

Walkthrough

Changes

Telemetry installation lifecycle

Layer / File(s) Summary
CLI hosting telemetry and environment propagation
packages/create-webiny-project/..., packages/telemetry/cli.js, packages/project/...
Interactive setup reports hosting type after resolution. CLI telemetry and admin build environments include the configured hosting type.
Browser telemetry identity and event fields
packages/telemetry/react.js
Admin telemetry uses the installation identifier and conditionally reports installation and hosting fields.
Post-build new-user state update
packages/project-server/...
The project server registers an admin post-build hook that sets the global newUser flag to false after a successful build.
Telemetry-based installation handoff
packages/app-admin/...
Eligible installation handoffs no longer require a CloudFront host or an alternate finish URL.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant CreateWebinyProject
  participant telemetry_cli
  participant ProjectBuildHooks
  participant telemetry_react
  participant ProjectServer
  CreateWebinyProject->>telemetry_cli: send hosting-aware project event
  ProjectBuildHooks->>telemetry_react: provide REACT_APP_WEBINY_HOSTING_TYPE
  telemetry_react->>telemetry_react: resolve installationId and emit admin event fields
  ProjectServer->>ProjectServer: set global newUser to false after admin build
Loading
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 8…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main telemetry changes: reporting hosting type and using a single installation ID.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch adrian/standalone-telemetry

Comment @coderabbitai help to get the list of available commands.

@adrians5j
adrians5j merged commit 26ff4c8 into next Sep 15, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant