Skip to content

Commit fa18cab

Browse files
SK-3225: Split SDK into common / skyvault / flowvault; add FlowVault Node SDK (#348)
* SK-3225: Split SDK into common/skyvault/flowvault; add FlowVault Node SDK Restructure the single `skyflow-node` package into an npm-workspaces monorepo mirroring the Java SDK split (common / skyvault / flowvault). - common (private, bundled into each published package): the contract-agnostic core — errors, utils, jwt, logging, service-account auth, config/credentials, generic validations, mutable SDK identity, Base* request/response/options classes, the BaseSkyflow facade base, and the generated auth client. Builds and tests in isolation. - skyvault -> skyflow-node (2.x): the existing contract, public API unchanged (74 exports, byte-for-byte). Its Skyflow/models extend the common Base* classes; keeps connections, detect and its data-plane generated client. - flowvault -> skyflow-flowvault-node (1.x): new unary v2 contract (insert/get/update/delete/query/detokenize/getTokens/uploadFiles/deleteFiles) wired to the generated v2 client; reconciled with the Java/Python flowvault SDKs. Each layer re-exports common and bundles it (tsup) into a self-contained tarball (verified via npm pack); the mutable SDK identity makes each product self-report its own name@version. Independent versioning: plain-semver tags publish skyflow-node, flowvault/* tags publish skyflow-flowvault-node. Tests: common 232, skyvault 878 (backward compatibility preserved), flowvault 132 (~99.85% coverage). Per-package api-extractor contract snapshots gate the public API of both SDKs. PR CI builds and tests both SDKs with one command. Docs: docs/sdk-split-plan.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * SK-3225: fix PR CI (eslint ignores + gitleaks allowlist for generated/dist) - eslint.config.mjs: ignore all generated dirs (**/_generated_/**, legacy leading-space ** / _generated_/**), all workspace test dirs (**/test/**), and build output across workspaces — the old ignores only matched the single-package paths. - Replace three banned magic-string comparisons in hand-written code (typeof === "symbol"/"function") with equivalent non-literal checks. - Rule/gitleaks.toml: allowlist the maintainer-owned generated dirs (example JWTs) and the bundled dist/ output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * SK-3225: fix skyvault jest — don't babel-transform compiled common/lib CI resolves the skyflow-common symlink to its realpath (outside node_modules), so babel-jest was transforming the compiled common/lib and injecting core-js (an undeclared dep) -> suites failed to run. Add `common/lib` to transformIgnorePatterns so the already-CJS common package is loaded as-is. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * SK-3225: add flowvault branch-coverage tests to clear codecov/project Covers the defensive edge branches the codecov/project gate flagged (net-new flowvault code dipped overall coverage 0.44% vs base): - empty/absent `records`/`response` bodies map to [] across all ops - per-record errors with no x-request-id header -> requestId null - getRequestId with an absent raw response - uploadFiles: non-Error PUT rejection, no-name/trailing-dot content-type, response record with no matching prepared columns - VaultClient: default log level, unrecognized cached auth type flowvault branches 92.51% -> 96.89% (statements/lines 100%); 132 -> 153 tests. No source/generated/config changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * SK-3225: flowvault insert/update — per-record table + separate tokens/data Aligns the Node flowvault insert/update contract with the Java/Python SDKs: - InsertRecord/UpdateRecord gain an optional per-record `table`; when any record sets it, a single insert/update call targets multiple tables (the generated InsertRecordData/UpdateRecordData already carry tableName). - Insert/Update response records now expose `tokens` and `data` separately instead of a single merged `fields`. Updated unit tests (incl. multi-table coverage) and the api-report snapshot. flowvault 156 tests pass (100% lines, branches 96.97%); common 232, skyvault 878 unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * SK-3225: add flowvault README + samples (parity with python flowvault) Expand flowvault/README.md into a full SDK guide (overview, install, auth, client init, all 9 operations, custom headers, error handling) and add flowvault/samples/ — one runnable TypeScript example per operation under vault-api/, plus service-account token-generation samples — mirroring the skyflow-flowvault-python samples. All samples type-check in strict mode against the built SDK types. flowvault-only; skyvault and common untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * SK-3225: flowvault timeout/retry config + wire custom request headers Config (parity with python flowvault, limited to what the Node generated client can honor): - VaultConfig gains optional timeout (seconds), maxRetries, and vaultUrl; the flowvault VaultClient forwards timeout/maxRetries as the generated request's timeoutInSeconds/maxRetries, and vaultUrl overrides the derived URL. (connect/read/write timeouts and retry-delay knobs are fixed in generated code.) Custom headers: - options.setRequestHeaders(...) is now actually wired — merged into every operation's request headers. Added the missing options params to delete, query, getTokens, uploadFiles, deleteFiles so headers work on all 9 ops. Samples/README updated; added tests (162 pass, 100% stmts/lines). flowvault + the shared VaultConfig only; skyvault untouched; api-report unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * SK-3225: flowvault parity — per-record insert upsert + multi-table get fields Closes two parity gaps vs the Java/Python flowvault SDKs (generated client already supports both; hand-written layer only): - InsertRecord gains optional per-record `upsert`; when any record sets it, each record sends its own upsert (falling back to the request-level one) and the request-level upsert is dropped. - GetRecord (multi-table get) gains optional `columns` / `columnRedactions` / `uniqueValues`, wired per record. Added tests (164 pass, 100% lines); api-report regenerated. flowvault-only; skyvault and common untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * SK-3225: flowvault update updateType + align README/samples with python - UpdateRequest gains an optional request-level `updateType` ("UPDATE" | "REPLACE"), wired to the generated UpdateRequest.updateType (no generated change needed). Closes the last parity gap vs Java/Python flowvault. - README + samples updated to match the python flowvault SDK: per-record `upsert` on insert, multi-table get per-record columns/redactions/uniqueValues, and update `updateType`. 166 tests pass (100% lines); api-report regenerated; samples type-check. flowvault-only; skyvault/common untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * SK-3225: fix flowvault vault URL — use skyvault.* host, not vault.* FlowVault (v3) is served from {clusterId}.skyvault.skyflowapis.{dev|com| -preview.com|tech}, a different subdomain than the shared getVaultURL (vault.*, used by skyvault/v2). The facade was reusing getVaultURL, so every call 404'd. Added a flowvault-local getFlowVaultURL with the four confirmed skyvault.* hosts and used it in buildVaultEntry. flowvault-only; skyvault/common untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * SK-3225: fix flowvault insert/update table placement + per-record error extraction Two bugs surfaced running the e2e against the live vault: - Insert/update sent tableName at BOTH the request level and per-record when any record carried its own table, which the API rejects (400 "table name ... at one place"). Now the request-level tableName is omitted whenever per-record tables are used (table lives in exactly one place), matching Java/Python. - A per-record failure returned with a 4xx HTTP status (e.g. delete/get/update of a non-existent id -> 404 whose body carries a records/response array) was thrown as a whole-call error. Now those per-record records are surfaced as a normal response (via unaryError), and genuine whole-call errors still throw. Added error-extraction tests for insert/get/update/delete/detokenize/getTokens/ deleteFiles. 174 tests pass (99.4% lines). flowvault-only; skyvault/common untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * SK-3225: flowvault get — allow empty ids when uniqueValues are set validateGetRequest rejected empty ids even for a get-by-unique-value, where the lookup is by GetOptions.setUniqueValues (not ids). It now accepts a single-table get with empty ids as long as uniqueValues are provided on the options ("ids OR uniqueValues", matching Java/Python flowvault). 175 tests pass. flowvault-only; skyvault/common untouched; api-report unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * SK-3225: flowvault custom headers — typed interceptor + CustomHeaderKey Replace the arbitrary setRequestHeaders(Record<string,string>) on options with the same typed mechanism the Python and Java FlowVault SDKs expose: a RequestInterceptor set via options.setInterceptor(), which receives a RequestContext and may attach only the supported headers in CustomHeaderKey (x-skyflow-account-id, x-skyflow-account-name, x-request-id). The context carries the operation name (INSERT, GET, ...) so headers can vary per call. - New: CustomHeaderKey enum, RequestContext (+ RequestInterceptor type), FlowVaultOptions base carrying the interceptor. - All 9 *Options now extend FlowVaultOptions; controller.requestOptions runs the interceptor per operation and maps CustomHeaderKey -> wire header. - README, sample, and tests updated; contract snapshot regenerated. common and skyvault are untouched. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * SK-3225: flowvault docs — vaultUrl override uses the skyvault.* host The vaultUrl config overrides the URL the SDK actually calls, and FlowVault is served from the skyvault.* subdomain (getFlowVaultURL), not vault.*. Fix the two override examples so a user who copies them points at the right host. The clusterId derivation comments keep vault.skyflowapis.com to match the Python SDK (that is where the clusterId is read from).
1 parent 0c8a2d4 commit fa18cab

600 files changed

Lines changed: 18045 additions & 2013 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/beta-release.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,5 +12,7 @@ jobs:
1212
build-and-deploy:
1313
uses: ./.github/workflows/common-release.yml
1414
with:
15-
tag: beta
15+
product: skyflow-node
16+
package-dir: skyvault
17+
release-type: beta
1618
secrets: inherit

‎.github/workflows/common-ci.yml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,11 @@ jobs:
2222
- name: Run ESLint
2323
run: npm run eslint
2424

25+
# One command builds BOTH SDKs (common -> skyvault/flowvault via project references).
26+
- name: Build all packages
27+
run: npm run build
28+
29+
# Fans out `npm test` across every workspace (common, skyvault, flowvault).
2530
- name: Run Tests
2631
env:
2732
SA_WITHOUT_CONTEXT: ${{ secrets.SA_WITHOUT_CONTEXT }}
@@ -33,6 +38,6 @@ jobs:
3338
uses: codecov/codecov-action@v2.1.0
3439
with:
3540
token: ${{ secrets.CODECOV_REPO_UPLOAD_TOKEN }}
36-
files: coverage/lcov.info
41+
files: skyvault/coverage/lcov.info,flowvault/coverage/lcov.info
3742
name: codecov-skyflow-node
3843
verbose: true

‎.github/workflows/common-release.yml‎

Lines changed: 52 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,30 @@
11
name: Common Release Workflow
22

3+
# Reusable release pipeline, parameterized per product.
4+
# product npm package name (skyflow-node | skyflow-flowvault-node)
5+
# package-dir workspace dir (skyvault | flowvault)
6+
# release-type public | beta | internal
7+
# version semver to release (caller strips any tag prefix, e.g. flowvault/1.0.0 -> 1.0.0)
8+
39
on:
410
workflow_call:
511
inputs:
6-
tag:
12+
product:
13+
required: true
14+
type: string
15+
package-dir:
16+
required: true
17+
type: string
18+
release-type:
19+
required: true
20+
type: string
21+
version:
722
required: false
823
type: string
924

1025
jobs:
1126
build-sdk:
1227
runs-on: ubuntu-latest
13-
1428
steps:
1529
- uses: actions/checkout@v4
1630
with:
@@ -24,68 +38,68 @@ jobs:
2438
- name: Install Packages
2539
run: npm install --ignore-scripts
2640

27-
- name: Build
28-
run: npm run build
41+
- name: Resolve Version
42+
id: version
43+
run: |
44+
if [ -n "${{ inputs.version }}" ]; then
45+
echo "value=${{ inputs.version }}" >> $GITHUB_OUTPUT
46+
else
47+
# Fallback: strip an optional "<dir>/" prefix from the pushed tag.
48+
echo "value=${GITHUB_REF_NAME##*/}" >> $GITHUB_OUTPUT
49+
fi
2950
30-
- name: Get Previous Tag
31-
id: previoustag
32-
uses: WyriHaximus/github-action-get-previous-tag@v1
33-
with:
34-
fallback: 1.0.0
51+
- name: Bump Version (${{ inputs.product }})
52+
run: |
53+
chmod +x ./scripts/bump_version.sh
54+
if [[ "${{ inputs.release-type }}" == "internal" ]]; then
55+
./scripts/bump_version.sh "${{ inputs.product }}" "${{ steps.version.outputs.value }}" "$(git rev-parse --short "$GITHUB_SHA")"
56+
else
57+
./scripts/bump_version.sh "${{ inputs.product }}" "${{ steps.version.outputs.value }}"
58+
fi
59+
60+
# Build everything (common -> layers) then bundle the published package
61+
# (inlines skyflow-common into a self-contained dist/).
62+
- name: Build
63+
run: |
64+
npm run build
65+
npm run build:dist --workspace ${{ inputs.package-dir }}
3566
3667
- name: Resolve Branch Name
3768
id: resolve-branch
38-
if: ${{ inputs.tag == 'beta' || inputs.tag == 'public' }}
69+
if: ${{ inputs.release-type == 'beta' || inputs.release-type == 'public' }}
3970
run: |
4071
TAG_COMMIT=$(git rev-list -n 1 ${{ github.ref_name }})
41-
4272
BRANCH_NAME=$(git branch -r --contains $TAG_COMMIT | grep -o 'origin/.*' | sed 's|origin/||' | head -n 1)
43-
4473
if [ -z "$BRANCH_NAME" ]; then
4574
echo "Error: Could not resolve branch for the tag."
4675
exit 1
4776
fi
48-
49-
echo "Resolved Branch Name: $BRANCH_NAME"
5077
echo "branch_name=$BRANCH_NAME" >> $GITHUB_ENV
5178
52-
- name: Bump Version
53-
run: |
54-
chmod +x ./scripts/bump_version.sh
55-
if ${{ inputs.tag == 'internal' }}; then
56-
./scripts/bump_version.sh "${{ steps.previoustag.outputs.tag }}" "$(git rev-parse --short "$GITHUB_SHA")"
57-
else
58-
./scripts/bump_version.sh "${{ steps.previoustag.outputs.tag }}"
59-
fi
60-
6179
- name: Commit Changes
6280
run: |
6381
git config user.name ${{ github.actor }}
6482
git config user.email ${{ github.actor }}@users.noreply.github.com
65-
if [[ "${{ inputs.tag }}" == "beta" || "${{ inputs.tag }}" == "public" ]]; then
83+
if [[ "${{ inputs.release-type }}" == "beta" || "${{ inputs.release-type }}" == "public" ]]; then
6684
git checkout ${{ env.branch_name }}
6785
fi
68-
git add package.json
69-
if [[ "${{ inputs.tag }}" == "beta" || "${{ inputs.tag }}" == "public" ]]; then
70-
git commit -m "[AUTOMATED] Release - ${{ steps.previoustag.outputs.tag }}"
86+
git add ${{ inputs.package-dir }}/package.json ${{ inputs.package-dir }}/src/version.ts
87+
if [[ "${{ inputs.release-type }}" == "beta" || "${{ inputs.release-type }}" == "public" ]]; then
88+
git commit -m "[AUTOMATED] Release ${{ inputs.product }} - ${{ steps.version.outputs.value }}" || echo "nothing to commit"
7189
git push origin ${{ env.branch_name }}
72-
elif [[ "${{ inputs.tag }}" == "internal" ]]; then
73-
git commit -m "[AUTOMATED] Private Release ${{ steps.previoustag.outputs.tag }}-dev.$(git rev-parse --short $GITHUB_SHA)"
90+
elif [[ "${{ inputs.release-type }}" == "internal" ]]; then
91+
git commit -m "[AUTOMATED] Private Release ${{ inputs.product }} ${{ steps.version.outputs.value }}-dev.$(git rev-parse --short $GITHUB_SHA)" || echo "nothing to commit"
7492
git push origin ${{ github.ref_name }} -f
7593
fi
7694
7795
- name: Publish
96+
working-directory: ${{ inputs.package-dir }}
7897
run: |
79-
if [[ "${{ inputs.tag }}" == "beta" ]]; then
98+
if [[ "${{ inputs.release-type }}" == "beta" ]]; then
8099
npm publish --tag beta
81-
elif [[ "${{ inputs.tag }}" == "public" ]]; then
82-
MAJOR_VERSION=$(node -p "require('./package.json').version.split('.')[0]")
83-
if [[ "$MAJOR_VERSION" == "1" ]]; then
84-
npm publish --tag v1
85-
else
86-
npm publish
87-
fi
88-
elif [[ "${{ inputs.tag }}" == "internal" ]]; then
100+
elif [[ "${{ inputs.release-type }}" == "public" ]]; then
101+
npm publish
102+
elif [[ "${{ inputs.release-type }}" == "internal" ]]; then
89103
JFROG_AUTH=$(echo -n "$JFROG_USERNAME:$JFROG_PASSWORD" | base64 -w 0)
90104
npm config set registry https://prekarilabs.jfrog.io/prekarilabs/api/npm/npm/
91105
npm config set //prekarilabs.jfrog.io/prekarilabs/api/npm/npm/:_auth "$JFROG_AUTH"

‎.github/workflows/contract-tests.yml‎

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,9 @@ jobs:
2424
- name: Install dependencies
2525
run: npm ci --ignore-scripts
2626

27-
- name: Verify API surface snapshot
27+
# Builds every package, bundles the two published dist/ surfaces, then
28+
# verifies BOTH SDKs' public-API snapshots (skyflow-node + skyflow-flowvault-node).
29+
- name: Verify API surface snapshots (both SDKs)
2830
run: npm run contract-snapshot-verify
2931

3032
- name: Show API surface diff
@@ -34,12 +36,18 @@ jobs:
3436
echo "Lines prefixed with '-' were REMOVED from the public API."
3537
echo "Lines prefixed with '+' were ADDED to the public API."
3638
echo ""
37-
diff -u api-report/skyflow-node.api.md temp/skyflow-node.api.md || true
39+
echo "===== skyflow-node (skyvault) ====="
40+
diff -u skyvault/api-report/skyflow-node.api.md skyvault/temp/skyflow-node.api.md || true
41+
echo ""
42+
echo "===== skyflow-flowvault-node (flowvault) ====="
43+
diff -u flowvault/api-report/skyflow-flowvault-node.api.md flowvault/temp/skyflow-flowvault-node.api.md || true
3844
39-
- name: Upload API surface diff on failure
45+
- name: Upload API surface diffs on failure
4046
if: failure()
4147
uses: actions/upload-artifact@v4
4248
with:
43-
name: api-surface-diff
44-
path: temp/skyflow-node.api.md
49+
name: api-surface-diffs
50+
path: |
51+
skyvault/temp/skyflow-node.api.md
52+
flowvault/temp/skyflow-flowvault-node.api.md
4553
retention-days: 7
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
name: FlowVault Public Beta Release
2+
3+
on:
4+
push:
5+
tags: 'flowvault/*.*.*-beta.*'
6+
paths-ignore:
7+
- "package.json"
8+
- "package-lock.json"
9+
- "*.md"
10+
11+
jobs:
12+
build-and-deploy:
13+
uses: ./.github/workflows/common-release.yml
14+
with:
15+
product: skyflow-flowvault-node
16+
package-dir: flowvault
17+
release-type: beta
18+
secrets: inherit
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
name: FlowVault Public Release
2+
3+
# Prefixed tags select the FlowVault product: flowvault/x.y.z -> skyflow-flowvault-node.
4+
on:
5+
push:
6+
tags: 'flowvault/[0-9]+.[0-9]+.[0-9]+'
7+
paths-ignore:
8+
- "package.json"
9+
- "package-lock.json"
10+
- "*.md"
11+
12+
jobs:
13+
call-common-workflow:
14+
uses: ./.github/workflows/common-release.yml
15+
with:
16+
product: skyflow-flowvault-node
17+
package-dir: flowvault
18+
release-type: public
19+
secrets: inherit

‎.github/workflows/internal-release.yml‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,8 @@
11
name: Internal Release
22

3+
# Branch-namespaced internal (JFrog) builds, routed per product:
4+
# flowvault-release/* -> skyflow-flowvault-node
5+
# skyvault-release/* | release/* -> skyflow-node (legacy release/* kept for skyvault)
36
on:
47
push:
58
tags-ignore:
@@ -10,10 +13,14 @@ on:
1013
- '*.md'
1114
branches:
1215
- release/*
16+
- skyvault-release/*
17+
- flowvault-release/*
1318

1419
jobs:
1520
call-common-workflow:
1621
uses: ./.github/workflows/common-release.yml
1722
with:
18-
tag: internal
23+
product: ${{ startsWith(github.ref_name, 'flowvault-release/') && 'skyflow-flowvault-node' || 'skyflow-node' }}
24+
package-dir: ${{ startsWith(github.ref_name, 'flowvault-release/') && 'flowvault' || 'skyvault' }}
25+
release-type: internal
1926
secrets: inherit

‎.github/workflows/release.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,5 +12,7 @@ jobs:
1212
call-common-workflow:
1313
uses: ./.github/workflows/common-release.yml
1414
with:
15-
tag: public
15+
product: skyflow-node
16+
package-dir: skyvault
17+
release-type: public
1618
secrets: inherit

‎.gitignore‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,7 @@
11
node_modules
22
lib
3+
dist
4+
*.tsbuildinfo
35
.vscode
4-
coverage
6+
coverage
7+
temp

‎Rule/gitleaks.toml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,13 @@ paths = [
5555
'''\.gem$''',
5656
'''verification-metadata\.xml''',
5757
'''Database.refactorlog''',
58+
# Maintainer-owned generated (Fern) clients contain example JWTs/tokens in
59+
# their doc comments. Allowlist every generated dir across the workspaces
60+
# (both the plain and the legacy leading-space directory names).
61+
'''(?:^|/)_generated_(?:/|$)''',
62+
'''(?:^|/) _generated_(?:/|$)''',
63+
# Bundled publish output inlines the generated clients (example JWTs).
64+
'''(?:^|/)dist/''',
5865
]
5966
stopwords = [
6067
"abcdefghijklmnopqrstuvwxyz",

0 commit comments

Comments
 (0)