Skip to content

✨ Feature: add gfs clone <source> <target>, and move the lazy-clone commands under gfs lazy - #201

Draft
medazizktata25 wants to merge 5 commits into
Guepard-Corp:mainfrom
medazizktata25:feat/gfs-clone-from-snapshot
Draft

medazizktata25 wants to merge 5 commits into
Guepard-Corp:mainfrom
medazizktata25:feat/gfs-clone-from-snapshot

Conversation

@medazizktata25

Copy link
Copy Markdown
Contributor

Related Issue

None filed.

What

Standalone gfs cannot make a new, independent database from a chosen snapshot of an existing one. It only has the storage primitive (gfs storage clone, which takes paths). Meanwhile the plain word clone is taken by the special case: gfs clone --from <url> is the lazy copy-on-read clone of an external database.

This PR adds gfs clone <source> <target> [--snapshot <id> | --branch <name>]. It is a full, independent copy of the database at <source>, taken from its latest snapshot by default, a given snapshot, or a branch tip. The lazy-clone commands move under one group:

Before After
gfs clone --from <url> [--snapshot] gfs lazy clone <url> [--freeze]
gfs fetch gfs lazy status
gfs pull gfs lazy pull
gfs remote gfs lazy source
gfs freeze gfs lazy freeze

The old spellings keep working for one release. They are hidden from help and print a rename warning on stderr only, so exit codes and --json stdout are unchanged.

How

Five commits, meant to be reviewed one at a time:

  1. feat(cli): group the lazy-clone commands under gfs lazy. Moves the lazy-only commands under gfs lazy and keeps the old spellings as hidden aliases.
  2. refactor(init): split creating a repository from starting its database. run() becomes create_repository() then start_database(), with no behaviour change, so a caller can fill the workspace before the first start. The record-before-start ordering on main stays inside start_database.
  3. feat(clone): gfs clone <source> <target> copies a database from a snapshot. Adds a new CloneFromSnapshotUseCase:
    • The snapshot is copied into the new workspace before its database first starts, using the same restore checkout uses (now one shared helper).
    • The copy starts on the source's provider, version, compute settings and saved credentials.
    • It begins with one commit naming its source, and carries none of the source's history.
    • Everything that can be refused is checked before the target exists, so a refusal leaves nothing behind: not a repository, no database, no snapshots, unknown id, missing snapshot data, existing target.
  4. feat(clone): gfs clone on Kubernetes seeds the volume before first start. On Kubernetes a snapshot is a VolumeSnapshot:
    • The new instance's data PVC is created from it before the StatefulSet exists, and the source's credentials Secret is adopted, so no blank database ever boots.
    • New port method Compute::provision_seeded; its default refuses. init gains start_database_seeded beside an unchanged start_database.
    • gfs clone refuses to cross runtimes.
  5. fix(clone): a seeded provision that fails removes the volume it made. If adopting the Secret or creating the StatefulSet fails after the PVC exists, nothing records the instance yet, so no later teardown could find it. It now tears down what this call made, keeping the source's snapshot.

Review Guide

  1. crates/applications/cli/src/lib.rs: the Lazy group, the hidden aliases, warn_renamed.
  2. crates/domain/src/usecases/repository/init_repo_usecase.rs: the split. Diff it against main to see the body is moved, not changed.
  3. crates/domain/src/usecases/repository/clone_from_snapshot_usecase.rs and cli/src/commands/cmd_clone_from_snapshot.rs: the new command.
  4. crates/domain/src/adapters/gfs_repository.rs: restore_workspace_from_snapshot, now shared by checkout and clone, plus seed_head_workspace.
  5. crates/adapters/compute-kubernetes/src/lib.rs, provision_seeded.

Testing

On Rust 1.93.1 (the pinned toolchain), macOS with Colima for Docker, using gfs-postgres:16/:17 images built from this tree.

  • cargo fmt --check: clean.

  • cargo clippy --locked --all-targets --all-features -- -D warnings: clean, run on Linux (rust:1.93.1-bookworm), as CI does, after the last commit.

  • Full cargo test --locked at commit 4: 884 passed, 2 failed, 3 ignored, across 60 test binaries, with no e2e test skipped for a missing image. Both failures fail identically on main (d5a0e83):

    • clone_range_and_temporal_hydration_elide: the test predates the asynchronous temporal copy; the fix is on its own branch and will come as a separate PR;
    • postgres_checkout_lifecycle: hint bits read as uncommitted work; the fix is on its own branch and will come as a separate PR.
  • After commit 5: gfs-domain (433) and gfs-compute-kubernetes (68) pass. The full suite was not re-run, since commit 5 only touches provision_seeded.

  • New e2e e2e_clone_from_snapshot (3 tests: SQLite, Postgres with custom credentials, and every refusal leaving no target):

    • all pass;
    • with the snapshot copy disabled, the two data tests fail and the refusal test still passes.
  • lazy_commands_have_one_home_and_the_old_names_still_work: fails when an old spelling is un-hidden, when its warning is dropped, or when the warning goes to stdout.

  • Init, unit tests:

    • 14 pass, including a new a_failed_start_leaves_a_seeded_instance_recorded;
    • with the seeded path starting before it records the instance, that test fails;
    • with run() skipping start_database, 7 of 13 existing tests fail.
  • Live on k3s (Ubuntu 24.04 arm64, openebs-zfs StorageClass and VolumeSnapshotClass): init, two commits, then gfs clone src copy --snapshot <first>. All 10 checks pass:

    • the clone took 8 s and created exactly one new StatefulSet;
    • the copy records its instance and mount point;
    • the data PVC's dataSource is the VolumeSnapshot, and the container never restarted;
    • the copy holds only the first commit's rows and has one commit;
    • checkout works on the copy, and the source is untouched;
    • destroy leaves nothing.

    Calibration: a build whose provision_seeded ignores the seed fails 3 checks (PVC not from the snapshot, copy empty, checkout finds no data).

  • Commit 5, live: a namespace ResourceQuota stops the StatefulSet from being created. Before the fix, the refused clone left {instance}-data behind; with it, nothing is left. The 10-check run above also passes with commit 5.

  • Not tested:

    • Docker Desktop, podman, a Linux Docker host, Windows;
    • MySQL and ClickHouse sources (only Postgres and SQLite);
    • cloning a lazy clone (the command says the copy stays lazy; no test covers it);
    • Kubernetes other than k3s with OpenEBS ZFS.

Documentation

  • README updated for gfs lazy … and gfs clone.
  • Not updated:
    • the website page crates/applications/website/src/pages/docs/commands/clone.rs still shows gfs clone --from (that spelling still works, with a warning);
    • CHANGELOG.

User Impact

  • New: gfs clone <source> <target> makes an independent copy from a snapshot. On Kubernetes it is a VolumeSnapshot clone with no blank boot.
  • Renamed: the lazy commands, under gfs lazy. Old spellings work for one release and warn on stderr. Removing them in the next minor release is the breaking change; this PR is not one.
  • Fixed: a seeded Kubernetes clone that fails part-way no longer leaves a PVC behind.

Checklist

  • Self-review completed
  • Tests pass locally (cargo test). Two fail, identically on main; see Testing.
  • Clippy passes (cargo clippy --all-targets --all-features -- -D warnings), on Linux.
  • Format check passes (cargo fmt --check)
  • This PR can be safely reverted. Reverting removes gfs clone <source> <target> and the gfs lazy spellings; the old lazy spellings work either way.

clone --from, fetch, pull, remote and freeze only work on a lazy clone
but sat at the top level as if they applied to every database, and they
held the word `clone` for the special case. They now live under
`gfs lazy clone|status|pull|source|freeze`; `lazy clone` takes the URL
as an argument and --freeze instead of --snapshot.

The old spellings stay for one release, hidden from help, with the same
exit codes and stdout and a rename warning on stderr only, so --json
consumers see no change.
run() is now create_repository() then start_database(), unchanged in
behaviour, so a caller can fill the workspace before the first start.
…pshot

A new, independent database holding the data of one snapshot of
another: --snapshot <id> or --branch <name>, the latest by default. The
snapshot is copied into the new workspace before its database first
starts, with the same restore checkout uses, and the database starts on
the source's provider, version, compute params and credentials. The
copy begins with one commit naming its source; it carries none of the
source's history.

Everything that can be refused (not a repository, no database, no
snapshots, unknown id, snapshot data missing, existing target) is
checked before the target is created, so a refusal leaves nothing.
Cloning a lazy clone gives a lazy copy, and the command says so.

`gfs clone --from <url>` still reaches the lazy clone for one release.
On Kubernetes a snapshot is a VolumeSnapshot, not a directory. The new
instance's data volume is now created from the source commit's
VolumeSnapshot before the instance exists, and the source's credentials
Secret is copied to it, so no blank database ever boots and nothing is
torn down and swapped underneath it.

Compute::provision_seeded (default: refuse) carries this; init gains
start_database_seeded beside start_database, whose signature is
unchanged. gfs clone picks the volume path when the source runs on
Kubernetes, refuses to cross runtimes, and records the copy's data
volume as init does. volumesnapshot_name_for_hash is now the one place
that names a snapshot's VolumeSnapshot.
provision_seeded creates the instance's data volume from the source
snapshot, then adopts the credentials Secret and creates the
StatefulSet. If either later step failed, the error returned with the
volume in place and nothing recording the instance, so no teardown
could find it: the PVC stayed until removed by hand.

Seen on k3s with a namespace quota that refuses the StatefulSet
("exceeded quota"): the clone failed and {instance}-data was left
behind. A failure after the volume exists now tears down what this
call made (StatefulSet, Service, data PVC, adopted Secret), keeping
the snapshot, which belongs to the source. Same run with this change:
the clone fails and no PVC is left; a normal clone is unchanged.

This branch is waiting to be deployed

1 waiting deployment
pr-validation — 2395e768 Waiting Oct 27, 2026 by medazizktata25 via build-and-test (ubuntu-latest) #296
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant