Re-running `vcspull import` requires restating every flag

[tony]

Summary

A reconciling import records where an entry came from, but not how it was selected. metadata.imported_from stores only a source tag such as "gitlab:mygroup" — not --mode, --workspace, --limit, --https, or any --skip-group value. Re-running the same import therefore means restating the full flag set exactly, so teams wrap vcspull import in a shell script and distribute that script to every developer.

Motivation

A team keeps roughly 25 customer repositories plus several shared ones in one GitLab group, arranged in a directory layout their local test setup depends on. Onboarding and a daily refresh run the same import, so the invocation lives in a wrapper script every developer copies:

$ vcspull import gitlab mygroup \
    --file ~/team.yaml \
    --mode org \
    --limit 0 \
    --skip-group documentation \
    --skip-group presales \
    --skip-group models \
    --workspace ~/code/mygroup \
    --sync \
    --yes

Nothing written to ~/team.yaml records any of it. Each entry carries one provenance field and no criteria:

~/code/mygroup/:
  api-server:
    repo: "git+ssh://git@gitlab.com/mygroup/api-server.git"
    metadata:
      imported_from: "gitlab:mygroup"

The script is not a convenience wrapper. It is the only place the selection criteria exist, every developer holds a copy, and the copies drift apart with nothing to detect it.

Dropping a flag on a later run is silent, and under --sync it rewrites the config. Omitting one --skip-group pulls that subgroup in. Mistyping --workspace is worse: the prune step walks every workspace in the file and removes anything carrying this source's tag that the run did not just fetch, so the previous workspace section is deleted and its repositories reappear under the new label, each of them now missing from disk at the path sync expects.

Omitting --limit 0 caps the fetch at the default of 100. The importer logs a truncation warning, the reconcile step then treats the truncated result as the remote's full contents, and entries past the cap are pruned. The warning is raised by the remote client while the prune decision is made in _run_import, and nothing connects the two. That point is read from source rather than reproduced against a group larger than the cap, and may be worth its own issue regardless of what happens here.

Proposal

Persist an import's selection criteria alongside the entries it produced, so a later refresh replays them without the operator knowing what the first run was. The shape is open — a per-source block in the config file, a separate sources document, or whatever fits the config-directory layout in #361.

Load-bearing constraints:

• A replay against an unchanged remote selects the same repositories the original invocation selected.
• Replaying does not require the operator to know or restate the original flags.
• A config written before this exists keeps importing and syncing unchanged.
• No credential is written to the config. Tokens stay in the environment.

Those four are the non-negotiable part. Everything else is open, including whether the criteria live in the repository config at all.

Acceptance:

• After an initial import, a refresh of the same source runs with no repetition of mode, workspace, limit, protocol, or exclusion flags.
• That refresh reconciles the same entry set the original flags produced, exclusions included.
• --dry-run reports which stored criteria a replay would apply, so a wrong or stale one is visible before anything is written.
• Two developers refreshing the same source against an unchanged remote converge on the same config without sharing a script.

Not doing: reading credentials from config, scheduling, or making import clone repositories.

Alternatives

Keep the wrapper script. It works today and asks nothing of vcspull, at the cost of every developer carrying a private copy of parameters the tool already had in hand once.

References

• Provenance is stamped as a single source tag: _common.py#L1051
• The reconcile and prune step that consumes it, walking every workspace in the file: _run_import
• Truncation is a warning on the remote client, not an input to prune: gitlab.py#L400
• Provenance tracking, documented: vcspull import
• vcspull import was planned in Import remote repos, ghorg-like fetching behavior #416 and shipped in feat: vcspull import #510; Settings file #361 proposes the config-directory layout this would live in