Apache Iggy is a persistent message streaming platform
in Rust. Thread-per-core shared-nothing, io_uring + compio.
Transports: QUIC, WebSocket, TCP (custom binary), HTTP (REST). SDKs:
Rust, .NET, Java, Python, Go, C++, Node.js. A connectors subsystem
ingests from / egresses to external systems via dlopened plugins.
Skills live under
.claude/skills/, with per-skill links under.agents/skills/for Codex. Load connectors-overview first for any change undercore/connectors/. Other subsystems follow the repo-wide principles in this file.
- Writing code for an issue not assigned to the user
(
gh issue view <N> --json assignees). If it isn't, or you can't confirm it, they must ask on the issue and wait to be assigned. - Bumping
iggy_connector_sdkMAJOR version or changing any FFI signature incore/connectors/sdk/src/{sink,source}.rs(breaks every pre-built plugin .so). - Modifying
core/binary_protocol/wire-format types tagged#[repr(C)](breaks every running client). - Touching the state-save path (
core/connectors/runtime/src/state.rs) in a way that changes the atomic-rename protocol (corruption risk). - Renaming default consumer groups, plugin path resolution rules, or any other operator-facing identifier.
- Force-pushing to
master, amending shipped commits, or skipping hooks with--no-verify/--no-gpg-sign.
Most PRs from people new to Iggy are written with an agent. The person opening the PR is responsible for it, and maintainers close PRs that read as a relay between reviewer and model. Read the AI Assistance, Bugs Found by an Agent and Close Policy sections of CONTRIBUTING.md before helping open a PR.
- The PR must link an issue the user was assigned to before any work started. See the STOP list above.
- Fill in every section of the PR template. Under AI Usage, name the agent and say what it wrote. Never leave the section out or write "None" when an agent wrote any of the change.
- Run
prek runbefore the PR is opened. If prek isn't installed, ask the user to install it (cargo install prek, thenprek install). Don't substitute the individual checks. - If a check can't run, tell the user which one and why, and treat the PR as not ready. Never write text in a PR asking a maintainer to run something.
- The user should write the rationale in their own words and be able to explain every change without asking you.
- New contributors keep one open PR at a time.
- If you found a bug, the first PR holds a failing test. The fix waits until the approach is agreed on the issue.
# Verification order is enforced by CI (.github/actions/rust/pre-merge).
cargo fmt --all # step 1
cargo sort --no-format --workspace # step 2 (--no-format mandatory)
cargo clippy --all-features --all-targets -- -D warnings # step 3
cargo test -p <crate> # step 4
# Other CI checks (mirror in pre-commit; pass --fix where supported)
./scripts/ci/taplo.sh # TOML
./scripts/ci/markdownlint.sh # Markdown
./scripts/ci/shellcheck.sh # Shell
./scripts/ci/license-headers.sh # Apache headers
./scripts/ci/trailing-whitespace.sh
./scripts/ci/trailing-newline.sh
./scripts/ci/sync-rustc-version.sh
./scripts/ci/python-sdk-version-sync.sh
./scripts/ci/uv-lock-check.sh
./scripts/ci/binary-artifacts.sh --check
./scripts/ci/third-party-licenses.sh --validate --manifest <path>
# Run things
cargo run --bin iggy-server -- --with-default-root-credentials
cargo run --bin iggy -- <subcommand>
IGGY_CONNECTORS_CONFIG_PATH=... cargo run --bin iggy-connectors
./scripts/run-bdd-tests.sh
./scripts/run-benches.shPre-commit hooks wire the same scripts. Install with pre-commit install (Python) or prek install (Rust drop-in) - both read
.pre-commit-config.yaml.
iggy/
├── core/
│ ├── server/ Iggy server binary (Viewstamped Replication)
│ ├── sdk/ Rust client SDK
│ ├── cli/ iggy CLI
│ ├── connectors/ Connectors runtime + SDK + sinks/sources
│ ├── common/ Shared types (IggyDuration, IggyTimestamp, ...)
│ ├── binary_protocol/ Wire format (stable)
│ ├── configs/ Config plumbing + ConfigEnv derive
│ ├── ai/mcp/ MCP server
│ ├── bench/ Benchmark suite
│ ├── integration/ Cross-crate integration tests
│ ├── simulator/ Deterministic simulator
│ ├── consensus/ Raft (Miri-checked unsafe iobuf)
│ ├── journal/ WAL / journal
│ ├── message_bus/ In-process bus
│ ├── shard/ Shard runtime
│ ├── harness_derive/ #[iggy_harness] proc macro
│ └── metadata, partitions, server_common, configs_derive, clock, tools
├── foreign/ Java / .NET / Python / Go / C++ / Node.js SDKs
├── bdd/ Cross-language BDD (Gherkin)
├── examples/ Per-SDK runnable examples
├── helm/ Helm charts
├── web/ Web UI
└── scripts/ CI + release + dev scripts
| Task | Location |
|---|---|
| Wire protocol | core/binary_protocol/ |
| Server | core/server/src/ |
| Rust client SDK | core/sdk/src/ |
| Connectors | core/connectors/ -> connector-* skills |
| Integration tests | core/integration/tests/ |
| Cross-SDK BDD | bdd/ |
| Benchmark suite | core/bench/ |
Each SKILL.md has YAML frontmatter (name, description). For
connectors work, load connectors-overview first as router.
Claude Code uses /skill-name; Codex uses $skill-name or /skills.
Keep one source under .claude/skills/ and link its directory from
.agents/skills/. Keep templates and references inside the source directory.
- connectors-overview - router + universal connector rules
- connector-runtime - FFI host, lifecycle, state, metrics
- connector-sdk - traits, decoders/encoders, retry
- connector-sink - sink plugin authoring
- connector-source - source plugin authoring
- connector-transform - transform authoring
- connector-testing - unit + integration test patterns
Review skills are user-invoked only. Keep disable-model-invocation: true
for Claude Code and policy.allow_implicit_invocation: false in
agents/openai.yaml for Codex. Do not replicate their steps without an
explicit invocation. When a non-trivial change passes verification,
suggest /team-review <target> in Claude Code or $team-review <target>
in Codex.
- connector-review - adversarial connectors PR/branch review with validation
- team-review - adversarial 4-expert PR/branch review, ~10 subagents per run
- team-review-slim - four-expert review without independent validation rounds
Review skills share execution instructions
for target preparation and client-specific agent tools. Run
./scripts/ci/skills.sh to check skill metadata, links, and invocation policies.
- Apache 2.0 header on every new source file: Follow the comment style configured in
licenserc.toml; common examples:.rsuses// ...,Cargo.tomland shell use# .... - Verification order:
fmt --all->sort --no-format --workspace->clippy --all-features --all-targets -- -D warnings->test. Plustaplo.sh --check. CI enforces all. - Always pass
--no-formattocargo sort. Without it multi-line feature arrays flatten andtaplore-expands them, churning the workspace. - Avoid LLM-slop tells: em dashes, gratuitous semicolons, hedging narrative, trailing summaries. Defaults, not absolute bans. Same rule for code, comments, commits, PR descriptions.
- Idiomatic Rust over free helpers.
FromStrnotparse_foo,Displaynotformat_foo,From/TryFromnotto_foo_from_bar. StructuredErrenums, notString. - Imports at the top. No inline
useinside functions or blocks. Group: std, external crates, thencrate::. - Precise names. No
b,p,t,m. Match literal API field names as log labels. - Comments explain WHY, not WHAT. Default to no comment. Add one only when a hidden constraint or workaround would surprise a future reader. Never reference the current task / PR.
- Never
cargo installwithout authorization. Toolchain pinned inrust-toolchain.toml. CI installs additional cargo tools on demand. If a required tool such as prek is missing, ask the user to install it rather than skipping the check. - No
unwrap()/expect()on Results from external I/O outside tests. tokio::sync::Mutex(notstd::sync::Mutex) wherever a lock is held across.await. Hold locks briefly: lock, clone, drop guard, then do I/O.- Code reads top to bottom. Public consts and public fns first, helpers below, deeper helpers below those.
- Zero clone in hot loops.
&self,&str,&[T].Vec::with_capacity(n)when size is known. - Forward-compatible config. New TOML fields use
#[serde(default)]orOption<T>.
| Tool | Where |
|---|---|
cargo fmt, cargo sort, cargo clippy |
pre-commit + CI |
cargo machete |
CI machete task |
cargo nextest + cargo-rail |
CI test-* DAG-scoped |
cargo llvm-cov |
CI coverage |
cargo http-registry |
CI verify-publish |
| Miri | CI miri (binary_protocol + consensus) |
taplo, markdownlint-cli, shellcheck |
pre-commit + CI |
typos |
pre-commit |
cargo about + license-checker-rseidelsohn |
scripts/ci/third-party-licenses.sh |
HawkEye / license-headers.sh |
pre-commit + CI |
uv, ruff |
pre-commit (Python) |
golangci-lint |
pre-push (Go) |
- Unit:
#[cfg(test)] mod testsat EOF ofsrc/lib.rs(or per-module). - In-crate integration:
<crate>/tests/. - Cross-crate (real infra):
core/integration/tests/. Harness incore/integration/src/harness/spawns realiggy-server(ServerHandle), connectors runtime (ConnectorsRuntimeHandle), MCP server (McpHandle), client (ClientHandle).#[iggy_harness]proc macro wires fixtures. Suites:cli/,cluster/,config_provider/,connectors/,data_integrity/,mcp/,sdk/,server/,state/,storage/. External backends insideconnectors/usetestcontainers-modules. Per-test logs land intest_logs/. - Cross-SDK BDD:
bdd/. Gherkin features against every SDK. Driver:scripts/run-bdd-tests.sh.
BDD test naming: given_X_when_Y_should_Z (3-part). Promote to 4-part given_X_when_Y_then_Z_should_W only with a distinct "then" intermediate. Be consistent inside a file.
Filter a single integration test by path. Build the binaries the harness launches first: it runs
whatever is already in target/, so a stale one fails as though the change under test broke it.
cargo build --bin iggy-server --bin iggy-connectors
cargo test -p integration -- connectors::runtime::benchmark::given_logging_format_json- Docker is required for most integration tests.
testcontainers-modulesspawns containers per test. cargo sortwithout--no-formatwill churn the workspace. Use the pre-commit hook.- Rust toolchain is pinned in
rust-toolchain.toml. Older systemcargofails on transitive deps. prefer rustup so the pin auto-resolves. test_logs/grows quickly. Wipe between major refactors.- Miri only covers
binary_protocol+consensus. Cannot emulateio_uringsyscalls. do not try to expand Miri to crates that pullcompio. - Integration crate has no
--testtarget. Filter by test path inside the singlemod.rsbinary. - The integration harness runs prebuilt binaries and never builds them.
Command::cargo_binonly resolves a path, so a staleiggy-server,iggy-connectorsoriggy-mcpis used silently. CI restores revision-checked artifacts from the runtime and Rust test builds; a local run does not build them, and a stale binary fails in ways that look like the change under test. Rebuild after anything that touches a launched binary or its config.
Created by server, connectors runtime, or tests. Gitignored. Safe to delete to reset:
local_data/- server datalocal_state/- connectors runtime state filestest_logs/- per-test logs +local_datasnapshotsperformance_results/- benchmark outputtarget/- cargo build
- Conventional commits with scope:
feat(connectors): .... Subject <= 72 chars. Allowed types/scopes enforced by.github/workflows/pr-title.yml. - Body: why before what. No bullet lists of files changed.
- Co-authored-by trailer when applicable.
- Workspace + SDK versions sync via
scripts/extract-version.shandscripts/ci/{sync-rust,python-sdk}-version{,-sync}.sh. - Crate publish chain verified by
scripts/verify-crates-publish.shagainstcargo-http-registry. - Third-party license enumeration:
scripts/ci/third-party-licenses.sh(ASF policy).