Skip to content

cli: poll messages as a consumer group, from a timestamp, and in follow mode #4446

Open

Description

@hubcio

Problem

iggy message poll reads one batch from one partition as a regular consumer.

  • It always polls as a regular consumer (core/cli/src/commands/binary_message/poll_messages.rs:64). You cannot read a topic the way a group member reads it.
  • It can start from --offset, --first, --last or --next (:55), but not from a timestamp. The SDK has PollingStrategy::timestamp (core/common/src/types/message/polling_strategy.rs:72).
  • It reads one batch and exits. To watch a live partition, you must run it in a shell loop, for example with --next --auto-commit. Each run connects and logs in again.

Proposal

  • Add -g, --consumer-group <GROUP>. The CLI joins the group and polls the partitions that the server assigns to it. On exit, it leaves the group, also after Ctrl-C or an error. -g conflicts with -c, --offset and <PARTITION_ID>.
  • Add --timestamp <TIME>. Accept an RFC 3339 time, Unix microseconds, or a duration back from now (15min).
  • Add --follow. Keep polling until Ctrl-C, and wait for --poll-interval (default 100ms) after an empty batch.
  • Build the group and follow modes on IggyConsumer (core/sdk/src/clients/consumer.rs). It already joins the group, tracks the offset of each partition, commits, and leaves the group on shutdown. Turn off its automatic group creation.

Notes

  • A member that disconnects without a leave keeps its partitions until consumer_group.session_timeout (default 30 s) expires.
  • A join moves partitions from other members. A partition with uncommitted polled messages moves only after its owner commits them, or after consumer_group.rebalancing_timeout (default 30 s). So a single -g poll on a busy group often gets no partition. server: consumer group member that joins while a revocation is pending never gets a partition #4391 can also leave a new member without a partition.
  • With --auto-commit, the group offset moves past the messages that the CLI prints, so the other members never get them. The help text must say this.
  • Over HTTP, -g cannot work, because a group join returns FeatureUnavailable (core/sdk/src/http/consumer_groups.rs:116). Reject -g with --transport http before the CLI connects.
  • The server has no long poll yet. When feat: add deferred polling timeout to PollMessages #3470 (deferred polling timeout) lands, use it in follow mode.

Activity

  1. added
    enhancementA change that improves our (or users) QoL
    tuiCommand line interface (iggy-cmd/iggy-cli) related change
    on Oct 8, 2026
  2. marvelefe commented on Oct 25, 2026

    @marvelefe

    I'd like to pick this up.

    I'd build it on IggyConsumer (core/sdk/src/clients/consumer.rs) as you suggest, turning off its automatic group creation, and implement in this order: --timestamp first (maps directly to PollingStrategy::timestamp), then -g with leave-on-exit/Ctrl-C/error, then --follow with the --poll-interval fallback since #3470 isn't landed. I'll reject -g with --transport http before connecting (group join returns FeatureUnavailable).

    Before I start: for --follow, do you want the interval-based fallback now and a switch to deferred polling when #3470 lands, or should I wait on #3470?

  3. justinmclean commented on Oct 25, 2026

    @justinmclean
    Member

    @marvelefe the --timestamp part was split out as #4461, which is assigned to de-authority with #4532 open, so start with -g. Your question on --follow (interval now, or wait for #3470) is for @hubcio.

  4. marvelefe commented on Oct 27, 2026

    @marvelefe

    Thanks @justinmclean — I'll drop --timestamp (covered by #4461/#4532) and start with -g. @hubcio, for --follow: use the --poll-interval fallback now and switch to deferred polling once #3470 lands, or wait? Could you assign #4446 to me?

  5. hubcio commented on Oct 27, 2026

    @hubcio
    ContributorAuthor

    @marvelefe deferred polling just landed

  6. marvelefe commented on Oct 27, 2026

    @marvelefe

    Got it @hubcio. I'll use deferred polling. Starting on -g now.

  7. marvelefe commented on Oct 29, 2026

    @marvelefe

    @hubcio The PR for -g/--consumer-group and --follow is open: #4680

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementA change that improves our (or users) QoLtuiCommand line interface (iggy-cmd/iggy-cli) related change

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions