Skip to content

[python] Add asyncio-native streaming API for bounded log reading #545

Description

@charlesdong1991

Search before asking

  • I searched in the issues and found nothing similar.

Description

LogScanner.to_arrow_batch_reader() currently returns a synchronous pyarrow.RecordBatchReader that blocks the calling thread on each __next__() call. This might be acceptable for Arrow interop (e.g., feeding into DuckDB, Polars etc), but is not suitable for asyncio-native Python code.

So, ideally we should add an async counterpart, e.g., async for batch in scanner.read_batches() which yields RecordBatch objects without blocking the event loop.

Willingness to contribute

  • I'm willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions