Skip to content

Stop DBFS iteration at EOF after file truncation - #1621

Draft
Kayvan-Zahiri wants to merge 1 commit into
databricks:mainfrom
Kayvan-Zahiri:codex/stop-dbfs-iteration-at-eof
Draft

Kayvan-Zahiri wants to merge 1 commit into
databricks:mainfrom
Kayvan-Zahiri:codex/stop-dbfs-iteration-at-eof

Conversation

@Kayvan-Zahiri

@Kayvan-Zahiri Kayvan-Zahiri commented Oct 28, 2026 •

Copy link
Copy Markdown

When a DBFS file is truncated after get_status, _DbfsIO.__iter__ continues using the cached size. An empty read does not advance the offset, so both iteration and read() keep requesting EOF indefinitely.

Stop iteration on an empty chunk. This preserves the existing direct read(size) and __next__ behavior and returns the bytes read before EOF.

Validation: eight public WorkspaceClient.dbfs.open HTTP-mocked cases cover empty/nonempty files, accurate/stale sizes, and iteration/read consumption. Four stale-size cases fail before the fix; all eight pass after it. All 22 DBFS mixin tests pass. Repository-wide Ruff lint and formatting checks pass, as does git diff --check. The full suite passes: 2,312 tests on Python 3.12 (13 warnings). No hosted Databricks workspace or full Python/OS matrix was used.

I understand this repository is a public mirror and maintainers re-apply accepted external changes internally.

Signed-off-by: Kayvan Zahiri <kzahiri@dons.usfca.edu>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant