Skip to content

BUG: write_csv fails midway and leaves a partial file when later rows carry extra keys #3303

Closed

Description

Description

write_csv in pyrit.common.csv_helper builds the CSV header from the keys of the first row only. Any later row carrying a key the first row lacks makes csv.DictWriter.writerows raise ValueError: dict contains fields not in fieldnames, and because the exception happens after the header and earlier rows were already written, the caller is left with a truncated partial file on disk.

This matters for the remote dataset cache path, which writes fetched rows through this helper: dataset rows can legitimately vary in which optional fields they carry, and a single wider row turns a cache refresh into a corrupt cache file.

Expected behavior

The header should be the union of keys across all rows in first-seen order, and rows missing a header key should get an empty cell instead of an error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions