Skip to content

Add a tuple read format for Map columns - #1005

Open
Kayvan-Zahiri wants to merge 1 commit into
ClickHouse:mainfrom
Kayvan-Zahiri:feat/949-map-tuple-format
Open

Add a tuple read format for Map columns#1005
Kayvan-Zahiri wants to merge 1 commit into
ClickHouse:mainfrom
Kayvan-Zahiri:feat/949-map-tuple-format

Conversation

@Kayvan-Zahiri

Copy link
Copy Markdown

Closes #949.

A ClickHouse Map is not a unique-key collection, so materializing every one as
a Python dict drops pairs. This adds an opt-in tuple read format that returns
the pairs as a list of tuples.

I checked this against the server source rather than inferring it from a JSON
round trip. At v26.3.21.7-lts:

  • src/Columns/ColumnMap.h describes ColumnMap as a column that stores a
    nested Array(Tuple(key, value)) column
  • src/DataTypes/DataTypeMap.cpp builds that nested type as
    DataTypeArray(DataTypeTuple({key_type, value_type}, {"keys", "values"}))
  • neither file deduplicates keys anywhere

So a list of (key, value) tuples is the server's own representation, not a
convention I picked.

Behavior:

read format map('k','1','k','2')
native (default, unchanged) {'k': '2'}
tuple [('k', '1'), ('k', '2')]

Enable per query with query_formats={'Map': 'tuple'} or globally with
set_read_format('Map', 'tuple').

Tests are parametrized over the shapes the issue names: top level Map,
Array(Map(...)), and a Map nested as another map's value. Each shape is
asserted in both formats, so the native cases pin that the default did not
move. Four fail without the change. pytest tests/unit_tests goes from 1453 to
1460 passing with the 37 pre-existing failures unchanged, ruff and mypy are
clean on the changed files.

Not included: insert side handling of duplicate keys, and sync/async integration
coverage against a live server, which I could not run here. Happy to add the
integration tests if you want them in this PR.

ClickHouse stores a Map as Array(Tuple(key, value)) and permits repeated
keys, so materializing every Map as a dict silently drops pairs. Reading
with the tuple format returns a list of (key, value) tuples instead,
which is the server's own representation.

The default read format stays native and still returns a dict, so
existing behavior is unchanged. Enable it per query with
query_formats={'Map': 'tuple'} or globally with set_read_format.

Confirmed against the server source at v26.3.21.7-lts: ColumnMap holds a
nested Array(Tuple(key, value)) column, DataTypeMap builds that nested
type from the key and value types, and neither deduplicates keys.

Closes ClickHouse#949
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Map(K, V) with duplicate keys silently loses entries (dict(zip(...)) collapses pairs)

1 participant