Do not bulk insert a VALUES list holding expressions - #1004
Do not bulk insert a VALUES list holding expressions#1004Kayvan-Zahiri wants to merge 2 commits into
Conversation
_try_bulk_insert checked only that the statement contained VALUES, never
what the values list held. The block writer carries one value per named
column, so a function call was dropped: VALUES (%s, hex(%s)) wrote the
raw parameter into the column with no error, and VALUES (%s, hex(unhex(
'AB'))) raised a misleading column count error instead of falling back.
The values list must now be parameter slots only, either pyformat
placeholders or server side {name:Type} bindings, and anything else
falls through to the row by row path that substitutes and lets the
server evaluate it. A bare VALUES with no list keeps the fast path,
since that is what the SQLAlchemy dialect emits.
Closes ClickHouse#934
|
Closing this. After review it does fix the reported function-expression examples, but it does not adequately address the root cause. This effectively replaces one lossy boolean ( A solid fix needs a fail-closed insert parser or planner of some kind. It should recognize only a safe subset, preserve the ordered relationship between target columns and parameter slots, validate row cardinality, reject or preserve modifiers, and fall back whenever equivalence cannot be guaranteed. The regression tests also need to verify stored values and I am leaving #934 open because the underlying bug still does exist and I am happy to discuss a fresh design there if you want to work through the details before implementing another patch. Thanks! |
Closes #934.
_try_bulk_insertchecked only that the statement containedVALUES, never whatthe values list held. The block writer carries one value per named column, so an
expression in there is lost:
VALUES (%s, hex(%s))hexVALUES (%s, hex(unhex('AB')))Insert data column count does not match column namesThe first case is the bad one: wrong data, silently.
A values list now has to be parameter slots only. That means pyformat
placeholders and also server side
{name:Type}bindings, which the existingtest_executemany_bulk_insert_keeps_percents_for_server_side_statementscoversand which my first attempt broke. Anything else falls through to the row by row
path, which already substitutes and lets the server evaluate the expression.
One case worth calling out: a bare
VALUESwith no list at all keeps the fastpath, because that is what the SQLAlchemy dialect emits and there is no
expression to lose.
test_cursor_bulk_insert_forwards_settingscaught that.Ten parametrized tests added, five for the fallback and five pinning the fast
path.
pytest tests/unit_testsgoes from 1453 to 1463 passing with the 37pre-existing failures unchanged.