[WIP][SPARK-57811][SQL] Support string to nanosecond-precision timestamp coercion in comparisons and predicates#57223
Open
stevomitric wants to merge 4 commits into
Open
Conversation
…oercion in comparisons and predicates
Type coercion previously handled only microsecond TimestampType when one
operand of a comparison or predicate is a string: the string was cast to
TimestampType so subsecond digits compared exactly. Nanosecond timestamp
types (Timestamp{LTZ,NTZ}NanosType) had no such arm, so a string compared
against a nanos column fell through to generic handling and lost the 7th-9th
fractional digits.
This adds the missing non-ANSI arms, mirroring the existing microsecond
handling: two Equality arms in StringPromotionTypeCoercion cast the string to
the concrete nanos operand type, and two findCommonTypeForBinaryComparison
cases in TypeCoercion return that nanos type for the range path. The concrete
family and precision of the timestamp operand are preserved, the arms are
ordered so equality fires before the range path, and legacy
castDatetimeToString is honored (range promotes both sides to string, equality
still casts the string to nanos). ANSI type coercion is intentionally left
unchanged.
Tests: TypeCoercionSuite "binary comparison with string promotion" gains
nanos assertions (all precisions, both families, plus a legacy-mode block that
exercises the new arms); TimestampNanosWideningSuiteBase adds an end-to-end
string-operand test; and the timestamp-{ltz,ntz}-nanos golden files add
equality/range/BETWEEN cases.
Co-authored-by: Isaac
…ion divergence Add a comment at the new findCommonTypeForBinaryComparison nanos arms noting that AnyTimestampNanoType covers both LTZ and NTZ, so the nanos types honor castDatetimeToString like micros TimestampType (LTZ) -- deliberately more precise than micros TimestampNTZType, which has no arm and falls through to the config-blind canPromoteAsInBinaryComparison case. Co-authored-by: Isaac
… string-nanos coercion arm
The existing golden cases in timestamp-{ntz,ltz}-nanos.sql run only in the
default config (ANSI on), where a string compared to a nanos timestamp
already coerces to the nanos type via the pre-existing string-promotion
fall-through -- so those cases are byte-identical on the parent commit and
do not lock in the arm this PR adds.
Add a case under `spark.sql.ansi.enabled=false` +
`spark.sql.legacy.typeCoercion.datetimeToString.enabled=true` -- the only
config where the new findCommonTypeForBinaryComparison arm changes the plan.
The regenerated analyzer-results show the split the arm produces: the range
comparison promotes the column to string (`cast(c as string) < '...'`,
matching micro TIMESTAMP), while equality still casts the string to the
nanos type (the Equality arm fires first). Reverting the production arm
makes these analyzer goldens fail, confirming they are non-vacuous.
Co-authored-by: Isaac
…s coercion tests Address review feedback: the `Equality` extractor used by the new StringPromotionTypeCoercion arms matches both `EqualTo` and `EqualNullSafe`, so the `<=>` operator is coerced identically -- but only `EqualTo` was asserted. Add `EqualNullSafe` ruleTest cases in both the default-config parameterized block and the legacy `castDatetimeToString` block, mirroring the existing `EqualTo` assertions. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Add non-ANSI type-coercion arms so that a string compared against a nanosecond-precision timestamp column (
TIMESTAMP_NTZ(p)/TIMESTAMP_LTZ(p), p in [7, 9]) is cast to that nanos type, mirroring the existing microsecondTimestampTypehandling (StringPromotionTypeCoercionequality arms +TypeCoercion.findCommonTypeForBinaryComparison).Why are the changes needed?
Micros TimestampType has string-coercion arms that honor ...datetimeToString.enabled (legacy → promote to string; equality → cast to timestamp). Nanos had none, so it fell through to config-blind AtomicType promotion. This adds the arms so nanos matches TimestampType's legacy behavior.
Does this PR introduce any user-facing change?
Only under legacy datetimeToString=true + ANSI off: range comparisons (<, BETWEEN, …) now promote to string (matching micros); equality unchanged. All other configs identical.
How was this patch tested?
Extended existing suites and added golden file tests.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code