protocol: add v2 protocol design by tkilias · Pull Request #18 · exasol/udf-runner-cpp

tkilias · 2026-08-14T08:08:16Z

Summary

Adds the v2 UDF protocol design and FlatBuffer schema.

The design separates low-level transport rules, generic call lifecycle, generic data-stream behavior, and high-level DB/UDFRunner orchestration. It includes Mermaid source diagrams and their rendered SVGs.

Impact

Defines the proposed v2 protocol contract, including framing, stream ownership, schema and record-batch ordering, flow control, transport bindings, close semantics, and high-level call behavior.

Validation

Rendered and visually inspected all updated Mermaid state diagrams.
Ran git diff --check before squashing the branch.

Make the high-level column definition the single source for Exasol type parameters and link to its Call Metadata contract from the Arrow mapping document. Remove repeated property ownership text and legacy exasol field metadata from the mapping examples. Keep only conversion-specific GeoArrow and interval extension metadata, and update validation assertions and rendered mapping documentation accordingly.

tkilias · 2026-08-28T12:57:19Z

+| `TIMESTAMP` | `Timestamp(unit, "")` | Select the smallest unit preserving declared precision. |
+| `TIMESTAMP WITH LOCAL TIME ZONE` | `Timestamp(unit, "UTC")` | Normalize Exasol's UTC-normalized value for transport. |
+| `DATE` | `Date(Day)` | Preserve calendar-day semantics. |
+| `CHAR` | `Utf8` | Preserve fixed-length padding and character metadata as Exasol semantics. |


we are not sending the padding or receiving the padding

tkilias · 2026-08-28T13:07:44Z

+- `DOUBLE PRECISION` maps to `FloatingPoint(Double)`.
+- `1 <= p <= 9` maps to `Decimal(32)` with `bit_width = 32`.
+- `10 <= p <= 18` maps to `Decimal(64)` with `bit_width = 64`.
+- `19 <= p <= 36` maps to `Decimal(128)` with `bit_width = 128`.


Suggested change

- `19 <= p <= 36` maps to `Decimal(128)` with `bit_width = 128`.

- `19 <= p <= 38` maps to `Decimal(128)` with `bit_width = 128`.

tkilias · 2026-08-28T13:53:10Z

+Every `Run` and Function call uses one `call_metadata` JSON payload. It may be sent on control stream `0` before any
+call is sent or between calls, or in the same `StreamMessage` as `OpenCall`. It conforms to
+[call_metadata.schema.json](../../../../../udf-runner-cpp/v2/json_schema/call_metadata.schema.json) and supplies the
+per-invocation execution context and iterator settings. Metadata sent before any call or between calls applies to
+subsequent calls. Metadata sent with `OpenCall` applies to that call and subsequent calls. The latest received value
+replaces the previous value. It is not sent during an active call.
+
+Column definitions are carried separately in one `column_metadata` JSON payload. It conforms to
+[column_metadata.schema.json](../../../../../udf-runner-cpp/v2/json_schema/column_metadata.schema.json) and may be
+sent on control stream `0` before any call is sent or between calls, or in the same `StreamMessage` as `OpenCall`.
+Metadata sent before any call or between calls applies to subsequent calls. Metadata sent with `OpenCall` applies to
+that call and subsequent calls. The latest received value replaces the previous value. It is not sent during an active
+call. Column `type` values use official Exasol type families, while `type_name` carries the
+complete parameterized Exasol SQL declaration. Their Arrow physical representation and metadata rules are defined in
+[type_mapping.md](type_mapping.md). The shared column-definition contract is defined in
+[column.schema.json](../../../../../udf-runner-cpp/v2/json_schema/column.schema.json) and is referenced by both column
+metadata and import specifications.


the rule for when the metadata can be sent and when it is valid or get overriden is identical can be explained ones

tkilias added 20 commits August 14, 2026 10:06

protocol: add v2 protocol design

db61309

protocol: add run batch correlation

12e29d2

protocol: mark completed run groups

695d2b1

protocol: define high-level payload contracts

95841aa

protocol: remove legacy metadata fields

486798b

Use Draft-07 v2 JSON schemas

35c21be

Reorganize v2 protocol design docs

90b51c4

docs(v2): define high-level Exasol-to-Arrow type mapping

ab08878

docs(v2): expose Exasol column type parameters

a4822d6

docs(v2): name Exasol type enum

d615498

docs(v2): centralize schemas and examples with CI validation

a53378c

Merge origin/main into documentation/design_v2_protocol

ffedcd5

docs(v2): map decimals to sized Arrow types

cf85adb

docs: consolidate v2 data stream references

834ff7f

docs: place call lifecycle under low-level protocol

5a08777

docs: consolidate low-level data stream references

b83f88f

some cleanup

2742461

docs: use fixed-width year-month interval storage

158f8c1

sort files into sub directories

a584dfa