Skip to content

Timeline layout v2 path configs are persisted into hoodie.properties of tables below version 8 #19562

Description

@yihua

A table created below table version 8 gets hoodie.timeline.path=timeline and hoodie.timeline.history.path=history written into its hoodie.properties, even though it does not use the layout those configs describe.

Both configs were introduced with timeline layout version 2 in 1.0.0 and are read only by TimelinePathProviderV2, which is selected exclusively at layout version 2 (table version 8 and above). A table below that version keeps its timeline directly under .hoodie (TimelinePathProviderV1 hardcodes it) and resolves its archived timeline through hoodie.archivelog.folder. Neither config has any meaning there.

The cause is that HoodieTableConfig.dropInvalidConfigs() only strips a config that declares a sinceVersion:

if (!configProperty.getSinceVersion().isPresent()) {
  return true;   // never dropped
}

RECORD_MERGE_MODE declares .sinceVersion("1.0.0") and is correctly dropped from a version 6 table; TIMELINE_PATH and TIMELINE_HISTORY_PATH declare nothing and survive.

The immediate effect is cosmetic, since every consumer resolves these through getStringOrDefault. It matters because table version 6 is the interoperable format used when writing with a 1.x binary for a 0.x reader, and the 0.x line has no definition for either key. Recording storage layout that the table does not have is also a trap for anything that later starts honouring the values.

Reproduce by creating a table with hoodie.write.table.version=6 and reading hoodie.properties.

Fix: declare the introducing version on both so the existing gate drops them below table version 8.

Note that hoodie.table.format looks similar but must not be handled this way: the table format SPI is orthogonal to the table version, nothing gates a custom format on table version 9, and no upgrade handler restores the config, so gating it on sinceVersion would permanently discard a custom format on any table below version 9.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:configConfiguration managementarea:coreCore functionalitypriority:mediumModerate impact; usability gaps

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions