feat(parquet): support reading list columns as Arrow large_list

[rahulsmahadev]

Summary

Implements the remaining LargeListArray gaps from #502, following the design suggested by @wgtmac in #513:

1. ValidateParquetSchemaEvolution now accepts LARGE_LIST wherever LIST is accepted, so schema projection works when the Arrow reader presents 64-bit offset list types.
2. A new reader property read.arrow.use-large-list (default: false) configures the Parquet reader via ArrowReaderProperties::set_list_type(::arrow::Type::LARGE_LIST) to decode list columns as large_list.

Since ToArrowSchema builds the reader's output schema with 32-bit lists (and ProjectRecordBatch dispatches on the output schema type), enabling the property also rewrites list fields in the output Arrow schema to large_list so the projection layer takes the ProjectLargeListArray path added in #502. The rewrite is local to the Parquet reader to avoid changing the ToArrowSchema signature used across writers and manifest readers.

Closes #513

Changes

• src/iceberg/parquet/parquet_schema_util.cc: accept LARGE_LIST for TypeId::kList in schema evolution validation.
• src/iceberg/file_reader.h: add ReaderProperties::kArrowUseLargeList (read.arrow.use-large-list, default false), following the kBatchSize pattern.
• src/iceberg/parquet/parquet_reader.cc: set ArrowReaderProperties::set_list_type when the property is enabled, and align the output Arrow schema (lists nested in structs/maps included) with the large_list arrays produced by the reader.

Test plan

• ParquetSchemaProjectionTest.ValidateSchemaEvolutionAllowsLargeList: large_list Arrow type validates against an Iceberg ListType.
• ParquetSchemaProjectionTest.ProjectLargeListType: projection over a SchemaManifest built with set_list_type(LARGE_LIST) (the same path BuildProjection uses in the reader).
• ParquetReaderTest.ReadListType: default behavior unchanged — list columns read as 32-bit offset list.
• ParquetReaderTest.ReadListAsLargeList: with read.arrow.use-large-list=true, the output schema exposes large_list and values round-trip correctly (verified via array slices since JSON parsing creates regular ListArray).

Note: my local environment lacks a C++23 toolchain (cmake 3.16/gcc 10), so I could not build locally; relying on CI to verify. All Arrow APIs used (set_list_type, large_list(field), MapType(key_field, item_field, keys_sorted), Field::WithType) were checked against the pinned Arrow 24.0.0 headers.

[rahulsmahadev]

@wgtmac this implements the design you suggested in #513 — would appreciate a review when you get a chance.

One note beyond the two items in the issue: the reader's output Arrow schema (built by ToArrowSchema, which uses 32-bit lists) also needed to be aligned so ProjectRecordBatch dispatches to the large_list path from #502. I kept that rewrite local to the Parquet reader to avoid changing the ToArrowSchema signature, but happy to take a different approach if you'd prefer.

[wgtmac]

Thanks for improving this, @rahulsmahadev! I think the long-term shape is to make this output-schema driven, ideally allowing callers to choose list vs large_list per projected column. Arrow’s Parquet reader only exposes a global set_list_type today, so keeping a global flag for this PR seems reasonable.

One API concern: since this option lives in generic ReaderProperties, it reads as format-independent. Either other readers such as Avro should honor it too, or the option should be clearly scoped/documented as Parquet-only.

One thing that is worth noting: when read.arrow.use-large-list is enabled, the reader always rewrites the output schema to large_list, but Arrow ignores set_list_type(LARGE_LIST) when the Parquet file contains serialized ARROW:schema metadata. In that case the actual batch can still contain ListArray, while ProjectRecordBatch dispatches as LargeListArray, causing a bad cast/crash. Please base the output schema on the actual reader schema or add a guard, and cover this with an ARROW:schema regression test.

[kamcheungting-db]

add comment:

Suggested change

	std::shared_ptr<::arrow::Field> UseLargeListField(
	// forward declaration to unblock cycle dependence.
	std::shared_ptr<::arrow::Field> UseLargeListField(

[kamcheungting-db]

this is duplicated logic.
let's add a method.

::arrow::FieldVector UseLargeListFields(const ::arrow::FieldVector& fields) {
  ::arrow::FieldVector rewritten;
  rewritten.reserve(fields.size());
  for (const auto& field : fields) {
    rewritten.push_back(UseLargeListField(field));
  }
  return rewritten;
}

[rahulsmahadev]

@wgtmac can you help review