Latest Results
oopsieslade/upgrade-python-deps feat(iceberg): read data files without field IDs via schema.name-mapping.default (#7618)
## Changes Made
Iceberg tables migrated with `add_files` keep parquet files without
field IDs, whose columns resolve by name through the
`schema.name-mapping.default` table property. Daft raised
`MissingParquetFieldIds` on them (#7446). It now applies the name
mapping:
- `IcebergDataSource` passes the property into `ParquetSourceConfig`,
which carries it with the field ID mapping as `FieldIdMapping`, so every
read path, including scan task splitting, gets it.
- For a file without field IDs,
`apply_field_ids_to_arrowrs_parquet_metadata` first assigns IDs by name
(aliases included; `element` / `key` / `value` for lists and maps), then
renames and filters by ID as before. Columns missing from the mapping
are dropped, as in Java.
- LIST / MAP groups annotated with only the converted type, as older
writers emit, are now recognized; their elements were silently dropped
otherwise.
- Files with field IDs and Delta column mapping are unchanged.
- Tests read `add_files` tables (flat, renamed column, nested types,
mixed with field-ID files, split scan tasks on Ray) and compare with
PyIceberg; a Rust test covers list / map / alias / converted-only
schemas.
## Related Issues
Closes #7617
## AI Disclosure
AI-assisted implementation. Latest Branches
0%
YuangGao:perf/iceberg-delete-files-from-manifest-list 0%
AYUSHNSUT:fix/rust-doctests 0%
© 2026 CodSpeed Technology