Storage Layers, File Formats, and Table Layout

Performance and correctness problems often share the same root design. The two most expensive ones have published price tags rather than cautionary tales.

The rows a dataset logically holds are only part of what it is. Its file boundaries, partition scheme, snapshot mechanism, retention policy, and metadata determine whether consumers can read it consistently.

ML workloads often scan a few columns across a long history, and the size of that advantage has been measured rather than asserted. Google ran the same single-field aggregation over 85 billion records on 3,000 nodes, once record-wise and once by column. Its Dremel paper reported the gap in 2010: “Dremel and MR-on-columns read about 0.5TB of compressed columnar data vs. 87TB read by MR-on-records.” The same paper marks the limit of its own result. The win holds “when few columns are read, the gains of columnar representation are of about an order of magnitude”, while the crossover back to record-wise storage “often lies at dozens of fields”. Columnar layout is not a virtue. It is a bet on narrow projections, and the paper says where the bet stops paying.

Physical layout also creates risk. Over-partitioning produces tiny files, mutable paths weaken reproducibility, and manual overwrites can expose partial results to readers.

The goal is not to choose one universal format. It is to align storage with access patterns, update behavior, reproducibility, governance, and recovery.

Row groups have a recommended size, and it is a number rather than a principle. “We recommend large row groups (512MB - 1GB). Since an entire row group might need to be read, we want it to completely fit on one HDFS block.”, says the Apache Parquet documentation. Larger groups mean longer sequential reads. They are not free on the write side: “Larger groups also require more buffering in the write path (or a two pass write)”. For pages the advice inverts: “We recommend 8KB for page sizes”, small enough for fine-grained reading.

The engines people actually run default far below that recommendation. The Amazon Athena User Guide: “Note that Parquet and ORC are internally organized by row groups (Parquet) and stripes (ORC). The default size for row groups is 128 MB, and for stripes, 64 MB.” It adds that “Decreasing the row group or stripe size to less than their default values is not recommended.” A 2023 benchmark of the shipping implementations found the two formats do not even count in the same unit: “(non-Java) Parquet uses a row-group size based on the number of rows (e.g., 1M rows) whereas ORC uses fixed physical storage size (e.g., 64 MB).” Between 512MB - 1GB, 128 MB and 1M rows there is a real decision. Nobody has made it for you.

A mature design avoids forcing every consumer to read raw sources or treating every derived table as permanent truth.

These are different layers of design. They can be combined rather than treated as exclusive brands.

Row-oriented records keep the fields of one record together. That suits transactional reads and message exchange, and it makes analytical scans read fields nobody asked for. Columnar files group values by column, which is why the Dremel measurement above exists at all. Their limitation for ML is not only that file-level updates require rewrite strategies. It is width. Zeng and colleagues projected a fixed 10 randomly chosen columns out of tables of growing width, and reported what happened: “As the number of attributes (i.e., features) in the table grows, the metadata parsing overhead increases almost linearly even though the number of projection columns stays fixed. This is because the footer structures in Parquet and ORC do not support efficient random access.” Meta built a format around that constraint. Its Nimble README says the format “is better suited for workloads that are wide in nature, such as tables with thousands of columns (or streams) which are commonly found in feature engineering workloads and training tables for machine learning”, and lists among its choices “Use Flatbuffers instead of thrift/protobuf to more efficiently access large metadata sections.” A feature table with thousands of columns is exactly the shape the benchmark penalises.

The third abstraction is a mechanism, not a brand. A table becomes a tracked set of files: “This table format tracks individual data files in a table instead of directories”, says the Apache Iceberg spec. Publication becomes one swap: “Table state is maintained in metadata files. All changes to table state create a new metadata file and replace the old metadata with an atomic swap.” The spec's goals answer the partial-overwrite failure listed at the top of this lesson in one sentence: “Serializable isolation -- Reads will be isolated from concurrent writes and always use a committed snapshot of a table’s data. Writes will support removing and adding files in a single operation and are never partially visible.” Partitioning stops being directory structure: “Partitioning will be table configuration. Reads will be planned using predicates on data values, not partition values. Tables will support evolving partition schemes.” And this is not a specification waiting for an implementer. The Amazon Athena User Guide says “Iceberg manages large collections of files as tables, and it supports modern analytical data lake operations such as record-level insert, update, delete, and time travel queries”, that “Athena supports Apache Iceberg version 1.4.2”, and that “Athena only creates and operates on Iceberg v2 tables”. Check that version number against your engine before you promise anyone time travel.

Begin with queries and lifecycle behavior rather than copying the source layout. Step 4 is the one with published numbers to check against rather than taste: a partition count inside the platform's ceiling, a partition size above its recommended floor, and a row-group or stripe size no smaller than the engine's default.