Open table formats make object storage behave like a transactional analytical table, but transactions do not eliminate housekeeping. Frequent writes create data files, delete files, manifests, logs, metadata versions, and snapshots. Over time, small files increase planning and open costs, retained history consumes storage, and failed jobs may leave files that no committed table state references. Lakehouse table maintenance is the controlled work that keeps these structures useful without deleting data needed by readers, writers, rollback, audit, or legal retention.
Compaction, snapshot expiration, metadata cleanup, and orphan file cleanup solve different problems and carry different risks. Running all four as a generic nightly optimize job obscures their preconditions. Apache Iceberg explicitly warns that orphan removal with a retention interval shorter than the expected duration of a write can corrupt a table by deleting in-progress files. A safe program begins with table-specific objectives, ownership, concurrency behavior, and recovery windows, then schedules only the operation supported by evidence.
Inventory table format, workload, and recovery needs
Record the format and version, catalog, object-store location, engines that read and write, write frequency, partitioning, branch or tag use, streaming checkpoints, time-travel consumers, and recovery promise. A Hudi Merge-on-Read table needs compaction semantics that do not apply to a Copy-on-Write table. Iceberg snapshot expiration reasons over references in its metadata. Delta optimization and vacuum have their own transaction-log rules. Do not translate a command by name across formats.
Define a minimum retained history from concrete dependencies: maximum query duration, maximum write duration, longest paused streaming reader, rollback objective, audit period, branch retention, and disaster-recovery lag. The safe retention floor is the largest applicable dependency plus margin, not the smallest storage target. If an engine can access storage paths directly and ignore table metadata, it may read files that cleanup legitimately removes; either migrate that access through the table format or treat it as an explicit blocker.
| Operation | Primary objective | What changes | Main safety boundary |
|---|---|---|---|
| Data-file compaction | Reduce file count and improve layout | Rewrites live records into new files | Concurrent update conflict and temporary IO |
| Manifest or metadata rewrite | Reduce planning overhead | Reorganizes metadata references | Format and engine compatibility |
| Snapshot expiration | Bound history and release unneeded files | Removes old snapshots from metadata | Rollback, time travel, branches, active readers |
| Orphan cleanup | Delete files absent from table metadata | Removes unreachable storage objects | In-flight writes and path equivalence |
| Hudi cleaning | Remove old file versions | Reclaims superseded file slices | Incremental readers and rollback retention |
Measure symptoms before choosing work
Collect file count, file-size distribution, bytes, snapshots or commits, manifests, delete-file count, planning time, scan time, partitions touched per write, object-store requests, maintenance duration, rewritten bytes, and conflicts. Segment by partition age and workload. A table with millions of well-pruned files may perform acceptably, while a small hot partition with thousands of tiny files may dominate latency. Use representative query profiles and metadata planning time rather than a universal target file size.
Establish triggers from trends. Compaction can begin when a partition's small-file count or delete-file ratio crosses a tested boundary. Snapshot expiration can run by age and retained-count policy. Orphan scanning may be less frequent because listing a large location is expensive. Alert on failed maintenance, rising backlog, conflict rate, storage reclaimed, and unexpected changes in live row counts. Maintenance that completes but rewrites nearly the whole table every day is itself a performance and cost defect.
Compact data files without changing truth
Compaction should preserve the live record set while changing physical layout. Select candidate partitions or file groups using size, age, delete density, and query demand. Bound bytes and tasks per run. Prefer incremental selection over scanning every partition. Coordinate with writers through the format's optimistic concurrency or table-service mechanism, and expect conflicts when a hot file group changes during rewrite. Retry from current metadata rather than forcing an obsolete plan.
File size is a tradeoff. Larger files reduce metadata and open overhead but increase rewrite work, reduce parallelism for small scans, and may amplify updates. Sorting or clustering can improve pruning but adds write cost and can decay as new data arrives. Delta Lake documents file compaction and data skipping optimizations, while Hudi distinguishes Merge-on-Read compaction from clustering and file sizing. Match the operation to the format and access pattern, then compare planning, scan, and write performance before and after.
| Signal | Possible action | Tradeoff | Validation |
|---|---|---|---|
| Many tiny files in hot partitions | Incremental file rewrite | Temporary read/write IO | File distribution and query latency |
| High delete-file ratio | Rewrite affected data or delete files | More frequent rewriting | Record count and scan CPU |
| Slow metadata planning | Rewrite manifests or metadata | Metadata commit contention | Planning-time profile |
| Poor predicate pruning | Sort or cluster selected partitions | Higher maintenance cost | Bytes scanned for representative filters |
| Compaction conflicts | Narrow scope or move schedule | Backlog may grow | Commit success and freshness |
Expire history from an explicit retention policy
Each Iceberg write creates a snapshot, and snapshots remain available until expired. Expiration removes old snapshots from metadata; data files are deleted only when no retained snapshot references them. Before expiring, enumerate protected branches, tags, audit holds, recovery points, and active consumers. Keep a minimum number as well as an age boundary so a burst of commits cannot remove nearly all recovery points. Test time-travel and rollback to the oldest promised point before shortening policy.
Retention should vary by table class. A reconstructible staging table can keep less history than a curated financial fact table. A high-frequency table may need a count cap to prevent metadata growth while still retaining a time window. Document whether backups protect data files independently; a catalog backup without referenced objects is not a usable recovery copy. Track the oldest retained snapshot or commit and expose it to consumers so the actual guarantee is visible.
Remove orphan files only after proving they are unreachable
Orphan files arise when tasks write objects but fail before a successful metadata commit, when metadata retention leaves untracked files, or when manual operations bypass normal cleanup. The operation compares table metadata references with objects under a location. It must not be treated as a filename sweep. Use the table format's supported action, a retention interval longer than every possible in-flight write, and a dry-run inventory when available. Pause if path authorities or URI forms have changed.
Iceberg warns that path string differences can make the same object appear different to metadata and filesystem listings, which can lead to data loss. Normalize and test catalog and filesystem views after migrations, bucket aliases, authority changes, or replication. Do not place unrelated application files beneath the table root. Review unusually large deletion candidates and maintain count and byte circuit breakers. If a run proposes deleting recent files or a substantial share of the table, stop for investigation.
Control concurrency and preserve recovery
Run maintenance through a service identity with only required catalog and storage permissions. Serialize operations that the format cannot safely combine, but avoid a global lock across unrelated tables. Use per-table leases to prevent duplicate schedulers, and let format transactions arbitrate with writers. Maintenance retries must recalculate candidates from current table state. Never reuse a stale file list after another commit because references may have changed.
Before scaling a policy, restore a representative table from catalog and object-store backup, roll back to a retained point, and verify streaming consumers after maintenance. Keep operation logs with table identifier, starting metadata version, policy version, candidate files, bytes, committed result, duration, conflicts, and errors. These records explain a storage drop and support incident reconstruction. They should not be the only recovery mechanism; retain independent backups according to the table's business criticality.
Build a risk-based maintenance schedule
Classify tables by criticality, update pattern, scale, and reconstructibility. Pilot each operation on a lower-risk table, compare metrics, and establish a resource budget so maintenance cannot starve ingestion or user queries. Stagger large tables and cap concurrent rewritten bytes. Schedule hot-partition compaction more often than cold-partition scanning. Snapshot expiration can be regular, while orphan cleanup is typically less frequent and more heavily reviewed.
Use policy as code with explicit exceptions. A table owner can request longer history, a delayed cleanup window, or a maintenance blackout with an expiry. Review tables that never run maintenance and those that always fail. Revisit thresholds after engine upgrades, partition changes, or new streaming writers. The aim is not maximum reclaimed storage; it is stable planning and query behavior with a known recovery envelope and no accidental loss.
Key takeaways
- Treat compaction, history expiration, metadata work, and orphan cleanup as distinct operations.
- Set retention from the longest reader, writer, rollback, audit, and replication dependency.
- Choose compaction from measured file and query behavior, not a universal file-size rule.
- Use supported metadata-aware cleanup with dry runs, age margins, and deletion circuit breakers.
- Recompute candidates after conflicts and never force a stale maintenance plan.
- Record maintenance evidence and test restoration before shortening recovery windows.
Frequently asked questions
How often should compaction run?
Run it when file count, delete density, planning time, or representative query cost crosses a tested threshold. Hot streaming partitions may need frequent incremental work; append-only or cold tables may need little. A fixed daily schedule without a candidate threshold often wastes compute.
Are snapshot expiration and orphan cleanup the same?
No. Snapshot expiration removes old table states and can release files no retained snapshot references. Orphan cleanup finds storage objects not referenced by table metadata, often from failed work. Their candidate logic and safety windows differ.
Can maintenance run while writers are active?
Often yes when the format and engine support transactional maintenance, but conflicts and operation-specific restrictions still apply. Use supported APIs, conservative retention, current metadata, and tested retry behavior. Never assume direct filesystem deletion is concurrent-write safe.
Conclusion
Good lakehouse maintenance is conservative about truth and deliberate about physical state. It rewrites files only when evidence shows a benefit, retains history for real dependencies, and deletes unreachable objects only after current metadata and in-flight work are protected. With table-level policies, bounded resource use, concurrency-aware retries, and tested recovery, maintenance preserves the performance promise of an open table format without turning cleanup into a data-loss event.