OverlayBD open-source contributions

Problem

LSMT and ZFile loaders needed to reject oversized index counts before calculating index-memory requirements.

Implementation

I added index-size limits in the LSMT and ZFile loading paths, including checks around read-only index creation.

Testing

I added regression cases for oversized LSMT and ZFile indexes and ran the targeted rejection tests.

Problem

Index-size checks needed consistent placement across metadata verification and recursive index merging.

Implementation

I moved ZFile size validation after metadata-integrity checks, added read-only LSMT validation in header verification, and handled oversized recursive merges at an exception boundary using the production limit.

Testing

I ran lsmt_test and zfile_test and removed the earlier test-only configurable merge limit.

Problem

ZFile compressed data blocks but stored the block-length index uncompressed.

Implementation

I added opt-in LZ4/ZSTD index compression and shared index writing across whole-file and streaming builders. Writers retain raw indexes when compression does not shrink them or exceeds codec limits. Readers validate metadata, sizes, checksums and decoded block lengths before exposing reads.

Testing

I added round-trip and malformed-input coverage across both codecs, writer paths, header/trailer paths, raw fallback and short reads. I ran the full zfile_test suite in a Debug build with QAT disabled.

Compression stays opt-in because older readers require raw indexes.