Compression

What Gets Compressed

HDF5 can only compress chunked datasets, so a few kinds of objects are always stored uncompressed:

Strings deserve special attention. By default, h5lite stores character vectors as fixed-length strings when they contain no NA and have similar lengths, and as variable-length strings otherwise. Fixed-length strings sit in the chunks and compress like any other data. Variable-length strings are kept in a separate area of the file, so the filter pipeline only ever sees small references to them. If you want text compressed, write it as fixed-length with as = "utf8[]". Fixed-length strings can’t hold NA, though.

words <- sample(c("control", "treatment", "placebo"), 1e4, replace = TRUE)
h5_write(words, file, "strings/fixed")
h5_inspect(file, "strings/fixed")
#> <HDF5 Dataset Properties>
#>   Type:    utf8[9]             Size:    87.89 KB
#>   Layout:  chunked             Disk:    20.87 KB
#>   Chunks:  [10000]             Ratio:   4.21x
#>   Pipeline: shuffle -> gzip

Very small datasets can end up a few bytes larger after compression because of filter overhead. This is harmless and isn’t worth optimizing.

Gzip

"gzip-[level]" (levels 1-9, default 5) is the universal standard. It isn’t the fastest or the strongest codec, but any HDF5 reader in any language can open it. Use it for files that you share, publish, or archive. Higher levels are slower and often barely smaller than level 5.

Zstandard and LZ4

• "zstd-[level]" (levels 1-22, default 3): Zstandard compresses about as well as gzip or better, in much less time. Levels 3-5 are a good balance. Levels above about 15 are much slower to write but no slower to read, which suits data that is written once and read many times.

• "lz4-[level]": LZ4 favors speed over ratio. Level 0 (the default) is standard LZ4. Levels 1-12 select the slower, stronger LZ4-HC mode.

Blosc2 and Bitshuffle

Bitshuffle rearranges data at the bit level before handing it to a compressor (see Shuffling). On numeric data, this often beats byte shuffling by a wide margin. You can get bitshuffling in two ways:

• "bshuf-lz4" and "bshuf-zstd-[level]" use the standalone Bitshuffle plugin with a built-in LZ4 or Zstandard compressor.

• "blosc2-[codec]-[level]" uses Blosc2, a meta-compressor that splits each chunk into cache-sized blocks, bitshuffles them, and compresses them with a sub-codec: "blosc2-zstd-5" or "blosc2-lz4-5", for example. Blosc2 also offers the optional delta and truncate-precision pre-filters, and it can run ZFP ("blosc2-zfp-...").

Blosc and Blosc2 can compress and decompress the blocks of each chunk in parallel. They use one thread unless you ask for more with h5_threads(), for example h5_threads(4), which sets the thread count for everything in h5lite that runs in parallel. Threads help most when compression is slow, as with high Zstandard levels, and with large chunks, but they speed up reading at the default 1 MB chunk size too. To change the default of one thread without calling h5_threads(), set the H5LITE_NTHREADS environment variable. Threads need hdf5lib 2.2.0.1 or later: when h5lite is built with hdf5lib 2.2.0.0, Blosc always uses one thread.

Szip

"szip-nn" and "szip-ec" are provided by the libaec library and are part of the core HDF5 filter set. Nearest Neighbor (nn) coding suits smooth, correlated data such as time series or gradients. Entropy Coding (ec) suits uncorrelated or categorical integer data. Szip works only on numeric data. For anything else, h5lite falls back to gzip (see Rules and Automatic Adjustments). Modern codecs usually beat szip, but it is still common in some scientific fields, such as Earth observation data.

Legacy Codecs

"blosc1-...", "bzip2-[level]", "lzf", and "snappy" are included mainly so you can read and write files for older pipelines that already use them. They are generally slower or weaker than the codecs above, so they aren’t recommended for new data.

Comparing Codecs on Your Own Data

Codec performance depends heavily on the data, so the most reliable way to choose is to try a few. This small helper writes the same object with each codec and reports the compression ratio:

compression_ratio <- function(x, codecs, ...) {
  tmp <- tempfile(fileext = ".h5")
  on.exit(unlink(tmp))
  sapply(codecs, function(codec) {
    h5_write(x, tmp, "x", compress = h5_compression(codec, ...))
    info <- h5_inspect(tmp, "x")
    round(info$uncompressed_size / info$storage_size, 2)
  })
}

codecs <- c("gzip", "zstd-5", "zstd-19", "lz4", "bshuf-lz4",
            "bshuf-zstd-5", "blosc2-lz4-5", "blosc2-zstd-5")

# A slowly varying sensor signal (float64)
compression_ratio(temps, codecs)
#>          gzip        zstd-5       zstd-19           lz4     bshuf-lz4 
#>          2.08          2.06          2.08          1.29          2.39 
#>  bshuf-zstd-5  blosc2-lz4-5 blosc2-zstd-5 
#>          2.46          1.31          2.47

# Small counts (stored as uint8)
compression_ratio(rpois(1e5, lambda = 3), codecs)
#>          gzip        zstd-5       zstd-19           lz4     bshuf-lz4 
#>          2.38          2.36          2.83          1.38          2.35 
#>  bshuf-zstd-5  blosc2-lz4-5 blosc2-zstd-5 
#>          2.46          2.44          2.49

To compare speed too, wrap the h5_write() and h5_read() calls in system.time(), or use a benchmarking package such as bench. Use realistically large data: on tiny inputs, the timings mostly measure overhead.

ZFP

ZFP is a compressor built specifically for numeric arrays. It has three lossy modes and one lossless mode:

• Accuracy ("zfp-acc-[tolerance]"): No value changes by more than the given absolute tolerance, such as "zfp-acc-0.001".
• Precision ("zfp-prec-[bits]"): Keeps a fixed number of bits of precision, so the error scales with each value’s magnitude, such as "zfp-prec-16".
• Rate ("zfp-rate-[bits]"): Stores exactly the given number of bits per value, which gives a predictable file size, such as "zfp-rate-8".
• Reversible ("zfp-rev"): Lossless.

h5_write(temps, file, "lossy/zfp", compress = "zfp-acc-0.01")

# No value has moved by more than 0.01
max(abs(h5_read(file, "lossy/zfp") - temps))
#> [1] 0.004433594

h5_inspect(file, "lossy/zfp")
#> <HDF5 Dataset Properties>
#>   Type:    float64             Size:    781.25 KB
#>   Layout:  chunked             Disk:    163.39 KB
#>   Chunks:  [100000]            Ratio:   4.78x
#>   Pipeline: zfp

ZFP has requirements that are easy to miss:

• It needs 32- or 64-bit numeric types: float32, float64, int32, or int64. Inside Blosc2 ("blosc2-zfp-..."), only float32 and float64 work. Because h5lite automatically stores whole numbers in the smallest type that fits (often uint8 or uint16), integer data usually needs an explicit as. When the type isn’t supported, h5lite quietly uses gzip instead, so check the result with h5_inspect():

  # Stored as uint16, so ZFP is skipped and gzip is used instead
  h5_write(1:1000, file, "lossy/zfp_skipped", compress = "zfp-rev")
  h5_inspect(file, "lossy/zfp_skipped")
  #> <HDF5 Dataset Properties>
  #>   Type:    uint16              Size:    1.95 KB
  #>   Layout:  chunked             Disk:    300.00 B
  #>   Chunks:  [1000]              Ratio:   6.67x
  #>   Pipeline: shuffle -> gzip
  
  # Storing as int32 lets ZFP run
  h5_write(1:1000, file, "lossy/zfp_int32", compress = "zfp-rev", as = "int32")
  h5_inspect(file, "lossy/zfp_int32")
  #> <HDF5 Dataset Properties>
  #>   Type:    int32               Size:    3.91 KB
  #>   Layout:  chunked             Disk:    1.53 KB
  #>   Chunks:  [1000]              Ratio:   2.56x
  #>   Pipeline: zfp

• It can’t store NA, NaN, or Inf. ZFP would turn them into arbitrary numbers, so h5lite writes data that contains them with the default pipeline (shuffle + gzip) instead. Those values come back exactly.

• Standalone ZFP compresses blocks of one to four dimensions. Data whose chunks have no dimension longer than one (a single value, say) or more than four is written with the default pipeline (shuffle + gzip) instead. With hdf5lib 2.2.0.1 or later, ZFP inside Blosc2 compresses chunks of any shape, but stores data of fewer than four values uncompressed in the same way.

• In Blosc2, accuracy mode supports only powers of ten. For "blosc2-zfp-acc-[tolerance]", the tolerance is rounded down to a power of ten (for example, 0.05 becomes 0.01).

Scale-Offset: int_packing and float_rounding

The Scale-Offset filter is built into HDF5, so it’s the one form of lossy compression that every HDF5 reader understands. It works on one chunk at a time and runs before the main compressor.

• int_packing = TRUE (lossless) subtracts each chunk’s minimum value and stores the remainders using the fewest bits that can represent them. This is effective for integers that span a narrow range, such as years, sensor counts, or IDs. Passing a number instead, such as int_packing = 8, forces that bit width. Use a fixed width only if you know the range fits.

• float_rounding = d (lossy) stores floating-point values as integer multiples of 10^-d, measured from the chunk’s minimum, and then bit-packs them. Every value comes back within half a unit of the d-th decimal place, for example within ±0.005 when d = 2. Because the steps are counted from the chunk’s minimum rather than from zero, the values you read back won’t necessarily print as round decimals.

# Integers in a narrow range: lossless
ids <- sample(100000:100100, 1e5, replace = TRUE)
h5_write(ids, file, "packed/ids", compress = h5_compression("gzip", int_packing = TRUE))
identical(h5_read(file, "packed/ids"), ids)
#> [1] TRUE

# Floats kept to 2 decimal places: lossy
readings <- rnorm(1e5, mean = 98.6, sd = 0.5)
h5_write(readings, file, "packed/readings",
         compress = h5_compression("gzip", float_rounding = 2))
max(abs(h5_read(file, "packed/readings") - readings))
#> [1] 0.004999984

h5_inspect(file, "packed/readings")
#> <HDF5 Dataset Properties>
#>   Type:    float64             Size:    781.25 KB
#>   Layout:  chunked             Disk:    109.92 KB
#>   Chunks:  [100000]            Ratio:   7.11x
#>   Pipeline: scaleoffset -> gzip

A few things to know:

• int_packing affects only datasets stored as integers, and float_rounding affects only datasets stored as floating point. You can set both in one configuration when you write a list that contains both kinds of data.
• Integer vectors that contain NA are stored as float64, so int_packing doesn’t apply to them.
• float_rounding can’t store NA or NaN. The filter would turn them into ordinary numbers, so h5lite skips float_rounding for any dataset that contains NA, NaN, or Inf.
• Scale-Offset turns off shuffling, and it can’t be combined with szip, ZFP, Bitshuffle, or the Blosc2 pre-filters.

Blosc2 Truncate Precision

With any Blosc2 codec except NDLZ and ZFP, blosc2_truncate = n keeps n bits of each floating-point value’s mantissa and zeroes the rest, which makes the data much more compressible. The relative error is about 2^-n. A float64 has 52 mantissa bits and a float32 has 23, so blosc2_truncate = 23 gives float64 data roughly single-precision accuracy while keeping the float64 type. A negative n drops that many bits instead: blosc2_truncate = -29 also keeps 23 bits of a float64.

Truncation only applies to float32 and float64 data. Integers, data frames, and float16/bfloat16 data are written without it, and so is data containing NA or NaN, since zeroing those bits would turn NA into NaN or Inf. So is float32 data when a negative n would drop all 23 of its mantissa bits.

x <- rnorm(1e5)
h5_write(x, file, "lossy/truncated",
         compress = h5_compression("blosc2-zstd-5", blosc2_truncate = 16))

# Largest relative error
max(abs((h5_read(file, "lossy/truncated") - x) / x))
#> [1] 1.523367e-05

h5_inspect(file, "lossy/truncated")
#> <HDF5 Dataset Properties>
#>   Type:    float64             Size:    781.25 KB
#>   Layout:  chunked             Disk:    261.72 KB
#>   Chunks:  [100000]            Ratio:   2.99x
#>   Pipeline: blosc2 [trunc(bits=16) -> bitshuffle -> zstd]

Shuffling

A shuffle filter doesn’t compress anything by itself. It reorders the data so that the compressor can do a better job. The byte shuffle groups the 1st byte of every value together, then the 2nd byte of every value, and so on:

• Integers: Small values stored in a wide type have many zero bytes. After shuffling, those zeros form long runs that compress extremely well.
• Floating point: Values of similar magnitude share their sign and exponent bytes. Grouped together, those bytes become highly repetitive.

Bitshuffle does the same thing one bit at a time, which exposes even more redundancy in numeric data.

h5lite picks the shuffle for you:

Blosc2 Delta

With any Blosc2 codec except NDLZ and ZFP, blosc2_delta = TRUE adds Blosc2’s delta pre-filter, which encodes values relative to a reference block before compression. It can help with slowly varying data, but it can also make things worse. Test it on your data with the helper from Comparing Codecs:

blosc2_codecs <- c("blosc2-zstd-5", "blosc2-lz4-5")
rbind(
  without_delta = compression_ratio(temps, blosc2_codecs),
  with_delta    = compression_ratio(temps, blosc2_codecs, blosc2_delta = TRUE) )
#>               blosc2-zstd-5 blosc2-lz4-5
#> without_delta          2.47         1.31
#> with_delta             2.45         1.44

Checksums

checksum = TRUE appends a Fletcher32 checksum to every chunk. Corrupted chunks are then detected when they are read, instead of silently returning bad values. The checksum is built into HDF5, costs very little, and works with every codec.

Rules and Automatic Adjustments

h5_compression() raises an error for settings that can never work:

• An unrecognized compression string, or a level outside the codec’s range.
• int_packing or float_rounding combined with an szip, ZFP, or bshuf- codec, or with blosc2_delta or blosc2_truncate.
• A blosc2_truncate of 0, or outside -51 to 52.

Other settings depend on the data being written, so h5_write() quietly adjusts them instead:

• Szip and ZFP fall back to the default pipeline (shuffle + gzip level 5) when the data type isn’t supported (see ZFP), or when the data is non-numeric, such as strings, data frames, or factors. Szip also falls back when the chunk’s fastest-changing dimension has fewer than 2 elements, and standalone ZFP when the chunks have no dimension longer than one, or more than four.
• Missing and non-finite values: ZFP falls back to the default pipeline, and float_rounding is skipped, for data that contains NA, NaN, or Inf, because those filters can’t store them. blosc2_truncate is skipped for data containing NA or NaN.
• Shuffling is skipped where it would be useless or harmful (see Shuffling).
• blosc2_delta and blosc2_truncate apply only to "blosc2-..." codecs other than "blosc2-ndlz" and "blosc2-zfp-...". With any other codec, they are ignored. blosc2_truncate also applies only to float32 and float64 data, and not to float32 data when it would drop all 23 mantissa bits.
• Scalars are never compressed.

When in doubt, h5_inspect() shows the pipeline that was actually applied.