Remote files and fsspec caching¶

Author: Luis Lopez @betolink

When working with large binary files stored in the cloud, tools like fsspec play a critical role in abstracting file access. These tools are input/output drivers to file systems. Libraries that understand how to read these binary formats do not know anything about remote files and fsspec is in charge of translating byte requests to remote byte requests (HTTP).

Most of the file formats in the scientific world were not designed for efficient cloud access as they require non-sequential reads, meaning client libraries can jump to specific byte ranges to read only what's needed, like a row group in a Parquet file or a variable slice in NetCDF. However, the way fsspec fetches data from remote sources is controlled by its caching strategy. This strategy determines how data is fetched, stored locally, and re-used. If this strategy isn't properly aligned with the access patterns of the file format, performance and reliability can (and will) quickly degrade.

Imagine reading a NetCDF file hosted on S3 with a default block cache of 5 MB. If a query requires multiple small reads scattered throughout the file, and each of these reads falls in a different 5 MB block, fsspec will repeatedly download new blocks, resulting in excessive data transfer and latency. Worse, if the file is very large and caching is disabled or limited (e.g. using simplecache with no chunk awareness), access may become painfully slow — or even fail if many tiny reads cause timeouts or bandwidth limits to be exceeded.

By tuning the cache size, choosing the right caching backend (like blockcache or filecache), and understanding how your format is read by your analysis tools (e.g., xarray, pandas, pyarrow), you can greatly reduce redundant downloads, improve throughput, and make interactive analysis over the cloud viable.

Aligning cache strategies on a per-file basis is not a simple task, therefore earthaccess won't try to fine tune your requests except for changing the default behavior in fsspec from readahead to blockcache that offers better access patterns and a more efficient cache invalidation strategy for binary files. Based on file size earthaccess will cache starting at 4MB up to 16MB for very large files (more than 1GB).

The max in-memory caching size will be 16MBx32 blocks = 512MB per file. All of this is configurable at runtime with the new open_kwargs parameter. Some examples of the usage of this new config dictionary are detailed in this notebook.

• Efficiency: Avoids repeated downloads of the same data during random-access reads.
• Performance: Reduces latency and bandwidth usage, especially for cloud-hosted files.
• Scalability: Supports parallel or repeated access patterns without overloading remote servers.

NetCDF and HDF5 files¶

A considerable amount of remote sensing data is stored in HDF5 and NetCDF formats. They are hierarchical, self-describing formats widely used in Earth science due to their ability to store multidimensional arrays, metadata, and complex data structures efficiently. However, their design assumes relatively fast random access to local storage, which becomes a bottleneck when accessed over high-latency networks like HTTP or S3.

HDF5 and NetCDF files are typically organized into chunks—contiguous blocks of data that are compressed and stored separately. When reading a subset of a dataset (e.g., a time slice or spatial region), only the relevant chunks should be fetched. But without proper caching, each byte-range request for a chunk can result in a separate HTTP GET request, leading to thousands of small requests (Ro to Rn in the figure) for large analyses. The diagram below illustrates how traditional HDF5 access results in scattered, inefficient requests, while cloud-optimized layouts enable sequential or sparse access with fewer round trips:

This is where fsspec's blockcache becomes essential. By aligning the cache block size with the HDF5/NetCDF chunk size (or a multiple thereof), we minimize redundant fetches and ensure that once a chunk is downloaded, it remains available for reuse during subsequent accesses. For example, if a NetCDF variable uses 1 MB chunks, a 4–16 MB block cache ensures that multiple chunks are retained per block, reducing round trips.

Additionally, because HDF5 files often contain shared metadata (e.g., dimensions, attributes, group structures), caching the beginning of the file (where some of this metadata resides) can dramatically speed up repeated opens or inspections. earthaccess leverages blockcache with sensible defaults to preserve both data and metadata across requests, making workflows using xarray or h5py over S3 much more performant.

Cloud optimized files (Zarr / COG / etc / Co-HDF5)¶

As cloud-native workflows mature, new formats and adaptations have emerged to address the limitations of traditional binary formats in object storage environments. These cloud-optimized formats are designed for efficient, parallel, and distributed access over HTTP/S3 with minimal latency.

• Zarr: A chunked, compressed, and directory-structured format that natively supports cloud storage. Each chunk is stored as a separate object, enabling true parallel I/O and eliminating the need for byte-range requests. When used with fsspec, Zarr datasets can be read efficiently even with minimal caching, since each chunk is an independent resource.

• Cloud Optimized GeoTIFF (COG): A GeoTIFF variant with internal tiling and overviews organized so that key image metadata is at the front of the file, and tiles are laid out sequentially. This allows fast previews and spatial subsetting via byte-range requests without downloading the entire file.

• Co-HDF5 (Cloud-Optimized HDF5): An emerging pattern where HDF5 files are rechunked and reorganized to improve cloud access performance. This includes placing metadata at the beginning, aligning chunks to network block boundaries, and minimizing cross-chunk references.

These formats work best when paired with intelligent caching strategies. While Zarr benefits less from large block caches (since each chunk is already a discrete object), COG and Co-HDF5 still rely heavily on byte-range requests and thus benefit significantly from blockcache with appropriately sized blocks.

Credit: Earthmover "What is Zarr"

Ultimately, the key to high-performance cloud data access lies in matching the caching strategy to the data layout, access pattern, and underlying format. With earthaccess, we aim to provide defaults that "just work" while giving advanced users full control via open_kwargs to tune behavior as needed.

💡 Pro Tip: Always inspect your data’s chunking structure (e.g., using zarr.tree() or h5py.File(...).visititems()) before setting cache parameters. Aligning cache and chunk sizes can yield 10x+ performance improvements.

xarray and earthaccess.open()¶

xarray has become the de facto library to open and analyze n-dimensional scientific data in Python, thus the use of fsspec to open our remote files has the advantage of being interoperable with any xarray backend(or file engine) that can accept "file-like" objects.

Now that we have results from CMR we can open the files, in this case a single HDF5 level 2 (non gridded) swath. One thing that this particular collection will highlight is that when dealing with non cloud-optimized files each dimension and variable will add overhead on the range requests. ATL03 contains more than 900 of them, and if we use libraries like xarray that try eagerly to fetch metadata to decode variables and dimensions our open_dataset() operation in xarray will be exponentially slow.

Let's open the file and inspect the new caching defaults from earthaccess!

Here we are seeing the new caching defaults when we open files using earthaccess, instead of readahead we are using the blockcache and earthaccess adjusted the block size to 8MB. This will helps us not re-fetching data when we in turn use xr.open_dataset()

Better caching¶

xarray decoded a few variables here and touched many parts of the file, what does this mean regardless file access? if we inspect the cache again will see the number of requests made and how many of them hit a cached block.

Defaults are bad for random access: testing readahead on HDF5, ATL03 data¶

Now let's explore what happens with fsspec defaults by overriding the config with open_kwargs.

This we can see we requested an order of magnitude more information with considerable more cache misses despite the block_size being similar in size. Most of it is due how readahead does cache invalidation for random access.

Old defaults:

• 3120435184 requested bytes
• 604 cache misses
• Wall time: 11min

New defaults:

• 184549376 requested bytes
• 22 cache misses
• Wall time: 29.7 s

Here we are seeing over an order of magnitude improvement without changing anything but the way we fetch data. Of course things can improve even further if we deal with netCDF4/HDF5 files with fewer variables and/or especially with cloud-optimized HDF5.

We can test that with version 7 of the same file.

Wait, what happened now?¶

We didn't optimize the cache and yet this file in version 7 took only ~17 seconds to open, why? well this is a cloud optimized HDF5 file, hence all the metadata fits in the first request fsspec does. But this is not always the case, depending on how big a file is and how many variables and metadata has. Now we's use new earthaccess defaults and see if we can improve this even more.

Note that cloud optimized HDF5 and NetCDF4 is a very recent and for now not a lot of datasets will be released in this format

Benchmarks¶

• blockcache is better than readahead for binary files (HDF5, netCDF4, HDF) that require random access. earthaccess now uses blockcache by default and the block size is adjusted by file size.
  • small files < 100MB: 4MB
  • medium files < 1GB: 8MB
  • big file: > 1GB: 16MB
• optimal caching depends very much on the access pattern and the internal chunking of a given dataset, as of now that information is not easily available beforehand, a database that contains this kind of information will be really helpful to the community.

Here is a table that shows the performance gains by this new caching schema for a few selected popular binary datasets in the NASA Earth archives.

CAUTION: Running this can take up to 1 hour if we are not in-region.

TL;DR; earthaccess.open() then vs now¶

All of this to say that with the upcoming release of earthaccess we'll enjoy speedups in our workflows from marginal to order of magnitude depending on the dataset.