Data paths

Wherever cellmap-flow takes a dataset – data_path in a YAML file, -d on the command line, the dashboard, a viewer layer’s path – it takes the same kinds of path, and reads the same metadata and voxels from a URL as from the same files on disk.

What a path can be

Format	Examples
zarr v2, OME-NGFF or funlib attributes	`/nrs/cellmap/data/jrc_hela-2/jrc_hela-2.zarr/recon-1/em/fibsem-uint8` `s3://janelia-cosem-datasets/jrc_hela-2/jrc_hela-2.zarr/recon-1/em/fibsem-uint8/s1` `https://janelia-cosem-datasets.s3.amazonaws.com/jrc_hela-2/jrc_hela-2.zarr/recon-1/em/fibsem-uint8` `gs://bucket/data.zarr/raw`
zarr v3 (OME-NGFF 0.5)	`/data/v3.zarr/em`, or the same at an `http(s)://`, `s3://` or `gs://` URL
N5	`/data/x.n5/em/fibsem-uint16`, `s3://janelia-cosem-datasets/jrc_hela-2/jrc_hela-2.n5/em/fibsem-uint16/s0`
neuroglancer precomputed	`precomputed:///data/volume`, `precomputed://gs://flyem-male-cns/em/em-clahe-jpeg`, `precomputed://https://storage.googleapis.com/flyem-male-cns/em/em-clahe-jpeg`, `precomputed://s3://bucket/volume`, and a bare `gs://flyem-male-cns/em/em-clahe-jpeg`

Paths run on past their container into the group or array inside it. The container ends at the last path component ending in .zarr or .n5; without one, it is the nearest directory up from the path with a .zgroup (on disk and at a URL alike), or the node’s own zarr.json for zarr v3.

A multiscale group (OME-NGFF multiscales, or a precomputed volume’s scales) is read at the level for the voxel size asked for, or at its first level when none is. A path to one level (…/s1, a precomputed volume’s …/s2) is read at that level.
A group without multiscales is read at its first array: the first subdirectory, sorted, that is one on disk; at a URL, which cannot be listed, its s0.
A shell-escaped space (my\ data.zarr) is unescaped in a local path, never in a URL.

A gs:// path is a precomputed volume unless it has a .zarr or .n5 component. That is how cellmap-flow has always read gs:// paths. A zarr whose names lack the suffix (such as CMIP6’s gs://cmip6/CMIP6/…/tas) is read through its public URL instead: https://storage.googleapis.com/cmip6/CMIP6/…/tas.

Remote data and credentials

Everything is read with tensorstore: no extra packages are needed for any scheme. Public data needs no credentials.

s3://

Read anonymously first, so a public bucket is read whatever AWS credentials are set up: S3 refuses even a public read signed with an expired or foreign key. When the bucket refuses an anonymous read, cellmap-flow tries again with the AWS default credentials: the AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY (and AWS_SESSION_TOKEN) environment variables, ~/.aws/credentials and ~/.aws/config (AWS_PROFILE picks the profile), then an EC2 instance’s role.

An S3-compatible store (MinIO, Ceph, Wasabi) is named with AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL, and its region with AWS_REGION or AWS_DEFAULT_REGION.

gs://

Read with Google application-default credentials when there are any: GOOGLE_APPLICATION_CREDENTIALS, or the file gcloud auth application-default login writes. Without them the read is anonymous. When Google Cloud Storage refuses those credentials (an expired or revoked login refuses even public buckets), the bucket is read at its public URL, https://storage.googleapis.com/<bucket>/<path>.

http:// and https://

Read as they are, with no credentials.

The way in that worked is remembered for each bucket, so only the first read of a private bucket is refused. A failed metadata request is retried for about 5 seconds before the path is reported unreadable.

The raw layer in the viewer

By default (wrap_raw: true) the raw layer is served by cellmap-flow itself, through the same reader as inference, so it works for every path above and shows the data through the input normalizers.

With wrap_raw: false neuroglancer reads the data in the browser, so the browser must be able to reach it. Its source is the dataset’s URL with the format in front: zarr://s3://…, zarr://gs://…, zarr://https://…, n5://…, or precomputed://gs://… (a precomputed volume, never one of its scales). The browser has none of the credentials above and cannot open local files, so a private bucket or a local path needs wrap_raw: true.