cellmap_flow.io.store

Where a dataset’s files are, and reading them: one tensorstore kvstore.

A location is a local directory or an http(s)://, s3:// or gs:// URL. io.metadata reads the JSON documents there (.zarray, attributes.json, zarr.json, a precomputed info) with read_json, and io.source opens the arrays on the same kvstores (with_access), so the metadata and the voxels come from the same place by the same route, and a URL is read the way the same files on disk are.

Credentials (docs/source/data_paths.rst) are tried only when a public read is refused:

s3://: anonymously first, so a public bucket is read whatever AWS credentials are configured (S3 refuses even a public read signed with a stale or foreign key); then with the AWS default chain (environment, ~/.aws, instance metadata). AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL names an S3-compatible endpoint, and AWS_REGION or AWS_DEFAULT_REGION its region.
gs://: Google application-default credentials if there are any, else anonymous (tensorstore’s own rule); then the bucket’s public https URL, for when those credentials are broken. TENSORSTORE_GCS_HTTP_URL, which tensorstore reads for another endpoint, moves that URL with it.
http(s):// and local paths: no credentials.

Which way worked is remembered per bucket, so only the first read of a private bucket is refused.

Attributes

`logger`
`URL_SCHEMES`
`GCS_PUBLIC_URL`
`T`

Functions

`is_url`(→ bool)
`kvstores`(→ List[dict])	The tensorstore kvstore specs that read `location`, a directory, in
`with_access`(→ T)	`attempt(spec)` with `location`'s kvstore specs in turn, moving on
`read`(→ Optional[bytes])	The file `key` in the directory `location`; None when it is not
`read_json`(→ Optional[dict])	The JSON document `key` in the directory `location`; None when it
`exists`(→ bool)

Module Contents

cellmap_flow.io.store.logger

cellmap_flow.io.store.URL_SCHEMES = ('http://', 'https://', 's3://', 'gs://')

cellmap_flow.io.store.GCS_PUBLIC_URL = 'https://storage.googleapis.com'

cellmap_flow.io.store.T

cellmap_flow.io.store.is_url(location: str) → bool

cellmap_flow.io.store.kvstores(location: str) → List[dict]: The tensorstore kvstore specs that read location, a directory, in the order to try them (see the module docstring). Each spec’s path ends in “/”, so a key read from it is a file in that directory.

cellmap_flow.io.store.with_access(location: str, attempt: Callable[[dict], T]) → T: attempt(spec) with location’s kvstore specs in turn, moving on only while access is refused; the one let in is tried first next time. Any other error, and the last refusal, is raised.

cellmap_flow.io.store.read(location: str, key: str) → bytes | None: The file key in the directory location; None when it is not there.

cellmap_flow.io.store.read_json(location: str, key: str) → dict | None: The JSON document key in the directory location; None when it is not there.

cellmap_flow.io.store.exists(location: str, key: str) → bool