cellmap_flow.finetune.data.reader

Reading a finetune patch: the annotation, and the raw around the same place.

Annotation voxel v spans corner_nm + v * output_voxel_size (the corner from session.volume.volume_corner_nm). A patch is whole annotation voxels, [c - size/2, c + size/2) around its centre c, and the raw is read around that same patch, so every label is paired with the raw it covers.

A one-slice patch, a 2D network’s, can be cut across y or x instead of z (slice_axes): a z–y slice for paint drawn in one. It is handed over turned to the network’s orientation, its slice first, so the network sees a side slice as it sees an x–y one.

Attributes

logger

Classes

PatchReader

Reads the annotation and raw patches of one volume, and normalizes the raw.

Functions

`slice_axes`(depth_axis) → Tuple[int, int, int]	The volume's axes in the order a patch cut across `depth_axis` is handed over.
`across`(sizes, depth_axis) → numpy.ndarray	`sizes` given in `slice_axes(depth_axis)` order, put in the volume's (z, y, x).
`input_normalizers`(input_norm_config) → list	The normalizer objects of an `input_norm` config; [] when it builds none.

Module Contents

cellmap_flow.finetune.data.reader.logger

cellmap_flow.finetune.data.reader.slice_axes(depth_axis: int) → Tuple[int, int, int]

The volume’s axes in the order a patch cut across depth_axis is handed over.

That axis first, where z is for a patch across z, then the other two in order: across x (a z–y slice), (x, z, y), rows z and columns y, which is how Cellpose’s 3D mode cuts that plane too (core.run_3D). A patch’s sizes are given in this order, and the patch is read with them put back in the volume’s (across).

cellmap_flow.finetune.data.reader.across(sizes, depth_axis: int) → numpy.ndarray: sizes given in slice_axes(depth_axis) order, put in the volume’s (z, y, x).

cellmap_flow.finetune.data.reader.input_normalizers(input_norm_config) → list

The normalizer objects of an input_norm config; [] when it builds none.

The config arrives in either shape: the name-keyed dict a YAML gives (e.g. {“MinMaxNormalizer”: {…}, “LambdaNormalizer”: {…}}), or the list of {“name”: …, …} dicts the dashboard POSTs – a list on purpose, because jsonify sorts dict keys and the order of these steps changes what they compute.

Do not coerce it with dict(). Over a list whose entries have exactly two keys it does not raise; it silently returns {“name”: “expression”}, which builds no normalizers at all, and the trainer would then train on raw uint8 while inference sees [-1, 1]. get_normalizations() understands both shapes, so it gets the config untouched.

class cellmap_flow.finetune.data.reader.PatchReader(volume_zarr_path: str, raw_dataset_path: str, input_size_voxels, output_size_voxels, input_voxel_size_nm, output_voxel_size_nm, corner_nm: numpy.ndarray, shape_voxels: numpy.ndarray, input_norm_config=None, resample: bool = False)

Reads the annotation and raw patches of one volume, and normalizes the raw.

The raw is normalized here, with the session’s own chain: the dashboard normalizes raw through its process_chain() before inference, but the trainer is a separate process whose chain is empty, so without this it would train on raw uint8 while inference sees normalized [-1, 1]. The channel of a multi-channel raw is the one that chain picks (its ChannelSelector) for the same reason: read through the empty chain it was channel 0, whichever one inference read.

The arrays are opened on first read. In training that is in a loader worker, after the dataset was pickled to it: no open tensorstore handle is ever pickled.

volume_zarr_path

raw_dataset_path

input_size

output_size

input_voxel_size

output_voxel_size

corner_nm

shape_voxels

input_norm_config

normalizers = []

resample = False

snap(centre_voxels: numpy.ndarray, depth_axis: int = 0) → numpy.ndarray

The nearest centre whose patch, [c - size/2, c + size/2), is whole voxels.

Without it the annotation patch would be cut at int(c - size/2) while the raw was read around c itself: half a voxel apart for an odd output size, and for a good region, whose centre falls anywhere. depth_axis is the axis the patch is cut across (slice_axes).

annotation(centre_voxels: numpy.ndarray, depth_axis: int = 0) → numpy.ndarray

The annotation patch around a snapped centre, in the volume’s own dtype.

Cut across depth_axis and handed over in slice_axes order, so its shape is output_size whichever axis it is cut across. Out-of-bounds voxels are 0, unannotated (worldarray pads a read past the array), which the trainer’s loss masks out when mask_unannotated is on.

The dtype is the volume’s because an instance-correction volume is uint16 or uint32 (instance id + 1): a uint8 patch wrapped its ids modulo 256, so instance 255 became unannotated, 256 background, and 257 merged with instance 1. The dataset casts the patch to float32, which holds every id below 2**24 exactly.

raw(centre_voxels: numpy.ndarray, depth_axis: int = 0) → numpy.ndarray

The input_size raw patch centred on the annotation patch, as stored.

Cut across depth_axis and turned as the annotation patch is. Read on the channel the session’s chain picks, with normalize=False, so no process-wide norms apply; normalize applies the session’s own afterwards.

The patch is a box of raw voxels: input_size of them, from the one the patch’s lower edge falls in. It used to go through a whole-nm Roi, whose Coordinate truncated the edge and the shape, so at 5.24 nm or 10.48 nm the patch came out a voxel short and started a voxel early. Where the edge is whole nm on a whole-nm grid (the 8 and 16 nm levels) both give the same box.

normalize(raw: numpy.ndarray) → numpy.ndarray: raw through the session’s normalizers, in order, as apply_norms() does for inference.