cellmap_flow.dashboard.finetune_utils
Helper functions for finetuning annotation workflows.
Handles MinIO server management, annotation zarr creation, and periodic synchronization of annotations between MinIO and local disk.
Attributes
Functions
|
Get or create a timestamped session directory for the given base output path. |
Get the local IP address for MinIO server. |
|
|
Find an available port pair for MinIO server (API on port, console on port+1). |
|
Create a correction zarr with OME-NGFF v0.4 metadata. |
|
Create a sparse annotation volume zarr covering the full dataset extent. |
|
Ensure MinIO is running and upload zarr file. |
|
Sync a single annotation crop from MinIO to local filesystem. |
|
Sync all annotations from MinIO to local disk. |
|
Extract a correction entry from a single annotated chunk in a sparse volume. |
|
Sync an annotation volume from MinIO, detect annotated chunks, extract corrections. |
Background thread function to periodically sync annotations from MinIO. |
|
Start the periodic annotation sync thread if not already running. |
Module Contents
- cellmap_flow.dashboard.finetune_utils.minio_state
- cellmap_flow.dashboard.finetune_utils.annotation_volumes
- cellmap_flow.dashboard.finetune_utils.output_sessions
- cellmap_flow.dashboard.finetune_utils.logger
- cellmap_flow.dashboard.finetune_utils.get_or_create_session_path(base_output_path: str) str
Get or create a timestamped session directory for the given base output path.
If a session already exists for this base path, reuse it. Otherwise, create a new timestamped subdirectory.
- Parameters:
base_output_path – Base output directory (e.g., “output/to/here”)
- Returns:
Timestamped session path (e.g., “output/to/here/20260213_123456”)
- cellmap_flow.dashboard.finetune_utils.get_local_ip()
Get the local IP address for MinIO server.
- cellmap_flow.dashboard.finetune_utils.find_available_port(start_port=9000)
Find an available port pair for MinIO server (API on port, console on port+1).
- cellmap_flow.dashboard.finetune_utils.create_correction_zarr(zarr_path, raw_crop_shape, raw_voxel_size, raw_offset, annotation_crop_shape, annotation_voxel_size, annotation_offset, dataset_path, model_name, output_channels, raw_dtype='uint8', create_mask=False)
Create a correction zarr with OME-NGFF v0.4 metadata.
- Structure:
- crop_id.zarr/
raw/s0/ (uint8, shape=raw_crop_shape) annotation/s0/ (uint8, shape=annotation_crop_shape) mask/s0/ (optional, uint8, shape=annotation_crop_shape) .zattrs (metadata)
- Returns:
bool, info: str)
- Return type:
(success
- cellmap_flow.dashboard.finetune_utils.create_annotation_volume_zarr(zarr_path, dataset_shape_voxels, output_voxel_size, dataset_offset_nm, chunk_size, dataset_path, model_name, input_size, input_voxel_size, claimed_output_voxel_size=None, claimed_input_voxel_size=None, input_norm_config=None)
Create a sparse annotation volume zarr covering the full dataset extent.
The volume has chunk_size = model output_size so each chunk maps to one training sample. Only metadata files are created (no chunk data), so the zarr is tiny regardless of dataset size.
Label scheme: 0=unannotated (ignored), 1=background, 2=foreground.
- Parameters:
output_voxel_size – the EFFECTIVE voxel sizes used for the actual grid alignment (typically the dataset’s closest available scale to the model’s claimed voxel size).
input_voxel_size – the EFFECTIVE voxel sizes used for the actual grid alignment (typically the dataset’s closest available scale to the model’s claimed voxel size).
claimed_output_voxel_size – optional — the model’s originally-declared voxel sizes, recorded for provenance.
claimed_input_voxel_size – optional — the model’s originally-declared voxel sizes, recorded for provenance.
- Returns:
bool, info: str)
- Return type:
(success
- cellmap_flow.dashboard.finetune_utils.ensure_minio_serving(zarr_path, crop_id, output_base_dir=None)
Ensure MinIO is running and upload zarr file.
- Parameters:
zarr_path – Path to zarr file to upload
crop_id – Unique identifier for the crop
output_base_dir – Base output directory (MinIO will use output_base_dir/.minio)
- Returns:
MinIO URL for the zarr file
- cellmap_flow.dashboard.finetune_utils.sync_annotation_from_minio(crop_id, force=False)
Sync a single annotation crop from MinIO to local filesystem.
- Parameters:
crop_id – Crop ID to sync
force – Force sync even if not modified
- Returns:
True if synced successfully
- Return type:
bool
- cellmap_flow.dashboard.finetune_utils.sync_all_annotations_from_minio(force: bool = True)
Sync all annotations from MinIO to local disk.
- Returns:
Number of annotations synced, or -1 if MinIO is not initialized.
- cellmap_flow.dashboard.finetune_utils.extract_correction_from_chunk(volume_id, chunk_indices, volume_metadata)
Extract a correction entry from a single annotated chunk in a sparse volume.
Reads the annotation chunk, extracts raw data with context padding, and creates a standard correction zarr entry compatible with CorrectionDataset.
- Parameters:
volume_id – Volume identifier
chunk_indices – Tuple (cz, cy, cx) of chunk indices
volume_metadata – Volume metadata dict
- Returns:
True if correction was created (chunk had annotations)
- Return type:
bool
- cellmap_flow.dashboard.finetune_utils.sync_annotation_volume_from_minio(volume_id, force=False)
Sync an annotation volume from MinIO, detect annotated chunks, extract corrections.
Steps: 1. Sync the full annotation zarr from MinIO to local disk 2. List chunk files in MinIO to find annotated chunks 3. For each new annotated chunk, extract raw data and create correction entry
- Returns:
True if any corrections were created
- Return type:
bool
- cellmap_flow.dashboard.finetune_utils.periodic_sync_annotations()
Background thread function to periodically sync annotations from MinIO.
- cellmap_flow.dashboard.finetune_utils.start_periodic_sync()
Start the periodic annotation sync thread if not already running.