cellmap_flow.dashboard.routes.finetune.common

Attributes

`logger`
`USER_PREFS_FILE`
`LOG_FILTER_PATTERNS`
`RESTART_PASSTHROUGH_KEYS`
`INSTANCE_TARGETS`
`SPARSE_MSE_NOTE`

Classes

TrainingSettings

The target and loss a job trains with (see training_settings).

Functions

`find_model_config`(model_name)
`current_chain`()	The dashboard's chains as step lists, `(input_norm, postprocess)`.
`viewer_position_and_scales`()
`session_store`()	The dashboard's sessions and volume registry (its session's dicts).
`ensure_corrections_storage`(output_path)
`load_user_prefs`()
`save_user_prefs`(prefs)
`resolve_finetune_session`(corrections_path_str)
`detect_sparse_annotations`(corrections_path)	Whether the session trains on painted (sparse) annotations.
`declared_output_type`(model_config)	The training target a model type declares (`finetune_output_type`), or None.
`annotation_dtype_for`(model_config)	The labels' dtype of a new annotation volume for `model_config`.
`autodetect_output_type`(model_config, output_type, offsets)
`training_settings`(*, output_type, loss_type, ...)	What a job trains with, from what was asked for and whether its session is sparse.
`build_restart_params`(data)
`get_lsf_job_id`(finetune_job)
`rewrite_minio_url_for_proxy`(minio_url[, request])	`minio_url` as the browser can reach it through a reverse proxy.
`write_volume_manifest`(volume)	Mark an annotation volume as trainable by writing its manifest.

Module Contents

cellmap_flow.dashboard.routes.finetune.common.logger

cellmap_flow.dashboard.routes.finetune.common.USER_PREFS_FILE

cellmap_flow.dashboard.routes.finetune.common.LOG_FILTER_PATTERNS = ['^\\s+base_model\\.\\S+\\.lora_', '^INFO:werkzeug:', '^Array metadata \\(scale=', '^Host...

cellmap_flow.dashboard.routes.finetune.common.RESTART_PASSTHROUGH_KEYS = ['lora_r', 'lora_alpha', 'num_epochs', 'batch_size', 'learning_rate', 'loss_type',...

cellmap_flow.dashboard.routes.finetune.common.find_model_config(model_name)

cellmap_flow.dashboard.routes.finetune.common.current_chain()

The dashboard’s chains as step lists, (input_norm, postprocess).

A new volume and a training manifest record them, so that the trainer normalizes its input as inference does, and the finetuned model’s YAML postprocesses as the dashboard does.

cellmap_flow.dashboard.routes.finetune.common.viewer_position_and_scales()

cellmap_flow.dashboard.routes.finetune.common.session_store(): The dashboard’s sessions and volume registry (its session’s dicts).

cellmap_flow.dashboard.routes.finetune.common.ensure_corrections_storage(output_path)

cellmap_flow.dashboard.routes.finetune.common.load_user_prefs()

cellmap_flow.dashboard.routes.finetune.common.save_user_prefs(prefs)

cellmap_flow.dashboard.routes.finetune.common.resolve_finetune_session(corrections_path_str)

cellmap_flow.dashboard.routes.finetune.common.detect_sparse_annotations(corrections_path)

Whether the session trains on painted (sparse) annotations.

Read from the volume the manifest points at: annotated voxels outside its imported crops are painted. This looked for per-chunk extracts marked source == “sparse_volume”, which are only written when there is no manifest – and every session has one now – so it was always False: the margin + distillation switch and the distance-model scribble guard in submit never fired, and mask_unannotated was never set. A session without a manifest cannot be trained, so it is not sparse either.

cellmap_flow.dashboard.routes.finetune.common.declared_output_type(model_config)

The training target a model type declares (finetune_output_type), or None.

Cellpose predicts flows, so its type says “flows”. A finetuned model trains as its base does, so its base_model_config is asked in turn (building that config builds no network).

cellmap_flow.dashboard.routes.finetune.common.INSTANCE_TARGETS = ('affinities', 'flows')

cellmap_flow.dashboard.routes.finetune.common.annotation_dtype_for(model_config)

The labels’ dtype of a new annotation volume for model_config.

uint8 (ids up to 255) for a model trained on foreground and background; uint16 (up to 65535) for one trained on instances (affinities, Cellpose’s flows), whose seeds number each object: 282 of a Cellpose model’s masks in one box did not fit a uint8 volume.

cellmap_flow.dashboard.routes.finetune.common.autodetect_output_type(model_config, output_type, offsets)

class cellmap_flow.dashboard.routes.finetune.common.TrainingSettings

The target and loss a job trains with (see training_settings).

output_type: str

loss_type: str

label_smoothing: float

distillation_lambda: float

mask_unannotated: bool

note: str = None

cellmap_flow.dashboard.routes.finetune.common.SPARSE_MSE_NOTE = 'Auto-switched to margin loss + distillation (lambda=0.5) for sparse annotations'

cellmap_flow.dashboard.routes.finetune.common.training_settings(*, output_type, loss_type, label_smoothing, distillation_lambda, sparse)

What a job trains with, from what was asked for and whether its session is sparse.

Some combinations cannot train, or would train the wrong thing, so they are replaced here, where the user sees it in the answer, rather than failing on the cluster: - sparse annotations with mse: margin loss, with distillation to the

base model at 0.5, which is how sparse annotations train (see the distance case below);

a distance target with sparse annotations: the interval loss, on the bounds the paint sets on each voxel’s distance, with distillation of at least 0.5 (see below);
a distance target otherwise: bce, without label smoothing;
a flow target (Cellpose): the flow loss, the only one it goes with, without label smoothing, sparse or not (it masks per channel itself).

A sparse session also masks its unannotated voxels out of the loss.

cellmap_flow.dashboard.routes.finetune.common.build_restart_params(data)

cellmap_flow.dashboard.routes.finetune.common.get_lsf_job_id(finetune_job)

cellmap_flow.dashboard.routes.finetune.common.rewrite_minio_url_for_proxy(minio_url, request=None)

minio_url as the browser can reach it through a reverse proxy.

session.minio.proxied_url, for request or else the Flask request being handled. Unless CELLMAP_FLOW_MINIO_PROXY_URL is set and the request came through a proxy (X-Forwarded-Host), and outside a request (a background import, a script), the URL is returned unchanged.

cellmap_flow.dashboard.routes.finetune.common.write_volume_manifest(volume)

Mark an annotation volume as trainable by writing its manifest.

create_dataloader requires this manifest: it is what points the trainer at the volume zarr to stream patches from, and it carries the patch geometry, the dense/sparse ratio and the dashboard’s chains. The good regions beside it are honoured through the same dataset. Without one, training raises rather than fall back to anything.

Returns the manifest path, or None when the volume record is too incomplete to describe (a resumed session whose .zattrs predates these fields, say). Nothing is guessed: such a session cannot be trained, and submit refuses it.