cellmap_flow.finetune.job_manager.submit

What a submission runs, and where.

The manager’s submit_finetuning_job validates a request and writes the run’s directory; this module answers the questions on the way:

resolve_model_type: the trainer’s --model-type for a model config, refusing one it cannot train; model_entry: the model as the trainer takes it with --model-entry, for the types that go that way;
find_checkpoint, count_corrections: the checkpoint to start from, and whether the session has anything to train on;
model_settings: the channels and voxel sizes to train with;
extract_data_path_from_corrections: the raw data the job’s server serves;
build_command: the trainer’s shell command line, which LSF runs on a GPU node (python -m cellmap_flow.finetune.finetune_cli);
launch: that command on LSF, or here when there is no bsub.

Attributes

`logger`
`FLAG_MODEL_TYPES`
`SYNCED_TEE`
`TEE_PYTHON`

Functions

`trainable_model_types`(→ list)	The model types the trainer can train: those that say so (`ModelConfig.finetunable`).
`resolve_model_type`(→ str)	Infer the finetuning CLI model type from the model config.
`model_entry`(→ Optional[dict])	The model as the trainer takes it with --model-entry, or None for a type
`find_checkpoint`(→ Optional[pathlib.Path])	The checkpoint to finetune from: `override`, else the config's own.
`count_corrections`(→ int)	How many corrections the session holds.
`model_settings`(model_config, warned)	(channels, input voxel size, output voxel size) to train `model_config` with.
`extract_data_path_from_corrections`(→ str)	Extract dataset path from corrections metadata.
`build_command`(→ str)	Build the shell command used to launch finetuning, for `bash -c`.
`launch`(job_name, command, queue, charge_group[, walltime])	Start `command` (build_command's) as the job `job_name`.

Module Contents

cellmap_flow.finetune.job_manager.submit.logger

cellmap_flow.finetune.job_manager.submit.FLAG_MODEL_TYPES

cellmap_flow.finetune.job_manager.submit.trainable_model_types() → list: The model types the trainer can train: those that say so (ModelConfig.finetunable).

cellmap_flow.finetune.job_manager.submit.resolve_model_type(model_config) → str

Infer the finetuning CLI model type from the model config.

Raises ValueError for a type the trainer cannot train, rather than submitting a GPU job whose argparse exits with code 2.

cellmap_flow.finetune.job_manager.submit.model_entry(model_config, model_type: str, checkpoint_path: pathlib.Path | None) → dict | None

The model as the trainer takes it with –model-entry, or None for a type that goes by its own flags (FLAG_MODEL_TYPES).

The entry is the model’s to_dict(), so the trainer builds the model the dashboard serves; checkpoint_path (find_checkpoint’s) replaces the entry’s own checkpoint, which is how an override reaches a Fly model. A model with an env always goes as an entry: no other flag carries the env, and without it the serving YAMLs the trainer writes would run the finetuned model in the default environment. Its type’s default_env needs no entry: the trainer builds the same type, and a finetuned model runs where its base type does.

cellmap_flow.finetune.job_manager.submit.find_checkpoint(model_config, override=None) → pathlib.Path | None

The checkpoint to finetune from: override, else the config’s own.

None when there is neither: a script model passes its script instead. Raises ValueError for a checkpoint that does not exist.

cellmap_flow.finetune.job_manager.submit.count_corrections(corrections_path: pathlib.Path) → int

How many corrections the session holds.

Raises ValueError when there is nothing to train on.

cellmap_flow.finetune.job_manager.submit.model_settings(model_config, warned: set)

(channels, input voxel size, output voxel size) to train model_config with.

Each is the config’s own, else its geometry’s, else a guess: channels [“mito”], 16 nm voxels. A guess is logged once per model; warned holds the names of the models already warned about.

cellmap_flow.finetune.job_manager.submit.extract_data_path_from_corrections(corrections_path: pathlib.Path) → str

Extract dataset path from corrections metadata.

The manifest’s raw_dataset_path first – it is what the trainer reads – and only then the first correction zarr’s attrs, which a crop zarr may not have.

cellmap_flow.finetune.job_manager.submit.SYNCED_TEE = ''

cellmap_flow.finetune.job_manager.submit.TEE_PYTHON = '/opt/hostedtoolcache/Python/3.11.16/x64/bin/python'

cellmap_flow.finetune.job_manager.submit.build_command(*, model_config, model_type: str, checkpoint_path: pathlib.Path | None, corrections_path: pathlib.Path, output_dir: pathlib.Path, log_file: pathlib.Path, channels: List[str], input_voxel_size: List[int], output_voxel_size: List[int], lora_r: int, num_epochs: int, batch_size: int, learning_rate: float, loss_type: str, label_smoothing: float, distillation_lambda: float | None, distillation_scope: str, margin: float, auto_serve: bool, serve_data_path: str | None, mask_unannotated: bool, balance_classes: bool, augment: bool, output_type: str, select_channel: int | None, offsets: str | None, models_dir: pathlib.Path | None = None, queue: str | None = None, charge_group: str | None = None) → str: Build the shell command used to launch finetuning, for bash -c.

cellmap_flow.finetune.job_manager.submit.launch(job_name: str, command: str, queue: str | None, charge_group: str | None, walltime: str | None = None)

Start command (build_command’s) as the job job_name.

On LSF with one GPU when bsub is available, for walltime (“HH:MM” or minutes) or else the site’s; otherwise as a process here. Returns its LSFJob or LocalJob; raises RuntimeError if it did not start.