cellmap_flow.finetune.job_manager.state

A finetune job’s record, its statuses, and what moves a job between them.

A job moves on: - what the scheduler (LSF, or the local process) says of it, which the

monitor asks every few seconds: on_scheduler_status. A job the scheduler says has completed becomes COMPLETED only once the monitor has found its export (monitor.complete_job), and FAILED if it has not;

the status markers the trainer prints in its log: on_status_marker;
the dashboard: a restart, which only a job that can_restart takes (start_iteration), and a cancel (FinetuneJobManager.cancel_job).

Once a job’s status is final (TERMINAL_STATUSES), neither the scheduler nor the log moves it again.

Attributes

`logger`
`TERMINAL_STATUSES`

Classes

`JobStatus`	Status of a finetuning job.
`FinetuneJob`	Track a finetuning job with metadata, status, and training progress.

Functions

`on_scheduler_status`(→ Optional[JobStatus])	Move `job` on what its scheduler says of it; once it has ended, how.
`on_status_marker`(→ None)	Move `job` on one of the trainer's status markers
`can_restart`(→ bool)	Whether `job`'s trainer can take a restart request.
`start_iteration`(→ None)	A new training iteration starts: its progress from zero, and its server

Module Contents

cellmap_flow.finetune.job_manager.state.logger

class cellmap_flow.finetune.job_manager.state.JobStatus(*args, **kwds)

Status of a finetuning job.

PENDING = 'PENDING'

RUNNING = 'RUNNING'

COMPLETED = 'COMPLETED'

FAILED = 'FAILED'

CANCELLED = 'CANCELLED'

WAITING_FOR_RESTART = 'WAITING_FOR_RESTART'

cellmap_flow.finetune.job_manager.state.TERMINAL_STATUSES

class cellmap_flow.finetune.job_manager.state.FinetuneJob

Track a finetuning job with metadata, status, and training progress.

Manages lifecycle from submission through completion, including inference server state.

job_id: str

lsf_job: cellmap_flow.jobs.lsf.LSFJob | None

model_name: str

output_dir: pathlib.Path

params: Dict[str, Any]

status: JobStatus

created_at: datetime.datetime

log_file: pathlib.Path

finetuned_model_name: str | None = None

model_yaml_path: pathlib.Path | None = None

current_epoch: int = 0

total_epochs: int = 10

latest_loss: float | None = None

inference_server_url: str | None = None

inference_server_ready: bool = False

corrections_path: pathlib.Path | None = None

cancel_requested: bool = False

to_dict() → Dict[str, Any]: Convert to dictionary for JSON serialization.

cellmap_flow.finetune.job_manager.state.on_scheduler_status(job: FinetuneJob, lsf_status) → JobStatus | None

Move job on what its scheduler says of it; once it has ended, how.

lsf_status is a jobs.spec.JobStatus. Returns None while the job lives, else the status it ended with. A job whose cancel was asked for has been cancelled however its end is reported. RUNNING moves a pending job on and leaves one waiting for a restart waiting; PENDING makes it pending.

COMPLETED is returned but not set: the finetune tab stops polling, and the log stream says done, at the first final status they see, so the job keeps its status until the monitor has checked what it exported.

cellmap_flow.finetune.job_manager.state.on_status_marker(job: FinetuneJob, marker: str) → None: Move job on one of the trainer’s status markers (markers.STATUS_MARKER_RE), unless its status is final.

cellmap_flow.finetune.job_manager.state.can_restart(job: FinetuneJob) → bool

Whether job’s trainer can take a restart request.

Only a trainer that is alive and waiting can take a restart. A COMPLETED job has exited: nothing reads the request, and the monitor that would have seen it through has stopped, so it used to sit at RUNNING forever. WAITING_FOR_RESTART also covers a later iteration that diverged: its server is up but not marked ready, and the restart the user needed was refused. A RUNNING job whose server is up is a trainer that predates the WAITING_FOR_RESTART marker.

cellmap_flow.finetune.job_manager.state.start_iteration(job: FinetuneJob) → None: A new training iteration starts: its progress from zero, and its server not ready again until it completes. The server’s URL is kept.