cellmap_flow.finetune.job_manager.restart

Asking a finetune job to train again, or to stop training early.

The trainer’s inference server, in the same process as its training loop, takes POST <server>/__control__/restart and .../__control__/stop with the job’s restart token (written into the run’s directory at submit; serving.restart_token) in the X-Restart-Token header. A restart’s body is {"restart": true, "timestamp", "params"}, params being the settings to change; a stop’s {"requested_at", "reason"}. When the server cannot be reached, or has no URL yet – it starts once the first iteration is trained – the same body goes into the run’s restart_signal.json or stop_signal.json, which the trainer also watches (finetune.session_loop). It is written whole or not at all (finetune.json_files.write_json_atomically): the trainer reads it as soon as it exists, and ends the job if a restart’s is not JSON. Jobs outlive dashboard upgrades, so both ways are a protocol: a trainer from before the stop endpoint answers it 404, and gets the file.

Attributes

logger

Functions

request_restart(job[, updated_params]) → None	Ask job's trainer to train again, with updated_params changed.
request_stop(job) → None	Ask job's trainer to stop training after the epoch it is in (Stop Early).

Module Contents

cellmap_flow.finetune.job_manager.restart.logger

cellmap_flow.finetune.job_manager.restart.request_restart(job: cellmap_flow.finetune.job_manager.state.FinetuneJob, updated_params: Dict[str, Any] | None = None) → None

Ask job’s trainer to train again, with updated_params changed.

The job is then RUNNING a new iteration from epoch 0, its server not ready until the iteration completes, and job.params updated. Raises ValueError, and sends nothing, unless the job can take a restart (state.can_restart).

cellmap_flow.finetune.job_manager.restart.request_stop(job: cellmap_flow.finetune.job_manager.state.FinetuneJob) → None

Ask job’s trainer to stop training after the epoch it is in (Stop Early).

It then exports what it has trained and serves it, as at the end of any iteration, and waits for a restart. Raises ValueError, and sends nothing, unless a training loop is running (the RUNNING job whose server is not serving a finished iteration; the browser’s backend refuses the same).