cellmap_flow.finetune.markers

The lines the trainer prints for the job manager to find in its log.

The training job’s loop (session_loop) and lora_trainer print these to stdout, which tee copies into training_log.txt; the job manager reads them back from there to follow the job, and the finetune tab reads the log it streams. The two sides can be different versions – a training job outlives a dashboard upgrade – so these only ever change compatibly.

The patterns are the ones the job manager has always used, unchanged.

Attributes

`FINETUNED_MODEL_YAML`
`TRAINING_ITERATION_COMPLETE`
`RESTART_FAILED`
`INFERENCE_SERVER_FAILED`
`RESTART_STATUS`
`TRAINING_DIVERGED`
`RESTARTING_TRAINING`
`WAITING_FOR_RESTART`
`EPOCH_START_RE`
`EPOCH_SUMMARY_RE`
`ITERATION_COMPLETE_RE`
`MODEL_YAML_RE`
`STATUS_MARKER_RE`
`SERVER_URL_RE`

Functions

emit(→ None)

Print marker (and its value) as a line of its own, at once.

Module Contents

cellmap_flow.finetune.markers.FINETUNED_MODEL_YAML = 'FINETUNED_MODEL_YAML:'

cellmap_flow.finetune.markers.TRAINING_ITERATION_COMPLETE = 'TRAINING_ITERATION_COMPLETE:'

cellmap_flow.finetune.markers.RESTART_FAILED = 'RESTART_FAILED:'

cellmap_flow.finetune.markers.INFERENCE_SERVER_FAILED = 'INFERENCE_SERVER_FAILED:'

cellmap_flow.finetune.markers.RESTART_STATUS = 'RESTART_STATUS:'

cellmap_flow.finetune.markers.TRAINING_DIVERGED = 'TRAINING_DIVERGED'

cellmap_flow.finetune.markers.RESTARTING_TRAINING = 'RESTARTING_TRAINING'

cellmap_flow.finetune.markers.WAITING_FOR_RESTART = 'WAITING_FOR_RESTART'

cellmap_flow.finetune.markers.EPOCH_START_RE

cellmap_flow.finetune.markers.EPOCH_SUMMARY_RE

cellmap_flow.finetune.markers.ITERATION_COMPLETE_RE

cellmap_flow.finetune.markers.MODEL_YAML_RE

cellmap_flow.finetune.markers.STATUS_MARKER_RE

cellmap_flow.finetune.markers.SERVER_URL_RE

cellmap_flow.finetune.markers.emit(marker: str, value=None) → None

Print marker (and its value) as a line of its own, at once.

Flushed, because the job manager is reading the log while the job runs.