cellmap_flow.finetune.job_manager.tailer

The trainer’s log, as the job manager reads it.

The trainer tees everything it prints into its run’s training_log.txt, and the lines the manager reads are the markers in finetune.markers. A job outlives a dashboard upgrade, so both sides of those lines are a protocol.

• LogTailer: the lines appended since the last read, whole lines only, and what all the lines read so far say of the finished iterations; the monitor follows a running job’s log with it, reading each line once.

• Iterations: what a log’s lines, read in order, say of the iterations the trainer finished: how many, and the last one’s model name and serving YAML.

• trainer_outputs_from_log: that model name and YAML, for a whole log.

• finished_iterations: how many iterations a log file says finished, which is what rehydration has to go on for a job LSF no longer knows.

Attributes

logger

Classes

Iterations	What a log's lines say of the iterations the trainer finished.
LogTailer	The lines appended to a log since the last read, whole lines only.

Functions

trainer_outputs_from_log(log_text)	(model name, serving YAML path) of the last iteration the log reports.
finished_iterations(log_file)	(how many iterations the log says finished, the last one's model name).

Module Contents

cellmap_flow.finetune.job_manager.tailer.logger

class cellmap_flow.finetune.job_manager.tailer.Iterations

What a log’s lines say of the iterations the trainer finished.

feed takes the lines in the order they were printed, in as many calls as they come in. Then count is how many TRAINING_ITERATION_COMPLETE markers they held, name the last one’s model name, and yaml_path its serving YAML: the trainer prints FINETUNED_MODEL_YAML just before the marker, and skips it when it could not write one, so a YAML is only the iteration’s if it came after the iteration before. Both are None before the first.

count = 0

name = None

yaml_path = None

feed(lines) → None

class cellmap_flow.finetune.job_manager.tailer.LogTailer(path)

The lines appended to a log since the last read, whole lines only.

A read can end mid-line (“Epoch 7/10 - Lo”, “TRAINING_ITERATION_COM”); parsed as it stood, the epoch’s loss or the marker would be lost or cut short. The incomplete end is left in the file, and read again once the rest of the line is written. position is the byte offset after the lines handed out.

The log is everything the run printed, the user’s code and its libraries included, so it need not be UTF-8: a byte that is not is decoded as U+FFFD. A whole line never ends inside a character, since no byte of a multi-byte UTF-8 character is a newline.

iterations (Iterations) holds what every line handed out so far says of the finished iterations, so what depends on the whole log never needs it read again.

tee writes one log for the job’s whole life, restarts included, so it only grows. If it shrinks, something replaced it, and reading starts over, iterations with it.

path

read(finished=False) → str

The complete lines written since the last call, or “” if none.

finished: nothing writes to the log any more (its job has ended), so a last line without its newline is complete, and is handed out too.

Raises OSError when the log cannot be read; nothing is consumed then, so the next call picks up from the same place.

cellmap_flow.finetune.job_manager.tailer.trainer_outputs_from_log(log_text: str)

(model name, serving YAML path) of the last iteration the log reports.

Either is None when the log has none; see Iterations.

cellmap_flow.finetune.job_manager.tailer.finished_iterations(log_file)

(how many iterations the log says finished, the last one’s model name).

Read a line at a time, since the log is everything the run printed. A log that is missing or cannot be read is no evidence: (0, None).