cellmap_flow.finetune.job_manager.tailer
========================================

.. py:module:: cellmap_flow.finetune.job_manager.tailer

.. autoapi-nested-parse::

   The trainer's log, as the job manager reads it.

   The trainer tees everything it prints into its run's ``training_log.txt``,
   and the lines the manager reads are the markers in ``finetune.markers``. A
   job outlives a dashboard upgrade, so both sides of those lines are a
   protocol.

   - ``LogTailer``: the lines appended since the last read, whole lines only,
     and what all the lines read so far say of the finished iterations; the
     monitor follows a running job's log with it, reading each line once.
   - ``Iterations``: what a log's lines, read in order, say of the iterations
     the trainer finished: how many, and the last one's model name and
     serving YAML.
   - ``trainer_outputs_from_log``: that model name and YAML, for a whole log.
   - ``finished_iterations``: how many iterations a log file says finished,
     which is what rehydration has to go on for a job LSF no longer knows.



Attributes
----------

.. autoapisummary::

   cellmap_flow.finetune.job_manager.tailer.logger


Classes
-------

.. autoapisummary::

   cellmap_flow.finetune.job_manager.tailer.Iterations
   cellmap_flow.finetune.job_manager.tailer.LogTailer


Functions
---------

.. autoapisummary::

   cellmap_flow.finetune.job_manager.tailer.trainer_outputs_from_log
   cellmap_flow.finetune.job_manager.tailer.finished_iterations


Module Contents
---------------

.. py:data:: logger

.. py:class:: Iterations

   What a log's lines say of the iterations the trainer finished.

   ``feed`` takes the lines in the order they were printed, in as many
   calls as they come in. Then ``count`` is how many
   TRAINING_ITERATION_COMPLETE markers they held, ``name`` the last one's
   model name, and ``yaml_path`` its serving YAML: the trainer prints
   FINETUNED_MODEL_YAML just before the marker, and skips it when it could
   not write one, so a YAML is only the iteration's if it came after the
   iteration before. Both are None before the first.


   .. py:attribute:: count
      :value: 0



   .. py:attribute:: name
      :value: None



   .. py:attribute:: yaml_path
      :value: None



   .. py:method:: feed(lines) -> None


.. py:class:: LogTailer(path)

   The lines appended to a log since the last read, whole lines only.

   A read can end mid-line ("Epoch 7/10 - Lo", "TRAINING_ITERATION_COM");
   parsed as it stood, the epoch's loss or the marker would be lost or cut
   short. The incomplete end is left in the file, and read again once the
   rest of the line is written. ``position`` is the byte offset after the
   lines handed out.

   The log is everything the run printed, the user's code and its libraries
   included, so it need not be UTF-8: a byte that is not is decoded as
   U+FFFD. A whole line never ends inside a character, since no byte of a
   multi-byte UTF-8 character is a newline.

   ``iterations`` (Iterations) holds what every line handed out so far says
   of the finished iterations, so what depends on the whole log never needs
   it read again.

   tee writes one log for the job's whole life, restarts included, so it only
   grows. If it shrinks, something replaced it, and reading starts over,
   ``iterations`` with it.


   .. py:attribute:: path


   .. py:method:: read(finished=False) -> str

      The complete lines written since the last call, or "" if none.

      ``finished``: nothing writes to the log any more (its job has
      ended), so a last line without its newline is complete, and is
      handed out too.

      Raises OSError when the log cannot be read; nothing is consumed then,
      so the next call picks up from the same place.



.. py:function:: trainer_outputs_from_log(log_text: str)

   (model name, serving YAML path) of the last iteration the log reports.

   Either is None when the log has none; see Iterations.


.. py:function:: finished_iterations(log_file)

   (how many iterations the log says finished, the last one's model name).

   Read a line at a time, since the log is everything the run printed. A log
   that is missing or cannot be read is no evidence: (0, None).