cellmap_flow.jobs.launch

Starting the inference servers the CLIs and the dashboard ask for.

The policy over jobs.lsf and jobs.local: start_hosts runs a server on LSF or on this machine, tries the other GPU queues when one does not start it, waits for its address and records the job in started_jobs(); install_cleanup_handlers kills the recorded jobs, and any still starting, on Ctrl+C, SIGTERM or SIGHUP.

Two deployment settings are read here, once, at import:

SERVER_COMMAND, how a server is started on a compute node, from CELLMAP_FLOW_SERVER_COMMAND. Callers read it from this module when they build a command (serving.launch), never a copy of it.
SERVER_LOG_DIR, where job logs go; tests and deployments point it elsewhere, and jobs.spec.default_log_dir reads it at submit time.

The job settings a start falls back on (the run limit, whether to try other queues) are jobs.settings.launcher_settings(), read when a job is started.

Attributes

`logger`
`SERVER_COMMAND`
`SERVER_LOG_DIR`
`PENDING_FALLBACK_SECONDS`
`STARTUP_TIMEOUT_SECONDS`
`FAILED_STARTS_KEPT`

Functions

`starting_jobs`(→ list)	The jobs submitted but not yet serving (waiting in their queue, or for
`failed_starts`(→ list)	The last failed start of each model name not started again since,
`started_jobs`(→ list)	The jobs this process started: the live list, which start_hosts
`cleanup_handler`(→ None)	Signal handler for graceful shutdown.
`install_cleanup_handlers`(→ bool)	Kill the tracked jobs on Ctrl+C, SIGTERM or SIGHUP. Returns whether installed.
`submit_bsub_job`(→ cellmap_flow.jobs.lsf.LSFJob)	Submit a shell command to LSF using bsub; see cellmap_flow.jobs.lsf.submit.
`start_hosts`(→ cellmap_flow.jobs.spec.Job)	Start a server job either via bsub or locally.

Module Contents

cellmap_flow.jobs.launch.logger

cellmap_flow.jobs.launch.SERVER_COMMAND

cellmap_flow.jobs.launch.SERVER_LOG_DIR

cellmap_flow.jobs.launch.PENDING_FALLBACK_SECONDS = 180

cellmap_flow.jobs.launch.STARTUP_TIMEOUT_SECONDS = 300

cellmap_flow.jobs.launch.FAILED_STARTS_KEPT = 5

cellmap_flow.jobs.launch.starting_jobs() → list: The jobs submitted but not yet serving (waiting in their queue, or for their server to come up), for the dashboard to show as starting.

cellmap_flow.jobs.launch.failed_starts() → list: The last failed start of each model name not started again since, for the dashboard to keep showing with its log.

cellmap_flow.jobs.launch.started_jobs() → list: The jobs this process started: the live list, which start_hosts appends to, cleanup_handler kills from, and the dashboard lists (its Session.jobs replaces the contents rather than the list).

cellmap_flow.jobs.launch.cleanup_handler(signum: int, frame) → None: Signal handler for graceful shutdown. Kills all tracked jobs, then exits with the conventional 128 + signal status, so a stopped run does not report success.

cellmap_flow.jobs.launch.install_cleanup_handlers() → bool

Kill the tracked jobs on Ctrl+C, SIGTERM or SIGHUP. Returns whether installed.

SIGHUP is what a dashboard gets when its terminal goes: the window is closed, the connection to it drops, or the interactive LSF session it runs in ends. Unhandled, it ended the process and left every server it had started running.

For entry points that launch jobs, called from their main thread. This used to happen when the module was imported, which set the handlers for every importer, and raised ValueError when the first import happened off the main thread (the lazy Session.finetune_job_manager, a dashboard request).

cellmap_flow.jobs.launch.submit_bsub_job(command: str, queue: str = _SITE.default_queue, charge_group: str | None = None, job_name: str = 'my_job', num_gpus: int = 1, num_cpus: int = 4, walltime: str | None = None, log_dir: pathlib.Path | None = None, bsub_timeout: float | None = lsf.BSUB_TIMEOUT_SECONDS, env: dict | None = None) → cellmap_flow.jobs.lsf.LSFJob

Submit a shell command to LSF using bsub; see cellmap_flow.jobs.lsf.submit.

Parameters:

command – Shell command to execute (run with bash -c)
queue – LSF queue name
charge_group – Project/chargeback group for billing
job_name – Name for the job
num_gpus – Number of GPUs to request
num_cpus – Number of CPUs to request
walltime – LSF run limit (“HH:MM” or minutes); None leaves the queue default
log_dir – Directory for the job’s <name>_<jobid>.log (stdout and stderr); defaults to SERVER_LOG_DIR
bsub_timeout – Seconds to wait for bsub to answer, or None to wait as long as it takes (an over-ratio request is held for minutes before bsub returns)
env – Variables to set in the job’s environment, on top of this process’s (which LSF copies into the job anyway)

Returns:

LSFJob object for the submitted job

Raises:

subprocess.CalledProcessError – If job submission fails
BsubTimeoutError – bsub did not answer and no new job with this name can be found. The caller must not simply submit again.

cellmap_flow.jobs.launch.start_hosts(command: str, queue: str = _SITE.default_queue, charge_group: str | None = None, job_name: str = 'example_job', use_https: bool = False, wait_for_host: bool = True, walltime: str | None = None, cycle_queues: bool | None = None, local: bool = False) → cellmap_flow.jobs.spec.Job

Start a server job either via bsub or locally.

It runs on this machine only when bsub is not installed, or when local asks for it. When bsub is there and every submission fails, this raises: the machine running the CLI or dashboard is often a login or submit node, which is no place for a GPU server.

Parameters:

command – Command to execute
queue – LSF queue name (for bsub)
charge_group – Project for billing (for bsub)
job_name – Name for the job
use_https – Whether to use HTTPS (adds cert/key flags)
wait_for_host – Whether to wait for host information before returning
walltime – LSF run limit (“HH:MM” or minutes); defaults to the launcher settings’ walltime, then the site’s
cycle_queues – Try other GPU queues when the requested one is busy or closed. Defaults to the launcher settings’ cycle_gpu_queues, which defaults to True.
local – Run on this machine even if bsub is available.

Returns:

Job object (LSFJob or LocalJob) with job information. job.queue is the queue it landed on. The settings are left alone: the queue the job fell back to is not what the next submission should ask for.

Raises:

JobStartError – bsub is available but no queue accepted the job.