cellmap_flow.jobs.launch ======================== .. py:module:: cellmap_flow.jobs.launch .. autoapi-nested-parse:: Starting the inference servers the CLIs and the dashboard ask for. The policy over ``jobs.lsf`` and ``jobs.local``: ``start_hosts`` runs a server on LSF or on this machine, tries the other GPU queues when one does not start it, waits for its address and records the job in ``started_jobs()``; ``install_cleanup_handlers`` kills the recorded jobs, and any still starting, on Ctrl+C, SIGTERM or SIGHUP. Two deployment settings are read here, once, at import: - ``SERVER_COMMAND``, how a server is started on a compute node, from ``CELLMAP_FLOW_SERVER_COMMAND``. Callers read it from this module when they build a command (serving.launch), never a copy of it. - ``SERVER_LOG_DIR``, where job logs go; tests and deployments point it elsewhere, and ``jobs.spec.default_log_dir`` reads it at submit time. The job settings a start falls back on (the run limit, whether to try other queues) are ``jobs.settings.launcher_settings()``, read when a job is started. Attributes ---------- .. autoapisummary:: cellmap_flow.jobs.launch.logger cellmap_flow.jobs.launch.SERVER_COMMAND cellmap_flow.jobs.launch.SERVER_LOG_DIR cellmap_flow.jobs.launch.PENDING_FALLBACK_SECONDS cellmap_flow.jobs.launch.STARTUP_TIMEOUT_SECONDS cellmap_flow.jobs.launch.FAILED_STARTS_KEPT Functions --------- .. autoapisummary:: cellmap_flow.jobs.launch.starting_jobs cellmap_flow.jobs.launch.failed_starts cellmap_flow.jobs.launch.started_jobs cellmap_flow.jobs.launch.cleanup_handler cellmap_flow.jobs.launch.install_cleanup_handlers cellmap_flow.jobs.launch.submit_bsub_job cellmap_flow.jobs.launch.start_hosts Module Contents --------------- .. py:data:: logger .. py:data:: SERVER_COMMAND .. py:data:: SERVER_LOG_DIR .. py:data:: PENDING_FALLBACK_SECONDS :value: 180 .. py:data:: STARTUP_TIMEOUT_SECONDS :value: 300 .. py:data:: FAILED_STARTS_KEPT :value: 5 .. py:function:: starting_jobs() -> list The jobs submitted but not yet serving (waiting in their queue, or for their server to come up), for the dashboard to show as starting. .. py:function:: failed_starts() -> list The last failed start of each model name not started again since, for the dashboard to keep showing with its log. .. py:function:: started_jobs() -> list The jobs this process started: the live list, which start_hosts appends to, cleanup_handler kills from, and the dashboard lists (its Session.jobs replaces the contents rather than the list). .. py:function:: cleanup_handler(signum: int, frame) -> None Signal handler for graceful shutdown. Kills all tracked jobs, then exits with the conventional 128 + signal status, so a stopped run does not report success. .. py:function:: install_cleanup_handlers() -> bool Kill the tracked jobs on Ctrl+C, SIGTERM or SIGHUP. Returns whether installed. SIGHUP is what a dashboard gets when its terminal goes: the window is closed, the connection to it drops, or the interactive LSF session it runs in ends. Unhandled, it ended the process and left every server it had started running. For entry points that launch jobs, called from their main thread. This used to happen when the module was imported, which set the handlers for every importer, and raised ValueError when the first import happened off the main thread (the lazy Session.finetune_job_manager, a dashboard request). .. py:function:: submit_bsub_job(command: str, queue: str = _SITE.default_queue, charge_group: Optional[str] = None, job_name: str = 'my_job', num_gpus: int = 1, num_cpus: int = 4, walltime: Optional[str] = None, log_dir: Optional[pathlib.Path] = None, bsub_timeout: Optional[float] = lsf.BSUB_TIMEOUT_SECONDS, env: Optional[dict] = None) -> cellmap_flow.jobs.lsf.LSFJob Submit a shell command to LSF using bsub; see cellmap_flow.jobs.lsf.submit. :param command: Shell command to execute (run with ``bash -c``) :param queue: LSF queue name :param charge_group: Project/chargeback group for billing :param job_name: Name for the job :param num_gpus: Number of GPUs to request :param num_cpus: Number of CPUs to request :param walltime: LSF run limit ("HH:MM" or minutes); None leaves the queue default :param log_dir: Directory for the job's _.log (stdout and stderr); defaults to SERVER_LOG_DIR :param bsub_timeout: Seconds to wait for bsub to answer, or None to wait as long as it takes (an over-ratio request is held for minutes before bsub returns) :param env: Variables to set in the job's environment, on top of this process's (which LSF copies into the job anyway) :returns: LSFJob object for the submitted job :raises subprocess.CalledProcessError: If job submission fails :raises BsubTimeoutError: bsub did not answer and no new job with this name can be found. The caller must not simply submit again. .. py:function:: start_hosts(command: str, queue: str = _SITE.default_queue, charge_group: Optional[str] = None, job_name: str = 'example_job', use_https: bool = False, wait_for_host: bool = True, walltime: Optional[str] = None, cycle_queues: Optional[bool] = None, local: bool = False) -> cellmap_flow.jobs.spec.Job Start a server job either via bsub or locally. It runs on this machine only when bsub is not installed, or when ``local`` asks for it. When bsub is there and every submission fails, this raises: the machine running the CLI or dashboard is often a login or submit node, which is no place for a GPU server. :param command: Command to execute :param queue: LSF queue name (for bsub) :param charge_group: Project for billing (for bsub) :param job_name: Name for the job :param use_https: Whether to use HTTPS (adds cert/key flags) :param wait_for_host: Whether to wait for host information before returning :param walltime: LSF run limit ("HH:MM" or minutes); defaults to the launcher settings' walltime, then the site's :param cycle_queues: Try other GPU queues when the requested one is busy or closed. Defaults to the launcher settings' cycle_gpu_queues, which defaults to True. :param local: Run on this machine even if bsub is available. :returns: Job object (LSFJob or LocalJob) with job information. ``job.queue`` is the queue it landed on. The settings are left alone: the queue the job fell back to is not what the next submission should ask for. :raises JobStartError: bsub is available but no queue accepted the job.