cellmap_flow.utils.job_extensions

Example extensions of the Job class for cloud providers.

This demonstrates how to extend the abstract Job class to support different execution environments like AWS Batch, Google Cloud, Azure Batch, etc.

Attributes

logger

Classes

`AWSBatchJob`	Job submitted to AWS Batch.
`GoogleCloudJob`	Job submitted to Google Cloud (Compute Engine or Batch).
`AzureBatchJob`	Job submitted to Azure Batch.
`SlurmJob`	Job submitted to SLURM cluster (common in HPC environments).

Functions

create_job_for_environment(...)

Factory function to create appropriate Job subclass based on environment.

Module Contents

cellmap_flow.utils.job_extensions.logger

class cellmap_flow.utils.job_extensions.AWSBatchJob(job_id: str, job_queue: str, model_name: str | None = None)

Job submitted to AWS Batch.

job_id

job_queue

kill() → None: Terminate the AWS Batch job.

get_status() → cellmap_flow.utils.bsub_utils.JobStatus: Query AWS Batch for job status.

wait_for_host(timeout: int = 300) → str | None

Monitor AWS Batch job logs for host information.

This would typically involve: 1. Getting the CloudWatch log stream for the job 2. Monitoring the logs for host information 3. Extracting the host URL

class cellmap_flow.utils.job_extensions.GoogleCloudJob(job_id: str, project_id: str, region: str, model_name: str | None = None)

Job submitted to Google Cloud (Compute Engine or Batch).

job_id

project_id

region

kill() → None: Terminate the Google Cloud job.

get_status() → cellmap_flow.utils.bsub_utils.JobStatus: Query Google Cloud for job status.

wait_for_host(timeout: int = 300) → str | None: Monitor Google Cloud job logs for host information.

class cellmap_flow.utils.job_extensions.AzureBatchJob(job_id: str, batch_account: str, pool_id: str, model_name: str | None = None)

Job submitted to Azure Batch.

job_id

batch_account

pool_id

kill() → None: Terminate the Azure Batch job.

get_status() → cellmap_flow.utils.bsub_utils.JobStatus: Query Azure Batch for job status.

wait_for_host(timeout: int = 300) → str | None: Monitor Azure Batch job for host information.

class cellmap_flow.utils.job_extensions.SlurmJob(job_id: str, model_name: str | None = None)

Job submitted to SLURM cluster (common in HPC environments).

job_id

kill() → None: Cancel the SLURM job using scancel.

get_status() → cellmap_flow.utils.bsub_utils.JobStatus: Query SLURM for job status using squeue/sacct.

wait_for_host(timeout: int = 300) → str | None: Monitor SLURM job output file for host information.

cellmap_flow.utils.job_extensions.create_job_for_environment(env_type: str, **kwargs) → cellmap_flow.utils.bsub_utils.Job

Factory function to create appropriate Job subclass based on environment.

Parameters:

env_type – Type of environment (‘lsf’, ‘aws’, ‘gcp’, ‘azure’, ‘slurm’, ‘local’)
**kwargs – Environment-specific parameters

Returns:

Appropriate Job subclass instance

Example

>>> job = create_job_for_environment('aws', job_id='12345', job_queue='my-queue')
>>> job = create_job_for_environment('lsf', job_id='67890')