cellmap_flow.models.envs

Running a model in an environment of its own.

Some models need packages that cannot share cellmap-flow’s environment: Cellpose 4 (Cellpose-SAM) wants a newer torch and numpy than the default environment’s cellpose 3 allows, and transformer models pin their own stacks. A model entry’s env says where such a model’s inference server, and its finetuning job, run instead:

models:
  my_model:
    type: script
    script_path: /path/to/my_model.py
    env: cellpose4

env is one of:

the name of an environment in cellmap-flow’s pixi.toml. The program runs as pixi run --frozen --manifest-path <pixi.toml> -e <name> ..., which installs the environment from the lockfile on first use (a server’s launcher installs it before submitting the job: jobs.launch._install_env_first). The manifest is the one in the checkout cellmap-flow is installed from, or CELLMAP_FLOW_PIXI_MANIFEST; pixi is PIXI_EXE (which pixi run sets), else the pixi on PATH.
an absolute path to a conda environment or virtualenv that has cellmap-flow installed. The program runs as <path>/bin/python -P -m <module>.

A value with a / in it, or starting with ~, is a path; anything else is a pixi environment’s name.

A name that ~/.cellmap_flow/envs.yaml maps to a path is that path: an alias, which wins over a pixi environment of the same name. It is how a machine without pixi (a conda-only cluster account) supplies the environments the model types default to:

cellpose4: /groups/lab/home/me/miniconda3/envs/cellpose4

The file is read each time it is needed, and CELLMAP_FLOW_ENVS_FILE names another.

A model type can name the environment its models run in when the entry gives none, as ModelConfig.default_env: a class attribute, or a property that decides per model. The environment a model runs in (model_env) is its entry’s env, else its type’s default_env, else this one; env: current runs it in this one whatever its type’s default. A default that cannot be used here falls back to this environment with a warning (_usable_default says when and why), where an explicit env that cannot be used is an error.

env is not a constructor argument of any model type, so it is not on the model’s own form or command line: registry.build_model takes it off the entry and sets ModelConfig.env, to_dict() and launch_entry write it back (never the type’s default, so an exported YAML says only what its author chose), and serving.launch takes it off again to put the server in the environment, so the server never sees it.

Only the server and the trainer move. Everything else (the dashboard, blockwise, infer --server-check) runs the model, when it runs it at all, in the process’s own environment.

Attributes

`logger`
`PIXI_MANIFEST_ENV`
`ALIASES_FILE_ENV`
`CURRENT`

Functions

`is_path`(→ bool)	Whether `env` names a directory rather than a pixi environment.
`aliases_file`(→ pathlib.Path)	`CELLMAP_FLOW_ENVS_FILE`, else `~/.cellmap_flow/envs.yaml`.
`aliases`(→ Dict[str, str])	The aliases file's names and their paths, `~` expanded; {} without one.
`pixi_manifest`(→ pathlib.Path)	The pixi.toml whose environments `env` names.
`pixi_environments`(→ Dict[str, dict])	Each environment `manifest` declares, as a table with `features`.
`pixi_program`(→ str)	The pixi to run: `PIXI_EXE`, else the one on PATH, else `"pixi"`.
`has_pixi`(→ bool)	Whether this machine has a pixi to run (`PIXI_EXE` or one on PATH).
`validate`(→ str)	`env` checked, with a path's `~` expanded; what the model keeps.
`prefix`(→ str)	The directory `env` is installed in.
`is_installed`(→ bool)	Whether `env` has a python yet (a pixi one is installed on first use).
`is_running_in`(→ bool)	Whether this process already runs in `env`: the server or trainer a
`effective`(→ Optional[str])	The environment a model runs in, or None for this one.
`model_env`(→ Optional[str])	`effective` of a model config's `env` and `default_env`.
`declared_default`(cls)	`cls.default_env` as the class declares it: a name, None, or the
`decides_per_model`(→ bool)	Whether a `declared_default` is decided per model, not one name.
`type_default`(→ Optional[str])	The default environment of `cls`'s models when it is one for all of
`install_argv`(→ List[str])	`pixi install` of `env` from pixi.lock as it is (`--frozen`), as
`pixi_env_of`(→ Optional[str])	The pixi environment `argv` runs in when it is `server_argv`'s
`server_argv`(→ List[str])	The program that serves a model in `env`, before its `--model` and `-d`.
`python`(→ List[str])	The python of `env`: a directory's `bin/python`, or `pixi run ... python`.
`python_argv`(→ List[str])	`python -P -m <module>` in `env`.
`lib_dir`(→ str)	The lib directory of `env`, for LD_LIBRARY_PATH.
`finetune_problem`(→ Optional[str])	Why the trainer cannot run in `env`, or None when it can (as far as

Module Contents

cellmap_flow.models.envs.logger

cellmap_flow.models.envs.PIXI_MANIFEST_ENV = 'CELLMAP_FLOW_PIXI_MANIFEST'

cellmap_flow.models.envs.ALIASES_FILE_ENV = 'CELLMAP_FLOW_ENVS_FILE'

cellmap_flow.models.envs.CURRENT = 'current'

cellmap_flow.models.envs.is_path(env: str) → bool: Whether env names a directory rather than a pixi environment.

cellmap_flow.models.envs.aliases_file() → pathlib.Path: CELLMAP_FLOW_ENVS_FILE, else ~/.cellmap_flow/envs.yaml.

cellmap_flow.models.envs.aliases() → Dict[str, str]

The aliases file’s names and their paths, ~ expanded; {} without one.

Read on each call, so an alias added while the dashboard runs is used by the next launch.

Raises:: ConfigError – the file is not a mapping of names to paths.

cellmap_flow.models.envs.pixi_manifest() → pathlib.Path

The pixi.toml whose environments env names.

CELLMAP_FLOW_PIXI_MANIFEST, else the one at the root of the checkout this package is installed from (pixi installs it editable from there).

cellmap_flow.models.envs.pixi_environments(manifest: pathlib.Path) → Dict[str, dict]

Each environment manifest declares, as a table with features.

default is always there, as pixi has it even when it is not declared. The list form (name = ["feature", ...]) becomes {"features": [...]}.

cellmap_flow.models.envs.pixi_program() → str: The pixi to run: PIXI_EXE, else the one on PATH, else "pixi".

cellmap_flow.models.envs.has_pixi() → bool: Whether this machine has a pixi to run (PIXI_EXE or one on PATH).

cellmap_flow.models.envs.validate(env, model_name: str = 'model') → str

env checked, with a path’s ~ expanded; what the model keeps.

An alias stays a name, so an exported YAML names the environment the same way on every machine, and each machine’s aliases file says where it is. current is kept as it is.

Raises:: ConfigError – env is not a string, a path (an alias’s included) is relative or has no bin/python, or no pixi.toml declares an environment by that name.

cellmap_flow.models.envs.prefix(env: str) → str

The directory env is installed in.

For a pixi environment that is where pixi installs it by default, .pixi/envs/<name> beside the manifest; a pixi configured with detached environments puts it elsewhere.

cellmap_flow.models.envs.is_installed(env: str) → bool: Whether env has a python yet (a pixi one is installed on first use).

cellmap_flow.models.envs.is_running_in(env: str) → bool: Whether this process already runs in env: the server or trainer a model was moved into, where building the model is what it is there for.

cellmap_flow.models.envs.effective(env, default_env, model_name: str = 'model') → str | None

The environment a model runs in, or None for this one.

env is the entry’s own (validated when the entry was read): itself, unless it is current. Without one, default_env, its type’s, when it can be used here (_usable_default).

cellmap_flow.models.envs.model_env(model_config) → str | None: effective of a model config’s env and default_env.

cellmap_flow.models.envs.declared_default(cls): cls.default_env as the class declares it: a name, None, or the property (or other descriptor) that decides it per model.

cellmap_flow.models.envs.decides_per_model(declared) → bool: Whether a declared_default is decided per model, not one name.

cellmap_flow.models.envs.type_default(cls) → str | None: The default environment of cls’s models when it is one for all of them, without building one; None when it has none or decides per model.

cellmap_flow.models.envs.install_argv(env: str) → List[str]: pixi install of env from pixi.lock as it is (--frozen), as pixi run --frozen would install it, so the lock is never rewritten.

cellmap_flow.models.envs.pixi_env_of(argv: List[str]) → str | None: The pixi environment argv runs in when it is server_argv’s pixi run ... -e <env> ..., else None.

cellmap_flow.models.envs.server_argv(env: str) → List[str]

The program that serves a model in env, before its --model and -d.

In a pixi environment that is the environment’s cellmap_flow serve, as the deployment’s own CELLMAP_FLOW_SERVER_COMMAND is. A directory need not be on PATH, so there it is its python running the CLI; -P keeps the job’s working directory off sys.path, so a job started inside another checkout does not run that checkout’s cellmap_flow.

cellmap_flow.models.envs.python(env: str) → List[str]: The python of env: a directory’s bin/python, or pixi run ... python.

cellmap_flow.models.envs.python_argv(env: str, module: str) → List[str]: python -P -m <module> in env.

cellmap_flow.models.envs.lib_dir(env: str) → str

The lib directory of env, for LD_LIBRARY_PATH.

A pixi environment’s is in its default place (prefix); with detached environments it is elsewhere, and the job then loads libraries as pixi run alone would.

cellmap_flow.models.envs.finetune_problem(env: str) → str | None

Why the trainer cannot run in env, or None when it can (as far as can be told cheaply).

A pixi environment is checked against the manifest: one of its features, the default feature included unless it sets no-default-feature, must install peft. A directory (a path or an alias) is not checked; a trainer that cannot import there fails in its job’s log.