cellmap_flow.blockwise.blockwise_processor

Attributes

logger

Classes

CellMapFlowBlockwiseProcessor

Functions

`precheck`(→ dict)	Check a blockwise task YAML without side effects.
`progress_marker`(→ pathlib.Path)	The file that records `block` as done: <tmp_dir>/<task>/<write ROI>.
`check_block`(→ bool)
`spawn_worker`(name, yaml_config, charge_group, queue[, ...])	A daisy spawn function that submits one blockwise worker to LSF.

Module Contents

cellmap_flow.blockwise.blockwise_processor.logger

cellmap_flow.blockwise.blockwise_processor.precheck(yaml_config: str) → dict

Check a blockwise task YAML without side effects.

Loads no model, writes nothing, and leaves g alone – unlike constructing CellMapFlowBlockwiseProcessor, which the dashboard used to do for this: that created the output arrays, loaded every model’s weights into the dashboard process (onto its GPU, if it had one), and replaced the dashboard’s live normalization and postprocessing with the task’s.

Checks the settings, that every model entry builds a model config, that json_data builds its normalizers and postprocessors, and that a local data_path exists. What needs the model’s geometry (output shapes, the channels) is left to the run itself.

Raises:: ConfigError – with the reason, when the file is not usable.

class cellmap_flow.blockwise.blockwise_processor.CellMapFlowBlockwiseProcessor(yaml_config: str, create=False)

config

yaml_config

input_path

charge_group

queue

output_path

workers

output_channels_is_dict

cpu_workers

walltime

track_progress

models

model_mode_str

model_merger

process_only

cross_channels_mode

model_config

block_shape

input_voxel_size

output_voxel_size

channels

task_name

dtype

inferencers = []

inferencer = None

idi_raw

output_arrays = []

grid_origin

full_output_roi

bounding_boxes

separate_zarrs

process_fn(block)

client()

run() → bool: Process every ROI; True only if every block of every ROI succeeded.

cellmap_flow.blockwise.blockwise_processor.progress_marker(tmp_dir, block: daisy.Block) → pathlib.Path

The file that records block as done: <tmp_dir>/<task>/<write ROI>.

One directory per daisy task, and named by the block’s write ROI rather than its index. The index is only unique within a task – daisy counts it from each task’s own total_roi, and every bounding box is its own task – so with one shared directory, finishing ROI 1 marked ROI 2’s blocks with the same indices as done. An index also names a different region once the block size changes; a write ROI does not.

cellmap_flow.blockwise.blockwise_processor.check_block(tmp_dir, block: daisy.Block) → bool

cellmap_flow.blockwise.blockwise_processor.spawn_worker(name, yaml_config, charge_group, queue, ncpu=12, walltime=None, log_dir=None)

A daisy spawn function that submits one blockwise worker to LSF.

Goes through jobs.launch.submit_bsub_job, so each worker gets a run limit (-W; the GPU queues’ own default is two hours), its own log file named after its job id, and a submission that fails loudly: a bsub error raises here, and daisy stops the run, rather than the master waiting forever for a worker that never started.

Logs go to log_dir, by default ./daisy_logs as before, resolved now so that every worker writes to the same place.