dacapo.blockwise.scheduler

Attributes

logger

Functions

run_blockwise(worker_file, total_roi, read_roi, write_roi)

Run a function in parallel over a large volume.

segment_blockwise(segment_function_file, context, ...)

Run a segmentation function in parallel over a large volume.

Module Contents

dacapo.blockwise.scheduler.logger
dacapo.blockwise.scheduler.run_blockwise(worker_file: str | upath.UPath, total_roi: funlib.geometry.Roi, read_roi: funlib.geometry.Roi, write_roi: funlib.geometry.Roi, num_workers: int = 16, max_retries: int = 1, timeout=None, upstream_tasks=None, *args, **kwargs)

Run a function in parallel over a large volume.

Parameters:

  • worker_file (str or Path) – The path to the file containing the necessary worker functions: spawn_worker and start_worker. Optionally, the file can also contain a check_function and an init_callback_fn.

  • total_roi (Roi) – The ROI to process.

  • read_roi (Roi) – The ROI to read from for a block.

  • write_roi (Roi) – The ROI to write to for a block.

  • num_workers (int) – The number of workers to use.

  • max_retries (int) – The maximum number of times a task will be retried if failed (either due to failed post check or application crashes or network failure)

  • *args – Additional positional arguments to pass to worker_function.

  • **kwargs – Additional keyword arguments to pass to worker_function.

Returns:

Bool.

Examples

>>> run_blockwise(worker_file, total_roi, read_roi, write_roi, num_workers, max_retries, timeout, upstream_tasks)
dacapo.blockwise.scheduler.segment_blockwise(segment_function_file: str | upath.UPath, context: funlib.geometry.Coordinate, total_roi: funlib.geometry.Roi, read_roi: funlib.geometry.Roi, write_roi: funlib.geometry.Roi, num_workers: int = 16, max_retries: int = 2, timeout=None, upstream_tasks=None, keep_tmpdir=False, *args, **kwargs)

Run a segmentation function in parallel over a large volume.

Parameters:

  • segment_function_file (str or Path) – The path to the file containing the necessary worker functions: spawn_worker and start_worker. Optionally, the file can also contain a check_function and an init_callback_fn.

  • context (Coordinate) – The context to add to the read and write ROI.

  • total_roi (Roi) – The ROI to process.

  • read_roi (Roi) – The ROI to read from for a block.

  • write_roi (Roi) – The ROI to write to for a block.

  • num_workers (int) – The number of workers to use.

  • max_retries (int) – The maximum number of times a task will be retried if failed (either due to failed post check or application crashes or network failure)

  • timeout (int) – The maximum time in seconds to wait for a worker to complete a task.

  • upstream_tasks (List) – List of upstream tasks.

  • keep_tmpdir (bool) – Whether to keep the temporary directory.

  • *args – Additional positional arguments to pass to worker_function.

  • **kwargs – Additional keyword arguments to pass to worker_function.

Returns:

Bool.

Examples

>>> segment_blockwise(segment_function_file, context, total_roi, read_roi, write_roi, num_workers, max_retries, timeout, upstream_tasks)