cellmap_flow.serving.probe ========================== .. py:module:: cellmap_flow.serving.probe .. autoapi-nested-parse:: Infer what postprocessing a model's raw output needs, from the output itself. A model's output activation is observable: feed it extreme inputs and see whether the result stays bounded. That tells us what activation is *already* baked into the weights, which is enough to catch the common misconfigurations (a SigmoidPostprocessor stacked on a model that already ends in a sigmoid, or a tanh-range model served without the rescale it needs). Deliberately does NOT try to decide what unbounded output *means*. Raw logits and signed-distance predictions are both unbounded and want opposite treatment, and nothing in the tensor distinguishes them -- see ``UNBOUNDED`` below. Attributes ---------- .. autoapisummary:: cellmap_flow.serving.probe.logger cellmap_flow.serving.probe.UNIT cellmap_flow.serving.probe.SIGNED_UNIT cellmap_flow.serving.probe.UNBOUNDED cellmap_flow.serving.probe.NEIGHBORHOOD_OFFSETS cellmap_flow.serving.probe.DACAPO_FRAMEWORKS cellmap_flow.serving.probe.CELLMAP_HF_ORG cellmap_flow.serving.probe.CELLMAP_MODEL_DIR cellmap_flow.serving.probe.SHIFT_TO_SIGNED Functions --------- .. autoapisummary:: cellmap_flow.serving.probe.classify_output_range cellmap_flow.serving.probe.looks_like_affinities cellmap_flow.serving.probe.suggest_affinity_chain cellmap_flow.serving.probe.suggest_postprocess_params cellmap_flow.serving.probe.output_display_range cellmap_flow.serving.probe.review_postprocess cellmap_flow.serving.probe.suggest_input_norm Module Contents --------------- .. py:data:: logger .. py:data:: UNIT :value: 'unit' .. py:data:: SIGNED_UNIT :value: 'signed_unit' .. py:data:: UNBOUNDED :value: 'unbounded' .. py:function:: classify_output_range(lo: float, hi: float) -> str Bucket an observed (min, max) output range into an activation class. .. py:function:: looks_like_affinities(out_channels=None, model_name='', channels_names=None) -> bool Heuristic: does this model emit affinities rather than a single map? Affinity models have one output channel per neighborhood offset (3 for nearest-neighbour, 9 with longer range), so a channel count of 3+ is the structural tell. Require a naming hint as well, since a multi-class semantic model also has several channels and wants entirely different handling. .. py:function:: suggest_affinity_chain(output_class: str) -> list The postprocessing an affinity model needs, in order. AffinityPostprocessor takes affinities in [0, 1], or the uint8 0-255 that DefaultPostprocessor produces (integer input is divided by 255). The chain suggested here goes through DefaultPostprocessor, the long-standing convention; a sigmoid feeding AffinityPostprocessor directly works too. .. py:data:: NEIGHBORHOOD_OFFSETS :value: [[1, 0, 0], [0, 1, 0], [0, 0, 1], [3, 0, 0], [0, 3, 0], [0, 0, 3], [9, 0, 0], [0, 9, 0], [0, 0, 9]] .. py:function:: suggest_postprocess_params(chain, output_class, out_channels=None) -> dict Parameter values for a suggested chain, from the range flowing into each step. The default parameters assume each step sees the range its most common predecessor produces, which stops being true as soon as the chain differs. The costly case is DefaultPostprocessor after a sigmoid: its defaults clip to [-1, 1] and map that to 0-255, so probabilities in [0, 1] land in [127.5, 255] -- half the range. AffinityPostprocessor then divides by 255 and subtracts its bias, and a probability of 0.0003 (a strongly repulsive edge) comes out at -0.002 instead of -0.5. The mutex watershed is left with essentially no repulsive edges and merges everything. Returns ``{postprocessor_name: {param: value}}``, containing only the steps whose defaults are wrong for this chain. .. py:function:: output_display_range(postprocess_steps, output_class) The value range a configured chain produces, for the viewer's shader. The raw layer has to sample percentiles because its distribution is unknown. A postprocessed output does not: the last step's arithmetic fixes the range exactly, so this is both cheaper and more accurate than sampling would be -- and sampling would mean running inference purely to choose a display range. ``postprocess_steps`` is a list of ``to_dict()`` dicts, so the real configured parameters are used rather than assumed defaults. Returns (lo, hi), or None when the chain leaves the range undetermined -- an unbounded model output with nothing applied, where logits could be any scale. Callers should leave neuroglancer's own range controls to it there. .. py:function:: review_postprocess(output_class: str, postprocess_names, out_channels=None, model_name='', channels_names=None) -> dict Compare an observed output class against the configured postprocessors. Returns ``{"level", "message", "suggest"}`` where ``level`` is one of "ok" / "suggest" / "warn", and ``suggest`` is a list of postprocessor names to apply (empty when the current configuration is already sensible). ``level == "warn"`` means the current chain is actively wrong, not merely unconventional. .. py:data:: DACAPO_FRAMEWORKS :value: ('dacapo',) .. py:data:: CELLMAP_HF_ORG :value: 'cellmap/' .. py:data:: CELLMAP_MODEL_DIR :value: '/models/cellmap/' .. py:data:: SHIFT_TO_SIGNED .. py:function:: suggest_input_norm(framework=None, raw_dtype='uint8', declared=None, source=None, data_range=None) -> dict Propose an input_norm config for a model. Priority: 1. ``declared`` -- an explicit input_norm in the model's own metadata. Nothing writes this today, but honour it when it appears. 2. The training framework's convention. DaCapo trains on [-1, 1], hence the extra ``x*2-1`` on top of the 0-1 rescale. 3. ``source`` -- where the model came from. A CellMap model follows the same [-1, 1] convention whatever its metadata calls the framework. 4. Nothing known about the model: assume [-1, 1] anyway, because it is what everything else in this repo trains on, and mark it low confidence so callers can present it as the assumption it is. Getting the scale wrong is not subtle -- the model sees inputs far outside its training range and predicts noise -- so a stated assumption beats silently feeding it raw uint8. ``data_range`` is the actual (min, max) of the raw data. Prefer it over ``raw_dtype``: uint8 data spans 0-255, but float data can span anything and its dtype says nothing useful. Returns ``{"input_norm": {...}, "order": [...], "reason", "confidence"}``. ``order`` is redundant with the dict's insertion order but not with what survives transport: Flask's jsonify sorts keys, which would silently turn MinMax-then-Lambda into Lambda-then-MinMax -- i.e. apply ``x*2-1`` to raw uint8 before the rescale. Callers must use ``order``.