Colors Processing Demo

This tutorial uses a deterministic color-analysis example to show how NIPACT follows a dataset through local processing, candidate selection, cohort fitting, and a downstream summary. Each source entity is one colored circle and one point on a 2D HSV disk, where hue is angle and saturation is radius.

The colored points stand in for records in a scientific workflow. Because the geometry is easy to inspect, you can see directly how NIPACT records inputs, cohorts, selected outputs, derivative changes, and trace evidence.

Tutorial Overview

The demo contains 200 color entities with IDs color_000 through color_199. It shows the same workflow ideas that matter in larger scientific analyses: defined inputs, declared cohorts, per-record processing, cohort-level fitting, selected results, derivative comparisons, and traceable inputs.

On this page you inspect:

• initial setup: the source population, cohort definitions, and local run directory
• workflow inspection: which steps exist, which output is selected, and which earlier steps support it
• step-by-step execution: entity-local transforms, candidate selection, cohort fit/apply, and the final sector summary
• base and derivative workflow comparison
• auditing examples: trace one analysis result, trace one source file, and summarize what changed in the derivative workflow

The figures below use bundled display files. The workflow graph mirrors the same demo declarations used by the CLI, and nipact workflow graph prints the same graph shape as JSON.

Initial Setup

Project Initialization

To run the demo yourself, initialize it through the NIPACT CLI. The project root contains the generated tutorial project. The runtime root holds local run state: prepared source data, run workspaces created by workflow run, generated manifests, published outputs, and runtime/database/registry.db.

The intended lifecycle is:

init -> prepare/register source data -> validate -> run selected workflow steps -> audit outputs

For packaged demos, --demo prepares the example source data during initialization and leaves the project ready for explicit validation. The selected project root and runtime root must be empty and must not contain each other. If --context is omitted during init, NIPACT uses the demo name as the context.

Initialize Demo Project

nipact init \
  --demo colors \
  --project-dir demos/colors/project \
  --runtime-dir demos/colors/runtime

project_root=demos/colors/project
runtime_root=demos/colors/runtime
context=colors
context_index=nipact.contexts.yaml
demo=colors
source_index=sources.yaml
manifest_count=2
source_file_count=1
source_data=data/color_source.json
source_manifest=init
init_entities=200
manifest_hash=287318ee136c4518
source_hash=1a5fba987c2e0ddf
databases=database/registry.db
runtime_dirs=data,database,outputs,manifests/generated
PASS: init

Before inspecting manifests or workflow examples, validate the context. Validation is read-only: it checks the project and runtime structure, configured source index, manifests, step and workflow files, deterministic counts and hashes, source content identity, referenced Python functions, and any current published outputs. Before execution, published_outputs=0 is valid.

Validate Demo Context

nipact validate --context colors

context=colors
project_root=demos/colors/project
runtime_root=demos/colors/runtime
validated_manifests=2
parsed_workflow_files=2
parsed_step_files=8
source_entities=200
published_outputs=0
PASS: validate

Demo Summary

Field	Value
Context	colors
Project root	demos/colors/project
Example runtime root	demos/colors/runtime
Project structure	generated config, sources.yaml, manifests, steps, workflows, and helpers
Runtime structure	generated subdirectories under the runtime root
Runtime database	runtime/database/registry.db
Docs publication snapshot	/artifacts/colors-processing-demo
Entities	200
Variants	base, red-qc-target
Docs artifacts	demo.json, workflow-graphs/overview.nipact.json

Project Directory Structure

Select a file to show contents below.

✓ Project Directory Structure

demos/colors/project/  # generated local project root├─ README.md                    # local setup notes├─                 # project config and context defaults├─                # source binding index├─ manifests/│  ├─       # logical manifest "init", full source population│  └─   # manually authored Pattern B fit cohort├─ steps/│  ├─ │  ├─ │  ├─ │  ├─ │  ├─ │  ├─ │  ├─ │  └─ └─ workflows/   ├─       # base composition declaration   └─   # derivative override declaration

✓ demos/colors/project/steps/color_local_transform.yaml

step_name: color_local_transform
pattern_kind: pattern_a
execution_role: transform
address_scope: entity
callable: nipact.examples.colors_processing_demo.runtime:color_local_transform_file
inputs:
  features:
    artifact: color_features.features
    dependency_role: source_input
params:
  target_theta: 2.0943951023931953  # 2pi / 3
  force: 0.18
  radius_gate: 0.35
outputs:
  local_color:
    extension: .json
    address_scope: entity

Source Index

The generated nipact.yaml points to sources.yaml with sources.index. sources.yaml maps the named color input to a path inside the runtime root:

demos/colors/project/sources.yaml

global:
colors_source: data/color_source.json

The recorded source path is data/color_source.json. Its resolved filesystem location is under the selected runtime root, for example demos/colors/runtime/data/color_source.json. The color_source step declares source_inputs: [colors_source], so NIPACT resolves which prepared source file to pass into the workflow.

Runtime Directory Structure

Runtime Directory Structure

<runtime-root>/        # local run location├─ data/                  # prepared source inputs│  └─ color_source.json├─ database/│  └─ registry.db         # recorded sources, cohorts, outputs, and lineage├─ outputs/               # published workflow outputs├─ runs/                  # created lazily by workflow run└─ manifests/ └─ generated/        # generated cohort definitions

Demo And Manifest Summary

Manifest Summary

Manifest	Role	Entities	Entity range	Manifest hash
init	source and analysis cohort	200	color_000..color_199	287318ee136c4518
demo-40	Pattern B fit cohort	40	color_160..color_199	9db06e41af119408

How to read the demo setup:

• init is the full source population and analysis cohort.
• init is a logical manifest name, not a file extension. Its authored file is demos/colors/project/manifests/init.yaml.
• demo-40 is the fit cohort used by the Pattern B fit step later in the page.
• base and red-qc-target are the two workflow variants compared later.
• demo.json backs the color figures, tables, and numeric summaries.
• workflow-graphs/overview.nipact.json backs the workflow topology graph.
• demos/colors/project is the generated local project root. Mutable local demo state should live under a selected runtime root such as demos/colors/runtime.
• data/, outputs/, and manifests/generated/ are created by nipact init. runs/ is created lazily by workflow run. runtime/database/registry.db records source identity, manifests, and published workflow outputs.
• sources.yaml stores source identity paths as data/.... Execution plans may use workspace-relative paths for Snakemake, but the recorded source identity keeps the data/... path.
• manifest_digest means the full 64-character SHA-256 digest. manifest_hash means the first 16 characters of that digest, which is the compact form shown in CLI output and summaries.
• Manifest identity tracks cohort membership, while source_digest tracks the exact source JSON content and output_digest tracks exact output content. Their 16-character source_hash and output_hash prefixes keep file names readable without reusing manifest_hash for unrelated content.

The demo calculations are implemented by Python functions. The generated project writes minimal YAML manifests and declaration files around those functions, and nipact validate reads those files as the actual project structure. It also checks the registry database and any published outputs recorded by workflow run.

Technical Detail: Python Implementation Map

This section maps the generated YAML files back to the small Python implementation. If you mainly want the analysis flow, skip to Inspect The Source Population; the workflow behavior is explained again in the figures and command outputs below.

Current Python Source Map

src/nipact/examples/colors_processing_demo/├─ model.py│  ├─ build_source_population -> color_source│  ├─ angular_pull            -> color_local_transform│  ├─ candidate_select        -> color_candidate_select│  ├─ fit_cohort              -> color_cohort_fit│  ├─ apply_cohort_fit        -> color_cohort_apply│  ├─ sector_label            -> color_sector_label│  └─ analyze_sectors         -> color_sector_analysis├─ demo_names.py│  ├─ fit_cohort_entity_ids   -> demo-40 manifest members│  └─ analysis_entity_ids     -> init manifest members├─ project_template.py│  ├─ source_payload          -> generated source data│  ├─ build_manifests         -> generated manifest declarations│  ├─ step_files              -> generated step declarations│  └─ workflow_files          -> generated workflow declarations└─ runtime.py ├─ import_color_source_file      -> color_source file adapter ├─ extract_color_features_file   -> color_features file adapter ├─ color_local_transform_file    -> color_local_transform file adapter ├─ color_candidate_select_file   -> color_candidate_select file adapter ├─ color_cohort_fit_file         -> color_cohort_fit file adapter ├─ color_cohort_apply_file       -> color_cohort_apply file adapter ├─ color_sector_label_file       -> color_sector_label file adapter └─ color_sector_analysis_file    -> color_sector_analysis file adapter

The generated project uses the Python implementation above while storing the step and workflow definitions in example-owned config files. project_template.py defines the manifests, step definitions, workflow definitions, and project config that nipact init --demo colors writes. The project directory viewer above shows those generated files one at a time. The workflow files are included here for orientation; the derivative comparison section returns to them when the page discusses workflows as combinations of steps.

How to read the selected project definitions:

• step_name identifies one analysis step and should match the step file stem.
• callable points from a declared step to the Python function that implements the calculation, or to a small demo wrapper around an existing model function. Current validation confirms that this reference imports and resolves to a callable, but it does not execute the callable.
• sources.index in nipact.yaml points to the source index file for the project.
• sources.yaml maps named source inputs to recorded source paths under data/....
• source_inputs appears only on source-import steps. External files enter the workflow through those steps; downstream transforms consume workflow outputs through normal inputs.
• params declares values that are recorded with the output, so parameter changes can be traced.
• inputs and outputs define named results that workflow graph and trace commands can inspect.
• manifest_binding records which cohort a step uses when it does not operate over the full source population.
• Workflow files combine these declared steps into selected outputs and derivative variants.
• The init entry under manifests: is the source manifest convention for project commands and the demo full-population name used by this page.
• paths.runtime is the single user-facing runtime location. In generated config, relative runtime paths resolve from nipact.yaml, not from the current shell directory.
• nipact.contexts.yaml maps the workspace context colors to demos/colors/project, so post-init commands can run from the workspace root with --context colors.
• NIPACT derives data/, outputs/, runs/, and manifests/generated/ beneath the runtime root. runtime/database/registry.db records context, source identity, manifests, hashes, and published output paths.

CLI command examples use the runtime context colors. The page slug and static artifact path stay colors-processing-demo for docs publication continuity.

Inspect The Source Population

The source manifest is the first dataset definition to inspect. It is named init by convention and contains 200 entities laid out as 20 angular bins by 10 radius bins. The layout is stratified and deterministic, so visual changes later can be attributed to workflow steps rather than random sampling.

WIP (tbd): List Manifests

nipact manifest list --context colors

manifests:
init:
  entities: 200
  entity_range: color_000..color_199
  manifest_hash: 287318ee136c4518
demo-40:
  entities: 40
  entity_range: color_160..color_199
  manifest_hash: 9db06e41af119408

WIP (tbd): Inspect Init Manifest

nipact manifest show init --context colors

manifest: init
entities: 200
entity_id_range: color_000..color_199
angular_bins: 20
radius_bins: 10
radius_range: 0.15..1.00
fixed_value: 0.95
display_seed: 20260519
first_entity:
id: color_000
hue_degrees: 9.0
radius: 0.15
last_entity:
id: color_199
hue_degrees: 351.0
radius: 1.00

How to read the WIP manifest summary:

• init is the full source population and the later analysis cohort.
• demo-40 is a sub-group cohort used later for the Pattern B fit.
• manifest_hash is a 16-character shortened version of the manifest_digest used in summaries and readable filenames.

Source Population and HSV Disk

The 200-entity source population is shown as colored circles and as positions on the HSV disk.

Loading colors...

Source Population Summary

Source field	Value
Entity count	200
Entity IDs	color_000 through color_199
Angular bins	20
Radius bins	10
Radius range	0.15 to 1.00
Value channel	Fixed at 0.95
Display-order seed	20260519

Workflow Definition And Inspection

The colors demo runs steps inside a named workflow. The --step flag selects a step output within that workflow; it is not a standalone step runner. Running a selected downstream step asks Snakemake to compute the earlier steps needed for that result.

Runnable: List Workflows

nipact workflow list \
--context colors

base
red-qc-target
PASS: workflow list

Runnable: List Base Workflow Steps

nipact workflow steps \
--context colors \
--workflow base

step=color_local_transform output=local_color
step=color_candidate_select output=selected_color
step=color_cohort_fit output=cohort_fit
step=color_cohort_apply output=cohort_color
step=color_sector_analysis output=sector_counts
PASS: workflow steps

Base Workflow Declaration

The base workflow lists declared steps in dependency order. A step entry with output_name can be selected with workflow run --step.

demos/colors/project/workflows/base.yaml

workflow_name: base
steps:
- step_name: color_source
- step_name: color_features
- step_name: color_local_transform
  output_name: local_color
- step_name: color_candidate_select
  output_name: selected_color
- step_name: color_cohort_fit
  output_name: cohort_fit
- step_name: color_cohort_apply
  output_name: cohort_color
- step_name: color_sector_label
- step_name: color_sector_analysis
  output_name: sector_counts

Runnable: Plan Base Workflow

nipact workflow plan \
--context colors \
--workflow base \
--step color_sector_analysis

workflow=base
step=color_sector_analysis
selected_output=sector_counts
steps=8
step=color_source pattern=pattern_a role=source_import outputs=source_color
step=color_features pattern=pattern_a role=transform outputs=features
step=color_local_transform pattern=pattern_a role=transform outputs=local_color
step=color_candidate_select pattern=pattern_a role=transform outputs=selected_color
step=color_cohort_fit pattern=pattern_b role=b_fit outputs=cohort_fit
step=color_cohort_apply pattern=pattern_b role=b_apply outputs=cohort_color
step=color_sector_label pattern=pattern_a role=transform outputs=sector_label
step=color_sector_analysis pattern=analysis role=analysis outputs=sector_counts
manifest_binding step=color_source role=source_population manifest=init manifest_hash=287318ee136c4518 entities=200
manifest_binding step=color_cohort_fit role=fit_cohort manifest=demo-40 manifest_hash=9db06e41af119408 entities=40
manifest_binding step=color_sector_analysis role=analysis_cohort manifest=init manifest_hash=287318ee136c4518 entities=200
overrides=0
warnings=0
PASS: workflow plan

Inspect The Workflow Graph

The base workflow is short, but it still shows which steps run, which outputs feed later steps, and which cohorts are used. color_features is a normal Pattern A transform in the graph, giving the workflow a named feature output before the local transform.

The command prints the full graph JSON. The output below is abbreviated to show the top-level fields.

Runnable: Inspect Workflow Graph

nipact workflow graph --context colors --workflow base --step color_sector_analysis

{
"workflow_name": "base",
"selected_step_name": "color_sector_analysis",
"selected_output_name": "sector_counts",
"terminal_step_kind": "analysis",
"nodes": ["... 8 compiled nodes ..."],
"artifacts": ["... 8 compiled artifacts ..."],
"edges": ["... 8 dependency edges ..."],
"manifest_bindings": ["... source_population, fit_cohort, analysis_cohort ..."],
"warnings": []
}

Graph-Derived Node View

Step	Kind	Role	Scope
color_source	pattern_a	source_import	entity
color_features	pattern_a	transform	entity
color_local_transform	pattern_a	transform	entity
color_candidate_select	pattern_a	transform	entity
color_cohort_fit	pattern_b	b_fit	cohort
color_cohort_apply	pattern_b	b_apply	entity
color_sector_label	pattern_a	transform	entity
color_sector_analysis	analysis	analysis	cohort

Graph-Derived Dependency View

source pathcolor_source.source_color└─ source_input -> color_features.features └─ source_input -> color_local_transform.local_color    └─ source_input -> color_candidate_select.selected_colorPattern B fan-incolor_candidate_select.selected_color├─ fit_input -> color_cohort_fit.cohort_fit└─ apply_input -> color_cohort_apply.cohort_colorcolor_cohort_fit.cohort_fit└─ collective_fit -> color_cohort_apply.cohort_coloranalysis tailcolor_cohort_apply.cohort_color└─ source_input -> color_sector_label.sector_label └─ analysis_input -> color_sector_analysis.sector_counts

Graph-Derived Manifest View

Step	Role	Manifest
color_source	source_population	init
color_cohort_fit	fit_cohort	demo-40
color_sector_analysis	analysis_cohort	init

How to read the graph:

• Read the graph left to right as a record of which outputs feed later steps.
• Pattern A steps act independently on each entity.
• Pattern B adds a cohort fit: selected_color feeds both the fit and apply sides, and cohort_fit also feeds the apply side.
• The manifest view shows which cohort supports source import, cohort fitting, and final analysis.
• The exact graph fields, such as pattern_kind, execution_role, address_scope, and dependency_role, are NIPACT's machine-readable labels for those relationships.

The graph gives the tutorial enough structure to show Pattern A, a composed Pattern A candidate-selection step, Pattern B fit/apply, a selected analysis step, and a derivative workflow comparison. color_sector_analysis is a cohort-level analysis step, and sector_counts is the selected result for the workflow runs below.

Colors Workflow Topology

Loading graph...

Base Workflow: Step By Step

The command examples below run selected steps from the base workflow. They let you check the workflow one selected target at a time. The output snippets show the CLI shape. Planning counters can vary with runtime state; they tell you whether NIPACT scheduled new jobs or could reuse previously recorded outputs.

Pattern A: Local Transform

The first transform is entity-local. Every point follows the same rule, but the amount of movement differs because the rule depends on the point's current angle and radius.

Runnable: Run Local Transform Step

nipact workflow run \
--context colors \
--workflow base \
--step color_local_transform \
--cores 1

NIPACT workflow run

context=colors
workflow=base
step=color_local_transform
selected_output=local_color
cores=1
dry_run=false
selected_outputs=200
WIP_jobs=<N>
WIP_reused_registered_artifacts=<N>
WIP_hydrated_inputs=<N>
existing_staged_outputs=0
run_workspace=demos/colors/runtime/runs/colors/base/color_local_transform
snakemake_log=demos/colors/runtime/runs/colors/base/color_local_transform/logs/snakemake.log
note=Registered upstream artifacts can be hydrated into the current run when their identity and digest checks pass.

Preparing run workspace...
Starting Snakemake...
Snakemake complete.
Publishing outputs...
Registry updated.

published_outputs=200
registry=updated
elapsed_seconds=1.234
PASS: workflow run

The transform definition is:

Field	Value
Function	model.angular_pull
Python implementation	src/nipact/examples/colors_processing_demo/model.py
Generated project file	demos/colors/project/steps/color_local_transform.yaml
Parameters	target_theta=2.0943951023931953 ($2\pi / 3$), force=0.18, radius_gate=0.35
Output	one entity-local ColorPoint per input entity

Transform logic:

Python

src/nipact/examples/colors_processing_demo/model.py

def angular_pull(
  point: ColorPoint,
  *,
  target_theta: float,
  force: float,
  radius_gate: float,
) -> ColorPoint:
  radius_weight = bounded_weight(point.radius, radius_gate)
  theta_delta = shortest_periodic_angle(wrap_angle(target_theta) - point.theta)
  theta_out = point.theta + clamp01(force) * radius_weight * theta_delta
  return make_point(
      point.entity_id,
      point.index,
      theta=theta_out,
      radius=point.radius,
      value=point.value,
  )

Math

For an input point $p=(\theta, r)$, target angle $\theta^\star$, force $f$, and radius gate $g$, the local transform keeps radius fixed and moves angle by a gated shortest-path rotation:

$$\begin{aligned} \operatorname{clip}_{01}(x) &= \min(1, \max(0, x)) \\ w(r, g) &= \begin{cases} 0, & \operatorname{clip}_{01}(r) \le \operatorname{clip}_{01}(g) \text{ or } \operatorname{clip}_{01}(g) \ge 1 \\ \dfrac{\operatorname{clip}_{01}(r) - \operatorname{clip}_{01}(g)} {1 - \operatorname{clip}_{01}(g)}, & \text{otherwise} \end{cases} \\ \Delta(\theta^\star, \theta) &= \left((\operatorname{wrap}_{2\pi}(\theta^\star) - \theta + \pi) \bmod 2\pi\right) - \pi \\ \theta' &= \theta + \operatorname{clip}_{01}(f) \, w(r, g) \, \Delta(\theta^\star, \theta) \\ r' &= r \end{aligned}$$

For this step, $\theta^\star = 2\pi / 3$, $f = 0.18$, and $g = 0.35$.

How to read the transform:

• The transform is declared once and applied independently to each entity.
• target_theta is the green-sector direction, equal to $2\pi / 3$.
• force controls the angular pull strength.
• radius_gate keeps low-radius entities from being pulled as strongly.
• The output remains entity-local: one input entity produces one transformed entity output.
• workflow run --step executes one selected workflow step with workflow-resolved parameters and inputs. Running a downstream step asks Snakemake to compute upstream dependencies as needed inside that run workspace.

Although the figure data is bundled with the documentation, the workflow order shown here is the order NIPACT records during execution. In the graph, color_local_transform runs after color_features.features.

The vector field below shows the direction and relative magnitude of that declared transform. The black dots show where the transformed entities land.

Pattern A Local Transform Focus

Loading graph...

Pattern A Local Transform

Use the before/after toggle to compare the source colors with the transformed output; the fixed vector field shows the angular pull applied to each point.

Loading colors...

Pattern A: Candidate Selection

color_candidate_select is still Pattern A because it has one input entity and one selected output entity. For each entity, the demo evaluates five possible outputs, scores each candidate against a QC target on the HSV disk, and keeps the closest candidate as the selected output.

This is a simple analogue of a step that tries several local methods and keeps the result that best satisfies a declared QC rule. The workflow graph still shows one step, while the candidate records explain why the final output was selected.

Runnable: Run Candidate Selection Step

nipact workflow run \
--context colors \
--workflow base \
--step color_candidate_select \
--cores 1

NIPACT workflow run

context=colors
workflow=base
step=color_candidate_select
selected_output=selected_color
cores=1
dry_run=false
selected_outputs=200
WIP_jobs=<N>
WIP_reused_registered_artifacts=<N>
WIP_hydrated_inputs=<N>
existing_staged_outputs=0
run_workspace=demos/colors/runtime/runs/colors/base/color_candidate_select
snakemake_log=demos/colors/runtime/runs/colors/base/color_candidate_select/logs/snakemake.log
note=Registered upstream artifacts can be hydrated into the current run when their identity and digest checks pass.

Preparing run workspace...
Starting Snakemake...
Snakemake complete.
Publishing outputs...
Registry updated.

published_outputs=200
registry=updated
elapsed_seconds=1.234
PASS: workflow run

Candidate Attempts

Candidate	Transform	Target theta
keep	identity	unchanged
red_candidate	angular_pull(force=0.35, radius_gate=0.35)	0.000000
yellow_candidate	angular_pull(force=0.35, radius_gate=0.35)	1.047198 (pi / 3)
green_candidate	angular_pull(force=0.35, radius_gate=0.35)	2.094395 (2pi / 3)
blue_candidate	angular_pull(force=0.35, radius_gate=0.35)	4.188790 (4pi / 3)

Selection Rule

Field	Value
QC target theta	2.094395 (2pi / 3)
QC target radius	0.80
Score	euclidean_distance(candidate_xy, qc_target_xy)
Selected output	candidate with the minimum score

Candidate attempts are shown in the figure data. They are not separate workflow graph nodes because the recorded workflow result is still one entity in and one selected entity out.

The WIP inspection example below shows how one entity's candidate evidence could be exposed by the CLI.

WIP (tbd): Inspect Candidate Evidence

nipact inspect step \
--context colors \
--workflow base \
--step color_candidate_select \
--entity-id color_148 \
--detail candidates

entity: color_148
workflow: base
input_after_local_transform:
theta: 2.596808
radius: 0.811111
hex: "#2ef28c"

candidates:
keep:
  score: 0.400623
red_candidate:
  score: 0.115109
yellow_candidate:
  score: 0.095374
green_candidate:
  score: 0.302625
blue_candidate:
  score: 0.699171

selected:
candidate: yellow_candidate
score: 0.095374
theta: 2.212054
hex: "#2ef244"

For color_148, the base workflow selects yellow_candidate, not green_candidate. The green QC target shifts the population distribution, but the nearest candidate for this example entity is yellow.

How to read the candidate-selection output:

• Candidate attempts explain the choice made inside one workflow-level step.
• The workflow still emits one selected output per input entity.
• The selected-candidate distribution shows the population-level effect of the QC target.
• One example entity can differ from the aggregate direction; that is normal when selection is based on local distance to the QC target.

Composed Pattern A Focus

Loading graph...

Internal Candidate Selection Evidence

For one entity, the figure shows five candidate attempts, their QC scores, and the selected output.

Loading candidate evidence...

After Candidate Selection

Use the before/after toggle to compare the local-transform inputs with the selected candidate outputs for the full population.

Loading colors...

Base Selection Counts

Selected candidate	Count
keep	60
red_candidate	14
yellow_candidate	34
green_candidate	74
blue_candidate	18

Pattern B: Cohort Fit And Entity-Local Apply

Pattern B introduces a cohort-level step. The fit step summarizes a defined cohort, then the apply step moves each entity output toward the fitted cohort centroid.

The fit-step command produces the cohort-scoped centroid.

Runnable: Run Cohort Fit Step

nipact workflow run \
--context colors \
--workflow base \
--step color_cohort_fit \
--cores 1

NIPACT workflow run

context=colors
workflow=base
step=color_cohort_fit
selected_output=cohort_fit
cores=1
dry_run=false
selected_outputs=1
WIP_jobs=<N>
WIP_reused_registered_artifacts=<N>
WIP_hydrated_inputs=<N>
existing_staged_outputs=0
run_workspace=demos/colors/runtime/runs/colors/base/color_cohort_fit
snakemake_log=demos/colors/runtime/runs/colors/base/color_cohort_fit/logs/snakemake.log
note=Registered upstream artifacts can be hydrated into the current run when their identity and digest checks pass.

Preparing run workspace...
Starting Snakemake...
Snakemake complete.
Publishing outputs...
Registry updated.

published_outputs=1
registry=updated
elapsed_seconds=1.234
PASS: workflow run

The apply-step command consumes the selected entity colors and the cohort fit.

Runnable: Run Cohort Apply Step

nipact workflow run \
--context colors \
--workflow base \
--step color_cohort_apply \
--cores 1

NIPACT workflow run

context=colors
workflow=base
step=color_cohort_apply
selected_output=cohort_color
cores=1
dry_run=false
selected_outputs=200
WIP_jobs=<N>
WIP_reused_registered_artifacts=<N>
WIP_hydrated_inputs=<N>
existing_staged_outputs=0
run_workspace=demos/colors/runtime/runs/colors/base/color_cohort_apply
snakemake_log=demos/colors/runtime/runs/colors/base/color_cohort_apply/logs/snakemake.log
note=Registered upstream artifacts can be hydrated into the current run when their identity and digest checks pass.

Preparing run workspace...
Starting Snakemake...
Snakemake complete.
Publishing outputs...
Registry updated.

published_outputs=200
registry=updated
elapsed_seconds=1.234
PASS: workflow run

demo-40 selects entity IDs from the outer two radial bins across all angular bins. The cohort fit is computed after candidate selection, so the selected entity colors feed both color_cohort_fit and color_cohort_apply; the apply step also consumes the resulting cohort fit.

Pattern B Dependency View

color_candidate_select.selected_color├─ fit_input -> color_cohort_fit.cohort_fit└─ apply_input -> color_cohort_apply.cohort_colorcolor_cohort_fit.cohort_fit└─ collective_fit -> color_cohort_apply.cohort_color

Pattern B Summary

Field	Value
Fit manifest	demo-40
Fit manifest role	fit_cohort
Fit entities	40
Fit entity range	color_160..color_199
Fit manifest hash	9db06e41af119408
Fit output	color_cohort_fit.cohort_fit
Apply output	color_cohort_apply.cohort_color
Apply strength	0.18

How to read the Pattern B output:

• The fit is cohort-scoped, not entity-local.
• The apply step returns entity-local outputs, but those outputs depend on the cohort fit.
• The fit cohort is part of the recorded meaning of the result: changing the cohort changes cohort_fit and every apply output that uses it.
• The workflow commands run this fit/apply pair as part of the selected workflow step.

Pattern B Focus

Loading graph...

Pattern B Fit and Apply

Use the view buttons to inspect the candidate-selected input population, the 40-entity fit cohort, and the post-apply outputs.

Loading colors...

Pattern B Node Summary

Step	Role	Addressing point
color_cohort_fit	b_fit	cohort-scoped fit over demo-40
color_cohort_apply	b_apply	entity outputs consuming selected colors and the cohort fit

Sector Analysis Output

The selected analysis output is descriptive: it counts how many final points land in three fixed annular arc regions.

Runnable: Run Sector Analysis Step

nipact workflow run \
--context colors \
--workflow base \
--step color_sector_analysis \
--cores 1

NIPACT workflow run

context=colors
workflow=base
step=color_sector_analysis
selected_output=sector_counts
cores=1
dry_run=false
selected_outputs=1
WIP_jobs=<N>
WIP_reused_registered_artifacts=<N>
WIP_hydrated_inputs=<N>
existing_staged_outputs=0
run_workspace=demos/colors/runtime/runs/colors/base/color_sector_analysis
snakemake_log=demos/colors/runtime/runs/colors/base/color_sector_analysis/logs/snakemake.log
note=Registered upstream artifacts can be hydrated into the current run when their identity and digest checks pass.

Preparing run workspace...
Starting Snakemake...
Snakemake complete.
Publishing outputs...
Registry updated.

published_outputs=1
registry=updated
elapsed_seconds=1.234
PASS: workflow run

Sector Definition

Field	Value
Analysis cohort	init
Red arc center	0.000000
Green arc center	2.094395 (2pi / 3)
Blue arc center	4.188790 (4pi / 3)
Arc half-width	0.523599 (pi / 6)
Minimum radius	0.35

The black sector fences show the counted regions. The population grid uses colored square borders for entities counted in the red, green, and blue arcs. This is a descriptive count, not a p-value or formal hypothesis test.

Sector Analysis Focus

Loading graph...

Final Analysis Surface

The base workflow final state shown with sector-labeled entity colors and sector boundaries on the HSV disk.

Loading colors...

Base Analysis Result

Count	Value
Red arc	8
Green arc	56
Blue arc	11
Outside named arcs	125
Red minus green	-48

How to read the sector analysis output:

• The sector definition determines what is counted as red, green, blue, or outside named arcs.
• init is the analysis cohort, so the sector counts cover all 200 source entities.
• red_minus_green is the compact comparison metric used in the derivative comparison.

Running the Full Base Workflow

The final selected base step is color_sector_analysis. If you run that command from a clean runtime without running the earlier selected steps first, NIPACT still asks Snakemake to compute the upstream dependency path needed for sector_counts.

You can check the same request with --dry-run. This asks Snakemake to plan the work without running jobs, publishing outputs, or updating runtime/database/registry.db. NIPACT may still create the run workspace and prepare reusable inputs so Snakemake can inspect the concrete file paths. If you run this from a clean runtime, existing_staged_outputs may be 0 instead of the value shown below.

Runnable: Dry-Run Full Base Workflow

nipact workflow run \
--context colors \
--workflow base \
--step color_sector_analysis \
--cores 1 \
--dry-run

NIPACT workflow run

context=colors
workflow=base
step=color_sector_analysis
selected_output=sector_counts
cores=1
dry_run=true
selected_outputs=1
WIP_jobs=<N>
WIP_reused_registered_artifacts=<N>
WIP_hydrated_inputs=<N>
existing_staged_outputs=1
run_workspace=demos/colors/runtime/runs/colors/base/color_sector_analysis
snakemake_log=demos/colors/runtime/runs/colors/base/color_sector_analysis/logs/snakemake.log
note=Registered upstream artifacts can be hydrated into the current run when their identity and digest checks pass.

Preparing run workspace...
Starting Snakemake...
Snakemake complete.

outputs_published=false
registry=not_updated
elapsed_seconds=0.456
PASS: workflow run

The non-dry-run command is the same final-step command shown in the sector analysis section. It publishes sector_counts and records the current output in runtime/database/registry.db.

Derivative Workflow Example

The derivative workflow extends the base workflow and changes one parameter in candidate selection. Everything else is inherited.

demos/colors/project/workflows/red-qc-target.yaml

workflow_name: red-qc-target
base_workflow: base
step_overrides:
color_candidate_select:
  params:
    qc_target_theta: 0.0

Runnable: Run Derivative Workflow

nipact workflow run \
--context colors \
--workflow red-qc-target \
--step color_sector_analysis \
--cores 1

NIPACT workflow run

context=colors
workflow=red-qc-target
step=color_sector_analysis
selected_output=sector_counts
cores=1
dry_run=false
selected_outputs=1
WIP_jobs=<N>
WIP_reused_registered_artifacts=<N>
WIP_hydrated_inputs=<N>
existing_staged_outputs=0
run_workspace=demos/colors/runtime/runs/colors/red-qc-target/color_sector_analysis
snakemake_log=demos/colors/runtime/runs/colors/red-qc-target/color_sector_analysis/logs/snakemake.log
note=Registered upstream artifacts can be hydrated into the current run when their identity and digest checks pass.

Preparing run workspace...
Starting Snakemake...
Snakemake complete.
Publishing outputs...
Registry updated.

published_outputs=1
registry=updated
elapsed_seconds=1.234
PASS: workflow run

That one parameter change affects selected candidates, the Pattern B cohort fit, downstream labels, and the sector-count result shown in the comparison below.

Workflow Diff

The workflow comparison connects one local parameter change to downstream sector-count changes and upstream steps that can be treated as unchanged. The table and figure below summarize this comparison. The nipact workflow diff command is WIP.

WIP (tbd): Diff Base And Derivative Workflows

nipact workflow diff \
--context colors \
--workflow base \
--other-workflow red-qc-target \
--step color_sector_analysis

changed_parameter:
step: color_candidate_select
field: qc_target_theta
base: 2.094395
red-qc-target: 0.000000

selected_candidate_count_delta:
keep: 0
red_candidate: +75
yellow_candidate: -11
green_candidate: -66
blue_candidate: +2

sector_analysis:
base:
  red: 8
  green: 56
  blue: 11
  other: 125
  red_minus_green: -48
red-qc-target:
  red: 40
  green: 21
  blue: 20
  other: 119
  red_minus_green: 19
delta:
  red: +32
  green: -35
  blue: +9
  other: -6
  red_minus_green: +67

modeled_reusable_upstream_steps:
- color_source
- color_features
- color_local_transform

modeled_changed_or_downstream_steps:
- color_candidate_select
- color_cohort_fit
- color_cohort_apply
- color_sector_label
- color_sector_analysis

Workflow	color_candidate_select.qc_target_theta	Red arc	Green arc	Blue arc	Red minus green
base	2pi / 3	8	56	11	-48
red-qc-target	0	40	21	20	19

Derivative Change Point

Loading graph...

Base vs Red-QC-Target Final Comparison

This figure compares final entity colors, sector counts, and HSV surfaces for the base and derivative workflows.

Loading derivative comparison...

How to read the workflow comparison:

• One parameter change alters candidate selection.
• Candidate selection changes the Pattern B cohort fit.
• The changed fit then affects entity-local apply outputs.
• Sector labels and sector counts change downstream.
• Source import, feature extraction, and local transform are reusable in the comparison.

Auditing Results

Trace The Analysis Result

The selected sector-count result is published by the base workflow run. trace reads the registry and follows the recorded dependencies backward from that selected output.

Runnable: Trace Analysis Result

nipact trace \
--context colors \
--workflow base \
--step color_sector_analysis \
--output sector_counts \
--address init

artifact_id=<id>
origin=workflow_output
is_published=true
workflow=base
step=color_sector_analysis
output=sector_counts
address=init
path=outputs/colors/base/color_sector_analysis/sector_counts/init.<output_hash>.json
content_digest=<digest>
output_hash=<output_hash>
parameter_hash=<parameter_hash>
upstream_artifacts=<count>
dependency_edges=<count>
manifest_bindings=<count>
provenance_status=complete
warnings=0
PASS: trace

How to read the trace output:

• origin=workflow_output confirms that the selected result came from a workflow run.
• is_published=true confirms that this selector resolved the current published workflow output.
• dependency_edges and manifest_bindings summarize the recorded evidence behind the result.
• provenance_status=complete means the registered upstream dependencies were found. If an upstream row is missing, trace reports provenance_status=degraded and prints warning lines.

For the full machine-readable graph, add --json to the same command.

Trace The Source File

The prepared colors source file is registered as a source artifact when it is used by a workflow run. This lets NIPACT connect an output back to the exact input file used to produce it.

Runnable: Trace Source Artifact

nipact trace \
--context colors \
--file-path data/color_source.json

artifact_id=<id>
origin=source
is_published=false
workflow=-
step=-
output=-
address=-
path=data/color_source.json
content_digest=<digest>
output_hash=<output_hash>
parameter_hash=-
upstream_artifacts=0
dependency_edges=0
manifest_bindings=0
provenance_status=complete
warnings=0
PASS: trace

How to read the source trace:

• Used source files are registered in artifacts with origin=source.
• They do not have workflow, step, output, address, or parameter fields unless a later source-binding convention adds that context.
• They can be selected by their recorded path inside the runtime root.
• They are not published workflow outputs, so is_published=false.

Inspect The Local GUI

After the workflow has published outputs, the same records can be inspected in the local read-only GUI. The GUI reads runtime/database/registry.db and shows the current project summary, workflows, manifests, registered outputs, output details, workflow topology, and focused lineage. Source files appear as artifacts with origin=source.

Runnable: Start Local GUI

nipact gui \
--context colors \
--port 8765

http://127.0.0.1:8765/

The GUI is a review tool. It reads the recorded project state so you can inspect what ran and which inputs support a selected result; workflow execution and file editing stay in the CLI and project files.

What Changed In The Derivative

The comparison summary connects the derivative workflow change to its downstream effects. The derivative changes only one setting, but that change is visible in the sector-count comparison. Runtime cache reuse is still scoped to the same workflow name, so the unchanged rows below describe modeled workflow continuity, not cross-workflow cache hits.

Audit point	Value
Source population	init, 200 stable entity IDs
Changed workflow field	color_candidate_select.qc_target_theta
Modeled unchanged upstream steps	color_source, color_features, color_local_transform
Changed or downstream-affected steps	color_candidate_select, color_cohort_fit, color_cohort_apply, color_sector_label, color_sector_analysis
Pattern B cohort	fit_cohort: demo-40
Analysis cohort	analysis_cohort: init
Comparison metric delta	red_minus_green: -48 to 19, delta +67

This introductory tutorial covers the core pattern used by the fMRI and DFC examples: define inputs and cohorts, run selected workflow outputs, compare a derivative workflow, and trace a result back to the files and steps that produced it.