Architecture

Topology in, locality out.

numaperf is a workspace of focused crates layered on one runtime topology model. Discover the machine once, then place memory, pin threads, and schedule work against the same source of truth — and measure whether it held.

data flow

            ┌───────────────────────────┐
            │   Topology::discover()    │   numaperf-topo
            │  nodes · cpus · distances │   (runtime, once)
            │  device → node map        │
            └────────────┬──────────────┘
                         │  read by every layer
        ┌────────────────┼─────────────────┬──────────────┐
        ▼                ▼                 ▼              ▼
  MemPolicy        ScopedPin        NumaExecutor    device buffers
  Bind/Pref/       (RAII pin,       per-node pools  on NIC/NVMe-
  Interleave/      restore on       node-scoped     local node
  Local +          drop)            stealing        numaperf-io
  Prefault::Touch  numaperf-        numaperf-sched
  numaperf-mem     affinity         numaperf-sharded
        └────────────────┴─────────────────┴──────────────┘
                         │
                         ▼
              ┌──────────────────────┐
              │  locality ratios,    │   numaperf-perf
              │  cross-node traffic, │   (did it hold?)
              │  health reports      │
              └──────────────────────┘

The layers

numaperf-topo

Topology model

The foundation. Discovers nodes, CPUs, distances, and device→node maps at runtime. Every other layer reads from it, so no component hard-codes a machine layout.

numaperf-mem

Memory placement

Turns a topology plus an explicit MemPolicy (Bind / Preferred / Interleave / Local) into concrete allocations, with Prefault::Touch to force placement at allocation time.

numaperf-affinity

Thread pinning

RAII affinity. ScopedPin constrains where a thread runs and restores the prior affinity on drop, so placement decisions survive refactors and pool transitions.

numaperf-sched · -sharded

Scheduling & data

NumaExecutor runs per-node worker pools with node-scoped stealing; NumaSharded<T> and ShardedCounter keep data per-node so work runs where its data already is.

numaperf-perf

Observability

Closes the loop. Locality ratios, health reports, and cross-node traffic counters turn "is it actually local?" into a measured number.

Why application-layer, not kernel heroics

On multi-socket servers, cross-node memory access latency is typically 2–3× local access (per the project README). The kernel’s default first-touch policy places a page on whichever node first writes it, which is rarely the node you want under load. numaperf’s position is that the fix is not a smarter opaque scheduler but explicit, observable placement at the application layer: you choose the policy, you pin the thread, and you measure the locality ratio.

Pinning vs scheduling

Two concerns hide under "scheduling". Pinning is a constraint on where work runs; scheduling is a policy for which work runs next. General-purpose async runtimes implement scheduling well and ignore pinning entirely. numaperf takes pinning back with ScopedPin, and offers a scheduler (NumaExecutor) that respects placement — while composing with your existing runtime for the control plane. The hybrid pattern: keep the general runtime for control, carve out numaperf workers for the latency-critical hot path, hand off over a bounded channel.

Incremental adoption

Because it is a workspace, you can adopt one crate at a time. Start with numaperf-topo to see the machine, add numaperf-mem and numaperf-affinity for one hot buffer, and reach for numaperf-sched only when you are ready to own scheduling. The glossary defines the terms above.