TacVar is a toolset for tackling variability in parallel running time measurement. It includes several tools to measure, quantify and filter unstable timing readings in multi-core processors and accelerators.
Here are core tools in TacVar:
- ParTES: Measuring the minimal 'measurable' running time of your server under different run-time environment.
- Vkern: Obeserving the deviation and error of your performance measurement results caused by timing fluctuation.
- Filter: Filtering a noisy running time distribution for better measurement accuracy.
Related publications:
Q. Liao and J. Lin, "TacVar: Tackling Variability in Short-Interval Timing Measurements on X86 Processors," 2024 IEEE 24th International Symposium on Cluster, Cloud and Internet Computing (CCGrid), Philadelphia, PA, USA, 2024, pp. 496-506, doi: 10.1109/CCGrid59990.2024.00062. (Best paper award)
Q. Liao, S. Zuo, Y. Wang and J. Lin. Timing Method and Evaluation Metrics for CPU Performance Variation Detections [J]. Chinese Journal of Computers，2024，47：456-472, doi: 10.11897/SP.J.1016.2024.00456. (Download Paper)
LIAO Qiucheng, ZHOU Yang, LIN Xinhua. Metrics and Tools for Evaluating the Deviation in Parallel Timing[J]. Computer Science, 2025, 52(5): 41-49. doi: 10.11896/jsjkx.241200053 (Download Paper)
VKern generates run time variations which can be configured by given statistical distributions. By using different timing methods to measure the run time of the specific distribution, one can investigate the instability caused by timing fluctuations in parallel short-interval timing. Furthermore, based on the timing fluctuation distribution sample, FilT can help estimate the real distribution of timing results by decoupling the timing fluctuation from measurement results.
This repository is a preliminary code base for validating the methodology. For broaden the usage of the method, it will soon be integrated into our open-source instrumentation-based profiling tool PerfHound.
$ ./filt.x --help
Usage: filt.x [OPTION...]
-g, --hz-ns=FREQ Tick per ns of the clock
-l, --plow=PROB Lowest threshold of possibility of a data bin
-m, --met-file=FILE Input measurement file
-n, --nsamp=NSAMP Number of samples in each optimization step
-s, --sample-file=FILE Input timing fluctuation file
-w, --width=TIME The least interval of a time bin (ns).
-x, --cut-x=PROB Cut the highest probability of met array
-y, --cut-y=PROB Cut the highest probability of timing fluctuation
array
-?, --help Give this help list
--usage Give a short usage message
Shared measurement support for NPB-MPI, NPB-OMP, and lmbench lives under src/measure/. Backends are selected at build time via each suite's tacvar.conf. Hot-path timer/counter reads use generated macros / static inline code (no runtime backend vtable). Benchmark kernel sources and their CFLAGS are not modified.
flowchart LR
Conf[tacvar.conf] --> Gen[gen_config.py]
Gen --> H[tacvar_generated_config.h]
H --> Lib[libtacvar_measure.a]
Lib --> Timers[Selected timer]
Lib --> Counters[Selected counter]
NPB[NPB timer API] --> Lib
LMB[lmbench start/stop] --> Lib
Lib --> CSV["data_YYYYMMDDTHHmmss/*.csv"]
Timer (TACVAR_TIMER) |
x86_64 | aarch64 | NPB-MPI | NPB-OMP | lmbench |
|---|---|---|---|---|---|
native |
yes | yes | → mpi_wtime |
→ omp_get_wtime |
→ gettimeofday |
gettimeofday |
yes | yes | no | no | yes |
omp_get_wtime |
yes | yes | no | yes | no |
mpi_wtime |
yes | yes | yes | no | no |
clock_gettime |
yes | yes | yes | yes | yes |
papi_get_real_nsec |
yes* | yes* | yes | yes | yes |
rdtsc / rdtscp / rdtscp_lfence / tsc_asym |
yes | no | yes | yes | yes |
cntvct_el0 / cntvct_el0_dmb |
no | yes† | yes | yes | yes |
Counter (TACVAR_COUNTER_BACKEND) |
Notes |
|---|---|
none |
Default; no hardware counters |
perf_event_open |
Userspace events (exclude_kernel=1); needs usable perf_event_paranoid |
papi_read |
Needs PAPI; set PAPI_HOME |
asm |
x86 RDPMC or ARM MRS via ph_enable_pmu kmod |
* Requires PAPI. † Requires non-zero TACVAR_NSTP (ns per CNTVCT tick). Arch is detected from the compiler (-dM -E), not from uname alone.
Header: src/measure/include/tacvar_measure.h.
tacvar_init(const tacvar_context_t *ctx)/tacvar_fini()— directory, CSV, selected backends; safe afterfork(PID reset).tacvar_csv_write_simple(...)— one CSV row per start/stop interval (called outside the timed region).- Hot path (from generated config):
TACVAR_TIMER_BEGIN/END/DELTA_NS,TACVAR_COUNTER_READ,TACVAR_COUNTER_DELTAS.
Naming: exported C symbols tacvar_*; macros/enums TACVAR_*; NPB/lmbench adapters tacvar_npb_* / tacvar_lmbench_*.
Suite paths:
suites/NPB3.4.4/NPB3.4-MPI/tacvar.confsuites/NPB3.4.4/NPB3.4-OMP/tacvar.confsuites/lmbench/tacvar.conf
Fields:
TACVAR_TIMER=native # see matrix above
TACVAR_NSTP=0 # ns/tick for CNTVCT; e.g. 10 for 100 MHz
TACVAR_COUNTER_BACKEND=none
TACVAR_COUNTER_COUNT=0 # must match number of names
TACVAR_COUNTER_NAMES= # comma-separated, e.g. cpu-cycles,instructions
TACVAR_OUTPUT_ROOT=. # CSV root relative to benchmark cwd
TACVAR_ENABLE_PER_STEP_TIMING=0 # NPB-MPI: 0=whole-kernel only; 1=+major-loop steps
TACVAR_ENABLE_PER_STEP_TIMING is independent of NPB’s NPB_TIMER_FLAG / timer.flag. Native NPB detail timers still update NPB’s accumulated report when that flag is set, but they are not written to TacVar event CSVs. TacVar records:
| Switch | Event CSV contents |
|---|---|
0 |
One whole-kernel interval per participating rank (region_id = NPB total slot, or 0 for DT’s all-rank total) |
1 |
That total plus each major outer-loop iteration/batch as region_id=1000 (TACVAR_REGION_STEP) |
Major-loop meanings (switch 1): MG V-cycle+resid; CG inverse-power iteration; SP/BT/LU one timestep; FT evolve+iFFT+checksum; IS one rank() pass; EP one local random batch (reserves rank-local np); DT one owned Source/Comparator/Sink body (idle ranks reserve 0 steps). Enabling switch 1 adds timer/counter overhead inside NPB’s official total; switch 0 is the whole-kernel baseline.
Change conf → rebuild the suite. Runtime does not switch backends. Invalid consumer/arch combinations are rejected by src/measure/tools/gen_config.py.
Dependencies: OpenMPI (NPB-MPI), optional PAPI (papi_get_real_nsec / papi_read), and for asm counters the ph_enable_pmu kmod (see src/kmod/). For perf_event_open as non-root with perf_event_paranoid >= 2, TacVar opens events with exclude_kernel=1.
Put MPI/PAPI on PATH / LD_LIBRARY_PATH (and set PAPI_HOME when needed) before building.
NPB-MPI / NPB-OMP (after config/make.def is present):
# Edit the suite tacvar.conf, then:
cd suites/NPB3.4.4/NPB3.4-OMP && make CG CLASS=S
cd suites/NPB3.4.4/NPB3.4-MPI && make IS CLASS=Slmbench (from suites/lmbench/src):
# export PAPI_HOME=... # if using PAPI backends
../scripts/build ../bin/$(../scripts/os)/lat_syscall
# or build lmbench.a / timing_o / enough with the same relative $O pathsStandalone measure library:
make -C src/measure CONF=suites/lmbench/tacvar.conf CONSUMER=lmbench \
OUTDIR=/tmp/tacvar_build CC=gcc \
PAPI_INC=$PAPI_HOME/include PAPI_LIB=$PAPI_HOME/libEach run creates one data_YYYYMMDDTHHmmss/ under the benchmark cwd (TACVAR_OUTPUT_ROOT). Override with TACVAR_DATA_DIR to reuse an existing directory (e.g. after fork/exec). The directory may be created at tacvar_init() so MPI ranks agree on one path, but event and metadata CSV files are created only during tacvar_fini(). Rows are buffered in memory (NPB-MPI pre-reserves exact capacity before timing; NPB-OMP/lmbench grow geometrically). Abnormal termination therefore loses buffered events. A forked child discards its inherited buffer without flushing the parent’s events.
Writers (rank / process / thread) get separate files:
<data_dir>/<suite>_<benchmark>_<class>_rRRRR_tTTTT_pPID.csv
<data_dir>/region_info.csv # rank/thread 0 only (NPB-MPI)
Example: npb-mpi_is_S_r0000_t0000_p12345.csv.
Base header (always present; comma-separated):
seq,suite,benchmark,class,test_tag,region_id,timer,
raw_start,raw_stop,elapsed_ns,rank,thread,pid,cpu_start,cpu_stop,
migrated,valid,source
| Column | Type | Meaning |
|---|---|---|
seq |
uint | Per-file row counter (1-based) |
suite |
string | npb-mpi, npb-omp, or lmbench |
benchmark |
string | e.g. is, cg, lat_syscall |
class |
string | NPB class letter (S…E); often X / empty for lmbench |
test_tag |
string | Optional sub-test label (may be empty) |
region_id |
int | Timed region (1000 = NPB major-loop step; else NPB total / lmbench) |
timer |
string | Build-time timer name (TACVAR_TIMER) |
raw_start / raw_stop |
uint64 | Backend-native tick / timestamp at begin/end |
elapsed_ns |
int64 | Duration in nanoseconds (TACVAR_TIMER_DELTA_NS); always ≥ 0 |
rank |
int | MPI rank, else 0 |
thread |
int | OpenMP thread id (updated at write), else 0 |
pid |
int | Writer process id |
cpu_start / cpu_stop |
int | sched_getcpu() at begin/end; −1 if unavailable |
migrated |
0/1 | 1 if both CPUs known and cpu_start != cpu_stop |
valid |
0/1 | 0 when migrated (asm counters unsafe across cores); else 1 |
source |
string | Optional caller source tag |
region_info.csv (NPB-MPI, once per run on rank 0) documents every logical NPB region, including legacy detail timers not recorded by TacVar:
suite,benchmark,class,test_tag,region_id,region_name,source_locations,description,active_when,recorded_by_tacvar
source_locations is semicolon-separated file:routine:start-end when one logical region has multiple call sites. Total rows use recorded_by_tacvar=1; detail rows use 0 with active_when=NPB_TIMER_FLAG; step rows use active_when=TACVAR_ENABLE_PER_STEP_TIMING=1.
Counter columns (only when TACVAR_COUNTER_COUNT > 0):
...,valid,source,counter_backend,<name>_start,<name>_stop,<name>_delta[,...]
counter_backend— build-time backend (perf_event_open,papi_read,asm).- For each name in
TACVAR_COUNTER_NAMES, three columns: raw start, raw stop, and modular wrap-safe delta (*_delta).
Example with cpu-cycles,instructions:
seq,suite,benchmark,class,test_tag,region_id,timer,raw_start,raw_stop,elapsed_ns,rank,thread,pid,cpu_start,cpu_stop,migrated,valid,counter_backend,cpu-cycles_start,cpu-cycles_stop,cpu-cycles_delta,instructions_start,instructions_stop,instructions_delta
1,npb-omp,cg,S,,0,clock_gettime,123...,456...,789,0,0,12345,3,3,0,1,perf_event_open,1000,2500,1500,8000,12000,4000
Notes:
- One row per start/stop interval; appended to an in-memory buffer outside the timed region and flushed at
tacvar_fini(). - Prefer binding (
mpirun --bind-to core,OMP_PROC_BIND,taskset) somigratedstays 0 for asm counters. - With
TACVAR_COUNTER_BACKEND=none, the file stops atsource(nocounter_backendcolumn). - EP Class large runs reserve
1+npevents before timing; allocation failure aborts viaMPI_Abortbefore the timed region. DT distinguishes BH/WH/SH viatest_tag.
Unchanged: NPB computational / MPI / OpenMP workload statements inside kernels (BT/CG/…), lmbench lat_* / bw_* kernels and bench.h body logic. NPB-MPI major loops may wrap unchanged bodies with tacvar_prepare / tacvar_step_* only.
Allowed adapters only: NPB common/c_timers.c, common/timers.f90, common/tacvar_npb.*, common/tacvar_npb_regions.*; lmbench lib_timing.c boundaries, tacvar_lmbench.*; suite Makefiles / scripts/build; src/measure/**.
Protected-path list for tests: src/measure/tests/protected_sources.txt. Workload hash guard: src/measure/tests/check_npb_workload.py.
# Unit / mock (config, deferred buffer, fork, region_info, workload guard)
bash src/measure/tests/run_unit_tests.sh
# Build every legal timer/counter for this arch (temp conf; does not edit suite confs)
bash src/measure/tests/run_backend_smoke.sh
# NPB: --build-only | --run-smoke
# Smoke includes OMP CG S, MPI IS S, plus MG/CG per-step (switch 0/1) with PAPI_TOT_CYC,PAPI_TOT_INS
bash suites/NPB3.4.4/test_tacvar.sh --run-smoke
# Pairwise NPB-MPI timer×reader matrix on MG Class S (PASS/SKIP/FAIL)
bash suites/NPB3.4.4/test_npb_backend_matrix.sh
# lmbench: lat_syscall null on one core; optional platform combo
bash suites/lmbench/scripts/test_tacvar.sh --run-smoke
# ARM host smoke (needs MPI/PAPI/kmod on that machine; uses PAPI_HOME=/home/hpckey/01-App/papi)
bash src/measure/tests/run_arm_tests.shx86 smoke matrix (scripted): native+none for all three suites; then lmbench tsc_asym+asm, NPB-OMP clock_gettime+perf_event_open, NPB-MPI mpi_wtime+papi_read. ARM: native+none, then lmbench cntvct_el0+perf_event_open, NPB-OMP cntvct_el0_dmb+asm, NPB-MPI mpi_wtime+perf_event_open (set TACVAR_NSTP / TACVAR_NSTP_ARM to ns per CNTVCT tick from CNTFRQ_EL0). Pairwise backend matrix covers every supported timer and reader at least once (not Cartesian); host permission / event-availability failures are SKIP, while compile/crash/malformed/zero-counter-after-init/verification failures are FAIL. If PAPI’s HW component is unavailable on a platform, use perf_event_open or asm instead of papi_read; papi_get_real_nsec may still build.
Acceptance: exit 0; NPB Verification = SUCCESSFUL; lmbench prints a syscall latency; CSV present with matching timer/counter names; with switch 1, MG Class S has 4 step rows/rank and CG Class S has 15; region_info.csv present; protected sources unchanged.
# NPB-OMP
cd suites/NPB3.4.4/NPB3.4-OMP
OMP_NUM_THREADS=4 OMP_PROC_BIND=true ./bin/cg.S.x
ls -d data_*
# NPB-MPI
cd suites/NPB3.4.4/NPB3.4-MPI
mpirun -np 4 --bind-to core ./bin/is.S.x
# lmbench
cd suites/lmbench
taskset -c 0 ./bin/$(cd src && ../scripts/os)/lat_syscall null| Symptom | Likely cause |
|---|---|
perf_event_open ... Permission denied |
Need exclude_kernel (already set) or lower perf_event_paranoid; or use papi_read / asm |
cannot open ph_enable_pmu |
Load kmod before asm |
cntvct rejected / bad ns |
Set TACVAR_NSTP (ns/tick); check CNTFRQ_EL0 |
| Stale counter symbols after conf change | Remove config/tacvar_build_$(uname -m) (NPB) or bin/$OS/tacvar_build + lib_timing.o (lmbench) and rebuild |
lmbench lat_rpc fails |
Missing libtirpc-devel; non-RPC tools still build via src/compat stubs |
Wrong bin/ OS dir for lmbench |
Run scripts/os from src/ so gnu-os is found |
PAPI_add_event ... Component ... disabled |
PAPI HW component unavailable on this CPU; use perf_event_open or asm |
setparams: cannot execute binary file |
NFS-shared x86/ARM objects; remove sys/setparams / common/*.o and rebuild on the current host |
For multi-timer × counter-profile experiments on NPB-MPI only (total NAS time, NPB detail timers, and TacVar per-step region_id=1000 distributions), use the reusable environment under utils/scripts/. Full commands, field tables, and troubleshooting live in utils/scripts/README.md. Agent workflow: SKILLS/test-npb-mpi-timer-comparison/SKILL.md.
Setup (Python ≥3.10, matplotlib; Times New Roman preferred for publication figures):
cd utils/scripts
uv sync
source .venv/bin/activateTwo-stage run (explicit campaign fields only — no silent host/class/timer defaults):
- Freeze
campaign.jsonand generate00_pretest.shwithpython3 generate_npb_timer_jobs.py init ...(orpretest --spec ...). - On the target host, run the pretest detached (
nohup bash 00_pretest.sh ...or./run_npb_mpi_pretest.sh <results-dir>). Expectpreflight/STATUS=PASSand matching digests. - Only then generate and run
01_fulltest.sh(generate_npb_timer_jobs.py fulltest/run_npb_mpi_fulltest.sh). Scripts restoretacvar.confon exit and refuse conflicting same-usermpirunjobs without killing them.
Analyze / plot an existing result tree (new campaign.json layouts or legacy runs/<timer>_<profile>/<kernel>/):
RESULTS=suites/NPB3.4.4/NPB3.4-MPI/results_<host>_class<C>_...
python3 utils/scripts/plot_npb_mpi_step_histograms.py "$RESULTS"
python3 utils/scripts/plot_npb_mpi_step_icdf.py "$RESULTS"
python3 utils/scripts/analyze_npb_timer_campaign.py "$RESULTS"
# writes summary/*.csv, summary/figures/*.{png,pdf,eps}, and REPORT.mdNotebook-style imports: sys.path.insert(0, "utils/scripts") then from npb_timer_campaign import load_campaign, analyze_campaign, .... Use --allow-font-fallback only when Times New Roman is unavailable (recorded in the report). Generated results_*/ trees and utils/scripts/.venv/ are gitignored.