StripEnergyThresholdFinder for per-strip slow and fast threshold extraction with diagnostics by JarredMRoberts · Pull Request #166 · cositools/nuclearizer

JarredMRoberts · 2026-06-23T05:33:43Z

Adds a standalone application, StripEnergyThresholdFinder, for computing per-strip slow and fast energy thresholds. Slow thresholds are determined from ADC spectra using a noise peak and trough method, while fast thresholds are determined from dt0/dt1 timing crossover. The tool reads calibrated HDF5 data via MModuleLoaderMeasurementsHDF, applies strip mapping and energy calibration from a YAML configuration, and produces ROOT diagnostic outputs (energy spectra with thresholds, dt0 vs dt1 per strip, and threshold distributions) along with CSV export files. The implementation is self-contained under apps/ and does not modify existing modules. Tested on COSI datasets with consistent threshold behavior and expected diagnostic results. Target branch is develop/em.

…extraction

JarredMRoberts · 2026-06-23T06:55:33Z

I still need to fix all of the code style issues and work on some optimizations to speed the code up a bit.

fhagemann · 2026-06-23T20:55:00Z

Is this different from #143 or making #143 obsolete?

Co-authored-by: Felix Hagemann <hagemann@berkeley.edu>

…nventions and integrate energy calibration improvements

fhagemann

Hi Jarred,
I had an extensive look at this app and finally got this to run.
I replied to all comments in this PR, and opened a separate PR onto your fork/branch, addressing some code changes to

get this app running
remove the helper classes by replacing them with existing nuclearizer/megalib code

Here is the PR onto your branch with detailed code changes: JarredMRoberts#1

zoglauer · 2026-08-17T21:09:02Z

I second not to have an additional parser linked.
You can use the nuclearizer/megalib XML file format instead; your file then would be:

<strip-threshold-finder>
    <input>
        <data_files>
            <file>/path/to/data/file.hdf5</file>
        </data_files>
        <calibration_file>/path/to/calibration/file.ecal</calibration_file>
        <tac_calibration_file>/path/to/tac/calibration/file.csv</tac_calibration_file>
        <strip_map>/path/to/strip/map/file.map</strip_map>
    </input>

    <analysis>
        <!-- number of skipped strips with minimum statistics -->
        <min_entries>10</min_entries> 
        
        <!-- If a threshold cannot be determined set threshold to default value -->
        <fallback_threshold_keV>20</fallback_threshold_keV> 
        
        <!-- limit for locating the low-energy noise peak -->
        <!-- most noise peaks should be between 100 and 250 -->
        <!-- Worst case the ADC max should be set to ~1000 -->
        <noise_search_max_adc>1800</noise_search_max_adc>
    </analysis>
    
    <output>
        <prefix>output_file_prefix</prefix>
    </output>
</strip-threshold-finder>

fhagemann

One more remark: I got this app to run and got reasonable results for the fast and slow thresholds only when using a dataset taken with NN off.
We might want to always filter out NN events, for this app to also run using datasets taken with NN on.

…nges from Felix

…test

…ise peak range from ADC to keV

…fig file error.

…test

fhagemann

This is a quick review with code-style comments, I will follow up with some more general comments

fhagemann

This is a quick review with code-style comments, I will follow up with some more general comments

Co-authored-by: Felix Hagemann <hagemann@berkeley.edu>

fhagemann

Some general comments:

do you have some figures to walk us through the pedestal/peak finding algorithms, to get an idea in what cases the pedestal, flattop or peak detection loops would do their thing?
we might want to avoid having hard-coded values distributed throughout the file and define them as constants at the top of the file (or member variables of the threshold app class) instead.
In the end, we might want to generate ONE threshold file per threshold type (fast, slow, hardware), instead of splitting them into LV and HV subfiles. I believe that the forward pipeline will require a combined file in the end anyhow

fhagemann · 2026-09-04T22:14:21Z

+        // --- dt0 vs dt1 separation ---
+        bool is_dt1 = (TAC > 8000); // initial threshold


Is this ever not the case?
Also, should this hard-coded value become some member variable or be defined at the very beginning of the file, instead of very far into the code?

Looking at this in more detail: shouldn't this be:

Suggested change

// --- dt0 vs dt1 separation ---

bool is_dt1 = (TAC > 8000); // initial threshold

// --- dt0 vs dt1 separation ---

bool is_dt1 = SH->HasFastTiming();

??

Tested on data from HP52406-1 (Am-241 lifetime data, taken with HDFv2.2: gse_20260217T115220.hdf5):

With the current TAC > 8000 line, I get nothing in HV strip 32 (no dt1 distribution, everything seems to be classified as dt0), resulting in the fallback fast threshold:

WARNING: FAST threshold could not be determined for Det 0 Side HV Strip 32: insufficient timing populations (dt0=88441, dt1=293, dt1 fraction=0.3302%). Using fallback threshold = 34 keV.

Changing it to SH->HasFastTiming() results in reasonable dt0 and dt1 distributions, but the fast threshold seems to be determined incorrectly:

This is my XML config file (using strip map and ecal file from the unit tests):

<?xml version="1.0" encoding="UTF-8"?> <StripEnergyThresholdFinder> <Input> <DataFiles> <DataFile>/home/hagemann/Downloads/gse_20260217T115220.hdf5</DataFile> </DataFiles> <CalibrationFile>/home/hagemann/Software/COSItools/nuclearizer/resource/unittestdata/406-1/hp52406-1.full.ecal</CalibrationFile> <StripMap>/home/hagemann/Software/COSItools/nuclearizer/resource/unittestdata/406-1/hp52406-1.stripmap.map</StripMap> </Input> <Output> <Prefix>output_file_prefix</Prefix> </Output> <Analysis> <MinEntries>10</MinEntries> <FallbackThresholdKeV>20</FallbackThresholdKeV> <NoiseSearchMaxKeV>40</NoiseSearchMaxKeV> <FastFallbackKeV>34</FastFallbackKeV> </Analysis> </StripEnergyThresholdFinder>

fhagemann · 2026-09-04T22:16:45Z

+      thresholds[R] = m_FallbackThreshold;
+
+	  // Mark the ADC value as invalid since we cannot easily convert back from
+	  // the fallback keV value to ADC bins
+      thresholdsADC[R] = -1.0;


The current implementation will give garbage values in thresholdsADC, but not in thresholds. Is this what we want?

fhagemann · 2026-09-04T22:16:56Z

+      continue;
+    }
+
+    hist->Smooth(3);


What is this doing?

fhagemann · 2026-09-04T22:25:08Z

+      if (c > peakCounts) {
+
+        // Still climbing: update candidate peak
+        peakCounts = c;
+        peakBin = b;
+        declineCount = 0;
+
+      } else {


Just checking that this is not an issue: some of the histogram bins seem empty in keV space, does this affect finding the leading edge?

fhagemann · 2026-09-04T22:27:12Z

+      // the threshold a few bins into the flat top.
+      thresholdBin = pedestalThresholdBin;
+
+    } else if (R.GetStripID() == 64) {


Avoid hard-coded numbers in the code and put them somewhere at the top of the file as GR_STRIP_ID (this code might be used, and maybe the GR strip ID is not 64 then).

fhagemann · 2026-09-04T22:29:06Z

+      // Require minimum statistics to avoid noise triggers
+      if (n0 + n1 < 50) {
+        continue;
+      }
+
+      if (n1 > n0) {
+        crossoverADC = ADC_val;
+        break;
+      }
+    }
+
+    int nbins = 21;
+
+    // Extend search window for stability
+    int searchMax = first_nonzero + 800;


There are a lot of hard-coded values here, that we might define as constants towards the top of the file:
50 or minimum number of counts, 21 for nbins and 800 for extending the window.

fhagemann · 2026-09-04T22:35:31Z

+  MString HVOutputCSVFileName = m_OutputPrefix + "_Slow_HV_thresholds.csv";
+  MString LVOutputCSVFileName = m_OutputPrefix + "_Slow_LV_thresholds.csv";
+
+  ofstream csv_HV(HVOutputCSVFileName);
+  ofstream csv_LV(LVOutputCSVFileName);
+
+  if (csv_HV.is_open() == false) {
+    cerr << "Error: Failed to open CSV output file for HV thresholds: "
+         << HVOutputCSVFileName << endl;
+    return;
+  }
+
+  if (csv_LV.is_open() == false) {
+    cerr << "Error: Failed to open CSV output file for LV thresholds: "
+         << LVOutputCSVFileName << endl;
+    return;
+  }


We might want to write everything to the same file instead of two separate ones.
Looks like the first token of every line already defines the side, so we should be able to write everything into one combined slow thresholds file (I think this is also what the forward pipeline would expect).

fhagemann · 2026-09-04T22:36:00Z

+  MString HardwareHVOutputCSVFileName =
+    m_OutputPrefix + "_Slow_Hardware_HV_thresholds.csv";
+
+  MString HardwareLVOutputCSVFileName =
+    m_OutputPrefix + "_Slow_Hardware_LV_thresholds.csv";
+
+  ofstream csv_Hardware_HV(HardwareHVOutputCSVFileName);
+  ofstream csv_Hardware_LV(HardwareLVOutputCSVFileName);
+
+  if (csv_Hardware_HV.is_open() == false) {
+    cerr << "Error: Failed to open CSV output file for HV hardware thresholds: "
+         << HardwareHVOutputCSVFileName << endl;
+    return;
+  }
+
+  if (csv_Hardware_LV.is_open() == false) {
+    cerr << "Error: Failed to open CSV output file for LV hardware thresholds: "
+         << HardwareLVOutputCSVFileName << endl;
+    return;
+  }


Also here: combine the two hardware thresholds file into one.

fhagemann · 2026-09-04T22:36:15Z

+  ofstream csv_TAC_HV(m_OutputPrefix + "_Fast_HV_thresholds.csv");
+  ofstream csv_TAC_LV(m_OutputPrefix + "_Fast_LV_thresholds.csv");
+
+  csv_TAC_HV << "detector_side,Strip,threshold_ADC,threshold_keV\n";
+  csv_TAC_LV << "detector_side,Strip,threshold_ADC,threshold_keV\n";
+
+  cout << "Writing FAST CSV entries: " << m_FastThresholds.size() << endl;
+
+  for (const auto& kv : m_FastThresholds) {
+    MReadOutElementDoubleStrip R = kv.first;
+    int Strip = R.GetStripID();
+
+    double thr_ADC = m_FastThresholdsADC[R];
+    double thr_keV = kv.second;
+
+    if (R.IsLowVoltageStrip() == true) {
+      csv_TAC_LV << "l," << Strip << "," << thr_ADC << "," << thr_keV << "\n";
+    } else {
+      csv_TAC_HV << "h," << Strip << "," << thr_ADC << "," << thr_keV << "\n";
+    }
+  }


And here: combine the two fast threshold files into one.

fhagemann

I can confirm that the app compiles and works when using multiple files.
You might want to remove the index from the XML file (it also works without it).

fhagemann · 2026-09-10T21:42:21Z

+      <DataFile index="0">/path/to/your/data/file0.hdf5</DataFile>
+      <DataFile index="1">/path/to/your/data/file1.hdf5</DataFile>


Suggested change

<DataFile index="0">/path/to/your/data/file0.hdf5</DataFile>

<DataFile index="1">/path/to/your/data/file1.hdf5</DataFile>

<DataFile>/path/to/your/data/file0.hdf5</DataFile>

<DataFile>/path/to/your/data/file1.hdf5</DataFile>

… Hardware). Added hardware diagnostics in ADC units

JarredMRoberts added 2 commits June 1, 2026 18:15

Add StripEnergyThresholdFinder app

6dc7104

Add StripEnergyThresholdFinder for per-strip slow and fast threshold …

2f699ef

…extraction

fhagemann reviewed Jun 23, 2026

View reviewed changes

fhagemann requested changes Jun 23, 2026

View reviewed changes

Comment thread apps/StripEnergyThresholdFinder.cxx Outdated

fhagemann marked this pull request as draft June 23, 2026 21:50

JarredMRoberts and others added 3 commits June 30, 2026 07:33

Update apps/StripEnergyThresholdFinder.cxx

dc2f2fb

Co-authored-by: Felix Hagemann <hagemann@berkeley.edu>

Update apps/StripEnergyThresholdFinder.cxx

808ad18

Co-authored-by: Felix Hagemann <hagemann@berkeley.edu>

Revision of StripEnergyThresholdFinder in-line with MEGAlib coding co…

a78bc0c

…nventions and integrate energy calibration improvements

JarredMRoberts marked this pull request as ready for review July 28, 2026 08:40

fhagemann reviewed Jul 28, 2026

View reviewed changes

Comment thread apps/StripEnergyThresholdFinder.cxx Outdated

fhagemann reviewed Aug 10, 2026

View reviewed changes

Comment thread apps/StripEnergyThresholdFinder.cxx Outdated

fhagemann mentioned this pull request Aug 12, 2026

[DEE] Add hardware thresholds to determine if a MDEEStripHit was a NearestNeighbor or not #190

Draft

fhagemann requested changes Aug 17, 2026

View reviewed changes

Comment thread apps/StripEnergyThresholdFinder.cxx Outdated

fhagemann reviewed Aug 17, 2026

View reviewed changes

Comment thread apps/StripEnergyThresholdFinder.cxx Outdated

ckierans mentioned this pull request Aug 18, 2026

Determine hardware threshold in Nuclearizer App #195

Open

JarredMRoberts and others added 10 commits August 18, 2026 08:21

Save working threshold finder implementation before merging other cha…

747ebbc

…nges from Felix

Remove left-over compiled StripEnergyThresholdFinder app

01a3e8f

Add yaml-cpp libraries and headers to Makefile

a2786a1

Fix typo when parsing YAML file

bc3193a

Replace StripKey with ReadOutElementDoubleStrip

c968fa3

Replace EnergyCalHelper with MModuleEnergyCalibration

aedf5c5

Remove obsolete TACCalHelper

1a41cd0

Set g_Verbosity to display error messages from nuclearizer modules

473d41a

Throw warnings if CSV files could not be opened

a4ffe29

Exclude nearest neighbor hits when reading data

05b50ab

fhagemann mentioned this pull request Aug 20, 2026

Replace helper classes by existing nuclearizer/megalib objects JarredMRoberts/nuclearizer#1

Merged

JarredMRoberts added 4 commits August 21, 2026 17:09

Improve strip threshold finder robustness and detector selection

bc85213

Improve strip threshold finder robustness and configuration

e383626

Clean up threshold finder configuration and documentation

ea7f190

Merge remote-tracking branch 'upstream/develop/em' into test/felix-la…

d00c721

…test

fhagemann reviewed Aug 23, 2026

View reviewed changes

Comment thread .github/workflows/tests.yaml

JarredMRoberts and others added 6 commits August 25, 2026 06:52

Pin yaml-cpp version in CI environment

2d674dd

Replace YAML threshold finder config with MEGAlib XML. Updated max no…

2e8c118

…ise peak range from ADC to keV

Add slow hardware threshold extraction and diagnostics. Fixed XML con…

73d3a96

…fig file error.

Merge remote-tracking branch 'upstream/develop/em' into test/felix-la…

3125d2d

…test

FIX: Syntax errors in CI test.yaml file

565f8ff

FIX: Remove YAML-related flags and libs from Makefile

2438194

fhagemann reviewed Sep 4, 2026

View reviewed changes

CHG: Apply code-style changes

8adaafa

Co-authored-by: Felix Hagemann <hagemann@berkeley.edu>

fhagemann requested changes Sep 4, 2026

View reviewed changes

CHG: More code-style changes

f68608a

fhagemann requested changes Sep 5, 2026

View reviewed changes

Comment thread apps/StripEnergyThresholdFinder.cxx Outdated

JarredMRoberts added 2 commits September 9, 2026 16:52

FIX: Process multiple threshold input files

fad7889

CHG: Allow threshold processing for all detectors

b2bf030

fhagemann reviewed Sep 10, 2026

View reviewed changes

Improved threshold output files (Three total: all Slow, all Fast, all…

40cee1e

… Hardware). Added hardware diagnostics in ADC units

		// --- dt0 vs dt1 separation ---
		bool is_dt1 = (TAC > 8000); // initial threshold

		<DataFile index="0">/path/to/your/data/file0.hdf5</DataFile>
		<DataFile index="1">/path/to/your/data/file1.hdf5</DataFile>

Uh oh!

Conversation

JarredMRoberts commented Jun 23, 2026

Uh oh!

JarredMRoberts commented Jun 23, 2026

Uh oh!

fhagemann commented Jun 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fhagemann left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zoglauer commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fhagemann left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

fhagemann left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fhagemann left a comment

Choose a reason for hiding this comment

Uh oh!

fhagemann left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fhagemann Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

fhagemann left a comment •

edited

Loading

zoglauer commented Aug 17, 2026 •

edited

Loading

fhagemann Sep 5, 2026 •

edited

Loading