Repository navigation
Flaky: tracker flush and logging-output tests fail intermittently on master #1371
Description
Three tests fail intermittently when the whole suite runs, across two files:
tests/test_emissions_tracker_flush.py::test_carbon_tracker_offline_flushtests/test_logging_output.py::test_carbon_tracker_offline_logging_output
The failures are row counts (0 == 1, 2 == 3), so an expected measurement did not land before the assertion. Each test passes in isolation, and it reproduces on master, not just on feature branches.
The most likely cause is how CI runs the suite. .github/workflows/test-package.yml:36 is:
uv run codecarbon monitor -- uv run task test-coverage
so a live tracker wraps the entire test run. It holds /tmp/.codecarbon.lock and writes its own emissions.csv in the repo root for as long as the tests execute, while the tests under it are starting and stopping their own trackers. That is a real interaction and it is not present when you run a single test locally.
Two smaller things that make it worse:
tests/test_logging_output.pybuilds three trackers with nooutput_dirand nooutput_file(lines 48-52, 64-69, and the@track_emissionsdecorator at 81-85), so they all default toemissions.csvin the current working directory. Nothing cleans that up.- Both files use a fixed filename in
tempfile.gettempdir()(emissions-test-TestCarbonTrackerFlush.csv,emissions-test-TestCarbonLoggingOutput.log) with no per-run suffix, andsetUp/tearDownboth delete it.
PeriodicScheduler chaining a threading.Timer per tick may also contribute, since a tracker a test does not stop keeps firing into the next test and tests/conftest.py only clears the hardware probe caches. Unverified. The re-arming timer goes away with #1339 / #1324, so it is worth re-measuring after one of those lands.
Likely fixes: give the logging test an explicit output_dir, make the temp filenames unique per process, and stop asserting on wall-clock-dependent row counts.
Activity
davidberenstein1957 commentedon Aug 12, 2026
Additional finding from a separate investigation, which narrows this considerably.
tests/test_emissions_tracker_flush.pyandtests/test_logging_output.pyare order/state-sensitive around a sharedemissions.csvin the repository root. They fail onorigin/masterunder some orderings and pass afterrm emissions.csv.So the first thing to try is not a scheduler fix but making these tests write to a per-test temporary directory instead of the repo root — and checking whether a stray
emissions.csvis being left behind by an earlier test in the run.That likely also explains why the failures are on row counts (
0 == 1,2 == 3): the assertions are counting rows in a file that a previous test already wrote to.The leaked-timer theory above may still be a contributing factor, but this one is cheaper to check and easier to fix.
davidberenstein1957 commentedon Aug 16, 2026
Correcting my comment above — it got the mechanism wrong, and I have updated the issue body accordingly.
The claim was that
tests/test_emissions_tracker_flush.pyandtests/test_logging_output.pyare sensitive to a sharedemissions.csvin the repository root because they write their output there. They do not. On currentmasterboth already write into the system temp directory:tests/test_emissions_tracker_flush.py:26-30—self.emissions_path = tempfile.gettempdir()tests/test_logging_output.py:27-31— sameSo "make these tests write to a temporary directory instead of the repo root" is not the fix; that part is already done.
What is actually shared:
emissions-test-TestCarbonTrackerFlush.csvandemissions-test-TestCarbonLoggingOutput.logare constant paths, deleted in bothsetUpandtearDown. Two concurrent runs of the suite on the same machine (xdist, parallel CI jobs, a second local run) race for the same file, which produces exactly the missing-row assertions seen.output_dir.tests/test_logging_output.py:47-52constructsEmissionsTracker(project_name=..., save_to_logger=True, logging_logger=...)withoutoutput_dir/output_file, so it writesemissions.csvinto the current working directory — the repository root. That is where the stray root-levelemissions.csvcomes from, and nothing cleans it up. So therm emissions.csvobservation was real; the attribution was not.Also worth flagging:
tests/conftest.pyonly clears the hardware/TDP/GPU probe caches. It does not cancel or count leftover timers, so the leaked-scheduler theory in the original report remains untested rather than supported.[-]Flaky: test_emissions_tracker_flush tests fail intermittently on master (order/shared-state dependent)[/-][+]Flaky: tracker flush and logging-output tests fail intermittently on master[/+]