Skip to content

test-build has no real dependency edge on installcheck -- races under make -j #107

Description

@jnasbyupgrade

Summary

test-build has no real dependency edge on installcheck in base.mk -- both only
share install as a prerequisite. Under make test (serial, -j1), test-build happens
to run after the outer install/installcheck (which runs the main suite plus any
test/install schedule), but this is an accident of TEST_DEPS list order interacting
with check-stale-expected's own forced edge onto installcheck (added in #79/#83) --
not a real guarantee. Under make -j, the two run concurrently, both invoking
pg_regress against the same live test database at the same time.

Why this matters

Two independent investigations of this in a downstream project (a pgxntool-based
extension) reached opposite conclusions about which runs first by reading base.mk's
declared dependencies -- which is itself a symptom of there being no real, enforced order.

Empirical proof

Added a timestamped marker (\! echo "... $(date +%s%N)" >> /tmp/order_marker.log) to both
a test/install/*.sql file and a test/build/*.sql file, then ran the real suite:

make test (serial, default):

install-roles 1787260491392933258
test-build    1787260493561075143   (~2.17s later)

Non-interleaved stdout confirms the outer installcheck (all test/install + test/sql
entries) completes fully before test-build's own recursive
$(MAKE) ... installcheck recipe even starts.

make -j4 test (parallel):

install-roles 1787260501485143693
test-build    1787260501671491375   (~186ms later)

stdout is now interleaved: test-build's own pg_regress invocation (installing files,
running its build test) runs while the outer schedule's remaining tests are still
executing against the same database -- a genuine race, not just a documentation gap.

Root cause

  • TEST_DEPS = testdeps, then (if enabled) check-stale-expected is appended, then
    test-build, then install installcheck.
  • check-stale-expected: installcheck (a real edge, added for check-stale-expected's installcheck dependency can run installcheck before install on a fresh tree #79) forces installcheck
    (and, via installcheck: install, install) to complete before Make can consider
    check-stale-expected "done" -- and since check-stale-expected precedes test-build in
    TEST_DEPS, under -j1 this happens to force installcheck before test-build gets its
    turn.
  • test-build's own rule is only test-build: install -- no edge to installcheck or
    check-stale-expected. Under -j, nothing stops Make from starting test-build (once
    install is satisfied) while installcheck is still running in another job slot.

Suggested fix

Add a real edge so the relationship is guaranteed rather than incidental, e.g.:

ifeq ($(PGXNTOOL_ENABLE_TEST_BUILD),yes)
test-build: installcheck
endif

(after installcheck's own definition, so the ordering is explicit regardless of
TEST_DEPS list position). Happy to open a PR if a maintainer confirms this is the
preferred shape -- open question whether test-build should run before or after the main
suite is a design decision maintainers should make explicitly rather than leave to list-order
accident; this issue is about there being no enforced order at all today, not which order
is "correct".

Update: see #108, which makes the case that test-build should gate installcheck (run
before it) rather than after -- there's no point running the full suite if the build itself
is broken. That's the direction being adopted; closing this issue as a duplicate of #108.

Workaround applied downstream

That project's Makefile added:

test-build: installcheck

directly (can't edit base.mk, a synced subtree) as an initial fix, later superseded by
installcheck: test-build per #108's corrected direction. Verified under both make test
and make -j4 test that either edge removes the interleaving -- the point of this issue was
that no edge existed, not which direction is correct.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions