Repository navigation
cabal bench should not run benchmarks in parallel #7557
Description
Activity
I think this would definitely be a welcome improvement.
I'm fairly sure benchmark parallelism is handled at the
cabal-installlevel.Cabalonly has build parallelism (through GHC)cabal bench -j1?Not necessarily: I'm still happy for cabal to build benchmarks in parallel.
[EDIT: I think I wrote this as a response to #10133, specifically interleaved benchmark output. Oops. Not so relevant here.]
What kind of solution would be acceptable here?
I guess cabal-install must already have some machinery for deconflicting parallel activity -- or maybe not?
I suppose the rough architecture would be
Loadingflowchart LR startbuild((start)) --> benches subgraph benches ["`benches *(output captured)*`"] bench1 bench2 bench3 end startbuild --> runningStatus runningStatus --> finish((finish)) bench1 & bench2 & bench3 --> finish finish --> summaryResults
We could default
cabal benchto-j1. One can always build the benchmarks in parallel with a separatecabal build. However, changing the defaults depending on the command is not great (there's some discussion about this in #5079 and #7883).I guess cabal-install must already have some machinery for deconflicting parallel activity -- or maybe not?
There's
Lock. SeeregisterLockfor an example.Another thing to consider is that parallel builds can also affect benchmark results. For example if
bench1is running whilebench2(or any other component) is building,bench1might still be affected.If anybody try tackling this, we'll try to help on the matrix channel. Pretty please!
please, consider reviewing a proposed fix for this here:
The comment I had meant to write on this thread is that sometimes benchmarks should be run in parallel. All my benchmarks in one project are load-independent; making them serial just wastes cycles. It would be fine with me if the default changed but I'd hope I could still choose to run them in parallel.
I don't quite follow how benchmarks can be load-independent.
E.g.
perfbased cpu-cycles / instruction counting ones.- Yeah, exactly that. I also have a max-bytes-retained benchmark somewhere, I think.
E.g.
perfbased cpu-cycles counting ones.CPU cycles are affected by cache misses and page faults, aren't they? This makes them very much load-dependent.
CPU cycles are affected by cache misses and page faults, aren't they?
And one can count cache misses and page faults too.
Or maybe I just want to run benchmarks to see that they work (as part of CI), not actually caring about the output that much.
The difference between test-suites and benchmarks is very blurry. I'm sure someone could find a reason to run all test-suites in serial too e.g. because they only have only one DB and don't want to write locks inside their tests: something you can do for you benchmarks too.
When there are several benchmark suites in a package (or - which happens more often - in a project),
cabal benchruns them all in parallel, which makes results completely useless. Couldcabalrun benchmarks sequentially?