[ci] strip pcms from wheels builds - #23071
Conversation
`TSystem::CompileMacro`, aka `ACLiC`, can generate a shared library from a ROOT macro. While doing so, it links the target library against all loaded libraries at time of calling.
When running from Python e.g. via `ProcessLine(".L macro.C+")`, the loaded libraries include those dynamically loaded by the Python interpreter.
When run on a MacOS system, these may also include so-called "private" Apple libraries. These libraries cannot be linked against by third-party code: the linker will fail with an error saying "product being built is not an allowed client of it".
One such situation has surfaced in a bug report to the homebrew distribution of ROOT Homebrew/homebrew-core#272324 related to the `libffi` library, but the issue is actually generic.
Unfortunately, Apple does not provide an authoritative list of "private" libraries and there is no direct way to ask a library whether linking against it will produce a linker error or not.
Aside from manually checking the list of allowed clients of each of the loaded libraries before linking, manually vetoing known "private" libraries seems the next-best thing. This commit proposes to remove the `libffi` library from the list of libraries `ACLiC` will link against when run on MacOS.
Test Results 4 files 4 suites 6h 33m 4s ⏱️ For more details on these failures, see this check. Results for commit 4c69dca. ♻️ This comment has been updated with latest results. |
|
What is the time taken by the rebuild of the pcms? |
| if (index >= 0) { | ||
| TString s = libs2(index, end); | ||
| if (s.Index(dynload) == kNPOS) { | ||
| if (s.Index(dynload) == kNPOS && s.Index(libffiMatch) == kNPOS) { |
There was a problem hiding this comment.
This seems to (also) be introduced in #22963.
However, the location of this code is somewhat surprising and I wonder if it should be not be with the other vetoing in TCling::RegisterLoadedSharedLibrary. (Or at the very least to code comment why this one is here and the other there).
There was a problem hiding this comment.
Indeed this was merged in master but not in the branch root-project:test_macos_wheel_v2 so I just cherry picked it for the tests to work
The PRs I have open on this branch (#22886 and #23071) are just experimenting with the macos wheels, not going into production as they are of course
|
@pcanal Around 20s for the first |
| for i, name in enumerate(names, 1): | ||
| _print_progress(i, len(names), name) | ||
| try: | ||
| declare(f"#pragma clang module import {name}") |
There was a problem hiding this comment.
Is this out the pcms are being rebuild? If so, I am not sure this is the same as the pcms produced by rootcling. The pcms produced by rootcling contains not only the 'real' pcm but also have an additional payload which was previously the rootpcm files which contains shortcuts to the class information (in the form of TProtoClass stored in a ROOT file) which allows to be able to do I/O without actually loading any interpreter information (i.e. the real pcm content) which improve run-time and drastically reduce the memory needed.
3 participants
Test a new strategy: ship ROOT wheels without pre-built C++ module
.pcmfiles, and rebuild them once on the target machine instead..pcmfiles are frozen against the system headers of the build machine creating portability issues to other machines with different versions. This PR strips them out at CI build time and adds a one-time warm-up script that rebuilds them locally on first import ROOT against whatever headers actually exist on the user's machine.