[build] Ship the clang builtin headers and prefer them at runtime - #33
[build] Ship the clang builtin headers and prefer them at runtime#33aaronj0 wants to merge 1 commit into
Conversation
Test Results
|
198d0e3 to
46a6799
Compare
| // #include. Prefer the bundled copy: it matches the build clang and | ||
| // needs no LLVM on the host. | ||
| std::string resourceDir = Paths.ClangIncludeDir; | ||
| if (resourceDir.empty() && Cpp::DetectResourceDir("clang").empty()) |
There was a problem hiding this comment.
Can we prioritize the system's resource directory.
First, check clang-version, then the one we include, then simple clang.
There was a problem hiding this comment.
No that would not work. If we build a binary for manylinux or OS X based on llvm22 and a user installs it on their system which happens to have a lower LLVM version (apple always has clang) that would crash the interpreter. The point is that cppjit should not depend on system LLVM.
There was a problem hiding this comment.
That is exactly the problem I described in last weeks meeting.
There was a problem hiding this comment.
No that would not work. If we build a binary for manylinux or OS X based on llvm22 and a user installs it on their system which happens to have a lower LLVM version (apple always has clang) that would crash the interpreter. The point is that cppjit should not depend on system LLVM.
That would fail the match of the resource dir folder name which incorporates the version. We can also protect against older clangs such a check...
There was a problem hiding this comment.
That would fail the match of the resource dir folder name which incorporates the version. We can also protect against older clangs such a check...
Can you point me to which match fail you are referring to? Yes we can incorporate a check if the major version matches but just to clarify, we decided to bundle the headers so that we don't run into this problem and introduce mechanisms to ensure that system or package manager LLVM's agree with what cppjit needs.
If we always prioritize system-installed LLVM, that goes against the very solution you proposed (and we agreed on) to bundle the headers, leading to a self contained binary and not depend on an arbitrary LLVM picked up at runtime which can differ based on user environments
There was a problem hiding this comment.
What the review suggestion here proposes is to prioritize the arbitrary clang headers detected at runtime over the one that we bundle and I disagree with that. If anything, system clang can be used as a fallback but not as the first pick for resource dir since there is no guarantee that exists or even work if it does.
There was a problem hiding this comment.
pip install --no-build-isolation -ve .
That's 1 package manager. Does that solution work for non-pip setups?
What the review suggestion here proposes is to prioritize the arbitrary clang headers detected at runtime over the one that we bundle and I disagree with that. If anything, system clang can be used as a fallback but not as the first pick for resource dir since there is no guarantee that exists or even work if it does.
Where my comments suggested that?
There was a problem hiding this comment.
That's 1 package manager. Does that solution work for non-pip setups?
Yes. Development mode works because this is not a pip-specific solution but in the the underlying build system. The installation layout is fixed and under our control irrespective of the package manager. Here is a pure CMake build, without pip, against an uninstalled developer build tree of LLVM:
$ python3 -m venv demo-venv # python to build against
$ cmake -S cppjit -B build \
-DCMAKE_BUILD_TYPE=Release \
-DLLVM_DIR=$HOME/llvm-project/release_build/lib/cmake/llvm \
-DPython_EXECUTABLE=$PWD/demo-venv/bin/python
$ cmake --build build -j$(nproc)
$ cmake --install build --prefix $PWD/cppjit-install
$ cp -r cppjit/python/cppjit cppjit-install/ # normally done by the wheelThe installed tree carries the builtin headers of that LLVM:
$ diff -r $HOME/llvm-project/release_build/lib/clang/21/include \
cppjit-install/cppjit_backend/lib/clang/21/include && echo IDENTICAL
IDENTICALSo irrespective of what LLVM we develop against, the runtime prefers the headers libclangCppInterOp was compiled against. In a developer's uninstalled build tree there are no bundled headers, and the loader falls back to DetectResourceDir, which is unchanged in this patch.
The result is self-contained either way. With an empty PATH (no clang visible to the process):
$ PYTHONPATH=$PWD/cppjit-install PATH=/nonexistent demo-venv/bin/python \
-c "import cppjit; cppjit.cppdef('#include <vector>\nint f(){ std::vector<int> v{41}; return v[0]+1; }'); print('f() =', cppjit.gbl.f())"
f() = 42The point: This does not depend on a package manager, but a property of the CMake install rules which is run by every install channel. At the end of the day, cppjit is a Python package and any PEP 517 (Python standard that separates the build frontend from the build backend) frontend, drives it through scikit-build-core. So this satisfies conda recipes, conda build environments, a linux distro specific package manager, and a bare cmake --install in a developers setup (with no Python packaging at all). We are satisfying the Python build standard. If any distribution or install does not contain the bundled headers (which is impossible unless someone removes them on purpose) we try the DetectResourceDir fallback with clang. We can improve that fallback to restrict to the supported LLVM version but that is outside the scope of this PR.
There was a problem hiding this comment.
Where my comments suggested that?
I was referring to the original review comment in this thread:
Can we prioritize the system's resource directory.
First, check clang-version, then the one we include, then simple clang.
There was a problem hiding this comment.
That's 1 package manager. Does that solution work for non-pip setups?
Yes. Development mode works because this is not a pip-specific solution but in the the underlying build system. The installation layout is fixed and under our control irrespective of the package manager. Here is a pure CMake build, without pip, against an uninstalled developer build tree of LLVM:
$ python3 -m venv demo-venv # python to build against $ cmake -S cppjit -B build \ -DCMAKE_BUILD_TYPE=Release \ -DLLVM_DIR=$HOME/llvm-project/release_build/lib/cmake/llvm \ -DPython_EXECUTABLE=$PWD/demo-venv/bin/python $ cmake --build build -j$(nproc) $ cmake --install build --prefix $PWD/cppjit-install $ cp -r cppjit/python/cppjit cppjit-install/ # normally done by the wheelThe installed tree carries the builtin headers of that LLVM:
$ diff -r $HOME/llvm-project/release_build/lib/clang/21/include \ cppjit-install/cppjit_backend/lib/clang/21/include && echo IDENTICAL IDENTICALSo irrespective of what LLVM we develop against, the runtime prefers the headers
libclangCppInterOpwas compiled against. In a developer's uninstalled build tree there are no bundled headers, and the loader falls back toDetectResourceDir, which is unchanged in this patch.The result is self-contained either way. With an empty
PATH(no clang visible to the process):$ PYTHONPATH=$PWD/cppjit-install PATH=/nonexistent demo-venv/bin/python \ -c "import cppjit; cppjit.cppdef('#include <vector>\nint f(){ std::vector<int> v{41}; return v[0]+1; }'); print('f() =', cppjit.gbl.f())" f() = 42The point: This does not depend on a package manager, but a property of the CMake install rules which is run by every install channel. At the end of the day, cppjit is a Python package and any PEP 517 (Python standard that separates the build frontend from the build backend) frontend, drives it through scikit-build-core. So this satisfies conda recipes, conda build environments, a linux distro specific package manager, and a bare
cmake --installin a developers setup (with no Python packaging at all). We are satisfying the Python build standard. If any distribution or install does not contain the bundled headers (which is impossible unless someone removes them on purpose) we try theDetectResourceDirfallback withclang. We can improve that fallback to restrict to the supported LLVM version but that is outside the scope of this PR.
Ok, that clarifies it -- can you update the PR description and the commit message? That would have helped me.
3 participants
This is the second part of the patch that ships the required runtime clang headers. With this change, a build/install of cppjit is self-contained and with the LLVM that is statically linked into
libClangCppInterOp.so, we drop the LLVM dependency on a target machine. The wheels build PR will exercise this on clean images without LLVM.As settled in review: the bundle is produced by the CMake install rules from the LLVM the build was pointed at, so it is identical to the headers
libClangCppInterOpwas compiled against under every install channel (pip, conda recipes, distro tooling, barecmake --install) and for development builds against anyLLVM_DIR. Onlyinclude/ships; the JIT reads nothing else from a resource directory. Setups without a bundle such as raw build trees, or distributions that strip vendored copies, fall back to the existing CppInterOpDetectResourceDirprobing, which was the previous default behaviour (unchanged). This increases wheel size by 3-5MB