Skip to content

[packaging] split native libs from the main jar - #834

Closed
rmannibucau wants to merge 1 commit into
duckdb:mainfrom
rmannibucau:dev/split-jars
Closed

[packaging] split native libs from the main jar#834
rmannibucau wants to merge 1 commit into
duckdb:mainfrom
rmannibucau:dev/split-jars

Conversation

@rmannibucau

@rmannibucau rmannibucau commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

it makes it fatty and bothering when creating distribution or docker images, also makes creating fatjars/shades quite slow for nothing

design: duckdb_jdbc = empty jar (for compatibility of pom but the jar is technically useless) with as dependency (pom) java lib + 4 historical native lib, duckdb_jdbc_java = java code, then one jar per native lib

note: review of release process is welcomed, I'm not 100% sure what I did is aligned on the practise

…ty and bothering when creating distribution or docker images, also makes creating fatjars/shades quite slow for nothing
@staticlibs

staticlibs commented Aug 24, 2026

Copy link
Copy Markdown
Member

Hi, thanks for the PR! Just the description is a little bit confusing - can you clarify what is it trying to achieve?

This part:

split native libs from the main jar

should be already covered by the existence of a -nolib jar (<classifier>nolib</classifier> in Maven).

@rmannibucau

rmannibucau commented Aug 24, 2026

Copy link
Copy Markdown
Contributor Author

totally missed the nolib jar so closing thanks!

edit: @staticlibs .so libs contain the java code? this kind of work but is ackward and require more audit work (think CVE linked to a jar, now it is linked to 7 jars or alike)

@staticlibs

Copy link
Copy Markdown
Member

.so libs contain the java code?

M, no, libduckdb_java.so_<platform_name> is an unmodified renamed native library (.so/.dylib/.dll) that contains a JNI wrapper inside it, with JNI symbols for JDBC and C API symbols for extensions - list.

There is a number of JAR files, -nolib one contains only the Java part of the JDBC driver and supports untrivial logic to load the native lib (see #450).

The "universal" (Linux x86_64, Windows x86_64 and Mac universal) JAR and platform specific JARs contain libduckdb_java.so_<platform_name> native binaries inside them. They are provided for convenience, mainly for Maven and BI users when a single dependency or a single JAR download are enough to start using DuckDB from Java without knowing anything about the native lib.

It is expected, that any non-trivial re-packaging process only consumes the -nolib JAR and handles native library as needed (for example, dropping it in FS next to -nolib JAR).

this kind of work but is ackward and require more audit work (think CVE linked to a jar, now it is linked to 7 jars or alike)

M, I would think a CVE would apply to the whole JDBC release (all artifacts in a Maven version) or even to all DuckDB clients (if CVE is inside the engine).

@rmannibucau

Copy link
Copy Markdown
Contributor Author

@staticlibs I was more referencing that:

unzip -l target/dependency/duckdb_jdbc-1.5.5.1-linux_amd64.jar 
Archive:  target/dependency/duckdb_jdbc-1.5.5.1-linux_amd64.jar
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2026-08-03 21:35   META-INF/
      681  2026-08-03 21:35   META-INF/MANIFEST.MF
       24  2026-08-03 21:35   META-INF/services/java.sql.Driver
     2436  2026-08-03 21:35   org/duckdb/DuckDBAppender$1.class
     8671  2026-08-03 21:35   org/duckdb/DuckDBAppender$Column.class
   ......
     1132  2026-08-03 21:35   org/duckdb/user/DuckDBUserStruct.class
 60780968  2026-08-03 21:49   libduckdb_java.so_linux_amd64

the common pactise is that classified jar only contains the .so the unclassified (or nolib is ok there) will load.
now we have the ~same code duplicated in all jars it seems

I would think a CVE would apply to the whole JDBC release

this is the case due to current packaging but downgrading the nolib jar could be a workaround (often) if only java code is affected - it is what I had in mind when I wrote previous message.

@staticlibs

Copy link
Copy Markdown
Member

Thanks for clarification! Yes, the point is clear now, the Java part is indeed included with platform JARs. I cannot immediately say whether it is better in include it or to require 2 dependencies. Current state may be more convenient to users who need to include only a single dependency. If the point deemed important - please feel free to open an issue to track it (right now it is not clear to me which variant is better).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants