diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index fd60cebe40..192845752a 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -97,6 +97,7 @@ These structural facts shape every module. Day-to-day build *commands* live in ` - **SDK levels** (`build-logic/.../build/config/BuildConfig.kt`): `COMPILE_SDK=36`, `MIN_SDK=28`, `TARGET_SDK=28`. **`TARGET_SDK` is deliberately pinned at 28:** higher targets enforce W^X (write-xor-execute), which blocks executing code from app-writable files. That is fatal for an on-device IDE that compiles and runs code (Gradle, `javac`, Termux binaries), so it is a hard requirement, not tech debt. `MIN_SDK_FOR_APPS_BUILT_WITH_COGO=16` is the floor for the apps a *user* builds with CoGo — distinct from CoGo's own `MIN_SDK`. - **Native asset bundling.** The on-device LLM (`llama-impl`) ships as a per-flavor native AAR, wired through the root `build.gradle.kts` (`bundleLlamaV8Assets` / `assembleV8Assets`, …); prebuilt per-flavor assets live under `assets/release/v7/` and `assets/release/v8/`. - **Native lib compression** (ADFA-2306, ADFA-4729). The app manifest hard-codes `android:extractNativeLibs="true"` (required: the installer must materialize libs in `nativeLibraryDir`, e.g. `libshizuku.so` is an executable the adb shell runs from there). That attribute overrides the `jniLibs.useLegacyPackaging` DSL, so AGP packages `lib//*.so` deflate-compressed in **every** APK — ~5.9 MB smaller (`libtree-sitter-kotlin.so` alone is 4.18 MB → 339 kB). The trap is the `recompressApk` post-step (release always, debug in CI only): its no-compress lists in `app/build.gradle.kts` must NOT contain `"so"`, or it silently re-stores the libs and undoes the saving — which is what ADFA-2306 fixed for release and ADFA-4729 for CI debug. Locally built debug APKs (including the e2e farm's) never run that step and were always fine. +- **Quick Build shares the Gradle distribution's toolchain** (ADFA-4931). `:app`'s `quickBuildDaemonZip` leaves the Kotlin compiler and eight other jars out of the daemon zip, because the on-device Gradle distribution already carries them; `QuickBuildArtifactStager` links them in at provision time. That makes the daemon's Kotlin version the distribution's, so `:quickbuild:daemon` takes its Kotlin artifacts at `org.adfa.constants.KOTLIN_VERSION`, and `:app:quickBuildDistJarCheck` fails the build when the distribution it packages lacks a linked jar. See [ADR 0016](docs/adr/0016-quick-build-compiles-outside-gradle.md). - **`app` package layout is by concern, not feature:** `activities`, `fragments`, `services`, `di`, `agent`, `viewmodel(s)`, `repositories`, `roomData`, `localWebServer`, `preferences`, `ui` (Compose screens live under `ui/compose`), `templates/manager` (the Manager screen's `.cgt`-parsing data layer, with direct filesystem access to `Environment.TEMPLATES_DIR` — distinct from the plugin-facing `IdeTemplateService` in `plugin-api`/`plugin-manager`), `utils`, …. ## Sibling Repositories & Asset Provenance diff --git a/app/build.gradle.kts b/app/build.gradle.kts index d0e1bfc9b5..1977240482 100644 --- a/app/build.gradle.kts +++ b/app/build.gradle.kts @@ -1,12 +1,17 @@ @file:Suppress("UnstableApiUsage") +import com.aayushatharva.brotli4j.Brotli4jLoader +import com.aayushatharva.brotli4j.decoder.BrotliInputStream import com.itsaky.androidide.build.config.BuildConfig import com.itsaky.androidide.desugaring.utils.JavaIOReplacements.applyJavaIOReplacements import com.itsaky.androidide.plugins.AndroidIDEAssetsPlugin +import com.itsaky.androidide.plugins.conf.hasBundledAssets import com.itsaky.androidide.plugins.tasks.AddFileToAssetsTask import org.adfa.constants.GRADLE_API_NAME_JAR_BR import org.adfa.constants.GRADLE_API_NAME_JAR_ZIP import org.adfa.constants.GRADLE_DISTRIBUTION_ARCHIVE_NAME +import org.adfa.constants.GRADLE_DISTRIBUTION_NAME +import org.adfa.constants.KOTLIN_VERSION import org.gradle.nativeplatform.platform.internal.DefaultNativePlatform import org.json.JSONObject import java.io.BufferedOutputStream @@ -27,6 +32,7 @@ import java.util.zip.CRC32 import java.util.zip.Deflater import java.util.zip.ZipEntry import java.util.zip.ZipFile +import java.util.zip.ZipInputStream import java.util.zip.ZipOutputStream plugins { @@ -507,16 +513,166 @@ dependencies { evaluationDependsOn(":quickbuild:runtime") evaluationDependsOn(":quickbuild:daemon") +// ADFA-4931: jars the daemon loads from the on-device Gradle distribution instead of shipping its +// own copy, since `gradle-dists/` and `quickbuild/` both sit under . +// quickBuildDaemonZip leaves these out and QuickBuildArtifactStager links the distribution's copy +// in under exactly these names, which is what the daemon jar's manifest Class-Path expects. +val quickBuildDistLinkedJars = + listOf( + "kotlin-compiler-embeddable-$KOTLIN_VERSION.jar", + "kotlin-stdlib-$KOTLIN_VERSION.jar", + "kotlin-build-tools-impl-$KOTLIN_VERSION.jar", + "kotlin-daemon-embeddable-$KOTLIN_VERSION.jar", + "kotlin-script-runtime-$KOTLIN_VERSION.jar", + "kotlin-reflect-$KOTLIN_VERSION.jar", + // These three carry their own version rather than KOTLIN_VERSION, pinned in + // libs.versions.toml to what the distribution ships so they can be linked from there. + "kotlinx-coroutines-core-jvm-${libs.kotlinx.coroutines.core.jvm.quickBuildDaemon.get().version}.jar", + "gson-${libs.gson.quickBuildDaemon.get().version}.jar", + "asm-${libs.ow2.asm.get().version}.jar", + ) + +// A broken link surfaces on device as a NoClassDefFoundError partway into the user's first +// compile, far from the change that caused it, so check both ends at build time: the daemon must +// resolve each name, and the distribution this build packages must carry each one. +val quickBuildReleaseDistribution = + rootProject.file("assets/release/common/data/common/$GRADLE_DISTRIBUTION_ARCHIVE_NAME.br") + +val quickBuildDistJarCheck = + tasks.register("quickBuildDistJarCheck") { + // Without the ordering, a build that downloads a new distribution can check the old one. + mustRunAfter("assetsDownloadDebug", "assetsDownloadRelease") + val expected = quickBuildDistLinkedJars + val distName = GRADLE_DISTRIBUTION_NAME + val archiveName = GRADLE_DISTRIBUTION_ARCHIVE_NAME + val debugArchive = rootProject.file("assets/$archiveName") + val releaseArchive = quickBuildReleaseDistribution + val classpath = + files( + rootProject + .project(":quickbuild:daemon") + .configurations + .named("runtimeClasspath"), + ) + inputs.files(classpath) + inputs + .files(debugArchive) + .withPropertyName("debugDistribution") + .optional(true) + inputs + .files(releaseArchive) + .withPropertyName("releaseDistribution") + .optional(true) + doLast { + val resolved = classpath.files.associateBy { it.name } + expected.sorted().forEach { name -> + if (name !in resolved) { + throw GradleException( + "$name is linked from the Gradle distribution but is not on the daemon's " + + "runtime classpath, so quickBuildDaemonZip's exclusion of that name " + + "matches nothing and the zip carries a copy the stager then links over. " + + "Point quickBuildDistLinkedJars at the name the daemon now resolves. " + + "Resolved: ${resolved.keys.sorted()}", + ) + } + } + + /** Fails unless every linked jar sits in the distribution's lib/ inside [archive]. */ + fun requireLinkedJars( + entries: Set, + archive: File, + ) { + expected.forEach { name -> + val entryName = "$distName/lib/$name" + if (entryName !in entries) { + throw GradleException( + "$entryName is missing from ${archive.name}. quickBuildDaemonZip " + + "excludes $name on the promise that the distribution supplies it, so " + + "the daemon would start with no copy at all. Either drop it from " + + "quickBuildDistLinkedJars and let the zip carry it, or correct the name.", + ) + } + } + logger.lifecycle( + "quickBuildDistJarCheck: found ${expected.size} of ${expected.size} linked jars " + + "in $distName/lib of ${archive.name}", + ) + } + + if (debugArchive.isFile) { + val entries = + ZipFile(debugArchive).use { zip -> + zip + .entries() + .asSequence() + .map { it.name } + .toSet() + } + requireLinkedJars(entries, debugArchive) + } + if (releaseArchive.isFile) { + Brotli4jLoader.ensureAvailability() + val entries = mutableSetOf() + ZipInputStream(BrotliInputStream(releaseArchive.inputStream().buffered())).use { zip -> + generateSequence { zip.nextEntry }.forEach { entries += it.name } + } + requireLinkedJars(entries, releaseArchive) + } + if (!debugArchive.isFile && !releaseArchive.isFile) { + logger.lifecycle( + "quickBuildDistJarCheck: NOT PACKAGED - 0 of ${expected.size} linked jars looked " + + "up. No Gradle distribution archive is on disk; a debug APK gets it from the " + + "assets zip, which ./gradlew :app:assetsDownloadDebug fetches to check.", + ) + } + } + } + +// A variant that bundles the distribution must not package the daemon zip until the check +// above has read it. Wired per variant below rather than inferred from task names. +val quickBuildBundledDistGate = + tasks.register("quickBuildBundledDistGate") { + dependsOn(quickBuildDistJarCheck) + val releaseArchive = quickBuildReleaseDistribution + doLast { + if (!releaseArchive.isFile) { + throw GradleException( + "${releaseArchive.absolutePath} is absent, so quickBuildDistJarCheck cannot read " + + "the Gradle distribution this variant packages. Run " + + ":app:assetsDownloadRelease first.", + ) + } + } + } + +// The names the stager links, travelling with the zip so the device side reads the build's +// list rather than repeating it. +val quickBuildDistLinkedJarList = + tasks.register("quickBuildDistLinkedJarList") { + val names = quickBuildDistLinkedJars.sorted() + val listFile = layout.buildDirectory.file("intermediates/quickbuild/dist-linked-jars.txt") + inputs.property("names", names) + outputs.file(listFile) + doLast { + listFile.get().asFile.writeText(names.joinToString("\n", postfix = "\n")) + } + } + val quickBuildDaemonZip = tasks.register("quickBuildDaemonZip") { archiveFileName.set("quickbuild-daemon.zip") destinationDirectory.set(layout.buildDirectory.dir("intermediates/quickbuild")) + dependsOn(quickBuildDistJarCheck) val daemonProject = rootProject.project(":quickbuild:daemon") dependsOn(daemonProject.tasks.named("daemonJar")) from(daemonProject.tasks.named("daemonJar")) // The daemon jar's manifest Class-Path names these by file name; they must sit // next to the jar after extraction. - from(daemonProject.configurations.named("runtimeClasspath")) + // ADFA-4931: except the ones the distribution already carries; the stager links those + // in at provision time. Scoped to this spec so it cannot filter the other from()s. + val linked = quickBuildDistLinkedJars.toSet() + from(daemonProject.configurations.named("runtimeClasspath")) { exclude { it.name in linked } } + from(quickBuildDistLinkedJarList) // Compose compiler plugin, version-matched to the daemon's compiler; the stable // name is the contract EnvironmentQuickBuildPaths.composeCompilerPlugin reads. from(daemonProject.configurations.named("composeCompilerPlugin")) { @@ -551,9 +707,14 @@ androidComponents.onVariants { variant -> AddFileToAssetsTask::outputDirectory, ) + // The same test that adds assets/release, and the distribution in it, to this variant. + val bundlesDistribution = hasBundledAssets(variant) val copyDaemonZip = tasks.register("copy${variantName}QuickBuildDaemonZip") { dependsOn(quickBuildDaemonZip) + if (bundlesDistribution) { + dependsOn(quickBuildBundledDistGate) + } inputFile.set(quickBuildDaemonZip.flatMap { it.archiveFile }) baseAssetsPath.set("data/common") } diff --git a/app/src/main/java/com/itsaky/androidide/quickbuild/QuickBuildArtifactStager.kt b/app/src/main/java/com/itsaky/androidide/quickbuild/QuickBuildArtifactStager.kt index a42fe8ba74..35b84b8e0a 100644 --- a/app/src/main/java/com/itsaky/androidide/quickbuild/QuickBuildArtifactStager.kt +++ b/app/src/main/java/com/itsaky/androidide/quickbuild/QuickBuildArtifactStager.kt @@ -3,27 +3,22 @@ package com.itsaky.androidide.quickbuild import android.content.Context import androidx.core.content.pm.PackageInfoCompat import com.itsaky.androidide.utils.Environment +import org.adfa.constants.GRADLE_DISTRIBUTION_NAME import org.slf4j.LoggerFactory import java.io.File import java.io.FileNotFoundException import java.io.IOException import java.io.InputStream +import java.nio.file.Files +import java.nio.file.Path import java.util.zip.ZipInputStream /** - * Extracts the quick-build artifacts from APK assets to `/quickbuild/` - the - * runtime AAR, and the daemon zip unpacked into `daemon/` (the daemon jar plus the runtime - * classpath its manifest Class-Path names). + * Stages the quick-build runtime AAR and daemon under `/quickbuild/`, linking + * the jars [LINKED_JARS_LIST] names in from the on-device Gradle distribution. * - * The AAR is copied on every call. The daemon directory is wiped and re-extracted only when - * the installed APK changed: a stamp file, written last so a crash mid-extract leaves none, - * records the package's versionCode and lastUpdateTime - not a version constant, which would - * serve a stale bundle when content changes without a bump; any install, an unchanged-version - * reinstall included, moves lastUpdateTime. That saves a 62 MB extraction per provision and - * rebaseline. It also keeps the one wipe path away from a live compile daemon, which loads - * the jars under `daemon/` lazily: the wipe runs only after an APK update, which force-stops - * the app and its child processes. No known path stages while a daemon is alive anyway - a - * rebaseline shuts it down before the Gradle build runs - so this is a guard, not a fix. + * The daemon is re-staged only when the APK was reinstalled or a linked jar stopped resolving, + * which skips a 1.9 MB extraction per provision and keeps the wipe away from a live daemon. */ object QuickBuildArtifactStager { private val log = LoggerFactory.getLogger("QB-ArtifactStager") @@ -34,6 +29,9 @@ object QuickBuildArtifactStager { /** Written last, after a complete extraction, so a crash mid-extract leaves no stamp. */ internal const val DAEMON_STAMP_FILE = ".staged-for-install" + /** Written into the daemon zip by :app's `quickBuildDistLinkedJarList`. */ + internal const val LINKED_JARS_LIST = "dist-linked-jars.txt" + /** @throws IOException when an asset is missing or extraction fails. */ @Throws(IOException::class) fun stage( @@ -46,7 +44,9 @@ object QuickBuildArtifactStager { } } - /** Identity of the installed APK; see the class doc for why lastUpdateTime and not a constant. */ + /** + * Identity of the installed APK. lastUpdateTime, unlike a version constant, moves on every reinstall. + */ private fun installStamp(context: Context): String { val info = context.packageManager.getPackageInfo(context.packageName, 0) return "${PackageInfoCompat.getLongVersionCode(info)}:${info.lastUpdateTime}" @@ -65,9 +65,10 @@ object QuickBuildArtifactStager { /** * Wipes and re-extracts [daemonDir] unless it already holds a complete extraction for - * [installStamp] - the stamp file matches and [daemonJar] is present. Internal so the JVM - * test can watch the skip, and the wipe, without an Android [Context]. + * [installStamp]: the stamp matches, [daemonJar] is present and every linked jar resolves. + * Internal so the JVM test can watch the skip, and the wipe, without an Android [Context]. * + * @param gradleDists exists so tests can point at a fake distribution. * @return whether an extraction ran. */ @Throws(IOException::class) @@ -75,10 +76,11 @@ object QuickBuildArtifactStager { installStamp: String, daemonDir: File, daemonJar: File, + gradleDists: File = Environment.GRADLE_DISTS, openZip: () -> InputStream, ): Boolean { val stamp = File(daemonDir, DAEMON_STAMP_FILE) - if (daemonJar.isFile && stamp.isFile && stamp.readText() == installStamp) { + if (daemonJar.isFile && stamp.isFile && stamp.readText() == installStamp && linkedJarsResolve(daemonDir)) { log.info("Daemon already staged for this install at {}", daemonDir) return false } @@ -88,6 +90,9 @@ object QuickBuildArtifactStager { Environment.mkdirIfNotExists(daemonDir) val count = extractDaemonZip(openZip(), daemonDir) + // Must run before the stamp is written, or a failed link would be stamped as done and + // never retried. + linkDistJars(daemonDir, gradleDists) stamp.writeText(installStamp) log.info("Staged {} daemon files into {}", count, daemonDir) return true @@ -132,4 +137,92 @@ object QuickBuildArtifactStager { return count } } + + /** + * Links the jars the APK leaves out into the daemon dir, where its manifest Class-Path + * expects them. + * + * @param gradleDists exists so tests can point at a fake distribution. + * @param createLink exists so tests can force the copy fallback. + * @throws FileNotFoundException if a jar or the list is missing, which fails loudly here + * rather than as a NoClassDefFoundError during the user's first compile. + */ + @Throws(IOException::class) + internal fun linkDistJars( + daemonDir: File, + gradleDists: File = Environment.GRADLE_DISTS, + createLink: (Path, Path) -> Unit = { link, existing -> Files.createSymbolicLink(link, existing) }, + ) { + val distLib = File(File(gradleDists, GRADLE_DISTRIBUTION_NAME), "lib") + for (jarName in readLinkedJarNames(daemonDir)) { + val source = File(distLib, jarName) + if (!source.isFile) { + throw FileNotFoundException( + "$jarName missing from the on-device Gradle distribution: $source. " + + "Quick Build loads it from there rather than from the APK.", + ) + } + + val target = File(daemonDir, jarName) + // Clear the path first: a link an earlier staging left would make the copy fallback + // follow it and truncate the distribution's own jar. + Files.deleteIfExists(target.toPath()) + try { + createLink(target.toPath(), source.toPath()) + } catch (e: IOException) { + copyInstead(source, target, e) + } catch (e: UnsupportedOperationException) { + copyInstead(source, target, e) + } + } + } + + /** Copies [source] to [target] when the filesystem refused a link, so the bytes still arrive. */ + private fun copyInstead( + source: File, + target: File, + cause: Exception, + ) { + log.warn("Symlink {} -> {} failed ({}), copying", target, source, cause.toString()) + source.inputStream().use { input -> target.outputStream().use { input.copyTo(it) } } + } + + /** + * Whether every linked jar in [daemonDir] still reaches a file. The stamp alone cannot say, + * since the assets installers delete and re-extract the distribution the links point into. + */ + private fun linkedJarsResolve(daemonDir: File): Boolean { + // An unreadable list re-stages too, and the re-stage's own read reports why. + val names = + try { + readLinkedJarNames(daemonDir) + } catch (e: IOException) { + return false + } + return names.all { File(daemonDir, it).isFile } + } + + /** The jar names the build left out of the zip for the distribution to supply. */ + private fun readLinkedJarNames(daemonDir: File): List { + val list = File(daemonDir, LINKED_JARS_LIST) + if (!list.isFile) { + throw FileNotFoundException( + "$LINKED_JARS_LIST is missing from $daemonDir, so the build never wrote it. It " + + "lists the jars Quick Build links from the Gradle distribution.", + ) + } + val names = list.readLines().map(String::trim).filter(String::isNotEmpty) + if (names.isEmpty()) { + throw FileNotFoundException( + "$LINKED_JARS_LIST at $list names no jars. It lists the jars Quick Build links " + + "from the Gradle distribution rather than shipping in the APK.", + ) + } + // Each name is used as a path on both sides of the link, so the same boundary + // extractDaemonZip guards applies here: anything but a bare file name escapes. + names.firstOrNull { File(it).name != it }?.let { name -> + throw IOException("Refusing linked jar name that is not a bare file name: $name") + } + return names + } } diff --git a/app/src/test/java/com/itsaky/androidide/quickbuild/QuickBuildArtifactStagerDistLinkTest.kt b/app/src/test/java/com/itsaky/androidide/quickbuild/QuickBuildArtifactStagerDistLinkTest.kt new file mode 100644 index 0000000000..a885b75340 --- /dev/null +++ b/app/src/test/java/com/itsaky/androidide/quickbuild/QuickBuildArtifactStagerDistLinkTest.kt @@ -0,0 +1,224 @@ +package com.itsaky.androidide.quickbuild + +import com.google.common.truth.Truth.assertThat +import org.adfa.constants.GRADLE_DISTRIBUTION_NAME +import org.adfa.constants.KOTLIN_VERSION +import org.junit.Rule +import org.junit.Test +import org.junit.rules.TemporaryFolder +import java.io.File +import java.io.FileNotFoundException +import java.io.IOException +import java.nio.file.Files + +/** + * The daemon now gets several jars from the Gradle distribution instead of the APK, so a link + * step that silently does nothing leaves its classpath naming files that are not there. + */ +class QuickBuildArtifactStagerDistLinkTest { + @get:Rule + val temp = TemporaryFolder() + + /** Stand-ins for the build-generated list, with two entries so a partial link step shows up. */ + private val jarNames = + listOf( + "kotlin-compiler-embeddable-$KOTLIN_VERSION.jar", + "kotlin-stdlib-$KOTLIN_VERSION.jar", + ) + + /** A staged daemon dir whose list names [names], as the build's list task writes it. */ + private fun daemonDirListing(names: List = jarNames): File { + val daemonDir = temp.newFolder("daemon") + File(daemonDir, QuickBuildArtifactStager.LINKED_JARS_LIST) + .writeText(names.joinToString("\n", postfix = "\n")) + return daemonDir + } + + /** A fake `gradle-dists/` holding one distribution whose lib/ carries [names]. */ + private fun gradleDistsWith(names: List = jarNames): File { + val dists = temp.newFolder("gradle-dists") + val lib = File(dists, "$GRADLE_DISTRIBUTION_NAME/lib") + lib.mkdirs() + names.forEach { File(lib, it).writeText("$it-bytes") } + return dists + } + + private fun stagedNames(daemonDir: File) = daemonDir.list()!!.toList() - QuickBuildArtifactStager.LINKED_JARS_LIST + + @Test + fun `every listed jar is readable from the daemon dir under the name the manifest expects`() { + val daemonDir = daemonDirListing() + + QuickBuildArtifactStager.linkDistJars(daemonDir, gradleDistsWith()) + + // Asserts content, not existence, because a symlink and a copy are both acceptable. + jarNames.forEach { name -> + assertThat(File(daemonDir, name).readText()).isEqualTo("$name-bytes") + } + } + + @Test + fun `nothing beyond the listed jars is staged`() { + val daemonDir = daemonDirListing() + + // Linking the whole lib/ would bury the daemon's classpath in Gradle's own jars. + QuickBuildArtifactStager.linkDistJars( + daemonDir, + gradleDistsWith(jarNames + "gradle-core-api-9.6.1.jar"), + ) + + assertThat(stagedNames(daemonDir)).containsExactlyElementsIn(jarNames) + } + + @Test + fun `a link rather than a copy is preferred so the bytes are not duplicated on device`() { + val daemonDir = daemonDirListing() + + QuickBuildArtifactStager.linkDistJars(daemonDir, gradleDistsWith()) + + // Where links work, we must not be paying for the bytes twice. + jarNames.forEach { name -> + assertThat(Files.isSymbolicLink(File(daemonDir, name).toPath())).isTrue() + } + } + + @Test + fun `a jar that cannot be linked is copied instead, so the daemon still has the bytes`() { + val daemonDir = daemonDirListing() + val refused = jarNames.first() + + // Stands in for a filesystem that has no symlinks, which throws rather than returning. + QuickBuildArtifactStager.linkDistJars(daemonDir, gradleDistsWith()) { link, existing -> + if (link.fileName.toString() == refused) throw UnsupportedOperationException("no symlinks here") + Files.createSymbolicLink(link, existing) + } + + assertThat(Files.isSymbolicLink(File(daemonDir, refused).toPath())).isFalse() + assertThat(File(daemonDir, refused).readText()).isEqualTo("$refused-bytes") + // The fallback is per jar, so one refused link must not cost the rest theirs. + assertThat(Files.isSymbolicLink(File(daemonDir, jarNames.last()).toPath())).isTrue() + assertThat(File(daemonDir, jarNames.last()).readText()).isEqualTo("${jarNames.last()}-bytes") + } + + @Test + fun `a link refused with an IO error is copied too, not just an unsupported one`() { + val daemonDir = daemonDirListing() + val refused = jarNames.first() + + // Stands in for a filesystem that supports links but refuses this one, such as a + // read-only mount or a denied permission. + QuickBuildArtifactStager.linkDistJars(daemonDir, gradleDistsWith()) { link, existing -> + if (link.fileName.toString() == refused) throw IOException("permission denied") + Files.createSymbolicLink(link, existing) + } + + assertThat(Files.isSymbolicLink(File(daemonDir, refused).toPath())).isFalse() + assertThat(File(daemonDir, refused).readText()).isEqualTo("$refused-bytes") + } + + @Test + fun `a listed name that is a path rather than a file name is refused`() { + // Resolves to a jar that is really there, so only the guard stops the link being made + // one directory above the daemon dir. + val escaping = "../lib/${jarNames.first()}" + val daemonDir = daemonDirListing(names = listOf(escaping)) + + // The list is extracted from the daemon zip, which extractDaemonZip already refuses to + // let escape; a name used as a path deserves the same guard. + val thrown = + runCatching { QuickBuildArtifactStager.linkDistJars(daemonDir, gradleDistsWith()) } + .exceptionOrNull() + + assertThat(thrown).isInstanceOf(IOException::class.java) + assertThat(thrown!!).hasMessageThat().contains("bare file name") + assertThat(thrown).hasMessageThat().contains(escaping) + assertThat(stagedNames(daemonDir)).isEmpty() + } + + @Test + fun `staging twice leaves the distribution's own jars intact`() { + val daemonDir = daemonDirListing() + val dists = gradleDistsWith() + QuickBuildArtifactStager.linkDistJars(daemonDir, dists) + + // Linking onto the previous run's links: the copy fallback would otherwise open them for + // write, truncating the distribution itself and breaking every Gradle build on the device. + QuickBuildArtifactStager.linkDistJars(daemonDir, dists) + + jarNames.forEach { name -> + val inDistribution = File(dists, "$GRADLE_DISTRIBUTION_NAME/lib/$name") + assertThat(inDistribution.readText()).isEqualTo("$name-bytes") + assertThat(File(daemonDir, name).readText()).isEqualTo("$name-bytes") + } + } + + @Test + fun `missing distribution fails loudly and names the path a human has to fix`() { + val daemonDir = daemonDirListing() + val emptyDists = temp.newFolder("gradle-dists") + + val thrown = + runCatching { QuickBuildArtifactStager.linkDistJars(daemonDir, emptyDists) } + .exceptionOrNull() + + // Failing here is deliberate, because the alternative surfaces mid-compile instead. + assertThat(thrown).isInstanceOf(FileNotFoundException::class.java) + assertThat(thrown!!).hasMessageThat().contains(GRADLE_DISTRIBUTION_NAME) + assertThat(thrown).hasMessageThat().contains(jarNames.first()) + assertThat(stagedNames(daemonDir)).isEmpty() + } + + @Test + fun `one listed jar missing from the distribution fails the whole staging`() { + val daemonDir = daemonDirListing() + + // The throw is what protects the caller here: the stamp is left unwritten, so the next + // staging wipes the jars this one already linked. + val thrown = + runCatching { + QuickBuildArtifactStager.linkDistJars(daemonDir, gradleDistsWith(jarNames.take(1))) + }.exceptionOrNull() + + assertThat(thrown).isInstanceOf(FileNotFoundException::class.java) + assertThat(thrown!!).hasMessageThat().contains(jarNames.last()) + } + + @Test + fun `a distribution directory for a different version does not satisfy the link`() { + val daemonDir = daemonDirListing() + val dists = temp.newFolder("gradle-dists") + File(dists, "gradle-0.0.1/lib").mkdirs() + jarNames.forEach { File(dists, "gradle-0.0.1/lib/$it").writeText("wrong distribution") } + + // The jars must come from the distribution this build bundles, not a neighbouring one. + val thrown = + runCatching { QuickBuildArtifactStager.linkDistJars(daemonDir, dists) } + .exceptionOrNull() + + assertThat(thrown).isInstanceOf(FileNotFoundException::class.java) + } + + @Test + fun `a daemon zip built without the list fails rather than linking nothing`() { + val daemonDir = temp.newFolder("daemon") + + // Without the list there is no way to tell a complete build from an empty one. + val thrown = + runCatching { QuickBuildArtifactStager.linkDistJars(daemonDir, gradleDistsWith()) } + .exceptionOrNull() + + assertThat(thrown).isInstanceOf(FileNotFoundException::class.java) + assertThat(thrown!!).hasMessageThat().contains(QuickBuildArtifactStager.LINKED_JARS_LIST) + } + + @Test + fun `an empty list is treated as a broken build, not as nothing to do`() { + val daemonDir = daemonDirListing(names = emptyList()) + + val thrown = + runCatching { QuickBuildArtifactStager.linkDistJars(daemonDir, gradleDistsWith()) } + .exceptionOrNull() + + assertThat(thrown).isInstanceOf(FileNotFoundException::class.java) + } +} diff --git a/app/src/test/java/com/itsaky/androidide/quickbuild/QuickBuildArtifactStagerTest.kt b/app/src/test/java/com/itsaky/androidide/quickbuild/QuickBuildArtifactStagerTest.kt index 5398da20cf..78518f4b5d 100644 --- a/app/src/test/java/com/itsaky/androidide/quickbuild/QuickBuildArtifactStagerTest.kt +++ b/app/src/test/java/com/itsaky/androidide/quickbuild/QuickBuildArtifactStagerTest.kt @@ -1,6 +1,8 @@ package com.itsaky.androidide.quickbuild import com.google.common.truth.Truth.assertThat +import org.adfa.constants.GRADLE_DISTRIBUTION_NAME +import org.adfa.constants.KOTLIN_VERSION import org.junit.Rule import org.junit.Test import org.junit.rules.TemporaryFolder @@ -13,12 +15,8 @@ import java.util.zip.ZipEntry import java.util.zip.ZipOutputStream /** - * The zip-slip guard is a security control: the daemon zip is a bundled asset today, but the - * extraction must never write outside the daemon dir no matter what the archive says. These - * tests watch the guard go red - a `../` entry must throw BEFORE any byte lands outside. - * - * The staging tests pin the other invariant: an already-staged daemon directory is left alone - * for the same install, because a rebaseline stages while the compile daemon is running off it. + * Covers two invariants: extraction never writes outside the daemon dir, and an already-staged + * directory is left alone for the same install. */ class QuickBuildArtifactStagerTest { @get:Rule @@ -105,10 +103,23 @@ class QuickBuildArtifactStagerTest { assertThat(thrown).isInstanceOf(FileNotFoundException::class.java) } + /** The one jar [daemonZip]'s linked-jars list names. */ + private val linkedJar = "kotlin-compiler-embeddable-$KOTLIN_VERSION.jar" + + /** A fake `gradle-dists/` for the link to point at, since the real one is unset off-device. */ + private val gradleDists: File by lazy { + val dists = tmp.newFolder("gradle-dists") + val lib = File(dists, "$GRADLE_DISTRIBUTION_NAME/lib") + assertThat(lib.mkdirs()).isTrue() + File(lib, linkedJar).writeText("linked-jar-bytes") + dists + } + private fun daemonZip(): ByteArrayInputStream = zipOf( "quickbuild-daemon.jar" to byteArrayOf(1, 2, 3), "lib/runtime.jar" to byteArrayOf(4, 5), + QuickBuildArtifactStager.LINKED_JARS_LIST to "$linkedJar\n".toByteArray(), ) @Test @@ -118,7 +129,7 @@ class QuickBuildArtifactStagerTest { var opened = 0 val ran = - QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar) { + QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar, gradleDists) { opened++ daemonZip() } @@ -129,17 +140,32 @@ class QuickBuildArtifactStagerTest { assertThat(File(daemonDir, QuickBuildArtifactStager.DAEMON_STAMP_FILE).readText()).isEqualTo("7:1000") } + /** + * Checks that staging actually calls the link step, which a rebase once left unreachable + * behind a `return` while the direct tests stayed green. + */ + @Test + fun `staging links the distribution's jars into the daemon dir`() { + val daemonDir = File(tmp.newFolder("home"), "daemon") + val jar = File(daemonDir, "quickbuild-daemon.jar") + + QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar, gradleDists) { daemonZip() } + + // Asserts content, not existence, because a symlink and a copy are both acceptable. + assertThat(File(daemonDir, linkedJar).readText()).isEqualTo("linked-jar-bytes") + } + @Test fun `a second stage for the same install leaves the directory untouched`() { val daemonDir = File(tmp.newFolder("home"), "daemon") val jar = File(daemonDir, "quickbuild-daemon.jar") - QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar) { daemonZip() } + QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar, gradleDists) { daemonZip() } // A file the running daemon could depend on: gone means the directory was wiped. val planted = File(daemonDir, "opened-by-a-live-daemon.jar").apply { writeBytes(byteArrayOf(9)) } var opened = 0 val ran = - QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar) { + QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar, gradleDists) { opened++ daemonZip() } @@ -153,10 +179,10 @@ class QuickBuildArtifactStagerTest { fun `a new install re-stages from scratch`() { val daemonDir = File(tmp.newFolder("home"), "daemon") val jar = File(daemonDir, "quickbuild-daemon.jar") - QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar) { daemonZip() } + QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar, gradleDists) { daemonZip() } val stale = File(daemonDir, "from-the-old-install.jar").apply { writeBytes(byteArrayOf(9)) } - val ran = QuickBuildArtifactStager.stageDaemonIfNeeded("7:2000", daemonDir, jar) { daemonZip() } + val ran = QuickBuildArtifactStager.stageDaemonIfNeeded("7:2000", daemonDir, jar, gradleDists) { daemonZip() } assertThat(ran).isTrue() assertThat(stale.exists()).isFalse() @@ -167,15 +193,77 @@ class QuickBuildArtifactStagerTest { fun `a matching stamp without the daemon jar re-stages`() { val daemonDir = File(tmp.newFolder("home"), "daemon") val jar = File(daemonDir, "quickbuild-daemon.jar") - QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar) { daemonZip() } + QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar, gradleDists) { daemonZip() } assertThat(jar.delete()).isTrue() - val ran = QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar) { daemonZip() } + val ran = QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar, gradleDists) { daemonZip() } assertThat(ran).isTrue() assertThat(jar.exists()).isTrue() } + @Test + fun `a matching stamp whose linked jar is gone re-stages`() { + val daemonDir = File(tmp.newFolder("home"), "daemon") + val jar = File(daemonDir, "quickbuild-daemon.jar") + QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar, gradleDists) { daemonZip() } + assertThat(File(daemonDir, linkedJar).delete()).isTrue() + + val ran = QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar, gradleDists) { daemonZip() } + + assertThat(ran).isTrue() + assertThat(File(daemonDir, linkedJar).readText()).isEqualTo("linked-jar-bytes") + } + + @Test + fun `a matching stamp whose linked-jar list is gone re-stages`() { + val daemonDir = File(tmp.newFolder("home"), "daemon") + val jar = File(daemonDir, "quickbuild-daemon.jar") + QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar, gradleDists) { daemonZip() } + assertThat(File(daemonDir, QuickBuildArtifactStager.LINKED_JARS_LIST).delete()).isTrue() + + val ran = QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar, gradleDists) { daemonZip() } + + assertThat(ran).isTrue() + assertThat(File(daemonDir, QuickBuildArtifactStager.LINKED_JARS_LIST).isFile).isTrue() + } + + @Test + fun `a matching stamp over a vanished distribution fails at staging, not mid-compile`() { + val daemonDir = File(tmp.newFolder("home"), "daemon") + val jar = File(daemonDir, "quickbuild-daemon.jar") + QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar, gradleDists) { daemonZip() } + // What an assets reinstall leaves for a moment: the link survives, its target does not. + assertThat(File(gradleDists, "$GRADLE_DISTRIBUTION_NAME/lib/$linkedJar").delete()).isTrue() + + val thrown = + runCatching { + QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar, gradleDists) { daemonZip() } + }.exceptionOrNull() + + assertThat(thrown).isInstanceOf(FileNotFoundException::class.java) + assertThat(thrown).hasMessageThat().contains(linkedJar) + } + + @Test + fun `a failed link leaves no stamp so the next stage retries`() { + val daemonDir = File(tmp.newFolder("home-link-fail"), "daemon") + val jar = File(daemonDir, "quickbuild-daemon.jar") + val emptyDists = tmp.newFolder("gradle-dists-empty") + + // Stamping a failed link would serve a daemon whose classpath names jars that are not + // there, and the early return would never stage again. + val thrown = + runCatching { + QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar, emptyDists) { daemonZip() } + }.exceptionOrNull() + assertThat(thrown).isInstanceOf(FileNotFoundException::class.java) + assertThat(File(daemonDir, QuickBuildArtifactStager.DAEMON_STAMP_FILE).exists()).isFalse() + + val ran = QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar, gradleDists) { daemonZip() } + assertThat(ran).isTrue() + } + @Test fun `a failed extraction leaves no stamp so the next stage retries`() { val daemonDir = File(tmp.newFolder("home"), "daemon") @@ -183,12 +271,12 @@ class QuickBuildArtifactStagerTest { val thrown = runCatching { - QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar) { zipOf("lib/" to null) } + QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar, gradleDists) { zipOf("lib/" to null) } }.exceptionOrNull() assertThat(thrown).isInstanceOf(FileNotFoundException::class.java) assertThat(File(daemonDir, QuickBuildArtifactStager.DAEMON_STAMP_FILE).exists()).isFalse() - val ran = QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar) { daemonZip() } + val ran = QuickBuildArtifactStager.stageDaemonIfNeeded("7:1000", daemonDir, jar, gradleDists) { daemonZip() } assertThat(ran).isTrue() } } diff --git a/build-info/build.gradle.kts b/build-info/build.gradle.kts index 076f62ce95..738c794c11 100644 --- a/build-info/build.gradle.kts +++ b/build-info/build.gradle.kts @@ -26,6 +26,7 @@ import com.itsaky.androidide.build.config.publishingVersion import com.itsaky.androidide.build.config.releaseVersion import com.itsaky.androidide.build.config.replaceContents import com.itsaky.androidide.build.config.simpleVersionName +import org.adfa.constants.GRADLE_DISTRIBUTION_VERSION plugins { //noinspection JavaPluginLanguageLevel @@ -75,11 +76,9 @@ tasks.create("generateBuildInfo") { "AGP_VERSION_LATEST" to libs.versions.agp.tooling .get(), - // The Gradle version AGP_VERSION_LATEST gets exercised against: the - // distribution the IDE bundles (GRADLE_DISTRIBUTION_VERSION in - // composite-builds/build-deps-common). AGP 9.3.1 refuses to configure - // on anything older than 9.5. - "AGP_VERSION_GRADLE_LATEST" to "9.6.1", + // Read from the constant rather than repeated: AGP refuses to configure on a Gradle + // distribution older than it expects, so a stale copy here fails every TestKit test. + "AGP_VERSION_GRADLE_LATEST" to GRADLE_DISTRIBUTION_VERSION, "SNAPSHOTS_REPOSITORY" to VersionUtils.SONATYPE_SNAPSHOTS_REPO, "PUBLIC_REPOSITORY" to VersionUtils.SONATYPE_PUBLIC_REPO, ), diff --git a/composite-builds/build-logic/plugins/src/main/java/com/itsaky/androidide/plugins/conf/AndroidModuleConf.kt b/composite-builds/build-logic/plugins/src/main/java/com/itsaky/androidide/plugins/conf/AndroidModuleConf.kt index 396d9f2de3..99e8efd862 100644 --- a/composite-builds/build-logic/plugins/src/main/java/com/itsaky/androidide/plugins/conf/AndroidModuleConf.kt +++ b/composite-builds/build-logic/plugins/src/main/java/com/itsaky/androidide/plugins/conf/AndroidModuleConf.kt @@ -65,7 +65,7 @@ private const val PROFILEABLE_BUILD_TYPE = "profileable" * This is `true` for non-debug builds and for [INSTRUMENTATION_BUILD_TYPE] builds. When updating this * value, please update the corresponding value in `AssetsInstaller.kt` in `:app` module. */ -internal fun hasBundledAssets(variant: Variant): Boolean = +fun hasBundledAssets(variant: Variant): Boolean = when (variant.buildType) { INSTRUMENTATION_BUILD_TYPE -> true PROFILEABLE_BUILD_TYPE -> false diff --git a/docs/adr/0016-quick-build-compiles-outside-gradle.md b/docs/adr/0016-quick-build-compiles-outside-gradle.md index f655661209..27f58febfc 100644 --- a/docs/adr/0016-quick-build-compiles-outside-gradle.md +++ b/docs/adr/0016-quick-build-compiles-outside-gradle.md @@ -25,7 +25,7 @@ The first three cannot be sped up or skipped. **Gradle handles what live reload cannot.** It still provisions the proxy app through the existing Tooling API path, and still builds every edit the classifier declines. Nothing a user installs or ships comes out of the daemon. -**One compiler, not two.** Quick Build needs Kotlin 2.3.x for faster, more robust incremental compilation. Until the rest of CoGo moves up, the APK carries two Kotlin compilers. The move is in review as ADFA-2602; unifying them is ADFA-4931. +**One compiler, not two (ADFA-4931).** The daemon no longer ships a Kotlin compiler. `:app`'s `quickBuildDaemonZip` leaves `kotlin-compiler-embeddable` and eight other jars out of the daemon zip, and `QuickBuildArtifactStager` links the on-device Gradle distribution's copies in at provision time - 57.6 MB off the debug v8 APK. The price is a version tie: the daemon's Kotlin must equal the distribution's, so `:quickbuild:daemon` takes its Kotlin artifacts at `KOTLIN_VERSION`, and `:app:quickBuildDistJarCheck` fails the build when the distribution lacks a linked jar. ## Consequences diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index def9994279..7aec580c07 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -242,6 +242,14 @@ androidx-work-ktx = { module = "androidx.work:work-runtime-ktx", version.ref = " # Google google-material = { module = "com.google.android.material:material", version = "1.12.0" } google-gson = { module = "com.google.code.gson:gson", version = "2.12.1" } +# The Quick Build daemon takes this from the on-device Gradle distribution rather than the APK, +# so the version is the distribution's choice, not ours. Kept separate from google-gson above so +# an app-wide bump cannot silently break that link. +gson-quickBuildDaemon = { module = "com.google.code.gson:gson", version = "2.13.1" } +# Same story for the coroutines the Kotlin compiler pulls in. Names the -jvm module rather than +# the KMP root because that is what the compiler depends on, and a constraint on the root would +# not govern it. +kotlinx-coroutines-core-jvm-quickBuildDaemon = { module = "org.jetbrains.kotlinx:kotlinx-coroutines-core-jvm", version = "1.10.2" } google-guava = { module = "com.google.guava:guava", version = "33.4.0-android" } google-auto-value-annotations = { module = "com.google.auto.value:auto-value-annotations", version = "1.11.0" } google-auto-value-ap = { module = "com.google.auto.value:auto-value", version = "1.11.0" } @@ -269,14 +277,17 @@ xml-jb-annotations = { module = "org.jetbrains:annotations", version = "24.1.0" # GIT git-jgit = { module = "org.eclipse.jgit:org.eclipse.jgit", version = "6.8.0.202311291450-r" } -# Quick Build daemon (ADFA-4128): Kotlin Build Tools API incremental engine -kotlin-buildToolsApi = { module = "org.jetbrains.kotlin:kotlin-build-tools-api", version.ref = "kotlin" } -kotlin-buildToolsImpl = { module = "org.jetbrains.kotlin:kotlin-build-tools-impl", version.ref = "kotlin" } -# Compose compiler plugin, version-matched to the daemon's compiler -kotlin-composeCompilerPluginEmbeddable = { module = "org.jetbrains.kotlin:kotlin-compose-compiler-plugin-embeddable", version.ref = "kotlin" } +# Quick Build daemon (ADFA-4128): Kotlin Build Tools API incremental engine, and the Compose +# compiler plugin matched to it. No version here: :quickbuild:daemon takes these at +# org.adfa.constants.KOTLIN_VERSION, the Kotlin the on-device Gradle distribution embeds. +kotlin-buildToolsApi = { module = "org.jetbrains.kotlin:kotlin-build-tools-api" } +kotlin-buildToolsImpl = { module = "org.jetbrains.kotlin:kotlin-build-tools-impl" } +kotlin-composeCompilerPluginEmbeddable = { module = "org.jetbrains.kotlin:kotlin-compose-compiler-plugin-embeddable" } # Compose runtime for the daemon's compose compile tests (see :quickbuild:daemon) composeRuntimeDaemonTests = { module = "androidx.compose.runtime:runtime-android", version = "1.7.3" } -ow2-asm = { module = "org.ow2.asm:asm", version = "9.7.1" } +# Only :quickbuild:daemon uses this, and it takes the jar from the on-device Gradle +# distribution rather than the APK - so the version is the distribution's, not ours. +ow2-asm = { module = "org.ow2.asm:asm", version = "9.9" } # Tests tests-junit = { module = "junit:junit", version = "4.13.2" } diff --git a/quickbuild/daemon/README.md b/quickbuild/daemon/README.md index df0817fd75..d86eb4ef99 100644 --- a/quickbuild/daemon/README.md +++ b/quickbuild/daemon/README.md @@ -125,9 +125,16 @@ mid-construction cannot leave a fingerprint describing snapshots that were never ## Traps - **Never print to stdout.** Use the injected `log` / `warn` channels, which reach stderr. -- **`kotlin-daemon-client` and `kotlin-daemon-embeddable` look like dead weight and are not.** - Excluding them throws `NoClassDefFoundError` from inside the in-process path. `build.gradle.kts` - records which exclusion is safe and why. +- **Nine of the jars beside the daemon jar on device are not in the APK.** `:app`'s + `quickBuildDaemonZip` leaves out `kotlin-compiler-embeddable` and eight others, and + `QuickBuildArtifactStager` links the on-device Gradle distribution's copies in under the same + names, which the manifest `Class-Path` matches by exact file name. So the daemon's Kotlin version + must equal the distribution's, which is why `build.gradle.kts` takes the Kotlin artifacts at + `KOTLIN_VERSION` rather than a catalog version. +- **`kotlin-daemon-embeddable` and `kotlin-reflect` look like dead weight and are not.** + Excluding either throws `NoClassDefFoundError` from inside the in-process path. `build.gradle.kts` + records which exclusion is safe and why. (`kotlin-daemon-client`, which this bullet used to name, + is not on the 2.3.21 graph at all.) - **A `Result.Failed` diagnostic list is bounded.** kotlinc emits one unresolved-reference error per use site, so a deleted dependency yields hundreds; the whole list rides one protocol line into a phone-screen panel. kotlinc, javac and aapt2 diagnostics cap at 50 entries each with a "+K more diff --git a/quickbuild/daemon/build.gradle.kts b/quickbuild/daemon/build.gradle.kts index 608fb04c49..61fa0fbf2f 100644 --- a/quickbuild/daemon/build.gradle.kts +++ b/quickbuild/daemon/build.gradle.kts @@ -1,3 +1,5 @@ +import org.adfa.constants.KOTLIN_VERSION + plugins { id("java-library") id("org.jetbrains.kotlin.jvm") @@ -102,29 +104,37 @@ dependencies { // against one protocol definition. api: the router/handler signatures expose them. api(projects.quickbuild.protocol) - implementation(libs.kotlin.buildToolsApi) - implementation(libs.google.gson) + // The Kotlin artifacts below take KOTLIN_VERSION, not a catalog version: ADFA-4931 links the + // on-device Gradle distribution's compiler, so the two must match by construction. + implementation("${libs.kotlin.buildToolsApi.get().module}:$KOTLIN_VERSION") + // Pinned to the on-device Gradle distribution's version, not the app's, because ADFA-4931 + // links gson from the distribution instead of shipping it in the APK. + implementation(libs.gson.quickBuildDaemon) // ACC_FINAL stripping on recompiled payload classes (proxies extend user classes). implementation(libs.ow2.asm) - // The BTA implementation + its runtime deps are loaded from the daemon's runtime - // classpath on device (staged alongside the jar), matched to the bundled compiler. - // kotlin-compiler-runner exists solely to launch/talk to a separate long-lived - // "Kotlin compile daemon" JVM over RMI, which IncrementalCompiler never does here - // (it always calls useInProcessStrategy()) - dead weight (~17 KB of the ~62 MB - // quickbuild-daemon.zip, ADFA-4128 size audit). - // kotlin-daemon-client and kotlin-daemon-embeddable looked like the same kind of - // dead weight but are NOT: BuildToolsApiBuildICReporter.reportCompileIteration (part - // of kotlin-build-tools-impl itself, on the in-process path) references - // org.jetbrains.kotlin.daemon.common.CompileIterationResult, which lives in - // kotlin-daemon-client - excluding it throws NoClassDefFoundError and failed 12/52 - // :quickbuild-daemon:test cases. Keep both. - runtimeOnly(libs.kotlin.buildToolsImpl) { + + constraints { + // kotlin-compiler-embeddable asks for 1.8.0, but the on-device distribution already runs + // that compiler against 1.10.2, so the upgrade is safe. + runtimeOnly(libs.kotlinx.coroutines.core.jvm.quickBuildDaemon) { + because("the on-device Gradle distribution ships this version, which ADFA-4931 links") + } + // kotlin-compiler-embeddable's POM asks for 1.6.10; the distribution's compiler runs on its + // own 2.3.21 copy, which ADFA-4931 links instead of shipping an older one. + runtimeOnly("org.jetbrains.kotlin:kotlin-reflect:$KOTLIN_VERSION") { + because("the on-device Gradle distribution ships this version, which ADFA-4931 links") + } + } + // kotlin-compiler-runner only drives a separate compile-daemon JVM over RMI, which this + // daemon never does; do not also exclude kotlin-daemon-embeddable or kotlin-reflect, which + // look just as unused but break every real compile. + runtimeOnly("${libs.kotlin.buildToolsImpl.get().module}:$KOTLIN_VERSION") { exclude(group = "org.jetbrains.kotlin", module = "kotlin-compiler-runner") } // Staged next to the daemon jar on device and passed as -Xplugin when the user // project uses Compose. - composeCompilerPlugin(libs.kotlin.composeCompilerPluginEmbeddable) + composeCompilerPlugin("${libs.kotlin.composeCompilerPluginEmbeddable.get().module}:$KOTLIN_VERSION") // The compose compile tests resolve a classpath from this; classes.jar is extracted // from the AAR at build time and never shipped. Names the -android artifact rather // than the KMP umbrella, which redirects via available-at - a redirect a diff --git a/quickbuild/docs/debugging.md b/quickbuild/docs/debugging.md index e04f432353..991e6f1b6f 100644 --- a/quickbuild/docs/debugging.md +++ b/quickbuild/docs/debugging.md @@ -179,7 +179,7 @@ to CoGo. | Path | What | How to read it | | --- | --- | --- | | `files/home/.cg/quickbuild/quickbuild-runtime.aar` | staged runtime AAR, re-staged every provision | `run-as com.itsaky.androidide` | -| `files/home/.cg/quickbuild/daemon/` | daemon jar, its full runtime classpath, `compose-compiler-plugin.jar`; deleted and re-extracted every provision | `run-as com.itsaky.androidide` | +| `files/home/.cg/quickbuild/daemon/` | daemon jar, its full runtime classpath, `compose-compiler-plugin.jar`; deleted and re-extracted when the installed APK changed. The jars named in `dist-linked-jars.txt`, the Kotlin compiler among them, are symlinks into the Gradle distribution's `lib/` rather than extracted files - `ls -l` here, since a broken one surfaces as a `NoClassDefFoundError` on the first compile | `run-as com.itsaky.androidide` | | `files/home/.cg/quickbuild/bench-events.jsonl` | session event log; bench flag only | `run-as com.itsaky.androidide` | | `no_backup/quickbuild-scratch/-<16 hex>/work` | executor payload staging | `run-as com.itsaky.androidide`; **deleted on teardown** | | `no_backup/quickbuild-scratch/-<16 hex>/out` | daemon output: classes, dex, relinked resources | as above | diff --git a/quickbuild/docs/pipeline.md b/quickbuild/docs/pipeline.md index 10447e183e..df7c35473e 100644 --- a/quickbuild/docs/pipeline.md +++ b/quickbuild/docs/pipeline.md @@ -356,7 +356,7 @@ flowchart TB Step 7's rebuild fallback re-enters the same runner through `rebuildProxyApp`. -1. [QuickBuildArtifactStager](../../app/src/main/java/com/itsaky/androidide/quickbuild/QuickBuildArtifactStager.kt) extracts the runtime AAR and the daemon zip from CoGo's APK assets into `/quickbuild/`. It re-extracts on **every** provision on purpose: a version-keyed marker serves a stale bundle when content changes without a version bump. +1. [QuickBuildArtifactStager](../../app/src/main/java/com/itsaky/androidide/quickbuild/QuickBuildArtifactStager.kt) extracts the runtime AAR and the daemon zip from CoGo's APK assets into `/quickbuild/`. The zip leaves out the nine jars the on-device Gradle distribution already carries, the Kotlin compiler among them; the stager reads their names from `dist-linked-jars.txt` and symlinks the distribution's copies in beside the daemon jar, copying instead if the filesystem refuses a symlink. It re-extracts when the installed APK changed, which a stamp file beside the daemon jar records. 2. [GradleQuickBuildProvisioner](../../app/src/main/java/com/itsaky/androidide/quickbuild/GradleQuickBuildProvisioner.kt) runs the proxy app build through CoGo's existing `BuildService.executeTasks`, then parses `setup.json` into [ProxyAppInfo](../core/src/main/java/org/appdevforall/cotg/quickbuild/data/ProxyAppInfo.kt). 3. [QuickBuildScratch](../core/src/main/java/org/appdevforall/cotg/quickbuild/data/QuickBuildScratch.kt) creates `noBackupFilesDir/quickbuild-scratch//{work,out}` and enforces the free-space floor. It must stay on app-private storage - see the README's "Build Scratch Lives in Faster Private Storage". 4. [QuickBuildClobberCheck](../core/src/main/java/org/appdevforall/cotg/quickbuild/service/provision/QuickBuildClobberCheck.kt) plus [RealIdInstall](../core/src/main/java/org/appdevforall/cotg/quickbuild/domain/reload/RealIdInstall.kt) read the installed package's `android:appComponentFactory` to decide whether the tap clobbers a Standard Run install. Stateless: an install or uninstall outside CoGo cannot leave it stale. @@ -637,7 +637,7 @@ Resources are relinked by [Aapt2Link](../daemon/src/main/kotlin/org/appdevforall ### Shrinking the daemon: what works, and what breaks it -The daemon ships as a ~62 MB zip inside CoGo's APK, almost all of it `kotlin-compiler-embeddable`. Findings from the ADFA-4128 shrink spike, kept here because the build task that implemented them has been removed as an unexecuted follow-up - redo the surgery from this if the size is worth reclaiming, and gate it on a green corpus run. +ADFA-4931 moved `kotlin-compiler-embeddable` and eight other jars out of the daemon zip; the daemon links the on-device Gradle distribution's copies instead. The findings below are from the ADFA-4128 shrink spike, kept here because the build task that implemented them has been removed as an unexecuted follow-up. They now target a jar inside that distribution, which CoGo downloads rather than builds - so reclaiming the space means repackaging the distribution, and it still needs a green corpus run. - **R8 is ruled out.** `R8 --classfile` shrank the compiler to 37 MB and produced a non-functional daemon: every Kotlin compile died with a `NoClassDefFoundError` initializing a core CLI diagnostics class, because tree-shaking cut a static-init dependency reached only reflectively. A compiler is exactly the kind of code R8 cannot reason about. - **Dropping whole never-loaded backend subtrees does work**, and is a different operation - plain jar surgery, not tree-shaking. Every class that survives is byte-identical, so nothing that remains can break on a reflective lookup. It removed **~5.7 MB from the compressed daemon zip**.