skip to content

How do you run TestKit functional tests across multiple Gradle versions, and why does it matter?

level: seniorimportance: should knowfreq 35%

answer

  1. withGradleVersion("x.y")
  2. withGradleInstallation / withGradleDistribution
  3. @ParameterizedTest over version list
  4. prove supported range, deprecations
  5. cache distributions in CI; slow set

basics

~10 s

Call withGradleVersion("8.5") on GradleRunner to run the build under a specific Gradle version. Parameterize the test over several versions to prove your plugin works across the range you support.

solid answer

~40 s

TestKit normally runs with the same Gradle version that runs the tests, but `GradleRunner.withGradleVersion("8.5")` (or `withGradleInstallation(file)` / `withGradleDistribution(uri)`) makes it download and run a *specific* version instead. Compatibility matters because a plugin advertises a supported Gradle range, and APIs deprecate or change behavior across versions; only running against each target proves the claim. The standard approach is a **parameterized test** (JUnit 5 `@ParameterizedTest` with `@ValueSource`/`@MethodSource`, or Spock's `where:`) iterating over the versions in your matrix, re-running the same functional assertions under each. In CI you often combine this with a build matrix. Pin the lowest supported version and the latest, plus any in between where behavior is known to shift.

code

kotlin · 11 lines
kotlin
@ParameterizedTest
@ValueSource(strings = ["7.6.4", "8.5", "8.7"])
fun `greet works on supported versions`(gradleVersion: String) {
    val result = GradleRunner.create()
        .withGradleVersion(gradleVersion)
        .withProjectDir(testProjectDir)
        .withArguments("greet")
        .withPluginClasspath()
        .build()
    assertEquals(TaskOutcome.SUCCESS, result.task(":greet")!!.outcome)
}

go deeper

for a junior

Know withGradleVersion(...) exists to pick a version.

for a middle

Parameterize a test across a small version list and run the same assertions.

for a senior

Define the support matrix to match published metadata, manage download/caching cost, and reason about API/behavior drift.

for a principal

Own the compatibility policy: which versions are supported, how the matrix maps to CI, and how deprecations are absorbed over time.

## Why cross-version testing A published plugin states which Gradle versions it supports. Between major/minor Gradle releases, APIs get deprecated then removed, configuration-cache and lazy-API semantics tighten, and task behaviors change. The only way to *prove* your plugin works on, say, 7.6 through 8.x is to actually execute it on each. ## The runner knobs - **`withGradleVersion("8.5")`** — TestKit downloads (and caches) that distribution and runs the build with it. - **`withGradleInstallation(file)`** — use a local Gradle install directory. - **`withGradleDistribution(uri)`** — use a distribution at a URI. If you set none, the build runs with the version embedded in the TestKit library on the test classpath. ## Parameterizing the test Drive the same assertions over a list of versions: ```kotlin @ParameterizedTest @ValueSource(strings = ["7.6.4", "8.5", "8.7"]) fun `plugin works across versions`(gradleVersion: String) { val result = GradleRunner.create() .withGradleVersion(gradleVersion) .withProjectDir(testProjectDir) .withArguments("greet") .withPluginClasspath() .build() assertEquals(TaskOutcome.SUCCESS, result.task(":greet")!!.outcome) } ``` ## Caveats - `withPluginClasspath()` injection has historically had limitations on very old Gradle versions; verify the lower bound you claim. - Downloading distributions costs time and network — cache them in CI (TestKit reuses a Gradle user home) and consider a separate slow test source set. - Match the *lowest supported* version to whatever your plugin's metadata / `attributes` declare, so tests and the published compatibility claim agree. ## CI strategy Pair the in-test parameterization with a CI matrix when you also want to vary the *JVM* running the tests, keeping the version loop inside the test for the Gradle dimension. The result is a documented, enforced support window rather than an aspirational one.

  • What method sets the Gradle version a TestKit build runs under?
    GradleRunner.withGradleVersion("x.y"), which downloads and caches that distribution; withGradleInstallation/withGradleDistribution are alternatives for local or URI-based distributions.
  • Which Gradle versions should you put in the matrix?
    At minimum the lowest version your plugin claims to support and the latest, plus any intermediate versions where relevant behavior or APIs changed.
  • Why is cross-version testing slow and how do you mitigate it?
    Each version may download a distribution and runs a full build. Mitigate by caching the Gradle user home in CI, splitting into a separate slow test source set, and limiting the matrix to representative versions.

saying these in an interview costs you the question

  • Claiming a support range without ever running tests on the lower bound.
  • Hardcoding only the current version and assuming older ones work.
  • Ignoring that withPluginClasspath() can behave differently on old Gradle versions.

context

resolving links…