skip to content

How do Cucumber-JVM's TestNG base class and CLI runner differ from a JUnit Platform suite?

level: middleimportance: nice to knowfreq 31%

answer

  1. Same engine, three different starters
  2. TestNG arrives through a data provider
  3. The CLI is only a main method
  4. An exit code is not a test report
  5. Per-scenario naming is the suite's advantage

basics

~20 s

AbstractTestNGCucumberTests feeds scenarios to one inherited test method through a data provider, so results arrive as repeated invocations. The command-line runner is not a test framework at all: it returns an exit code the build must interpret.

solid answer

~50 s

The TestNG route is a subclass: extend `io.cucumber.testng.AbstractTestNGCucumberTests` and configure it with the TestNG options annotation. The base class exposes a data provider of scenarios and one inherited test method that runs each, so TestNG reports one method invoked many times rather than a named test per scenario, and concurrency is set by making that data provider parallel — not by Cucumber's parallel keys. The command-line route runs `io.cucumber.core.cli.Main` as an ordinary Java program: no test framework is involved, results reach the build only as a **process exit code**, and any report has to be attached deliberately. The platform suite is the one that gives per-scenario tests with real names, automatic build failure, and single-scenario re-run from an IDE — which is why it is the default choice unless an existing TestNG estate says otherwise.

code

java · 17 lines
java
package com.vinylmarket.acceptance;

import io.cucumber.testng.AbstractTestNGCucumberTests;
import io.cucumber.testng.CucumberOptions;
import org.testng.annotations.DataProvider;

@CucumberOptions(
        features = "classpath:com/vinylmarket/acceptance",
        glue = "com.vinylmarket.acceptance.steps")
public class RunCucumberTest extends AbstractTestNGCucumberTests {

    @Override
    @DataProvider(parallel = true)
    public Object[][] scenarios() {
        return super.scenarios();
    }
}

go deeper

for a junior

Know that more than one entry point exists and that the platform suite is the usual choice. Being able to name the command-line runner as an alternative is enough at this level.

for a middle

Explain how scenarios reach TestNG through a data provider, why that flattens reporting, and why a command-line run communicates only through an exit code.

for a senior

Be ready to diagnose a pipeline that never goes red because a command-line exit code is swallowed, and to justify migrating an inherited TestNG suite or deliberately keeping it.

for a principal

Own the decision for the organisation: one entry point across repositories, or a documented exception for the estate that has TestNG infrastructure worth keeping, and what the migration costs if you standardise later.

## Three ways into the same engine Cucumber-JVM separates *what runs* from *what starts the run*. The Gherkin parsing, glue discovery and step matching are identical across entry points; what changes is who owns the process and where the results end up. | Route | How scenarios appear | How the build learns of failure | Concurrency configured by | |---|---|---|---| | JUnit Platform suite | one platform test per scenario, named | ordinary test failure | Cucumber's parallel keys | | TestNG base class | one inherited method invoked per scenario | TestNG result adopted by the build's TestNG support | the data provider's parallel flag | | Command-line runner | console output plus whatever formatter you attach | process exit code | the runner's thread option | ## The TestNG route You write a class that extends `AbstractTestNGCucumberTests` and carry configuration on the TestNG-flavoured Cucumber options annotation. The base class supplies two things: a data provider that yields the discovered scenarios, and a test method that executes one scenario per invocation. That shape has real consequences: - **Reporting granularity.** TestNG sees one method invoked N times. Reports that group by method name show a single row with N results, rather than N named tests, unless the reporting layer understands data-provider parameters. - **Concurrency.** Because scheduling belongs to TestNG's data provider, you enable it by overriding the provider method and marking it parallel. Cucumber's own parallel configuration keys are not what drives this route. - **Ecosystem fit.** If a team already has TestNG suite XML files, groups, listeners and retry analysers, this route inherits all of it for free. That, and only that, is the reason to choose it today. ## The command-line runner `io.cucumber.core.cli.Main` is a `main` method. You put the test classpath together yourself and pass options as arguments — `--glue` for the packages, `--tags` for a tag expression, `--threads` for concurrency, `--dry-run` to walk discovery without executing step bodies — followed by the features to run. What you gain is independence: nothing about the run assumes a test framework, which suits a container image whose only job is to execute a pack, a tool that shells out to Cucumber, or a smoke check against an already-deployed environment. What you take on is everything the framework used to do: 1. **Interpreting the result.** The build sees a process that exited zero or non-zero. If the script swallows the exit code, a red run is a green build, and this is by far the most common mistake with this route. 2. **Producing a report.** No formatter is attached unless you attach one, so a failed run may leave nothing behind but console output. 3. **Assembling the classpath.** Something has to hand the JVM the compiled test classes, the glue and every dependency. 4. **Re-running one scenario.** There is no test tree to click; you narrow with a tag or name filter instead. ## What does not change between them It is worth being precise about the part that is identical, because interviews often probe whether you think the entry point alters behaviour. It does not. Across all three routes: - Gherkin parsing and outline expansion are the same, so a scenario means the same thing everywhere; - glue discovery works the same way, from the same package list; - step matching, hooks and tag filtering behave identically; - the same `cucumber.*` option vocabulary applies, only written in a different place; - a step that fails fails for the same reason and produces the same result. So a migration between entry points is a change to how a run is *started and reported*, never to what the scenarios assert. That makes it a low-risk change to plan and an easy one to justify — the risk lives entirely in the pipeline wiring around it. ## Why the platform suite is usually right The suite route wins on the things you feel daily rather than the things you notice once: - each scenario is a first-class test with a name, so a failure in a report says which scenario failed without opening the log; - failures fail the build with no glue code and no exit-code handling; - an IDE can run one scenario straight from the feature file, because that scenario exists as a platform test; - Cucumber's own configuration keys apply uniformly, so the same knobs work in the suite class and in a properties file. The honest summary for an interview: choose the platform suite by default; choose TestNG when an existing TestNG estate is worth more than per-scenario naming; choose the command-line runner when there is deliberately no build around the run at all. And say which one you mean whenever you use the word "runner", because in this ecosystem it can mean any of the three plus the framework underneath.

  • Your pipeline runs Cucumber through the command-line runner and never goes red. What is wrong?
    The script is discarding the exit code — a pipe, a trailing command, or an explicit ignore. The command-line runner communicates failure only through the process result, so the job step must propagate it. Attaching a report format as well gives you something to inspect afterwards, but the exit code is what makes the build honest.
  • When would you still start a new suite on the TestNG route?
    When the surrounding estate is TestNG: existing suite XML, groups, listeners and retry infrastructure that the team relies on. Inheriting that machinery can outweigh the loss of per-scenario naming. For a new project with no such estate, the platform suite gives better reporting and better IDE support for less setup.
  • How does concurrency differ between the TestNG route and a platform suite?
    On a platform suite, Cucumber's engine schedules scenarios and its own parallel configuration sizes the pool. On the TestNG route, scheduling belongs to the data provider, so concurrency is enabled by making that provider parallel. Same requirement on the glue either way: no shared mutable state, and no scenario depending on another's leftovers.

saying these in an interview costs you the question

  • Expects a command-line run to fail the build automatically
  • Thinks TestNG generates one test class per feature file
  • Uses Cucumber's parallel keys to parallelise the TestNG route
  • Says runner without saying which of the three
  • Believes the entry point changes how steps are matched