skip to content

In TestNG, how do you tag test methods with groups and then run only one group, such as smoke, from testng.xml or the command line?

level: juniorimportance: must knowfreq 62%

answer

  1. a label on the annotation
  2. class labels plus method labels
  3. groups, run, include, exclude
  4. regular expressions, not shell wildcards
  5. exclusion preempts inclusion

basics

~20 s

Add groups = {"smoke"} to @Test, then select it with <groups><run><include name="smoke"/></run></groups> in testng.xml or -groups smoke on TestNG's command line. Group names match as regular expressions, and an excluded group beats an included one.

solid answer

~40 s

A TestNG group is a string label on `@Test(groups = {"smoke", "regression"})`. Put `@Test(groups = ...)` on the class and its public methods inherit the label, with method-level groups added on top. The suite file picks what runs: inside a `<test>` element, `<groups><run>` holds `<include name="smoke"/>` and `<exclude name="broken"/>`, and groups named under `<suite>` accumulate with each `<test>`'s own. TestNG's command-line runner takes the same choice as `-groups` and `-excludegroups`, each a comma-separated list. Names are matched as Java regular expressions, so `windows.*` works and a bare `*` does not. A method in both an included and an excluded group is left out, which is why a `broken` group is a clean way to park tests. Switching sets needs no recompile, only a different suite file or flag.

code

java · 13 lines
java
import org.testng.annotations.Test;

@Test(groups = "checkout")
public class CheckoutTests {

  @Test(groups = {"smoke"})
  public void cartTotalIsShown() { }

  @Test(groups = {"regression", "broken"})
  public void couponStacking() { }

  public void taxRounding() { }
}

go deeper

for a junior

Recall the groups attribute on @Test, the groups/run/include/exclude block in testng.xml, and that the names are regular expressions.

for a middle

Explain the precedence: exclusion beats inclusion, an include makes untagged methods drop out, suite and test groups accumulate, class and method groups combine.

for a senior

Show how you would lay out labels so the smoke and regression slices stay predictable, and how a broken group quarantines failures without deleting them.

for a principal

Talk about naming discipline: a small, documented label vocabulary with hierarchical names, so pattern selection stays safe as the suite grows.

## What a TestNG group is A **group** in TestNG is nothing more than a string label attached to a test method through the `groups` attribute of `@Test`. A method can carry any number of labels, and the labels change nothing until a run filters on them or a dependency names them. That is the point: you tag once in code and decide at run time which slice of the suite to execute, without recompiling. - **Method level:** `@Test(groups = {"smoke", "regression"})` puts one method in two groups. - **Class level:** `@Test(groups = "checkout")` on the class gives that label to every public method of the class, and a method-level `groups` list adds to it rather than replacing it. - **No group:** a method with no label is still a test; whether it runs depends on the filter, as the table below shows. ## Selecting groups in the suite file The usual place to choose groups is the suite file, `testng.xml`. Inside a `<test>` element, a `<groups>` block holds a `<run>` element, and `<run>` holds `<include>` and `<exclude>` entries: ```xml <suite name="Nightly"> <test name="Smoke"> <groups> <run> <include name="smoke"/> <exclude name="broken"/> </run> </groups> <classes> <class name="com.shop.tests.CheckoutTests"/> </classes> </test> </suite> ``` Groups can also be declared under `<suite>`; they apply to every `<test>` below it and **accumulate** with the `<test>`'s own, so including `a` at suite level and `b` in a `<test>` runs both. A `<define>` element builds a **group of groups** (a meta-group), for example an `all` group that includes `smoke` and `regression`, so the run block can name one label. ## Regular expressions, not wildcards Every name in `<include>`, `<exclude>`, and the command-line flags is treated as a **Java regular expression**. That has two consequences worth saying out loud in an interview: 1. `windows.*` matches `windows.checkintest` and `windows.functest`, because `.*` means "any characters". 2. A shell-style `*` on its own is not "everything"; the regular expression for anything is `.*`. Naming groups hierarchically (`api.smoke`, `ui.smoke`) pays off precisely because a pattern can then select a family of them. ## How include and exclude combine TestNG's selector evaluates exclusions after inclusions and lets them win: a method that matches an excluded group is dropped even if another of its groups is included. | Filter configured | Method in included group | Method in excluded group | Method with no group | |---|---|---|---| | include only | runs | n/a | does not run | | exclude only | n/a | does not run | runs | | include and exclude | runs unless also excluded | does not run | does not run | The last column surprises people: once any `<include>` is present, an unlabelled method falls outside the selection. With only an `<exclude>`, everything not excluded still runs. This is also why a `broken` group works as a parking lot: add the label to a failing method, keep `<exclude name="broken"/>` in the suite, and the rest of the run stays clean while the broken tests remain visible in code. ## Selecting from the command line TestNG's own runner class, `org.testng.TestNG`, accepts `-groups` and `-excludegroups`, each a comma-separated list of names. They are convenient for a one-off run against a class with `-testclass`. In a pipeline the common shape is two suite files, or one suite with two `<test>` elements: a fast `smoke` slice on every change and the full `regression` slice nightly. How a build tool forwards these choices is that tool's configuration, not TestNG's. ## A worked nightly split A common layout for a pipeline that splits a fast slice from a full one: 1. Every test that guards a critical path carries `smoke` in addition to its feature label. 2. Every test carries `regression`, often set at class level so nobody forgets it. 3. The per-commit job's suite includes `smoke` and excludes `broken`. 4. The nightly job's suite includes `regression` and excludes `broken`. Because `smoke` tests are also `regression` tests, the nightly run is a superset of the per-commit run, and a quarantined test drops out of both slices through one label. ## Common mistakes - Expecting `*` to behave like a shell glob. - Assuming an include of `smoke` still runs unlabelled helper tests. - Believing a method in both an included and an excluded group runs because the include came first. - Using groups as an ordering device; groups select, they do not order. Ordering is the job of dependencies. JUnit Jupiter's nearest counterpart is `@Tag` with tag expressions; TestNG groups differ in being matched by regular expression and being definable in the suite file itself, including meta-groups.

  • In TestNG, a method is tagged both "smoke" and "flaky"; the suite includes smoke and excludes flaky. Does it run?
    No. TestNG's selector lets exclusions preempt inclusions: once a method matches any excluded group it is dropped, whichever of its other groups is included. That makes an exclude list a reliable way to quarantine a label across every slice that would otherwise pick the method up.
  • What does a <define> element inside <groups> give you in a TestNG suite file?
    A meta-group: a named group made of other groups, for example all = smoke + regression. The run block can then include that one name, and changing what all means is a one-line edit in the suite file rather than a retag of test code.

  • A bare * in an include selects every group.
  • With an include of smoke, untagged tests still run.
  • If a method is both included and excluded, the include wins.
  • Changing which groups run requires recompiling the tests.
  • Class-level groups are replaced by method-level groups, not combined.