After upgrading TestNG from 7.9 to 7.12, a suite that ran JUnit 4 classes through TestNG reports far fewer tests. What happened, and how do you fix it?
answer
- deprecated, then removed
- the junit attribute now only warns
- a message on standard error
- docs page lags the code
- one platform, several engines
basics
~20 sTestNG 7.10.0 removed its ability to run JUnit 3 and 4 tests, deprecated since 7.7.0. junit="true" now only prints a removal message, so those classes quietly stop running. Run them on the JUnit Platform, or migrate them.
solid answer
~40 sTestNG deprecated running JUnit tests in 7.7.0 and removed it in 7.10.0. In 7.12 the suite file's `junit="true"` attribute on `<suite>` or `<test>` is still parsed, but it only prints a message to standard error saying the ability has been removed and pointing at the JUnit team's TestNG engine; the `-mixed` command-line flag is still accepted but no longer used. TestNG finds no TestNG test methods in a JUnit 4 class, so those tests just stop executing and the count drops. The documentation site's JUnit-tests page still describes the old behaviour, which is why teams are surprised. Fixes: run both frameworks on the JUnit Platform, JUnit 4 classes on the deprecated Vintage engine and TestNG classes on the TestNG engine; or migrate the JUnit 4 classes; pinning 7.9 is only a stopgap.
code
xml · 7 lines<suite name="legacy-regression">
<test name="junit4-orders" junit="true">
<classes>
<class name="com.example.LegacyOrderTest"/>
</classes>
</test>
</suite>go deeper
Recall that TestNG once ran JUnit 3 and 4 tests and no longer does since 7.10.0.
Explain what remains in 7.12: the junit attribute only prints a removal message, -mixed is accepted but unused, and JUnit classes contribute no tests.
Diagnose from the test-count drop and the console message, distrust the lagging docs page, and choose between platform hosting, migration and a time-boxed pin.
Use the episode to set policy: count guards in CI, release-note review for test frameworks, and deprecations treated as dated work.
## What changed in TestNG For many years **TestNG** could act as a runner for **JUnit 3 and JUnit 4** tests. A `testng.xml` `<test>` (or `<suite>`) element with `junit="true"`, or a "mixed" mode that auto-detected the kind of each class, let a team keep old JUnit tests and write new ones in TestNG under one runner. The release history shows how that ended: 1. **7.7.0** deprecated the support ("Deprecate support for running JUnit tests", GITHUB-2847). 2. **7.10.0** removed it ("Remove deprecated JUnit related support in TestNG", GITHUB-3022). 3. **7.12.0**, the current release, still carries the removal. Any upgrade that crosses 7.10 therefore stops running JUnit classes through TestNG. ## What you see after the upgrade The removal is quiet rather than loud, which is what makes it a senior diagnosis question: - **The suite file still parses.** In 7.12, TestNG's suite-file parser still reads the `junit` attribute on both `<suite>` and `<test>`, but all it does with it is print a message to **standard error**: the ability to run JUnit tests "was deprecated in v7.7.0 and is NOW being removed permanently", followed by a pointer to the JUnit team's `testng-engine`. - **The `-mixed` flag is vestigial.** TestNG's command-line runner still accepts `-mixed`, but the parsed value is not used for anything. - **The JUnit classes stop executing.** A JUnit 4 class carries `org.junit` annotations, which TestNG does not treat as test methods, so the classes contribute no tests. - **The test count drops.** The clearest signals are the console message (easy to miss in a long CI log) and a sharp fall in the number of tests run. A further trap: the TestNG **documentation site's "JUnit tests" page** still says TestNG can run JUnit 3 and JUnit 4 tests with `junit="true"`. The source code and `CHANGES.txt` win over the docs here. ## Diagnosing it - Compare the number of tests run before and after the upgrade, per module. - Search the CI log for the removal message. - Search the suite files for `junit="true"` and the build or scripts for `-mixed`. - Confirm the missing tests are the JUnit-annotated ones, not TestNG classes excluded for another reason. - Check which TestNG version the build actually resolved, since a transitive upgrade can cross 7.10 without anyone editing a version number. - Look at the result files: the JUnit classes are not reported as failed or skipped, they are simply absent, which is why a pass-rate dashboard stays green. ## Fix options | Option | What it means | Trade-off | |---|---|---| | Pin TestNG at 7.9 | Keep the old behaviour | A stopgap: no later fixes, and the support was already deprecated | | Run both on the JUnit Platform | JUnit 4 classes on the **Vintage** engine, TestNG classes on the **TestNG engine** maintained by the JUnit team | One launch hosts both; Vintage is itself deprecated, so this is a bridge | | Migrate the JUnit 4 classes to TestNG | Rewrite annotations and assertions | TestNG's `AssertJUnit` keeps JUnit's expected-first argument order during the move | | Migrate the JUnit 4 classes to JUnit Jupiter | Leave TestNG for the TestNG suite | Two runners remain, but each is current | The JUnit Platform route is the one TestNG's own message recommends. It is described here in words only; how a build tool selects those engines belongs to the build-tool topics. ## Preventing the next silent drop - **Guard the test count.** A CI check that fails when a module's executed-test count falls sharply catches this class of problem for any runner. - **Read the release notes of a test framework like production code.** `CHANGES.txt` announced the deprecation in 7.7.0, three minor releases before the removal. - **Treat deprecation as a deadline.** The 7.7.0 deprecation was the moment to plan the JUnit 4 classes' future. The broader lesson for comparing runners: a framework's ability to host the other framework's tests is a feature that can be withdrawn, so a mixed codebase should rely on a platform built for several engines, not on one runner impersonating another.
- Why is pinning TestNG at 7.9 not a real fix?It freezes the suite on a release whose JUnit support was already deprecated, so it forgoes every later TestNG fix and only postpones the decision. It is acceptable as a short stopgap with a written trigger, such as finishing the JUnit 4 migration or moving both suites onto the JUnit Platform.
- Does the JUnit Platform route make the JUnit 4 classes future-proof?Not by itself. The Vintage engine that runs JUnit 4 classes on the platform is deprecated in current JUnit and meant for temporary use while migrating. It buys time and a single launch for both frameworks, but the JUnit 4 classes still need migrating, to JUnit Jupiter or to TestNG.
It is like a building that stops forwarding mail for a former tenant: the old address still exists, a notice is pinned in the lobby, and the letters simply stop arriving unless someone reads the notice.
- TestNG 7.12 still runs JUnit 4 classes when junit="true" is set
- TestNG rejects a suite file containing junit="true" with a parse error
- The TestNG documentation site is authoritative on current JUnit support
- The -mixed flag still makes TestNG auto-detect and run JUnit classes
- A green build after the upgrade proves every test still ran
Assumes TestNG 7.12; running JUnit 3/4 tests was deprecated in 7.7.0 and removed in 7.10.0, so any upgrade across 7.10 drops them.