Testing Tools

It is a composed annotation: your own annotation type that is itself annotated with JUnit's @Test, @Tag, @ExtendWith and so on. Jupiter looks up annotations recursively, so anything meta-annotated with @Test is discovered and run as a test.

@Disabled skips a test instead of running it, reporting it as skipped rather than failed. On a method it skips that method; on a class it skips every test in the class, including its nested classes. The string argument is the reason, shown in reports and IDEs.

Annotate it with org.junit.jupiter.api.Test. The method must not be private, static or abstract, and must return void. Parameters are allowed only if a ParameterResolver supplies them (TestInfo, TestReporter, @TempDir...). The class must not be abstract and needs a single constructor; neither class nor method has to be public.

Annotate the test with JUnit Jupiter's @Timeout, e.g. @Timeout(value = 500, unit = TimeUnit.MILLISECONDS). The default unit is seconds. When the limit elapses the test fails with a TimeoutException naming the method and the configured duration.

Call Assumptions.assumeTrue(condition) (or assumeFalse) at the top of the test. If the condition does not hold, a TestAbortedException is thrown, the rest of the method never runs, and the test is reported as aborted or skipped — the build stays green rather than red.

The mock's default answer decides. Mockito's built-in default, Answers.RETURNS_DEFAULTS, gives 0 for numeric primitives, false for boolean, an empty collection, Optional or Stream for those types, and null for every other object. Nothing throws.

Use eq() only when the same call already uses another matcher, because matchers and raw values cannot be mixed. If no argument needs a wildcard, pass plain values - eq() everywhere is noise with identical behaviour.

An ArgumentCaptor grabs the actual argument a mock received so you can assert on it. Create it with ArgumentCaptor.forClass(X.class) or @Captor, pass captor.capture() inside verify(...), then read captor.getValue() and assert normally.

A matcher is a placeholder used inside Mockito's when() or verify() saying 'any argument fitting this description' instead of one exact value. any() matches anything, anyString() any non-null String, anyInt() any int. Use them when the value is irrelevant or unpredictable.

No real code runs. Mockito returns the type's default value: null for objects, 0 for numbers, false for boolean, and an empty collection, Optional or stream for those types. Void methods do nothing. The call is recorded so you can verify it later.

allow(obj).to receive(:msg) permits a message and configures its reply; the example passes whether or not it is sent. expect(obj).to receive(:msg) also demands it: unless it arrives exactly once by the end of the example, the example fails.

describe and context both create an example group and behave identically; it defines one example in a group. A nested group is a subclass of its parent, inheriting let, hooks and helpers, and each example runs in a fresh group instance.

eq passes when actual == expected, eql when actual.eql?(expected), and equal or be(x) only when both are the same object (equal?). Bare be with no argument passes for any truthy value. eq is the default for values.

A plain double accepts stubs for any method name, so it stays green when the real class changes. instance_double(SmsGateway) only lets you stub or expect instance methods SmsGateway really defines, and checks calls against their parameters.

let is lazy, running its block on first call and memoizing the value for one example. let! also adds a before hook, so it always runs. An instance variable set in before always runs too, and reads as nil if misspelled.

Minitest runs every public method whose name starts with test_ in a Minitest::Test subclass, each on a fresh instance. setup runs before every test and teardown after it, even when the test failed.

Run the file with ruby -Ilib:test, and filter tests with --include (alias -n) using a name or a /regexp/, or --exclude. Under rake use N=, X= or A=; Minitest 6's minitest command also accepts file.rb:line.

assert_raises(ErrorClass) { code } passes when the block raises that class or a subclass, and returns the exception so you can assert on its message. No exception, or a different one, fails the test.

expect(name, retval, args) records one expected call and its return value; each actual call consumes one expectation. verify raises MockExpectationError for any expected call that never happened; unexpected arguments raise it at call time, unknown methods raise NoMethodError.

Time.stub(:now, fixed_time) { ... } redefines now on Time's singleton class for the length of the block, returning fixed_time or calling it if it is callable; an ensure clause restores the original method even if the block fails.

<testsuites> names sets of test directories or files; bootstrap names a PHP script, usually vendor/autoload.php, included once before tests load; <source> declares your first-party code, which coverage and issue reporting are scoped to.

Name the method with method(), then attach an answer: willReturn() for fixed values, willReturnMap() to pick by arguments, willReturnCallback() to compute from them, and willThrowException() to throw.

assertSame checks identity with ===: same type and value, and for objects the same instance. assertEquals checks loose equality: 90 equals '90', and separate objects with equal properties are equal. Loose checks can pass when the type is wrong.

#[DataProvider('name')] points a test at a public static method returning an iterable of argument arrays. PHPUnit calls it while building the suite, before any setUp(), so there is no instance to call it on; each array becomes one test run.

createStub() returns a Stub that only answers calls with configured values; createMock() returns a MockObject that can also carry expects() rules that PHPUnit enforces. Stub collaborators that feed data in, mock those whose calls are the outcome.

toBe() checks type and value and, for objects, the same instance; toEqual() checks equal value loosely, so '1' equals 1 and two equal objects pass. not inverts only the next expectation, and and() starts a new expectation on another value.

$this is the PHPUnit test case instance the closure is bound to, a fresh one per test. beforeEach() stores state on it, such as $this->cart. pest()->extend(Base::class)->in('Feature') makes tests in that folder bind to your base class instead.

test() and it() both register one test from a description and a closure; it() only prefixes the description with 'it'. describe() groups tests under a shared name prefix and scopes the hooks and datasets declared inside it to that group.

Write arch()->expect('App\Http\Controllers')->not->toUse([...]) naming the database namespace and classes such as PDO, or turn it around with arch()->expect('App\Database')->toOnlyBeUsedIn('App\Repositories'); Pest fails the test listing every file that breaks the rule.

--parallel runs the whole suite across several processes through ParaTest, so it belongs in CI; --tia (Pest 5) records which files each test touches and later runs only affected tests, replaying cached results, so it is meant for local runs.

Cypress 16 launches Chrome-family browsers - Chrome, Chrome for Testing, Chromium and Edge - plus Firefox, experimental WebKit, and the deprecated bundled Electron. Choose one per run with the --browser flag, or set defaultBrowser in the Cypress config.

Cypress runs inside the browser and obeys the same-origin policy, so a test is bound to one origin and cannot touch a document from a different one. Wrap the commands for the new origin in a cy.origin() block.

Cypress drives a single tab and has no tab-switching command, so assert the link rather than follow it: check its href and target attributes, request the URL directly with cy.request, or strip the target attribute so the click navigates the current tab.

Spec code is evaluated inside the browser, so console.log() writes to that browser's DevTools console. The terminal belongs to Cypress's separate Node process, which only prints what code in cypress.config.js and its Node handlers log.

Cypress ends the request itself and answers 503, adding an x-cypress-matched-blocked-host header naming the pattern that matched. The request is never sent, so it resolves immediately instead of waiting on a third party.

It compares an element's accessibility tree, meaning the roles, accessible names and a few ARIA states of its non-hidden nodes, against a YAML template. It ignores pixels, CSS, tag names, class names and hidden nodes.

In outcome they are the same. Both retry until the element is hidden or absent from the DOM, and both pass when it never existed at all. The difference is readability and the failure message, not semantics.

Playwright captures the page, writes that capture to the snapshots folder as the new reference image, and still fails the test. A later run, or an automatic retry, compares against the file just written and passes.

A soft assertion records its failure and lets the test keep running, while a plain expect throws and stops the test at that line. Both end with the test reported as failed.

Playwright's web-first matchers return a promise. Without await, the test body runs on and the test finishes before the matcher decides anything, so the assertion can never fail it — the error is dropped or blamed on a later test.

Take the dialog with driver.switchTo().alert(), which returns an Alert. Call getText() for its message, then accept() to press OK or dismiss() to press Cancel. Use sendKeys only on a prompt, before accepting it.

Set the Chrome preference download.default_directory to an absolute path and download.prompt_for_download to false, passed as the prefs experimental option on ChromeOptions. The browser then writes the file into that folder on whichever machine runs the browser.

Nothing functional separates them. Both send the same WebDriver Navigate To command and load the URL in the current tab. navigate() exists because the object it returns also carries back, forward and refresh.

In a per-user cache folder, by default ~/.cache/selenium, and no. Selenium 4's Selenium Manager reuses a matching cached driver on later runs and downloads only when the cache has no suitable copy. SE_CACHE_PATH moves the folder.

Selenium Manager runs automatically. The binary bundled in the Selenium 4 client detects the installed Chrome, works out the matching chromedriver, downloads it if the local cache lacks it, and returns the path, so the session starts with no PATH entry and no setup code.

JMeter reads the file's first line and uses its fields as the variable names. That line is consumed as a header, never handed to a thread as data, and the Delimiter field must match the header's separator.

JMeter function calls take the form ${__functionName(arg1,arg2)}. The double underscore marks a built-in, the parentheses hold comma-separated positional arguments, and a function that takes none may drop them: ${__UUID}.

Template $1$ inserts the text captured by the first parenthesised group of the Regular Expression field. JMeter builds the stored value from the Template, so $0$ means the whole match and $1$ the first group.

Add a JSON Extractor as a child of the sampler that returns the confirmation body, give it a variable name and a JSON Path expression, then reference the captured value as ${orderId} in the next request.

A JMeter variable is a per-thread copy that no other thread can read or change. A JMeter property is a single JVM-wide value that every thread in the run shares.

In k6 v2, rate is how many iterations the executor starts during each timeUnit period, timeUnit is that period (default 1s), and duration is how long the scenario keeps starting them.

k6 turns root vus and iterations into a single scenario named default that uses the shared-iterations executor, so iterations is the total across all VUs. A root duration set alongside them becomes that scenario's maxDuration.

k6's constant-vus executor holds a fixed number of VUs for a set time and takes vus plus duration. ramping-vus moves the count through a schedule and takes startVUs, stages and gracefulRampDown. Neither accepts the other's keys.

preAllocatedVUs is the number of VUs k6 initializes before an arrival-rate scenario starts; maxVUs is the hard ceiling that pool may grow to mid-run. Omitted, maxVUs equals preAllocatedVUs, so the pool never grows.

k6 has four metric types: Counter, Gauge, Rate and Trend. Among the built-ins, vus and vus_max are Gauges, http_reqs and iterations are Counters, checks and http_req_failed are Rates, and every duration such as http_req_duration is a Trend.

Maven runs mvn gatling:test, Gradle runs ./gradlew gatlingRun, sbt runs sbt Gatling/testOnly with a class pattern, and the JavaScript CLI runs npx gatling run. Each build tool names its own entry point; there is no shared Gatling command.

Store each flow in a chain value returned by a standalone exec call, then hand both to scenario("Shopper").exec(browse, checkout). Chains are ordinary values, so one scenario composes as many of them as you like, in order.

repeat(n) runs the wrapped chain n times; during(d) runs it until d has elapsed. Java, Kotlin and JavaScript attach the body with .on(...), while Scala passes it in a second parameter list.

A Gatling simulation in Java or Kotlin extends io.gatling.javaapi.core.Simulation and needs two imports per DSL family: the package wildcard for the types, plus a static import of CoreDsl and HttpDsl for the DSL methods.

Three: @gatling.io/core for the DSL and the simulation function, @gatling.io/http for the HTTP protocol builder, and @gatling.io/cli, a dev dependency that supplies the gatling command you invoke as npx gatling.

Assign a scheme to the static field RestAssured.authentication, for example RestAssured.basic(user, pass). It applies to any request that sets no scheme of its own. A single request opts out with given().auth().none().

Read the session id off the login response with sessionId(), then hand it to the next call as given().sessionId(value). REST Assured writes it as the JSESSIONID cookie. One SessionFilter instance added to both requests does that capture-and-replay for you automatically.

Use given().auth().oauth2(token) - REST Assured has no bearer method at all. With the default HEADER signature that call installs a PreemptiveOAuth2HeaderScheme, which writes a plain Authorization: Bearer header on the request, with no signing and no challenge round trip.

REST Assured is a Java library for testing HTTP/REST APIs. You write one fluent chain: given() sets up the request (headers, body, params, auth), when() fires the HTTP call (get/post/put/delete), then() asserts on the response (status, headers, body values).

REST Assured buffers the log instead of printing it when you chain given().log().ifValidationFails() on the request or then().log().ifValidationFails() on the response. The buffered text reaches the console only if a then() expectation fails. RestAssured.enableLoggingOfRequestAndResponseIfValidationFails() switches both sides on globally.

Declare Kotest's beforeTest { } and afterTest { } blocks in the spec body. beforeTest receives the TestCase about to run; afterTest receives a TestCase/TestResult pair. Both are suspend lambdas, so suspending setup and cleanup work directly.

Every row runs. Kotest collects the failures instead of stopping at the first one and then throws a single error that lists each failing row with its header/value pairs and that row's assertion error, so you see all bad rows in one run.

Kotest asks the element for a stable identifier: a WithDataTestName implementation wins, then an @IsStableType-annotated class's toString, then a data class's toString. Anything else falls back to a type-derived name, so every case ends up named the same.

Kotest invokes every test body — and every lifecycle callback like beforeTest and beforeSpec — inside a coroutine, so suspending calls are made directly with no wrapper builder. The test scope is itself a CoroutineScope, and Kotest enforces the test's configured timeout around that coroutine.

Kotest specs are executed by Kotest's own TestEngine, not JUnit Jupiter, and Jupiter annotations only mean something to the Jupiter engine. Use Kotest's own facilities instead: beforeTest/afterTest/beforeSpec, .config(enabled = false) or the x-prefixed builders, @Ignored on the spec, and Kotest Tag objects.

It reflects over the given instance, creates a mock or spy for every MockK-annotated property and assigns it. Without it those lateinit properties are never set, so the first use throws UninitializedPropertyAccessException — the annotations do nothing on their own.

It initialises annotated mock properties on each new test instance, resolves MockK-annotated test parameters, and afterwards runs unmockkAll() plus clearAllMocks() — after every test method, or after the whole class when the instance lifecycle is PER_CLASS.

anyConstructed<Foo>() is not an instance — it is a single shared handler for the class. Every Foo constructed while the class is mocked routes its calls through it, so one stub applies to all of them. Methods you never stub fall through to the real implementation.

No new instance: MockK patches the existing singleton in place, so its identity and state are unchanged and every holder of a reference sees the patch. Unstubbed methods still run the real implementation — an object mock is a spy, not a strict mock.

Pass the JVM name of the file facade class: package plus source file name with Kt appended. Greetings.kt in com.acme becomes com.acme.GreetingsKt. A @file:JvmName annotation on that file replaces the name, and the string must match exactly.

Pact is consumer-driven: the consumer records a pact file and the provider must satisfy it. Spring Cloud Contract is producer-first: the provider authors the contract and ships stubs. Bi-directional cross-checks a provider specification against consumer-recorded expectations, coupling the teams least.

An asynchronous Pact message pact verifies that a producer emits a payload the named consumer's handler can decode. It covers payload shape and message metadata only. Topic, broker, delivery and serialization on the wire stay unproven.

The Pact Broker stores pacts as a relation: each pact belongs to a named application and a specific version, and each verification result is recorded against that pact and the provider version that ran it. A folder stores only bytes.

A Pact consumer test starts a local mock HTTP server, registers the expected interactions on it, and runs the real client code against that server. The pact JSON file is written only when every registered interaction was requested and matched.

A Pact verification run loads pact files from a broker, a URL or a local folder, then per interaction applies the provider state, sends the recorded request to the running provider, and matches the real response against the recorded expectations.

Gherkin recognises a fixed keyword set - Feature, Rule, Background, Scenario, Scenario Outline, Examples and the Given/When/Then/And/But step keywords. Other text is free-form description where the grammar allows it, and a parse error that fails the whole file anywhere else.

It selects scenarios tagged @smoke that are not also tagged @wip. Cucumber tag expressions are boolean, with lowercase and, or, not and parentheses. Tags written on Feature, Rule, Scenario, Scenario Outline and Examples lines are inherited downward.

The glue option lists package names that Cucumber scans recursively for step definitions and hooks — packages, never file paths. Point it at the wrong package and every step of every scenario reports undefined, not just one line.

Cucumber substitutes an Examples value into the Scenario Outline's name, into step text, into the cells of a data table attached to a step, and into a doc-string body. Tags, comments and Background are never templated.

Cucumber-JVM registers a formatter with --plugin name:path or the cucumber.plugin property; cucumber-js uses --format name:path; Behave pairs --format with --outfile. Console and HTML output is written for people, while JSON, JUnit XML and message output is written for tools.

They build the specification Testcontainers sends to Docker when the container is created: withExposedPorts marks container-internal ports to publish on random free host ports, withEnv sets environment variables, and withCommand overrides the image's default command.

A module container ships the right image defaults, the environment wiring the image expects, a database-aware readiness check, and typed accessors — getJdbcUrl, getUsername, getPassword, getDatabaseName — that already contain the randomly mapped host port.

Testcontainers publishes each exposed container port on a random free port on the Docker host, so the number differs every run and cannot be hardcoded. getHost() matters too, because the daemon is not always on localhost.

Testcontainers probes an ordered chain of client strategies at startup: the tc.host override, the DOCKER_HOST environment variable with its TLS settings, then the Unix socket at /var/run/docker.sock. A CI runner needs one reachable Docker endpoint, not Docker Desktop.

Module containers assert that the image name matches the one they were written for. Wrap a mirrored image with DockerImageName.parse(name).asCompatibleSubstituteFor("postgres") to declare it behaves like upstream, and construction succeeds.

Assignment records an owner on one cycle item, so it leaves the shared pile and appears in that tester's filtered queue. It changes nobody's result: an assigned item stays untested until an outcome is recorded.

The denominator is the set of requirements the panel's saved filter selects; the numerator is how many of those satisfy the panel's configured covered condition. The unit counted is the requirement, not the test case.

A variant table holds one row of values per run of the same stored case. Placeholders in the step text name a column, and the repository substitutes that row's value, producing one execution per row.

Two things, as a pair: which tracker project receives the item, and which item type it is created as. Trackers attach mandatory fields, closed value lists and workflow to that pair, so the mapping is defined against it.

Filing from a failed execution pre-fills the draft ticket with the run's context: which case failed, which cycle it ran in, the environment or build under test, who recorded the result, and the comment and evidence captured at failure time.

MockServer keeps a journal of every request it receives, and MockServerClient.verify asserts against it. Pass a request definition describing the POST plus a VerificationTimes count. When nothing matches, verify throws an AssertionError listing the requests the journal did record.

WireMock's --no-request-journal flag stops the server recording the requests it receives. Because nothing is recorded, verification is not weakened but abolished: the disabled journal raises RequestJournalDisabledException. A verify() call after that flag can never pass, whatever the client sent.

With makeStubsPersistent set to false, WireMock keeps the regenerated stubs in memory only: they serve requests while the instance lives and vanish on restart. Its default, true, writes each one as a JSON file under the mappings directory.

WireMock's admin API at /__admin/mappings lists every registered stub on GET and registers one new stub from the posted body on POST. DELETE on the same path removes all of them. Any HTTP client can drive it while the server runs.

A hosted mock service is a stub server a vendor runs for you. Its canned replies live in an account on their platform, reachable at an address they control. What you give up is custody: review, rollback and availability.

The service makes the recording on its own machine: a capture process beside the browser grabs the virtual screen for the whole session. A test asks with one setting in the session request and fetches the finished file afterwards.

No. The worker count is a local setting that decides how many sessions your suite tries to hold at once; the ceiling belongs to the provider account and is enforced there, whatever your runner is configured to do.

W3C WebDriver reserves the colon for extension capabilities, so a provider's settings must sit under prefixed keys; gathering them into a single map is convention, not a rule. Loose non-standard keys are not capabilities the specification recognises at all.

A tunnel carries traffic for a defined set of names; anything outside that set the remote browser fetches the ordinary way. The danger is silence: an unlisted internal name can resolve publicly and the test passes against the wrong site.

Selenoid starts a fresh Docker container per WebDriver session, proxies the session to the driver inside, then force-removes it on delete. Nothing carries over: no profile, cookies, downloads or stray process, unless browsers.json mounts a host directory in.

Karate's contains asserts a subset. Against a JSON object, every expected key must exist with an equal value while extra actual keys are ignored. Against a JSON array, every expected item must appear somewhere, in any order.

Karate's match == compares the entire payload in one recursive step: every key and value must correspond exactly. An extra role key fails the step, because == is a strict whole-payload comparison, not a subset check.

The left side of a Karate match step is a variable name, a JsonPath or XPath rooted at a variable such as response.name, a function or method call, or parenthesised JavaScript. A bare $ stands for the response, and a left side that starts with / is a bare XPath rooted there too.

Put marker strings where the unpredictable values would go: match response == { id: '#uuid', createdAt: '#string', name: 'Billie' }. A marker asserts the value's type or shape instead of its content, so one step still checks every key.

#? runs the JavaScript that follows it and passes only when the result is true. Karate binds _ to the actual value at that position, so '#? _ > 0' asserts that the value there is greater than zero.