Why does a hosted browser provider know your session ran but not whether your test passed?

juniornice to knowfreq 50%

answer

commands crossed the wire, expectations did not
a clean close is not a pass
the comparison happens in your process
two records, each authoritative for something
your own call carries the verdict

basics

~20 s

A hosted provider sees what crossed the automation connection: a session opening, its commands, and a close. Your assertion compares an expected value with an actual one inside your own process, so the verdict has to be sent separately.

solid answer

~50 s

The connection that drives a rented browser carries commands and their results — go here, click that, read this element's text — and nothing else. Whether the text that came back was the text you expected is a comparison your own process makes after the value arrives, so it never crosses the wire at all. The service can therefore report quite faithfully that a session opened, was held for some wall-clock, and ended, while having no idea that your driving-test booking suite's slot-confirmation check failed. That is why suites push a verdict on a separate, out-of-band call: a short request to the service that names the session, says pass or fail, and carries a case name a human recognises. Skip it and the sessions in the account are hard to tell apart, which is precisely the wrong position to be in when you are hunting the one that broke.

The connection that drives a rented browser carries commands and their results — go here, click that, read this element's text — and nothing else. Whether the text that came back was the text you *expected* is a comparison your own process makes after the value arrives, so it never crosses the wire at all. The service can therefore report quite faithfully that a session opened, was held for some wall-clock, and ended, while having no idea that your driving-test booking suite's slot-confirmation check failed. That is why suites push a verdict on a **separate, out-of-band call**: a short request to the service that names the session, says pass or fail, and carries a case name a human recognises. Skip it and the sessions in the account are hard to tell apart, which is precisely the wrong position to be in when you are hunting the one that broke.

code

java · 16 lines

java

// The automation connection carried only these, in order:
//   create a session on the rented browser
//   go to the booking service's slot search
//   click the earliest slot at the chosen test centre
//   read the confirmation banner's text
//   end the session
// None of that is a verdict. The verdict is made here, locally:
String banner = bookingPage.confirmationText();
boolean passed = "Slot booked".equals(banner);

// ...so a call of our own is what carries it across:
outcomeChannel.send(
    session.remoteId(),                    // handed back when the session opened
    "book-earliest-slot-at-chosen-centre", // our stable case name
    passed,
    passed ? null : "banner read: " + banner);

go deeper

for a junior

Be ready to say in one sentence what the far side can and cannot see. The commands crossed the wire; the comparison that decided pass or fail did not, because it happened in your process after the value came back.

for a middle

Know the shape of the call that fixes it: a separate request naming the session, the verdict, and a stable case name. Be able to say where in a harness it belongs and why not in each test.

for a senior

Expect to be asked what a team loses by never sending one. The tests are unaffected; triage is what degrades, and it degrades slowly enough that nobody files a ticket about it.

for a principal

Be prepared to argue why this belongs in shared harness code owned by one team rather than in each suite. The cost of skipping it is paid by whoever is on call months later, never by whoever skipped it.

Two things are happening, and only one of them is on the wire

A test running against a hosted browser holds a single connection for the life of the session. The harness sends a command, the service performs it on the rented browser, and a result comes back. Create a session with these settings. Go to this address. Find the element matching this selector. Read its text. End the session. That traffic is a faithful record of what the browser was asked to do and what it answered, and a verdict is not in it anywhere.

The reason is that a verdict is not a browser operation. When your driving-test booking suite reads the confirmation banner after claiming the earliest slot at a chosen test centre, the service performs a read and hands back a string. The comparison — is this the string we expected? — happens afterwards, in your process, in a line of your own code. The expectation was never sent anywhere, so nothing on the far side can act on it.

The service can observe	Only your harness knows
that a session was created, and with what settings	what the case was called
each command it received, and what each returned	which returned value was the one under test
how long the session was held, and how it ended	whether that value matched the expectation
that the client stopped sending commands	whether this failure is new or already known

"It ended cleanly" is not "it passed"

A session that ends without error means the client finished and closed it. That happens on a pass. It also happens when the harness caught a failure, reported it locally and tidied up; when a case was abandoned after the session had already opened; and when a run was cancelled between two steps. Those are different outcomes that leave a similar trace on the far side, and the difference lives entirely in your process.

It cuts both ways. The service's record is authoritative for what it did — when the session started, how long it was held, how it ended. Your harness is authoritative for the verdict. Neither is complete alone, and annotation is the act of joining them.

The shape of the call that closes the gap

A suite closes the gap by making a second request to the service — a request about the session rather than a command inside it. Described as a shape rather than as any product's surface, it carries:

the session it refers to, named with the identifier the service handed back when the session was created, which your test could not have predicted beforehand;
a verdict, reduced to whatever the service is willing to accept;
a case name stable enough that the same case produces the same name on the next run;
usually a short reason, when the verdict is a failure.

What matters is that it is your call, made by your code, at a moment your code chooses.

What you actually observe when nobody pushes one

The symptoms are mild individually and expensive together:

a list of sessions in which the interesting one is not marked out from the rest;
a failing case in your report with no obvious way to reach whatever the far side kept for that session, because the handle you would need is an identifier nobody wrote down;
a colleague opening the account, seeing activity, and being unable to say whether the suite is healthy;
an engineer re-running the suite to reproduce something that was already captured.

Note what is not a symptom: the tests are unaffected. A suite with no annotation passes and fails exactly as it did before, which is why this is so easy to leave undone — nothing breaks, triage just gets slower every month.

What to do about it

Put the push in the harness, not in the tests. One place that knows how a case ended is a place you can fix once; a reporting line copied into every test is a line that is missing from the next test somebody writes.
Send a case name your framework already computes, not a title somebody typed. A name that changes with the wording is not a name you can search on next month.
Record the service's session identifier in your own report too, so the link works both ways. Naming a stored artefact from the test belongs with recordings; here it is enough to keep the identifier the reply gave you.
Treat the push as best-effort for the test's own result. The verdict is already true whether or not the far side hears about it; a reporting failure should be logged loudly and must not turn a passing case red.

The one habit to build

Assume nothing is reported unless your code reports it. A rented fleet is a competent witness to its own work and a blind one to yours, and configuring the far side does not change that: the missing information never left your runner.

## Two things are happening, and only one of them is on the wire A test running against a hosted browser holds a single connection for the life of the session. The harness sends a command, the service performs it on the rented browser, and a result comes back. Create a session with these settings. Go to this address. Find the element matching this selector. Read its text. End the session. That traffic is a faithful record of what the browser was asked to do and what it answered, and a **verdict is not in it anywhere**. The reason is that a verdict is not a browser operation. When your driving-test booking suite reads the confirmation banner after claiming the earliest slot at a chosen test centre, the service performs a read and hands back a string. The comparison — *is this the string we expected?* — happens afterwards, in your process, in a line of your own code. The expectation was never sent anywhere, so nothing on the far side can act on it. | The service can observe | Only your harness knows | |---|---| | that a session was created, and with what settings | what the case was called | | each command it received, and what each returned | which returned value was the one under test | | how long the session was held, and how it ended | whether that value matched the expectation | | that the client stopped sending commands | whether this failure is new or already known | ## "It ended cleanly" is not "it passed" A session that ends without error means the client finished and closed it. That happens on a pass. It also happens when the harness caught a failure, reported it locally and tidied up; when a case was abandoned after the session had already opened; and when a run was cancelled between two steps. Those are different outcomes that leave a similar trace on the far side, and the difference lives entirely in your process. It cuts both ways. The service's record is authoritative for **what it did** — when the session started, how long it was held, how it ended. Your harness is authoritative for **the verdict**. Neither is complete alone, and annotation is the act of joining them. ## The shape of the call that closes the gap A suite closes the gap by making a second request to the service — a request *about* the session rather than a command *inside* it. Described as a shape rather than as any product's surface, it carries: - the session it refers to, named with the identifier the service handed back when the session was created, which your test could not have predicted beforehand; - a verdict, reduced to whatever the service is willing to accept; - a case name stable enough that the same case produces the same name on the next run; - usually a short reason, when the verdict is a failure. What matters is that it is **your** call, made by your code, at a moment your code chooses. ## What you actually observe when nobody pushes one The symptoms are mild individually and expensive together: - a list of sessions in which the interesting one is not marked out from the rest; - a failing case in your report with no obvious way to reach whatever the far side kept for that session, because the handle you would need is an identifier nobody wrote down; - a colleague opening the account, seeing activity, and being unable to say whether the suite is healthy; - an engineer re-running the suite to reproduce something that was already captured. Note what is *not* a symptom: the tests are unaffected. A suite with no annotation passes and fails exactly as it did before, which is why this is so easy to leave undone — nothing breaks, triage just gets slower every month. ## What to do about it 1. **Put the push in the harness, not in the tests.** One place that knows how a case ended is a place you can fix once; a reporting line copied into every test is a line that is missing from the next test somebody writes. 2. **Send a case name your framework already computes**, not a title somebody typed. A name that changes with the wording is not a name you can search on next month. 3. **Record the service's session identifier in your own report too**, so the link works both ways. Naming a stored artefact from the test belongs with recordings; here it is enough to keep the identifier the reply gave you. 4. **Treat the push as best-effort for the test's own result.** The verdict is already true whether or not the far side hears about it; a reporting failure should be logged loudly and must not turn a passing case red. ## The one habit to build Assume nothing is reported unless your code reports it. A rented fleet is a competent witness to its own work and a blind one to yours, and configuring the far side does not change that: the missing information never left your runner.

If the provider cannot see the verdict, how does it decide a session is finished?
By its own signals, not by yours: the client asked for the session to end, the connection dropped, or a limit it enforces on idle sessions expired with no further commands. All three are observations about traffic. None of them says anything about your assertion, which is why a session that ended perfectly normally can belong to a case that failed hard.
Does sending the verdict change anything about how the test itself runs?
No, and that is the point worth internalising. The annotation is a separate request made after the outcome is known, so a suite with no annotation passes and fails identically. What changes is what a human can do afterwards: find the session that matters, group runs by case, and reach the material the far side kept for it.

saying these in an interview costs you the question

Assuming a session that closed cleanly means the test passed
Believing the provider infers pass or fail from the commands sent
Thinking the verdict travels automatically with the session
Putting the reporting call in individual tests rather than the harness
Treating the provider's session record as the suite's source of truth

Why does a hosted browser provider know your session ran but not whether your test passed?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

Two things are happening, and only one of them is on the wire

"It ended cleanly" is not "it passed"

The shape of the call that closes the gap

What you actually observe when nobody pushes one

What to do about it

The one habit to build

▾common follow-ups2 questions

▾red flags5 flags

Two things are happening, and only one of them is on the wire

"It ended cleanly" is not "it passed"

The shape of the call that closes the gap

What you actually observe when nobody pushes one

What to do about it

The one habit to build

answer

go deeper