When does a check belong on a vendor's hosted schedule rather than in your own pipeline, and what do you pay for it?
answer
- Placement question, not a tooling question
- Two unique properties: no change, outside origin
- Everything else is preference, not structure
- Cheap to attribute, or nobody triages it
- Budget the hosted set; adding one spends from it
basics
~20 sPut a check there only when it must run with no change behind it, from outside your network. You pay with an unreviewed second definition, a credential in an account you do not hold, and a borrowed clock.
solid answer
~50 sThere are exactly two things this arrangement gives that your own pipeline structurally cannot: **execution when nobody pushed**, covering the long quiet between deploys, and **an origin outside your network**, exercising the path a stranger takes in. A check that needs neither belongs where changes happen, because there it gets attribution, review and a revert for free. The price is real: a second copy of your suite that version control never sees, a working credential held in somebody else's account, a verdict that is not in your build, an ownership vacuum, and a dependency on their availability — plus the fact that the whole thing degrades toward green rather than red. So keep the hosted set small, narrow and cheap to attribute; a broad regression suite on somebody else's clock produces failures nobody can own. Whether a check is a good health indicator is a separate measurement question.
go deeper
Remember the two things this placement uniquely offers: a run happens even when nobody pushed, and the request arrives from outside your network. Checks that need neither belong where your changes are.
Explain the tradeoff concretely: an unreviewed second definition, a credential stored where your rotation does not reach, and a verdict that never lands in a build, weighed against continuous outside coverage.
Argue for a small, narrow hosted set whose failures are cheap to attribute, with named owners, rotation covered in the runbook, and something outside the account that notices the runs have stopped.
Own the strategy: how much operational definition lives in vendor accounts, whether to run the same vantage point on infrastructure you control instead, and how you cap and audit the set over time.
## Frame the decision as placement, not tooling The weak version of this discussion is "should we use hosted scheduled runs?" The strong version is: **for each check, where should it be driven from, and what does that placement cost?** A collection is portable; the interesting choice is who starts it, from where, and where the answer is written down. Once you frame it that way, the decision has a small number of inputs. ## What the hosted placement uniquely provides Only two properties are genuinely unavailable to a run you drive yourself: 1. **Execution with no change behind it.** Your pipeline runs because somebody pushed. Between deploys — nights, weekends, a quiet quarter — nothing exercises the system. Failures that arrive without a code change (an expiry, a rotation, a dependency changing behaviour, a hand edit) are invisible to a change-triggered runner by construction. 2. **An origin outside your own network.** The request arrives the way a stranger's does, through public name resolution, your edge, and whatever sits in front of the service. A run started inside cannot exercise that path no matter how good the collection is. Everything else people cite — convenience, no infrastructure to run, a shared place to see results — is a preference, not a structural property. Say that out loud; it separates a judgement answer from a product pitch. ## What you pay | Cost | Concretely | |---|---| | **A second definition** | what runs, how often, against what, held in an account with no diff, no review and no revert | | **A live credential elsewhere** | a working identity against your environment, stored where your rotation process does not reach | | **A detached verdict** | the result is not in your build, so nothing links it to a change or an owner | | **A borrowed clock** | your feedback loop now depends on another party's availability | | **Silent decay** | drift, narrowing and pausing all present as continuous green | The last row is the one that decides how big the hosted set should be. Costs that announce themselves are manageable; costs that look like success compound. ## A usable decision rule - **If the check only makes sense against a change, keep it where changes happen.** Anything asserting "this behaviour is correct" belongs beside the code that produces it, where a failure names a commit. - **If the check must be true continuously, and true from outside, place it on the hosted clock.** Reachability of the paths your customers actually depend on is the archetype. - **Prefer checks that are cheap to attribute.** A narrow check whose failure has a small number of possible causes can be triaged by whoever is around. A broad suite whose failure could mean anything will be triaged by nobody. - **Prefer paths that change rarely.** Every change to a hosted-checked path means remembering to republish a copy in an account, which is exactly the step people forget. - **Cap the set deliberately.** Decide how many hosted checks you are willing to own, and treat adding one as spending from that budget rather than as a free addition. ## The questions a principal is expected to ask 1. **Who owns each hosted check by name, and where is that written?** If nobody can be named, the check should not exist, because the mechanism will not name anyone for you. 2. **What happens when the vendor's platform, or the account itself, becomes unavailable?** The check stops, silently, and its silence looks like health. What outside that account notices? 3. **How does a credential rotation reach the copy stored there?** If the runbook does not mention it, every rotation will break these checks and everyone will re-learn it in triage. 4. **How does a newcomer discover these checks exist?** They are not in the repository unless you put a note there. 5. **Could we own this vantage point ourselves?** Running a check from a network you control but outside your production perimeter is a real alternative with a different cost profile: more to operate, less to lose track of. Deciding between them is the actual strategic question, and neither answer is universally right. ## The boundaries Two adjacent debates are not yours to settle here. **What makes a good indicator of service health**, and how a synthetic check compares with real client telemetry, is a service-level measurement subject. **How failures are routed, aggregated and displayed** is an observability design subject. Placement — which checks run on whose clock, from which side of your perimeter, and what that costs in review, ownership and trust — is the part this question is about, and keeping the three apart is itself part of the answer.
- Why is a broad regression suite a poor candidate for somebody else's clock?Because its failures are expensive to attribute and no mechanism assigns an owner. A wide suite can go red for any of dozens of reasons, with no commit to point at and no author on the hook, so it becomes noise that people learn to ignore or pause. Narrow checks with few possible causes survive that environment; broad ones do not.
- What alternative gives you an outside origin without a definition living in a vendor account?Driving the same collection yourself from a network you control that sits outside your production perimeter. You keep review, history and ownership, and the credential stays in your own store; you pay by operating that runner and its clock. Choosing between the two is the real tradeoff — convenience against custody — and neither answer is right for every organisation.
- How large should the hosted set be, and how would you argue for a cap?Small enough that every check in it has a named owner and a bounded set of causes. Argue the cap from the decay properties: each hosted check adds unreviewed definition, a credential outside your rotation, and a silence you must detect. Those costs do not announce themselves, so the only control is deciding in advance how many you will carry.
saying these in an interview costs you the question
- Justifies the placement with convenience rather than a structural property
- Puts the whole regression suite on somebody else's clock
- Never asks who owns each hosted check by name
- Omits the stored credential from the rotation runbook
- Assumes the vendor's clock is always available to run it
- Ignores that a self-hosted external runner is an alternative