You inherit an Apache httpd server whose single httpd.conf defines 120 virtual hosts, and every change requires a full restart. How would you restructure that configuration and the process for changing it?
answer
- one file per site, explicit order
- the first vhost is a decision
- configtest needs the same module set
- diff the resolved map, not the source
- graceful reload, checked before signalling
basics
~20 sSplit it into one file per virtual host pulled in by Include, pin the catch-all vhost first through the file naming, generate the repetitive parts from a template, validate with apachectl configtest and an apachectl -S diff in CI, and apply changes with a graceful reload instead of a restart.
solid answer
~60 sThe problems in one monolith are blast radius, ordering and review. I would split it into `conf.d/vhosts/*.conf`, one file per site, pulled in with `IncludeOptional`, keeping only genuinely global settings in the root file. Because the glob loads lexically and the first vhost for an address:port is the fallback for unmatched hostnames, I would name the catch-all so it sorts first — that ordering is now a contract, not an accident. Repetitive vhosts should be generated from a template rather than copy-pasted, so a security header change is one edit. For the process: `apachectl configtest` in CI on an image with the same module set, plus a diff of `apachectl -S` output before and after so an unintended change to the vhost map fails the build. Then apply with a graceful reload — `apachectl` runs a config check before signalling, so a syntax error refuses the reload rather than dropping the site. The residual risk stays real: one bad file breaks parsing for all 120 sites, which is an argument for splitting genuinely independent tenants across servers.
code
bash · 13 lines#!/usr/bin/env bash
# CI gate: parse must succeed and the vhost map must not change unexpectedly
set -euo pipefail
apachectl configtest
apachectl -S > /tmp/vhosts.new 2>&1
if ! diff -u ./expected-vhost-map.txt /tmp/vhosts.new; then
echo "vhost map changed; review the diff and update the baseline deliberately" >&2
exit 1
fi
echo "config valid and vhost map unchanged"go deeper
Know that Apache config can be split into files pulled in with Include, and that you should always run a config check before reloading rather than restarting and hoping.
Explain the mechanics you would rely on: IncludeOptional globs loading in lexical order, configtest reporting the failing file and line, and graceful reload letting in-flight requests finish.
Demonstrate that you gate on the resolved vhost map rather than syntax alone, that you validate against the production module set, and that you sequence a migration so restructuring and behaviour change never land together.
Own the failure-domain argument: say plainly that splitting files buys reviewability and rollback but not isolation, and make the call about which tenants justify separate servers rather than letting tidier configuration imply a guarantee it does not provide.
## What is actually wrong with the monolith It is not the file size. Three specific properties make a single 120-vhost file expensive to own: 1. **Blast radius on parse.** Apache parses the assembled configuration as one unit. A typo in the twelfth vhost prevents the server from starting for all 120. Splitting into files does not remove this — the includes are still assembled into one config — but it makes the failing unit small, reviewable and independently revertible. 2. **Ordering is invisible.** The first vhost for an address:port is the default that answers unmatched hostnames, and matching is first-match. In one long file that is whichever block someone happened to paste highest. Nothing marks it, and nothing warns when it changes. 3. **Change review does not scale.** A diff touching one site is buried in a file every site shares, so ownership, approval and rollback all become server-wide operations for a per-site change. ## The target layout ``` /etc/httpd/conf/httpd.conf # global only: Listen, MPM, logging, defaults /etc/httpd/conf.d/00-defaults.conf /etc/httpd/conf.d/vhosts/000-catchall.conf /etc/httpd/conf.d/vhosts/010-app.example.com.conf /etc/httpd/conf.d/vhosts/020-shop.example.com.conf ``` with ```apacheconf IncludeOptional conf.d/vhosts/*.conf ``` The numeric prefixes make load order explicit and reviewable rather than emergent from hostname spelling. `000-catchall.conf` deliberately occupies the default-vhost slot: a `ServerName` nobody resolves, no real document root, a 404. That converts "which site answers for an unknown Host" from an accident into a decision. One caution on `IncludeOptional`: it does not fail when the glob matches nothing. That is what you want for optional drop-ins and exactly what you do not want for the directory holding every site, where an empty match should be loud. Assert the file count separately in your deployment check. ## Generate the repetition Across 120 vhosts, most content is identical: TLS settings, security headers via mod_headers, log formats, proxy boilerplate. Copy-paste means a change to any of it is 120 edits with a long tail of stragglers. Two workable approaches: - A shared snippet pulled into each vhost with `Include conf.d/snippets/tls-common.conf`, keeping per-site files down to what is genuinely per-site. - Template generation in configuration management, with the site list as data and the config as an artifact nobody hand-edits. The second scales further and gives you a diff of generated output to review, which is what you actually want to gate on. ## The change process **Validate on a matching image.** `apachectl configtest` parses everything and names the file and line of any error — but it can only validate directives whose modules are loaded. Run it in a container built from the same base with the same modules enabled, or it will pass on directives the production server would reject as unknown. **Diff the resolved map.** `apachectl -S` prints the vhost map per address:port, with source file and line, and marks the default server. Capturing that before and after a change and failing the build on an unexpected diff catches exactly the class of error configtest cannot: a valid config that routes hostnames differently, or promotes a different vhost to default. **Apply gracefully.** A graceful reload has the parent re-read the configuration and retire workers as they finish, so in-flight requests are not cut. Going through `apachectl`/`systemctl reload` rather than signalling by hand matters: `apachectl` checks the configuration before signalling on restart and graceful, so a broken config fails the reload instead of taking down a running server. Signals sent directly bypass that. **Roll back by file.** With one file per site, rollback is reverting one file and reloading, not restoring a monolith that has since accumulated other people's changes. ## The trade you should name out loud Splitting files improves reviewability and rollback; it does not change the fact that 120 sites share one process, one configuration parse, one module set and one failure domain. If some of those tenants have genuinely different availability requirements or different change cadences, the honest answer is that config hygiene is a mitigation and separate servers are the fix. Say which you are buying. Restructuring is worth doing regardless — but pitching it as isolation when it is only tidiness is how people are surprised later. ## Sequencing the migration Do it mechanically and prove equivalence at each step: split the file without editing content, confirm `apachectl -S` output is byte-identical, reload; then introduce the explicit catch-all and confirm the map changes only where intended; then factor shared snippets; then move to generation. Each step is independently revertible, and none of them mixes restructuring with behaviour change — which is the discipline that keeps a config migration from becoming an outage.
- Why is diffing apachectl -S output more valuable in CI than configtest alone?configtest proves the assembled configuration parses; it says nothing about what the configuration means. A perfectly valid change can move which vhost is the default, capture a hostname with a wildcard alias, or leave a site unreachable. `apachectl -S` prints the resolved map per address:port with source lines, so diffing it catches semantic drift that syntax checking cannot see.
- What is the risk of using IncludeOptional for the directory that holds every vhost?It silently succeeds when the glob matches nothing. If a deployment misplaces the directory or a permissions change hides it, Apache starts cleanly serving only the default vhost, and health checks on the main page may still pass. Use it for genuinely optional drop-ins, and assert the expected file count in the deployment check so an empty match fails loudly.
- You have split the files and tightened the process. What have you still not fixed?The shared failure domain. All 120 sites remain one process, one configuration parse, one module set and one reload. A fatal parse error, a bad module upgrade or a crash still affects everyone. File splitting buys reviewability and quick rollback, not isolation. If some tenants need genuinely independent availability or change cadence, that is a separate-servers decision, and it should be named as such rather than assumed to follow from tidier config.
- How would you sequence the migration so it cannot cause an outage?Mechanically first: split the file with no content edits and prove `apachectl -S` output is unchanged before reloading. Then introduce the explicit catch-all and verify the map changes only where intended. Then factor shared snippets, then move to generated config. Each step is separately revertible, and no step mixes restructuring with behaviour change — which is what usually turns a config migration into an incident.
saying these in an interview costs you the question
- Assuming configtest proves the configuration behaves correctly
- Treating include order as cosmetic when it picks the default vhost
- Believing file splitting isolates tenants from each other
- Restarting rather than gracefully reloading for config changes
- Validating config on an image with a different module set