skip to content

A CI job cannot clone a private repository over SSH. How do you diagnose it at the Git level?

level: seniorimportance: must knowfreq 50%

answer

  1. Make it verbose before making it different
  2. Was a key even offered?
  3. Headless means nothing can prompt
  4. Trusting the server is a separate step
  5. Read access working is not write access

basics

~20 s

Reproduce the exact command with verbose SSH via GIT_SSH_COMMAND="ssh -v", then check in order: which key was offered, whether the agent is reachable, key file permissions and passphrase, the host key in known_hosts, and whether the key has write as well as read access.

solid answer

~50 s

Work from the transport outwards. First reproduce non-interactively: `GIT_SSH_COMMAND="ssh -vvv" git clone <url>` shows which identity files ssh offered and which the server accepted. If no key was offered, the job has none — check `SSH_AUTH_SOCK` and `ssh-add -l` for the agent, the key path in `~/.ssh/config` (`IdentityFile`, `IdentitiesOnly`), and file permissions, since ssh refuses a private key that is group- or world-readable. If a passphrase-protected key is used without an agent, a headless job hangs or fails. Next, host verification: an unknown host key makes a non-interactive job fail rather than prompt, so `known_hosts` must be seeded. If clone works but push is rejected, the key is authenticated but not authorized for writes. On the HTTPS side the analogues are `GIT_TERMINAL_PROMPT=0` to fail fast instead of hanging, plus `GIT_TRACE=1` and `GIT_CURL_VERBOSE=1`. Finally check `url.insteadOf` rewrites and submodule URLs, which often send a job to a transport it has no credentials for.

code

bash · 6 lines
bash
GIT_SSH_COMMAND="ssh -vvv" git clone [email protected]:acme/app.git 2>&1 | grep -E 'Offering|Authentications|denied'
ssh-add -l
git config --show-origin --get-regexp '^url\.'

# HTTPS equivalent: fail fast instead of hanging on a prompt
GIT_TERMINAL_PROMPT=0 GIT_TRACE=1 git clone https://example.com/acme/app.git

go deeper

for a junior

Know the common causes to check: is a key present, are its permissions right, and is the host in known_hosts. Know that verbose ssh output is available through GIT_SSH_COMMAND.

for a middle

Walk the diagnosis in order and explain what each step rules out, including why a passphrase-protected key behaves differently in a headless job than in your terminal.

for a senior

Separate authentication from authorization, find configuration inherited from the base image, and reject fixes that trade away host verification for a green build.

for a principal

Own the pattern rather than the incident: how credentials and host keys are provisioned to runners, so this class of failure is configured once instead of debugged repeatedly.

## Reproduce before theorizing The first mistake is debugging in the pipeline UI. Get the exact command and environment, run it in the same image, and make it verbose. For SSH the single most useful lever is `GIT_SSH_COMMAND="ssh -vvv" git clone <url>` (or `core.sshCommand` in config), because it prints the identity files considered, the ones offered, and the server's response. ## The ordered checklist **1. Is any key being offered?** The verbose log shows `Offering public key: ...` lines. None means ssh found no identity. Causes: no agent (`SSH_AUTH_SOCK` unset, `ssh-add -l` empty), a key at a non-default path with no `IdentityFile` entry, or `IdentitiesOnly=yes` limiting ssh to keys the config names explicitly. **2. Are the permissions acceptable?** ssh refuses to use a private key file that is readable by group or others. Copying a key into a container with permissive modes is a top-three cause of "permission denied (publickey)" in CI. **3. Is the key passphrase-protected?** In an interactive shell it prompts and you never notice. Headless, it hangs or fails immediately. Either use an unencrypted key held only for the job's lifetime, or unlock it into an agent at the start of the job. **4. Is the host key known?** SSH verifies the server against `known_hosts`. A first connection to an unrecognized host cannot prompt in a non-interactive job, so the clone fails or stalls. The correct fix is to seed `known_hosts` with the host's published key. Disabling `StrictHostKeyChecking` makes the symptom vanish and the machine-in-the-middle protection with it — be prepared to say so in an interview. **5. Is it authentication or authorization?** If `git clone` and `git fetch` succeed but `git push` is rejected by the server, the key is accepted — it simply is not granted write access. That is a permissions question on the host, not a Git or ssh misconfiguration, and hunting for it in ssh logs wastes hours. ## HTTPS variants of the same failure If the remote is HTTPS, the equivalents are: - `GIT_TERMINAL_PROMPT=0` — turn a hidden interactive prompt into an immediate error, which is why jobs "hang forever at cloning". - `GIT_TRACE=1` and `GIT_CURL_VERBOSE=1` (or `GIT_TRACE_CURL=1`) — show the requests and their status. - A credential helper on the image serving a stale token; check with `git credential fill`. - `http.proxy` or an environment proxy variable sending traffic somewhere unexpected. ## The two traps that hide in plain sight **URL rewrites.** A `url.<base>.insteadOf` rule in a global or system config — sometimes inherited from a base image — can silently redirect an HTTPS URL to SSH or the reverse, so the job authenticates with a credential it does not have. `git config --show-origin --get-regexp '^url\.'` finds them. **Submodules.** The superproject clones fine over the transport the job is set up for, then `git submodule update --init` follows the URLs recorded in `.gitmodules`, which may use a different transport entirely. The failure appears one step after the successful clone and looks unrelated. ## How to present this in an interview Say the order out loud: reproduce verbosely, confirm an identity is offered, confirm the identity is accepted, confirm the host is trusted, then separate authentication from authorization, and only then look at rewrites and submodules. An answer that jumps straight to disabling host-key checking signals someone who fixes symptoms.

  • The job hangs at cloning instead of failing. What is the most likely cause?
    Something is waiting for input that will never arrive: an HTTPS credential prompt, an ssh passphrase prompt, or an unknown-host confirmation. Set `GIT_TERMINAL_PROMPT=0` so Git errors out instead of prompting, ensure any key is unencrypted or already unlocked in an agent, and seed `known_hosts` so host verification cannot ask a question.
  • Clone works but push is rejected. Where do you look?
    Not at ssh. Successful authentication for a fetch proves the key is accepted, so a rejected push is an authorization decision on the server side about write access for that identity. Confirm by checking whether a read-only operation on the same remote still succeeds, then take it up as a permissions question rather than a transport one.
  • Why is disabling StrictHostKeyChecking a poor fix?
    Host-key verification is what stops a machine-in-the-middle from impersonating the server. Disabling it makes the error go away by accepting any server that answers, which in CI means a job could fetch code from an impostor and then build and run it. Seed known_hosts with the host's published key instead.
  • The superproject clones but submodules fail. Why?
    Submodule URLs come from `.gitmodules` and often use a different transport than the job was configured for — typically SSH URLs in a job that only has an HTTPS token. Either normalize the transport with a `url.<base>.insteadOf` rewrite in the job, or provide credentials for the transport the submodules actually name.

saying these in an interview costs you the question

  • Disables host key checking as the first fix
  • Confuses a rejected push with a broken key
  • Never checks whether a key was offered at all
  • Ignores inherited url.insteadOf rules in the image
  • Assumes an interactive shell reproduces a headless job

context

resolving links…