skip to content

An IKEv2 tunnel between two different makers' gateways will not come up; how do the failing exchange and its notify point you to the mismatch?

level: seniorimportance: should knowfreq 30%

answer

  1. which exchange, which side, which notify
  2. silence is not a refusal
  3. IKE SA up, Child SA missing
  4. certificates make IKE_AUTH large
  5. first-exchange errors are unauthenticated

basics

~20 s

Locate the failure by exchange: NO_PROPOSAL_CHOSEN in IKE_SA_INIT is an algorithm mismatch, AUTHENTICATION_FAILED in IKE_AUTH a secret, identity or certificate mismatch, TS_UNACCEPTABLE a selector mismatch with the IKE SA up, and an unanswered IKE_AUTH often dropped fragments.

solid answer

~40 s

Work exchange by exchange, ideally from the responder's log. In `IKE_SA_INIT`, `NO_PROPOSAL_CHOSEN` means no proposal shares one transform of every type — encryption, PRF, integrity, Diffie-Hellman group; `INVALID_KE_PAYLOAD` alone is a normal retry with the named group, not a fault. In `IKE_AUTH`, `AUTHENTICATION_FAILED` means no IKE SA: a different pre-shared key, an ID type or value the peer has no policy for (an address on one side, an FQDN expected on the other), or a certificate that does not verify. `TS_UNACCEPTABLE`, or `NO_PROPOSAL_CHOSEN` for the Child SA, leave the IKE SA up with nothing protected, so compare traffic selectors and ESP transforms. If `IKE_SA_INIT` completes but `IKE_AUTH` is never answered, suspect a large certificate-bearing message fragmented and dropped on the path; RFC 7383's IKE fragmentation fixes that.

go deeper

for a junior

Recall that IKEv2 setup has two exchanges and that a failure in each one points to a different kind of mismatch.

for a middle

Map each notify to its exchange and meaning: NO_PROPOSAL_CHOSEN, INVALID_KE_PAYLOAD, AUTHENTICATION_FAILED, TS_UNACCEPTABLE, and know which ones leave the IKE SA standing.

for a senior

Diagnose from the responder's side, separate normal retries from faults, and recognise silent IKE_AUTH loss from fragmented certificate-bearing messages and its RFC 7383 remedy.

for a principal

Push interoperability problems upstream: agree a written IKEv2 profile with peers — suites, groups, ID types, fragmentation support — so tunnel bring-up becomes a checklist, not a debugging session.

## Why the exchange tells you the layer Two gateways from different makers share only RFC 7296; their configuration screens use different words for the same IKEv2 parameters. The protocol, however, fails in a structured way. Each exchange settles one layer of the agreement, and each failure is reported with a named **Notify** message: | Exchange | Settles | Failure you see | What it points at | |---|---|---|---| | `IKE_SA_INIT` | IKE SA algorithms, Diffie-Hellman, nonces | `NO_PROPOSAL_CHOSEN` | no common IKE suite | | `IKE_SA_INIT` | — | `INVALID_KE_PAYLOAD` then success | normal: initiator guessed another group | | `IKE_SA_INIT` | — | `COOKIE` then success | normal: responder under load | | `IKE_AUTH` | identities and proof | `AUTHENTICATION_FAILED` | secret, ID or certificate | | `IKE_AUTH` | first Child SA | `NO_PROPOSAL_CHOSEN`, `TS_UNACCEPTABLE` | ESP suite or traffic selectors | | `IKE_AUTH` | — | no response at all | path drops large or fragmented messages | ## Step one: IKE_SA_INIT The initiator sends proposals; the responder must accept **exactly one transform of each type** present in one proposal, or reject them all with `NO_PROPOSAL_CHOSEN`. Mismatches therefore come from the transform types RFC 7296 defines for the IKE SA: - **encryption** (type 1), including key length attributes such as AES-128 against AES-256; - **PRF** (type 2); - **integrity** (type 3), absent when an AEAD cipher is used; - **Diffie-Hellman group** (type 4), which RFC 9370 renames Key Exchange Method. `INVALID_KE_PAYLOAD` is not this failure. It says the responder picked a group, from the initiator's own proposals, other than the one the `KEi` used; the initiator retries in that group. When the group lists share nothing, the answer is `NO_PROPOSAL_CHOSEN` instead, because the group is one of the transform types. RFC 8247 makes group 14 (2048-bit MODP) a MUST and group 19 (256-bit random ECP) a SHOULD, so check first for a peer limited to older groups such as 2 or 5, which it marks SHOULD NOT. Errors in this exchange are **unauthenticated**. RFC 7296 tells the recipient to keep trying for a while rather than act at once, so a single forged notify should not be read as the answer. ## Step two: IKE_AUTH and authentication `AUTHENTICATION_FAILED` covers every reason authentication can fail — RFC 7296 lists an invalid shared secret, an invalid ID, an untrusted certificate issuer, a revoked or expired certificate — and when it is returned the IKE SA is **not created**. The usual culprits between makers: 1. **Pre-shared key differs** — including one side storing a hex string and the other treating the same characters as ASCII. 2. **Identity mismatch** — one gateway sends `ID_IPV4_ADDR` and the other's policy expects an `ID_FQDN` or `ID_DER_ASN1_DN`. RFC 7296 uses the ID only to fetch the peer's policy and credentials, so an unexpected ID finds none. 3. **Certificate problems** — an ID that does not match the certificate's subjectAltName (RFC 4945), or a chain the peer cannot validate (a PKI matter, but it surfaces here). The initiator's view is thinner: RFC 7296 says a responder-side error comes back in the protected response, usually as its only payload, so whatever detail exists is in the responder's records. ## Step three: the first Child SA If authentication succeeds but the Child SA does not, RFC 7296 keeps the IKE SA alive. `NO_PROPOSAL_CHOSEN` here means the ESP transforms differ; `TS_UNACCEPTABLE` means the responder's policy accepts no part of the proposed traffic selectors — typically subnets entered differently on each side. The symptom is an established IKE SA with no traffic flowing. (How selectors map into each gateway's security policy database is the security-association subject; here it is enough to compare what each side proposes.) ## Silence instead of a notify - **No reply to `IKE_SA_INIT`** — UDP port 500 blocked on the path, or the initiator aimed at the wrong peer address. - **`IKE_SA_INIT` completes, `IKE_AUTH` gets no answer** — RFC 7383 notes that `IKE_AUTH` messages carrying certificates reach several KB, exceed the path MTU and are fragmented at IP level, and some devices drop fragments. Both peers sending `IKEV2_FRAGMENTATION_SUPPORTED` in `IKE_SA_INIT` lets them split messages into Encrypted Fragment payloads instead. ## A working order 1. Confirm both sides run IKEv2 and reach each other on UDP 500. 2. Read the responder's log for the first notify, and note which exchange carried it. 3. Compare only the parameters that exchange negotiates. 4. Change one side at a time, and re-test from a fresh `IKE_SA_INIT`.

  • Why should an IKEv2 initiator not give up at the first error notify in IKE_SA_INIT?
    Because every error in that exchange is unauthenticated: no keys exist yet, so anyone on the path could have sent it. RFC 7296 says the recipient should keep trying for some time before giving up and act immediately only where the specification defines a corrective step, as for `COOKIE`, `INVALID_KE_PAYLOAD` and `INVALID_MAJOR_VERSION`.
  • An IKEv2 IKE SA is established but no traffic passes; where do you look?
    At the Child SA negotiated in `IKE_AUTH`. A `NO_PROPOSAL_CHOSEN` there means the ESP transforms differ, and `TS_UNACCEPTABLE` means the responder accepted no part of the traffic selectors; in both cases RFC 7296 keeps the IKE SA. A narrowed selector set is another cause: the Child SA exists but covers only part of the traffic one side expects.

saying these in an interview costs you the question

  • INVALID_KE_PAYLOAD means the two gateways share no Diffie-Hellman group.
  • An established IKE SA proves the tunnel is passing traffic.
  • AUTHENTICATION_FAILED always means the pre-shared keys differ.
  • A tunnel that stalls in IKE_AUTH must have an algorithm mismatch.
  • One unauthenticated error notify is reason to stop retrying immediately.