A rarely covered language framing returns refused content, but degraded - is that a finding?

### One result, two claims - keep them apart A framing in an under-covered language or register comes back with content that would be declined in plain dominant-language phrasing, but the content is wrong in places. Whether this is a finding depends on which claim you are making. **Claim A: the trained refusal did not engage for this form.** The degraded answer supports this directly. The model attempted the task instead of declining it, which is behaviour about the alignment coverage, not about the quality of the output. This is a real result and it is the one worth filing. **Claim B: an adversary obtains content they can act on.** The degraded answer does not support this. Severity in this family is graded by what survived, and a fluent artefact with the operative specifics wrong is not the same outcome as a clean answer. Reporting the two as one thing is the most common error here, and it pushes triage in both wrong directions at once: over-fix on a result that was mostly noise, or dismiss a result that was substantively complete. ### It is a measurement, so give it a denominator Generation is probabilistic and these regions are thin rather than empty, so the same framing may be declined on the next attempt. A result with no denominator cannot be compared with anything - not with a re-run, not with another form, not with the same form after a model update. What belongs in the record: - how many attempts were made; - how many produced substantive output rather than a refusal or drift; - the scoring rule for "substantive" - who judged it and against what; - the date and the deployment it was run against. One success in five is a legitimate result. It reports a weak trained preference in that region. What it does not support is any claim of reliable access, and a reader who sees only "it works" will make exactly that inference. ### The verification problem is the hard part The judgement "partly wrong" requires someone who reads the form. Where the whole point of the framing is that the team has no such reader, the quality assessment is itself unverified - and so is the claim that the content is objectionable at all, since a fluent-looking output could be off-topic. Two traps follow: - **Back-translation is not neutral.** Passing the output through another model or a translation service to read it adds a second lossy transformation, one that can smooth over incoherence or supply detail the original never contained. A back-translated transcript is evidence about the translation, not only about the original. - **An unread output cannot be re-scored.** If nobody can evaluate it, the entry cannot be confirmed later either, which affects whether it should be carried at all. The honest move is to state the verification status alongside the result: judged by a fluent reader, judged by machine translation, or unjudged. ### Is it a bug or a design limit? Both framings appear in triage, and the distinction is genuine. A specific form being thin is a coverage state that moves - it may close on its own as alignment data is extended. The general property, that competence generalises further than safety data, is structural: it is recreated wherever a capability reaches users ahead of its demonstrations. A report that names only the specific form invites a narrow response scoped to that form; a report that names the property alongside it says what class of result to expect next. Filing at both levels is what distinguishes a useful report from an anecdote. ### What a strong answer sounds like State what fired and what did not. Give the rate and the sample. Say who judged the output and in what language. Grade severity on surviving substance rather than on the presence of text. And say plainly which of the two claims you are making, because an interviewer is listening for whether you can tell them apart under pressure.