How do you build audio understanding when the signal is non-speech, like machine noise?

## The failure that starts the conversation Audio pipelines are usually built for speech, and the default reflex is to transcribe first and reason over text. Applied to a factory-floor recording or a field microphone, that reflex produces an empty string or, worse, a hallucinated fragment of speech, and the entire signal disappears before any reasoning begins. Recognising that non-speech audio needs a different path is most of the question. ## What the signal actually is Sound is a waveform, and almost every approach starts by turning it into a time-frequency representation, typically a mel-spectrogram, which is effectively an image of how energy is distributed across frequencies over time. A bearing developing a fault shows up as energy at particular harmonics that were not there last month. A bird call is a distinctive shape in that image. Once you see the problem this way, much of the toolkit is the vision toolkit. ## Path one: a general audio-capable model Frontier multimodal models accept audio directly, alongside text and in some cases video, and are billed per second of media rather than per token of transcript. They can describe what they hear in open vocabulary, answer questions about it, and combine it with other context in the same request. This path wins when the categories are not known in advance, when you want a description rather than a label, when the volume is modest, and when the audio needs to be reasoned about together with something else, for example a recording paired with the machine's telemetry or with the video of the same moment. It is weakest for fine-grained discrimination inside a narrow domain, where a general model has seen far less of your specific acoustic world than a targeted model has, and for anything requiring a stable, calibrated score you can threshold. ## Path two: a purpose-built classifier A supervised classifier over spectrogram features, or an audio-text embedding model that lets you match a clip against natural-language label descriptions without training examples, gives you something the general model does not: a number. Fixed taxonomy, calibrated probabilities, a threshold you can tune against a precision and recall target, and a per-clip cost low enough to run continuously on every microphone. This path wins for always-on detection over high volumes, for regulated or safety contexts where a tunable operating point is required, and for narrow domains such as industrial condition monitoring or bioacoustic species identification, where domain-trained models are strong and general models are mediocre. ## The pattern most systems converge on Cascade them. Run the cheap classifier continuously as a detector tuned for recall, and pass only flagged clips to the general model for description, context and triage. That keeps the always-on cost proportional to the cheap model and the reasoning cost proportional to the rare event. It also gives you two independent signals to disagree, which is a useful escalation trigger. ## What actually determines success Acoustic conditions, not model choice. The same bearing recorded through a cheap microphone three metres away in a reverberant hall is a different problem from a contact sensor on the housing. Background machinery, compression artefacts from the recording chain, sample rate and clipping all move accuracy more than the difference between two candidate models. That is why the only meaningful evaluation is on your own recordings, from your own hardware, in your own environment, with a labelled set that includes the near-misses and the confusable sources. Two further practicalities. Class imbalance is severe, since the anomaly you care about is rare by definition, so headline accuracy is meaningless and you should be quoting recall at a fixed false-positive rate. And labels are expensive, because non-speech audio often needs a domain expert to label at all, which frequently makes a zero-shot or description-matching approach attractive at the start, with a supervised model trained later on the data the first system helped you collect. ## Where this connects to video When the sound accompanies footage, native video input can carry both into one context, billed per second of audio on top of the sampled frames. That is often the cheapest way to catch an event that is audible before it is visible, and it is a good argument against stripping the audio track reflexively when the environment is one where sound leads.