Domain-Specific Languages

## Where the grammar lives The core distinction is where the language's grammar lives. - An **internal DSL**, sometimes called an embedded DSL, has no grammar of its own at all: it is ordinary source code in a general-purpose host language (Kotlin, Ruby, Java, JavaScript) that has been shaped, through naming conventions, method chaining, trailing lambdas, or operator overloading, to read like a small vocabulary for one problem. - An **external DSL**, by contrast, defines an entirely new concrete syntax with its own keywords, punctuation, and structure, described by a grammar (see EBNF-style rules). That grammar is fed into a lexer and parser - hand-written or generated by a tool such as `ANTLR` or `Xtext` - which turns source text into an abstract syntax tree specific to the DSL. When you write a Gradle `build.gradle.kts` file with `plugins { kotlin("jvm") }` or an RSpec test with `describe "Cart" do ... end`, you are writing completely ordinary Kotlin or Ruby - it compiles/parses with the host compiler, gets syntax highlighting and autocomplete from the existing IDE plugin for that language, and can freely call any other library in the ecosystem. `SQL`, regular expressions, Dockerfiles, and Terraform's HCL are all external DSLs: none of them are valid programs in any general-purpose language, and none of their tooling (syntax highlighting, linting, autocomplete) comes for free from an existing language plugin. ## Why a DSL exists at all DSLs of either kind exist to close the gap between how domain experts think about a problem and how a general-purpose language forces them to express it. A general-purpose language is optimized for expressing arbitrary computation; a well-designed DSL is optimized for expressing one narrow domain's concepts directly, cutting out incidental complexity (loops, type declarations, boilerplate) that has nothing to do with the domain itself. - A pricing-rules DSL lets a business analyst write `if customer.tier == GOLD then discount 10%` instead of navigating a general-purpose object model. - A build DSL lets a developer describe *what* artifacts to produce instead of *how* to invoke each compiler step by hand. ## The trade-off The trade-off between the two approaches is fundamentally a **build-cost-versus-freedom** trade-off. - **Internal DSLs are dramatically cheaper to stand up:** there's no grammar to design, no parser to write and maintain, and the host IDE's existing plugin supplies syntax coloring, error squiggles, and refactoring tools automatically. The price is that the DSL's syntax is bounded by what the host language's grammar allows - you cannot invent new keywords or infix operators the host doesn't support, and every DSL 'sentence' is still fundamentally a function call or object construction under the hood, which can leak host-language noise (parentheses, generics, exception stack traces) into what was supposed to be a clean domain notation. - **External DSLs remove that ceiling entirely:** you can design exactly the notation the domain calls for, including notations aimed at non-programmers, and you can enforce much stricter validation because the grammar itself simply disallows constructs you don't want expressible. The cost is that you now own an entire miniature language-tooling stack - parser, error recovery, editor integration, versioning of the grammar itself - that has to be built and maintained indefinitely. ## Failure modes 1. **On the internal side.** A classic failure mode on the internal-DSL side is when the abstraction leaks so badly that users start writing arbitrary host-language logic inside what was meant to be a declarative configuration surface - for example a build script that does network calls or non-deterministic branching during evaluation, defeating the whole point of having a declarative DSL and making builds unreproducible. 2. **On the external side.** A classic failure mode on the external-DSL side is under-investing in tooling: a hand-rolled parser with cryptic 'unexpected token' errors and no IDE support turns the DSL into a second, undocumented programming language that only its original author can debug, which becomes an organizational liability once that person moves on. ## A concrete contrast A concrete real-world contrast: **Gradle** deliberately supports both a Groovy and a Kotlin internal DSL for build scripts precisely because its authors (developers) benefit from full language power, IDE tooling, and the ability to drop into imperative code when needed. **Terraform**, by contrast, chose HCL, a purpose-built external DSL, because its authors wanted infrastructure descriptions to stay declarative, statically analyzable, and safely diff-able in code review - properties that are much easier to guarantee when the grammar itself forbids arbitrary imperative logic rather than merely discouraging it by convention.