skip to content

Why should you always specify a charset when converting between bytes and text in Java, and what goes wrong if you don't?

level: middleimportance: must knowfreq 70%

answer

  1. Bytes have no meaning as text until a charset interprets them
  2. Omitting charset -> JVM default -> different result per machine (mojibake)
  3. Standardize on StandardCharsets.UTF_8 everywhere
  4. JEP 400 (JDK 18) made default UTF-8; FileReader got charset ctor in JDK 11
  5. Wrong encoding breaks length/indexing for multibyte chars

basics

~20 s

Bytes are not text until you choose a character encoding (a charset) to interpret them. If you don't specify one, older Java uses the machine's default, so the same code produces different or corrupted text on different machines. Always pass an explicit charset, normally UTF-8.

solid answer

~40 s

A charset (encoding) is the rulebook mapping characters to bytes and back, e.g. UTF-8. Any conversion — InputStreamReader, OutputStreamWriter, new String(bytes), String.getBytes(), FileReader/FileWriter, Files.readString — involves a charset. Historically these used the JVM's *default* charset, derived from the OS/locale, so a file written as UTF-8 on Linux could be read as windows-1252 elsewhere, mangling accented characters or emoji (mojibake). The fix is to always pass an explicit charset, and standardize on StandardCharsets.UTF_8. As of JDK 18 (JEP 400) the default charset is UTF-8 everywhere, which removes one class of bugs, but file.encoding can still be overridden and legacy APIs like FileReader gained charset-taking constructors only in JDK 11 — so explicit-charset remains the rule for portable, correct code. Wrong encoding also breaks length and indexing because multibyte characters span several bytes.

code

java · 15 lines
java
import java.nio.charset.StandardCharsets;
import java.nio.file.*;

// BAD: depends on the JVM default charset -> non-portable, can corrupt non-ASCII
String risky = new String(Files.readAllBytes(path));
Writer risky2 = new FileWriter("out.txt"); // pre-JDK11: always default charset

// GOOD: explicit, portable, correct everywhere
String text = Files.readString(path, StandardCharsets.UTF_8);
try (var w = Files.newBufferedWriter(path, StandardCharsets.UTF_8)) {
    w.write("café 😀"); // 'café 😀' round-trips correctly
}

// Bridging a byte stream to text with an explicit charset
var reader = new java.io.InputStreamReader(in, StandardCharsets.UTF_8);

go deeper

for a junior

Understands that a charset is needed to turn bytes into text and should pass UTF-8 instead of relying on defaults.

for a middle

Can name the conversion points (readers/writers, getBytes/new String), explain mojibake from default-charset reliance, and prescribe StandardCharsets.UTF_8.

for a senior

Explains the JEP 400 default change, the JDK 11 FileReader charset constructors, length/indexing issues with multibyte text, and lossy round-trips.

for a principal

Sets organization-wide encoding policy (UTF-8 end to end, build/runtime enforcement), reasons about Unicode normalization, BOMs, and interop boundaries with external systems.

## Bytes vs. characters — the core idea A computer stores everything as **bytes** (numbers 0–255). Human-readable **text** is a sequence of **characters** ('A', 'é', '世', '😀'). To store or transmit text you must turn characters into bytes; to read text back you must turn bytes into characters. The rulebook that defines this mapping is a **character encoding**, called a **charset** in Java (modeled by `java.nio.charset.Charset`). Examples: **US-ASCII** (one byte, only 128 characters), **ISO-8859-1 / windows-1252** (one byte, 256 characters, Western European), and **UTF-8** (variable length, 1–4 bytes, encodes all of Unicode — every character in every language plus emoji). ## Why the same bytes can become different text The bytes alone don't tell you the encoding. The character 'é' is the single byte `0xE9` in windows-1252, but the two bytes `0xC3 0xA9` in UTF-8. So if you write 'é' as UTF-8 and another program reads those two bytes as windows-1252, it shows 'é' — garbled text known as **mojibake**. There is no way to recover the intent without knowing the encoding both sides used. ## Where conversions happen in Java Every byte<->text boundary picks a charset, explicitly or implicitly: - `InputStreamReader` / `OutputStreamWriter` — the bridge between byte streams and character streams. - `new String(byte[])` and `String.getBytes()` — decode/encode in memory. - `FileReader` / `FileWriter` — convenience readers/writers over files. - `Files.readString` / `Files.writeString` / `Files.newBufferedReader`. - `PrintWriter`, `Scanner`, `Properties.load`, parsing HTTP bodies, etc. ## The bug: relying on the default charset When you omit the charset, older Java uses the **default charset**, computed at JVM startup from the OS and locale (the `file.encoding` system property). That means the *same* program behaves differently per environment: a developer on a UTF-8 Linux box writes a file that a Windows machine (historically windows-1252) reads wrongly, or a CI server with a different locale corrupts data. These bugs are nasty because they pass on the author's machine and fail elsewhere, often only on non-ASCII characters. ## The modern landscape (JEP 400) A key date: **JDK 18 (2022) made the default charset UTF-8 on every platform** via JEP 400. That eliminates much of the historical pain. But three caveats keep 'always specify' the right habit: (1) you may run on JDK 8/11/17 for years; (2) `file.encoding` can still be set on the command line to override it; (3) `Console`/native encoding can differ. Also relevant: **`FileReader`/`FileWriter` only gained charset-accepting constructors in JDK 11** — before that they *always* used the default, which is exactly why they were dangerous. ## The fix Always pass an explicit `Charset`, and standardize on **`StandardCharsets.UTF_8`** (a constant, no checked exception, no typo-prone string). Prefer the NIO helpers (`Files.readString(path, UTF_8)`, `Files.newBufferedWriter(path, UTF_8)`) or wrap byte streams with `new InputStreamReader(in, UTF_8)`. ## Secondary correctness issues With a wrong/variable encoding, **byte length != character count** for multibyte text, so `getBytes().length`, buffer sizing, and any byte-offset indexing break. Round-tripping (decode then re-encode) can also be **lossy** if the charset can't represent a character; UTF-8 avoids this by covering all of Unicode. There's also a subtlety: `String` internally is UTF-16, and characters beyond the Basic Multilingual Plane (like many emoji) are **surrogate pairs** taking two `char`s — but that's a `String`-internals point, separate from the external byte encoding you choose.

  • Did JDK 18 (JEP 400) make explicit charsets unnecessary?
    No. It made UTF-8 the default everywhere, which removes a big source of bugs, but file.encoding can still be overridden, you may run on older JDKs, and explicit charsets document intent and stay correct regardless of environment. Specifying remains best practice.
  • Why is StandardCharsets.UTF_8 preferred over the string "UTF-8" or Charset.forName("UTF-8")?
    StandardCharsets.UTF_8 is a compile-time constant: no typos, no UnsupportedEncodingException to catch, and slightly faster. The string-based overloads can throw at runtime and are typo-prone.

A charset is a shared codebook between two spies. If the sender encodes with codebook A and the receiver decodes with codebook B, the message comes out as gibberish even though the paper (the bytes) is identical. Agreeing on one codebook (UTF-8) is the whole point.

saying these in an interview costs you the question

  • Assuming bytes 'are' text or that there is one universal encoding
  • Using FileReader/FileWriter or new String(bytes) with no charset
  • Hardcoding the locale-dependent default and shipping it
  • Believing JEP 400 means you never need to specify a charset again
  • Confusing the external byte encoding with String's internal UTF-16 representation

context

resolving links…