Data Encoding and Storage
Turning Go values into bytes and rows and back: json tags and streaming, the marshaler interfaces, XML, CSV, gob and binary, database/sql's pool and transactions, and decoding hostile input.
part ofGo (Golang)overview, primer and where to startread it →on this pageshowhide
explore
- JSON Marshaling33 questions
- Evolving a Wire Struct4 questions
- Struct Tags and Mapping5 questions
- Types on the Wire5 questions
- Streaming Encoders and Decoders4 questions
- Custom Marshalers and RawMessage5 questions
- Nulls, Omitted and Defaults5 questions
- Stricter v2 Decoding5 questions
- Conversion Interfaces9 questions
- Text and Binary Marshalers5 questions
- Stringer Versus Marshaler4 questions
- Non-JSON Wire Formats32 questions
- XML Structs and Tags4 questions
- XML Token Streaming5 questions
- CSV Reading and Quoting5 questions
- Compressed Streams5 questions
- Byte Order with encoding/binary4 questions
- Self-Describing gob Streams5 questions
- Base64 and Hex Codecs4 questions
- Decoding Untrusted Input14 questions
- Size Limits and Caps5 questions
- Validating Parsed Structs4 questions
- Unknown Fields and Drift5 questions
- Relational Access31 questions
- Prepared Statement Lifetime4 questions
- sql.DB as a Pool4 questions
- Query, Scan and Rows5 questions
- NULLs and Custom Scanners4 questions
- Transactions and Rollback4 questions
- Pool Sizing and Exhaustion5 questions
- Placeholders and Prepared Statements5 questions
questions
119 · 5 sectionsWhat concrete Go types does json.Unmarshal produce when the target is an `any`?
basics
~10 sSix concrete types, and no others: map[string]any for a JSON object, []any for an array, string for a string, float64 for every number, bool for true and false, and an untyped nil for null.
In Go, what does implementing MarshalJSON on a type change about how encoding/json encodes it?
basics
~10 sImplementing MarshalJSON() ([]byte, error) makes a type satisfy json.Marshaler, so encoding/json calls that method instead of walking the type's fields, and splices the JSON bytes it returns into the output.
In encoding/json, what happens to the emitted JSON key when you rename a struct field?
basics
~20 sWithout a json tag, encoding/json emits the Go field name as the key, so renaming the field renames the key and old readers see nothing. A json tag pins the wire name, freeing the Go field to be renamed.
In encoding/json, what does the omitempty struct tag option omit, and what does it leave in?
basics
~20 sIn encoding/json, omitempty skips a field whose value is false, 0, an empty string, a nil pointer or interface, or an empty array, slice or map. It never skips a zero-valued struct such as time.Time{}, and it only affects encoding.
When should you use json.NewDecoder(r).Decode(&v) instead of json.Unmarshal on a []byte?
basics
~20 sUse json.NewDecoder when the JSON arrives as a stream, such as an HTTP body or a file, so it is decoded as bytes arrive. Use json.Unmarshal when you already hold the whole document in memory as a []byte.
Why does adding a String() method to a type leave json.Marshal's output unchanged?
basics
~10 sfmt.Stringer is consulted only by the fmt package. encoding/json never looks for String; it looks for json.Marshaler and then encoding.TextMarshaler, and a type with neither is encoded from its structure, field by field.
Why does a time.Duration field marshal to a bare number in encoding/json, and how do you get "1m30s"?
basics
~20 stime.Duration is a named int64 counting nanoseconds and implements no marshaler interface, so encoding/json encodes the underlying integer: 90*time.Second becomes 90000000000. To emit "1m30s", define your own duration type with MarshalText and an UnmarshalText that calls time.ParseDuration.
What does implementing encoding.TextMarshaler and TextUnmarshaler change about a type's encoding/json output?
basics
~20 sencoding/json writes the type as one JSON string built from the bytes MarshalText returns, instead of encoding its fields or its underlying number. Decoding hands the unquoted bytes back to UnmarshalText. json adds the quotes and escaping itself.
In encoding/json, what may a map's key type be, and what does MarshalText have to do with it?
basics
~20 sJSON object keys are strings, so encoding/json accepts a map key type only if it is a string kind, an integer kind, or implements encoding.TextMarshaler. Anything else fails with an unsupported-type error. Decoding needs the mirror: a string, an integer, or TextUnmarshaler.
Why does fmt.Println(v) skip a String() method declared with a pointer receiver?
basics
~20 sOnly the pointer type's method set contains a pointer-receiver method, so the value copy stored in fmt's interface argument does not satisfy fmt.Stringer. fmt cannot take the address of that copy, so it falls back to default formatting.
Why does a []byte struct field marshal to a base64 string in encoding/json?
basics
~20 sJSON has no type for raw bytes, so encoding/json marshals a []byte field as a base64 string using the standard padded alphabet. A nil []byte becomes null and an empty one becomes an empty string.
What do binary.BigEndian.Uint32 and binary.LittleEndian.Uint32 return for the bytes 00 00 01 00?
basics
~10 sbinary.BigEndian.Uint32 reads the most significant byte first and returns 256. binary.LittleEndian.Uint32 reads the least significant byte first and returns 65536. Same four bytes, opposite orders, different numbers.
Why must a gzip.Writer be closed before the file is valid, and what breaks if its Close error is dropped?
basics
~20 sA gzip.Writer buffers compressed data and writes the gzip trailer, a CRC32 plus the uncompressed length, only on Close. Skip Close and the file is truncated; ignore Close's error and a failed final write vanishes silently.
In Go's encoding/csv, when would you use Reader.Read in a loop instead of Reader.ReadAll?
basics
~20 sReadAll builds the whole file as one [][]string in memory and returns nil records if any row fails to parse. Read hands back one record per call and ends with io.EOF, so large or partly malformed files stay workable.
What is Go's encoding/gob package, and how do you encode and decode a value with it?
basics
~20 sencoding/gob is Go's own binary serialization format. Wrap a writer with gob.NewEncoder and call Encode to send a value; wrap a reader with gob.NewDecoder and call Decode with a pointer to receive it. Only exported fields travel.
In a Go HTTP handler, what does http.MaxBytesReader do when a client sends a body larger than the limit?
basics
~20 shttp.MaxBytesReader wraps a request body so any read past the given byte limit fails with a *http.MaxBytesError instead of continuing. It caps what one request can allocate, but the handler must still write the 413 response itself.
Why does json.Unmarshal leave a struct field at zero when a key in the JSON is misspelled, and return no error?
basics
~20 sencoding/json skips any JSON object key that matches no field of the destination struct. A misspelled key is discarded, the field keeps its zero value, and Unmarshal returns nil. Silence is the documented default, not a bug.
Why does json.Unmarshal succeed when a required field is missing from the JSON input?
basics
~20 sencoding/json has no required-field concept. A key absent from the input is simply never assigned, so the struct field keeps its zero value and Unmarshal returns nil. Requiredness is a rule your own code must check after decoding.
Why is checking r.ContentLength not a safe size cap on an untrusted HTTP request body in Go?
basics
~20 sr.ContentLength only reports what the client declared. A chunked request declares nothing and arrives as -1, so a length check silently passes exactly the requests whose size is unknown. Only a cap on bytes actually read bounds the allocation.
What does json.Decoder.DisallowUnknownFields change about a Decode call that reads an unexpected key?
basics
~20 sIt makes a key that matches no field of the destination struct an error instead of a silent skip. Decode then returns an error naming the first such key, for example: json: unknown field "maxConnnections".
Why does scanning a SQL NULL into a Go string variable fail in database/sql, and what do you scan into instead?
basics
~20 sA Go string has no value that means absent, so Scan returns an error saying converting NULL to string is unsupported. Declare the variable as sql.NullString or as *string, then check Valid or check for nil before using it.
In Go's database/sql, why pass values as arguments after the query string instead of fmt.Sprintf-ing them into the SQL?
basics
~20 sdatabase/sql never splices arguments into the SQL text. The statement and the values reach the driver separately, so the database parses the query before it sees any value. Anything you fmt.Sprintf in becomes SQL syntax rather than data.
Does sql.Open actually connect to the database, and how do you prove connectivity?
basics
~10 sNo. sql.Open only validates its arguments and hands back a connection-pool handle; it usually dials nothing. Call db.PingContext with a timeout to force a real connection and prove the database is reachable.
In Go's database/sql, what does db.PrepareContext return and why must you Close it?
basics
~20 sIt returns a *sql.Stmt, a handle to a statement the database server has already parsed and is holding open. Stmt.Close discards it on every pooled connection it was prepared on; skip that Close and the server-side state stays.
In Go's database/sql, how do you run a multi-row query and read every row?
basics
~10 sCall db.QueryContext, defer rows.Close(), then loop while rows.Next() calling rows.Scan with a pointer per selected column, and check rows.Err() after the loop. Next returning false means end of data or a failure.