In PHP, how do stream filters and the compress.zlib:// wrapper let you gzip or base64-encode a large file while streaming it?
answer
- filters sit on read and write chains
- stream_filter_append() and STREAM_FILTER_WRITE
- convert.base64-encode, zlib.deflate
- php://filter/read=…/resource=…
- compress.zlib:// writes a real .gz file
basics
~20 sstream_filter_append() attaches a filter such as convert.base64-encode or zlib.deflate to a handle's read or write chain, so data is transformed chunk by chunk. compress.zlib:// opens a gzip file as a stream. Either way stream_copy_to_stream() moves the data without loading it whole.
solid answer
~40 sA **stream filter** transforms data as it passes through a handle. `stream_filter_append($h, 'convert.base64-encode', STREAM_FILTER_WRITE)` adds one to the write chain, so every `fwrite()` is encoded on the way out; `STREAM_FILTER_READ` works on reads, and by default the chain follows the open mode. `stream_get_filters()` lists what is available: `string.rot13`, `string.toupper`, `string.tolower`, `convert.*`, `zlib.*` when zlib is loaded, and others. For one-shot functions there is `php://filter/read=convert.base64-encode/resource=big.bin`. Note that `zlib.deflate` writes raw DEFLATE data with no gzip header; to produce a `.gz` file, open `compress.zlib://export.csv.gz` with `'wb'` and copy into it with `stream_copy_to_stream()`. Buffered output from a filter is flushed when the filter is removed or the stream is closed, so close before using the result.
go deeper
Recall stream_filter_append(), the read and write chains, and a few built-in filters such as convert.base64-encode and zlib.deflate.
Explain how the mode picks the chain, why removing the filter flushes it, and how php://filter applies filters to one-shot functions.
Show you stream large transforms with stream_copy_to_stream(), choose compress.zlib:// for real gzip files, and verify filters and wrappers exist on the target build.
Weigh in-process streaming transforms against handing compression or encoding to the web server, a queue worker or storage, based on CPU cost and request latency.
## Filters: transform data in flight A **stream filter** is a piece of code attached to an open stream that processes data in chunks as it is read or written. Each stream has a **read chain** and a **write chain**: - `stream_filter_append($stream, $name, $mode = 0, $params)` adds a filter at the end of a chain; `stream_filter_prepend()` adds it at the start. - `$mode` is `STREAM_FILTER_READ`, `STREAM_FILTER_WRITE` or `STREAM_FILTER_ALL`. With the default, the filter goes on the read chain if the stream was opened for reading and on the write chain if opened for writing. - The function returns a filter resource, or `false` if the filter name is unknown; `stream_filter_remove($filter)` detaches it and flushes what it still holds. Because each filter sees one chunk at a time, memory stays small no matter how large the file is. ## Built-in filters `stream_get_filters()` shows what the running PHP offers. Common ones: | Filter | Does | |---|---| | `string.rot13`, `string.toupper`, `string.tolower` | simple text transforms | | `convert.base64-encode`, `convert.base64-decode` | Base64 | | `convert.quoted-printable-encode`, `-decode` | quoted-printable | | `zlib.deflate`, `zlib.inflate` | raw DEFLATE (zlib extension) | | `bzip2.compress`, `bzip2.decompress` | bzip2 (bz2 extension) | | `convert.iconv.*` | character-set conversion (iconv extension) | | `dechunk` | decodes HTTP chunked transfer encoding | `zlib.deflate` produces **raw** DEFLATE data by default (no gzip header or checksum trailer), which is not a `.gz` file. Its parameters (`level`, `window`, `memory`) can change that, but for gzip files the wrapper below is simpler. ## compress.zlib:// for gzip files With the zlib extension, `compress.zlib://path` opens a gzip file as a stream, the equivalent of `gzopen()` but usable with every file function. Reading decompresses, writing compresses: ```php <?php declare(strict_types=1); $in = fopen('/var/app/exports/ledger.csv', 'rb'); $out = fopen('compress.zlib:///var/app/exports/ledger.csv.gz', 'wb'); if ($in === false || $out === false) { throw new RuntimeException('cannot open export files'); } stream_copy_to_stream($in, $out); fclose($in); fclose($out); // finishes the gzip stream ``` `stream_copy_to_stream($from, $to)` copies in chunks and returns the number of bytes copied, or `false`. Neither side is ever fully in memory. Note the three slashes: `compress.zlib://` followed by the absolute path `/var/…`. ## Base64 while streaming ```php <?php declare(strict_types=1); $in = fopen('/var/app/exports/ledger.csv.gz', 'rb'); $out = fopen('php://temp', 'w+b'); $filter = stream_filter_append($out, 'convert.base64-encode', STREAM_FILTER_WRITE); stream_copy_to_stream($in, $out); stream_filter_remove($filter); // flush the last partial Base64 group rewind($out); ``` The same can be done for a read with the `php://filter` meta-wrapper, which applies filters at open time and suits one-shot functions: ```php $encoded = file_get_contents('php://filter/read=convert.base64-encode/resource=/var/app/logo.png'); ``` `php://filter` accepts `read=`, `write=` and a required `resource=` at the end; several filters are separated by `|`. ## Pitfalls 1. **Unflushed output.** Encoders and compressors hold partial data. Remove the filter or close the stream before reading the result; otherwise the last bytes are missing. 2. **Wrong chain.** A filter on the read chain does nothing to `fwrite()` calls, and vice versa; pass the mode explicitly. 3. **Seeking.** Seeking does not reset a filter's buffered state, so treat a stream as forward-only while a filter is attached. 4. **Missing extension.** `zlib.*`, `bzip2.*` and `compress.zlib://` exist only when the matching extension is loaded; check `stream_get_filters()` and `stream_get_wrappers()`. ## When filters are the right tool - **Large payloads with a simple transform**: encoding an attachment, compressing an export, converting a character set chunk by chunk. - **One-shot functions that need a transform**: `php://filter` lets `readfile()` or `file_get_contents()` apply one without a handle. - **Pipelines of several steps**: filters stack in order, for example decompress then change case. They are the wrong tool when the transform needs the whole document at once, such as parsing JSON or XML, or when a single function call like `base64_encode()` on a small string is clearer. ## Writing your own A class extending `php_user_filter` and implementing `filter($in, $out, &$consumed, bool $closing): int` can be registered with `stream_filter_register('app.redact', RedactFilter::class)` and then appended like a built-in. It returns `PSFS_PASS_ON`, `PSFS_FEED_ME` or `PSFS_ERR_FATAL`.
- Why is the output of the zlib.deflate filter not readable by gunzip?By default the filter emits raw DEFLATE data: no gzip header, no CRC and length trailer. `gunzip` expects the gzip container. Either write through `compress.zlib://file.gz`, which produces a real gzip file, or pass the filter a `window` parameter that selects gzip framing, which is easier to get wrong.
- What goes wrong if you read a php://temp buffer before removing a base64 write filter?Base64 works on 3-byte groups, so the filter keeps up to two trailing bytes waiting for more input. Until the filter is removed or the stream closed, those bytes are not written and the encoded output is truncated. Call `stream_filter_remove($filter)` first, then `rewind()` and read.
- How do you apply a filter to file_get_contents(), which never gives you a handle?Use the `php://filter` meta-wrapper: `file_get_contents('php://filter/read=string.toupper|convert.base64-encode/resource=/path/file')`. The `read=` list is applied in order as the one-shot function reads, and `resource=` must come last.
A stream filter is a water filter fitted to a pipe: water is treated as it flows, whatever the size of the tank. Pull the filter out and the last drops still inside it come through; leave it in and they stay trapped.
saying these in an interview costs you the question
- zlib.deflate produces a standard .gz file that gunzip can read.
- Stream filters load the whole file before transforming it.
- A filter appended to the read chain also transforms fwrite() calls.
- Filtered output is complete as soon as the last fwrite() returns.
- compress.zlib:// works even when the zlib extension is not loaded.