From tokens to terabytes: Building reactive generative media pipelines
For the first three years of the generative AI wave, the output of a model was a string. You called an API, you got tokens back, you rendered them in a chat window or wrote them to a row in Postgres. The economics of that pipeline were dominated by inference cost. Storage was a rounding error.
That era is over. The output of a modern generative pipeline is an asset: a 4K video clip, a stem-separated audio track, a 50-megapixel product render, a 3D mesh with PBR textures. Generative AI has gone from text-centric to asset-centric, and the architectural center of gravity has moved with it. The teams building durable advantages in generative media right now are the ones treating their storage layer as a pipeline component rather than a destination.
This is a good problem. It is the problem you get when your pipeline works.
Asset-centric changes the shape of the system
Text pipelines are stateless in practice. A prompt goes in, a response comes out and the interesting state lives in a database. You can rebuild almost any artifact by re-running the call.
Media pipelines are not like that. Every stage produces a large, opaque binary that the next stage consumes. A single finished deliverable might traverse a dozen of them: prompt expansion, base generation, upscale, frame interpolation, color pass, audio generation, mix, mux, transcode to delivery formats, thumbnail extraction. Each stage writes an intermediate. Each intermediate is expensive enough to regenerate that you keep it.
The result is a system where the objects are the state. Your object store stops being a place you put things when you are finished and becomes the substrate the pipeline runs on.
Adoption is past experimentation, and the volume is in production
Advertising has the clearest numbers. IAB’s 2026 Digital Video Ad Spend and Strategy Report finds that nearly two in three digital video buyers now use generative AI for creative, up from half in 2025. A third of their ad assets are expected to involve generative AI this year, up from a quarter in 2025, with buyers projecting 43 percent by 2027. That is happening inside a U.S. digital video ad market IAB projects will pass $80 billion in 2026, growing 11 percent year over year, nearly 20 percent faster than the total ad market.
The interesting detail for architects is what the creative is used for. IAB’s prior-year data showed buyers reaching for generative AI specifically to produce audience-specific versions of an ad, visual style variations and contextually adapted cuts. That is not one asset per campaign. That is a matrix.
Games are the instructive counterexample. GDC’s 2026 State of the Game Industry puts generative AI use at 36 percent of industry professionals and 30 percent at game studios specifically, but the usage breakdown is dominated by language models rather than media generation: research and brainstorming at 81 percent, code assistance and routine writing at 47 percent each, prototyping at 35 percent. The most-used tools are ChatGPT, Gemini and Copilot. Sentiment is sharply negative, with 52 percent saying generative AI is having a negative impact on the industry, rising to 64 percent among visual and technical artists. Asset-centric pipelines have not landed in games the way they have in advertising, and the constraint is as much workforce and provenance as it is tooling.
Elsewhere, the pattern holds even where the survey data is thinner: e-commerce teams generating on-model imagery per SKU per segment, localization pipelines producing dubbed and lip-synced variants per market, previsualization work that used to require an art department.
What the adopting categories have in common is that none of them produce one asset per request. They produce a set. The pipeline is judged on how many viable options it surfaces, which means a better pipeline is, definitionally, a pipeline that writes more bytes.
Reactive architecture, because the model layer will not hold still
The model landscape resets on a cadence measured in weeks. A new video model ships with better temporal coherence. A new audio model handles multilingual prosody properly. A new image model finally gets text rendering right. If your pipeline requires an engineering sprint to adopt a new model, you are structurally behind teams whose pipelines do not.
Reactive architecture is the answer, and it means two specific things.
- Model-agnostic stages. Each stage of the pipeline should express a contract in terms of inputs and outputs, not in terms of a vendor. A generation stage takes a prompt and conditioning assets and produces a video at a declared resolution and duration. Which model backs it is configuration. Swapping providers should be a config change and an eval run, not a refactor.
- Event-driven orchestration. Polling-based orchestration couples your stages to a scheduler and makes each new stage a change to the control plane. Event-driven orchestration inverts it: a stage completes, it writes its output, the write itself is the signal that the next stage should start. Adding a stage means subscribing to an event, not modifying a DAG definition that six other teams depend on.
This is where storage stops being passive. Object storage that emits events on write lets your bucket act as the message bus for the pipeline. B2 Event Notifications send a signed HTTP POST to a webhook endpoint when objects are created, updated or deleted, with rules scoped per bucket and filterable by prefix. That prefix filter is the part that matters architecturally: if your bucket is organized by stage, a rule on stage/upscale/ is a subscription to “upscale finished” without any code knowing what upscale is. Custom headers on the notification carry auth tokens or context to the target, so the endpoint can be a queue, a serverless function or a workflow platform rather than a service you had to build.
A completed upscale triggers the color pass. A completed mux triggers the transcode fan-out. A completed transcode triggers the CDN warm and the catalog write. The storage layer sequences the work, which removes an entire class of orchestration glue from your codebase and removes polling latency along with it.
Quality improvements arrive as file size increases
Every generation of media models improves along axes that all cost bytes. Resolution goes up. Frame rate goes up. Duration limits extend. Bit depth and color fidelity improve. Audio moves from mono to multi-channel. Compression artifacts that were acceptable at 720p are not acceptable at 4K, so teams move to higher bitrates and, for anything entering a post pipeline, to intermediate codecs.
The arithmetic is worth doing explicitly. A 10-second clip in a delivery-grade H.264 4K encode at 50 Mbps is roughly 60 MB. The same ten seconds as a ProRes 422 HQ intermediate, which Apple targets at 884 Mbps for 3840×2160 at 30p, is 1.1 GB. That is roughly 18 times the size, and intermediates are exactly what you keep between stages. Now assume your pipeline generates eight candidates per brief because your creative director wants options, and each candidate produces four intermediates before final. That is one brief consuming tens of gigabytes.
Nobody plans for that in a proof of concept. Everybody encounters it in month four of production.
The iteration multiplier
Here is the part that surprises teams: robustness and storage growth are the same curve.
A fragile pipeline produces one output per request because that is all it can manage. A robust pipeline produces candidates, keeps the rejects for training and eval, versions every asset so a creative decision can be reverted, retains intermediates so a late note does not require regenerating from the prompt, and derives proxies, thumbnails and per-platform cuts from every approved master.
Each of those is the correct engineering decision. Together they mean that improving your pipeline increases your storage footprint superlinearly relative to your output volume. If your unit economics assume storage scales with delivered assets, they are wrong. Storage scales with attempts multiplied by stages multiplied by versions multiplied by derivatives.
This is why storage strategy has to be a design input rather than a line item you discover on an invoice. The two things that turn it from a manageable cost into a structural problem are egress pricing and the absence of a lifecycle policy. Egress hurts most in the exact architecture described above because a multi-stage pipeline repeatedly reads its own intermediates, and a distribution layer constantly reads masters. When every read carries a metered charge, the pipeline design that produces the best creative output is also the one that produces the worst bill, and teams start making architectural compromises to protect margin. Lifecycle policy hurts by omission: if you never decide what an intermediate is worth after 30 days, you pay to keep all of them forever.
What to put in place now
If generative media is core to what you are building, four decisions determine whether your storage layer accelerates you or constrains you:
- Choose a storage economic model that does not penalize reads. Understand your egress terms before your architecture depends on them. A pipeline that reads its own outputs at every stage is a read-heavy workload, and pricing that assumes write-once, read-rarely does not fit it. Model the ratio you actually expect: egress as a multiple of stored volume, not as an absolute. That ratio is the number to design against.
- Make writes trigger work. Use object-level event notifications as the pipeline’s signaling mechanism. This buys you loose coupling, lower latency between stages and the ability to add a stage without touching the orchestrator.
- Attach metadata at write time. Model version, prompt hash, parent asset, generation parameters, approval state. Metadata written at generation time is nearly free. Reconstructing provenance across a million objects later is not; provenance is what makes your rejected candidates usable as eval data and a training signal.
- Define lifecycle policy per artifact class. Masters, approved derivatives, intermediates and rejects have different retention values. Encode that as policy on day one rather than as a cleanup project in year two.
Which points at the useful way to think about the storage layer: in a pipeline where everything else churns, it is the constant. Models turn over every few weeks. Stages get swapped, added and reordered around them. Output volume compounds with every quality improvement. The one layer absorbing all of that without being redesigned is the one holding the assets, so it is worth choosing based on the characteristics that stay true while the rest moves. That is what we built B2 for. Always hot, so no stage waits on a rehydration to read what the last one wrote. No retention minimum or file size floor, so intermediates that were always disposable cost what they used. Egress scales to what you store rather than metered per read, so a pipeline that reads its own output is not penalized for being good at its job. The architecture above it should change every quarter. The storage underneath it should not have to.
The opportunity
The teams that will win in generative media are not the ones with privileged access to a model. Model access is converging toward commodities. The advantage is in the pipeline: how fast you can adopt a better model, how many candidates you can afford to generate, how much history you retain to evaluate and fine-tune against, and how cheaply you can move all of it.
Every one of those is a storage architecture question. Treat the storage layer as an active participant in the workflow and it becomes the thing that lets you iterate faster than your competition. Treat it as a bucket you dump finished files into, and it becomes the ceiling on how good your pipeline is allowed to get.
The assets are the product now. Architect accordingly.












