How On-Device AI Works (and Why We Built It This Way)

The Privacy Problem With Cloud AI

Most AI tools work the same way: you type something into a text box, it travels across the internet to a company's server, a language model processes it on their hardware, and the result comes back to your screen. This happens in milliseconds, and it feels instant — but what actually happened during that round trip matters more than most people realize.

Your input — which might be a product idea, a business concept, a half-formed thought you are not ready to share — was sent to a server you do not control, processed by a model you cannot inspect, and potentially logged under a privacy policy you did not read. The company operating that server now has your idea. They might use it to train future models. They might store it indefinitely. They might be compelled to share it in response to a legal request.

For a tool designed to capture early-stage, sometimes half-formed product ideas, this is a dealbreaker. The ideas you put into a PRD generator are your competitive advantage. They are the concepts you are exploring before you have decided whether to pursue them. Sending them to a third-party server defeats the purpose of having a private tool in the first place.

What "On-Device" Actually Means

When we say the AI runs on your device, we mean it literally. The language model — the file of numbers that understands your input and generates the PRD — is downloaded once to your browser and cached locally. After that first download, every calculation happens on your own hardware. Your input never leaves your computer. There is no request to an AI server during inference, because your own machine is the AI server for that moment.

The flow works like this: the browser downloads the model file, caches it so the next visit is instant, compiles it into something your hardware can execute, and then runs your input through it to produce the result. The interesting engineering is in that third and fourth step, because a browser was never designed to do heavy math, and two technologies had to arrive before this was practical.

The Two Engines: WebGPU and WebAssembly

Running a language model is mostly a huge pile of matrix multiplication — the kind of math that GPUs are built for. There are two ways to do this in a browser, and modern browser AI uses both.

WebGPU WebGPU is the fast path. It gives web pages direct access to your graphics card. A GPU is built to do thousands of small calculations at once, which is exactly the shape of neural-network math. When WebGPU is available, the model runs on your GPU and feels genuinely responsive — tokens appear as fast as you can read them.

WebAssembly WebAssembly is the compatibility path. It is a way to run compiled, near-native code on the CPU. It is slower for this kind of work than WebGPU, but it runs almost everywhere. When a GPU is not available — on older hardware, in certain browsers, or on devices with limited graphics capability — WebAssembly keeps the tool working. It degrades gracefully instead of failing.

Most browser AI tools automatically detect which engine is available and use the best one. If your browser supports WebGPU, you get GPU-accelerated inference. If not, you get WebAssembly. Either way, the model runs on your device.

How the Model Fits in Your Browser

A full-size language model can be several gigabytes. That is too large to load into a browser tab without preparation. The solution is quantization — compressing the model's numbers from high precision (16 or 32 bits per number) down to lower precision (4 or 8 bits). This shrinks the model dramatically — a model that was several gigabytes at full precision might become under a gigabyte quantized — with a small, usually acceptable drop in quality.

Quantization is the single most important technique for browser AI. It is what lets a laptop or a phone hold a language model in memory at all. A 4-bit version of a model that would otherwise require a server GPU can run on a consumer laptop with 8GB of RAM.

The compressed model is downloaded once, cached in your browser's storage (typically IndexedDB), and reused across sessions. The first visit takes a moment while the model downloads. After that, it loads from cache — near-instant, even offline.

What Happens to Your Data

Nothing. This is the core architectural guarantee. When you type your product idea into the tool, it stays in your browser's memory. The inference happens locally. The result appears on your screen. No HTTP request carries your input to an external server. No response payload carries it back. There is nothing to intercept, log, or leak during the inference process.

This is not a policy promise or a contractual guarantee — it is an architectural fact. The model is on your device. Your input is processed on your device. The output is displayed on your device. At no point does your idea travel across the internet to a server you do not control.

For a tool that handles early-stage product concepts — ideas that might become your next startup, your next feature, your next competitive advantage — this matters. The privacy is not a feature we added. It is the reason we built the tool this way.

Why Not Just Use a Cloud API?

Cloud AI APIs are faster to build with, cheaper to start, and produce higher-quality output from larger models. We could have built a PRD generator in a weekend using OpenAI or Anthropic's API. It would have been easier to build, easier to maintain, and the output would have been arguably better.

But it would have required every user to send their product ideas to a third-party server. For a tool whose entire value proposition is capturing early-stage thinking privately, that is a contradiction we were not willing to accept.

The tradeoff is real: browser-based models are smaller, sometimes slower, and occasionally produce lower-quality output than the largest cloud models. But the privacy guarantee is absolute. For many users — founders exploring concepts, PMs documenting internal tools, freelancers writing specs for clients — that tradeoff is worth it.

The Fallback: When Browser AI Is Not Available

Not every browser or device supports the technology this requires. Older browsers, devices with very limited memory, or environments where WebGPU and WebAssembly are both unavailable will not be able to run the on-device model.

When that happens, the tool does not stop working. It falls back to a lighter, still fully local, rule-based generation mode. This fallback does not use a language model at all — it uses structured templates and deterministic logic to produce a PRD from your input. The output is more formulaic, but it is still private, still local, and still useful.

We would rather produce a good-enough document locally than a perfect one remotely. The privacy boundary is non-negotiable.

The Bigger Picture

On-device AI is not just a privacy feature. It is a fundamentally different architecture that changes what is possible. Offline capability — the tool works without an internet connection after the initial model download. Zero latency — no network round-trip means the response feels instant. Zero cost — no per-request API fees means the tool can be free forever.

The browser is becoming an edge compute node. WebGPU gives web pages access to GPU compute. WebAssembly gives them near-native CPU performance. Together, they turn a browser tab into an environment capable of running real AI models — not toy demos, but useful, private, offline-capable intelligence.

We built the Free PRD Generator on this architecture because we believe the best tools are the ones that respect your data by design, not by policy. Your product ideas are yours. They should stay on your device.