Run AI on Your Website. Skip the API Bill. (WebLLM)

0:00 Usually, when someone uses AI on your website, their message travels to a server, and another request gets added to your bill. Web LLM can move that generation onto the visitor's own GPU.

0:11 So, how does a browser run an AI model, and what are you giving up? The useful promise is no inference API bill for requests that run locally. Your website still needs hosting, the model needs

0:22 delivering, and somebody's computer still does the work. Take a writing assistant that rewrites selected text. Someone highlights a sentence, clicks rewrite, and gets a version to review.

0:33 Each use adds another generation request. In the usual cloud setup, the browser sends that selection to your back end. Your back end calls an inference provider, which runs the model

0:43 and sends the result back along the same route. That provider can charge for the input and output. A popular rewrite button can become a recurring expense. Web LLM changes where that request

0:53 finishes. Your website provides the editor and loads a supported model. Once it's ready, the selection goes to an inference engine inside the browser,

1:01 which returns an answer without calling your inference provider. If the website isn't paying for the computation, who is, and is that trade worth it? Follow the rewrite button through its first

1:10 visit, when the browser version has to prepare the model. Before it can answer, the browser needs the model. The model's learned parameters are stored in files called weights. Web LLM also needs a

1:21 compatible model library, the tokenizer that turns text into model inputs, and configuration telling those pieces how to work together. Those assets arrive over the internet. In the project's

1:32 standard configuration, model files come from Hugging Face, while compiled model libraries have their own URLs. A developer can choose other locations. The model wasn't already hiding inside

1:42 the browser. Web LLM downloads missing assets and prepares the runtime. Its documentation warns that a first load can take substantial time. Show loading progress before someone wonders

1:53 whether the rewrite button has broken. By default, Web LLM caches model assets using the browser's cache API. That's storage managed by the browser on the visitor's device. So, a later load can

2:05 reuse those files instead of downloading them again. It still needs to prepare the model for execution, so cache doesn't mean instantly ready. Think of the cached weights as a book on a shelf.

2:15 To use them, the runtime still has to open the book and put the working material on the desk. Here, the shelf is browser storage and the desk is working memory. They're different resources with

2:26 different limits. Browser storage also isn't a permanent installation promise. People can clear site data and browsers can evict stored data when space is tight. Your app needs to handle a

2:36 missing cache, and if you want the whole website to reopen offline, its page and application code need an offline loading strategy, too. To test offline generation, load the app and model,

2:47 disconnect the network, then submit a fresh rewrite. Keep the original and the new answer visible. An old answer in a chat window wouldn't prove generation happened offline. Once loaded, the model

2:58 doesn't need an internet round trip for each new token. So, what turns a sentence into a different sentence inside a webpage? There are three main parts to keep track of. The weights

3:07 contain the learned parameters. Web LLM coordinates the inference runtime. WebGPU gives that runtime access to the device's GPU for the numerical work. The page supplies the instruction and shows

3:18 the response as it arrives. WebAssembly helps with CPU work in the runtime. It's a way to run compiled code inside the browser. While WebGPU handles GPU operations, the browser is providing the

3:30 execution environment. It isn't supplying a language model of its own. The particular model still has to be supported by the engine. For example, select this sentence: We might ship on

3:40 Friday if the tests pass. Ask the assistant to make it clearer while preserving its meaning. That last instruction matters. A rewrite that promises a Friday release has changed

3:49 the product's message, even if it sounds more confident. First, the runtime turns the instruction and selected sentence into tokens, the chunks of text the model works with. It processes that

3:59 input using the loaded weights. That initial processing is often called prefill. It prepares the state the model needs to start generating a response. Then the runtime runs the model to

4:09 produce a next token, adds it to the response, and continues from that updated state. Web GPU carries the GPU operations to the visitor's hardware. Text appears in the editor as those

4:20 generated pieces return until generation reaches a stopping condition or its output limit. The weights stay available while that happens. The browser isn't downloading a fresh model for every

4:30 word. It's repeatedly applying the same learned parameters to an input and a growing response. That's why moving the weights onto the device can remove the server call from this part of the

4:39 feature. Heavy processing still has to coexist with typing, scrolling, and clicking. Web LLM supports running its engine in a web worker, a separate thread inside the browser. The interface

4:50 sends the worker a request. The worker manages generation, and messages bring the output back to the page. That separation helps keep the interface responsive. It doesn't create a second

5:00 GPU or reduce the model's memory needs. You can give the user a working cancel button while generation is running, but the computer still has a real job to finish or stop. Now the useful part is

5:11 deciding which jobs deserve that download. Rewriting a short selection is a reasonable task to evaluate because the input is bounded and the user can compare the suggestion with the

5:20 original. Keep the original available and let the user accept or reject the change. For our Friday sentence, check that the rewrite preserves uncertainty and the condition about passing tests.

5:31 Might and if aren't clutter to remove. Turning them into a firm promise fails this test. The same editor could summarize a selected passage. Evaluate whether the summary keeps the

5:42 important facts, exceptions, and names. It only receives the text your application gives it, so selecting one paragraph doesn't automatically give the model the rest of the document or

5:51 current information from the web. You could also ask it to classify a message, perhaps as a bug report or a feature request. Check the chosen label against examples you've reviewed. If you

6:02 need a fixed response format, Web LLM supports structured output, but a valid format doesn't guarantee the classification is correct. Build a collection of real inputs, including

6:12 awkward ones, and decide what counts as a useful answer. Run the same collection when you change the model or its settings. Model size sets a limit on how easy that will be to deliver. A larger

6:22 model may offer capabilities you need, but the visitor has to download its files and fit its working state into memory. Shrinking the weights can help. That's where quantization comes in,

6:32 storing parameters with fewer bits of precision. Take an illustrative model with 1 billion parameters. At 16 bits per parameter, the raw weights account for 2 GB. At 4 bits, that arithmetic

6:44 becomes half a gigabyte. That's a storage calculation before extra metadata and runtime memory. It isn't a measurement of a particular Web LLM model. The actual memory requirement

6:54 also includes working buffers and the state used while processing the prompt and generating text. For attention models, keeping more context can increase that state. So, a download that

7:04 fits on disk can still fail when the runtime tries to allocate memory for it. Lower precision can affect output quality, too. The file getting smaller doesn't tell you whether the Friday

7:13 sentence still means the same thing. Recheck the task after changing the model's precision. Otherwise, you might save memory by making the feature less useful, which is a fairly expensive way

7:23 to save money. Compatibility is another check to make before downloading a large file. Web GPU needs a suitable browser and device environment, and production pages

7:32 need a secure context, normally HTTPS. Detecting the API is only the start. The request for a usable GPU adapter can still fail. The model can also require capabilities

7:43 that the available adapter doesn't provide. Web LLM's model records include requirements such as GPU features alongside memory estimates. Treat those estimates as guidance, then

7:54 test loading and generation on the devices your visitors actually use. The user's machine pays in memory, processing time, and energy. That can mean battery use in competition with

8:04 other work. Measure these costs on representative devices with the model you intend to ship. But your side of the bill changes, too. Delivering a large model file has bandwidth costs under the

8:14 hosting arrangement you choose. Model updates can require another download. If visitors only need one short rewrite, the setup cost may be a poor exchange for avoiding that single cloud request.

8:25 Privacy needs the same attention to the whole application. The inference engine can process the selection locally, but analytics, error reporting, save documents, or a cloud

8:35 fallback can still send text away. Inspect those routes before saying that a person's writing stays on their device. A network inspection should include loading, generation, and what

8:45 happens afterward. An offline answer shows that this generation didn't need the network. It doesn't prove the app will never upload anything when connectivity returns. Keep that claim as

8:55 narrow as the evidence you actually collected. To evaluate the feature, separate the first visit from a return visit. On the first, record the download and the time until the assistant is

9:04 ready. On the return visit, check what gets reused. Then measure the wait for the first visible output and the time until a complete useful rewrite. Check quality alongside timing, and keep

9:15 failed loads in the results. Give unsupported devices a clear outcome, and preserve the selected text if initialization fails. For this writing assistant, local

9:25 inference is worth trying when the chosen model passes the task tests, the device can run it comfortably, and repeated use makes the download worthwhile. Cloud inference makes sense

9:34 when the needed capability or device support isn't there. A hybrid can offer both routes explicitly. If local loading fails, offer a server option and explain that it sends the selection off the

9:44 device. Let the user choose that route before the text moves. You can keep the editor useful while respecting the privacy choice made at the rewrite button.

9:53 Local generations don't create an inference provider charge. The visitor supplies the computing resources and accepts the download. Start with one bounded task, test its answers and its

10:03 device costs, then choose where that work belongs.