Run Your Own AI Coder on a 6GB GPU (No Subscription)

0:00 This app throws a 500 error on every request, every single one. It's a small web API. The tests are read and I haven't touched a line of it yet. Here's the part that matters. I'm not opening a

0:11 chat window in the cloud to fix it. Everything today runs on this one machine. 16 gigs of RAM and a graphics card with 6 gigs of its own memory. That's a setup a lot of you already own.

0:22 And the promise you keep hearing is that a computer like this can run AI locally. Now, it can. I want to answer something narrower. Can the model that's small enough to fit actually finish real

0:32 coding work? So, I'm giving local AI three jobs on this codebase. Fix the broken API. Add a feature on top of it. Then repair the mess it makes doing the first two. Loading a model is the easy

0:43 part, and we'll do it. The hard part is what's left over. On a 6 gig card, the model isn't the only thing fighting for space. And that leftover room turns out to be the whole story. Quick map of

0:53 where we're going. First, the four models worth trying on a card this size. Then, the one number that decides which of them can actually work, and you can figure it out before you download

1:03 anything. After that, the exact setup, the app that runs the model, and how it talks to my editor. Then, the three jobs, and at the end, the one I'd actually start with, plus the two I

1:13 treat as experiments. One thing up front, because it'll save you a download. The size on a model's page is not the memory it runs in. That gap is where most six gig setups fall over. And

1:23 it's the number we keep coming back to. Start with the short list. Four models, each small enough to at least load on this card. And each one here for a different reason. First, the practical

1:33 starting point. It's called Neo Horse 1, the 4 billion version. The people who built it aimed it straight at coding, following instructions, and running as an agent. That's the exact job I've got.

1:44 The build I'd grab is the 5bit one. Just over 3 GB on disk on a 6 gig card that leaves real room to spare. That leftover space is the whole game here, so keep an eye on it. Its makers do report strong

1:57 scores. I'm not going to repeat those as gospel. A benchmark on their hardware tells you almost nothing about how the shrunk down version behaves on yours. Finding that out is what the three jobs

2:07 are for. Second, a direct competitor a lot of you already know. Sparkx 2.5, also 4 billion, and its 4-bit build is even smaller, 2.6 GB. Getting it running used to be the hard part. Its current

2:20 page now lists native support in the mainstream tools, so there's no special runtime to chase anymore. Both of these are 4 billion parameter models, small. The bet baked into this whole video is

2:31 that small and coding focused beats big in general on a card like this. We're going to test that bet head on. Third model, and it's the obvious next question. What if bigger actually helps

2:41 enough to be worth it? Same family, Neoorse again, but the 9 billion version. Its 4-bit build is 5.6 GB. Put that number next to the card. 5.6 gigs of model on a card with 6 gigs of

2:54 memory. It nearly fills the whole thing by itself before the model has done a single thing. That leftover space I told you to watch, there's almost none left. So, the 9 billion model is a stretch,

3:05 and I'm calling it that on purpose. It'll likely have to push part of itself off the card and onto your regular RAM run by the processor, which is slower. Whether the better answers are worth the

3:15 wait is a real question. And job 3 is where we find out. Fourth, and this one's just strange, a 27 billion parameter model on this card. It's called Bonsai 2, and the download is

3:27 5.95 GB, a 27 billion model in under 6 gigs. That's a clever compression trick, and this channel already pulled it apart in its own video. So today it's just a challenger. Two catches come with it. It

3:39 needs a patched runtime from Prism, the lab that made it, because the normal tools reject its file outright. And a small download does not mean it sits comfortably on a 6 gig card. So that's

3:50 the field. Two small coding models, one bigger sibling, and one wild card. All four fit on disk. The rest of this video is really one question. Which of them fit where it actually counts inside the

4:01 6 gigs of memory on the card? Here's that one number, and you can work it out before you download a thing. It's how much has to sit in the graphics card's memory at the same moment. Start with

4:11 the card. 6 gigs, but you never get all six. The desktop, the screen you're looking at, the drivers, they take a few hundred megabytes off the top before you load anything. Call it around 5 gigs you

4:22 can actually spend. Treat that as a planning figure, not a guarantee. Now, the model, take the 5-bit Neoorse, just over 3 gigs. That's only the weights, the model sitting still. Load it and

4:33 you've spent three of your five. So far, so comfortable. But a model that's only sitting there is useless. The moment it reads your code and starts working, it needs a second pool of memory called the

4:43 context cache. Think of it like the notes you keep on the desk so you don't reread the whole book every time. It's the model's short-term memory. Every token of your file and every token of

4:53 its own reply, kept close so it doesn't have to reread everything at each step. and that cash grows with how much you feed it. This is a setting every guide mentions. Start at 4,000 tokens of

5:04 context and the cash stays small. Push it to 8,000 and it roughly doubles. On a tight card, that's the line between fitting and not fitting. So, run the sum for Neo Horse 4 billion. Say you load it

5:16 and hand it a short file at 4,000 tokens of context. Three gigs of weights plus a small cache plus a bit of working scratch space. It lands under five with room to breathe. the whole thing stays

5:27 on the card. And that's exactly why it's fast and why it's the one to start with. Spark tells the same story with even more headroom because its 4-bit build is only 2.6 gigs. That leaves space for a

5:38 bigger context or just more slack. Both of these run with the lights on. Now run that identical sum for the 9 billion model and it comes apart. 5.6 gigs of weights on a card where you had about

5:49 five to spend. The weights alone don't fit before you've added any context cash at all. So, what does the software do? It splits the model. Some layers stay on the fast graphics card and the rest get

6:00 pushed back onto your system RAM run by the processor. It still works. Your 16 gigs of system memory are the safety net that catches the overflow, but every layer that's off the card is on the slow

6:11 road. That's the real trade with the bigger model. You might get a smarter answer, but part of it is crawling along the slow path, so you wait longer for it. I genuinely don't know yet whether

6:21 that's worth it for coding. That's the point of running it. Then there's the 27 billion Bonsai. And this is a trap I warned you about. Its file is 5.95 gigs. So on paper, it looks like it just

6:32 squeaks onto a 6 gig card, but that's the download size, not the running size. Underneath the compression, it's still a 27 billion parameter model. The scratch space and the per token cache it wants

6:43 are sized for a big model, not a small one. It also needs that prism runtime to load at all. Nobody has shown it living happily inside 6 gigs on a card like this. So, I'm treating full residency as

6:54 unproven, not as a feature on the box. One more lever worth knowing since it's the reason these files are small in the first place. A model normally stores each number in 16 bits. These builds

7:04 squeeze that down hard. Four bits, 5 bits, 8 bits per number. Fewer bits means a smaller file and a little less precision. The four and five bit builds are the sweet spot for a card this size.

7:15 8bit is sharper but bigger and it's a stretch here. So the memory math already sorts the field before a single test runs. The two small models fit and stay fast. The 9 billion spills onto the slow

7:27 path. The 27 billion is a question mark. Time to set it up and actually put them to work. The setup is three pieces and it's simpler than it sounds. An app to run the model, your editor, and a bridge

7:39 between the two. The app is LM Studio. It's the thing that loads the model file and serves it. and it's the one that gives you the two dials that matter most. The context size and how many of

7:49 the models layers go on the graphics card versus the processor. It also has a memory estimate built in, so you can check whether a model will fit before you commit to loading it. On a 6 gig

7:59 card, you'll reach for that check more than any other button. The editor is VS Code, and the bridge is an extension called continue. Continue wires your editor to the model running inside LM

8:09 Studio, so the AI can see your files and propose edits right where you're working. Here's why this exact stack matters. To fix a bug, the AI can't just chat about it. First, it reads the file.

8:20 Then, it writes a change to that file. Then, it runs the test to see whether the change worked. That read, edit, run loop is driven by tool calls, and LM Studio supports them. A chatbot just

8:31 suggests a fix in words. A model that runs the loop opens the file and does it, and that's what actually codes before the big test. The settings I'd lock in one model loaded at a time so it

8:42 gets the entire card to itself. Context at 4,000 to start and 8,000 only if that memory check says there's room. Feed it the files that actually matter and the test output, not the whole repository.

8:54 Keep its attention narrow. And write down the exact build, the context size, and the offload split for every run. So a result means something when you compare it later. One more, and it's the

9:05 one people skip. Use the reasoning setting the model's own page recommends and put a hard limit on it. A time cap or a cap on how much it's allowed to write. A small model left to think

9:15 forever will happily think forever. Then before any real job, one tiny rehearsal, point it at a single file, ask for one small change, and let it check itself against a single test. If that little

9:27 loop doesn't hold together, nothing bigger will. Prove the plumbing before you trust it with anything. Two quick setup footnotes for Spark. Check your versions. It needs recent builds of the

9:37 tools, roughly the early September releases or newer. Older versions simply won't recognize it. The exact numbers are on screen. And Bonsai is the one that breaks the simple setup. It won't

9:48 load in the stock tools at all. You need Prism's patch build or the file just gets refused. Factor that friction in before you decide it's worth the trouble. Setup's done. One model on the

9:58 card, the editor wired in, the loop rehearsed. Now, the three jobs from the easy one up to the one that breaks small models. Three jobs chosen so each one probes something different. This isn't a

10:09 benchmark. It's real work on a real broken codebase. The kind of thing you actually do on a Tuesday. Job one, the broken API. The app throws that 500 on every request, and the failing test

10:20 points straight at the endpoint. I hand the model the root file and the test output. And I watch one thing above everything else. Can it hold the file and the test in memory at once? find the

10:30 bug and produce a fix that turns a test green. That's the whole loop in a single pass. And it's the fairest test there is of whether a four billion model can code at all. The memory math says the two

10:41 small models should breeze through this one. There's room on the card to hold a short file and a short test with plenty to spare. So if a model struggles here, it isn't a memory problem, it's the

10:51 model. Job two is harder. Add a feature. That means touching more than one file and holding more of the codebase in your head at the same time. This is where that 4,000 token context limit starts to

11:03 bite. Watch what happens as the context fills up. Either the model keeps track across the files or it forgets what it changed two steps ago and starts contradicting itself. And on the 9

11:13 billion model, this is exactly where the slow path shows up because now there's more to hold and less room to hold it in. Job three is the real one. Repair its own mistakes. I take the messiest

11:24 thing it produced, feed the new failing test straight back in, and ask it to fix what it broke. No hints from me. This is the agent test. A good model reads the new error, forms a theory, and tries an

11:35 actual fix. A weaker one guesses, breaks something else, and starts going in circles. Whether a small local model can climb out of a hole it dug itself is the thing that decides whether it saves you

11:46 time or wastes it. And this is precisely where the benchmark scores stop being any help. A model can look strong on a oneshot coding test and still fall apart the second it has to fix its own second

11:56 mistake. The only way to know is to watch it try. So that's what we do. So that's the rig. Same three jobs, same codebase, one model at a time. Every setting written down. In each run,

12:07 you're really watching three things. Whether it fits, whether it finishes, and whether it's quick enough that you'd actually stick around for the answer. I'll show you each run the way it

12:16 happened. Failures and all. No editing around the ugly parts. The point was never a leaderboard. It's whether this setup on this card turns into something you'd leave running while you grab a

12:26 coffee or something you'd close after 10 minutes. By the end of the three jobs, the picture's clear enough to make a call. So, let me make it. Here's where I land. If you've got a 6 gig card, start

12:36 with the small coding models. Neo Horse 4 billion in the 5bit build or Spark 4 billion in the 4-bit build. They're the only two that fit on the card with room left over to actually work. For a

12:47 developer on a machine like this, 16 gigs of RAM and a six gig card, who wants coding help that doesn't phone home and doesn't bill you every month, one of those two is your daily driver.

12:57 Pick whichever one handles your own three jobs better, the 9 billion sibling. Treat it as a weekend experiment, not a daily tool. It spills onto the slow path on this card, so

13:06 you're trading speed for maybe smarter answers. Reach for it once a small model is working, and only when the small model's answers aren't good enough. The 27 billion bonsai is fascinating and the

13:18 trick behind it is real, but the file size flatters it. Until someone shows it running fully on a card like this, it's a lab curiosity here, not a setup I'd hand you. A download fitting on disk is

13:29 not the same as a model fitting in memory. If there's one habit to break, it's reading the download size and stopping there. That 5.95 gig file that looks like it fits is the perfect

13:39 example. The number that decides your setup was never the file size. It's how much room is left on the card once the model starts thinking. Which brings us right back to that broken API from the

13:50 start. Your PC can run AI. That part's settled. It loads. It runs. Whether it can actually code comes down to one thing you can now work out for yourself before you download anything. Does the

14:02 model that fits your card have room left over to do the job on this machine? The answer is yes, with the small model for real coding help that you own outright. And no, not yet. if you were hoping that

14:13 27 billion headline number was going to fit inside your six gigs. So, do the memory sum first. It takes 5 minutes and it tells you which of these is worth the download and which one's going to spend

14:23 your afternoon swapping to slow memory. If you want the follow-up, I'm taking the same setup up to a bigger graphics card next to find where the small models stop being enough. Subscribe and I'll

14:33 bring you along. One real setup like this every week.