5000 Hours of Building AI in Just 17 Minutes

I've spent over 5,000 hours building with AI, and I've used it to build a seven-figure agency, teach more than 400,000 people, and automate my entire business. So, in this video, I'm going to give you the biggest lessons from those 5,000 hours, so you don't have to spend years making the same mistakes that I made. So, let's get into it. All right, lesson number one. Right now, if you want to get into AI, it kind of feels like there are two main paths, which are either to start your own agency and go find clients, or to become the AI person at the company you already work at. And I do think that both of these are genuinely great options. The way that I got into the space was I actually wanted to be the AI person at my company when I was working at Goldman Sachs, but I could just tell that it was going to take too long and there was a lot of red tape. So, I ended up leaving, and that's when I built my own AI agency. But I do think it's important to acknowledge both paths and find the one that's right for you. But there is one thing that's true for both paths, which is it's getting much harder to actually stand out. Building with AI has gotten so easy that everyone has a portfolio now. Everyone's watched the same tutorials and built the same demos. So, from the outside, it all kind of looks the same. And the businesses on the other side can't tell who's actually legit versus who just threw together a demo last weekend based on a YouTube tutorial. So, the way you stand out is you stop collecting builds and you start collecting receipts. A portfolio says, "I built this thing," but a receipt says, "Here's what this thing actually did for the business." So, every time you build something, document the outcome. The process took this many hours before, it now takes this many. We were missing this many leads, now we catch all of them. Even on a tiny project, even if it's just for yourself, just write the number down and then record a quick walk-through. Because the person with five screenshots of workflows looks exactly like everyone else, and the person with three real outcomes stands out. It's also exactly why we're building out the certification program at AI Automation Society, because this trust gap is the biggest pain that we're seeing on both sides right now, for businesses and for service providers. All right, lesson two. Tools don't matter. And I'm saying this as someone who built basically his entire YouTube channel on one or two tools. For the longest time, my main thing was n8n, and it's what a lot of you guys probably know me from. But these days, my main tool is Cloud Code. But the point I'm trying to make is that new and better tools will come out all the time, and that's just never going to stop. But the tool was never the valuable part. It's what you've learned and how you think that's actually valuable. Just think about everything that ended and taught me. What an API call is, where things tend to break, how to read an error and actually fix that error. None of that went away when I switched. I just carried it all straight over into Cloud Code. And the same goes for what you build. So a lot of you guys right now have been building an AI operating system like me and you're totally fine because it's really just folders and markdown files and instructions. Whether you want to run that on Cloud Code or Codex or Hermes or whatever comes out next month, it all just transfers over. And if for some reason your setup doesn't, then just make sure you're building it in a way where it is actually tool agnostic. So don't wait around for the perfect tool and don't stress when the one that you love gets replaced. Just get obsessed with the skills underneath it. Like how to communicate clearly, how to break down the problem, and how to find a better solution when the first one flops. Okay, moving on to lesson three. Being AI native isn't about how much you know. It's not about how many models you can name or how many tools you've used. It's what you reach for first. So when something lands on your desk, is your default, I wonder if AI can do this. Let me try that. Or do you just grind it out manually the way that you always have? And the question, can AI do this, is never a binary. It's not yes or no. The real question is, to what extent can AI do this? If it gets you 70% of the way there and then you have to handle the rest, that's a huge win. If it only knocks out the first 25%, still a win because you're still way ahead of the person who's doing it 100% manually. And also that answer's never final because the models and the tools around them are literally the worst they'll ever be right now. So just remember, being AI native isn't how much you know, it's what your brain defaults to. Okay, lesson number four. Most people think the AI itself is the valuable part. Like whoever's got the best prompts or the best model or the fanciest skills is going to win. But that's backwards. The AI is the one thing that everyone has equal access to. Think about it. If everyone can use the same Opus model, then wouldn't everyone be getting the exact same results? No, because everyone brings a different system and a different expertise to that AI model and to the AI harness. So think about it like an accountant building a skill to put together a budget. They're going to be 10 times better at this than someone who's never touched a spreadsheet because they know what a good budget looks like and where people mess up in that budgeting process. So, here's a really practical way that that shows up in everything that I do. Over 5,000 hours, I've stepped on a ton of landmines and now I know how to never step on those landmines again. So, one thing I always do in my builds is I negative prompt. I negative prompt my skills, my systems, my instructions, all my prompting. I'm constantly telling the AI what not to do. That list of don'ts is really just my experience written down, my failures written down and it's stuff that a beginner probably has no of knowing to add. And this isn't just some hack that I made up. Go look at Anthropic's own documentation on how to prompt Claude. And a lot of those examples are loaded with negative prompts. Stuff like, "Don't add features beyond what was asked." or "Don't add error handling for scenarios that can't happen." And this is really what context engineering is all about. Now, everyone's got a slightly different definition of that term, but this is the simplified way that I like to think about it. If you think back to the previous lesson, everyone gets handed the same model and the same harness, which is just, you know, the app that's wrapped around the model like Claude code, for example. So, context engineering is everything that you put on top of that stuff. All the knowledge you feed it, the way you prompt it, the instructions, the skills, the systems you build around it. It's basically just the way that you apply your own brain onto that powerful AI model. Okay, lesson five. Stop talking to your AI and start managing it. Most people will type a request, they'll get an output and if it's bad, they'll just decide the AI isn't smart enough yet. But the people who are getting gold out of these tools, they're not just typing better prompts, they're managing better. So, instead of saying, "Hey, write me this." or "Go research this." I give it a problem and I let it tell me how it wants to solve it and I make it ask me questions until it's completely sure that it understands what I want before it builds anything. And one more thing about these models, they're kind of trained to please you. It's been proven that they're a little bit sycophantic. So, if you ask one what it thinks of your plan, it might just tell you that it's great because that's what you want to hear. But every plan has blind spots. So, make these AI models play devil's advocate. I'll have different AI models attack my own plan from multiple personas, like a skeptical customer, a competitor, an engineer who has to maintain the actual thing because every angle will catch a hole that the other angles didn't. So, what comes out the other side is way more thorough than anything that you'd get by just asking one AI model, is this good? And then after all that, give it a clear finish line, so it knows exactly what done looks like and doesn't stop when it's only halfway there. And that way it can start to delegate work to different sub-agents, and you have a plan all the way from upfront idea all the way into a verified finish line execution. And that will cut your back and forth in half. So, the AI is not your chat buddy, it is your newest employee and it's able to hire and delegate to other little AI employees. And it's only going to be as good as how good you are at managing and orchestrating that. All right, in lesson six, just to kind of piggyback off the previous one, verification. And this might be the most important one for anyone who is building AI agents. When you ask AI to do something, what you want is to have that thing be 100% done. What you usually get is more like 60 or 70% and then you give it feedback, it fixes something, you give it more feedback, and eventually you just kind of claw your way up to like 90 or 95% of the way there. But what if the AI could verify its own work and it wouldn't stop verifying until that condition was actually met and it was confident in that deliverable. And then you could basically just shoot off one prompt and then wait and then you get back a deliverable that's already at 90 or 95% of the way there. And the setup for this is way simpler than you might think. You basically just ask yourself, if a human handed me this work, how would I review it? How would I give it that stamp of approval? Would I just look at it? Would I test it? Would I go through the sign up process? Whatever you would manually do to check the work, the AI can probably do for you. It can operate a browser, it can write a bunch of tests, it can analyze outputs, it can look at things from different perspectives. So, for example, when I'm building a website, I'll have it go into a screenshot loop. Make sure everything is in bounds, make sure it looks good on mobile, make sure there's nothing wrong. And then I'll have it click around and make sure all the buttons work and make sure the forms are actually formatted correctly and send to the right webhook. So, start making all of your AIs verify the stuff that's getting done. Make them prove to you that the work is actually complete. Okay, lesson seven. If AI has access to something, you have to assume that it's going to use it. And I learned this one the hard way. We had an agent that sent an email to about 150,000 people with a discount code, which obviously it wasn't supposed to do. Nobody told it to do that. It saw a task sitting on our to-do list and interpreted that as write and send a discount code to the whole list. And then it just did it. And the lesson underneath that is there's a huge difference between a prompt permissioning layer and a tool permissioning layer. Because you can tell your agent never send emails, only write drafts. But if it still has a send email tool, then it can still send emails. And you need to assume that it's going to one day. These models are non-deterministic, meaning you can run the same thing 100 times and you can get 100 different results. You swap out the model and now the whole system is going to behave differently, interpret your skills differently, stuff like that. So, a rule that lives in the prompt is just a suggestion, but a rule that lives in the tools is an actual restriction. So, do things like scoped API keys. An API key is basically just a password that your agent uses to log into a service and then you can scope it so the key itself physically can only open a certain amount of doors. So, you could give an agent a key that allows it to draft emails, but never even allows it to send. Like, would you ever hand a new hire a credit card and say, "Hey, don't use this." Like, the card works and like you could go buy things, but just don't. You probably wouldn't do that. So, look at everything that your agent can touch, every tool, every database, every file, every key, and assume that it's going to use all of that one day. And if you're not the one building this stuff, then this is the question you ask whoever is. What can this thing actually do on its own? Can it send or can it only draft? And if the answer scares you, then fix the access, not the prompt. All right, lesson eight. When you build an agent and it works, all you've actually proven is that it worked one time on one output. And like I just said, these models are non-deterministic, so you really have no idea what your success rate is going to be across 100 real runs. So, the fix is running AI Evals, and it's way less fancy than it sounds. So, here's a quick example. We had a support agent that had to do a bunch of research. So, it had to look up the customer, look up things in the database, and then write a response. And what we did is we collected 500 examples of actual manual human-written responses that we knew were good. And that was basically our source of truth, our golden data set. And we were able to feed that golden data set back into the system and we could score how many times the agent actually met the expectations of, you know, success. So, when you're running these evals, if the answer is completely objective, you can just use code to grade it. But, a lot of times it needs a little bit of reasoning to see if the answer is actually correct, and that's where you can do an LLM as a judge. So, using an AI model as the judge. So, the way that you design the evaluation really just comes down to how a human would actually evaluate the work and what that success criteria actually is. But, once that's set up, you can now make one small tweak and run the whole evaluation again. So, you're able to turn your hypothesis into an actual definitive yes or no answer because now you know if the change improved the system or made it worse. Whether that change is a change to the prompt or a change to the tool configuration or even swapping out the AI model entirely. Because the thing is, sometimes a tweak that you are sure is going to actually make an improvement to the system is going to drop the score. So, you can't just base it on a gut feeling. You have to actually prove it. And you'd much rather catch that in your own testing rather than in the real world in front of customers and stuff like that because it'll actually give you the ground truth to say, "Hey, this worked." or "Hey, this didn't." and you can now be confident moving something from development into real production. So, before your next agent touches anything real, collect real examples with known good answers, even if you just have to start with 20 good examples, and then score every version against them. But, obviously, the more examples in the golden data set, the better. All right, lesson nine. Think about a business like a pipe. On the front end, we have water coming in, and that's the traffic. So, all the leads and the attention that comes into the business. And on the back end, the water flowing out is the profit, the lifetime revenue that they actually keep. Now, two things can go wrong in this pipe. You can have a clog where something like halfway backs everything up, or you can have a leak where money is just escaping out the side and it just never makes it all the way through. So, your job is to come in and find both the clogs and the leaks. And after hundreds of builds, the biggest thing I've learned is that the thing that the stakeholder usually asks for never the actual constraint. And when I say stakeholder, I just mean whoever you're doing the work for. So, if you run an agency, it's your client. If you are the AI person inside a company, it's your boss. Either way, they'll come to you and go, "I need a chatbot." or "I need this specific automation." But, that's just what they think the fix is. The real value is finding the actual clog or the leak that they didn't even see themselves. Literally targeting the constraints because that's the only true way a business grows. So, before you build anything, don't just take the order and run with it. Walk through the processes and find where they back up and find where money's leaking out and ask, "Where are they losing the most time or the most money?" And when you show up with that frame, something shifts in the way that these stakeholders see you. Instead of seeing you as an order taker who just builds what's requested, you become a consultant who actually cares about the growth of the business and the bottom line. So, find the clog, find the leak, and just figure out how to fix that pipe. Okay, lesson number 10. Once you've found the clog, you don't just start building. Every project needs a Northstar, which is one metric that you're trying to move, and you have to pick it before you start building anything. So, think about it like this. When a business hires an ad agency, the deal is super clear. Hey, you know, we're going to spend $10,000 a week on ads and we're going to bring you $50,000 a week in additional revenue from those ads. Anyone can look at it and be like, "Yeah, that was worth it." But AI projects usually aren't framed that way. It's not super clear because a lot of times these projects are designed to save time or cut costs. So, that makes the bottom line impact a little bit fuzzy. It is your job to make that impact clear. And the way that you do that is you have a baseline to target. So, say the business is getting five leads a week right now. You go to them and you say, "Hey, if we build this AI automation and get you from five leads a week to 15 leads a week within two months, would that be a success? Would that meaningfully move your revenue or your margins?" And if everyone on the team says, "Yeah, that would totally be a success." You now have your Northstar for that project. You're building towards that one number, and when you deploy, everyone can see whether that number has moved up or down. That one conversation changes everything about the project because now the build has a finish line, your stakeholder knows exactly what they got, and you now have a much more solid case study with a real number in it instead of just saying, "Hey, we automated this X, Y, and Z." All right, lesson number 11, and this one decides whether your AI work makes money or loses it. It's about tokens. Tokens are the units that these AI models charge you by. Every word going in and every word coming out is going to cost you a little bit. And the mistake that a lot of people are making is that they throw the biggest, smartest, most expensive model at every single step of the process, and they never think about it again. But, the smart move is matching the model to that specific task. So, reading through a few hundred thousand words of articles and pulling out a one-paragraph summary is grunt work. That's something that a cheap, fast model like Haiku could do for pennies. It would be overkill to give that to Fable 5, for example. But, maybe the final piece of reasoning where you take that summary and then you apply it to the business, and you have to use some strategic decision-making, maybe that's where you bring in Fable. And once you internalize that idea, you can build it right into your systems. It's basically called model routing. Every task gets routed to the cheapest model that can actually handle that task. So, then the expensive ones only get called in when the job really needs it. Same output quality, and the bill can drop by 10 times or more. And this is only going to become more important because local models are getting much better and much smaller, and some of the stuff you can literally run completely free. So, the ceiling on match the model to the task keeps rising. All right, last one, lesson 12. We were all kind of raised on the same script. Get the degree, get the title, and then you're finally allowed to do the work. But, that whole thing is kind of running in a reverse right now. Every single person that I've watched get pulled into an AI role or promoted into one was already doing the work before the role even existed. And the work was only half of it. Their co-workers and their managers started seeing them as the AI person at the company. And this isn't because they were some crazy expert training models, you know, in their basement. It's just relative to everyone else in the building, they were the ones actually experimenting, running little tests, staying up-to-date with the news, and bringing AI projects into the business while everyone else was still just kind of like sitting around. So, the proof comes first now. Just take one annoying repetitive task from your job, the thing that you dread doing every week, and build an automation for it. And then, after you build stuff, show it around. Show it to your team, show it to your boss, and show them the actual impact that it's having on your day-to-day work. And pretty soon, they will be building a seat around what you're already doing. Or, if you're going for your first client, build something for yourself first so you walk in with proof instead of promises. All right, so those were the 12 lessons that I've learned after 5,000-plus hours of building with AI. And with these lessons, you'll learn how to use AI and make money much faster. So, what I did is I broke all of this down into a free resource guide that you guys can access for completely free in my free school community. The link for that is down in the description. You'll also find full courses, resources, skills, and a community of over 400,000 people who are building with AI. But, that is going to do it for today. So, if you guys enjoyed the video or you learned something new, please give it a like. It helps me out a ton. And as always, I appreciate you guys making it to the end of the video. I'll see you in the next one. Thanks, everyone.