The Caveman Prompting Challenge

0:00 Back in the day, we had browsers and we would go and navigate to a place and get on there and look around, put it into another spreadsheet and yeah, a spreadsheet, then manipulated in some

0:12 way and then send it off >> so that it could go onto a PowerPoint and >> all of that >> goes out the window.

0:20 >> It goes completely out the window. And so, yeah, caveman, why use many word when few do trick, right? Like that's the bread and butter of the whole thing, >> man. Even I used GitHub the other day

0:30 and I was like, why would anybody go on this UI and expect to do anything? It's so hard. >> I think the the trouble that I'm predicting that we'll we'll run into is

0:42 what I call like the the latent shopper problem. You know, when you just like walk into a store and you don't know specifically what you're looking for and the person comes up and they ask, "Hey,

0:51 can I help you find anything? [music] All right. So, now I'm here with James, the global head of AI enablement at MetLife.

1:10 Give me the breakdown. What do you work on? >> We focus on just giving our business partners like the tools they need to to get their jobs done, right? And so in in

1:22 the world of AI today, that means getting them AI, right? And so uh what does that mean to to a large global organization, it it really is understanding all the various laws,

1:35 regulations that go into effect for different regions around the world, different countries. um understanding, you know, customer rights and and important aspects of of

1:49 of governing how new technology can be applied to old business problems at a scale uh and faster than than we've done before. So, it's you know, day-to-day it's a lot of

2:03 meetings. [laughter] I still push code. Um, you know, I've I open up poll requests. I review my my engineers pull requests, things like that. Um, like just today, I set up a

2:15 GitHub action for looking at, you know, some agent skills that we're developing. Whatever the problem is, that's that's what we solved. When you say figuring out the different pieces in different

2:29 countries, it reminds me of these games that my daughter played in this book where, you know, it would be like a drawing and then the actual name or word on the other side and you have to like

2:43 draw a line to it. [laughter] >> Yeah. >> I imagine you just have a bunch of laws and rules and regulations on one side and then all right, this maps to the EU.

2:52 this maps to the US except in probably a more automated fashion. >> Yeah, I mean we we do have a lot of these processes automated. The the nice thing is that we have experts in the

3:04 region >> who are obviously the experts >> um who work closely with the government regulators, you know, people like that to understand what the rules of using

3:14 these tools either are currently or are going to be. Um so that way we can understand how we can how we can apply this technology in uh in safe and reasonable way to make you know our

3:24 customers more confident about being our customer. >> And you're also working in one of the PHOPS working groups right the AI one. >> Yeah. Yeah.

3:33 >> What does that entail? >> So the the PHOps Foundation obviously uh does a lot of stuff with AI because it costs money and it's hot. Um and so the early on the the question was well how

3:47 do we just even start thinking about honestly I think the the impetus question was uh how do I know what model to use for which workload >> right I think this is back with like

3:57 chatbt 3.5 um and obviously that was a while ago >> only got more complex >> it only got a little bit more complex with only just a few couple million more

4:06 models that you could choose from um so uh so really that was that was what it started with and what we noticed is there's generally the trade-off between the cost of a model, the speed of the

4:17 model, and the accuracy of the model. Um, and so balancing these things for your specific workload was was what the the working group was trying to put out information about is like here's what we

4:29 see across all of our PHOPS member organizations >> and here is here's what we found. We don't know. We don't know for certain, but this is what we've noticed and and

4:39 here's, you know, the trends, things like that. And so we've we publish white papers. We uh you know throw together code repositories to to demonstrate you know how to how to make a a token

4:51 counting and like chargeback system. Not as a not as a way to be sort of the the end all beall authority but as a way to start that conversation because a lot of people just don't know and they just

5:01 need something to read. So, this is a bit of a tangent, but I've challenged a few of my friends, and I've always wanted to see this because of my own workloads. Uh, and maybe I will

5:13 challenge you to go and do this. We'll see. I want to know what the maximum amount of the caveman skill I can use is before the actual output just falls off a cliff.

5:28 >> Yeah. So, like could somebody benchmark how far I can push the caveman skill? And there's like just the different tasks that you have. And so easy task maybe you could do it on a few different

5:42 metrics like the complexity of the tasks and then also the amount of back and forth >> that you're getting. But I know caveman skill is huge for this whole saving

5:53 money. It's one area, one vector you can use to just like save on costs, >> right? >> And if you push it too far, then it's kind of crap. And it's especially crap

6:04 when you need to have more information or you're like compacting the uh the whole conversation and then you need that. And so I want to know what like how far can I push it?

6:20 >> Yeah. Yeah, I mean it it really depends on the model that you're using, which is like I guess the lamest answer. Like it'd be nice to say like it's about that far, right? Like [laughter]

6:30 >> but unfortunately >> Oh, I'm sure like you could do it with a bunch of different models and then you'll see oh this one you can push it to this far that far at least for now.

6:38 But just as you have those benchmarks that will come out when folks launch their models and they say >> we got this high on SWE. >> Yeah. It's like I want to see what's

6:50 cave bench. >> What's cavebench? No. [laughter] Yeah, it it's it's tricky to know, but I think it it does strike at the the core sort of the whole point of like the

7:00 Phops Foundation and like what the the tokconomics foundation is is striving to do, which is trying to get the most value out of the the tool of the moment, the LLM, the generative

7:15 AI capabilities >> with uh you know, just maximizing that ratio, right? Um, and so yeah, caveman, why use many word when few do trick, right? Like that's the that's the like

7:29 bread and butter of the whole thing, right? If I can get a really good response out of just like a few input tokens or if I can get a really good response and it's just a really small

7:40 output, you're saving money on both sides. Yeah. Right. And so as these models sort of mature and ironically, right, Anthropic just released a new model literally today. Um the the new

7:53 Fable model. Um >> but they said they said something weird like, "Oh, you're going to have this price for or you're going to have this model at this price until the end of

8:02 June." Did you see that? >> I did. Yeah. It's like they're it I think they're probably I think they got burned a couple times when they released a new model and started charging people

8:11 for it right away and it you know the usage spiked and the the cluster that's running this model couldn't keep up with demand. >> Oh, so you think they're doing it

8:20 higher? >> I I think they're I think they released it with uh with a sort of acceptable price or like free sort of you know kick the wheels pricing. uh with with the

8:34 anticipation of like it may not be perfect when it launches. It may have some issues or some weird regressions that will work out for the next what is it 14 days or something like that. Um

8:46 and as they learn about that they're I I think they're trying to get you know developer sentiment kind of back aligned with what it was which was you know much more positively thinking about the the

8:59 company models. Yeah. And like you know when your system goes down for half a day, people start thinking like well what am I doing? >> Maybe we should

9:08 >> maybe we should have look at codeex. >> Maybe we should have other back other things. And >> I think it's a natural thing. Um it's sort it feels like to me a test drive,

9:16 right? They're s they're letting you take the model for a test drive, see what it can do and then you know ideally it proves itself and you're like yeah that's worth it. and and I think the

9:26 business value of me using Fable would be, you know, is worth the squeeze. I think that's what they're going for. >> And how do you plan for these kind of like wrenches that get thrown into the

9:38 mix as you're enabling the developers across the whole company and then you get something like that where it's like, okay, cool. We can't trust this pricing at all. Uh, so we can use it, we can try

9:51 and build something on it, but at the end of the month, we're going to have to reassess. >> Well, it's not so much that we don't trust the pricing. It's just that the

9:59 the consumption is difficult to to sort of tie down, right? So, like I mean, at the end of the day, if you burn a million tokens, apparently it'll be $50, right? And

10:10 >> that's that's pretty easy to understand. The the problem is is understanding what those million tokens did. And so when the wrench gets thrown and you know 10 minutes after the model release we have

10:22 developers asking for it. It's like yeah guys like we just learned about it too. Um you know we're we're working on partnering with these model providers to to give us

10:33 >> heads up or like early access stuff like that. But at the same time they're they're doing the same thing, right? A lot of times the models they're building are cutting edge and and they've been

10:42 using them just for a couple months themselves. So, like I think it's just kind of this new strange world where um the release of models and the new capabilities will will probably just

10:53 sort of be something you just learn to roll with the punches with. >> Um I think they'll continue to leaprog each other in capabilities here for the next year or so and I think he'll

11:04 probably taper off >> and you can you know time stamp this right now and then a year from now when I'm completely wrong you can uh you know I'll point and laugh. It's fine. Who is

11:13 that guy? You see that Substack of the guy who just goes and fact checkcks people's like claims? >> Yeah, >> there's a whole uh there's a whole

11:22 newsletter on it. I can't remember the name, but incredible Substack because he'll just be like, "You were wrong. You were wrong. Look, you said that this was going to happen on this day 10 years

11:33 ago." No, still not happening. >> Yeah. I I mean I Yeah. How could you not know the future? It's just it's very short-sighted of them for the future. >> Exactly.

11:43 >> Jeez. >> Uh so how are you thinking about then the value that the folks in the company are getting with AI? Like what are the metrics that you're feeling very

11:57 confident about recognizing and and viewing as strong signals that there's value being produced here? >> We look at a couple things. one and it's the most easy to to sort of measure is

12:09 are we avoiding costs somehow, right? Like did we are are we able to somehow through the use of AI not spend money somewhere else? >> Um and that can that can look like a lot

12:23 of different things. Um so you know maybe we are internalizing a tool, maybe we've rebuilt a capability or or you know you name it. Um, that's one way that we looked at it. This the second

12:37 way we look at it is can we can we drive business outcomes faster and like that's like the most corporate response ever. But like it's so broad, right? It would be way really easy if we

12:48 were just a widget factory >> and we could look at the pile of widgets on the other side of the factory and say like, "Yep, we made more of them." >> Um, but unfortunately we don't just make

12:56 widgets. Um, and so that's the that's sort of the driving conversation that we have with our various business stakeholders is like what is your widget, right? What is your unit metric

13:07 that you care about? And ironically, normally when you do have this conversation and you find one unit metric or thing that really, you know, aligns to revenue or a KPI or something

13:18 like that, it'll satisfy one stakeholder, but then the next stakeholder in the next conversation will say, "Well, that's not the right one. M

13:26 >> and what I actually care about is this. But what's handy is that once you find at least one unit metric and you start sort of advertising that as the the source of

13:35 >> the north star, >> the north star, people are all like they'll get upset and they'll be like, "No, that's not what I care about." And you're like, "Haha, excellent. I've got

13:42 you now. So this is what you care about." And you just kind of go through and collect all the unit metrics and you start, you know, it very much aligns to the core principle of of FinOps, which

13:51 is understanding like what is that unit of value that the business really cares about. Um, and then you're able to uh eke it out because it is a different way of thinking, right? Like if you're

14:02 thinking quarterly, it doesn't necessarily align to an individual thing that happens a thousand times a day, right? Um, and so I think those are the two two main ways we're looking at value

14:15 very well. >> So you get this this key northstar metric and all different kinds of folks have their different metrics.

14:24 Then how do you tie that back to the AI is the idea of however much or however they're changing that metric like I didn't quite make the connection.

14:37 >> Yeah. So like if a process consumes $10 worth of tokens and this is just for AI, right? If a process just consumes $10 worth of tokens, then you can say that you you can go to your

14:53 business person and say, "Hey, to get one of these outcomes, it cost us $10. Do you like that or not?" Right? And they'll say, "Yeah, that's great. Keep doing that." And it's like, "Okay,

15:06 perfect. conversation over. Or they can say, "That's kind of expensive. Like, can we do it for $9?" Because like, $9 that would that works a lot better with our current like revenue

15:18 estimates. Um, and then the conversation becomes a more classic sort of tech optimization conversation as opposed to like this weird ethereal AI conversation

15:31 >> where you can go from, okay, we need to save $1 per unit of work. What are our options? We can do a cheaper model. We can do various like prompt efficiency

15:46 techniques. >> Caveman. caveman we can do uh you know but but with all this comes different trade-offs right where like if you have a a cheaper model it might be slower

15:59 >> you might have to run it twice to get you know the sort of uh certainty that you need about that output right which in effect doubles your costs right so it becomes this sort of uh balancing act

16:12 optimization problem where um there's a lot of knobs and levers that can then be pulled. But at least you have that metric that you can track with a goal,

16:23 right? I think once you have a goal in mind, you're able to really drive towards achieving it and testing whether or not, you know, a change actually pushed you closer to that goal or moved

16:34 you further away. >> And does this also work when you're in this exploratory nature of trying to create something? Because feel like a lot of times when I'm first building

16:45 something, maybe it's going to be exploration green field type of stuff. But after I've done that, then I'm much more confident in all right, I've built it.

16:58 Maybe it wasn't the most optimized way to build it, but now that I've done it, I can look back and I can say, "Yeah, it was a little bloated." But >> yeah,

17:07 >> I mean that's the classic PHOps sort of story and that's what PHOps people do, right? We we optimize existing workloads to meet the demand for for what needs to be

17:19 done. Um, >> honestly, it's kind of like what humans do all the time anyways, right? We are very good at optimizing systems. Um and so yeah, I think generally the

17:31 like my my approach is use the smartest model I can for the initial work and then if it you know knocks it out of the park the question is a do I need to do this task again

17:43 >> and if I do go down one level of a model right if it's a 300 billion parameter model maybe I go down to a 70 right just using like the standard open source model counts or not parameter counts

17:54 right and like maybe instead of um instead of leveraging the 300 billion parameter model, if I can get the same output from that 70 and uh just sprinkle in a little bit more of a like a harness

18:09 agent to harness capability. Hey, that's not bad, right? And and it's that optimization that you just sort of slowly chip away at. um that ironically AI is really good at sort of doing for

18:21 you which is kind of weird because traditionally the cloud was not so good at telling you how to improve itself right like if you have a Lambda function running somewhere

18:32 >> it's not going to without help from the cloud function cloud provider they're not going to say hey this serverless function that you've got running needs to be tuned a bit right that's a that's

18:42 a that's a thing that they provide you whereas with AI you can literally just ask the same model. Hey, how do I make this work a little bit better? And you know, it'll say it'll take a look at it

18:53 and say, "Oh, we could jump here or move there or whatever." And usually it works. >> That is a great point. You don't get that with traditional fin.

19:04 [clears throat] >> It's like this weird self bootstrapping capability. Like once you have AI plugged into whatever that capability is, you can just keep asking it about

19:16 that capability and like where where certain inefficiencies you've noticed or where inefficiencies perhaps it's noticed. >> Um, and you can really like accelerate

19:26 really fast. You can go from zero to 100 like that. um in terms of the capabilities that you were originally going for. And then once once you're kind of elevated, you the human out of

19:38 the sort of toil of that process, you can actually start thinking about what what is the problem that we're actually trying to solve and is the process that I've put in place originally, is that

19:48 actually solving that problem or is there like a different >> angle that I need to come at it from? And what's nice is that with obviously AI, you can just have it like iterate

19:58 rapidly and say, "Okay, let's try this new angle and it might not work, right? But at least at least you can spend the time or you don't have to spend the time rather um to to even explore it, right?

20:10 You just kind of like, hey, go try it, come back, run it, run a model on that for a thousand a thousand invocations and see if it actually pans out." Well, yeah, that's kind of what my questioning

20:23 and the reasoning was on the last one with that's like exploratory nature in a way. And maybe you have five [snorts] explorations before you land on one. [clears throat] Do those five still

20:36 count as this, hey, we need to do it for $10. this that's part of the $10 when you're going to the >> CFO or whoever to say this costs five $10 even though

20:50 >> it's like at the base it was a lot of exploration and then you chose one and you went vertical with it and you you shipped that one version but as part of that whole package.

21:02 Yeah, I I think most I mean I don't I would assume that most CFOs wouldn't necessarily classify the exploratory work as like a production workload, right? Like

21:17 >> so it's different. It's like R&D versus actual production. Yeah. >> Yeah. That that's the way I look at it. I'm sure if you get a CFO in here, they'd say something like totally

21:26 different. But well, I guess where it starts to get tricky is how can you classify those tokens as these are R&D tokens? This whole exploration that I did

21:39 >> that shouldn't be counted towards the total cost and you're going back and you're counting what was part of the final build versus what was the exploration, right? You know what I

21:50 mean? Like >> Yeah. Yeah. The way that I think about it is that's just personal productivity. Right? Like obviously it's not directly what we classically think of as

22:00 productivity, but like if I'm able to personally iterate and build a thing locally or like experiment with a model, >> you know, on my machine to somehow not not the inference is remote, but like

22:15 >> if I'm able to build the thing on my machine and explore it that way, that's one thing, right? It's kind of like building software locally on your laptop. you can run the mpm test command

22:25 and like validate that the new feature works, right? But when you actually push it to production, there's a whole pipeline process sort of difference between the thing, you know, it runs on

22:38 my computer is totally different than it runs on any computer. >> Um, and so I think that's sort of the same framework that I apply it with. It's or or that I think about it with is

22:48 there's this sort of monthly budget, right? um that can be applied to each developer as they're exploring

23:00 and then that is specifically separate from the actual workloads that you know are actually running and doing things like specifically like task oriented right so I think the the R&D model does

23:15 kind of work really well there it's just difficult I think they've started getting really close to each other so that's the that's the trouble. >> Yeah. Especially as you've you're

23:26 kicking off like 10 different cloud agents >> trying to build different things and like you were saying like hey can we run a thousand simulations on this to try

23:37 and find the best way to go about it. But it could be one way of looking at it is like you said once you merge it to main then you know this is the production workload. We're going to go

23:49 and do that. However, I know that like some teams it's just constant like merging domain and some of them are exploration still but then they'll go and a new

24:04 PR gets merged so fast over top of it that it's like you can't really separate the two like you're saying like it's so unclear how the agents are are working and which

24:18 ones are working in what capacity. >> Yeah. The experiment becomes the product. Yeah. >> It's like wait what? Yeah. Which I mean honestly every product is an experiment

24:27 or at least should be >> I I would argue should be >> drafted as an experiment. Like will this product actually succeed in this market? >> Yes or no? And then if it succeeds then

24:37 you know it did. >> Yeah. >> But obviously you have to be able to measure like what it was about that product that made it succeed. And that's

24:45 where again I think the the the unit economic or the unit metric for that particular product really comes into play, right? How is it actually moving your company's bottom line? And and

24:56 that's where I think opening up the door for all these devs to start exploring and building and iterating while still having the the traditional software development life cycle guard rails in

25:07 place even for agents, right? um is super critical because >> yeah you can not not every experiment is a really good idea, right? Like um and and so there is still this higher level

25:22 I would say product thinking that is required in order to make sure that it aligns to your overall strategy um while also uh being a good investment, right?

25:33 Because there can be big investments that just don't align at all to your strategy. Well, and there's a lot of things that I do with my agents, especially like the personal Hermes

25:43 agents, >> that I would never do as a human, but since it's possible with an agent, I'm like, "Oh, sweet. Of course, I want to know every single like Reddit subreddit

25:55 that I follow and what's popular there and hacker news like summaries of the day and all of that in my daily briefing, you know?" So, it goes and it scour the internet and then gives me

26:06 this briefing. And maybe every once in a while I'll do that. Or who am I kidding? Like I do that when I'm trying to take a break from quote

26:18 unquote work. But there's a lot of I guess uh things that I do with the agents that I wouldn't do as a human, but since I have the possibility to do them, why not?

26:29 >> Yeah. I mean, that's what complicated organizations are all about anyways, right? It's, you know, you can have a a business that has all these different departments that

26:41 all do things that one person would not be able to do or not even have the mental faculties to do in the first place. >> But that's the whole point, right? The

26:49 the the point of delegation is to get specificity towards the problem that you are all trying to solve, right? And so the I think basically what you're describing

27:00 is the you know sort kind of organizational theory right where the the complexity of the problems you're able to investigate sort of you know expands

27:14 because you have the capability to investigate them and it's like that's cool that's kind of that's naturally human that's like what we do put us in a dark room we'll start feeling around

27:22 seeing what's going on and that's just what we do right Um, and so I I hear that a lot from people. It's like, I've I've I've never even thought about doing X, Y, or Z thing, but now now I have my

27:34 Mac Mini sitting on my desk, and so of course I'm going to have it do it, right? Like, why why wouldn't you? >> Yeah. >> Um,

27:39 >> I I guess the question then becomes how do you look at that from a governance perspective? because I can make the claim that this is very valuable for our northstar metric, but

27:54 is it really like moving the needle and how much is it spending? And so like I'm just trying to have these frameworks in my mind as far as the costbenefit analysis or that triangle that you

28:06 mapped out for from PinOps and the and the spend match that up with the metrics. But then I'm like, man, there's all these really potentially costly personal productivity agents that I

28:23 can let loose and have them doing things just because they can, >> but like how do you bring that back into like how do you put the reins on it or

28:35 how do you how do you even know if it's worth doing? >> Yeah. I mean, I think that's the it's really [clears throat] there isn't one single way to answer it, right? There's

28:45 layers to it. Um, just like how when you know the personal computer first landed on somebody's desk, the the hope was they would use that to just crunch numbers all day, right? And then what do

28:57 you know? They invented the browser and like everything just went up in flames, right? Um, but it I think it's the kind of getting back to the the previous question which was like I never would

29:10 have even thought about collecting all this data and understanding what my favorite subreddit is talking about today. Um, sort of is a reflection of what we can do with with a governance

29:21 perspective as well, right? like the it it means different things to be able to um to to demonstrate that an agent is able to uh you know bring extra value to a company.

29:35 It it's another thing to ask whether or not it should be the one bringing that value to the company. Um, and so it it really depends, I think, on each individual sort of risk tolerance, and I

29:49 mean like individual company or individual person, >> what they're comfortable doing. Um, you know, in highly regulated spaces, we, you know, luckily we have laws that kind

29:58 of tell us where where we have to be. >> Um, and and that's sort of where we start with with the governance aspect. And then from there there's again getting back to delegation there's a lot

30:11 of different stakeholders in play that make sure that the the use of AI is something that uh that should be done the the you know the type of data being sent to the model should be sent to the

30:23 model um the type of you know the type of server that we're using should be the correct server right like it just kind of the more you use it the more questions you know that there are to ask

30:35 and answer and it just sort of snowballs from Yeah, cuz I think about governance in different ways and I'm sure there's so many different layers to governance and it's almost

30:47 like peeling back an onion >> as you're saying like I know there's the data residency governance or just like policies that you have to keep in place and I also was talking to some folks

31:01 about how their platform will enable the certain policies that company has with spinning up a GPU cluster or spinning up a Kubernetes cluster

31:13 >> so that it's done the way that that company wants it to do >> instead of just hoping that a skill will be able to be as effective. Yeah. >> Yeah.

31:26 >> Which, you know, it's it's great. Usually it works, but if it's something as delicate as this like need to get it right every time, you can't always trust a skill, right?

31:39 >> And and so you want to have it be a little bit more deterministic. So how you put those policies and that governance in place, but again, I I imagine there's a bunch of different

31:50 layers of governance that you're seeing every day. I'd love for you to like talk me through some of those. Well, it it really rhymes with a lot of the existing stuff that we had in the cloud, right?

32:01 There's the proactive prevention, right? Like for AWS, for example, don't make a public bucket, >> right? So, how do I how can I proactively prevent people from making a

32:12 public bucket? >> If a person can't make a public bucket, >> an agent can't either, right? >> And so, or at least it shouldn't be like I'm not going to say that anyways. Uh

32:24 >> it can't. Yeah. And then you never know >> that's on the cloud provider. That's not me, right? Like uh so if if a person can't make a a public bucket, then the agent can't. Um and then there's

32:35 reactive, right? Where if something does not align to your specific policies, how do you then automatically react to it such that it can sort of become in line with your policies? Right? Maybe it's

32:49 like a to your point about Kubernetes like a pod configuration is out of out of spec. I don't know, right? There's so many different ways to configure Kubernetes.

32:58 >> Um, and like I think defining those things in code, right? Policy as code is really important. So that way, you know, because the agent's not going to be able to call up, well, I guess technically

33:12 the agent could call up your infosc partner and ask, hey, how am I supposed to configure this Kubernetes cluster, >> but the infosc partner probably has better things to be doing at that time

33:21 and probably shouldn't be talking to your agent anyways. Um, but like if it's policy as code, the agent can just simply read the code and say, "Oh, okay. this is why either a I'm being blocked

33:33 by the proactive defenses or b why I'm being flagged by the reactive defenses >> in terms of like whatever the heck it is I just provisioned um >> so I I I think

33:45 if you if you remove the word agent from all this AI agent get rid of it um the same rulebook sort of applies for cloud provisioning as it did for you know letting humans pro provision stuff

33:57 >> you wouldn't let a human just go into your production workload and start spinning up Kubernetes clusters and things like that, right? >> Burn in GPUs.

34:04 >> Burn in GPUs. If if you wouldn't let a human just go in and do it, then you certainly shouldn't let an AI agent go in and do it. There should be, >> you know, governance and like SDLC. It's

34:16 all software at the end of the day. Now, um I guess unless you're plugging in a GPU somewhere, that's that's that's hardware, but >> beyond that, right, it's all software at

34:25 the end of the day. And so, um, if if it can be described in a in a document, like literally a text document somewhere or a configuration file somewhere, run it through a pipeline, right? Like

34:38 >> don't do it manually. And if you're uh an AI provider and you are forcing people to do things manually, stop it, >> right? Like give us the APIs to be able to control that so that way we can

34:52 properly govern these big capabilities. now without having to to manually click around, right? Like let's not go down the path of click ops like we did with the cloud

35:03 >> cuz getting out of that was [sighs] >> painful >> painful um man even I used GitHub the other day and I was like why would anybody go on this UI and expect to do

35:16 anything? It's so hard. I had to go and ask the LLM how to do what I'm doing and I ended up just being like, "Ah, you know what? Like, let's just ask Claude Code to do it for me because clicking

35:30 around is not fun." >> That's the Yeah. And that's the weirdest thing I think that strikes me about this current sort of wave of of AI is it seems like the the sort of modern UI

35:41 that we're that we're really used to as humans is kind of it's feeling more and more obsolete to me, right? Yeah. like that. And that's what I'm kind of reconciling internally like with myself

35:54 is like why do I need a a dashboard that tells me the top 10 token consuming applications of the day when I can just write a command line tool, give it to my LLM, and ask it what the top 10 things

36:11 are. And then I don't need a web server that hosts this UI anymore. I don't have to worry about, you know, cookies in my browser like logging in and things like that because my identity is on my laptop

36:23 already, >> right? So, like, why bother clicking around on this ancient UI >> that >> hasn't had an update in forever? Just

36:31 give me the just give me the API, give me a CLI, and I'll give it to my agent and it'll call it on my behalf. >> Yeah. or even potentially just getting stuff brought into your

36:43 chatbot or whatever like LM provider you're using through MCP apps. >> Yeah. >> And you get to see it right there in the conversation,

36:53 >> right? And it's like have the it's sort of an explosion of capabilities that I don't think a lot of people realize, right? If you have like an information security dashboard and you have a PinOps

37:05 dashboard and you have a cloud configuration dashboard for all your different applications and things like that, right? Like as a human, you have to go one

37:14 >> two three and then you bring the information back and you consolidate it all into one sort of mentally digestible thing. But with an MCP app or something like that, you just you are already at

37:26 the center of this. you ask the question and it goes out and finds these different things for you on your behalf. >> Yeah. >> And like sort of processes it for you.

37:35 It cuts it so much more efficient, right? Um and I I think that's sort of where a lot of these capabilities are going to go is like um there's going to be this unified sort of way of

37:49 interacting with things and that's going to be where we work in the future. it's going to be sort of in this hub and the questions that we ask will involve activating different connections between

38:03 systems sometimes in ways that we wouldn't have done ourselves, right? And and that's kind of one of the interesting things is is as we look at a lot of these AI capabilities

38:13 um often the human way of doing it is not the way that the AI would do it, right? like gathering all the data before you know in some sort of like mass collection process is a very human

38:26 way of doing it. Whereas an AI might do it on the fly, right? It'll go it'll start the problem with going to system A. It'll chew on that information and then it'll go to system B and system C

38:35 when it needs to as opposed to like a human-driven process would sort of frontload all that activities, right? Like I I think about like filing my taxes, right? A lot of the work that I

38:44 did was going to all my different bank accounts and downloading, you know, all the forms that they they they publish. And it's like, oh man, >> what if instead you just sort of went

38:56 there on demand for each bank, right, to to download that file and like that's sort of the way that that I see AI helping humans in the future. >> Yeah. I've heard my buddy Simba talk

39:07 about how the chat interface is the new browser >> and it very much feels like in line with what you're talking about. We have a chat that we can ask things of and it

39:20 will populate the information as needed. >> Yeah. >> And it's the specific information that we look for. It's not a dashboard that has 20 items and maybe we only need

39:31 three of them. >> Yeah. It's like, "No, I'm going to ask directly to that." And back in the day, we had browsers and we would go and navigate to a place and get on there and

39:42 look around and try and click around their UI and get what we need. >> Yeah. [laughter] or get that data and then bring it back to, you know, put it in our head and then maybe go or

39:54 translate that data or kind of like do something with that data, put it into another >> piece of software >> spreadsheet

40:03 >> and yeah, a spreadsheet then manipulate it in some way and then send it off >> so that it could go onto a PowerPoint and >> all of that

40:13 goes out the window. >> It goes completely out the window, right? like you just go you say make me a make me a slide deck that can that shows the top 10 you know token users by

40:24 security vulnerability right like pulling that data manually would be absolutely insane in any like big company and so like being able to sort of

40:33 >> just do that on the fly is going to be critical in in years going forward. I think the the trouble that that I find or that I I'm predicting that we'll we'll run into is uh what I call like

40:44 the the latent shopper problem. You know, when you just like walk into a store and you don't know specifically what you're looking for and the person comes up and they ask, "Hey, can I help

40:53 you find anything?" And you're like >> in your head you're like, "Yeah, I'm kind of looking for like where the >> the dark chocolate aisle is, but like I'm just kind of browsing. I don't know

41:02 necessarily what I want." Like I don't know how AI solves that problem because there's a lot of like you and I are describing known problems that we want to use data that we know about to be

41:12 solved. The problem that I'm talking about now is like what about the problems that we don't know that we have that you do get from like >> looking at that dashboard

41:22 >> that dashboard and you're like what the heck is that spike over there right like if you don't have that dashboard if you don't see that data >> to inspire new ideas

41:31 >> yeah what do we [clears throat] lose and so >> I think it's going to be balanced there's a whole lot of stuff that's that's going to be you know trading off

41:37 here and there obviously LLM can just look at a big old CSV and see the spike in the data even though it's just like a 100 thousand rows, >> but it it just it feels different,

41:47 right? If you don't see the data, you can't necessarily know what questions to ask. And I think >> that will be sort of the tension in the future is like how do you know when to

41:57 put the data that you're interacting with on a more dedicated dashboard versus just letting the LLM sort of look at it and say, "Hey, here's the top

42:08 three, you know, >> most important >> most pieces. Here's your punch list." >> Yeah. Yeah. you you have to be intimate with the data on some level because

42:17 that's where you gain those insights >> and only exploring it manually sometimes is like the best way to >> yeah to sort of do that right like there's a lot of

42:25 >> I think once we're removed from the toil of that everyday sort of work >> the the ability for creativity you know in solving these problems comes out naturally and I think that'll be really

42:38 interesting to see >> you know where we are in a here what what it looks like for work actually to be done with these tools when we're no longer

42:47 >> slogging through spreadsheets >> in a year. Ha, that's if [laughter] >> we're sugging through different spreadsheets. Um but you know like h how how will that change fundamentally how

42:59 we approach problems and how we solve problems in in in a business in society at the world at large right it's going to be really interesting list. >> Yeah. Yeah. you you have to be intimate

43:09 with the data on some level because that's where you gain those insights >> and only exploring it manually sometimes is like the best way to yeah >> to sort of do that right like there's a

43:20 lot of >> I think once we're removed from the toil of that everyday sort of work >> the the ability for creativity you know in solving these problems comes out

43:32 naturally and I think that'll be really interesting to see you know where we are in a here what what it looks like for work actually to be done with these tools when we're no longer slogging

43:44 through spreadsheets in a year. H that's if [laughter] >> we're slogging through different spreadsheets. Um but you know like h how how will that change fundamentally how

43:55 we approach problems and how we solve problems in in in a business in society at the world at large right it's going to be really interesting.