Insight ON Your AI Assumptions Were Built on Subsidized Pricing. Now What?

Audio transcript:

Your AI Assumptions Were Built on Subsidized Pricing. Now What?

Juan Orlandini (00:02):

The real damage, it spreads itself throughout the organization. And I'll use that example that I said about that CEO that told me that he would never hire a junior developer again. That's actually a reflection of the thinking of how he's going to staff and build. If you start looking at that, there's a cost assumption to that as well. It was like, I can do it faster and I can also do it cheaper. And both of those are not necessarily true.

Jillian Viner (00:34):

You made the decision to invest in AI. You did the hard work of user adoption, transformation, and made it scale. And then the billing changed. And now you're looking at a pretty big invoice wondering what to do next. I'm Jillian Viner, and this is Insight on AI Tokenomics. Let's go. Should we start by complaining about Token Maxing and then you.

Juan (00:57):

Well, just say this token maxing thing is kind of - Token

Jillian (01:00):

Maxing thing sucks.

Juan (01:02):

When Fable five first came back, I blew it in 15 minutes with one prompt. But it was actually, I was doing some coding and it was a complicated prompt, but it was one prompt and I blew through the entire.

Jillian (01:14):

What did you do? Did it even finish the task?

Juan (01:17):

It didn't, but at the time it was still on the consumption, not the consumption per token thing. So I just said, now you got to wait until next week.

Jillian (01:27):

I always wanted to see like, okay, I'm maxing out my session. I've got an hour and 45 minutes before I can start over. I'm like, okay, what can I do for an hour and 45 minutes? It's a really weird, I don't know, way to think about projects. I find myself multitasking even more and more these days.

Juan (01:43):

And then multi-agent or multi-account, and then you start losing track of context of where the hell was I working on this.

Jillian (01:54):

Send, yes.

Juan (01:55):

So I was working on that. I know, but I don't know which one I was working on that was it.

Jillian (01:59):

I feel so validated that you have that problem too.

Juan (02:01):

Everybody does.

Jillian (02:02):

No, but you have a great analogy for this. You wrote about it in a blog not long ago, and I want to hear the story of how you've come to this epiphany about Token maxing and really what that means for.

Juan (02:16):

So it's really not about Token maxing, but what the consequences of Token maxing have done to the foundations of how we operate as a business. And this is what happened to me, true story. It was about 10 years ago. It might be a little bit longer than that. I hear a scream from my wife. My wife pays all the bills. I don't know how much anything is. And I come and she goes, "Oh my God, look at the water bill." I go, "What happened?" She goes, "It's $1,500." I'm like, "Is that a lot?" I don't know.

(02:49):

I knew it was a lot. So I'm like, "Oh God, no." So we call the city, city comes out, they send. And so where I live, Roswell, Georgia, the city was there in 20 minutes. It was amazing. So props to Roswell. They come out and they're like, "Yeah, the problems between the street and your house, it's fine at the meter," which is where they stop at. I'm like, "Oh, great." So call a contractor and they find out that the line that runs from the street underneath my driveway to my house, somewhere in there, there's a leak. So the fix is to run a whole new line. So dig a trench, go under the driveway, do all this other stuff. Three, $4,000 later, I got a new water line. I can cook, shower, all that stuff. Great. Next bill comes in, another $700. We caught it right between building cycles.

(03:39):

So I'm at like $5,000. I'm like, oh, this is terrible. No big deal. Move on. Home ownership. A couple months go by and I watch my wife drive in the driveway. We live at the top of the hill and she's got the kids in the car. My wife's in the car, and I see this driveway sink a couple inches as she's driving her kids into the house. And I go, "That's not good." So I call another contractor. Well, all that water had washed out the dirt underneath my driveway. And there was a two-foot gap between the dirt and the cement. So there was literally nothing supporting that driveway. The

Jillian (04:22):

Foundation's got underneath.

Juan (04:24):

Like I said, I live in the top of a hill, so I had to build up a retaining wall and bring in dirt, bring in all this cement. So six and a half trucks of cement. I don't know how many trucks of dirt and all this stuff. $30,000 later, I got the nicest looking driveway in the neighborhood.

Jillian (04:44):

At least the strongest one.

Juan (04:46):

Nobody buys driveways. So that's $30,000. It's just gone.

Jillian (04:51):

Oh, you can't even see it.

Juan (04:52):

And so when this token economy really started coming around, it actually reminded me of that story because the very first shock that we got is when the Frontier Labs turned off the all-you-can-eat subscription models that they had and turned on the metered pricing that was going on, just kind of like water, there was meter. And just like that, we got sticker shock and we're all like, "Oh God, we're going to put things in place." What my fear is, and this is starting to get validated out there, is that because we had made an assumption that the water was free, tokens are free, were easily digestible. We've actually not looked at what that's really done to our thinking and our structures of our organizations. So there's expectations around code development, about efficiency, around agentic workflows that are now being asked to be run, timelines for delivery, training.

(05:53):

Heck, I even had one company tell me that they will never hire junior developers again because they thought that these AI systems were going to take on all the junior development things. And I'm like, "Well, you might want to now because it's cheaper to buy a human than to pay for the tokens."

Jillian (06:10):

Isn't that ironic? Kind of a full circle moment.

Juan (06:12):

Yeah. Well, there's another insidious problem there is that at some point the senior people are going to be gone because they either choose another job or they retire or whatever. Who's going to be the next one? So you got to have the train of development. That's a whole nother conversation. And so I fear that what's happened to a lot of our companies are us and everybody else, is that the foundation got washed and we haven't even been made aware of that yet. So we've been spending time with customers talking about this. I'm like, "Hey, at least take a look. Take a look under the driveway and see if your foundations have been washed out. Have you made the right assumptions based on what tokens used to cost versus what they're going to cost into the future?"

Jillian (06:53):

What are they finding?

Juan (06:55):

More and more that they're finding out that they need to reevaluate both priorities, expectations, timelines, and maybe reset some of the things like hiring and training and all that other stuff. Because it has been so pervasive over the last couple, three years have been all AI. Everybody knows that. So it's embedded itself everywhere.

Jillian (07:16):

Yeah. I got to be a little bit of a pessimist for a second here because I imagine this is so frustrating. Organizations have put so much time, effort, resources into figuring out how to do AI transformation, change workflows, really make sure that AI is part of their systems. And now it's like we got to go back and almost start over because the foundation that we built is not cost-effective. It's not sustainable on this new pricing model.

Juan (07:42):

Yes and no. There's a couple of levers that we have in order to adjust what that cost model is. And this is true. There's a difference in price between the lowest end AI models and the highest end models, even in the frontier lab providers, of up to $4,500 difference. 4,500 times difference, not dollars, between the lowest cost models and the highest cost models. So as a prompt, so for example, what's the weather today? It could cost you a dollar or it could cost you $4,500. Same provider just based on which model you choose to ask that question of.

Jillian (08:22):

So model education is huge.

Juan (08:24):

Model selection. Correct. 100%. Both. Yep. So there's that. So part of it is like, hey, make sure that you're using the right model for the right kind of prompt and the right kind of task that you're trying to build around. And for many automation tasks, you don't need a deep reasoning model that's got all the knowledge of all of humanity. You need something a little bit more constrained and those are cheaper to operate and infer with, so use that. And so you got to do some of that. The other one is not all models need to be run in the Frontier Labs. Some of them could be run from some of the hyperscaler cloud providers that are not the Frontier Labs. So go with an Azure GCP or an AWS. They also offer API-based AI services. And then they tend to be a little bit lower because they tend to be not exactly on the Frontier Lab stuff, although some of them are.

(09:21):

Anyway, so it gets a little bit messy, but you get the idea. And then the other opportunity there is there's nothing that says you can't do those models on-premises or even on device. And that's actually the transition that we're seeing is more and more customers or clients are taking a look at it like, "Hey, for a set of workloads, does it make sense for me to buy my own hardware and use my own hardware plus software tooling to do that inferencing on a constrained price? Because once you buy it and once you build the tooling around it, your cost is very constrained. It's power and cooling and that's it. So you can amortize that over a much broader range of things. So there's a couple tricks in there, things that we need to worry about, but for sure you can do that. On-device inferencing is also becoming a bigger thing.

(10:09):

You remember a year and a half ago, we were having conversations around -

Jillian (10:13):

On-device AI.

Juan (10:14):

On-device AI. Now we're starting to understand. It's like, hey,

Jillian (10:18):

That

Juan (10:19):

Was the future. This starting to make sense. And I think we talked about this back then. Silicon provides capability. Software takes a little longer to be able to consume that capability. I think we're going to start seeing that happening on the device side of the house as well. So if you look at what Apple and Microsoft, I mean all of the device makers are doing, they're embedding AI into all of their offerings and all of that is being done on device. So yeah, there's hope. Long-winded answer. There's definitely hope. You just got to be a lot smarter than what you were doing before.

Jillian (10:58):

When you're trying to have that economics conversation with business leaders who maybe they understand the cloud economics, but the token economics is maybe not quite clicking. How do you translate that?

Juan (11:09):

Everybody's talking token economics and token maxing is out of the door, and we're just laughing about that and all that. The question is really, what am I going to do about it? You have capabilities around metering and mapping the token consumption so we can actually understand who and why and do attribution. Are you attributing this to the right project, to the right thing? Is this returning money or investment? Or is this giving you the value that you're paying for? Another way to think about it is if your tokens cost you a thousand dollars, but you make $2,000 with them, I'd pay $1,000 to make 2,000. But if I'm paying $1,000 for my tokens and I make $500 or make $0, that's kind of dumb. I'm not going to do that. And without this visibility, you don't know. You don't know what you're doing.

Jillian (12:03):

Yeah. That's an interesting point because you don't know whether or not you're profiting off of AI unless you can measure it to know whether or not the cost is useful. For organizations that have gotten that sticker shock, what is the first thing that they do or should they do? Because I think the temptation is to just run to the top and turn it off.

Juan (12:24):

It is. And actually there's well-published on stories where that's actually happened. They've been all over the news. Uber blew through their token budget in three months, what they had budgeted for the entire year. Actually one just came out, I think last week, the US Army blew through their entire budget as well in five months, six months. And now they're instituting constraints on their users. And this is for 3.3 million people. Army's kind of big here. And knee-jerk reaction is turn it off and then start turning it back on and dribs and drabs. And that's not wrong. I mean, if you don't know at all who was using that and what for, well now you start hearing a lot of screaming and yelling. But a more intelligent approach is maybe to start implementing things like, hey, all of these AI providers have telemetry that they provide.

(13:24):

Instrument that telemetry and start understanding where in all the consumption's happening and then start mapping it back, that attribution that I was saying where you can actually start mapping it back to business function or projects at least, or maybe even code repositories that are being worked on. And then you can actually start saying, no, that thing is actually, I need full token max on that one. On that one, we're wasting our time. We should be doing something else over here. And so instrument would be hugely important. And what's amazing is, like I said, we've built tools, there's other tools from all the providers. This is going to be a fixed problem here pretty shortly. Knee-jerk reaction, always probably not the right answer.

Jillian (14:06):

If the bill is the only visible problem that people are seeing, what's the part that doesn't show up on that invoice?

Juan (14:14):

That's a good question.

Jillian (14:15):

Where does the real damage of all this accumulate?

Juan (14:18):

Yeah, the real damage, it spreads itself throughout the organization. And I'll use that example that I said about that CEO that told me that he would never hire a junior developer again. That's actually a reflection of the thinking of how he's going to staff and build. And then in his mind, I'm certain what he was looking at, the reason he was looking at it is like, look how much faster I can bring out new products and features. Well, if you start looking at that, there's a cost assumption to that as well. It was like, I can do it faster and I can also do it cheaper. And both of those are not necessarily true. And so it's that foundational thinking about how you plan that actually needs to be reevaluated to make sure that you're looking at it that way. Same thing for say in the legal department where you were thinking it's like, "Hey, we're going to be able to build this thing and it's going to be able to process all these legal documents way faster." That might be true, but it might cost you so much money that it still might be better for you to consider having a junior legal department or interns or whatever.

(15:34):

At least do the first calling the first pass through those documents. I mean, there's all these foundational layers that we've made these assumptions, staffing, training, mentorship. That's a great one. So in AI, in coding, not in an AI, there's been largely two kinds of training that happens for programmers, developers, coders. One is the training that educates you on how to do the basics of programming. Here's how you do Java, here's how you do Python, here's how you do whatever language that they're teaching you, C++, whatever. And there's a zillion of them. And you can or could, can earn a competent living because writing and developing that software, once you were given the spec, that required effort and time and energy. So there was an army of people that did this. I call them coders, and this is not a negative on that word at all or to that career.

(16:37):

There's another group of training that happens to be more of the computer science side, what I call a computer science, where you taught fundamentals of algorithms, algorithm design, the complexity theory, compiler construction, operating system theory, networking. I mean, all these things. Very abstract things. But the reason they're taught is because it teaches you how to think at a system level and understand that kind of thing. And then what you do with that cohort of workforce is you start educating them on the business value of that capability that then gets implemented by the coders. So they're the architects that come out of that group. And I'm painting with broad brushes here. Well, that training on the business context, that tends to be more mentorship because it's hard to teach. I couldn't take somebody straight out of Arizona State or Caltech or MIT, Pop Insight and have that person understand Insight as a business.That's not what they teach in school.

(17:46):

So you spend the next couple years being taught what that happens, and then you start mapping that back to what you did get trained into school, and then you start generating value. Well, in this AI, this washing of the foundations, we said we're going to get rid of the coders. And these programmers, the developers, the ones that actually are computer scientists or whatever, we're going to have them now start doing UI, UX. We're going to have them be doing product ownership, product management, all these things that they're not in any shape, trained for, or possibly even want to do. And then on top of that, we're going to have them understand how that actually maps to the business functions that they're trying to serve. And that's a huge tall order. So you're washing out the foundation by burning those people out. And by the way, you got rid of a whole workforce that might actually become your next generation architects that are actually understood.

(18:42):

So that's what I'm talking about is these foundational things that are not obvious at first glance until you start digging underneath the covers to what assumptions you made and what those really transform themselves into.

Jillian (18:53):

Yeah, you're talking years and years of a domino effect of decisions. That's right. When an organization is today now starting to build maybe an agyptic system, what are some of the lessons learned that they should be thinking about as they're doing this?

Juan (19:12):

Lessons learned, I'm going to harp on this again. You got to measure, measure, measure, measure. You got to make sure that you're instrumenting, understanding. But I'll also say lesson learned is that this AI transformation is going to continue to happen. So don't stop the education, don't stop the onboarding of these capabilities. Because two years ago we were saying, "Hey, the person that's going to take your job is not AI. It's going to be the person that knows how to use AI." That's still true. 100%. So you can't stop on that side of the house. Two years ago, we were also saying, "Hey, if you're not using AI, the company that is using AI will outpace you." That is still true. So lesson learned there is as well. It's like, okay, you got to be more disciplined, but continue to invest and evolve your organization to adopt this amazing technology.

Jillian (20:09):

Good advice. And by the way, we just interviewed our interns who are leaving, and all of them agree that they would not go work for a company that was not using AI. They see it as a red flag.

Juan (20:19):

Yeah. Honestly, because a lot of the work that we were asking of our interns, they don't need to do anymore. They can do some more really more interesting things and we can educate them faster on the business value side of us.

Jillian (20:33):

You mentioned, and you mentioned this a couple times, and it's kind of common knowledge, pick the right model for the right job. Is there a mental framework or something? I know you're doing that, not just vibes. What advice do you go off to choose the right model?

Juan (20:46):

Well, there's art and science to it. Me personally, as I actually start with the lowest and then work my way up because I'm kind of a cheap skate.

Jillian (21:00):

Maybe it's a good thing these days. Serving you well.

Juan (21:03):

Yeah. If in my mind I know that it's going to be really difficult to answer this problem, I'll go with the highest and then work my way back down. But that's just my algorithm. Fortunately, there's actual tooling that's being created that is intelligently routing these things. And they will do some pretty clever things. Well, they'll actually analyze the prompt that you're generating. And based on what they know about prompts and model capabilities, they're routing that prompt to the thing automatically for you. So start looking for those things. There's routers and smart model routing and all these other things that are being put into place that will redirect those things for you. So rather than you trying to be smart, there's things that are going to be smart for you and can do this better for you. So

Jillian (21:54):

When Jillian keeps picking Opus for the weather, it's going to automatically -

Juan (21:59):

Hey, I do that too. No judgment.

Jillian (22:02):

Well, it's painful when you forget to change it and ask a dumb question. Or you're mid-conversation, you're like, oh, I got to change model.You can't change your mouth. It's a bad idea. You made a really excellent point earlier about the long-term impacts of this, the foundation that you spoke of that really got to pay attention to pipeline. When you're thinking again about the foundation underneath this token maximum problem, the surprise bill, what's another element of that foundation that business leaders really need to be paying attention to that maybe is not obvious today? I mean, is it really just about the talent pipeline and making sure that you've got incoming talent, talent in the right places?

Juan (22:51):

So structurally to the organization, there's definitely not just talent. Go look at how maybe you've decided you're going to run finance into the future. Go look at some of the contracts that you signed with some of your ISV providers that actually have their own AI models that they now have to actually charge you for the right price for them. Because think about how many startups got built over the last couple years with the assumption that AI tokens were essentially cheap or free. And now that they're not, they're going to have to actually pass that money over to you or that bill, or they're not going to be surviving. And we're starting to see a lot of startups go away just because they can't afford the bill that is now being charged of them. Their business model is no longer good. So you got to re-look at all of that side of the house.

(23:47):

Even the big SaaS providers, they all have AI agentic capabilities inside of them. And usually that was just a feature that was included or there was a small upcharge. I don't know if that's going to be true anymore. You better start renegotiating all of that as well. There's a number of things that are coming on that side of the house. And if that's going to change the equation for you, you're no longer going to be using the agentic systems. Boy, you better understand what that really means for the team that spent the last two years figuring out how to use the agentic system. So there's all these things that are underneath the covers, right?

Jillian (24:25):

Gosh, it gets real hairy. All right. You want to play a little game with me?

Juan (24:27):

Sure.

Jillian (24:28):

We haven't done this in a while. It's called red light, green light. All

Juan (24:30):

Right.

Jillian (24:30):

I'm going to read off a few statements and you're going to tell me if it's a red light, meaning, ooh, we stop that behavior right now. Okay. Or if it's green light, keep doing it or do more of it. Okay. And if it's a yellow light. You can do a yellow light. That's okay. All right. Are you ready?

Juan (24:42):

Yes. All

Jillian (24:43):

Right. We use the most powerful model available by default because we want the best output.

Juan (24:48):

Red light. It's easy.

Jillian (24:51):

We set a hard spending cap on AI, so costs don't get out of control.

Juan (24:55):

Yellow light.

Jillian (24:56):

Why? What do you

Juan (24:57):

Do? You understand what you're getting back for that investment. And if you put a hard cap, you might or might not be getting the right amount. I don't know.

Jillian (25:08):

Fair enough. We let individual teams choose their own AI tools and figure out what works for them.

Juan (25:14):

Red light.

Jillian (25:15):

Why?

Juan (25:19):

We actually have a product that we've actually developed ourselves to help customers through this. Because what ends up happening is if you let everybody pick everything, you end up with a whole bunch of snowflakes all over the place. And that FinOps team, I've been telling people to go, their job is impossible. You cannot do that. So we've actually been helping clients understand like, "Hey, go look at these use cases and sequence them based on how you can actually build a platform, a capabilities matrix, so that the next project that you work on starts leveraging the investment that you made on the first one, and then the one after that." And you get a compounding interest on your investment so that by the time you get to your 10th project or 12th project, you've actually started to really actually get the benefit of all of that accumulated capability and wisdom, internal wisdom.

(26:13):

So that's an absolutely red light. You need to be smarter than that.

Jillian (26:18):

Okay. We're moving our most promising AI pilots into production as fast as possible.

Juan (26:26):

Yellow light. Flashing. How

Jillian (26:29):

Do we make it green?

Juan (26:30):

Flashing yellow light. Yield. Why are we yielding? Proceed with clashing. It's the conversation we just had is are your assumptions as to how these AI systems are going to be operated, where they're going to get their intelligence out of, and the cost model for those? Have you really looked at those? If you have, cool, green light. If you haven't, yellow light, red light.

Jillian (27:01):

Finally, we're waiting until token pricing stabilizes before we commit to a model routing strategy.

Juan (27:06):

Nope. Red light.

Jillian (27:08):

Why?

Juan (27:11):

You already know that there's models that are more than good enough today that you can literally run on your laptop or on a small data center device or even a large data center device that are going to be significantly cheaper, but just as capable as the frontier models. There is no reason for you not to start implementing some model routing of some kind today. And every large organization that I've spoken with that has done that has seen huge dividends pay off on that.

Jillian (27:42):

There are still organizations who are sitting on the sidelines right now with AI and haven't even completed a rollout of Microsoft Copilot. I'm going to use that example because a lot of organizations run on Microsoft. Well, Gemini too. If they're a Google shop, you got Gemini. Are you finding that this new pricing model with AI, the token usage, is that deterring companies more? Especially when you're talking about the budgets and I got to boost my security. And how are they addressing this conversation?

Juan (28:16):

So it's actually making customers be smarter about it than they were because in the prior pricing models from everybody else, it was a yes or no. Do I want this or not? And now it's actually how much do I want? So in prior pricing models, whether with Microsoft, Google and others and Office productivity, AI kind of a thing, you would typically say, "Hey, I want 5,000 seats of this thing." And that turned on all capabilities for those 5,000 seats or whatever it is, or 500 or whatever you bought. Well, now it's not just all 5,000 seats. Of those 5,000 seats, how many do I want to have Supermax, ultra, super duper? And how many do I want just the bare essentials that'll get them through digesting email or whatever? And so you're going to have to be a lot more nuanced and much more intelligent in your consumption.

(29:12):

And yeah, there's still going to be organizations to this day, there are organizations that are still just in the infancy of their adoption of their AI capabilities. This has not helped that because it's added confusion, but it's probably healthy because the true cost of things are actually coming to light and there won't be an even bigger surprise down the road.

Jillian (29:36):

Yeah. Parker Johnson, who's our Agentic AI field CCO, described this moment as a forcing function, the usage-based billing. What's your catchphrase for this moment?

Juan (29:47):

Well, Parker nailed it. That's exactly what it is. It's a forcing function for understanding what the true cost of tokens and all that really are. My personal belief though is that We're fortunate that it happened when it did, because if the labs had waited another six months or another year and continued the subscription subsidized model, these things would've spread even deeper into our roots and it would've been even harder for us to actually get our fingers around how to manage this. I actually think it's a good thing. Thank you. It's painful right now, but in the long run we're better off for it. I

Jillian (30:31):

Feel like we're just all going back to the gym. We just were at the buffet and now like, okay, we got to us.

Juan (30:38):

I got to go back there.

Jillian (30:41):

Well, and thank you so much for your time today. It was very enlightening. Yeah,

Juan (30:43):

Thank you.

Speaker 3 (30:45):

Thanks for listening to this episode of Insight On. If today's conversation sparked an idea or raised a challenge you're facing, head to insight.com. You'll find the resources, case studies, and real world solutions to help you lead with clarity. If you found this episode to be helpful, be sure to follow Insight On, leave a review, and share it with a colleague. It's how we grow the conversation and help more leaders make better tech decisions. Discover more at insight.com. The views and opinions expressed in this podcast are of those of the host and the guests and do not necessarily reflect on the official policy or position of Insight or its affiliates. This content is for informational purposes only, should not be considered as professional or legal advice.