Insight ON The Agent Architecture Debate Happening Inside a 20-Year AI Company

Audio transcript:

The Agent Architecture Debate Happening Inside a 20-Year AI Company

Aakriti Bhargava (00:02):

When generative AI came out, obviously there was a hackathon. We had great results and we started building this and we started integrating that within our product. I think that was the easy part, right? I think the hard part and something that actually we didn't anticipate is the security aspect of it. And it sounds like a no-brainer right now, but I'll tell you why it was something that we didn't anticipate.

Jillian Viner (00:24):

I love that she says, "I know that sounds like a no-brainer because that's exactly the kind of honest reflection that you don't always get. " Welcome back to InsightOn. I'm Jillian Viner and we're wrapping up our series on building solid foundations today and we're staying at Revionics for this one, which is intentional. So in our last episode, you just heard from Patrick Lee about the infrastructure transformation. Now we're going one layer up into the product and engineering organization with Akrate Vargaba. She's the VP of product engineering and AI. Akrate leads the team that actually builds Revionics AI, the models, the agentic systems, the generative AI experiences that their retail customers use every day. And what makes her perspective so valuable is that she's been doing applied AI for over two decades. So when she tells you what generative AI still managed to catch her off guard, well, that's worth paying close attention to.

All right, here's Akrasi. Akhti, thank you for joining us today. It's so good to have you. I'm

Aakriti (01:22):

Excited to be here.

Jillian (01:23):

So just for context for our listeners, tell me a litle bit about Revionics and what you do.

Aakriti (01:28):

So Revionics has been in the market for like 20, 25 years. We are a price AI price optimization company and what that means is we help our customers who are retailers get price recommendations. We have customers all across the globe and I lead engineering and AI at Repuonix. So

Jillian (01:45):

Long time AI, it's part of your DNA, part of your origin story. I would think that because you have 20, 25 years of experience with AI in your products that when generative AI came onto the scene, you guys were just like, "Let's do it. We got it. We know how this works." And it was easy to integrate, right?

Aakriti (02:01):

I would say partially. So let me tell you what exactly happened. We're a organization that experiments a lot. We are innovating all the time with AI and obviously this is applied AI. This is not at a foundational level. And so when generative AI came out, obviously there was a hackathon, we had great results and we started building this and we started integrating that within our product. I think that was the easy part. I think the hard part and something that actually we didn't anticipate is the security aspect of it. And it sounds like a no-brainer right now, but I'll tell you why it was like something that we didn't anticipate. So we've thought about security because we work with really big enterprises. We've thought about security from an AI perspective for a really long time. Segregation of data, models being trained only on their data, segregation of logs, all of that.

And so that was very well defined and that our customers understood that the minute generative AI came into the picture, it's a little different. You have generalized examples. You have everyone asking the data agent at the same time, are the logs being segregated? So we got a lot of questionnaires and explaining to them the difference and the fact that we already did certain things the right way, I think that was a little bit of challenge. Now at least there's a lot more in the market where people are understanding how things work. So it's eased out a little bit. But in the start, I think that was definitely way ... It was a lot more than I expected it to be.

Jillian (03:29):

Concerns that the data was being shared with other competitors.

Aakriti (03:32):

Yeah, concerns about how their questions were being segregated, how the questions that they were asking weren't affecting the generative AI, the prompts of other customers. It was all of that where everyone thought we were training these models, but at the end of the day, it was foundational models we were using from Gemini from Google and then you're creating these prompts on top of it. So I think that it was a little different than traditional AI the way people think about it. So there was just definitely an education there.

Jillian (04:01):

Yeah. The hackathon is interesting and fun.

Aakriti (04:04):

I'm

Jillian (04:04):

Curious, is there something that you're doing today that was not possible without Generative AI, we'll say a year ago?

Aakriti (04:11):

Yeah. I honestly think the insights, right? So we're actually going to be talking a little bit about our release of our pricing agent system. But what we've been doing is a lot of our customers are pricing analysts or merchandisers and they have small teams. And so we want to help them scale, we want to help them be more efficient and they're looking at data all the time. So they can now just go to our SaaS application and just be like, "Hey, tell me the top performing products or tell me products that are priced higher than my competitor." They don't have to go to different dashboards, different filters. We just kind of give them the data right at their fingertips, helping them make quick decisions, quick insights. And that was just something we weren't able to do a year, year and a half ago. Amazing.

Jillian (04:56):

So you mentioned Gemini. There's a question that every leader who is looking to integrate AI into their products is eventually going to have to answer, which is sort of like this buy versus build. The models that are on the shelf right now are incredible, but there's something to be said about having your proprietary data. How did you guys come to the decision about whether buy verse build and how you expanded your generative AI

Aakriti (05:19):

Use? Yeah. I think that's an interesting question because obviously when we think of bioverse as build, I think that's a little different from a pricing AI perspective. We're a company that gives out our AI platform. We produce accurate price recommendations and every time we're talking to clients, sometimes there are other competitors, but a lot of times we're like they want to build it themselves. And so I think that's a different kind of tactic. They want to build

Jillian (05:42):

Your product

Aakriti (05:43):

Themselves. They want to build ... Yeah, exactly. Which is different, sorry, which is different from basically saying, "Hey, I want to go down the generative AI. Should I be fine-tuning these models?" So two different things. When we talk of builds versus buy on our pricing AI site, I think the one thing that I'll say is because we've been in the market for 20 years, we've really honed our AI platform. If you think about it, if you think of just context, it is data that we've been exposed to in general like COVID kind of market changes that we've been looking at and all of this has been infused into our AI platform. So if you went and asked a ChatGPT, "Hey, help me build this pricing platform." From a AI perspective, they would not give you a very accurate answer. So that's from a build and buy, build versus buy from our product standpoint.

I think when you talk about generative AI, I think given that we've built everything, you would think that, oh, fine-tuning would be the way to go. And when it started, a lot of people were talking about, should we be doing that or should we not? I would say that there's already a rat race on the foundational model level. I don't think it makes sense to get into it, but what needs to be focused on is the engineering around it, the agentic system, because a lot of the reliability and being able to put it into production comes from the engineering around the models. Yes, the models are super important, but the Googles and the antopics of the world and the open AIs of the world are already in that rat race.

Jillian (07:07):

Yeah. Let's talk a litle bit about the production side because I understand that your team is having a healthy internal debate right now about where AI product development is heading next. On one side, you have the case for deterministic reliable agent behavior. On the other, the case for letting AI write and execute code on the fly. Why is that debate happening and where do you stand on it?

Aakriti (07:33):

Okay. So if you asked me this at the end of December last year, I would be like, "Hey, determinism is the way to go. " But with some of the stuff that Claude's been doing, I think it's become very apparent and evident that there's been a shift and that shift is how do you make your agents more reliable, but at the same time giving your agentic system the LLMs a lot more freedom to be able to work on the edge cases. So what we think about when we are building these solutions are three to four different things. So the first one is security. When you are giving the agent a lot of freedom to be able to do certain things like talk to the databases, copy files, et cetera, there is a general sense of security, there's a general lack of guardrails there. And so you need to be very clear about what you want your agentic system to have fu kind of power over.

So that's one. I think cost is the second one. When you are letting your agent do everything, i.e. It's an LLM call, cost increases. That is something that everyone's been talking about. What is the trade off in terms of the effort that you're putting in and the costs and the money that you're spending? I think the third one, and I think this is honestly one of the bigger ones is reliability. So whenever you're building a system, you try to code for 100% of the use cases, but that takes more time and so lesser speed. But what if you could do something where you know 70% of the use cases, you know this always happens. So you built for that deterministically. The last 30% you let the LLM kind of use their own, I call it LLM Magic, use their own brain to be able to do that.

That not only increases your speed, it also reduces your cost because it's not like you're doing 100% letting the LLM do everything. And so I think all in all, that kind of gets you to a line which is a pretty balanced in terms of cost, security and reliability within production. So that's how we're thinking about it. I honestly think it depends use case to use case. I don't think there's a clear answer. I also think on the fly code execution is pretty new. People are still figuring out how to do that. Obviously you have stuff from Amazon and AWS and Google where you can build some of these things. But I do think that as people are building their own agents, that's something that they're kind of figuring out because it's not easy to do.

Jillian (09:51):

If you're not in the engineering team doing this, do you need to be worried about this? Do you need to be informed of

Aakriti (09:58):

How

Jillian (09:58):

Agents are being built? Why?

Aakriti (10:00):

Very much so because I think at the end of the day, we all know that engineering, software engineers, it's a really good career to get into programming, et cetera, et cetera. But right now, I think software code is becoming a commodity. It's so easy to generate code. And so using that in a productive way is super important. And I do think that organizations have to be very clear about what their goals are with generative AI. It could be external. So for us, it's always been external experiences or customer experiences. Also internal productivity is not something that we've focused on as much. We've only started focusing on in the last eight months. But as you're an industry leader, engineering head, I think it's very important because it helps just in general increase your productivity. How quick are you getting products in the market competitive edge? There's just a lot there.

And I do think this is super important because there was a paradigm shift and if you're not going ahead with it, you would be left behind, I think. Yeah.

Jillian (11:06):

How has agents changed your org way that you go

Aakriti (11:10):

To market? Yeah, I can tell you from an engineering perspective and then I can tell you from a business perspective as well. So from an engineering perspective, I think productivity is key and it's hard to track that honestly. A lot of people are like, "Oh, I'm going to just get a code assist Gemini subscription. I'm going to get a CloudCode subscription." But figuring out are you just talking to it and moving in circles or how is your productivity increasing? I think has been kind of not challenging, but it's something that we think keeps evolving. For us, it's been a few things. I think code generation has become a lot easier, but code review is still not. That's still becoming a bottleneck. So how are we pushing code faster through the pipeline? That's something that we've really tried to focus on. I think the second one is documentation.

I do think that being in the market for 20 years is pro and con where you have just a lot of expertise and domain knowledge that maybe was documented well, maybe wasn't. And so that's really helped us build our knowledge base a little better. I think from a business perspective, I think just being able to do a lot from a product standpoint in terms of looking at the design videos, being able to summarize that, not spending too much time on it or just like we get a lot of RFPs from customers from prospects and being able to fill that out process that used to take, I don't know, three to four weeks now just takes a few days. I think that's pretty big.

Jillian (12:45):

Who doesn't love doing documentation?

Aakriti (12:47):

I'm sorry?

Jillian (12:48):

Who doesn't love doing documentation?

Aakriti (12:49):

I know, I know. I mean, that's something that I keep telling my engineers and I'm honestly also bad at it, but this has just really helped me get back on track. I've

Jillian (12:57):

Never met someone who does code who's great at doing documentation. So that alone feels like a major win. And of course the speed you mentioned, just being able to do things like answer RFPs faster. Has it reshaped your team at all, like the types of roles that you are looking to hire or ...

Aakriti (13:17):

I think it's a little early for that. I think the way I think about it is more about how are we making every engineer more productive and that productivity and that Gauge is different at each level. I do think it's a lot more challenge. It is very easy to get into this weird thing where a junior engineer might not necessarily know how to solve something and they can just go ask Gemini or Claude to do that. And I do think that is ... Then you're producing something you don't understand, which is there is a pretty big con there. I do think being productive at more senior levels has really helped our engineers there. But I think at a junior level, I do think it's a little detrimental in my eyes because you have to be ... We were pushed into the deep end, you learned it the hard way and I do think that helps structure thinking and just like how you would solve problems.

I do think that's going to hampering that to an extent. So not from a role perspective, but generally our thing has been for each person, how does it make you productive? Because everyone's good and bad at certain things. And so it's about figuring that sweet spot for the person. That's how we've been thinking about it.

Jillian (14:23):

Yeah. Interesting. Go learn things the hard way before you can do the shortcuts.

Aakriti (14:26):

I mean, for the junior engineers, I mean, how else are you going to build the muscle? Problem solving is really a thing that I look for when I'm hiring people and I think some of these tools have made people dependent and it's very easy for you to say, okay, build me a system that does this, but you don't know why certain engineering decisions were made. So I think of them as you can tell them what to do, but how to do it still, a big part of it is on the architecture, et cetera. And so that's a very fin balance.

Jillian (14:55):

Yeah. Let's talk about trust for a moment. Again, your company really started with AI doing AI pricing. You've invested years getting your customers to trust that. It feels like generative AI has really sparked a new conversation about trust and even pieces of AI that people previously wouldn't even bat an eyelash at, now everything is put into question. So from your perspective, did the trust that you had from just the original business, did that carry over when generative AI came to the scene? And if not, what did you learn in that process that other organizations can take with them?

Aakriti (15:35):

So I'd say that even trust and adoption for our normal our pricing AI, it's not been easy because you're asking people to kind of trust the outputs that are coming out of a model. And so we've done a few things on this side of the pond that we tried to translate to the other. So it's just been model outputs that are transparent, i.e. We have different elasticity, seasonality, so different effects kind of broken down, just more transparency within our tool. So how did you get to an answer? Just not what the answer is. And I think the third thing is just people, right? People trying, helping our customers understand why certain recommendations were given. And I think we try to do the same from a gen AI perspective because imagine you're used to going into multiple dashboards and kind of figuring out the answers yourself.

Suddenly you're asking someone, it just gives you an answer. It doesn't give you any explanation as to why, it doesn't give you how it got to the answer. It's hard to trust it. And so the same principles we've had to apply here, which is how did we get to this answer? Okay, what was the supporting data? What was the assumptions? So just imagine that we talked about the insights example in the start I think where we said that we allow users to just come to our tool and ask certain data questions, but we've had to be very clear of how we got to that answer. And obviously there are a lot of questions that our business peeps need to answer as well. But I do think some of it has translated but not everything because it is a different way of thinking about it and it's a different mindset because it's being applied to things they never thought it would be applied to.

So I think it's work in progress.

Jillian (17:10):

You've been in the space for a long time AI is to use the cliche word disruptive. And Google has a campaign right now, new ways of working.

Speaker 3 (17:19):

Oh, interesting.

Jillian (17:19):

What have you had to, or what have your teams had to unlearn in the process and what do you think engineers are not talking enough about?

Aakriti (17:30):

That's interesting. Yeah. So I feel like the aha. So I think a lot of my engineers tend to kind of focus a lot on looking at every line of code, which is great. So from a merge request standpoint, and I keep coming back to this, but I think that the more technically you are, the harder it is for you to trust something that just comes that gives you an answer. So when you're telling the agent to basically, the agentic system or whatever, to basically give you an answer on like, "Hey, build me a system," and it gives you a piece of code, I think making sure that the engineers trusted what came out of it, I think that was a little bit of a learning curve because you know how to do it, someone else is doing it, you cross-check their work. I think that's been a part and parcel because I think initially the coding agents weren't as good.

I think in the last four to five months it's kind of really taken off. And I think going from how it used to be to what it is now, I think unlearning that, hey, things can improve. I think that's been key and people need to just leave what their general misconceptions were initially. I think that's been something that the engineers are still getting used to, but now they're way more open to it. But also there are some engineers that aren't open to AI and I think just kind of working through with them, if that makes sense.

Jillian (18:55):

Yeah. Interesting. Yeah. Akrate, thank you so much for your time today. I've learned so much from you.

Aakriti (19:00):

Yeah, thank you. This was super fun.

Speaker 3 (19:02):

Thanks for listening to this episode of Insight On. If today's conversation sparked an idea or raised a challenge you're facing, head to insight.com. You'll find the resources, case studies, and real world solutions to help you lead with clarity. If you've found this episode to be helpful, be sure to follow InsightOn, leave a review and share it with a colleague. It's how we grow the conversation and help more leaders make better tech decisions. Discover more at insight.com. The views and opinions expressed in this podcast are of those of the host and the guests and do not necessarily reflect on the official policy or position of Insight or its affiliates. This content is for informational purposes only, should not be considered as professional or legal advice.