From hype to control: How to build a private AI strategy

[00:00:00] Will: When you think about any environment, any company, no company wants their customer records to go out. No company wants their own financial data or their own IP to be released to the wild. The only way you can guarantee control over anything is to put it in the box that you own, that you have the key for the padlock.

[00:00:23] Ollie: Hello, I’m Ollie and I’m a solution architect at Xtravirt.

[00:00:25] Will: Hi, my name’s Will Rodbard. I am a master architect at VMware by Broadcom, and I’ve been with Broadcom for about 15, 16 years now.

[00:00:34] Ollie: So we’re here today to talk about private AI, the public cloud, private cloud, and running AI on premise, and then possibly how Xtravirt can help you run all these things.

[00:00:44] Ollie: Thanks, Will, for being here. I know we talk a lot about AI in terms of the whole private AI piece. Where do you think people are going with it today? Where do you think the market’s going? What sort of thing do you see? I feel like you see a lot of people making decisions that are perhaps reactionary rather than maybe strategic.

[00:01:00] Ollie: How would you summarise based on what you’ve seen?

[00:01:01] Will: It’s a good question. I think that you have a lot of fear. You know, people are making knee jerk decisions based on an assumption or a fear that everybody else is already doing it. You have some customers that are way down the path already, but I’d say the large proportion of customers are still non AI or just discovering AI.

[00:01:25] Ollie: Do you think there’s maybe an assumption that organisations think other people are further ahead than they are, and everyone’s sort of thinking everyone’s kind of overtaking me here. But I personally feel perhaps the reality is everyone’s playing the cards quite close to their chest, and there’s only really a few people that are kind of out there if we’re honest.

[00:01:43] Will: Yeah, I think some of that is driven by… everyone has the word AI in their title now, their job title, the product title, AI is everywhere. You see adverts on TV about AI chat, GPT, Gemini, Grok, it’s all over the place, right? So there’s definitely a fear that everyone else is doing it, but most enterprises have been priced out of starting AI or they’ve been too reticent to start because the cost is high to start.

[00:02:13] Ollie: When I look at customers that we’ve deployed AI solutions for, the ones that we are working for, I split them into two tranches of customers: the people who’ve deployed a cool demo, but it’s kind of stopped there. And to really be successful, you’ve got the customers who’ve identified a problem, put a cost against it, and then produced a solution to fix that problem.

[00:02:32] Ollie: The real success is those companies that can identify that actual use case and nail that use case down. We need to be cognisant of some blockers to adoption. If you’re managing the risk, you’re managing the risk to the business. What do you think the main concerns are in terms of actually adopting these frontier AI models?

[00:02:55] Will: Privacy and security concerns.

[00:02:57] Ollie: Yeah.

[00:02:58] Will: Cost and not falling foul of regulation or legal constraints. So from a cost point of view, cost to develop the application. Last year or the year before, there was a stat that came out, only 4% of applications developed for generative AI or in generative AI made it to market.

[00:03:18] Will: And you think of the expense of doing it, how much that cost the business to all the businesses to invest, if only 5% of them make it. You know, so that’s gone up now with the ease of use, but it’s still, it’s not every application makes it to market. Second, privacy and security. Right? If you start at the model, the large language model with a lot of them, you don’t know what the data was used to train on it.

[00:03:40] Will: You don’t know how the model is coming up with this. It’s newly created information or answer. How can you trust it? So as a company, you don’t want to create a chat bot that then offends all the customers, or…

[00:03:52] Ollie: could you imagine

[00:03:53] Will: …an online banking application that gives out somebody else’s balance, for example?

[00:03:57] Will: So ensuring that the model itself is good fit for purpose, but then also to make it work for you, you have to add your own data.

[00:04:05] Ollie: Yeah.

[00:04:05] Will: So you have to input something, and you don’t necessarily want to be sending it out to wherever.

[00:04:11] Ollie: Mm.

[00:04:11] Will: So you want to maintain that control, whether that control is somewhere else, but you have full control of that product, that data, or you are governed by regulatory compliance to ensure that no personally identifiable data goes out.

[00:04:25] Will: Or you can have an audit trail of it and make sure nobody’s touching it, which is fine. And then Agentic AI comes along where agents are doing things for agents. So the control and the compliance around that and audit trail is very important.

[00:04:41] Ollie: I think the solving the three main pillars are control, compliance and cost that people are really interested in.

[00:04:47] Ollie: Yeah. One of the interesting ones that we see a lot of is a lot of talk about data sovereignty. The sovereignty bit itself is who has the keys to your data? And we see a lot across a variety of customers that are concerned, because I believe it’s the US CLOUD Act basically says that any company headquartered in the States can, if it’s a public cloud provider, have access to your data.

[00:05:08] Ollie: You know, you pair that with …

[00:05:09] Will: Patriot Act.

[00:05:10] Ollie: Thank you. And you tie them together and ultimately it does paint a bit of a worrying picture in terms of who can access your data. Now we’ve got clients that have sensitive legal matters. They don’t want the US government to know to bill or subpoena the hosting company.

[00:05:22] Ollie: What do you think future proofing looks like for these Gen AI solutions?

[00:05:28] Will: We all know this now. It’s not cloud only, or it’s not on-prem only. It’s the best place for the workload, but I think that something that gives you choice and flexibility. So for a long time customers have flipped between single vendor strategy and multiple vendor strategy, right?

[00:05:46] Will: So we are all in with this one vendor, but they don’t give us everything we want. We are going for every product under the sun from all these different vendors, but now it’s too complicated to manage. They don’t fit together and work together very well. So I think flexibility like an open platform that allows you to plug and play or develop from one place to another while still maintaining control.

[00:06:08] Will: The more different platforms you have, the more different processes you need, the more people you need, the bigger the risk profile and therefore the bigger the cost as well to manage and maintain. But if you go down a single vendor strategy, which may work for a lot of people, you could still be squeezing everything into a box, so that it’s a trade-off really between choice and flexibility, and simplicity to manage.

[00:06:34] Will: So I would say start with something that you can develop and keep changing and keep adapting as your needs and your business grows and changes. And don’t think, oh, we’ve chosen this vendor today, this is the vendor always. Or this is the tool we use. Be less religious about it. People’s perception of cloud is, it’s a thing, it’s freedom of choice.

[00:06:53] Will: It’s ability to get what you want rather than being told what you can have. And the leaders of the hyperscalers, the cloud service providers have done an amazing job at that and it’s choice. And they’re aware that if they don’t make it today, you’ll go somewhere else.

[00:07:09] Ollie: Yep.

[00:07:09] Will: But if you are on-prem, you have the ability to control cost.

[00:07:16] Ollie: What do you think some of the common myths or misconceptions are?

[00:07:18] Will: Common myths are that to do AI you have to be a data scientist.

[00:07:23] Ollie: Yeah.

[00:07:23] Will: Right. It’s been designed so you don’t have to be a data scientist. The models are built by data scientists. That’s highly complex. Mathematical algorithms that’s basically what a large language model is, and it’s modelled on how the brain works.

[00:07:35] Will: An AI application is still just an application. It’s still an interface requiring access to data to serve a purpose. Whether that’s a human asking or an agent asking, or one software asking from another, API calls, that sort of thing. So at its heart, it needs infrastructure to run.

[00:07:56] Will: And as an infrastructure company as well as a software company, we are trying to make that easy to serve.

[00:08:02] Ollie: And I think for me, one of the myths is cost as well. I think people think it’s very expensive to get started with a private AI on-premise solution. And I can see where it comes from.

[00:08:12] Ollie: You know, the cost of a single GPU is within what the, the tens of thousands?

[00:08:17] Will: Yeah. Anything from seven, five, $7,000 to 50, $60,000 for one GPU, right? Yeah. So you’re right. Cost is a big factor. Talking about hyperscalers, and cloud service providers, they are a fantastic place to work out what you need from an infrastructure standpoint, right?

[00:08:34] Will: Because they’ve all sunk billions and billions of dollars into the hardware. So you can go along and rent it for a minute, an hour, a week, a month, a day, whatever, and go, that doesn’t work for me, but this does. Right, now I know what to buy on premises. So you’re making a more informed decision on cost before you actually have to make it.

[00:08:55] Ollie: What do you think are the main points or the main things to consider are around control and sovereignty in the public cloud. When you are using company data in the public cloud?

[00:09:04] Will: Controls in and of itself is the main driver. And it covers lots of things. So there’s control over access to data, there’s control over loss of data, and there’s control over cost.

[00:09:17] Ollie: Hmm.

[00:09:17] Will: When you think about any environment, any company, no company wants their customer records to go out. No company wants their own financial data or their own IP to be released to the wild. So having control over that, the only way you can guarantee control over anything is to put it in the box that you own, that you have the key for the padlock.

[00:09:41] Will: So if you want to make a decision, if you want to cut the connection or you want to do anything, you have the power to do it. As soon as you give parts of control away, you lose overall control and control over a cost. You can only control cost if you are in charge and in control over who can do what and when.

[00:10:02] Ollie: Yep.

[00:10:03] Will: What are the common use cases you see?

[00:10:05] Ollie: So, the biggest one to start with, and the easiest one to start with because you’re not betting the business on it and it’s entirely self-contained, that we’ve seen is a documentation pipeline or a documentation search tool. You know, lots of businesses have documents all over the place, different formats, different repositories, different everything.

[00:10:22] Will: Drowning in data.

[00:10:23] Ollie: Yeah. Tonnes of great data, but no real way of making much use of it. So we’ve done this internally ourselves. We’ve got an internal lab, a couple of graphics cards, GPUs, they weren’t the most expensive things in the world, but they were still a reasonable cost.

[00:10:40] Ollie: …and we deployed them on VCF 9, so the private AI platform itself. And we built something called a RAG solution.

[00:10:49] Will: So what is RAG just for everyone to understand?

[00:10:52] Ollie: RAG stands for retrieval augmented generation. So you would typically take a large language model, and what it does is there’ll be a database of all your documents chunked up in various different ways.

[00:11:02] Ollie: And when you query the large language model, it augments the query that’s passed the large item with relevant documents. So will feed back with an informed answer on your documentation.

[00:11:12] Will: I’m an infrastructure person. My skill is, clearly talking, but my skill is never being the most technical person in the room.

[00:11:19] Will: But I have enough knowledge to be dangerous as my friend says. But I don’t need to be a data scientist. I don’t need to be a programmer to build that system.

[00:11:30] Ollie: So 18 months from now what does the business look like that got their a AI strategy right?

[00:11:34] Will: I would love to see more new things being invented and developed by companies that were struggling to do that because of the traditional ways, I suppose .

[00:11:45] Ollie: The people who get it right will be able to do more with the existing more workforce.

[00:11:48] Ollie: Yeah, exactly. Some things… people will still be required, but I think that there’s a lot of discussion around this at the moment and I certainly see people augmenting their workflows with it massively and that churn that would take them maybe a week to do want, for a better term, the busy work…

[00:12:04] Ollie: …they can automate, do that and actually spend more time doing stuff that they enjoy doing and in return the business reacts faster and has happier employees.

[00:12:14] Will: Ollie for all those companies out there, how do they start that journey? How do they learn? How do they adapt?

[00:12:19] Ollie: I think the first thing is they ultimately don’t know what they don’t know.

[00:12:23] Ollie: So a case of taking stock of the business, deciding what it is that they actually want to do, and working back from a set of solutions rather than wanting to install a shiny toy. And where we come in as Xtravirt is we provide the platform that runs the workloads.

[00:12:39] Ollie: We can speak to you, the customer, and use our expertise from… we’ve deployed various different private AI solutions across the country, and work back from there. We do work closely with data scientists. We’re able to speak their language and provide them a platform that they can then use and consume.

[00:12:54] Ollie: If you did want to go down that route, ultimately, I think it’s a case of we’ll bring the expertise, in terms of what we’ve seen in the market. The fact that we’ve got actually quite a good close working relationship with yourself over at Broadcom and building on it from there.

[00:13:09] Ollie: Where would you advise people to start?

[00:13:10] Will: At the end. Work backwards as always. Yeah, absolutely. So have a goal in mind. Have something, a problem you’re trying to fix. So identify a problem and, AI might not be the answer, on premises might not be the answer, public cloud might not be the answer.

[00:13:25] Will: What is the answer? What’s the problem? How can we resolve or mitigate that issue? And then look for the skills, look for people that already have the skills. We don’t have them internally, we go somewhere else. So Xtravirt, for example, infrastructure, skills, software development, whatever it happens to be, resource augmentation, and then have that plan in mind and work towards it as an enterprise.

[00:13:51] Will: I know I’ve got to do AI. Everyone’s doing AI. I’m definitely dragged into AI, but I don’t know what, I don’t know.

[00:13:56] Ollie: Yeah.

[00:13:57] Will: How can Xtravirt help me?

[00:13:58] Ollie: We typically like to work with businesses to enhance the value that they deliver from their existing investments. We work with them very closely, throughout the entire lifecycle of the technology stack itself.

[00:14:09] Ollie: We’d like to sit down with you, work through from the start, gather your requirements and ultimately we deliver various platforms, various different use cases. The answer might not be AI, it might be a case of, we’ll look, we’ll sit down with you, look at your processes, look at your business, and work out how we can be effective to work with you.

[00:14:27] Ollie: We provide a platform that data scientists consume. We work with them. We provide an awful lot of supporting infrastructure and supporting services around that as well to really enable you to, if you do so choose, to consume those on-premise AI services. You know, the key one that we talked about before was that RAG solution for your documentation.

[00:14:46] Ollie: That’s usually a great one to get started. So we had a quite an interesting conversation today. What would you say you’ve really learned about?

[00:14:53] Will: I’d say that to me, going to a well-rounded, well-skilled sort of company, a resource augmentation technology company, to help me identify if I have a problem and, if I have, how to resolve that issue.

[00:15:06] Will: So I know that I don’t know everything. In fact, I know the older I get, I know less and that’s key. But I know that there are people out there that will know the answer and it won’t always be the first person I speak to, but I have to start that conversation. It’s…

[00:15:20] Ollie: …knowing who to ask

[00:15:21] Will: as well, right?

[00:15:21] Will: Yeah, absolutely. I know I’ve got a problem and actually it turns out the problem I thought I had, I don’t have, or it won’t be an issue in six months anyway. What I always find interesting is that I never know what use cases are going to come back. So, you know the nice thing about generative AI is that what you want to deliver or what you want to build is limitless.

[00:15:43] Ollie: Oh yeah.

[00:15:44] Will: So, you know, if you have an idea, there’s no such thing anymore as like a completely wild idea.

[00:15:49] Ollie: No

[00:15:49] Will: …because you can build it or you can ask somebody else to build it and deliver it. Would you say that Xtravirt are ideally placed to help me with whatever challenge I have from an infrastructure perspective?

[00:16:00] Ollie: I would say Xtravirt…

[00:16:01] Ollie: …occupy a unique place in the marketplace being the massive Broadcom partner that we are and you know, the expertise we have across a various or vast amount of customers. It’s not just AI solutions, you know, that we deliver. It’s all sorts of on-premise infrastructure.

[00:16:18] Ollie: A lot of customers are just started getting started with this. And that’s certainly what we’ve seen and that’s what we want people to come to us. You know, people will be able to put you in touch with the right people. We usually have a wide roster of experts to talk to. The solution might not be an AI solution, or it might be a massive private AI on premise deployment.

[00:16:38] Ollie: You know, we don’t know until we get in there and start talking to you. Great talking to you Will, I’m sure we’ll see each other again in the future.

[00:16:45] Will: Thanks mate. Yeah, I’ll ping you and we’ll meet up in London again and we’ll do another one of these.

[00:16:50] Ollie: Sounds like a plan.