Transcript:
Patti Harman (00:06):
Welcome to this edition of the Dig In Podcast. We're so glad you could join us today. I'm Patti Harman, Editor-in-Chief of Digital Insurance.
The adoption of artificial intelligence across the insurance ecosystem is rapidly changing and companies that want to remain competitive are finding that AI adoption and integration are mandatory across their operations. This changes how company departments interact. It affects a company's risk profile for managing and protecting data. It can significantly improve and personalize the customer experience, and overall, AI can just be an invaluable tool.
Here to discuss all of this and more today is Daniel Herrington, Chief AI Officer for The Zebra. Previously, he was the senior product manager at Google and he has some great insights to share on how to integrate AI and what roadblocks or issues can affect its use. Thank you so much for joining us today, Daniel.
Daniel Herrington (01:08):
Yeah, thanks, Patti. I'm so excited to do this podcast with you. Thanks for having me on.
Patti Harman (01:13):
It's such a hot topic and your insights are going to be so invaluable. So I've noticed that more and more companies are adding a chief AI officer to their teams. Can you explain for our listeners what this position entails? Do you work across multiple departments, sales, service, underwriting claims? What does it look like in a company?
Daniel Herrington (01:36):
Yeah, I have to pinch myself sometimes to realize that we get to have chief AI officers now. I've been working in the machine learning AI space for a long time, and it's such a cool time and space to look across all industries and things like even insurance and regulated spaces are hiring chief AI officers.
So first off, I have to stop and say, "I can't believe this is real and I get to do this because it's just so much fun to do. " And it's been a dream to be able to have this type of role. The second off is we don't know. We don't know what the chief AI officer roles are. Yeah, they're hiring. And I think it's rapidly changing depending on the industries and what gets introduced. I think when I first started talking with the team of The Zebra, being an AI officer or being the AI product team was all about how do you put a chatbot on the website?
(02:33):
And we've seen this technology grow just so fast, so quickly that the chief AI officers are finding their hands in everything from how do you organize the data to how you interact with the customers to how you write and develop the code, getting to touch all sorts of different areas of the business.
Patti Harman (02:49):
Wow, definitely no silos when it comes to implementing AI. Where do you fit into the corporate structure? Do you usually interact with the chief technology officer, the CISOs, other technology leadership counterparts? What does that kind of look like day-to-day?
Daniel Herrington (03:09):
Yeah, I can tell you how we're doing it at The Zebra. And I have a few colleagues that I've spoken to see how they're setting up their org structures as well. Currently, I'm reporting to the CEO and my colleagues are the CTO, the chief data officer team. My direct reports are non-existent. So I operate as a separate entity so that I can move fast and work across these different teams.
Originally, my mandates were working with the VP of data, working with the CTO, trying to understand how we could build products. And quickly it evolved into, oh, how do we reach into all the different areas of the company and interact? I've seen a lot of other areas. My colleagues in the travel space are operating in these CIO plus CAIO roles. So they're starting in the chief information officer seat as that's something that's familiar. And then taking on the artificial intelligence officer role with that to try to identify where those two roles separate in the future or if they become one.
(04:14):
I see that as a path that others are doing as well. But it's very clear that you need to have a close interaction with the data team and you need a very close interaction with the customer team, whether that's coming from the product side or the engineering side.
Patti Harman (04:28):
And you have to be really flexible because there's just, as you said, there's just so much changing on a daily basis. What are some of the biggest challenges for companies when it comes to trying to integrate AI into their operations? What are you seeing?
Daniel Herrington (04:45):
I think we like to compare the AI transformation to the digital transformation. So many of us went through that as technologists of moving into mobile even going from desktop to mobile interfaces and how we had to reinvent our processes on that.
The AI transformation is happening so much faster and it enables small teams to do such big leaps so quickly that I don't think we're going to be afforded this slow-moving change because the customers are adopting these tools so much faster. I'd have to go pull the numbers and maybe we'll get a chance to do it at some point. I'd love to look at what the mobile adoption speeds look like, the time to first million users and mobile time to first billion. I've got to imagine that the AI trends are just setting records there and that's causing companies to have to do radical shifts and not being able to plan for these transformations.
Patti Harman (05:48):
I would agree with you on that. I saw some numbers, but that was like six months ago. So now those numbers are super, super outdated, but the adoption, you're right, has been absolutely insane. And with AI changing almost daily, how do you stay ahead of it or at least abreast of how it's changing? Because what it was last week, a new version is probably already out and it's already changing what can be done and how it's handled.
Daniel Herrington (06:18):
Yeah, I'll talk about that on a few fronts. One on the diversity of tools that we use, and maybe we'll start there. Previously, when you wanted to pick a stack on how you did development, entire companies would sit down and decide, this is our stack. Most traditional, we're going to use the lamp stack. We are going to build this way. These are the development tools we're going to use.
As you mentioned, things change so quickly. The state of the art changes so quickly. If you're deep into agentic coding, you may be using CloudCode one week and then find out that Google Anti-Gravity becomes the best the next week, and then OpenAI's Codex becomes the best the next week. And so you can't just pick one platform as a company and say, "This is the one we're going to use for all of our developers or all of our designers." You kind of got to keep a foot into each one of them and kind of keep a warm contract to see which way things go because month to month they can change.
(07:23):
The second is the education part of it. How do you learn about these tools? And we've had to go full crowdsource inside The Zebra. Originally, I guess in the beginning of February, we did a Vibethon event where various members of the Zebra team actually across the company were able to pick up tools and try to vibe code, build products for our customers that we saw would be useful. And originally, I did some of that training content for the team so that we would just get started on the tools, but we quickly had to change that into a shared space because people were learning new techniques so quickly and sharing that content internally. There wasn't just a course that they could go take to learn how to use these tools. The companies hadn't released those yet. So we had to teach each other and go back to being excited to learn, which for so many engineers that came naturally, but it was pretty amazing how naturally it came to just the whole Zebra culture.
(08:28):
There were so many curious people that just wanted to learn to learn these tools. And it's reaping benefits by how quickly they can pick up the new ones and stay on top.
Patti Harman (08:37):
I love hearing that because the insurance industry has always had this reputation of being really slow to change. And I think the pandemic kind of turned that on its ear. And now with the adoption and the use of AI, it's like you're going to be left in the dust if you're not really making this a priority. And I think a lot of carriers are seeing that.
As you're integrating AI, is part of your role risk management in terms of protecting the data and the processes? And I don't know if you get into the coding part and making sure that the algorithms are correct so that the outcomes are correct as well.
Daniel Herrington (09:19):
Yeah. The risk mitigation and risk acceptance, I'm squarely in the middle of, I believe it's my responsibility as the chief AI officer to help our legal and the rest of our executive team understand exactly what type of risk we may be taking and where the holes and just where all the unknowns are. There's a lot of unknown unknowns here, and we at least need to make the known unknowns apparent. And that's the first point of where we're diving in.
Patti Harman (09:51):
When you're getting this information, how do you verify that what the AI is providing is accurate to your teams?
Daniel Herrington (09:59):
Yeah, there's a whole topic on grounding in evals that I think we should spend some time on, Patti. In the ensuring that the AIs don't hallucinate is going to be a challenge that we perpetually deal with. I would assume for the next three to five years, if not longer, initially, the hallucinations were easy to spot. The joke was that AI was either spooky or kooky, and it was very kooky sometimes in those hallucinations you could see from a mile away. The hallucinations are still there. They're just better at hiding. And so the opportunity for AI to mislead us is becoming harder and harder to spot, whether it's in coding, whether it's in customer service agent tooling that you're building, whether it's in recommendations tooling that you're building for your own team, it's getting harder and harder to spot those things.
Patti Harman (10:59):
Yes. I was talking to some folks last week and we were talking about the use of AI in terms of insurance fraud and different things. And I said, it's even changed that because it's no longer the email from the Egyptian or Iranian prince or whatever. It's like now you're getting messages from people that you know and it's very realistic. And I think it's the same thing with AI in terms of being able to identify some of these hallucinations.
So we're going to take a short break now. We'll be back in just a few minutes.
Welcome back to the Dig In Podcast. We're chatting with Daniel Harrington, Chief AI Officer at The Zebra about the benefits and risks of adopting AI across your company. So we've talked about how excited people are at The Zebra about using it. What have been some of the challenges in terms of getting employees to use AI that you've seen?
Daniel Herrington (12:01):
Yeah. Before the break, we talked a lot about the learning curve and how there's not a ton of content out there. And so much has to be self-learned. So it's got to be based on a willpower and a desire to learn. So I think one of the biggest challenges is none of this is spoon ready. There's not a clear paved path on how you use this tooling. And so there's just a feeling from just about everyone, even from myself of, I don't think I'm doing this right. The outcomes look good. I need to go bounce this off another person to make sure we agree and align. And that seems to be one of the biggest bottlenecks for just getting people to adopt it. The second one is also we talked about just before the break, the hallucinations and the evals are something that I'm very passionate about making sure we solve for and do correctly.
(12:56):
And we spend a lot of time testing the different LLMs that we use and build to ensure that the responses from the LLMs are fine-tuned for the scenarios that we care about. We want to make sure that we're meeting regulatory guardrails, meeting compliance or any risk that we may be introducing, as well as just making sure it's factually correct when we have responses and when we spend a lot of time doing real evals based on transcripts from customers, as well as synthetic evals of transcripts that we think could exist in the future and ensuring that they respond the way we want them to.
Patti Harman (13:30):
Yes. We've been talking just within our different editorial teams about how we can integrate AI. And I will admit there are times when it changes what I'm writing or other things. I'm like, " AI, stop. No, you don't know this yet. "And then I think, oh no, it probably understands what I'm saying, which again is a little bit scary.
What are some of the biggest issues when it comes to integrating AI into your operations? And I don't know if it's like inputting the data or setting up the guardrails or upskilling or even protecting the data that you have. I'm sure there are a lot of different aspects to that.
Daniel Herrington (14:10):
Yeah. And data protection is obviously an extremely large one. We're working with some very sensitive customer data and we want to ensure that that's as safe as possible. The AI tooling has the ability to really throw gasoline on the fire here. So we've got to make sure that it's in a controlled environment that we're not letting things out there. Guardrails on how things sell. We're studying on how humans have sold policies for the last, gosh, 10 years at The Zebra. So we've got tons of really useful data here. We need to make sure that that's a model that we want to use and that we're teaching the model not to reward hack or just maybe to define that the model will find the best way to achieve its outcome if that may not be what we define as best. So it will find a way to not lose deals, as an example, will be to hang up on every customer because it didn't lose that deal.
(15:12):
So we have to watch out for the reward hacking on the model and ground with a human. One of the features of our industry is that we have humans that can maintain an advisory role to oversee what the AI is doing. And in many other industries, they're going full on without that. And fortunately, I don't see it as a blocker in our industry. I see it as a great feature that we get to work alongside humans all across this process.
Patti Harman (15:40):
I agree because when it comes down to it, insurance is still very much relationship based and that human empathy and input and everything else just plays a key role in it. One of the things I think is kind of exciting is how it's changing and improving customer service. Can you explain for our listeners how that integration into company operations is just really kind of changing what we're seeing in the customer service realm?
Daniel Herrington (16:10):
Yeah. The automation parts of AI, I think most people can visualize. So all of the data entry processes, all of the different things that we had to do on the backend to go pull documents or coalesce data, I think it's pretty common for people to say," I see how AI makes that great. "And so when you're speaking with an advisor, whether it's from a service standpoint or a sales standpoint, they have so much more data at their fingertips that previously they had to manually type in and go get and automate that from the policy sides and the content sides.
But the other side of it that I think we often miss is how much extra context and personalization that's available to our advisors that wasn't previously available in their interactions. When people would come through, say a Google search or even through the funnel, they were searching for, " I need car insurance, "those two keywords.
(17:15):
And they were coming to our advisors with a little bit of information, but for the most part, the context was very limited. Because we're able to interact with these customers via these LLMs, their searches on Google are significantly more complex. I need car insurance for my Honda Civic in Austin, Texas. I park the car outside. It's a 2022.That's the keyword search that we have coming in and there's so much extra context by the time they get to the advisor. You mentioned it's relationship based. We're giving our advisors all of that extra content and context to build those relationships with the customers, and that's just a really cool change in the process that didn't exist before.
Patti Harman (18:00):
Yes. I would totally agree with that because when you talk to somebody and they know what you're looking for, what you're worried about, it just totally changes the conversation and it sets the advisor up as an expert who can really answer the questions that are being asked. Where do you think AI has the most potential for the insurance industry? And I should probably put the caveat at this point in time, because it's changing so fast.
Daniel Herrington (18:30):
Yeah. So first off, The Zebra is a marketplace for people to come and search for insurance, and they can purchase insurance with our licensed advisors. We don't get into the underwriting portion of the process or the claims management side of that. Those areas are super interesting, and I think there's tons of opportunity. The only reason I'm not bringing them up is because that's not something we focus on. We spend all of our time in helping people demystify and understand and maintain the education of what is it that I'm buying and then help them actually make that purchase. And so that's where I get most excited is because of where we spend our time at The Zebra. I am so excited about agent-to-agent as an opportunity in the future. I think that's near term. I don't think it's that far out. So when we talk about all this extra context that we're able to provide to our licensed advisors about the type of car that someone purchased or why they're looking for insurance, if we think of an agent to agent world, imagine one layer higher when they're actually purchasing that vehicle.
(19:37):
Imagine they're purchasing it on an Agentic experience. So I've previously worked in e-commerce for used cars. I built recommendations, engines, and pricings and cars, used cars in a previous role, and we had basic filters. You came in and you said," Hey, I'm looking for a car that's under $30,000 and I'd like a Toyota." And now we have people searching with so much more context. I need a car for a family of three that can fit a new baby stroller and is easy to get a car seat in and out of. That context that will be brought into the Agentic shopping experience for car purchasing is going to be so valuable in passing the context into the insurance buying. And so being able to take all of that agentic knowledge that happens in the car purchasing and passing it over to our licensed advisors and an agent to agent flow is just such an awesome opportunity to take the friction out of the customer engagement angle and have that hyper-personalized best coverage, best price available for every customer.
Patti Harman (20:38):
I agree. I just saw a study that came out of JD Power and what they found was that the customer experience has really improved. AI is helping with that. And so being able to do the hyper-personalized policies and give them that information, I think it all goes to giving them just a much better customer experience overall. There are some concerns that AI might take people's jobs. What are the risks that you see with its use and do you think that they offset the benefits?
Daniel Herrington (21:12):
I think it's a naive assumption to say that there won't be some jobs taken by AI. It's very reasonable to say that with every major technological shift, we've had job displacement and creation. We see so much demand for software developers right now in a time period where AI software development is absolutely exploding. And so there will probably be some roles that go through a transformation. In the insurance space, I'm sure there is a set. What I see though is a demand for more and more understanding of how we do data science models, how we do interaction layers with the customers. We're increasing the size of our engineering and data science team so that we can build these personalization tasks for the customers. So I think it's inevitable that we'll see a shift in the roles and that the students that are coming out of high school today will have completely different roles when they hit the main parts of their careers as AI helps shift and change what it is that we need to solve for.
Patti Harman (22:22):
Yes. I would totally agree with that assessment. It's sort of like when the internet came out and you realized, oh, I don't have to go to the library now to do research. I can just type it in on my computer, this is what I'm looking for and things will pop up. And I think that it's going to change how we work and what we do, what the workday looks like and what the opportunities are.
We have covered a lot in the last couple of minutes. Is there anything I haven't asked you, whether it's about your job or AI or just something that you want our listeners to know?
Daniel Herrington (22:55):
We spoke briefly about the evals and the impact there. I really think that's something, it's an opportunity for us to make a miss here, but I think it's also an opportunity to partner with the regulatory and learn some incredible things. I got to work on some of the efficiency of the Waymo models at Google, which is super interesting. And it's been really cool to see them deployed. The Zebra's in Austin, Texas, and there's Waymos everywhere there now. I think as an analogy, Waymo has uncovered so many places where a bush is were covering the stop sign and it has identified ways of making human driving so much safer. And I'm excited to see what we find as we dig into synthetic evals and we start driving those roads with our personalization agents and engines. I think we're going to find a lot of these places where the bushes have overgrown the stop signs, and I'm so excited to work with the regulatory agencies on this to figure out what this needs to look like and get to be thought leaders in that space too.
Patti Harman (24:02):
Wow. That really is exciting. Thank you so much, Daniel, for giving our listeners a great overview of AI and how they can integrate it into the operations and what issues to watch out for as part of that process.
Thank you to you for listening to The Dig In Podcast. I produced this episode with audio production by Anna Mints. Special thanks this week to Daniel Herrington of the Zebra for joining us. Please rate us, review us, and subscribe to our content at www.dig-in.com/subscribe. From Digital Insurance, I'm Patti Harman, and thank you for listening.
