CI/CD gets us pretty excited anyway, but it's not often we get to talk about how it improves something as cool as a luxury car. Chris Hill, Head of Systems Engineering for Infotainment at Jaguar Land Rover, recently shared his own team's journey from feedback loops of 4-6 weeks to just 30 minutes, in this inspiring talk from DevOps Enterprise Summit London 2018.
Key takeaways from Chris' talk
What's needed for transformation
How you respond to complaints matters
"Equally if not more important than the complaint itself, is the response or reaction to the complaint. 'Can I bring a complaint, that I know my voice is heard and that somebody cares about resolving my issue?'"
"'I asked the ops team three weeks ago to add a build dependency on the build servers, and it still hasn't been added. I'm just going to go back to building on my own.' This complaint obviously is a knife right to the heart because you feel like you've started to regress. But what I like about this complaint is it led to a behavioral change as well as a technical change. We decided instead of continuing the same direction, to move to ephemeral Docker containers to run all of our builds. With ephemeral Docker containers we defined every piece of build infrastructure as code. We used packer recipes to find a Docker container, and every app developer could now change the underlying infrastructure which built their application. They were empowered. They now had the self service to do their lifecycle on their own. And you're never going to receive the ops complaint because you've handed over the keys."
Efficient feedback loops are critical
"Our feedback loops were 4-6 weeks. Could you imagine writing code today and six weeks from now being told whether or not it works or is broken? I don't remember the shirt that I wore yesterday, let alone what I had for breakfast this morning, let alone what I wrote six weeks ago, and chances are I've been working on features for the last six weeks, and for me to try to unpick what I was thinking at that point could be a huge context-switch penalty."
"Infotainment also had a significantly higher number of contributors – up to 1,000 contributors. And what we noticed is that contributions don't come linearly, they come in bursts. We actually found that Thursdays were the day that most of our developers committed on. And when we had manual code reviews, if we didn't have reviewers ready on a Thursday, we would create our own backlog."
Deployments don't have to be limited to a traditional release cycle
"How could we change the game? Instead of ditching the combustion engine, we ditched the dealership visits, and we implemented software over the air. And this huge Linux distribution that we build upwards towards 700 times per day in a continuous integration pattern, on a dev branch or a master branch, or a release branch, we can now deliver to every vehicle in the form of small, incremental deltas. We can also deliver it to the vehicle while you're driving, and not interrupt your daily life. In fact I showed Gene yesterday, we started a download and an install while I was driving, and the entire thing happened in the background. Jeff even made the comment, 'This is blue-green deployment for vehicles.'"
"One of my favorite indicators is deploys per day, per developer. But I was always embarrassed to share ours because it was always below one. All of our new software wouldn't actually make it to vehicles; it was always batched together. Now I'm happy to say we can deploy, and we have been in our engineering environment, 50-70 times per day of each individual piece of software to a target or to a vehicle."
"No longer are deployments limited to a traditional software release cycle. We've now skirted every single process to get a technician a new piece of software, and bother somebody else's day – one of our owners – to come into a dealership and spend an hour waiting for their vehicle to be done. We've now empowered the customer to be their own technician."
