WARREN, Michigan — General Motors Co. has developed a software-testing lab inside its Global Technical Center that it’s replicating elsewhere, including in GM’s Silicon Valley office, to prevent bugs from entering new vehicles and affecting customers.
The lab was born out of GM’s software struggles, which included a stop sale on the Chevrolet Blazer EV for intermittent issues with in-vehicle screens and DC fast charging at certain public stations. It’s one of the new software testing “innovations” at GM “that’s rapidly increased the efficiency and rigor of software testing, catching 10 times the number of defects in development, and doing it earlier in the process,” Dave Richardson, GM’s senior vice president of software and services engineering, told investors and analysts at the automaker’s early October Investor Day event.
GM isn’t alone in the battle to keep software glitches out of vehicles. Automakers from Tesla Inc. to GM’s crosstown rival Stellantis NV and Volkswagen AG have all dealt with software woes, with some leading to recalls.
In fact, more than 41.6% of all vehicles recalled so far in 2024 involved software issues, up from 14.9% in 2023, according to Envorso, a Detroit-based software consulting firm that tracks software recalls. This year’s software recalls were driven by two each from Tesla and Stellantis.
Automakers have historically handled software in ways that doesn’t mesh with what’s needed for today’s vehicles, experts say.
“It’s kind of the difference between (Apple co-founder) Steve Jobs and Henry Ford. Henry Ford thought about how you did an assembly line and how you put all the pieces together on the assembly line. Jobs would think about it from the experience and how the software integrates with the hardware and working that experience through before you do manufacturing,” said Todd Warren, a senior adviser at Envorso who specializes in software strategy for the automotive industry.
Part of the issue is that legacy automakers assemble parts that they get from suppliers — and those parts include software.
“Quite literally, they put all those pieces together on a table and wire them together and then wonder why the software doesn’t work,” Warren said. “You can’t do it that way. … The further you get from when the software developer types in the code to when you find the bug, the more expensive and slow it is to fix that bug.”
That’s why automakers like GM are rethinking their software strategies by shifting the software testing “left,” as Richardson talked about during Investor Day, because “if you’re finding those bugs when you’re driving the car, it’s way too late. It’s hard to track down, it’s slow to fix.”
Following its series of software woes with the Blazer and other new EV products, GM’s software team is focused on that “shift to the left” and has moved software testing and validation “as early in the development process as possible before all software components are integrated into the final product,” Richardson explained at the Investor Day event.
To facilitate the shift, GM integrated tech-industry standard tooling and processes for developers to write better code, tracked oversight of quality, introduced testing automation, began testing systems in the cloud and “built a global network of software quality labs, where we have hardware benches accessible by developers anywhere in the world running suites of automated tests,” said Richardson, who joined GM in September 2023 and was promoted to his current position in June.
GM is replicating the Warren lab at its technical center in Canada and at its recently opened Mountain View Technical Center in California.
Some of the changes have led to difficult decisions. In August, GM said it had laid off more than 1,000 salaried employees in its software and services organization globally — including more than 600 employees working at the Global Technical Center in Warren — in an effort to streamline the division’s operations under new leadership.
Richardson told investors the workforce reduction “was absolutely critical to GM’s future in software, and we will continue to make bold choices to move faster, to pivot when needed, and prioritize investing in what will have the greatest impact. We have simplified our team structures, to remove unnecessary layers, avoid duplication and enable speed.”
Inside the lab
In GM’s software quality lab, which opened one year ago, there are “benches” or cockpits for every vehicle the company produces. Software is tested on one module, then graduates to a bench before it’s put into a full-vehicle integrated bench.
“You’re checking the code at each section, which ensures better quality over the testing,” said Amy Talerico, director of test infrastructure, software labs, vehicle fleets at GM. “Any software for these modules comes here to be tested, regardless of recall. Certainly, if there’s a recall remedy in play, it’ll come here to be checked to make sure it’s confirmation of good quality before it ships out.”
Before the lab, developers maintained their own benches, Talerico explained, but that could lead to confusion about who tested which software set.
“We did an organizational change because really, our organizations were too siloed, individuals working on individual things, including on benches, right?” Talerico said. “So software developers kind of working in their own little lane and not understanding how they interact with one another when they come into a vehicle platform.”
At the lab, which can be accessed remotely, the team collects data on the assets there and in other labs to track progress.
The real evidence of the lab’s effects is evident to Talerico in GM’s recent launches of the Chevrolet Silverado EV, Chevrolet Equinox EV and GMC Sierra EV.
“The feedback from the customers is really the proof point that what we’re doing here matters and is making a difference,” she said. “Especially with respect to the Blazer. That’s something we’re not proud of, but we really doubled down our efforts and got serious about our organization, about how we test software.”
What’s behind the challenges?
Software in vehicles isn’t new, but the industry has faced challenges in developing software with modern methods.
In the past, the approach was to use “deeply embedded software,” so vehicle functions like anti-lock brakes or sequential turn signals had individual electronic control units, or ECUs, with software for each, explained Sam Abuelsamid, principal e-mobility analyst at market research firm Guidehouse Inc.
“All of those things had been traditionally developed independently of each other. They don’t talk to each other,” he said. “The software is tied, typically, directly to that hardware, that ECU, that it’s running on.
“What we’re seeing now is a transition to a different kind of electronic architecture, away from having these distributed ECUs around the car, so where you’ve got 100 or more computers installed somewhere in the car, to getting down to a small handful of computers, usually with one or two large centralized computers,” Abuelsamid said. “… and then maybe some zone controllers, and then having a common software platform, and all of the software from all those individual ECUs is being brought into that central compute.”
Embedded software wasn’t designed to be updated. With the transition to software-defined vehicles, there’s an expectation that the software will be updated to provide new features and functions over time.
“You have to look at how you develop the software in a whole different way,” Abuelsamid said. “You need different kinds of processes. You need a different kind of organization to develop and test and maintain that software and update it over the life of the car.”
Now automakers have to transition from having small teams work on specific functions to a larger organization responsible for the software platform, which can be challenging, especially when adding new tech hires who aren’t used to some of the limitations in the auto industry, Abuelsamid said.
“One of the challenges is bringing in software developers from these other industries to the auto industry, in those other industries, developers have more freedom to make changes … on a more frequent basis, without having to worry so much about the consequences if something broke,” Abuelsamid said. “But when that happens in a car, the consequences of that kind of failure are much more severe.”
The changes GM is making appear to be steps in the right direction for Abuelsamid, but “whether it actually is successful remains to be seen.”
Like GM, other automakers have made changes to address the software woes they’ve experienced. Volkswagen, for example, is partnering with EV startup Rivian Automotive Inc. In June, the companies announced a joint venture, which they launched Tuesday, to create next-generation software-defined vehicle platforms to be used in both automakers’ future EVs.
“Increasingly, automakers that were counting on doing this on their own are now recognizing, ‘OK, this is a lot harder than we thought,’” Abuelsamid said. “And they’re working with partners that have more expertise in that kind of software development.”
