How Caterpillar and Nvidia Put Voice AI Inside a Construction Machine

Big iron met big computing power at CES — and the result talks back.

At CES this week, Caterpillar and Nvidia showed how AI is moving beyond screens and into steel. The companies demonstrated how AI is now running directly inside construction machines, starting with a Cat 306 CR Mini Excavator that can respond to voice commands and assist operators in real time.

The collaboration puts Nvidia’s edge AI technology at the center of Caterpillar’s push to make heavy equipment smarter, safer, and easier to use on jobsites around the world.

During Caterpillar’s CES keynote, a live demo showed an operator speaking directly to the excavator. The request was processed on the machine itself, without relying on a cloud connection.

According to Nvidia, the system runs on NVIDIA Jetson Thor, an edge AI platform designed for industrial and robotic systems. Speech recognition and responses are handled by NVIDIA Riva, using Nvidia’s Nemotron speech models. A compact language model interprets requests locally, while Caterpillar’s Helios data platform provides trusted machine data and context.

Caterpillar CEO Joe Creed used the moment to connect AI with the company’s broader role in the global economy.

“Caterpillar builds and powers the invisible layer of the world’s modern tech stack,” Creed said during the keynote. “That’s the work Caterpillar does, at scale, all around the world,” Creed emphasized that AI is not replacing physical work but enhancing it, helping operators stay safer, make better decisions, and work more efficiently in demanding environments.

Beyond the jobsite

The collaboration extends past construction equipment. Caterpillar is also using Nvidia Omniverse tools and OpenUSD standards to build digital twins of multiple US factories. These virtual models allow teams to simulate production changes, test schedules, and optimize material flow before making physical adjustments.

Caterpillar said it’s already piloting these factory digital twins as part of broader efforts to modernize manufacturing and supply chain operations.

Data, autonomy, and what comes next

Caterpillar executives say the AI systems also open the door to better use of machine data. The company’s equipment already sends large volumes of information back to Caterpillar, which can be used to improve planning, automation, and future designs.

The company has long operated fully autonomous vehicles in mining and views the current construction pilots as a step toward broader automation across its equipment lineup.

Nvidia, for its part, sees partnerships like this as central to its strategy. In a comment to TechCrunch, Deepu Talla said, “Physical AI is the next wave of AI.” He added that Nvidia is focused on building the computing platforms needed to train, simulate, and deploy AI into machines, “whether [that’s] an autonomous car or a Caterpillar machine.”

Alongside the technology push, Caterpillar announced a $100 million commitment over five years for workforce training and education. This includes a $25 million Global Workforce Innovation challenge to prepare workers for AI-enabled industrial systems.

Also read: Nvidia unveiled Alpamayo, a ‘thinking’ autonomous vehicle AI that can explain why it’s making a decision.