MIT engineers design a better controller for operating construction diggers

Anyone who’s ever wrestled with a claw machine at an arcade can appreciate the difficulty in pulling and pushing on joysticks, in just the right way, to get a mechanical arm to scoop up that one special toy. Coordinating the joysticks and connecting their movement to the claw’s motion is a type of “mental mapping” that is not immediately intuitive. (And it’s what arcade owners depend on to bring players back, again and again). 

In fact, the mechanics of a claw machine are broadly similar to driving an excavator: An operator uses joysticks to control the digger’s boom, arm, and bucket, and the direction of its cab. But an excavator’s maneuvers are far more complex than anything an arcade claw can do. Operators must learn more complicated mental maps to direct a digger to move rocks, grade soil, clear debris, and dig foundations, among other essential on-site jobs. Indeed, it can often take years for operators to build up expertise in maneuvering the heavy machines.

MIT engineers are looking to shorten the learning curve for excavator operators with a new training interface. Instead of using joysticks, the team has designed a more intuitive controller, which itself resembles a miniature excavator’s arm and bucket. Trainees grasp the device and use their arm and hand to make it move the way an excavator does. A digital excavator projected on an immersive six-screen display mirrors the trainee’s movements in a virtual environment.

“This is a more intuitive way to command the machine,” says Hermano Krebs, principal research scientist in MIT’s Department of Mechanical Engineering. “With this new interface, we can eliminate a lot of the mental maps that an operator would need to build in order to operate an excavator.” 

Krebs sees the interface as a faster way to train excavator operators, as well as a new way to physically operate the machines, both on-site and remotely. 

“Instead of having joysticks, you might have this miniature arm on the side, where the operator would place their own arm, kind of like an exoskeleton, which would allow them to operate the excavator in the cab,” Krebs says. “If work has to be done in a difficult or unsafe environment, you could have an operator sitting off-site in a trailer and using this arm to remotely tele-operate the excavator.”

The team reports its open-access results this week in the Journal of Computing and Civil Engineering. MIT co-authors include Moises Alencastre-Miranda, Joao Buzzatto, and Eran Beeri Bamani, along with collaborators from Sumitomo Heavy Industries, an industrial machinery manufacturer based in Japan. 

A machine mimic

At MIT, Krebs’ group works on human-robot interactions, with a longtime focus on physical rehabilitation. Through this work, the team has accumulated knowledge about the ways in which humans control their limbs and how they can most intuitively interact with machines. 

In 2018, Krebs struck up a collaboration with researchers at Sumitomo Heavy Industries, who were looking for a faster way to train excavator operators. They noted that in Japan, the population of heavy machinery operators is aging rapidly; training their replacements takes time. 

Operators typically learn by driving actual excavators on a controlled driving course. As they operate the machine, novices must learn to relate the actions of the excavator’s joysticks with the movements of the arm, bucket, and cab. Coordinating these actions to carry out actual tasks adds another level of complexity that can take months to years to master. 

The team reasoned that if they could eliminate the need for this mental map, they might significantly shorten the training process. To do so, they looked for a more natural way to control the machine, as an alternative to the traditional joysticks. They soon landed on the mechanical arm design, reasoning that the physical resemblance to the digger’s own arm and bucket could enable operators to mimic and control the excavator’s movements directly, without much mental translation. 

Over the next few years, the researchers worked to build the mechanical arm, along with the software to pair its movements with a virtual simulation of an excavator. The combination of the mechanical arm and the virtual simulator constitutes a new training and control platform for digger operators, which the team has named the “World-Space Interface.”

“‘World-space’ refers to everything in the world that is outside of yourself, or in this case, outside of the excavator’s cab,” Krebs explains. “Normally, operators have to build a mental map of how to manipulate things in the world-space. But now, we can just mime picking up rocks or dirt, and the computer will do that translation to the world-space for us.”

Construction on day one

For their new study, the team ran training experiments with volunteers who used the World-Space Interface (WSI) as well as a more traditional, joystick-based excavator simulator. The researchers developed virtual simulations of 15 realistic excavation environments, including construction sites, highways, forest roads, riverbanks, mining areas, and urban and rural settings. Each virtual environment was associated with various excavation tasks, such as scooping and dumping sand or gravel, digging and grading trenches, clearing debris from roads, removing tree branches from water edges, and breaking up rocks. 

The team designed the experiment to resemble the tasks that an operator typically performs during a weeklong excavator driving course. For one hour each day for seven days, volunteers — both expert and novice — operated the WSI and the joystick counterpart, training on tasks with increasing difficulty. 

The researchers then compared the volunteers’ performance before and after the training period. For the joystick simulator, they found that novices were consistently worse than experts, though they did improve over the training period. In comparison, the team found that with the new World-Space Interface, novices were just as good as experts from the start. 

“In this case, joysticks are a non-intuitive way to control and coordinate the machine,” says study co-author and MIT postdoc Joao Buzzatto. “This is the first interface that does not require me to command the excavator with joysticks.”

The team is now working to add haptics, or feeling to the WSI’s physical arm. The idea is that, as an operator uses the arm to mime an action such as picking up a pile of rocks, the arm will generate a force in response, as if the operator can feel the heaviness of the rocks, as confirmation that the excavator is indeed picking them up. 

“Haptics would make this an even more intuitive system,” says co-author and visiting engineer Solmon Jeong.

The team says the new training interface can be a more natural alternative to excavator simulators that the construction industry is currently exploring. Companies such as Caterpillar, Hyundai, and Komatsu are developing virtual simulators, both to help train operators before they go on-site, and to one day remotely control excavators from a distance. However, these simulators are largely based on traditional joystick controllers that still take time to learn. 

If the team’s new arm-and-bucket controller were incorporated, as an appendage in an excavator cab, or in a virtual, teleoperational simulator, the researchers envision that even first-time operators could get to work, from day one. 

This research was supported, in part, by Sumitomo Heavy Industries.