IEEE websites place cookies on your device to give you the best user experience. By using our websites, you agree to the placement of these cookies. To learn more, read our Privacy Policy.
Computer-vision systems use shapes, colors, and even labels to identify materials at superhuman speeds
The Amp Cortex, a highspeed robotic sorting system guided by artificial intelligence, identifies materials by category on a conveyor belt. To date, systems in operation have recognized more than 50 billion objects in various permutations.
It’s Tuesday night. In front of your house sits a large blue bin, full of newspaper, cardboard, bottles, cans, foil take-out trays, and empty yogurt containers. You may feel virtuous, thinking you’re doing your part to reduce waste. But after you rinse out that yogurt container and toss it into the bin, you probably don’t think much about it ever again.
The truth about recycling in many parts of the United States and much of Europe is sobering. Tomorrow morning, the contents of the recycling bin will be dumped into a truck and taken to the recycling facility to be sorted. Most of the material will head off for processing and eventual use in new products. But a lot of it will end up in a landfill.
So how much of the material that goes into the typical bin avoids a trip to landfill? For countries that do curbside recycling, the number—called the recovery rate—appears to average around 70 to 90 percent, though widespread data isn’t available. That doesn’t seem bad. But in some municipalities, it can go as low as 40 percent.
What’s worse, only a small quantity of all recyclables makes it into the bins—just 32 percent in the United States and 10 to 15 percent globally. That’s a lot of material made from finite resources that needlessly goes to waste.
We have to do better than that. Right now, the recycling industry is facing a financial crisis, thanks to falling prices for sorted recyclables as well as policy, enacted by China in 2018, which restricts the import of many materials destined for recycling and shuts out most recyclables originating in the United States.
There is a way to do better. Using computer vision, machine learning, and robots to identify and sort recycled material, we can improve the accuracy of automatic sorting machines, reduce the need for human intervention, and boost overall recovery rates.
My company, Amp Robotics, based in Louisville, Colo., is developing hardware and software that relies on image analysis to sort recyclables with far higher accuracy and recovery rates than are typical for conventional systems. Other companies are similarly working to apply AI and robotics to recycling, including Bulk Handling Systems, Machinex, and Tomra. To date, the technology has been installed in hundreds of sorting facilities around the world. Expanding its use will prevent waste and help the environment by keeping recyclables out of landfills and making them easier to reprocess and reuse.
Before I explain how AI will improve recycling, let’s look at how recycled materials were sorted in the past and how they’re being sorted in most parts of the world today.
When recycling began in the 1960s, the task of sorting fell to the consumer—newspapers in one bundle, cardboard in another, and glass and cans in their own separate bins. That turned out to be too much of a hassle for many people and limited the amount of recyclable materials gathered.
In the 1970s, many cities took away the multiple bins and replaced them with a single container, with sorting happening downstream. This “single stream” recycling boosted participation, and it is now the dominant form of recycling in developed countries.
Moving the task of sorting further downstream led to the building of sorting facilities. To do the actual sorting, recycling entrepreneurs adapted equipment from the mining and agriculture industries, filling in with human labor as necessary. These sorting systems had no computer intelligence, relying instead on the physical properties of materials to separate them. Glass, for example, can be broken into tiny pieces and then sifted and collected. Cardboard is rigid and light—it can glide over a series of mechanical camlike disks, while other, denser materials fall in between the disks. Ferrous metals can be magnetically separated from other materials; magnetism can also be induced in nonferrous items, like aluminum, using a large eddy current.
By the 1990s, hyperspectral imaging, developed by NASA and first launched in a satellite in 1972, was becoming commercially viable and began to show up in the recycling world. Unlike human eyes, which mostly see in combinations of red, green, and blue, hyperspectral sensors divide images into many more spectral bands. The technology’s ability to distinguish between different types of plastics changed the game for recyclers, bringing not only optical sensing but computer intelligence into the process. Programmable optical sorters were also developed to separate paper products, distinguishing, say, newspaper from junk mail.
So today, much of the sorting is automated. These systems generally sort to 80 to 95 percent purity—that is, 5 to 20 percent of the output shouldn’t be there. For the output to be profitable, however, the purity must be higher than 95 percent; below this threshold, the value drops, and often it’s worth nothing. So humans manually clean up each of the streams, picking out stray objects before the material is compressed and baled for shipping.
Despite all the automated and manual sorting, about 10 to 30 percent of the material that enters the facility ultimately ends up in a landfill. In most cases, more than half of that material is recyclable and worth money but was simply missed.
We’ve pushed the current systems as far as they can go. Only AI can do better.
Getting AI into the recycling business means combining pick-and-place robots with accurate real-time object detection . Pick-and-place robots combined with computer vision systems are used in manufacturing to grab particular objects , but they generally are just looking repeatedly for a single item, or for a few items of known shapes and under controlled lighting conditions. Recycling , though, involves infinite variability in the kinds, shapes, and orientations of the objects traveling down the conveyor belt, requiring nearly instantaneous identification along with the quick dispatch of a new trajectory to the robot arm.
AI-based systems guide robotic arms to grab materials from a stream of mixed recyclables and place them in the correct bins. Here, a tandem robot system operates at a Waste Connections recycling facility [top], and a single robot arm [bottom] recovers a piece of corrugated cardboard. The United States does a pretty good job when it comes to cardboard: In 2021, 91.4 percent of discarded cardboard was recycled, according to the American Forest and Paper Association.AMP Robotics
My company first began using AI in 2016 to extract empty cartons from other recyclables at a facility in Colorado; today, we have systems installed in more than 25 U.S. states and six countries. We weren’t the first company to try AI sorting, but it hadn’t previously been used commercially. And we have steadily expanded the types of recyclables our systems can recognize and sort.
AI makes it theoretically possible to recover all of the recyclables from a mixed-material stream at accuracy approaching 100 percent, entirely based on image analysis. If an AI-based sorting system can see an object, it can accurately sort it.
Consider a particularly challenging material for today’s recycling sorters: high-density polyethylene (HDPE), a plastic commonly used for detergent bottles and milk jugs. (In the United States, Europe, and China, HDPE products are labeled as No. 2 recyclables.) In a system that relies on hyperspectral imaging, batches of HDPE tend to be mixed with other plastics and may have paper or plastic labels, making it difficult for the hyperspectral imagers to detect the underlying object’s chemical composition.
An AI-driven computer-vision system, by contrast, can determine that a bottle is HDPE and not something else by recognizing its packaging. Such a system can also use attributes like color, opacity, and form factor to increase detection accuracy, and even sort by color or specific product, reducing the amount of reprocessing needed. Though the system doesn’t attempt to understand the meaning of words on labels, the words are part of an item’s visual attributes.
We at AMP Robotics have built systems that can do this kind of sorting. In the future, AI systems could also sort by combinations of material and by original use, enabling food-grade materials to be separated from containers that held household cleaners, and paper contaminated with food waste to be separated from clean paper.
Training a neural network to detect objects in the recycling stream is not easy. It is at least several orders of magnitude more challenging than recognizing faces in a photograph, because there can be a nearly infinite variety of ways that recyclable materials can be deformed, and the system has to recognize the permutations.
Today’s recycling facilities use mechanical sorting, optical hyperspectral sorting, and human workers. Here’s what typically happens after the recycling truck leaves your house with the contents of your blue bin.
Trucks unload on a concrete pad, called the tip floor. A front-end loader scoops up material in bulk and dumps it onto a conveyor belt, typically at a rate of 30 to 60 tonnes per hour.
The first stage is the presort. Human workers remove large or problematic items that shouldn’t have made it onto collection trucks in the first place—bicycles, big pieces of plastic film, propane canisters, car transmissions.
Sorting machines that rely on optical hyperspectral imaging or human workers separate fiber (office paper, cardboard, magazines—referred to as 2D products, as they are mostly flat) from the remaining plastics and metals. In the case of the optical sorters, cameras stare down at the material rolling down the conveyor belt, detect an object made of the target substance, and then send a message to activate a bank of electronically controllable solenoids to divert the object into a collection bin.
The nonfiber materials pass through a mechanical system with densely packed camlike wheels. Large items glide past while small items, like that recyclable fork you thoughtfully deposited in your blue bin, slip through, headed straight for landfill—they are just too small to be sorted. Machines also smash glass, which falls to the bottom and is screened out.
The rest of the stream then passes under overhead magnets, which collect items made of ferrous metals, and an eddy-current-inducing machine, which jolts nonferrous metals to another collection area.
At this point, mostly plastics remain. More hyperspectral sorters, in series, can pull off plastics one type—like the HDPE of detergent bottles and the PET of water bottles—at a time.
Finally, whatever is left—between 10 to 30 percent of what came in on the trucks—goes to landfill.
In the future, AI-driven robotic sorting systems and AI inspection systems could replace human workers at most points in this process. In the diagram, red icons indicate where AI-driven robotic systems could replace human workers and a blue icon indicates where an AI auditing system could make a final check on the success of the sorting effort.
It’s hard enough to train a neural network to identify all the different types of bottles of laundry detergent on the market today, but it’s an entirely different challenge when you consider the physical deformations that these objects can undergo by the time they reach a recycling facility. They can be folded, torn, or smashed. Mixed into a stream of other objects, a bottle might have only a corner visible. Fluids or food waste might obscure the material.
We train our systems by giving them images of materials belonging to each category, sourced from recycling facilities around the world. My company now has the world’s largest data set of recyclable material images for use in machine learning.
Using this data, our models learn to identify recyclables in the same way their human counterparts do, by spotting patterns and features that distinguish different materials. We continuously collect random samples from all the facilities that use our systems, and then annotate them, add them to our database, and retrain our neural networks. We also test our networks to find models that perform best on target material and do targeted additional training on materials that our systems have trouble identifying correctly.
In general, neural networks are susceptible to learning the wrong thing. Pictures of cows are associated with milk packaging, which is commonly produced as a fiber carton or HDPE container. But milk products can also be packaged in other plastics; for example, single-serving milk bottles may look like the HDPE of gallon jugs but are usually made from an opaque form of the PET (polyethylene terephthalate) used for water bottles. Cows don’t always mean fiber or HDPE, in other words.
There is also the challenge of staying up to date with the continual changes in consumer packaging. Any mechanism that relies on visual observation to learn associations between packaging and material types will need to consume a steady stream of data to ensure that objects are classified accurately.
But we can get these systems to work. Right now, our systems do really well on certain categories—more than 98 percent accuracy on aluminum cans—and are getting better at distinguishing nuances like color, opacity, and initial use (spotting those food-grade plastics).
Now that AI-based systems are ready to take on your recyclables, how might things change? Certainly, they will boost the use of robotics, which is only minimally used in the recycling industry today. Given the perpetual worker shortage in this dull and dirty business, automation is a path worth taking.
AI can also help us understand how well today’s existing sorting processes are doing and how we can improve them. Today, we have a very crude understanding of the operational efficiency of sorting facilities—we weigh trucks on the way in and weigh the output on the way out. No facility can tell you the purity of the products with any certainty; they only audit quality periodically by breaking open random bales. But if you placed an AI-powered vision system over the inputs and outputs of relevant parts of the sorting process, you’d gain a holistic view of what material is flowing where. This level of scrutiny is just beginning in hundreds of facilities around the world, and it should lead to greater efficiency in recycling operations. Being able to digitize the real-time flow of recyclables with precision and consistency also provides opportunities to better understand which recyclable materials are and are not currently being recycled and then to identify gaps that will allow facilities to improve their recycling systems overall.
Sorting Robot Picking Mixed PlasticsAMP Robotics
But to really unleash the power of AI on the recycling process, we need to rethink the entire sorting process. Today, recycling operations typically whittle down the mixed stream of materials to the target material by removing nontarget material—they do a “negative sort,” in other words. Instead, using AI vision systems with robotic pickers, we can perform a “positive sort.” Instead of removing nontarget material, we identify each object in a stream and select the target material.
To be sure, our recovery rate and purity are only as good as our algorithms. Those numbers continue to improve as our systems gain more experience in the world and our training data set continues to grow. We expect to eventually hit purity and recovery rates of 100 percent.
The implications of moving from more mechanical systems to AI are profound. Rather than coarsely sorting to 80 percent purity and then manually cleaning up the stream to 95 percent purity, a facility can reach the target purity on the first pass. And instead of having a unique sorting mechanism handling each type of material, a sorting machine can change targets just by a switch in algorithm.
The use of AI also means that we can recover materials long ignored for economic reasons. Until now, it was only economically viable for facilities to pursue the most abundant, high-value items in the waste stream. But with machine-learning systems that do positive sorting on a wider variety of materials, we can start to capture a greater diversity of material at little or no overhead to the business. That’s good for the planet.
We are beginning to see a few AI-based secondary recycling facilities go into operation, with Amp’s technology first coming online in Denver in late 2020. These systems are currently used where material has already passed through a traditional sort, seeking high-value materials missed or low-value materials that can be sorted in novel ways and therefore find new markets.
Thanks to AI, the industry is beginning to chip away at the mountain of recyclables that end up in landfills each year—a mountain containing billions of tons of recyclables representing billions of dollars lost and nonrenewable resources wasted.
This article appears in the July 2022 print issue as “AI Takes a Dumpster Dive .”
Jason Calaiaro is director of hardware engineering for AMP Robotics, in Louisville, Colo. Before joining AMP, he founded Marble, now CAT Robotics, where he pioneered robots for last-mile delivery. He also developed aerial transportation drones at Matternet, the first FAA drone airline, and served as chief information officer and director of propulsion at Astrobotic Technology, which is slated to be the first private company to land on the moon in late 2022
The only thing I see that wasn't addressed is using product optical bar codes to assist sorting. Perhaps bar codes could be more prominently placed or enhanced to assist sorting and material identification.
This is very interesting and hopeful. Consumers still need to do more by emptying containers and presorting by trash and recyclables and keeping plastic bags out of the recycling waste stream.
Consumers will be better motivated with container deposit laws., which reduce litter.
The goal is reduce, reuse, and recycle in decreasing priority. Thus there is good potential in getting back to refillable containers - even if they are made of heavy duty plastic instead of glass. This is being done in some countries.
The author mentions getting to 100% purity of recyclable materials; that's not possible, but that goal can be approached.
Finally, there will still be a need for some human labor to oversee the automation, to improve it, and to fix it when it breaks down.
The four weigh in on climate change, education programs, and diversity
Joanna Goodrich is the assistant editor of The Institute, covering the work and accomplishments of IEEE members and IEEE and technology-related events. She has a master's degree in health communications from Rutgers University, in New Brunswick, N.J.
Life Fellow Thomas Coughlin, Senior Members Kathleen Kramer and Maike Luiken, and Life Fellow Kazuhiro Kosuge are running for 2023 President-Elect.
Two virtual events were held in June and July for members to get to know the four candidates running for 2023 IEEE president-elect. President Ray Liu asked Thomas M. Coughlin, Kazuhiro Kosuge, Kathleen A. Kramer, and Maike Luiken questions submitted by members on issues important to them.
The candidates were asked about their plans for increasing diversity, equity, and inclusion at IEEE; expanding science, technology, engineering, and math education programs; and ways to attract and retain members. They also spoke about IEEE’s role in addressing the global climate crisis.
The candidates were given 90 seconds to answer each question. The following is a summary of some of their answers. Both events are available on demand on IEEE.tv.
IEEE is committed to increasing DEI within the organization as well as in the engineering profession as a whole. How the organization will go about doing so was top of mind for some members.
Coughlin said diversity gives the organization “strength and multiple perspectives,” and he said he believes that is “important for IEEE members to be able to develop technology to help solve the world’s problems.” One of his goals, he said, is to increase the number of women who volunteer and hold leadership positions.
Kramer said one of her priorities should she be elected would be to make inclusivity at IEEE more effective and equitable for “diverse and global members as the organization advances technology.” She said the organization needs to understand just how diverse and inclusive IEEE regions and sections are and make them “accountable for increasing their efforts.”
Coughlin and Kramer serve on a committee in the IEEE Santa Clara Valley (Calif.) Section that is working to incentivize the chapters to add more female speakers at their events.
Luiken said gender diversity within IEEE is improving but said the organization’s leaders need to “keep their eyes open” to what other types of diversity—including age, cultural, and geographic—need to be increased within the organization. If elected, she said, she will focus on increasing the number of women who become life members because, she said, right now women make up only 2 percent of that membership designation. (To achieve “life” status, a member must be at least 65 years old and have been a member of IEEE for such a period of time that the sum of their age and their years of membership equals or exceeds 100.)
Kosuge said IEEE sections and regions play an important role in increasing diversity and inclusion within the organization. He would tailor IEEE’s strategies for increasing DEI to each region so that they better respect local cultures.
Kramer said her main priority as president-elect would be to inspire and engage the next generation of IEEE members. She acknowledged that the organization has been increasing such efforts but said she wants to do more, as she believes it’s essential for IEEE to play a role in preuniversity education.
Coughlin praised current IEEE programs such as TryEngineering, which provides teachers, parents, and students with a variety of educational resources including hands-on classroom activities. But, he said, IEEE also should produce educational materials for 3- and 4-year-olds. One example he shared was the IEEE science-kit program being offered at libraries in Region 4. The kits teach elementary school pupils what engineers do, as well as how to code, build robots, design video games, and create animations.
“We need to show [schoolchildren] the value of a professional society when they are young,” Coughlin said.
He said he also wants IEEE to support STEM competitions and events held by other organizations such as First Robotics and the Maker Faire. In the First Robotics competitions, student teams design, program, and build robots. At Maker Faire conventions, people engage in hands-on DIY and STEM activities.
Luiken agreed with Coughlin about the importance of making educational materials available earlier to youngsters. She said she also wants to give members more opportunities to go into classrooms and share their expertise through talks or hands-on activities. She said the organization should offer more resources to teachers.
Kosuge reiterated that IEEE has many “amazing” educational programs, and he said he wants to make them available globally and in multiple languages.
The candidates were asked how IEEE could do a better job of encouraging children from underrepresented groups and underserved communities to pursue a STEM career.
Kramer said IEEE needs to offer more mentorship opportunities.
Kosuge said the organization should start at the local level. Chapters and sections could host events or go to schools to engage students.
Luiken pointed out that several such programs already exist, such as IEEE REACH, a program that provides teachers with resources to help them explain the history of technology and the roles played by engineers. She said the program should increase its efforts to distribute the materials more widely. She added that IEEE “needs to work with other organizations around the world, such as UNESCO, to make the materials available to underprivileged communities.”
Coughlin agreed and said IEEE should be collaborating with groups such as the U.S. National Society of Black Engineers to conduct outreach programs.
One of Kosuge’s priorities would be to ensure IEEE’s products and services meet members’ needs—which he believes would help with retention. One of his plans would be to offer services in languages other than English. Many lectures, courses, and publications are offered only in English, he noted. He said that offering publications and services in other languages would not only help IEEE attract new members but also retain current ones.
Coughlin said it’s critical to attract and engage student members because the average age of IEEE members has been increasing. If he could accomplish one thing as president-elect, he said, it would be to make the organization valuable to the next generation of engineers. He said he plans to do so by establishing more local sections and chapters to show young engineers the value of being a member. He suggested that IEEE should sponsor career fairs at universities to attract young engineers.
“If they got a job because of the career fair,” he said, “they may have a bigger interest in becoming a member.”
Kramer said that to retain members, IEEE needs to change the way it communicates with them about membership.
“Membership has become a cost rather than a value,” she said.
Luiken suggested that the organization personalize its interaction with its members.
All the candidates weighed in with suggestions on how to better engage industry professionals. Kosuge said one way is to offer continuing education courses designed for engineers working in industry.
Coughlin agreed that such courses are important because “technology is changing all the time, and we need to address that.” Another one of his strategies is for IEEE to hold senior member elevation events at companies.
If Kramer becomes president-elect, she said, she will develop a strategy to increase IEEE’s member engagement.
Luiken said that to better connect with engineers, IEEE needs to understand what they value and what their goals are.
“Common goals give the organization an opportunity to engage with students, industry professionals, and young professionals,” she said.
She added that she wants to create a program whereby IEEE members—in industry and academia—work with local organizations and government agencies to develop goals.
How IEEE can help address the climate crisis was a popular topic at both forums.
Luiken said one of her goals would be to “mobilize IEEE to help answer the challenges of climate change and build a more sustainable planet.” She wants to create more opportunities for members to share their technical know-how with the world—which she said would “make a true difference.”
“Knowledge breeds responsibility,” she said, “and it’s our responsibility to serve.”
Coughlin said IEEE could increase its climate-change impact by increasing its outreach.
“We need to engage with other parts of the world and other organizations such as the Engineering Change Lab–USA,” he said. “This would help us address climate change and help other organizations see us as a valuable resource.” Engineers and technical organizations use the Engineering Change Lab as a platform to share their perspectives and collaborate on projects.
Coughlin noted that an IEEE ad hoc committee on climate change recently was formed. He reiterated the fact that the organization has the capabilities to impact public policy on climate change worldwide and that “members can be important agents on technical details necessary to deal with the global threat.”
Kramer agreed with Coughlin that increasing IEEE’s outreach is key. The organization needs to play a role in directing the technical aspects of sustainable development and be involved in creating policy, she said.
Kosuge said he wants to create a technology road map to provide government agencies and other organizations with steps to tackle the climate crisis.
To learn more about the candidates and to read their position statements, visit the IEEE President-Elect Candidates website.
The annual IEEE election process begins on 15 August. Be sure to check your email for electronic voting instructions, or your mailbox for your ballot. For more information about the offices up for election, the process of getting on the ballot, and deadlines, visit the IEEE Annual Election page or write to elections@ieee.org.
Your weekly selection of awesome robot videos
Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion.
It’s awesome to see a mobile (ish?) manipulator doing useful commercial work, but I have to wonder about this system where bottles are stocked on one shelf (presumably by a human?) and then have to be constantly moved to a different shelf nearby. Is there not a better way of doing it?
Curiosity has spent 10 years on Mars, which is both awesome and vaguely shocking since I was at JPL a decade ago watching it happen. Here are a couple of videos very briefly summarizing what’s been going on, and what’s to come.
I (still) really appreciate what Canvas is doing, solving a specific problem with tangible value. Robots doing what robots are good at!
I do not understand who these people are that they have this massive trailer containing a robot to make them (just the two of them!) coffee, but the shape of the robot pleases me, so we’ll let it slide.
Look, delivery drones are certainly a technical challenge, and making them safe and robust is important. But I’m still waiting for someone to show me that doing consumer delivery is sustainable and makes sense.
The latest from RoMeLa at ICRA 2022: "Design and Control of a Miniature Bipedal Robot with Proprioceptive Actuation for Dynamic Behaviors."
You’ll have to sit through a lot of “okays” in this video, but it’s worth it for the last few seconds for what happens after the robot finally finishes its kitting task.
FIRST is great, but the competitions they put on are always superchaotic (and confusing) to watch.
You may have done some reliability testing on your robot, but if you really want to know what it takes to build something durable that you can bring back from the dead, check out this video from Ray Billings on a postmortem after one of the best BattleBots in the world got knocked out in the ring.
Marco Hutter’s keynote speech from ICRA 2022.
With IP law firms under pressure to meet client expectations more efficiently, practices are turning to creative workflow solutions and new staffing models
With IP law firms under increasing pressure to meet client expectations faster and more efficiently, many practices are turning to creative workflow solutions and new staffing models. Register now for this free webinar.
Join us for the upcoming webinar, Driving IP Law firm growth amidst staffing and market challenges, as our in-house experts, with combined 40+ years of industry knowledge, share key learnings from the experiences of our IP law firm customers, including the considerations for getting it right and ensuring quality outcomes.
Topics that will be covered: