Comparing waste detection approaches on extended TACO dataset

Differentiating waste instances under a single class label is challenging task

Detection combines two tasks: localization and classification of the objects in the image.

We have been struggling with the task of detecting garbage in the environment for almost 5 months. In our previous post we described some metrics to compare performance achieved by different models. Now is the time to present our first results.

Let’s get back to the data

In the Detect Waste in Pomerania project we intended to base our research on TACO dataset - an open image dataset of waste in the environment. As we wrote in one of our first posts, we converted TACO categories to detect waste categories according to the obligatory segregation rules of the city of Gdańsk. Additionally, around 3k images were tagged by our partner Epinote with bounding boxes. In this way, we started with waste detection performed for 6 classes - paper, glass, metal and plastic, bio, non-recyclable, and other - on an extended TACO dataset (almost 4.5k images).

Unfortunately, there are types of waste that do not fall into any of above-mentioned categories - very crowded photos with garbage obscured by various objects, where it was not possible to tag individual garbage, and trash that could not be clearly identified due to their state of degradation. We have created another category for it - “unknown”.

Crowded and hard to recognize waste from TACO dataset.

Detecting only metal and plastic in 7-class detection challenge

We conducted our experiments on an EfficientDet network, written in PyTorch. EfficientDet is a family of object detection models that came out of the Google Brain team. It is based on the EfficientNet paper, which achieves one of the best performances on the image classification task. In our case EffcientDet-D2 gave us the best possible results. Smaller architectures got lower scores, whereas bigger ones quickly overfitted. We hypothesize this is because the dataset itself is small (roughly 4.5k images) and highly imbalanced.

Average Precision (AP) at IoU=0.5 achieved for TACO with EfficientDet-D2.

The best result was recorded for the “metal and plastic” class. It is not surprising at all, considering the fact that a given class makes up almost 45% of our imbalanced dataset. Generalizing, our model treated almost every object it found as a plastic one. The worst results were achieved for the smallest class - “bio” with only 69 instances (0.5% of the dataset). The “other” class was also not very successful due to the large variety of other item’s kinds (from rubble to medicines).

Some example instances from poorly recognized class “other”, like medicines, rubble, and batteries.

Sometimes less is more: 1-class detection task

Detection combines two tasks: localization and classification of the objects in the image. We have noticed that our model performs quite well when localizing waste but works poorly with classification. Moreover, we have massive data from OpenLitterMap, with thousands of images labeled for the multi-label classification task (few categories in one image). We have decided to try single class detection instead, and separate classification. This allowed us to obtain an Average Precision (AP) on one litter class above 50%, which is a much better result than the one reported in the article describing the detection on one-class TACO (16%).

Average Precision achieved for one-class TACO with EfficientDet of different sizes.

Some example predictions made by: (a) 1-class, and (b) 7-class approach.

Future works

Good results, obtained for detection in a single class, prompted us to dive into these studies also on other datasets - representing the detection or segmentation of waste in various environments. This is the first post in a series describing our results for garbage detection on public datasets. Look out for more soon.

Sylwia Majchrowska
Sylwia Majchrowska
Deep Learning Researcher, PhD Candidate in Physics, STEM Teacher

PhD student in Physics at Wroclaw University of Science and Technology, and Deep Learning Researcher at NeuroSYS. She is a big fan of Python, eager to learn more about data science and machine learning.

Agnieszka Mikołajczyk
Agnieszka Mikołajczyk
Passionate PhD Candidate, ML researcher at Voicelab, PI in pre-doctoral research grant

Author of scientific papers and publications interested in the issue of machine learning, explainable AI, image analysis as the way of improving people’s everyday life. A PhD student who cares about the further development and training opportunities within interests.