Mobile Logo Detection in the Wild

This journey starts in late 2016. Part of the data science team’s job was to develop new methods and sources for gaining insights about consumers, approaches that market research companies or brand companies could use to understand better what consumers like and do. In market research, it is common to pay large groups of people (panels) to allow themselves to be tracked online, for example, or for scanning and uploading their receipts.

One idea to measure people’s exposure to brand logos and see what brands they use in their everyday lives is using smart glasses. Why would we want to do that? Traditional market research methods like surveys and phone interviews are becoming less and less popular among consumers. They are also prone to error since our memories aren't perfect. In other projects, we have shown that detecting logos in social media images is a cool and effective way to see consumer trends or measure the influence of marketing campaigns (here is an example) – but what about the world outside of social media?

Tackling that question is where the idea with the glasses came in. Using smart glasses would enable the detection of every brand logo a person sees in a day. A whole panel of carefully chosen people would reveal insights about what brands people buy and use, how frequently they do so, what brands they see in print, online, on TV, in billboard advertisements, etc. We’d be able to collect from all visual sources at once. Building on that idea, by classifying the context of a logo, e.g., “web page”, “billboard” or “real product”, really cool insights could potentially be derived that would be pretty hard to gain using conventional methods and technologies.

Incidentally, I have never looked back to the iPhone world. I was astonished by how developer-friendly Android was (and how much saner its notifications are on the lock screen). But I digress.

Eventually, I got a brand logo classification model, trained on one of the proprietary datasets we had used in previous projects to work with CNNdroid. But I was still using the phone’s camera. Now we needed something like Google Glasses.

As a more complicated alternative, the image transfer could be done much faster via WiFi, however running a WiFi access point on the mobile phone drained the battery of the glasses much quicker. We only got about 1.5 hours of operating time in this mode. Bluetooth was used to control the WiFi transmission, and it was quite sensitive to electromagnetic interference. Other Bluetooth devices in proximity of the glasses would often cause the Bluetooth stack to crash, requiring a restart of both the Bluetooth and the WiFi stack in a defined order. Occasionally though, the WiFi stack would not come up again, and the glasses had to be powered off and back on.

This unreliability that could hit at any moment was something we needed to address. But before worrying too much about the odd Bluetooth interference, we needed to become able to detect logos: The glasses would be of little use if you had to get your nose close to a logo just to have it recognized.

In June 2017, Google released and highly praised their MobileNets, computer vision models specifically designed to run tasks like object detection on mobile devices. MobileNets are Single Shot MultiBox Detector models, using a different kind of single shot detector. Releasing them was a very welcome step as it largely democratized computer vision on mobile devices. Google offered pre-trained models in different “sizes” to balance the trade-off between model performance and runtime performance. Their APIs also contained everything for running the models on mobile devices.

Since training and incorporating a MobileNet into our mobile app would have meant a considerable amount of effort, I had not tried them immediately when I read about them. YOLO was performing well. However, I was intrigued – even more so after my visit to CVPR 2017.

In July 2017, I was fortunate enough to attend the CVPR conference, one of the biggest computer vision conferences in the world, in Honolulu, Hawaii. The conference was purely amazing and the most interesting, exciting, and inspiring one I have ever attended. The location was breathtakingly beautiful and idyllic, too. It felt like a week in paradise, for mind, body, and soul.

I was also lucky enough and super proud to meet Fei-Fei Li, a superstar in the computer vision, deep learning, and artificial intelligence worlds. That called for a picture (which I will put here as soon as I get permission from her).

Being very interested in Google’s MobileNets, I was excited when I heard that Google was about to show them in action at their stand. They demonstrated object detection of little toy cars, toy giraffes, etc. So when demo time arrived, I went to the Google stand.

When I saw the demonstration and the reactions of the crowd, I just couldn’t help myself. I had to show that I, too, worked on mobile object detection. So I went to a vending machine nearby, returned to the Google stand with a can of Coke in my hand, and approached the Googlers with something along the lines of “Yay, I found the object detection demo stand… let me demo some mobile object detection for you!” I then raised my Coca-Cola can, took the Google Pixel phone out of my pocket, and launched our YOLOdroid app. I was quite confident, and the demo worked brilliantly. YOLOdroid spotted the logo and tracked it as I moved the can through the air. The Google people seemed to like it and played around with it for a while.

But the best part was that the little demo got me the opportunity to talk to one of the MobileNet paper authors. So I asked him: How do the MobileNets compare to a (tiny) YOLO model? The essence of his answer was, basically (paraphrased):

That sounded promising. It would mean that there was the possibility of getting even better logo detection without sacrificing detection speed. I packed a pair of colorful Google sunglasses into my goodie bag and decided I would give MobileNets a try when I got back then went on with the conference.

To my good fortune, the author of YOLO was there too! He presented his YOLO 9000 paper, which later received the best paper honorable mention. After his talk, I approached and asked him how YOLO compared to the Google MobileNets, to which he replied (paraphrased):

These two answers contradicted each other. If MobileNets are more accurate but running at the same speed, how could Tiny YOLO be faster? I realized that there was no easy answer and that I would have to put both models to the test.

Fired up and all excited from the CVPR experience, I decided on two major things to do on my way back:

1. Replace the Sony glasses. Instead, I wanted to use a simple USB webcam if possible
2. Try out Google’s MobileNets

Once again, I converted our dataset and put together a training pipeline. I fine-tuned Google’s largest, best-performing MobileNet, implemented the Single Shot MultiBox Detector in the Android app, and tested the MobileNet in the office. My impression was that it didn’t recognize logos as well as the YOLO model, so I started to investigate.

Performance metrics of deep learning models can be misleading. They depend on many factors, especially on the datasets used. If the test dataset is too similar to the training dataset, you might get top performance scores, but your model might have problems generalizing, which you’ll find out when testing your model with real-world data.

When you use the same sources for training and test datasets, both suffer from the same biases. For instance, if you collect images of logos from printed magazines only, then split your data into training and test sets, both datasets suffer from the “how logos look in magazines” bias. When you apply your logo detection to images snapped with your phone while walking outside on a rainy day in bad lighting conditions, you might find that your performance metrics don’t mean anything in this context. They accurately reflect only how well your model might perform on magazine images.

While it seems evident that logo detection trained on magazine images might only work well on images of magazines, we often get carried away by the difficulty to acquire enough training data and forget about potential inherent biases of their sources.

The following comparison is effective, but it is not a fair one. Both models under test stem from base models trained on different datasets (Pascal VOC for Tiny YOLO vs. MSCOCO for MobileNets), and we might not have done a perfect job in fine-tuning either of them. However, we did our best to get reasonably good performance out of both models and to select reasonable parameters when testing.

We evaluated both models on the same dataset in our pursuit of answering the two long-standing, highly related questions:

• Which one performs better: Tiny YOLO or MobileNet?
• Which model should we use for our logo detection?

Aaand the answer is: it depends! Let’s look at some performance metrics:

It was a hard decision to make: Do we want to find more logos, accepting more mistakes, or do we want fewer errors but to miss more logos?

Given that both recall rates (66% vs. 56%) were far from perfect, we tended to opt for finding more logos and accepting a few more mistakes.

Another critical question was: How well do the models work for small logos? When trying to catch logos "in the wild" – while walking outside – the logos in our photos would be relatively small, depending on their distance away. Ideally, we’d want to recognize logos from as far away as possible. When humans can clearly identify a logo, our model should do so too – under ideal circumstances.

Now that we had the technicalities all sorted out, it was time to do something useful with the logo detection glasses: We decided on doing a small case study to verify the approach of capturing consumers’ brand touchpoints via “smart glasses”. We also wanted to contrast the results from automatic logo detection with a more traditional market research method: a survey.

The first idea was to use the glasses while shopping, which we had to discard because taking pictures in shops is not permitted in Germany unless you have permission. We didn’t have it, and it was unrealistic that we would get permission from every shop that we would have planned to visit.

So we changed the scenario: participants in our case study would go on a fictional (window) shopping stroll in the city of Nuremberg, taking a pre-defined route while wearing our glasses and smartphones. We would make sure we knew in advance all the brand logos on our route that were part of the case study. At one defined part of the shopping route, the participants would get a choice to visit either a café or a fast-food chain, to take a small break and get rewarded for their participation with a coffee or beverage. Apart from that, they were to behave as naturally as possible. We would not inform them what the case study was about.

After the shopping stroll, they would need to fill out a short survey asking sociodemographic questions and questions about brands and brand touchpoints. For instance, they were asked to name ten brands that they had seen on the trip, how they liked and used our selected brands, which touchpoints with the selected brands they could recall, the order of these touchpoints, etc.

During the guided walk, the logo detection would track, for every detected logo:

• timestamp
• geolocation
• logo
• logo size and location in the image

In combination with the timestamp, the logo size and location in the image can tell whether someone had been moving toward or away from a logo.

Our next task was to identify suitable brands that were also detectable on a walk through the city, and for which we already had some training images. We settled on five brands, two of which had both a graphical logo and a text part. We would identify text and logo individually for such brands, making seven detectable classes in total.

Our first training approach was to use a different proprietary dataset of 15 brands with 21 classes (some had logo and text to recognize) that we had used in the past. The results were discouraging: 86% precision but only 22% recall on average. Detecting only 22% of logos was not acceptable.

We had already known that the dataset was problematic. It contained lots of small logos, cut-off logos, etc., which were difficult even for server-grade object detection models. So we threw away all the problematic images and collected new ones.

We used three sources: social media images, images found in a web search, and images captured from our glasses!

I had prepared a second pair of webcam sunglasses and went on a field trip through Nuremberg with a working student, both of us wearing our logo detection gear. We tried to capture the logos of our desired brands by looking at them along the way.

To my surprise, nobody even gave us a funny look. People were so busy that hardly anyone noticed we were wearing webcam glasses with electronics and a cable. It was midsummer, so wearing sunglasses was no surprise. Because it was so bright outside, it was difficult to see the green LED on the webcam module that might have attracted attention. Only one young couple smiled at us, which I attributed to us wearing webcam-enabled glasses, but I couldn't even tell if that was the reason.

Even when we conducted the case study, walking along our defined route with 26 people in total (in groups of one to two participants), nobody ever got a funny look or a comment. I was there, I was the guide, and I looked out for potential conflicts with a potentially camera-shy or technophobic person.

Anyway, at the end of our data collection, we annotated the images, drawing bounding box rectangles around every logo, and associating a logo class with each box. Then, we re-trained Tiny YOLO with the following results:

• 90% precision: the detected logo was correct in 90% of all detections
• 76% recall: the model detected 76% of all logos in the test set

We were happy with these results. However, these only showed us the performance on the test dataset. We needed to find out how the model would perform “in the wild.”

As I described above, both a working student and I went to collect additional logo images. However, we only used my images for training, giving us over one hour’s worth of her pictures snapped every second along the way to test on. After annotating the images, we ran the logo detection model on them.

We also drew bounding rectangles around the detected logos and created a little time-lapse video of the result. While I cannot share the entire video, I can present its start and end frames, and a few snippets:

While the whole case study itself is outside the scope of this article about mobile logo detection, here are the key findings:

The Google Maps image above shows a heatmap of all logo detections during the study conducted with 26 participants.

The key findings were:

• The logo detection captures more brand touchpoints than participants could recall from memory: The participants did not remember 35% of the detected touchpoints.
• Recalling brand touchpoints in the correct order was hard for the probands: There is only a 39% match between the detected sequence of brand touchpoints and the one given by the participants.