The Black Box Problem

A lot of times, the use of neural networks is criticized because it’s known as a “black box”. The meaning can be hard to parse for someone unfamiliar because it’s metaphorical. Picture an opaque black box - you can’t see through it at all. You put a purple cube into the box and out comes a green sphere. You have no idea how the black box turned the purple cube into a green sphere; it’s as if a magic trick took place.

This is an issue with neural networks. We know what goes into the neural network, and what comes out, but not what happens within the neural network. From an outside perspective, this can sound quite terrifying. A machine is learning using techniques we’re not privy to. Could this be the start of an AI takeover?

Well, no. We don’t know exactly what neural networks use to make their decisions but this isn’t magic at all, in fact it’s quite common in learning. Especially with the language barrier between humans and machines.

Machines “think” quite differently than we do. How would you determine that there’s a dog in the picture? Maybe you would think well, there’s a four legged, furry animal, with two pointy ears, and a long snout. These are features of a dog. Neural networks have different features the find important. Usually these are some form of gradients of pixel values. In simpler terms, they like to find unique edges and shapes. This doesn’t really mean much to people, but it’s how machines define a dog. So this is our black box: we put in an image of a four-legged fury creature, with pointy ears and a long snout into our neural network, and out comes the label “dog”, even if we don’t know exactly how the machine came to that conclusion.

It’s really not all that mysterious when you think about it. Even different people perceive the same images differently. I recently drew the comparison to how my husband and I view the color red. My husband is red-green colorblind, so he often can’t tell if I’m holding up a red shirt. However, he can often pick out a red shirt in a line up of other colors. I don’t really understand how he sees red and he can’t understand how I see red. When I’ve asked him to describe it he says: it’s a different shade from green so even though they look similar, I can tell when they’re side by side. This is how I think neural nets see too: not quite sure how people can see, but able to draw comparisons from patterns using their own forms of perception.

We often give neural networks a lot more credit than we should. Machines are extraordinarily excellent at detecting patterns. Much like the human brain is, but we often don’t notice because we’re lucky enough that our brains do it automatically. What is one of the greatest strengths of machines, is also one of their greatest weaknesses.

Let’s go back to the dog images. Now I want you to picture an image of a Siberian Husky and a wolf. Could you tell the difference? You probably could. But do you know why you can tell the difference? That’s a bit harder to explain. Maybe the wolf just appears larger, more wild. But this is harder for a machine to grasp. Yet, as Hannah Fry explains in her book Hello World, an AI excelled at this task! The AI could easily tell the difference between a husky and a wolf. That is, until the researchers showed the AI pictures of wolves in summer time. Then they realized the AI was classifying the image as a “wolf” purely due to snow being in the background. Not due to any feature about the animal at all!

In a similar trial, a self driving vehicle failed a test in a similar fashion. It was doing well until it came to a bridge, where the self-driving vehicle promptly started swerving, forcing the researchers to take the wheel before they plummeted. After reviewing their data, they realized their test images for roads all contained grass on either side of the road. So, when the vehicle tried to cross the bridge, it did not recognize it as a road because there was no grass.

That’s something humans do much better than machines. We’re able to parse and conceptualize data. We know that even though there’s no grass, the bridge is just a different type of road. A good thing too, or else travel would be much more difficult.

Tying this back to the black box problem, we don’t know exactly what’s happening under the hood of AI, but it’s not all smoke and mirrors either. The boring answer is it’s a lot of math and educated guessing. That’s why we have to experiment so much with AI, we don’t quite understand what they see and vice versa.