Neural Nets 2

TUTORIAL ON MACHINE LEARNING AND NEURAL NETWORKS (PAGE 2)

Tutorial Pages

1. Intro
2. Network Inputs
3. Network Activation
4. Network Outputs

The previous page ended by mentioning that neural net classifiers can fail for two independent reasons:

1. The neural network itself is ill-formulated, so no matter how much information is provided it cannot learn to solve the classification problem.
2. The input data is insufficient in quantity or format to solve the classification problem.

That said, we found that a single neuron with input from a single feature was sufficient to solve the previous classification problem. Let's try another one. Without changing anything yet, press the play button on the Tensorflow playground below and let it run for about 5 seconds, and then press pause...

Tensorflow

---

{{#widget:Tensorflow2}}

Finished?

How'd it do? This time around is one neuron with input from a single feature (the dim1 data: AGE) performing still performing well in the separation task? If so, an orange-colored background should have formed behind the orange dots, while a blue-colored background should have formed behind the blue dots.

It probably didn't do so well this time around, right?

Reset Tensorflow and try adding more neurons, and add an additional layer with several neurons, and rerun.

How'd it do? Still can't solve the classification problem right? Seems like we need to add an additional feature as input, let's try some of those. Here is the full set of possible network inputs...

Input Features

X1	AGE
X2	SCORE
X12	AGE2
X22	SCORE2
X1X2	AGE × SCORE
sin(X1)	sin(AGE)
sin(X2)	sin(SCORE)

Start by running X₁ and X₂ together, just to see what happens. Then feel free to test out whatever combinations you think might work.

Input Combinations

---

Take a close look at the input options in Figure-1 on the right. There are a bunch of X variables with subscripts and superscripts, and next to each is a box with various color gradients. For now, let's focus on just two of those symbols, and what they mean to us...

Input Features

X1	AGE
X2	SCORE

These are parsed such that subscripts (X₁ , X₂ ,... X_i ) represent each predictor variable, like AGE and SCORE. As you can see, the first two input options X₁ and X₂ are just X_AGE and X_SCORE. Note that since X₁ is plotted on the x-axis, it has a color gradient that changes horizontally, but is constant in the vertical dimension. Conversely the X₂ feature plotted on the y-axis has a vertical color gradient. To clarify why this happens...

If the only thing we know about these study participants is their AGE, X₁, we can only make a 1-D plot with each person's age along the x-axis, such that [ x = AGE_i , y = 0 ]. If you take a look at Figure-2, it should be clear that when information is collapsed onto its single dimension and plotted along the x-axis, the best line we can draw to separate the dim-1 data will be orthogonal to the x-axis (a vertical line). As you move horizontally along the x-axis your categorical guess will likely change, along with the confidence in that guess, which is precisely what is being represented by the color gradient. On the other hand, knowing nothing about exam score, moving up and down on the y-axis will have no effect on your decision, which is why color is constant in the y-dimension.

When the neural net only gets input about a single feature of each person in the dataset, its synaptic weights will only adapt output along that one dimension. Thus, if for example the network sees that a person is 3 years above the dataset average (considering the data has been mean deviated and centered), it won't matter what that person's cognitive SCORE was (since the neural net doesn't have access to that info), the network will always make the same guess for anyone 3 years above average age. This is why color is constant at x=3 for any y value.

Error creating thumbnail: File missing

Figure 2

This isn't a shortcoming of having just one single neuron in the entire network. You could add as many neurons and layers as you want (go ahead and try it)...... if the network only gets input about one feature dimension, the output will be the same, whether there is 1 neuron, or 1 billion. To realize this fact, pretend you can only see the dots as they are plotted in along the number line in 1D (in Figure 2); if we were unable to see the 2D cluster clouds above that line, the billions of neurons in our brain would tell us to draw the classification line in basically the same place as that one single neuron in our artificial neural net. This is a very interesting concept worth noting: neural net classifiers can fail for two very different reasons.

(1) The neural network itself might be ill-formulated in such a way that, no matter how much information you provide, it cannot seem to learn to solve the classification problem. (2) On the other hand, you might have implemented an apposite deep neural network; yet if the input data is insufficient to solve the classification problem, it will appear to you that this potentially very good neural network performs like garbage.

“ A perfectly capable neural net might end up performing like garbage because, with the info you were feeding in, it never stood a chance. ” — anonymous social worker

With that said, there are ways to help prevent that later scenario from happening. These involve doing things like you see for the rest of the input features.

Continue to Neural Nets Tutorial Page 2

Below I've embedded another Tensorflow neural net playground.

Outputs

---

More directly, it is the value spit-out by the activation function of the 'output layer'. Here, since we only have a single layer, our hidden 'hidden layer' and 'output layer' are one in the same. The output function of our neuron is known as the tanh function.

The tanh function is an extremely common choice for an output function in artificial neural network machine learning frameworks because it yields a nice sigmoid shape, and no matter the magnitude of its inputs, the output from the tanh function is bounded between { 0 : 1}. These are very desirable properties for neural net nodes. Here you see the tanh function evaluated across various x-dim inputs...

Error creating thumbnail: File missing

see tanh on wolfram alpha for many details about tanh function.

Tanh produces a sigmoid output over the range {-2 : 2}, and automatically evaluates to exact values when its argument is the natural logarithm. Speaking of the natural log, that is another very common choice of output function for the same reasons as tanh.

For now, let's not belabor the point that our neuron (and in going forward, all our neurons) are using the tanh function. Maybe just keep this in mind if you're wondering what sorts of numbers are travelling along the axons of these neurons, and ultimately those colored gradients underneath the dots.

This tutorial continues on the next page. Don't worry about playing around too much with the TensorFlow GUI, there will be plenty of that on the next page, and those that follow.