Transcript Artificial Neural Networks (ANN)

Artificial Neural Networks
(ANN)
Artificial Neural Networks (ANN)
X1
X2
X3
Y
Input
1
1
1
1
0
0
0
0
0
0
1
1
0
1
1
0
0
1
0
1
1
0
1
0
0
1
1
1
0
0
1
0
X1
Black box
Output
X2
X3
Output Y is 1 if at least two of the three inputs are equal to 1.
Y
Artificial Neural Networks (ANN)
X1
X2
X3
Y
1
1
1
1
0
0
0
0
0
0
1
1
0
1
1
0
0
1
0
1
1
0
1
0
0
1
1
1
0
0
1
0
Input
nodes
Black box
X1
X2
X3
Output
node
0.3
0.3
0.3

t=0.4
Y  I (0.3 X 1  0.3 X 2  0.3 X 3  0.4  0)
1
where I ( z )  
0
if z is true
otherwise
Y
Artificial Neural Networks (ANN)
• Model is an assembly of
inter-connected nodes
and weighted links
• Output node sums up
each of its input value
according to the weights
of its links
• Compare output node
against some threshold t
Input
nodes
Black box
X1
Output
node
w1
w2
X2

Y
w3
X3
t
Perceptron Model
Y  I (  wi X i  t )
i
or
Y  sign (  wi X i  t )
i
General Structure of ANN
x1
x2
x3
Input
Layer
x4
x5
Input
I1
I2
Hidden
Layer
I3
Neuron i
Output
wi1
wi2
wi3
Si
Activation
function
g(Si )
Oi
threshold, t
Output
Layer
Training ANN means learning
the weights of the neurons
y
Oi
Algorithm for learning ANN
• Initialize the weights (w0, w1, …, wk)
• Adjust the weights in such a way that the
output of ANN is consistent with class
labels of training examples
2
– Objective function: E   Yi  f ( wi , X i )
i
– Find the weights wi’s that minimize the above
objective function
• e.g., backpropagation algorithm
Classification by
Backpropagation
• Backpropagation: A neural network learning algorithm
• Started by psychologists and neurobiologists to develop
and test computational analogues of neurons
• A neural network: A set of connected input/output units
where each connection has a weight associated with it
• During the learning phase, the network learns by
adjusting the weights so as to be able to predict the
correct class label of the input tuples
• Also referred to as connectionist learning due to the
connections between units
Neural Network as a Classifier
• Weakness
– Long training time
– Require a number of parameters typically best determined
empirically, e.g., the network topology or ``structure."
– Poor interpretability: Difficult to interpret the symbolic meaning
behind the learned weights and of ``hidden units" in the network
• Strength
–
–
–
–
–
–
High tolerance to noisy data
Ability to classify untrained patterns
Well-suited for continuous-valued inputs and outputs
Successful on a wide array of real-world data
Algorithms are inherently parallel
Techniques have recently been developed for the extraction of
rules from trained neural networks
Backpropagation
• Iteratively process a set of training tuples & compare the
network's prediction with the actual known target value
• For each training tuple, the weights are modified to
minimize the mean squared error between the network's
prediction and the actual target value
• Modifications are made in the “backwards” direction: from
the output layer, through each hidden layer down to the
first hidden layer, hence “backpropagation”
• Steps
–
–
–
–
Initialize weights (to small random #s) and biases in the network
Propagate the inputs forward (by applying activation function)
Backpropagate the error (by updating weights and biases)
Terminating condition (when error is very small, etc.)
Software
•
•
•
•
•
Excel + VBA
SPSS Clementine
SQL Server
Programming
Other Statistic Package …