Issue 59

D. Bui-Ngoc et alii, Frattura ed Integrità Strutturale, 59 (2022) 461-470; DOI: 10.3221/IGF-ESIS.59.30

For each layer, the forward propagation is calculated by the following Eqn. (1) as below:

  1 l N





  l 1 conv w , s ik i l 1 1

i

i





x

b

(1)

k

k



i

 1 l ik w is the kernel from

th i neural at layer  1 l to th k neural at layer l ,  1 l

l k x is the input,

th i

i s is the output of

where

th k neural at layer l . The intermediate output l

l k b is the bias of the

neural at layer  1 l ,

k y can then be calculated based

   l l k k y f x .

on the activation function   . f as:

Back propagation algorithm is used to train the network based on identifying the gradient of the loss function   E y from the weights of the CNN. The derivative of the error with respect to each weight is calculated by Eqn. (2) as below:



E

, l i k

 

w

(2)

, l i k



w

The weight is then calculated based on the computation of the gradients of layers as below:

   * , , , w w w l l l i k i k i k

(3)

* , w l i k is the weight of the next iteration. Details of the calculation can be seen in [23].

where  is the learning rate,

Recurrent Neural Network Recurrent neural network (RNN) is a class of ANN in which the outputs from neurons are used as feedback to the neurons of the previous layer.RNN has been proved to have various advantages in data processing: It has the ability to process input data of any length; Model size does not increase when the number of input increases; Calculation process can make use of the old information; Weights are shared throughout the processing.Fig. 1 below presents a common RNN structure:

Figure 1: Recurrent neural network structure

In RNN, the hidden state at time t- t h can be calculated in the Eqn. (4) as below:

 h f

  1 (Ux V ) t t h

(4)

t

where t h is a hidden state at time t, t x is an input at time t. f is a linear function liked tang hyperbolic (tanh) or ReLU. For the first hidden state, the initial  1 t h is assigned to zero. t o is output at time t and can be used as:

 softmax(

o

Vh

)

(5)

t

t

463