Neural Network Module

The Neural Network module contains the following three core classes:

• Var

• DenseLayer

• NeuralNetworkClassifier

These classes combined allow for training a neural network on any dataset.

class scripts.models.neural_net.DenseLayer(n_in, n_out, activation, name='DenseLayer')

Bases: object

DenseLayer data structure that works as a building block for the Neural Network class.

This class serves as a core building block to implement a Neural Network.

Its main functionality is to take an input X and produce an output Y as a linear combination of the weights and the input (passed through a non linear activation function)

Parameters

• n_in (int) – The number of inputs that the layer expects (e.g. number of features for the first layer)

• n_out (int) – The number of output neurons

• activation (str) – The non-linear activation function that is used when forwarding an input through the layer. Possible values are ‘relu’, ‘tanh’ or ‘softmax’

• name (str, optional) – Name of the layer

dim()

Returns the dimensions of the weights matrix and the bias vector

Returns

Return type

tuple

forward(X)

Computes a forward pass through the layer

Parameters

X (2d array of Var instances) – n x n_in where n is the number of provided samples

Notes

Computes the forward pass of the dense layer: For each output neuron, j, it computes: activation(weights[i][j]*inputs[i] + bias[j])

Returns

Return type

n x n_out array of Vars where n is the number of provided samples.

neurons()

Return the number of neurons

Returns

Return type

int

num_params()

Returns the number of parameters

Returns

Return type

int

parameters()

Returns all the vars of the layer (weights + biases) as a single flat list

Returns

n x 1 where n is a sum of the number of weights and the number of biases

Return type

1d array

class scripts.models.neural_net.NeuralNetworkClassifier(layers=[], loss='cross_entropy', name='NeuralNetworkClassifier')

Bases: scripts.base._base_classifier.BaseClassifier

This class serves as a high level client facing API. It allows to specify all hyper parameters to initialise a feed forward neural network

Parameters

• layers (list) – a list of DenseLayer

• loss (str, optional) – loss function to be minimised (default is ‘cross_entropy’)

• name (str, optional) – Name for the classifier

X

Data points to used to train the neural network

Type

2d array

y

Target classes

Type

1d array

y_hot

One hot encoded target classes

Type

2d array

n

number of data points (X.shape[0])

Type

int

p

number of features (X.shape[1])

Type

int

fitted

Boolean to see whether the classifier has been fit

Type

bool

parameters

array of all the parameters

Type

1d array

_parameters()

Returns all the parameters of the layers as a 1d np array

Returns

Return type

1d array of length n, where n is the number of parameters

_total_parameters()

Number of parameters

Returns

Return type

int, the number of parameters the Neural Network has

add(layer)

Add a dense layer behind the current last dense layer

layer : DenseLayer

fit(X, y, num_batches=1, epochs=1000, lr=0.01, verbose=0)

The training loop of the neural network

Parameters

• X (2d array) – n x p matrix of data points, where n is the number of data points and p the number of features

• y (1d array) – n x 1 vector of target classes, where n is the number of data points

num_batchesint, optional

Number of batches used to train the network in every epoch. The parameters are being updated after every forward pass of a batch. Default value is 1 which corresponds to feeding the entire data set X in every epoch

epochsint, optional

Number of training loops (default is 1000)

lrfloat, optional

learning rate for the gradient descent step

verboseint, optional

Prints out during the training (0, 1 or 2)

k

number of unique classes

Type

int

loss_history

Type

list

accuracy_history

Type

list

forward(X)

Compute the forward pass

Parameters

X (2d array) – n x n_in sample to forward through the network, n_in must correspond to the first DenseLayer

Notes

Computes the forward pass of the MLP: x = layer(x) for each layer in layers

Returns

• n x n_out array where n is the number of data points and n_out

• corresponds to the number of neurons in the last DenseLayer

predict(X)

Predict class labels

Parameters

X (2d array) –

Returns

• 1d array of predicted labels of size n, where n is the number of

• data points

predict_proba(X)

Predict probabilities for each class

Parameters

X (2d array) –

Notes

For all the data points, find the probabilities of belonging to all the k classes

Returns

• 2d array of probabilities of size n x k, where n is the number of

• data points and k the number of classes

summary()

Summary of the neural network

Returns

• str, summary of the network (name,weights,biases,layers,number of

• parameters)

class scripts.models.neural_net.Var(val, parents=None)

Bases: object

A variable which holds a number and enables gradient computations.

Adapted from Rasmus Berg Palm repository.

Parameters

• val (float or int) – The actual value of the number.

• parents (list, optional) – List where each item has the following structure (Var, gradient)

grad

Partial derivative of this variable with respect to the variable from which the bakcward method was called.

Type

float or int

Raises

AssertionError – If the val is not float or int.

backward()

Compute gradient of this Var with respect to all its parents.

exp()

Peform exp function.

log()

Peform log function. (base 10)

relu()

Peform Relu activation function.

Relu is defined as follows:

\[relu(x) = max(0, x)\]

tanh()

Peform Tanh activation function.