Inside Ai
Multi-Layer Perception Time Series
Apply State of the Art Deep Learning MLP models for predicting the sequence of numbers/time series data.
Hi, how are you doing? I hope it's great.
Today let’s understand something new and advanced that I have learned. Deep Learning’s state-of-the-art Multi-layer Perception Models (MLP in short) for Time Series Forecasting.
So what we will learn in this article.
1.) MLP for univariate time series forecasting.
2.) Multivariate MLP models further divided into
a.) Multiple Input series
b.) Multiple Parallel series
3.) Multi step MLP Models
4.) Multi step Multi-Step MLP Models further divided into
a.) Multiple Input Multi-Step Output
b.) Multiple Parallel Input and Multi-Step Output.
It’s a long list. Let’s get started shall we?
- ) Multi-layer Perception for Uni-Variate Time series forecasting.
To illustrate the MLP univariate time series forecasting we will take real-life case data. One of the datasets that got my attention is COVID-19 Dataset and thought let’s check MLP’s accuracy level, how good is it with real-life cases.
So i quickly grabbed the data from the link https://github.com/imdevskp/covid-19-india-data if you can find the dataset don’t worry, you can ask me.
Now if we view the ‘complete.csv’ dataset we will get columns of detailed day level data like new cases, confirmed cases, state, dates.
We will take only 2 columns ~ ‘confirmed cases’ and the ‘dates’.
We need to modify the 2 columns to provide us cumulative of confirmed cases and the count of days.
Let’s load our dataset. And yes we need to convert if we have multidimensional array data [ [12],[13],…….[19] ] to 1D/simple list format [12,13,…….19] by using flatten() function.
from pandas import read_csv
dataframe =read.csv('covid19processed_data.csv', usecols=[0], engine='python')
dataset = dataframe.values#converting multidimensional array to single dim
data = dataset.flatten()
raw_seq = data.tolist()
if you don’t want to use the above data you can simply create a sequence of list() as for this MLP model we only need a sequence of lists.
#optional
raw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]As we all know in machine learning we have the concept of dependent as well as the independent variable. So what we need to do is to divide the input series into multiple inputs. That is creating a series of 3 steps back that will be used as input / independent variables X and one step is used as output Y variable for the one-step prediction. Thus it will look like this.
Below the split_sequence() function will convert our data into a sequence of X and Y where each sample/sequence of X has the output in a single time step Y.
#split a univariate sequence into samples
def split_sequence(sequence, n_steps):
X, y = list(), list()
for i in range(len(sequence)):
#find the end of this pattern
end_ix = i + n_steps
#check if we are beyond the sequence
if end_ix > len(sequence)-1:
break#input and output sample data
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)#choose a number of time steps
n_steps = 3
#split into samples
X, y = split_sequence(raw_seq, n_steps)
Now we know how to prepare the dataset for MLP univariate time series forecasting. It’s time to build our first simple MLP model.
#define model
model = Sequential()
model.add(Dense(100, activation='relu', input_dim=n_steps))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')Here we initialize our model as Sequential() then 1 hidden layer with 100 output dimensions/layers then activation relu for non-linearity and the input_dimensions = n_steps will be 3 as we have 3 steps back. Technically the model will view each time step as a separate feature. Then we have Dense (1) as we need 1 output layer with optimizer Adam version of stochastic gradient descent, and a parameter ‘mse’ to calculate loss.
#fit model
model.fit(X, y, epochs=3000, verbose=0)It's time to fit our data into the model with epochs = 2000, and verbose = 1 verbose gives us the animation while displaying the processed output.
Our model is done, it's time to do some predictions.
The model is expecting an input shape of 2 dimensions [ samples, features ] So we will be modifying our input value to 2 dimensions[ samples, features ]
#Sequence of array, This are actually the last 3 values from our dataset so that we can predict the next step that will be our 156th day
x_input = array([13758429,14377942,15046478])
x_input = x_input.reshape((1, n_steps))
yhat = model.predict(x_input, verbose=0)
print(yhat)Output
[[15733087.]]
Alright we have our 156th-day cumulative sum of confirmed cases and since this is cumulative we will minus to get the actual value 15733087–15046478 = 686609
Now let redo again for 157th to verify with the website.
#prediction for 157th day
x_input = array([14377942,15046478,15733087])
x_input = x_input.reshape((1, n_steps))
yhat = model.predict(x_input, verbose=1)
print(yhat)output
[[16455969.]]
And again since it’s a cumulative value the Actual incremental value for the 157th day i.e. for 6th July 20 will be 16455969–15733087 = 722,882
Cool! it's not even the end of the day we are very close to the actual numbers.
Well, what we have learned so far is that the simple Multi-layer perception model is also very effective in identifying the pattern of the next sequence of numbers.
However, do observe whenever we fit/train our model.fit() we will see the ‘loss and accuracy’ level of the model trained at are very high(loss) and low(acc_score) if we are doing for huge super random numbers. And if we do it with a few simple random numbers like raw_seq=10,20,30….90 we will get a high accuracy_score in just 2000 epochs WHY? It's becoz there is less randomness in the sequence of 10,20,30,….90.
And when we have huge numbers with very high randomness the algorithm will take lots of epochs to learn and adjust the weights which are not always practical in nature. So what we can do is get a close approximation and estimation value of the next event. Try it out with raw_seq= 10,20,30…90 with epoch = 1500 or more, you will get very high accurate result = 100
let’s put all of the pieces together
Congratulations! We have successfully implemented our first Multi-Layer Perception MLP deep learning model with a real live case study.
I myself was very skeptical about the concept will work or not, but seems to me after doing various trials with random case studies found it quite effective.
Special Thanks to Jason from Machine Learning Mastery for enlightening us with this concept.
Next, we have Multivariate Multiple Input MLP Models Time Series.
Multivariate Multiple input MLP time series.
This type of analysis is very helpful when we have more than one observation for each time step.
Let’s understand this with the help of an example where we have 2 retail sales records for the month of Jan and we need to predict the next immediate sales value.
Or we can take a sample of a few simple random numbers to understand its performance.
in_seq1 = array([5, 10, 15, 20, 25, 30, 35, 40, 45])
in_seq2 = array([15,25, 35, 45, 55, 65, 75, 85,95])#with an
out_sequence = out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])
Now we have the sample data we need to convert/reshape our array data into the proper format as a single dataset where each row is a time and each column is a separate time series.
#convert to [rows, columns] structure
in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))#horizontally stack columns
dataset = hstack((in_seq1, in_seq2, out_seq))
print(dataset)
We need to structure these data into samples with input and output samples.
Before we split the data into samples maintain the order of observations across the 2 inputs. Now if we choose 3 input time steps. Then the first sample will look like this
Note: we will also find that in transforming the data into input/output samples we have discard some values from the output because we don’t have values in the input at prior time steps.
Don’t worry we have defined a function that will take care of all transformations of our input/output samples into X and Y
def split_sequences(sequences, n_steps):
X, y = list(), list()
for i in range(len(sequences)):
#find the end of this pattern
end_ix = i + n_steps
#check if we are beyond the dataset
if end_ix > len(sequences):
break
#collect input and output pattern data
seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1, -1]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)The last step before we fit out data is we must flatten the shape of the input samples.
#flatten input
n_input = X.shape[1] * X.shape[2]
X = X.reshape((X.shape[0], n_input))For example, if each input sample is of 3 rows 2 columns after we flatten it becomes.
Now we are all set to fit our data into the model.
#define model
model = Sequential()
model.add(Dense(100, activation='relu', input_dim=n_input))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse',metrics = ["accuracy"])#fit model
model.fit(X, y, epochs=2500, verbose=1)
Done… we have 100% accuracy of predicting the next value.
To predict the next value we also need the data to follow the same model data structure/format i.e. what have in X
#demonstrate prediction
x_input1 = array([[5, 15], [10, 25], [15, 35]])
x_input2 = array([[7, 17], [12, 27], [17, 37]])x_input 1= x_input1.reshape((1, n_input))
y1 = model.predict(x_input1, verbose=1)
print(y1)x_input 2= x_input2.reshape((1, n_input))
y2 = model.predict(x_input2, verbose=1)
print(y2)Output:
[[49.980164]]
[[53.5025]]
Done…..we have high accuracy next sequence value.
Let's put all the pieces together.
Now this can also be achieved by separately processing using each input series and the output of each sub models later can be combined before making the prediction of the final output. We can refer this as Multi headed input MLP model.
So we will define each input series in a separate model that expects the same input data format
#first input model
visible1 = Input(shape=(n_steps,))
dense1 = Dense(100, activation='relu')(visible1)#second input model
visible2 = Input(shape=(n_steps,))
dense2 = Dense(100, activation='relu')(visible2)
Now we will merge both the sub models
#merge input models
merge = concatenate([dense1, dense2])
output = Dense(1)(merge)One more step is we need to tie up the input and output together.
model = Model(inputs=[visible1, visible2], outputs=output)
Here’s the schematic beautifully explained for how this model work like.
Finally, we will fit the model.
model.fit([X1, X2], y, epochs=2500, verbose=1)Done, our model is complete with 100% accuracy.
Time to predict
#demonstrate prediction
x_input1 = array([[5, 15], [10, 25], [15, 35]])
x1 = x_input1[:, 0].reshape((1, n_steps))
x2 = x_input1[:, 1].reshape((1, n_steps))
y1 = model.predict([x1, x2], verbose=0)
print(y1)x_input2 = array([[7, 17], [12, 27], [17, 37]])
x1 = x_input2[:, 0].reshape((1, n_steps))
x2 = x_input2[:, 1].reshape((1, n_steps))
y2 = model.predict([x1, x2], verbose=0)
print(y2)Output
[[50.00611]]
[[53.589615]]
Alright, we will have the same or very close to our previous answer. This method sometimes offers more flexibility, and control over management and performance.
Let’s put all the pieces together
Next, we have something interesting where we will have multiple parallel observations/inputs and a value must be predicted for each.
Multi-variate Multiple Parallel Output
For this, we have another split function to give the data a desired multiple parallel outputs for each observation
#split a multivariate sequence into samples
def split_sequences(sequences, n_steps):
X, y = list(), list()
for i in range(len(sequences)):
#find the end of this pattern
end_ix = i + n_steps
#check if we are beyond the dataset
if end_ix > len(sequences)-1:
break
#gather input and output parts of the pattern
seq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix, :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)Let’s put all of the codes together and will observe the output.
Output [[15.00675 34.943497 49.50155 ]]
Yup! We have a high accuracy result. Okay if we arrange the x_input1 data.
We can verbally calculate the next sequence. Thus we can validate the accuracy of our model.
Often when we will analyze our time series data we will require predicting more than one-time step. So this is referred to as Multi-Step MLP models. There are also different versions of Multi-Step MLP models like
a.) Univariate/Single input Multi Step Output
b.) Multi-variate/Multiple Input Multi Step Output
c.) Multiple Parallel Input Multi Step Series
Let’s start with Univariate Multi-Step Output
[5, 10, 15, 20, 25, 30, 35, 40, 45] using this example for a sequence of data, we will transform the values in X Y format like before where
X will have 3-time steps as input and Y will now have 2-time steps as output. Thus we could use the last three time steps as input and forecast the next 2 time steps.
Example: Input: [35,40,45] Output: [ 50,55]
The split-sequence function will replicate the behavior of splitting the data into X and Y for any input series of data.
# split a univariate sequence into samples
def split_sequence(sequence, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out
# check if we are beyond the sequence
if out_end_ix > len(sequence):
break
#collect input and output pattern data
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix:out_end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)After that it’s all similar to One-Step, the only difference out here is we need to define the n_steps_out while defining the model.
#define model
model = Sequential()
model.add(Dense(100, activation='relu', input_dim=n_steps_in))
model.add(Dense(n_steps_out))That’s it we are ready to test out the model.
#univariate multi-step MLP
from numpy import array
from keras.models import Sequential
from keras.layers import Dense
#split a univariate sequence into samples
def split_sequence(sequence, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequence)):
#find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out
#check if we are beyond the sequence
if out_end_ix > len(sequence):
break
#collect input and output pattern data
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix:out_end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)
#define input sequence
raw_seq = [5, 10, 15, 20, 25, 30, 35, 40, 45]#choose a number of time steps
n_steps_in, n_steps_out = 3, 2
#split into samples
X, y = split_sequence(raw_seq, n_steps_in, n_steps_out)
#define model
model = Sequential()
model.add(Dense(100, activation='relu', input_dim=n_steps_in))
model.add(Dense(n_steps_out))
model.compile(optimizer='adam', loss='mse', metrics = ["accuracy"])
#fit model
model.fit(X, y, epochs=4500, verbose=1)
'''if we wish to save
model.save_weights('F:/DeepLearning/mlp_univariate_MultiStep.h5')
model.load_weights('F:/DeepLearning/mlp_univariate_MultiStep.h5')
'''#prediction
x_input = array([50, 55, 60])
x_input = x_input.reshape((1, n_steps_in))
y = model.predict(x_input, verbose=1)
print(y)
Here we are, we have successfully forecasted the next 2-time step. This time the accuracy is 2% low I believe its epoch value. Well, I believe you already know how to get the optimal epoch value to train the model.
Try with epochs=1830 you will get the better result [[66.24881 71.47211]]
Now, what if we have more than 2 observations i.e. Multiple inputs and we need to predict the next 2-time step just like we did above Multivariate Multiple input series where the output series are dependent upon the input series.
We will call it as Multivariate/Multiple input Multi-Step MLP Series
Here too the sequence function that will take care of all from formatting the multiple input series to a proper shape and size.
#split a multivariate sequence into samples
def split_sequences(sequences, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequences)):
# find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out-1
# check if we are beyond the dataset
if out_end_ix > len(sequences):
break
#collect input and output pattern data
seq_x, seq_y = sequences[i:end_ix, :-1], sequences[end_ix-1:out_end_ix, -1]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)#define input sequence
in_seq1 = array([5, 10, 15, 20, 25, 30, 35, 40, 45])
in_seq2 = array([15,25, 35, 45, 55, 65, 75, 85,95])
out_seq = array([in_seq1[i]+in_seq2[i] for i in range(len(in_seq1))])
#convert to [rows, columns] structure
in_seq1 = in_seq1.reshape((len(in_seq1), 1))
in_seq2 = in_seq2.reshape((len(in_seq2), 1))
out_seq = out_seq.reshape((len(out_seq), 1))
#horizontally stack columns
dataset = hstack((in_seq1, in_seq2, out_seq))
#choose a number of time steps
n_steps_in, n_steps_out = 3, 2
#convert into input/output
X, y = split_sequences(dataset, n_steps_in, n_steps_out)
Then we have to flatten the input X here’s what the structure of data looks like after we apply to flatten.
#flatten input
n_input = X.shape[1] * X.shape[2]
X = X.reshape((X.shape[0], n_input))
Let’s put all of the codes together.
Running the model predicts the next 2 time steps of the output beyond the dataset.
Alright, we are very close to the actual results by 0.1. it should be 50 and 65 but we got 49.9 and 64.9
Last but not least we have Multiple Parallel Input and Multi-Step Output.
Sometimes we may require predicting multiple time steps of each time series observation, then the last 3 time steps from each of the time series observations will be an input to the model and predict the next time steps of each observation as output.
Here’s the visual representation of X and Y / input and output.
We have a sequence function to do the data transformation.
#split a multivariate sequence into samples
def split_sequences(sequences, n_steps_in, n_steps_out):
X, y = list(), list()
for i in range(len(sequences)):
#find the end of this pattern
end_ix = i + n_steps_in
out_end_ix = end_ix + n_steps_out
#check if we are beyond the dataset
if out_end_ix > len(sequences):
break
#collect input and output pattern data
seq_x, seq_y = sequences[i:end_ix, :], sequences[end_ix:out_end_ix, :]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)The only major difference we have here is to flatten both X and Y
#flatten input
n_input = X.shape[1] * X.shape[2]
X = X.reshape((X.shape[0], n_input))
#flatten output
n_output = y.shape[1] * y.shape[2]
y = y.reshape((y.shape[0], n_output))That’s it! If we predict an array([[35, 75,110], [40, 85,125], [45, 95,140]]) it will give us output of [[ 49.85181 106.10115 156.77332 56.124226 117.06872 172.24738 ]] if we manually validate we know it’s pretty close to the actual result.
The full code
Finally we are done…. It’s been a long way. I hope I’m to explain the MLP concepts. Feel Free to discuss any new concepts or innovative ways to use MLP for our day to day solutions.
Next, I’m going to write about LSTM and replicate the same example above to validate its better performance on Time Series Data. Stay Tune!
I hope you enjoyed it.
Some of my alternative internet presences are Facebook, Instagram, Udemy, Blogger, Issuu, and more.
Also available on Quora @ https://www.quora.com/profile/Rupak-Bob-Roy
have a good day…
~ Be Happy and Enjoy!
