In 2009, Escalante etal. (2009) proposed a machine learning pipeline that included selecting a preprocessing algorithm, a feature selection algorithm, a classifier and, all their hyper-parameters. Their approach used a modified Particle Swarm Optimization (PSO) to deal with the limited configuration space and was called PSMS system. Although the authors found that they could apply their method to different datasets without domain knowledge, most of the datasets used had unrelated attributes. In order to avoid overfitting, the authors proposed using k-cross-validation, and then the approach was extended with a custom assembling strategy that combined the best solutions from multiple generations (Escalante, Montes, and Sucar 2010).

Later, Sun et al. extended the idea of PSMS and proposed the unification of the PSO algorithm and the Genetic Algorithm (GA) (2013). This approach was called GPS (which stands for GA-PSO-FMS). A GA were was used to optimize the pipeline structure, while the PSO for the hyperparameter optimization of each pipeline. The pipeline proposed by the authors included selecting from a pool of methods such as data sampling, data cleansing, feature transformation, feature selection, and classification. The datasets used for evaluating GPS were characterized by a high number of instances, thus causing an increase in the computational cost during the loss function evaluation. Therefore, the authors proposed the use of an internal binary tree structure to speed up the GPS system.

Another interesting line of research is the application of multi-objective evolutionary algorithms. One of these approaches is the Multi-objective Support Vector Machine Model Selection (MOSVMMS) (Rosales-Pérez, Escalante, Gonzalez, Reyes-Garcia, and Coello-Coello 2013), where the search is guided by a Non-dominated Sorted Genetic Algorithm-II (NSGA-II).

The authors built a pipeline formed by feature selection, pre-processing, and classification tasks focused only in the SVM classifier. The models were evaluated under bias and variance trade-off as prime objective functions. This approach was only tested on thirteen binary classification problems. Two extensions of this approach were reported, the first called Multi-Objective Model Type Selection (MOMTS), where a multi-objective evolutionary algorithm based on decomposition (MOEA/D) was used instead of the NSGA-II (Rosales-Pérez, Gonzalez, Coello-Coello, Escalante, and Reyes-Garcia 2014). MOMTS focused only on selecting classification models without other involved tasks. However, the authors explored the idea of measuring complexity models through the Vapnik-Chervonenkis dimension, which could have a high computational cost as the dimension of the datasets grows.

For that reason, the second extension proposed by Rosales-Pérez, Gonzalez, Coello, Escalante, and Reyes-Garcia was the Surrogate Assisted Multi-Objective Model Selection (SAMOMS) (2015), in which a pipeline structure is considered, (preprocessing, feature selection, and classification). They proposed a surrogate assistant to speed up the fitness evaluation.

The Tree-Based Pipeline Optimization Tool (TPOT) is an open-source software package for configuring pipelines in a more flexible manner (Olson et al. 2016). TPOT uses a genetic programming algorithm for optimizing structures and hyperparameters. The main operator included in TPOT has supervised classification, feature preprocessing operators, and feature selection operators, all ofthem taken from scikit-learn. The main drawback of TPOT is considering unconstrained search, where resources can be spent on generating and evaluating invalid solutions.

So far, these approaches do not present evidence of the treatment of time-series databases. Most of them use a fixed pipeline length in sequential steps. TPOT is, to date, the approach that stands out for optimizing the design of pipes. Early efforts for an approach that suggests automated pipelines for time series can be found a previous work proposed by Pérez-Castro, Acosta-Mesa, Mezura-Montes, and Cruz-Ramírez (2015). The authors proposed using a micro version of differential evolution to solve the FMS problem, and they suggested optimized pipelines. In that work, smoothing, time series representation, and classification through the k-nearest neighbor algorithm are only considered. This work is an extension of the work mentioned above.

In this article, a part of the well-known collection of univariate time series databases is used (Keogh et al. 2011). The essential characteristics of those databases are summarized in Table 1.

Table 1

Source: Authors

A brief description of each database is presented below:

Beef: This database consists of five classes of beef spectrograms acquired from raw samples, cooked using two different cooking regimes. Each beef class represents a differing degree of contamination with offal (Al-Jowder, Kemsley, and Wilson 2002).

It can be seen that most databases describe real phenomena. A behavior analysis of time-series databases was carried out. This analysis consisted of observing three characteristics: a) class separation (CS), b) noise level (NL), and c) the similarity between the training and testing sets (SBS). The mean, median, and average of these per class for each database were computed and plotted from raw databases. From the visualization, the three characteristics of the above-listed were ranked. CS 1 means non-separable, and a value of 3 means easily separable. NL can take values between 1-5, where 1 means low noise and five high noise. SBM was measured in a range of 1 to 3, where 1 represents low similarity, and 3 means high similarity. The results of this analysis are summarized in Table 2.

Table 2

Note: Acronyms: CS (Class Separation); NL (Noise Level); SBS (similarity between the training and testing sets)

It is important to note that the data was not pre-processed for the experimental stage.

In this work, four main tasks are considered to build a learning pipeline for time series which involves solving the FMS problem.

Smoothing: It is usually used to soften out the irregular roughness to see a clearer signal. This task does not provide a model, but it can be a promising first step in describing various series components (Giron-Sierra 2018). It is common to use the term filter to describe the smoothing procedure. Moving Average (Baijal, Singh, Rani, and Agarwal 2016), the Savitzky-Golay filter (Savitzky and Golay 1964), and Local Regression (with and without weights) are considered with related hyper-parameters (Cleveland and Loader 1996).

Table 3 shows a summary of the available methods for each pipeline tasks and their related hyperparameters.

Table 3

Note: Acronyms: S (Smoothing); R (Representation); NR (Numerosity Reduction); C (Classification); TLS (Time-series length); SG (Savitzky-Golay Filter); MA (Moving Average); LRw (Local Regression-lowess); LRe (Local Regression-loess); SAX(Aggregate approximation); PAA (Piecewise Aggregate Approximation); PCA (Principal Component Analysis); INSIGHT (Instance Selection based on Graph-coverage and Hubness for Time-series); MC1 (Monte Carlo 1); KNN-ED (K-Nearest Neighbor-Euclidean Distance); KNN-LBDTW (K-Nearest Neighbor-Lower Bounding Dynamic Time Warping); NB (Naive Bayes); DT (Decision Tree); AB (AdaBoost); SVM (Support Vector Machine)

A candidate solution for the learning pipeline for time-series databases is represented as a vector in this work. Each vector can be formed by continuous values, binary values, or mixed values (continuous and discrete values).

Continuous encoding: Each potential solution is encoded as a continuous vector which is formed as in Equation (2).

Where depicts each position within a particular vector; and X _j,s Є [1;4], X _j,r Є [1;3], X _j,e Є [1;2], and x _j , _c Є [1;6] represent the ID of the selected smoothing, time-series representation, numerosity reduction, and classification available methods, respectively.

h _j,1 ...,ns, h _j,1,…, nr, h _j,1,…,ne , and h _j , _1,…,nc encode the set of hyperparameters related to the overall available methods, where ns, nr, ne, and nc represent the number of hyperparameters per type of task into the learning pipeline that has different limits. Each position can take random continuous values according to Equation (3), which determines a value between the lower and upper bounds of each hyperparameter, described in Table 3.

In Equation (3), lb _i is a lower bound, ubi an upper bound, and rand represents a random value between 0 and 1. Figure 3 shows an example of the structure of a continuous vector solution. Black boxes represent the positions that encode the selected method according to the task in the learning pipeline (smoothing, time-series representation, numerosity reduction, and classification), and the white boxes encode their related hyperparameters. For continuous encoding, vectors of 24 dimensions are considered to represent a learning pipeline, which is equivalent to a candidate solution of FMS problem for time-series databases.

Figure 2 Source: Authors

Mixed encoding: Mixed encoding consists of a vector of 24 dimensions, as shown in Figure 2. However, for this option, both continuous and discrete values are permitted.

Continuous values are generated by Equation (3), according to the limits of each position. In contrast, discrete values are generated by randi(), the function of MATLAB language that returns integer values drawn from a discrete uniform distribution, where limits are also respected.

Binary encoding: Binary encoding consists of a vector formed by binary values (0 or 1). These values can be grouped into binary strings that represent continuous or discrete values.

The length of a particular binary string depends on the boundary of values to be expressed. Binary string length lj is computed with Equation (4), where int expresses a integer value, log ₂ is the log base 2, ub the upper boundary, lb the lower boundary, and precision is a constant that means the number of decimal places to encode.

Then, the overall binary vector length bvl to encode a potential pipeline solution is the concatenation of each binary string. Equation (5) states how it is computed, where D is the amount of continuous or discrete values that can be encoded as binary strings, and l _j is the maximum length of these binary strings.

If a mixed vector structure is considered containing 24 values that represent a potential pipeline solution, which respects the boundaries of the values presented in Table 3, then vector with a length of 169 positions is required. It can be seen in Figure 3 that the first three binary values correspond to a binary string representing integer values between 1 and 4 that are the number of available smoothing methods. The next binary strings encode the rest of the values.

Figure 3 Source: Authors

A decoding process is needed to compute the quality of each binary encode solution. Decoding is performed for each binary string x_s of complete binary vector according to Equation (6), where lbj is the lower boundary used for this binary string, ub _j is the upper boundary used for this binary string, x_int is the result of traditional binary to decimal conversion, and l _j is the binary string length performed obtained from (4)

The Cross Validation Error Rate (CVER) is used as the fitness function f _x to evaluate the quality of a learning pipeline under a particular time-series database. Equation (7) describes f_x, where a represents the portion of instances in the time-series database that was incorrectly classified, and b is the total number of instances in such database. k depicts the number of stratified subsamples (folds) chosen randomly but with roughly equal size in the cross validation method that is adopted to avoid over-fitting.

Population-based metaheuristics such as evolutionary algorithms have a reduced population version that has proven to be efficient for solving large scale optimization problems (Olguín-Carbajal et al. 2019, Salehinejad, Rahnamayan, and Tizhoosh 2017). The reduced population versions usually are denoted with the prefix u Besides the small population, μ algorithms are characterized by a restart mechanism to avoid stagnation.

The μ-DE cycle and conventional operations, based on the scaled difference between two vectors of a population set, remain the same as in the classical DE. Usually, the population size in μ -DE can take a value between four and six vectors (Viveros Jiménez, Mezura Montes, and Gelbukh 2012, Caraffini, Neri, and Poikolainen 2013). Regarding the restart mechanism, μ -DE requires randomly replacing the N worst vectors each R generations. In this paper, the u-DE proposed by Parsopoulos (2009) is used as a population-based metaheuristic.

Algorithm 1 summarizes the main steps of the adopted u-DE. Step six shows the mutation and combination process, for that, different variants such as rand/1/bin, rand/1/exp, best/1/bin, and best/1/exp are used in the experimentation.

This results in four versions to solve FMS problem, called P-DEMS1 (Population-Differential Evolution Model Selection 1, using rand/1/bin), P-DEMS2 (Population-Differential Evolution Model Selection 2, using rand/1/exp), P-DEMS3 (Population-Differential Evolution Model Selection 3, using best/1/bin), and P-DEMS4 (Population-Differential Evolution Model Selection 4, using best/1/exp).

LS, a single-point optimization metaheuristic, is considered to be the oldest and most straightforward method (Talbi 2009). However, it has recently been used to train complex structures of neural networks and examine their hyperparam-eters for successful image classification (Aly et al. 2019). The LS algorithm used in this paper is briefly described in Algorithm 2.

For each iteration of the LS, a single solution s is replaced by a neighbor as long as the objective function is improved. Otherwise, the original solution is preserved. The search stops when all candidate neighbors are worse than the current solution, meaning that a local optimum is reached.

Step five (Algorithm 2) corresponds to the operator that generates the N neighbors of a slightly varied solution, according to the type of solution encoding. The neighbors are generated based on nvar Є [1; D] modifications that equivalent to the selected random positions. For example, Figure 4a shows a binary vector where nvar = 3. Thus, 3 positions are switched (0 instead 1 or vice-versa).

Figure 4 Source: Authors

On the other hand, when mixed encoding is used, the nvar selected values are replaced with new values that are within boundaries of their corresponding variables (Figure 4b).

Two versions of LS are adopted as search engines: S-LSMS1 (Single-Local Search Model Selection, where binary encoding is used) and S-LSMS2 (Single-Local Search Model Selection, where mixed encoding is used).

Table 5 shows the final numerical results of CVER obtained by the six metaheuristics versions. The reported values correspond to the average of five trials evaluated in the testing set of each database.

Table 5

Note: Acronyms: P-DEMS1 (Population-Differential Evolution Model Selection 1 with rand/1/bin); P-DEMS2 (Population-Differential Evolution Model Selection 2 with rand/1/exp); P-DEMS3 (Population-Differential Evolution Model Selection 3 with best/1/bin), and P-DEMS4 (Population-Differential Evolution Model Selection 4 with best/1/exp); S-LSMS1 (Single-Local Search Model Selection with binary encoding); S-LSMS2 (Single-Local Search Model Selection with mixed encoding). Values to the right of ± represent the standard deviation, the values in parentheses represent the ranks computed by the Friedman test, and values in parentheses to the left mean the lowest values found or the best ranking.

Due to the fact that the samples have a non-normal distribution, and multiple comparisons are needed, the non-parametric Friedman test was used (García, Fernández, Luengo, and Herrera 2010). The related samples are the performances of the metaheuristics measured across the same data sets. The Friedman tests evaluates the following null hypothesis: all methods obtain similar results with non-significant differences.

In the Friedman test, numerical results are converted to ranks. Thus, it ranks the metaheuristics for each problem separately. The best performing metaheuristic should have rank 1, the second best, rank 2, etc., as shown in Table 5. When there are ties, average ranks are computed. With six compared metaheuristics and ten databases, the p-value computed by the Friedman test was 0,183, which means that the null hypothesis is accepted. Thus, there are no significant differences found among the compared metaheuristics.

However, according to the average rank shown in Table 5, the S-LSMS1 (SM with binary representation) was the highest rank in most of the databases. It was followed by the P-DEMS1 (PM based on μ-DE, where the base vector is randomly chosen, and a binomial crossover is used).

To enhance statistical validation, the Tukey post-hoc test based on the Friedman results was applied by using the best and median values obtained over the five runs for each metaheuristic over the whole databases. Figures 6 and 7 show the results of this test, where the x-axis exhibits the confidence interval of mean ranks (given by the Friedman test) and the y-axis shows the name of each metaheuristic compared. Using the best and median values, the test yielded a p-value = 0,005 and a p-value = 0,250, respectively. In the case of Figure 6, there was a significant difference between S-LSMS1 and P-DEMS4.

Figure 6 Source: Authors

Figure 7 Source: Authors

Meanwhile, in Figure 7, there are no significant differences between the metaheuristics. Finally, a pairwise comparison was conducted to determine which of the metaheuristics exhibit a different performance against a selected control metaheuristic, namely S-MSLS1 because it was the best ranked.

The non-parametric 95% confidence Wilcoxon rank sum test was applied to the numerical results of the six metaheuristics for each database. Table 4 shows the numerical results of the pairwise comparison. The metaheuristics are sorted according to the average rank provided by the Friedman test.

The results in Table 6 show that the S-LSMS1 technique was able to provide the most competitive results among the compared metaheuristics. S-LSMS1 outperformed P-DEMS1 in two (out often) databases Lightning-7 and Trace, while P-DEMS1 outperformed S-LSMS1 in Coffee and GunPoint. S-LSMS1 outperformed P-DEMS2 in four databases (Beef, ECG200, Lightning-7, and Trace), while P-DEMS2 outperformed S-LSMS1 in just the Coffee database.

Table 6

Note: Acronyms: P-DEMS1 (Population-Differential Evolution Model Selection 1 with rand/1/bin); P-DEMS2 (Population-Differential Evolution Model Selection 2 with rand/1/exp); P-DEMS3 (Population-Differential Evolution Model Selection 3 with best/1/bin), and P-DEMS4 (Population-Differential Evolution Model Selection 4 with best/1/exp); S-LSMS1 (Single-Local Search Model Selection with binary encoding); S-LSMS2 (Single-Local Search Model Selection with mixed encoding). (-) means that there was a significant difference favoring the control metaheuristic. (+) implies that there was a significant difference favoring the compared metaheuristic. (=) means that no significant difference was observed between the compared metaheuristics. Values in parentheses to the left mean the best values found.

S-LSMS1 outperformed S-LSMS2 in Beef and Lightning-7, and was beaten in the Coffee database. S-LSMS1 outperformed P-DEMS4 in four databases (Beef, ECG200, Lightning-7, and Trace) and was outperformed in just one (Coffee). In summary, S-LSMS1 was able to obtain the best numerical values at least in eight often databases (Beef, CBF, ECG200, Face-Four, Lightning-2, Lightning-7, OliveOil, and Trace). Finally, P-DEMS3 was outperformed by S-LSMS1 in three databases (ECG200, Lightning-7, and Trace) and outperformed in just one (Coffee).

Table 7 shows the best pipelines suggested by each compared approach for each database. The third column details the pipeline models. Despite the fact that differences were observed in the solution models, there are interesting similarities.

Table 7

Note: Acronyms: S (Smoothing); R (Representation); NR (Numerosity Reduction); C (Classification); TLS (Time-series length); SG (Savitzky-Golay Filter); MA (Moving Average); LRw (Local Regression-lowess); LRe (Local Regression-loess); SAX(Aggregate approximation); PAA (Piecewise Aggregate Approximation); PCA (Principal Component Analysis); INSIGHT (Instance Selection based on Graph-coverage and Hubness for Time-series); MC1 (Monte Carlo 1); KNN-ED (K-Nearest Neighbor-Euclidean Distance); KNN-LBDTW (K-Nearest Neighbor-Lower Bounding Dynamic Time Warping); NB (Naive Bayes); DT (Decision Tree); AB (AdaBoost); SVM (Support Vector Machine); P-DEMS1 (Population-Differential Evolution Model Selection 1 with rand/1/bin); P-DEMS2 (Population-Differential Evolution Model Selection 2 with rand/1/exp); P-DEMS3 (Population-Differential Evolution Model Selection 3 with best/1/bin), and P-DEMS4 (Population-Differential Evolution Model Selection 4 with best/1/exp); S-LSMS1 (Single-Local Search Model Selection with binary encoding); S-LSMS2 (Single-Local Search Model Selection with mixed encoding). Symbols in parentheses mean the spent runtime during training and the subscripts represent the performance of the pipeline in terms of classification error. (|) means high runtime > 933 minutes, (•) means medium runtime > 272 and < 873 minutes, (J.) means low runtime < 272 minutes. Subscripts next to symbols: g means good, r means regular, and b means bad performance.

Regarding the smoothing task, Moving Average was the most preferred. PAA was the most commonly used and suggested method for time series representation, while INSIGHT was the most popular numerosity reduction technique.

As for the classification task, the decision tree and the Ad-aBoost (with decision trees as the weak learners) appeared as the most suitable. From the resulting final models, it can be seen that there were some evaluated databases with different models with similar performance values.

Databases such as Beef, Gun-Point, and Lightning-7 were detected as possible multimodal problems. They reported more diversification in the selected methods and their related hyperparameters.

Runtime varies considerably due to the different features of the temporal databases and the selected methods for carrying out a specific sub-task. Overall, P-DEMS3 reported the lowest runtime computational cost, while S-LSMS1 was the most expensive approach.

However, S-LSMS1 reported the best performance during training and competitive results in the testing phase. Figure 15 shows a graphical example of the suggested pipeline that was applied to the CBF database. It can be seen that the average behavior of the original CBF database remains after the processing originated by the applied pipeline. A significant dimensionality reduction was observed.

Figure 15 Note: Acronyms: P-Metaheuristics (Population-based Metaheuristics); S-Metaheuristics (Single point search Metaheuristics); SG (Savitzky-Golay Filter); MA (Moving Average); LRw (Local Regression-lowess); LRe (Local Regression-loess); SAX(Aggregate approXimation); PAA (Piecewise Aggregate Approximation); PCA (Principal Component Analysis); INSIGHT (Instance Selection based on Graph-coverage and Hubness for Time-series); MC1 (Monte Carlo 1); KNN1 (K-Nearest Neighbor-Euclidean Distance); KNN2 (K-Nearest Neighbor-Lower Bounding Dynamic Time Warping); NB (Naive Bayes); DT (Decision Tree); AB (AdaBoost); SVM (Support Vector Machine). Source: Authors

In order to enhance the analysis of the preferred solutions, a selection frequency analysis of the methods considered by the approaches for the FMS problem in time-series databases was made. Figure 16 shows the average frequency, for each recognized method, computed from five trials over all databases for each metaheuristic.

Figure 16 Note: (1) The average behavior of the original testing database is plotted that is compared to the average smoothed testing test. (2) The averaged smoothed testing database is plotted before the time-series representation process. (3) The averaged smoothed testing database is plotted after the time-series representation and numerosity reduction processes were applied. Source: Authors

The frequency results for population-baed meteheuristics were based on 601 200 evaluated pipeline-models, while single-point search metaheuristics were based on 300 600 models. Regarding the smoothing options, Moving Average method, was the most solicited by population-based meteheuristics, while the Savitzky-Golay filter was the most preferred by single-point search metaheuristics.

For time-series representation, the PAA method was the most preferred for both population-based and single-based metaheuristics. INSIGHT was the most selected numerosity reduction method. Regarding the classifiers, it can be confirmed that Adaboost was the most suitable classifier, while KNN1 was the less preferred.