his.sePublications
Change search
Link to record
Permanent link

Direct link
BETA
Löfström, Tuve
Alternative names
Publications (10 of 10) Show all publications
Löfström, T., Johansson, U. & Boström, H. (2009). Ensemble Member Selection Using Multi-Objective Optimization. In: Proceedings of IEEE Symposium on Computational Intelligence and Data Mining (CIDM): . Paper presented at 2009 IEEE Symposium on Computational Intelligence and Data Mining (CIDM 2009) proceedings : March 30-April 2, 2009, Sheraton Music City Hotel, Nashville, TN, USA (pp. 245-251). IEEE conference proceedings
Open this publication in new window or tab >>Ensemble Member Selection Using Multi-Objective Optimization
2009 (English)In: Proceedings of IEEE Symposium on Computational Intelligence and Data Mining (CIDM), IEEE conference proceedings, 2009, p. 245-251Conference paper, Published paper (Refereed)
Abstract [en]

Both theory and a wealth of empirical studies have established that ensembles are more accurate than single predictive models. Unfortunately, the problem of how to maximize ensemble accuracy is, especially for classification, far from solved. In essence, the key problem is to find a suitable criterion, typically based on training or selection set performance, highly correlated with ensemble accuracy on novel data. Several studies have, however, shown that it is difficult to come up with a single measure, such as ensemble or base classifier selection set accuracy, or some measure based on diversity, that is a good general predictor for ensemble test accuracy. This paper presents a novel technique that for each learning task searches for the most effective combination of given atomic measures, by means of a genetic algorithm. Ensembles built from either neural networks or random forests were empirically evaluated on 30 UCI datasets. The experimental results show that when using the generated combined optimization criteria to rank candidate ensembles, a higher test set accuracy for the top ranked ensemble was achieved, compared to using ensemble accuracy on selection data alone. Furthermore, when creating ensembles from a pool of neural networks, the use of the generated combined criteria was shown to generally outperform the use of estimated ensemble accuracy as the single optimization criterion.

Place, publisher, year, edition, pages
IEEE conference proceedings, 2009
National Category
Computer and Information Sciences
Research subject
Technology
Identifiers
urn:nbn:se:his:diva-3212 (URN)10.1109/CIDM.2009.4938656 (DOI)000271487700035 ()2-s2.0-67650434708 (Scopus ID)978-1-4244-2765-9 (ISBN)
Conference
2009 IEEE Symposium on Computational Intelligence and Data Mining (CIDM 2009) proceedings : March 30-April 2, 2009, Sheraton Music City Hotel, Nashville, TN, USA
Available from: 2009-06-26 Created: 2009-06-26 Last updated: 2018-01-13Bibliographically approved
Löfström, T. (2009). Utilizing Diversity and Performance Measures for Ensemble Creation. (Licentiate dissertation). Örebro universitet
Open this publication in new window or tab >>Utilizing Diversity and Performance Measures for Ensemble Creation
2009 (English)Licentiate thesis, comprehensive summary (Other academic)
Abstract [en]

An ensemble is a composite model, aggregating multiple base models into one predictive model. An ensemble prediction, consequently, is a function of all included base models. Both theory and a wealth of empirical studies have established that ensembles are generally more accurate than single predictive models. The main motivation for using ensembles is the fact that combining several models will eliminate uncorrelated base classifier errors. This reasoning, however, requires the base classifiers to commit their errors on different instances – clearly there is no point in combining identical models. Informally, the key term diversity means that the base classifiers commit their errors independently of each other. The problem addressed in this thesis is how to maximize ensemble performance by analyzing how diversity can be utilized when creating ensembles. A series of studies, addressing different facets of the question, is presented. The results show that ensemble accuracy and the diversity measure difficulty are the two individually best measures to use as optimization criterion when selecting ensemble members. However, the results further suggest that combinations of several measures are most often better as optimization criteria than single measures. A novel method to find a useful combination of measures is proposed in the end. Furthermore, the results show that it is very difficult to estimate predictive performance on unseen data based on results achieved with available data. Finally, it is also shown that implicit diversity achieved by varied ANN architecture or by using resampling of features is beneficial for ensemble performance.

Place, publisher, year, edition, pages
Örebro universitet, 2009. p. 106
Series
Studies from the School of Science and Technology at Örebro University ; 2
Keywords
Ensemble Learning, Machine Learning, Diversity, Artificial Neural Networks, Data Mining, Information Fusion
National Category
Computer Sciences
Research subject
Technology
Identifiers
urn:nbn:se:his:diva-2920 (URN)
Presentation
(Swedish)
Available from: 2009-05-07 Created: 2009-03-27 Last updated: 2018-01-13Bibliographically approved
Johansson, U., Sönströd, C., Löfström, T. & Boström, H. (2008). Chipper - A Novel Algorithm for Concept Description. In: Frontiers in Artificial Intelligence and Applications. Paper presented at 10th Scandinavian Conference on Artificial Intelligence, SCAI 2008;Stockholm;26 May 2008through28 May 2008 (pp. 133-140). IOS Press
Open this publication in new window or tab >>Chipper - A Novel Algorithm for Concept Description
2008 (English)In: Frontiers in Artificial Intelligence and Applications, IOS Press, 2008, p. 133-140Conference paper, Published paper (Refereed)
Abstract [en]

In this paper, several demands placed on concept description algorithms are identified and discussed. The most important criterion is the ability to produce compact rule sets that, in a natural and accurate way, describe the most important relationships in the underlying domain. An algorithm based on the identified criteria is presented and evaluated. The algorithm, named Chipper, produces decision lists, where each rule covers a maximum number of remaining instances while meeting requested accuracy requirements. In the experiments, Chipper is evaluated on nine UCI data sets. The main result is that Chipper produces compact and understandable rule sets, clearly fulfilling the overall goal of concept description. In the experiments, Chipper’s accuracy is similar to standard decision tree and rule induction algorithms, while rule sets have superior comprehensibility.

Place, publisher, year, edition, pages
IOS Press, 2008
Series
Frontiers in Artificial Intelligence and Applications, ISSN 0922-6389, 1879-8314 ; 173
Research subject
Technology
Identifiers
urn:nbn:se:his:diva-3614 (URN)000273520700017 ()2-s2.0-84867569402 (Scopus ID)978-1-58603-867-0 (ISBN)
Conference
10th Scandinavian Conference on Artificial Intelligence, SCAI 2008;Stockholm;26 May 2008through28 May 2008
Available from: 2010-01-29 Created: 2010-01-29 Last updated: 2017-11-27
Johansson, U., Löfström, T. & Niklasson, L. (2008). Evaluating Standard Techniques for Implicit Diversity. In: Takashi Washio, Einoshin Suzuki, Kai Ming Ting, Akihiro Inokuchi (Ed.), Advances in Knowledge Discovery and Data Mining: 12th Pacific-Asia Conference, PAKDD 2008 Osaka, Japan, May 20-23, 2008 Proceedings. Paper presented at 12th Pacific-Asia Conference, PAKDD 2008 Osaka, Japan, May 20-23, 2008 (pp. 592-599). Springer Berlin/Heidelberg
Open this publication in new window or tab >>Evaluating Standard Techniques for Implicit Diversity
2008 (English)In: Advances in Knowledge Discovery and Data Mining: 12th Pacific-Asia Conference, PAKDD 2008 Osaka, Japan, May 20-23, 2008 Proceedings / [ed] Takashi Washio, Einoshin Suzuki, Kai Ming Ting, Akihiro Inokuchi, Springer Berlin/Heidelberg, 2008, p. 592-599Conference paper, Published paper (Refereed)
Abstract [en]

When performing predictive modeling, ensembles are often utilized in order to boost accuracy. The problem of how to maximize ensemble accuracy is, however, far from solved. In particular, the relationship between ensemble diversity and accuracy is, especially for classification, not completely understood. More specifically, the fact that ensemble diversity and base classifier accuracy are highly correlated, makes it necessary to balance these properties instead of just maximizing diversity. In this study, three standard techniques to obtain implicit diversity in neural network ensembles are evaluated using 14 UCI data sets. The experiments show that standard resampling; i.e. dividing the training data by instances, produces more diverse models, but at the expense of base classifier accuracy, thus resulting in less accurate ensembles. Building ensembles using neural networks with heterogeneous architectures improves test set accuracies, but without actually increasing the diversity. The results regarding resampling using features are inconclusive, the ensembles become more diverse, but the level of test set accuracies is unchanged. For the setups evaluated, ensemble training accuracy and base classifier training accuracy are positively correlated with ensemble test accuracy, but the opposite holds for diversity; i.e. ensembles with low diversity are generally more accurate.

Place, publisher, year, edition, pages
Springer Berlin/Heidelberg, 2008
Series
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), ISSN 0302-9743, E-ISSN 1611-3349 ; 5012
National Category
Computer Sciences
Research subject
Technology
Identifiers
urn:nbn:se:his:diva-2797 (URN)10.1007/978-3-540-68125-0_54 (DOI)000256127100053 ()2-s2.0-44649182764 (Scopus ID)978-3-540-68124-3 (ISBN)978-3-540-68125-0 (ISBN)
Conference
12th Pacific-Asia Conference, PAKDD 2008 Osaka, Japan, May 20-23, 2008
Note

Also part of the Lecture Notes in Artificial Intelligence book sub series (LNAI, volume 5012)

Available from: 2009-03-02 Created: 2009-03-02 Last updated: 2019-10-08Bibliographically approved
Löfström, T., Johansson, U., Sönströd, C., König, R. & Niklasson, L. (Eds.). (2007). Proceedings of SAIS 2007: The 24th Annual Workshop of the Swedish Artificial Intelligence Society, Borås, May 22-23, 2007. University College of Borås
Open this publication in new window or tab >>Proceedings of SAIS 2007: The 24th Annual Workshop of the Swedish Artificial Intelligence Society, Borås, May 22-23, 2007
Show others...
2007 (English)Conference proceedings (editor) (Other academic)
Place, publisher, year, edition, pages
University College of Borås, 2007
Research subject
Technology
Identifiers
urn:nbn:se:his:diva-3701 (URN)
Available from: 2010-02-16 Created: 2010-02-16 Last updated: 2017-11-27
Johansson, U., Löfström, T., König, R. & Niklasson, L. (2006). Accurate Neural Network Ensembles Using Genetic Programming. In: Proceedings of SAIS: The 23rd Annual Workshop of the Swedish Artificial Intelligence Society. Swedish Artificial Intelligence Society - SAIS, Umeå universitet
Open this publication in new window or tab >>Accurate Neural Network Ensembles Using Genetic Programming
2006 (English)In: Proceedings of SAIS: The 23rd Annual Workshop of the Swedish Artificial Intelligence Society, Swedish Artificial Intelligence Society - SAIS, Umeå universitet , 2006Conference paper, Published paper (Other academic)
Abstract [en]

Abstract: In this paper we present and evaluate a novel algorithm for ensemble creation. The main idea of the algorithm is to first independently train a fixed number of neural networks (here ten) and then use genetic programming to combine these networks into an ensemble. The use of genetic programming makes it possible to not only consider ensembles of different sizes, but also to use ensembles as intermediate building blocks. The final result is therefore more correctly described as an ensemble of neural network ensembles. The experiments show that the proposed method, when evaluated on 22 publicly available data sets, obtains very high accuracy, clearly outperforming the other methods evaluated. In this study several micro techniques are used, and we believe that they all contribute to the increased performance.

One such micro technique, aimed at reducing overtraining, is the training method, called tombola training, used during genetic evolution. When using tombola training, training data is regularly resampled into new parts, called training groups. Each ensemble is then evaluated on every training group and the actual fitness is determined solely from the result on the hardest part.

Place, publisher, year, edition, pages
Swedish Artificial Intelligence Society - SAIS, Umeå universitet, 2006
Series
UMINF, ISSN 0348-0542
Identifiers
urn:nbn:se:his:diva-1914 (URN)
Available from: 2007-03-22 Created: 2007-03-22 Last updated: 2017-11-27
Löfström, T., König, R., Johansson, U., Niklasson, L., Strand, M. & Ziemke, T. (2006). Benefits of Relating the Retail Domain to Information Fusion. In: 9th International Conference on Information Fusion: IEEE ISIF. Paper presented at 9th International Conference on Information Fusion, ICIF '06, Florence (Italy), July 10-13, 2006 (pp. Article number 4085930). IEEE conference proceedings
Open this publication in new window or tab >>Benefits of Relating the Retail Domain to Information Fusion
Show others...
2006 (English)In: 9th International Conference on Information Fusion: IEEE ISIF, IEEE conference proceedings, 2006, p. Article number 4085930-Conference paper, Published paper (Refereed)
Place, publisher, year, edition, pages
IEEE conference proceedings, 2006
Identifiers
urn:nbn:se:his:diva-1956 (URN)2-s2.0-50149109426 (Scopus ID)0-9721844-6-5 (ISBN)
Conference
9th International Conference on Information Fusion, ICIF '06, Florence (Italy), July 10-13, 2006
Available from: 2008-04-11 Created: 2008-04-11 Last updated: 2017-11-27Bibliographically approved
Johansson, U., Löfström, T., König, R. & Niklasson, L. (2006). Building Neural Network Ensembles using Genetic Programming. In: The International Joint Conference on Neural Networks 2006. Paper presented at International Joint Conference on Neural Networks 2006, IJCNN '06;Vancouver, BC;16 July 2006through21 July 2006 (pp. 2239-2244). IEEE Press
Open this publication in new window or tab >>Building Neural Network Ensembles using Genetic Programming
2006 (English)In: The International Joint Conference on Neural Networks 2006, IEEE Press, 2006, p. 2239-2244Conference paper, Published paper (Refereed)
Abstract [en]

algorithm for ensemble creation. The main idea of the algorithm is to first independently train a fixed number of neural networks (here ten) and then use genetic programming to combine these networks into an ensemble. The use of genetic programming makes it possible to not only consider ensembles of different sizes, but also to use ensembles as intermediate building blocks. The final result is therefore more correctly described as an ensemble of neural network ensembles. The experiments show that the proposed method, when evaluated on 22 publicly available data sets, obtains very high accuracy, clearly outperforming the other methods evaluated. In this study several micro techniques are used, and we believe that they all contribute to the increased performance. One such micro technique, aimed at reducing overtraining, is the training method, called tombola training, used during genetic evolution. When using tombola training, training data is regularly resampled into new parts, called training groups. Each ensemble is then evaluated on every training group and the actual fitness is determined solely from the result on the hardest part.

Place, publisher, year, edition, pages
IEEE Press, 2006
Identifiers
urn:nbn:se:his:diva-1806 (URN)10.1109/IJCNN.2006.246836 (DOI)000245125902029 ()2-s2.0-38049049329 (Scopus ID)
Conference
International Joint Conference on Neural Networks 2006, IJCNN '06;Vancouver, BC;16 July 2006through21 July 2006
Available from: 2007-10-10 Created: 2007-10-10 Last updated: 2017-11-27
Johansson, U., Löfström, T., König, R. & Niklasson, L. (2006). Genetically Evolved Trees Representing Ensembles. In: Artificial intelligence and soft computing - ICAISC 2006: 8th international conference, Zakopane, Poland, June 25 - 29, 2006 ; proceedings (pp. 613-622).
Open this publication in new window or tab >>Genetically Evolved Trees Representing Ensembles
2006 (English)In: Artificial intelligence and soft computing - ICAISC 2006: 8th international conference, Zakopane, Poland, June 25 - 29, 2006 ; proceedings, 2006, p. 613-622Conference paper, Published paper (Refereed)
Abstract [en]

We have recently proposed a novel algorithm for ensemble creation called GEMS (Genetic Ensemble Member Selection). GEMS first trains a fixed number of neural networks (here twenty) and then uses genetic programming to combine these networks into an ensemble. The use of genetic programming makes it possible for GEMS to not only consider ensembles of different sizes, but also to use ensembles as intermediate building blocks. In this paper, which is the first extensive study of GEMS, the representation language is extended to include tests partitioning the data, further increasing flexibility. In addition, several micro techniques are applied to reduce overfitting, which appears to be the main problem for this powerful algorithm. The experiments show that GEMS, when evaluated on 15 publicly available data sets, obtains very high accuracy, clearly outperforming both straightforward ensemble designs and standard decision tree algorithms.

Series
Lecture Notes in Computer Science, ISSN 0302-9743 ; 4029
National Category
Engineering and Technology
Research subject
Technology
Identifiers
urn:nbn:se:his:diva-1587 (URN)10.1007/11785231_64 (DOI)000239600000064 ()2-s2.0-33746239343 (Scopus ID)978-3-540-35748-3 (ISBN)
Available from: 2008-02-08 Created: 2008-02-08 Last updated: 2017-11-27
Johansson, U., Löfström, T., König, R., Sönströd, C. & Niklasson, L. (2006). Rule Extraction from Opaque Models: A Slightly Different Perspective. In: 6th International Conference on Machine Learning and Applications (pp. 22-27). IEEE Computer Society
Open this publication in new window or tab >>Rule Extraction from Opaque Models: A Slightly Different Perspective
Show others...
2006 (English)In: 6th International Conference on Machine Learning and Applications, IEEE Computer Society, 2006, p. 22-27Conference paper, Published paper (Refereed)
Abstract [en]

When performing predictive modeling, the key criterion is always accuracy. With this in mind, complex techniques like neural networks or ensembles are normally used, resulting in opaque models impossible to interpret. When models need to be comprehensible, accuracy is often sacrificed by using simpler techniques directly producing transparent models; a tradeoff termed the accuracy vs. comprehensibility tradeoff. In order to reduce this tradeoff, the opaque model can be transformed into another, interpretable, model; an activity termed rule extraction. In this paper, it is argued that rule extraction algorithms should gain from using oracle data; i.e. test set instances, together with corresponding predictions from the opaque model. The experiments, using 17 publicly available data sets, clearly show that rules extracted using only oracle data were significantly more accurate than both rules extracted by the same algorithm, using training data, and standard decision tree algorithms. In addition, the same rules were also significantly more compact; thus providing better comprehensibility. The overall implication is that rules extracted in this fashion will explain the predictions made on novel data better than rules extracted in the standard way; i.e. using training data only.

Place, publisher, year, edition, pages
IEEE Computer Society, 2006
Identifiers
urn:nbn:se:his:diva-1952 (URN)10.1109/ICMLA.2006.46 (DOI)000244477800004 ()2-s2.0-40349090116 (Scopus ID)0-7695-2735-3 (ISBN)
Available from: 2008-04-11 Created: 2008-04-11 Last updated: 2017-11-27
Organisations

Search in DiVA

Show all publications