Högskolan i Skövde

The local environment and land usages have changed a lot during the past one hundred years. Historical documents and materials are crucial in understanding and following these changes. Historical documents are, therefore, an important piece in the understanding of the impact and consequences of land usage change. This, in turn, is important in the search of restoration projects that can be conducted to turn and reduce harmful and unsustainable effects originating from changes in the land-usage. This work extracts information on the historical location and geographical distribution of wetlands, from hand-drawn maps. This is achieved by using deep learning (DL), and more specifically a convolutional neural network (CNN). The CNN model is trained on a manually pre-labelled dataset on historical wetlands in the area of Jönköping county in Sweden. These are all extracted from the historical map called “Generalstabskartan”. The presented CNN performs well and achieves a F1-score of 0.886 when evaluated using a 10-fold cross validation over the data. The trained models are additionally used to generate a GIS layer of the presumable historical geographical distribution of wetlands for the area that is depicted in the southern collection in Generalstabskartan, which covers the southern half of Sweden. This GIS layer is released as an open resource and can be freely used. To summarise, the presented results show that CNNs can be a useful tool in the extraction and digitalisation of non-textual information in historical documents, such as historical maps. A modern GIS material that can be used to further understand the past land-usage change is produced within this research. Previously, no material of this detail and extent have been available, due to the large effort needed to manually create such. However, with the presented resource better quantifications and estimations of historical wetlands that have been lost can be made. 

A Basic Oxygen Furnace (BOF) for steel making is a complex industrial process that is difficult to monitor due to the harsh environment, so the collected production data is very limited given the process complexity. Also, such production data has a low degree of variability. An accurate machine learning (ML) model for predicting production outcome requires both large and varied data, so utilising data from multiple BOFs will allow for more capable ML models, since both the amount and variability of data increases. Data collection setups for different BOFs are different, such that data sets are not compatible to directly join for ML training. Our approach is to let a neural network benefit from these collection differences in a joint training model. We present a neural network-based approach that simultaneously and jointly co-trains on several data sets. Our novelty is that the first network layer finds an internal representation of each individual BOF, while the other layers use this representation to concurrently learn a common BOF model. Our evaluation shows that the prediction accuracy of the common model increases compared to separate models trained on individual furnaces’ data sets. It is clear that multiple data sets can be utilised this way to increase model accuracy for better production prediction performance. For the industry, this means that the amount of available data for model training increases and thereby more capable ML models can be trained when having access to multiple data sets describing the same or similar manufacturing processes. 

The research in this thesis is focused on how deep learning models can be designed and implemented to better emulate and integrate the heuristic reasoning process of human experts. Several case studies, within the domains of steel making and drug discovery, are conducted in order to evaluate the performance of these models, in comparison to models that do not consider human expert knowledge from the targeted domain. These case studies focus on separate problems in targeted industries and, for example, deals with predictions of the outcome in the melting of steel and the rolling of steel sheets. This thesis also deals with property predictions of molecules and the generation of new drug candidates.

The research in this thesis addresses three main qustions. Firstly, the impact of different data representations and especially representations that are close to how human experts represent the addressed problem is studied. Secondly, focus is put on how the internal structure of data-driven algorithms can be designed to address problems in the same way as human experts. Finally, it is investigated how constraints, which are specifiedby experts, can be integrated into models to get models that generalise better and prevent non-feasible extrapolation. These questions are addressed in empirical case studies, where the model development is steered by knowledge about the problem provided by domain experts. In the presented cases, the developed algorithms performed significantly better, under the performance metric used, than in other commonly used machine learning models. Hence, this thesis shows the need for the involvement of domain experts in the design of models possessing high performance. This requires models that can both be applied to various data representations and be designed to compute and reason in multiple ways. Hence, the models must be flexible, so that they can be designed to reason in different ways and be applied to different data representations that correspond to the mental representations that is used by the human experts for the targeted problem.

The conclusion of this thesis is that such models can be designed using artificial neural networks, and that these models perform better than conventional machine learning models, when applied to new data. Such models can be designed so that they can be applied to complex representations of the data, representing the targeted problem better. More information regarding the problem can also be provided by designing the model in such a way that the reasoning of human experts is emulated. This can either bedone by emulating the expert behaviour in computational steps or by specifying known boundaries of the problem. The main conclusion is that models that are applied to datarepresentations that capture the real world problem in a good way, emulate the steps that human experts take to solve the problem, or know the limitations around the problem, have more information about the problem, and therefore, have an edge over other models, ultimately resulting in better performance.

The problem of using machine learning (ML) to predict the process endpoint for a Basic Oxygen Furnace (BOF) process used for steelmaking has been largely studied. However, current research often lacks both the usage of a rich dataset and does not address revealing influential factors that explain the process. The process is complex and difficult to control and has a multi-objective target endpoint with a proper range of heat temperature combined with sufficiently low levels of carbon and phosphorus. Reaching this endpoint requires skilled process operators, who are manually controlling the heat throughout the process by using both implicit and explicit control variables in their decisions. Trained ML models can reach good BOF target prediction results, but it is still a challenge to extract the influential factors that are significant to the ML prediction accuracy. Thus, it becomes a challenge to explain and validate an ML prediction model that claims to capture the process well. This paper makes use of a complex and full production dataset to evaluate and compare different approaches for understanding how the data can determine the process target prediction. One approach is based on the collected process data and the other on the ML approach trained on that data to find the influential factors. These complementary approaches aim to explain the BOF process to reveal actionable information on how to improve process control.

Basic oxygen steel making is a complex chemical and physical industrial process that reduces a mix of pig iron and recycled scrap into low-carbon steel. Good understanding of the process and the ability to predict how it will evolve requires long operator experience, but this can be augmented with process target prediction systems. Such systems may use machine learning to learn a model of the process based on a long process history, and have an advantage in that they can make useof vastly more process parameters than operators can comprehend. While it has become less of a challenge to build such prediction systems using machine learning algorithms, actual production implementations are rare. The hidden reasoning of complex prediction model and lack of transparency prevents operator trust, even for models that show high accuracy predictions. To express model behaviour and thereby increasing transparency we develop a reinforcement learning (RL) based agent approach, which task is to generate short polynomials that can explain the model of the process from what it has learnt from process data. The RL agent is rewarded on how well it generates polynomials that can predict the process from a smaller subset of the process parameters. Agent training is done with the REINFORCE algorithm, which enables the sampling of multiple concurrently plausible polynomials. Having multiple polynomials, process developers can evaluate several alternative and plausible explanations, as observed in the historic process data. The presented approach gives both a trained generative model and a set of polynomials that can explain the process. The performance of the polynomials is as good as or better than more complex and less interpretable models. Further, the relative simplicity of the resulting polynomials allows good generalisation to fit new instances of data. The best of the resulting polynomials in our evaluation achieves a better R 2 score on the test set in comparison to the other machine learning models evaluated.

Artificial intelligence (AI) is nowadays included into an increasing number of critical systems. Inclusion of AI in such systems may, however, pose a risk, since it is, still, infeasible to build AI systems that know how to function well in situations that differ greatly from what the AI has seen before. Therefore, it is crucial that future AI systems have the ability to not only function well in known domains, but also understand and show when they are uncertain when facing something unknown. In this paper, we evaluate four different methods that have been proposed to correctly quantifying uncertainty when the AI model is faced with new samples. We investigate the behaviour of these models when they are applied to samples far from what these models have seen before, and if they correctly attribute those samples with high uncertainty. We also examine if incorrectly classified samples are attributed with an higher uncertainty than correctly classified samples. The major finding from this simple experiment is, surprisingly, that the evaluated methods capture the uncertainty differently and the correlation between the quantified uncertainty of the models is low. This inconsistency is something that needs to be further understood and solved before AI can be used in critical applications in a trustworthy and safe manner. © 2020, Springer Nature Switzerland AG.

The steel-making process in a Basic Oxygen Furnace (BOF) must meet a combination of target values such as the final melt temperature and upper limits of the carbon and phosphorus content of the final melt with minimum material loss. An optimal blow end time (cut-off point), where these targets are met, often relies on the experience and skill of the operators who control the process, using both collected sensor readings and an implicit understanding of how the process develops. If the precision of hitting the optimal cut-off point can be improved, this immediately increases productivity as well as material and energy efficiency, thus decreasing environmental impact and cost. We examine the usage of standard machine learning models to predict the end-point targets using a full production dataset. Various causes of prediction uncertainty are explored and isolated using a combination of raw data and engineered features. In this study, we reach robust temperature, carbon, and phosphorus prediction hit rates of 88, 92, and 89 pct, respectively, using a large production dataset. © 2020, The Author(s).

Data science applications often need to deal with data that does not fit into the standard entity-attribute-value model. In this chapter we discuss three of these other types of data. We discuss texts, images and graphs. The importance of social media is one of the reason for the interest on graphs as they are a way to represent social networks and, in general, any type of interaction between people. In this chapter we present examples of tools that can be used to extract information and, thus, analyze these three types of data. In particular, we discuss topic modeling using a hierarchical statistical model as a way to extract relevant topics from texts, image analysis using convolutional neural networks, and measures and visual methods to summarize information from graphs.

In medicinal chemistry programs it is key to design and make compounds that are efficacious and safe. This is a long, complex, and difficult multiparameter optimization process, often including several properties with orthogonal trends. New methods for the automated design of compounds against profiles of multiple properties are thus of great value. Here we present a fragment-based reinforcement learning approach based on an actor-critic model, for the generation of novel molecules with optimal properties. The actor and the critic are both modeled with bidirectional long short-term memory (LSTM) networks. The AI method learns how to generate new compounds with desired properties by starting from an initial set of lead molecules and then improving these by replacing some of their fragments. A balanced binary tree based on the similarity of fragments is used in the generative process to bias the output toward structurally similar molecules. The method is demonstrated by a case study showing that 93% of the generated molecules are chemically valid and more than a third satisfy the targeted objectives, while there were none in the initial set.

We present a flexible deep convolutional neural network method for the analyse of arbitrary sized graph structures representing molecules. The method makes use of RDKit, an open-source cheminformatics software, allowing the incorporation of any global molecular (such as molecular charge) and local (such as atom type) information. We evaluate the method on the Side Effect Resource (SIDER) v4.1 dataset and show that it significantly outperforms another recently proposed method based on deep convolutional neural networks. We also reflect on how different types of information and input data affect the predictive power of our model. This reflection highlights several open problems that should be solved to further improve the use of deep learning within cheminformatics.