MULTI-TASK DEEP LEARNING WITH INCOMPLETE TRAINING SAMPLES FOR THE IMAGE-BASED PREDICTION OF VARIABLES DESCRIBING SILK FABRICS

Abstract. This paper presents a method for the classification of images of silk fabrics with the aim to predict properties such as the place and time of origin and the production technique. The proposed method was developed in the context of the EU project SILKNOW (http://silknow.eu/). In the context of classification, we address the problem of limited as well as not fully labelled data and investigate the connection between the distinct variables. A pre-trained Convolutional Neural Network (CNN) is used for the feature extraction and a classification network realizing Multi-task learning (MTL) is trained based on these features. The training procedure is adapted to enable the consideration of images that do not have a label for all tasks. Additionally, MTL with fully labeled training data is investigated for the classification of silk fabrics. The impact of both MTL approaches is compared to singletask learning based on two different class structures. We achieve overall accuracies of 92–95 % and average F1-scores of 88–90 % in our best experiments.