Filters

2017 - 20202

More filters

2019 - 201912020 - 20221
Reset filters

Settings

Subject: deep learning × WGL Institute: TIB ×

Search Results

Now showing 1 - 2 of 2
  • Thumbnail Image
    Item
    Estimating the information gap between textual and visual representations
    (New York City : Association for Computing Machinery, 2017) Henning, Christian; Ewerth, Ralph
    Photos, drawings, figures, etc. supplement textual information in various kinds of media, for example, in web news or scientific pub- lications. In this respect, the intended effect of an image can be quite different, e.g., providing additional information, focusing on certain details of surrounding text, or simply being a general il- lustration of a topic. As a consequence, the semantic correlation between information of different modalities can vary noticeably, too. Moreover, cross-modal interrelations are often hard to describe in a precise way. The variety of possible interrelations of textual and graphical information and the question, how they can be de- scribed and automatically estimated have not been addressed yet by previous work. In this paper, we present several contributions to close this gap. First, we introduce two measures to describe cross- modal interrelations: cross-modal mutual information (CMI) and semantic correlation (SC). Second, a novel approach relying on deep learning is suggested to estimate CMI and SC of textual and visual information. Third, three diverse datasets are leveraged to learn an appropriate deep neural network model for the demanding task. The system has been evaluated on a challenging test set and the experimental results demonstrate the feasibility of the approach.
  • Thumbnail Image
    Item
    Classification of important segments in educational videos using multimodal features
    (Aachen, Germany : RWTH Aachen, 2020) Ghauri, Junaid Ahmed; Hakimov, Sherzod; Ewerth, Ralph; Conrad, Stefan; Tiddi, Ilaria
    Videos are a commonly-used type of content in learning during Web search. Many e-learning platforms provide quality content, but sometimes educational videos are long and cover many topics. Humans are good in extracting important sec-tions from videos, but it remains a significant challenge for computers. In this paper, we address the problem of assigning importance scores to video segments, that is how much information they contain with respect to the overall topic of an educational video. We present an annotation tool and a new dataset of annotated educational videos collected from popular online learning platforms. Moreover, we propose a multimodal neural architecture that utilizes state-of-the-art audio, visual and textual features. Our experiments investigate the impact of visual and temporal information, as well as the combination of multimodal features on importance prediction.