A Multimodal Approach for Semantic Patent Image Retrieval
dc.bibliographicCitation.bookTitle | Proceedings of the 2nd Workshop on Patent Text Mining and Semantic Technologies (PatentSemTech) 2021 co-located with the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021) | eng |
dc.bibliographicCitation.seriesTitle | CEUR Workshop Proceedings ; 2909 | eng |
dc.bibliographicCitation.volume | 2909 | eng |
dc.contributor.author | Pustu-Iren, Kader | |
dc.contributor.author | Bruns, Gerrit | |
dc.contributor.author | Ewerth, Ralph | |
dc.date.accessioned | 2021-12-22T16:11:27Z | |
dc.date.available | 2021-12-22T16:11:27Z | |
dc.date.issued | 2021 | |
dc.description.abstract | Patent images such as technical drawings contain valuable information and are frequently used by experts to compare patents. However, current approaches to patent information retrieval are largely focused on textual information. Consequently, we review previous work on patent retrieval with a focus on illustrations in figures. In this paper, we report on work in progress for a novel approach for patent image retrieval that uses deep multimodal features. Scene text spotting and optical character recognition are employed to extract numerals from an image to subsequently identify references to corresponding sentences in the patent document. Furthermore, we use a neural state-of-the-art CLIP model to extract structural features from illustrations and additionally derive textual features from the related patent text using a sentence transformer model. To fuse our multimodal features for similarity search we apply re-ranking according to averaged or maximum scores. In our experiments, we compare the impact of different modalities on the task of similarity search for patent images. The experimental results suggest that patent image retrieval can be successfully performed using the proposed feature sets, while the best results are achieved when combining the features of both modalities. | eng |
dc.description.version | publishedVersion | eng |
dc.identifier.uri | https://oa.tib.eu/renate/handle/123456789/7795 | |
dc.identifier.uri | https://doi.org/10.34657/6842 | |
dc.language.iso | eng | eng |
dc.publisher | Aachen, Germany : RWTH Aachen | eng |
dc.relation.essn | 1613-0073 | |
dc.relation.uri | http://ceur-ws.org/Vol-2909/paper6.pdf | |
dc.rights.license | CC BY 4.0 Unported | eng |
dc.rights.uri | https://creativecommons.org/licenses/by/4.0/ | eng |
dc.subject.ddc | 020 | eng |
dc.subject.gnd | Konferenzschrift | ger |
dc.subject.other | Patent Image Similarity Search | eng |
dc.subject.other | Deep Learning | eng |
dc.subject.other | Mulitmodal Feature Representations | eng |
dc.subject.other | Scene Text Spotting | eng |
dc.title | A Multimodal Approach for Semantic Patent Image Retrieval | eng |
dc.type | BookPart | eng |
dc.type | Text | eng |
dcterms.event | PatentSemTech 2021 - Patent Text Mining and Semantic Technologies, July 15th 2021, online | |
tib.accessRights | openAccess | eng |
wgl.contributor | TIB | eng |
wgl.subject | Informatik | eng |
wgl.type | Buchkapitel / Sammelwerksbeitrag | eng |
wgl.type | Konferenzbeitrag | eng |
Files
Original bundle
1 - 1 of 1