Embedding large-scale graph and text-based datasets with LLMs

Abstract

Large text data sets, such as publications, websites, and other text-based media, inherit two distinct types of features: (1) the text itself, its information conveyed through semantics, and (2) its relationship to other texts through links, references, or shared attributes. While the latter can be described as a graph structure, enabling us to utilize tools and methods from graph theory, as well as conventional classification methods, the former has recently gained new potential through the use of LLM embedding models. Demonstrating these possibilities and their practicability, we investigate the Web of Science dataset, containing 56 million scientific publications through the lens of our proposed embedding method, revealing a self-structured landscape of texts. Furthermore, we discuss strategies for combining these emerging methods with traditional graph-based approaches, potentially compensating for each other’s shortcomings.