Overview

Structured JSON extraction from ingested content

As content is ingested into Graphlit, it will be automatically prepared - either by text extraction or audio transcription, and will prepare a JSON formatted file, stored on Azure blob storage.

This JSON file will be used by the platform as the source for LLM text embeddings, as well as to provide wider context for search 'hits' during the Retrieval Augmented Generation (RAG) pipeline during LLM conversations.

Document JSON

Transcript JSON

Formatted Text

In addition to requesting the URI for the JSON file, you can request the entire extracted text.

query QueryContents($filter: ContentFilter!) {
  contents(filter: $filter) {
    results {
      id
      text
    }
  }
}

{
  "results": [
    {
      "text": "JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021\nUnifying Large Language Models and Knowledge Graphs: A Roadmap\nShirui Pan, Senior Member, IEEE, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, Xindong Wu, Fellow, IEEE\nAbstract-Large language models (LLMs), such as ChatGPT and GPT4, are making new waves in the field of natural language processing and artificial intelligence, due to their emergent ability and generalizability. However, LLMs are black-box models, which often fall short of capturing and accessing factual knowledge. In contrast, Knowledge Graphs (KGs), Wikipedia and Huapu for example, are structured knowledge models that explicitly store rich factual knowledge. KGs can enhance LLMs by providing external knowledge for inference and interpretability. Meanwhile, KGs are difficult to construct and evolving by nature, which challenges the existing methods in KGs to generate new facts and represent unseen knowledge. Therefore, it is complementary to unify LLMs and KGs together and simultaneously leverage their advantages. In this article, we present a forward-looking roadmap for the unification of LLMs and KGs. Our roadmap consists of three general frameworks, namely, 1) KG-enhanced LLMs, which incorporate KGs during the pre-training and inference phases of LLMs, or for the purpose of enhancing understanding of the knowledge learned by LLMs; 2) LLM-augmented KGs, that leverage LLMs for different KG tasks such as embedding, completion, construction, graph-to-text generation, and question answering; and 3) Synergized LLMs + KGs, in which LLMs and KGs play equal roles and work in a mutually beneficial way to enhance both LLMs and KGs for bidirectional reasoning driven by both data and knowledge. We review and summarize existing efforts within these three frameworks in our roadmap and pinpoint their future research directions. ...",
      "id": "3b02a191-c845-4b99-b711-2d1bc7cc880f"
    }
  ]
}

Or you can request a formatted markdown version of the extracted JSON. This uses the role of the text chunk to infer the proper Markdown formatting.

query QueryContents($filter: ContentFilter!) {
  contents(filter: $filter) {
    results {
      id
      markdown
    }
  }
}

{
  "results": [
    {
      "markdown": "JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021\n\n## Unifying Large Language Models and Knowledge Graphs: A Roadmap\n\nShirui Pan, Senior Member, IEEE, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, Xindong Wu, Fellow, IEEE\n\nAbstract-Large language models (LLMs), such as ChatGPT and GPT4, are making new waves in the field of natural language processing and artificial intelligence, due to their emergent ability and generalizability. However, LLMs are black-box models, which often fall short of capturing and accessing factual knowledge. In contrast, Knowledge Graphs (KGs), Wikipedia and Huapu for example, are structured knowledge models that explicitly store rich factual knowledge. KGs can enhance LLMs by providing external knowledge for inference and interpretability. Meanwhile, KGs are difficult to construct and evolving by nature, which challenges the existing methods in KGs to generate new facts and represent unseen knowledge. Therefore, it is complementary to unify LLMs and KGs together and simultaneously leverage their advantages. In this article, we present a forward-looking roadmap for the unification of LLMs and KGs. Our roadmap consists of three general frameworks, namely, 1) KG-enhanced LLMs, which incorporate KGs during the pre-training and inference phases of LLMs, or for the purpose of enhancing understanding of the knowledge learned by LLMs; 2) LLM-augmented KGs, that leverage LLMs for different KG tasks such as embedding, completion, construction, graph-to-text generation, and question answering; and 3) Synergized LLMs + KGs, in which LLMs and KGs play equal roles and work in a mutually beneficial way to enhance both LLMs and KGs for bidirectional reasoning driven by both data and knowledge. We review and summarize existing efforts within these three frameworks in our roadmap and pinpoint their future research directions.\n",
      "id": "3b02a191-c845-4b99-b711-2d1bc7cc880f"
    }
  ]
}