Create Feed With Workflow

Create Web feed with workflow to crawl links.

When ingesting content into Graphlit, you often will want to configure how the content is processed. Via the Workflow entity, you can specify the stages of the content workflow, which gives fine-grained control over operations like text summarization, entity extraction, and link crawling.

In this example, we will create a workflow to crawl the links found in the Web pages of a Web feed.

First, we call createWorkflow mutation, with the enrichment stage configured to crawl a maximum of 10 Web links per-content, and ignoring the content's domain.

Then, we call createFeed mutation, and pass the ID of the workflow to be used.

Finally, we call the content query to view one of the crawled Web pages.

The content shown has the URI https://arxiv.org/abs/2303.10130, and that page is not within the sitemap of the feed URI https://openai.com/blog, so it was crawled from the links found on one of the feed's Web pages.

If no workflow is specified with the createFeed mutation, Graphlit will look to see if the project has a default workflow assigned. If one was assigned, it will use that, and if not, it will process the content with the built-in workflow stages (which simply indexes metadata, and prepare content for semantic search and conversations).

Create Enrichment Workflow

Mutation:

mutation CreateWorkflow($workflow: WorkflowInput!) {
  createWorkflow(workflow: $workflow) {
    id
    name
    state
    enrichment {
      link {
        enableCrawling
        allowedDomains
        excludedDomains
        allowedLinks
        excludedLinks
        allowedFiles
        excludedFiles
        allowContentDomain
      }
    }
  }
}

Variables:

{
  "workflow": {
    "enrichment": {
      "link": {
        "enableCrawling": true,
        "allowedLinks": [
          "WEB"
        ],
        "allowContentDomain": false,
        "maximumLinks": 10
      }
    },
    "name": "Enrichment Workflow"
  }
}

Response:

{
  "enrichment": {
    "link": {
      "enableCrawling": true,
      "allowedLinks": [
        "WEB"
      ],
      "allowContentDomain": false
    }
  },
  "id": "d8875dd0-7a3b-45f9-b8bb-cc45fb04d5c3",
  "name": "Enrichment Workflow",
  "state": "ENABLED"
}

Create Web Feed

Mutation:

mutation CreateFeed($feed: FeedInput!) {
  createFeed(feed: $feed) {
    id
    name
    state
    type
  }
}

Variables:

{
  "feed": {
    "type": "WEB",
    "web": {
      "uri": "https://openai.com/blog",
      "readLimit": 10
    },
    "workflow": {
      "id": "d8875dd0-7a3b-45f9-b8bb-cc45fb04d5c3"
    },
    "schedulePolicy": {
      "recurrenceType": "ONCE"
    },
    "name": "Feed With Workflow"
  }
}

Response:

{
  "type": "WEB",
  "id": "1b6c0901-81c4-457e-bd35-c31367bc2799",
  "name": "Feed With Workflow",
  "state": "ENABLED"
}

Query Contents

Query:

query QueryContents($filter: ContentFilter!) {
  contents(filter: $filter) {
    results {
      id
      name
      creationDate
      owner {
        id
      }
      state
      originalDate
      finishedDate
      workflowDuration
      uri
      text
      type
      fileType
      mimeType
      fileName
      fileSize
      masterUri
      mezzanineUri
      transcriptUri
      links {
        uri
        linkType
      }
      document {
        title
        subject
        summary
        author
        publisher
        description
        keywords
        pageCount
      }
    }
  }
}

Variables:

{
  "filter": {
    "queryType": "SIMPLE",
    "searchType": "VECTOR",
    "offset": 0,
    "limit": 1
  }
}

Response:

{
  "results": [
    {
      "type": "PAGE",
      "links": [
        {
          "uri": "https://www.cornell.edu/",
          "linkType": "WEB"
        },
        {
          "uri": "https://info.arxiv.org/about/ourmembers.html",
          "linkType": "FILE"
        },
        {
          "uri": "https://info.arxiv.org/about/donate.html",
          "linkType": "FILE"
        },
        {
          "uri": "https://info.arxiv.org/help/index.html",
          "linkType": "FILE"
        },
        {
          "uri": "https://arxiv.org/search/advanced",
          "linkType": "WEB"
        },
        {
          "uri": "https://arxiv.org/",
          "linkType": "WEB"
        },
        {
          "uri": "https://arxiv.org/login",
          "linkType": "WEB"
        },
        {
          "uri": "https://info.arxiv.org/about/index.html",
          "linkType": "FILE"
        },
        {
          "uri": "https://arxiv.org/abs/2303.10130v1",
          "linkType": "WEB"
        },
        {
          "uri": "https://arxiv.org/search/econ?searchtype=author&query=Eloundou%2C+T",
          "linkType": "WEB"
        },
        {
          "uri": "https://arxiv.org/search/econ?searchtype=author&query=Manning%2C+S",
          "linkType": "WEB"
        },
        {
          "uri": "https://arxiv.org/search/econ?searchtype=author&query=Mishkin%2C+P",
          "linkType": "WEB"
        },
        {
          "uri": "https://arxiv.org/search/econ?searchtype=author&query=Rock%2C+D",
          "linkType": "WEB"
        },
        {
          "uri": "https://arxiv.org/abs/2303.10130",
          "linkType": "WEB"
        },
        {
          "uri": "https://arxiv.org/abs/2303.10130v5",
          "linkType": "WEB"
        },
        {
          "uri": "https://doi.org/10.48550/arXiv.2303.10130",
          "linkType": "WEB"
        },
        {
          "uri": "http://creativecommons.org/licenses/by-sa/4.0/",
          "linkType": "WEB"
        },
        {
          "uri": "https://ui.adsabs.harvard.edu/abs/arXiv:2303.10130",
          "linkType": "WEB"
        },
        {
          "uri": "https://scholar.google.com/scholar_lookup?arxiv_id=2303.10130",
          "linkType": "WEB"
        },
        {
          "uri": "https://api.semanticscholar.org/arXiv:2303.10130",
          "linkType": "WEB"
        },
        {
          "uri": "https://info.arxiv.org/help/trackback.html",
          "linkType": "FILE"
        },
        {
          "uri": "https://static.arxiv.org/static/browse/0.3.4/css/cite.css",
          "linkType": "WEB"
        },
        {
          "uri": "https://arxiv.org/ct?url=http%3A%2F%2Fwww.bibsonomy.org%2FBibtexHandler%3FrequTask%3Dupload%26url%3Dhttps%3A%2F%2Farxiv.org%2Fabs%2F2303.10130%26description%3DGPTs+are+GPTs%3A+An+Early+Look+at+the+Labor+Market+Impact+Potential+of+Large+Language+Models&v=51542aa8",
          "linkType": "WEB"
        },
        {
          "uri": "https://arxiv.org/ct?url=https%3A%2F%2Freddit.com%2Fsubmit%3Furl%3Dhttps%3A%2F%2Farxiv.org%2Fabs%2F2303.10130%26title%3DGPTs+are+GPTs%3A+An+Early+Look+at+the+Labor+Market+Impact+Potential+of+Large+Language+Models&v=43ad3eb4",
          "linkType": "WEB"
        },
        {
          "uri": "https://info.arxiv.org/labs/showcase.html#arxiv-bibliographic-explorer",
          "linkType": "FILE"
        },
        {
          "uri": "https://www.litmaps.co/",
          "linkType": "WEB"
        },
        {
          "uri": "https://www.scite.ai/",
          "linkType": "WEB"
        },
        {
          "uri": "https://www.catalyzex.com/",
          "linkType": "WEB"
        },
        {
          "uri": "https://dagshub.com/",
          "linkType": "WEB"
        },
        {
          "uri": "https://paperswithcode.com/",
          "linkType": "WEB"
        },
        {
          "uri": "https://sciencecast.org/welcome",
          "linkType": "WEB"
        },
        {
          "uri": "https://replicate.com/docs/arxiv/about",
          "linkType": "WEB"
        },
        {
          "uri": "https://huggingface.co/docs/hub/spaces",
          "linkType": "WEB"
        },
        {
          "uri": "https://influencemap.cmlab.dev/",
          "linkType": "WEB"
        },
        {
          "uri": "https://www.connectedpapers.com/about",
          "linkType": "WEB"
        },
        {
          "uri": "https://core.ac.uk/services/recommender",
          "linkType": "WEB"
        },
        {
          "uri": "https://info.arxiv.org/labs/index.html",
          "linkType": "FILE"
        },
        {
          "uri": "https://info.arxiv.org/help/mathjax.html",
          "linkType": "FILE"
        },
        {
          "uri": "https://info.arxiv.org/help/contact.html",
          "linkType": "FILE"
        },
        {
          "uri": "https://info.arxiv.org/help/subscribe.html",
          "linkType": "FILE"
        },
        {
          "uri": "https://info.arxiv.org/help/license/index.html",
          "linkType": "FILE"
        },
        {
          "uri": "https://info.arxiv.org/help/policies/privacy_policy.html",
          "linkType": "FILE"
        },
        {
          "uri": "https://info.arxiv.org/help/web_accessibility.html",
          "linkType": "FILE"
        },
        {
          "uri": "https://status.arxiv.org/",
          "linkType": "WEB"
        },
        {
          "uri": "https://subscribe.sorryapp.com/24846f03/email/new",
          "linkType": "WEB"
        },
        {
          "uri": "https://subscribe.sorryapp.com/24846f03/slack/new",
          "linkType": "WEB"
        }
      ],
      "mimeType": "text/html",
      "fileType": "DOCUMENT",
      "fileName": "2303.10130.htm",
      "fileSize": 47924,
      "masterUri": "https://graphlit202309044a4fa477.blob.core.windows.net/files/e4899d7c-407f-4532-89ad-cb18a00feb87/2303.10130.htm?sv=2023-01-03&se=2023-09-07T02%3A06%3A22Z&sr=c&sp=rl&sig=yGJcvB%2FkBvuszIiXbPRwDlzXugzU97eiXEQDJT3xQFY%3D",
      "mezzanineUri": "https://graphlit202309044a4fa477.blob.core.windows.net/files/e4899d7c-407f-4532-89ad-cb18a00feb87/Mezzanine/2303.10130.json?sv=2023-01-03&se=2023-09-07T02%3A06%3A22Z&sr=c&sp=rl&sig=yGJcvB%2FkBvuszIiXbPRwDlzXugzU97eiXEQDJT3xQFY%3D",
      "document": {
        "title": "[2303.10130] GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models",
        "description": "We investigate the potential implications of large language models (LLMs),\nsuch as Generative Pre-trained Transformers (GPTs), on the U.S. labor market,\nfocusing on the increased capabilities arising from LLM-powered software\ncompared to LLMs on their own. Using a new rubric, we assess occupations based\non their alignment with LLM capabilities, integrating both human expertise and\nGPT-4 classifications. Our findings reveal that around 80% of the U.S.\nworkforce could have at least 10% of their work tasks affected by the\nintroduction of LLMs, while approximately 19% of workers may see at least 50%\nof their tasks impacted. We do not make predictions about the development or\nadoption timeline of such LLMs. The projected effects span all wage levels,\nwith higher-income jobs potentially facing greater exposure to LLM capabilities\nand LLM-powered software. Significantly, these impacts are not restricted to\nindustries with higher recent productivity growth. Our analysis suggests that,\nwith access to an LLM, about 15% of all worker tasks in the US could be\ncompleted significantly faster at the same level of quality. When incorporating\nsoftware and tooling built on top of LLMs, this share increases to between 47\nand 56% of all tasks. This finding implies that LLM-powered software will have\na substantial effect on scaling the economic impacts of the underlying models.\nWe conclude that LLMs such as GPTs exhibit traits of general-purpose\ntechnologies, indicating that they could have considerable economic, social,\nand policy implications."
      },
      "uri": "https://arxiv.org/abs/2303.10130",
      "id": "e4899d7c-407f-4532-89ad-cb18a00feb87",
      "name": "https://arxiv.org/abs/2303.10130",
      "state": "FINISHED",
      "creationDate": "2023-09-06T20:05:30Z",
      "finishedDate": "2023-09-06T20:05:46Z",
      "workflowDuration": "PT16.3506442S",
      "owner": {
        "id": "530a3721-3273-44b4-bff4-e87218143164"
      }
    }
  ]
}

Queries

Mutations

Objects