Feeds

Create, manage and query Feeds.

Overview

In Graphlit, Content can be ingested directly, one-by-one, with the ingestUri and ingestText mutations, but commonly you will want to ingest content from a cloud storage folder, a SharePoint library, RSS feed, or an entire website.

For bulk ingestion, you can create a feed, which will enumerate all content at the specified location, and automatically ingest the content into your Graphlit project.

Feeds can be scheduled as a one-time sweep, or as a recurring sweep, which will continually ingest new content as it gets noticed by the feed. (We call this the scheduling policy for the feed.)

You can use feeds to ingest any type of file, including PDFs, images, videos or audio recordings, which are supported by Graphlit. Or you can use a feed to ingest RSS or Reddit posts, or audio from podcasts.

Each feed type supports a variety of configuration options, such as hyperlink following, limiting the number of ingested items, and scheduling policies.

All the current and planned feed types are listed here.

More information about feed configuration can be found here.

Create Feed:

Disable Feed

The disableFeed mutation disables a feed by utilizing the id parameter, and it returns the ID and state of the disabled feed.

Feed disabling applies to feeds with recurrent schedule policies, and acts as a pause button for the feed, where it will no longer continue to look for new content from the feed.

Mutation:

mutation DisableFeed($id: ID!) {
  disableFeed(id: $id) {
    id
    state
  }
}

Variables:

{
  "id": "fa83bbee-c30e-49d0-87bd-66ce6386829d"
}

Response:

{
  "id": "fa83bbee-c30e-49d0-87bd-66ce6386829d",
  "state": "DISABLED"
}

Enable Feed

The enableFeed mutation enables a feed by utilizing the id parameter, and it returns the ID and state of the enabled feed.

Feed enabling applies to feeds with recurrent schedule policies, and acts as a play button for the feed, where it will start to look for new content from the feed. It will attempt to catch up on ingesting any content it missed while it was paused.

Mutation:

mutation EnableFeed($id: ID!) {
  enableFeed(id: $id) {
    id
    state
  }
}

Variables:

{
  "id": "fa83bbee-c30e-49d0-87bd-66ce6386829d"
}

Response:

{
  "id": "fa83bbee-c30e-49d0-87bd-66ce6386829d",
  "state": "ENABLED"
}

Content ingested from feeds are linked to the source feed to maintain data lineage.

In order to prevent orphaned references from content to feeds, Graphlit does not support deleting feeds without deleting the linked content.

When you delete a feed, the linked content will also be deleted.

For most use cases, you will want to disable a feed - not delete - when done with the feed. You are able to query feeds by their state to return enabled feeds only.

Delete Feed

The deleteFeed mutation allows the deletion of a feed by utilizing the id parameter, and it returns the ID and state of the deleted feed.

This does not delete the contents sourced from the feed, only the feed itself.

Mutation:

mutation DeleteFeed($id: ID!) {
  deleteFeed(id: $id) {
    id
    state
  }
}

Variables:

{
  "id": "266fa32b-98ea-44a9-8f8d-1874ade413f3"
}

Response:

{
  "id": "266fa32b-98ea-44a9-8f8d-1874ade413f3",
  "state": "DELETED"
}

Delete Feeds

If you have multiple feeds you want to delete, you can use the deleteFeeds mutation, and pass an array of IDs for the feeds you wish to delete.

Mutation:

mutation DeleteFeeds($ids: [ID!]!) {
  deleteFeeds(ids: $ids) {
    id
    state
  }
}

Variables:

{
  "ids": [ "f16fd151-be51-4b10-bec0-ceb535bf229d", "decbb9f5-e74a-41b8-9fe3-d31de8818769" ]
}

Response:

[ 
  {
    "id": "f16fd151-be51-4b10-bec0-ceb535bf229d",
    "state": "DELETED"
  },
  {
    "id": "decbb9f5-e74a-41b8-9fe3-d31de8818769",
    "state": "DELETED"
  }
]

Delete All Feeds

While developing and testing your application, you may want to delete all feeds in your project.

You can use the deleteAllFeeds mutation. This does not take any additional variables, and will delete all feeds in the project or tenant (depending on the JWT). The mutation returns an array of deleted feeds.

Mutation:

mutation DeleteAllFeeds($ids: [ID!]!) {
  deleteAllFeeds(ids: $ids) {
    id
    state
  }
}

Response:

[ 
  {
    "id": "f16fd151-be51-4b10-bec0-ceb535bf229d",
    "state": "DELETED"
  },
  {
    "id": "decbb9f5-e74a-41b8-9fe3-d31de8818769",
    "state": "DELETED"
  }
]

Get Feed

The feed query allows you to retrieve specific details of a feed by providing the id parameter, including the ID, name, state, owner ID, and type associated with the feed.

Query:

query GetFeed($id: ID!) {
  feed(id: $id) {
    id
    name
    creationDate
    state
    owner {
      id
    }
    type
    reddit {
      allowLinkEnrichment
      allowedDomains
      allowedLinks
      excludedLinks
      allowedFiles
      excludedFiles
      listingLimit
      durationLimit
      subredditName
    }
    lastPostDate
    lastReadDate
    readCount
    schedulePolicy {
      recurrenceType
      repeatInterval
    }
  }
}

Variables:

{
  "id": "fa83bbee-c30e-49d0-87bd-66ce6386829d"
}

Response:

{
  "type": "REDDIT",
  "reddit": {
    "subredditName": "OpenAI"
  },
  "lastPostDate": "2023-06-02T13:15:44Z",
  "lastReadDate": "2023-07-07T04:15:58Z",
  "readCount": 332,
  "schedulePolicy": {
    "recurrenceType": "REPEAT",
    "repeatInterval": "PT30S"
  },
  "id": "fa83bbee-c30e-49d0-87bd-66ce6386829d",
  "name": "OpenAI Reddit",
  "state": "RUNNING",
  "creationDate": "2023-07-07T04:04:24Z",
  "owner": {
    "id": "9422b73d-f8d6-4faf-b7a9-152250c862a4"
  }
}

Query Feeds

Query Feeds

The feeds query allows you to retrieve all feeds. It returns a list of feed results, including the ID, name, creation date, state, owner ID, and type for each feed.

Query:

query QueryFeeds($filter: FeedFilter!) {
  feeds(filter: $filter) {
    results {
      id
      name
      creationDate
      state
      owner {
        id
      }
      type
      reddit {
        subredditName
      }
      lastPostDate
      lastReadDate
      readCount
      schedulePolicy {
        recurrenceType
        repeatInterval
      }
    }
  }
}

Variables:

{
  "filter": {
    "offset": 0,
    "limit": 100
  }
}

Response:

{
  "results": [
    {
      "type": "REDDIT",
      "reddit": {
        "subredditName": "OpenAI"
      },
      "lastPostDate": "2023-05-31T22:11:12Z",
      "lastReadDate": "2023-07-07T04:19:38Z",
      "readCount": 427,
      "schedulePolicy": {
        "recurrenceType": "REPEAT",
        "repeatInterval": "PT30S"
      },
      "id": "fa83bbee-c30e-49d0-87bd-66ce6386829d",
      "name": "OpenAI Reddit",
      "state": "RUNNING",
      "creationDate": "2023-07-07T04:04:24Z",
      "owner": {
        "id": "9422b73d-f8d6-4faf-b7a9-152250c862a4"
      }
    }
  ]
}

Query Feeds By Name

The feeds query allows you to retrieve feeds based on a specific filter criteria, via the name parameter. In this example, the name is set to "RSS." It returns a list of feed results containing the ID, name, creation date, state, owner ID, and type for each matching feed.

Query:

query QueryFeeds($filter: FeedFilter!) {
  feeds(filter: $filter) {
    results {
      id
      name
      creationDate
      state
      owner {
        id
      }
      type
      reddit {
        subredditName
      }
      lastPostDate
      lastReadDate
      readCount
      schedulePolicy {
        recurrenceType
        repeatInterval
      }
    }
  }
}

Variables:

{
  "filter": {
    "name": "OpenAI",
    "offset": 0,
    "limit": 100
  }
}

Response:

{
  "results": [
    {
      "type": "REDDIT",
      "reddit": {
        "subredditName": "OpenAI"
      },
      "lastPostDate": "2023-05-31T14:56:00Z",
      "lastReadDate": "2023-07-07T04:20:23Z",
      "readCount": 441,
      "schedulePolicy": {
        "recurrenceType": "REPEAT",
        "repeatInterval": "PT30S"
      },
      "id": "fa83bbee-c30e-49d0-87bd-66ce6386829d",
      "name": "OpenAI Reddit",
      "state": "RUNNING",
      "creationDate": "2023-07-07T04:04:24Z",
      "owner": {
        "id": "9422b73d-f8d6-4faf-b7a9-152250c862a4"
      }
    }
  ]
}

Query Feeds By Type

The feeds query allows you to retrieve feeds based on a specific filter criteria, via the types parameter. In this example, the type is set to REDDIT. It returns a list of feed results containing the ID, name, creation date, state, owner ID, and type for each matching feed.

Query:

query QueryFeeds($filter: FeedFilter!) {
  feeds(filter: $filter) {
    results {
      id
      name
      creationDate
      state
      owner {
        id
      }
      type
      reddit {
        subredditName
      }
      lastPostDate
      lastReadDate
      readCount
      schedulePolicy {
        recurrenceType
        repeatInterval
      }
    }
  }
}

Variables:

{
  "filter": {
    "types": [
      "REDDIT"
    ],
    "offset": 0,
    "limit": 100
  }
}

Response:

{
  "results": [
    {
      "type": "REDDIT",
      "reddit": {
        "subredditName": "OpenAI"
      },
      "lastPostDate": "1970-01-01T00:00:00Z",
      "lastReadDate": "2023-07-07T05:34:53Z",
      "readCount": 925,
      "schedulePolicy": {
        "recurrenceType": "REPEAT",
        "repeatInterval": "PT30S"
      },
      "id": "fa83bbee-c30e-49d0-87bd-66ce6386829d",
      "name": "OpenAI Reddit",
      "state": "RUNNING",
      "creationDate": "2023-07-07T04:04:24Z",
      "owner": {
        "id": "9422b73d-f8d6-4faf-b7a9-152250c862a4"
      }
    }
  ]
}

Last updated 15 days ago