Replicate

Configure Replicate specification.

When using Replicate LLMs, you have the choice of using a built-in Graphlit model, which accrues credits for usage, or using your own Replicate API key.

Graphlit currently supports these built-in Replicate models via the model field: MISTRAL_7B_INSTRUCT, MISTRAL_7B, LLAMA_2_7B_CHAT, LLAMA_2_7B, LLAMA_2_13B_CHAT, LLAMA_2_13B, LLAMA_2_70B_CHAT, and LLAMA_2_70B.

Currently, Replicate models can only be used with the promptSpecifications mutation, and are not supported for conversations with promptConversation. The Llama 2 and Mistral models are not yet reliable for returning JSON, which is a requirement for the current Graphlit RAG implementation.

You can find the list of current Replicate models here, and use the versioned model name syntax such as meta/llama-2-7b-chat:13c3cdee13ee059ab779f0291d29054dab00a47dad8261375654de5540165fb0.

By assigning the model field to CUSTOM, you also will need to assign the key and modelName to use your own Replicate developer account.

Create Built-In Model Specification

Mutation:

mutation CreateSpecification($specification: SpecificationInput!) {
  createSpecification(specification: $specification) {
    id
    name
    state
    type
    serviceType
  }
}

Variables:

{
  "specification": {
    "type": "COMPLETION",
    "serviceType": "REPLICATE",
    "replicate": {
      "model": "LLAMA_2_70B_CHAT"
    },
    "name": "Replicate"
  }
}

Response:

{
  "type": "COMPLETION",
  "serviceType": "REPLICATE",
  "id": "a87ce6cd-75f4-4391-8d00-33149e9afc86",
  "name": "Replicate",
  "state": "ENABLED"
}

Create Custom Model Specification

Mutation:

mutation CreateSpecification($specification: SpecificationInput!) {
  createSpecification(specification: $specification) {
    id
    name
    state
    type
    serviceType
  }
}

Variables:

{
  "specification": {
    "type": "COMPLETION",
    "serviceType": "REPLICATE",
    "replicate": {
      "key": "redacted",
      "modelName": "meta/llama-2-7b-chat:13c3cdee13ee059ab779f0291d29054dab00a47dad8261375654de5540165fb0",
      "model": "CUSTOM"
    },
    "name": "Replicate Custom"
  }
}

Response:

{
  "type": "COMPLETION",
  "serviceType": "Replicate",
  "id": "3c7d107c-cc9f-4c2f-8ac5-0b0b7ddd2ddf",
  "name": "Replicate Custom",
  "state": "ENABLED"
}