# What is Graphlit? **The context layer for AI agents** Give your AI agents the context they need to work reliably. One API for ingestion, extraction, storage, and retrieval - organizational knowledge with entities, relationships, and temporal state. ## What is Graphlit? Graphlit is the **context layer for AI agents** - providing organizational knowledge infrastructure, data ingestion from 30+ sources, and intelligent retrieval. Whether you're building with Mastra, Agno, Vercel AI SDK, or custom code, Graphlit handles the hard parts so you can focus on your agent's logic and UX. {% hint style="info" %} **šŸ”Œ Works with your framework**: Use Graphlit's [MCP server](/mcp-integration/mcp-integration) to give any MCP-enabled framework instant access to 30+ feeds, audio/video processing, semantic search, and knowledge graphs. Or use our TypeScript/Python/C# SDKs directly. {% endhint %} **Context layer vs traditional RAG:** | Feature | Traditional RAG | Graphlit Context Layer | | ------------------- | ------------------------------------ | --------------------------------------------- | | **Memory** | Stateless (forgets between sessions) | Persistent organizational memory | | **Understanding** | Text chunk retrieval | Entities + relationships + temporal state | | **Recall** | Keyword/similarity matching | Graph traversal + semantic search | | **Knowledge** | Document vectors only | Knowledge graph + vectors | | **Infrastructure** | 7+ services to integrate | Complete platform (one API) + MCP integration | | **Processing** | Manual pipeline setup | Automatic extraction workflows | | **Personalization** | None (treats all users the same) | Per-user knowledge graphs | | **Citations** | Basic text snippets | Entity-linked with provenance | Think of it this way: RAG is like searching through filing cabinets. A context layer is like having a knowledgeable assistant who understands your organization. ## Why Developers Choose Graphlit ### The Problem Building data infrastructure for AI agents means integrating: * Vector database (Pinecone, Weaviate) * Document parsers (Unstructured, LlamaParse) * Entity extraction (spaCy, custom LLMs) * Embedding models (OpenAI, Cohere) * Storage (S3, Azure Blob) * Search (Elasticsearch) * OAuth connectors for data sources (Slack, Gmail, etc.) * Sync infrastructure (polling, webhooks, rate limits) **Result**: 3-20 months of integration work before building your actual agent application. ### The Graphlit Solution ```typescript import { Graphlit } from 'graphlit-client'; async function main() { const graphlit = new Graphlit(); // Ingest a document const content = await graphlit.ingestUri( 'https://arxiv.org/pdf/1706.03762.pdf', 'Attention Paper', undefined, undefined, true // Wait for processing ); // Ask questions about it const conversation = await graphlit.createConversation({ name: 'Q&A Session', filter: { contents: [{ id: content.ingestUri.id }] } }); const answer = await graphlit.promptConversation( 'What are the key innovations?', conversation.createConversation.id ); console.log(answer.promptConversation.message?.message); } main(); ``` **One API. No assembly required.** *** ## Complete Platform Features Graphlit provides everything you need to build production AI applications - from data ingestion to advanced processing: | Capability | Memory-Only Platforms | Graphlit | | ----------------------- | ------------------------------------- | --------------------------------------------------------------------------------------- | | **Data Feeds** | Manual ingestion | **30+ feeds** (Slack, Gmail, GitHub, S3, RSS, etc.) - OAuth, API keys, or public | | **Automatic Sync** | Manual upload | **Continuous polling** (30 sec to hours, configurable per feed) | | **Audio Processing** | Basic or not available | **Transcription + speaker diarization** (Speaker #1, #2, etc.) via Deepgram, AssemblyAI | | **Video Processing** | Not available | **Audio extraction + transcription** (available) + frame analysis (coming soon) | | **Document Processing** | Text extraction | **Vision OCR + layout preservation** (handles complex tables, diagrams) | | **Web Capabilities** | Not available | **Web crawling, screenshots, search integration** (Tavily, Exa) | | **Workflows** | Fixed pipeline | **Customizable multi-stage pipelines** (preparation + extraction stages) | | **Publishing** | Retrieval only | **Audio generation, summaries, Markdown export** (TTS, content transformation) | | **Knowledge Graph** | Vectors only (some have basic graphs) | **Schema.org entities + relationships** with temporal context | | **Search Types** | Vector similarity | **Hybrid: vector + graph + keyword** | | **Advanced Filtering** | Basic metadata filters | **Geo-spatial, image similarity, entity-based, temporal, boolean (AND/OR)** | | **Production Features** | Basic user scoping | **Per-user isolation (userId parameter), collections, specifications** | **Real example:** Building a Slack assistant with Graphlit vs memory-only platforms: **With Graphlit:** ```typescript // 1. Setup OAuth connector (one-time) const feed = await graphlit.createFeed({ name: 'Team Slack', type: FeedTypes.Slack, slack: { type: FeedListingTypes.Past } }); // āœ… All messages automatically synced, indexed, and searchable ``` **With Memory-Only Platform:** ```typescript // 1. Build Slack OAuth integration yourself // 2. Poll Slack API yourself // 3. Handle rate limits yourself // 4. Parse messages yourself // 5. Upload to memory platform // 6. Repeat for every data source // āŒ Weeks of integration work per connector ``` **The difference:** Graphlit provides a complete platform - from data ingestion through processing to retrieval - so you can focus on building your application. *** ## What Can You Build? ### AI Agents with Memory Customer support agents that remember every interaction and have full context from past conversations.\ ā†’ [Build an agent in 7 minutes](/getting-started/quickstart) ### Production SaaS Applications [Zine](https://zine.ai) runs in production on Graphlit with growing user base and multi-source data sync.\ ā†’ [See the architecture](/examples/zine-case-study) ### Knowledge Extraction Systems Automatically extract people, organizations, and relationships from any content.\ ā†’ [Extract knowledge graphs](/tutorials/knowledge-graph) ## Quick Start (TypeScript) {% hint style="info" %} **SDK availability**: Python, TypeScript, and .NET SDKs available. All quickstart examples use TypeScript. Click "Convert to Python/ā€‹.NET" links throughout for instant language conversion via [Ask Graphlit](/resources/ask-graphlit). {% endhint %} {% hint style="success" %} **Launch checklist:** 1. [Sign up](/account-setup-one-time/signup) (30 seconds) 2. [Create project](/account-setup-one-time/create-project) (1 minute) 3. [Get credentials](/account-setup-one-time/credentials) (1 minute) 4. āœ… **Verify setup**: Run `hello.ts` (Step 1 below) 5. [Quickstart: Your First Agent](/getting-started/quickstart) (7 minutes) {% endhint %} **Key terms you'll use:** * **Content** ā€“ anything you've ingested (files, web pages, emails) * **Conversation** ā€“ AI session that remembers prior messages and retrieved context * **Specification** ā€“ which LLM + settings to use (model, temperature, etc.) ### 1. Say Hello to Graphlit Verify your credentials work: ```typescript import { Graphlit } from 'graphlit-client'; const graphlit = new Graphlit(); async function main() { const project = await graphlit.getProject(); console.log(`āœ… Connected: ${project.project.name}`); } main(); ``` Run: `npx tsx hello.ts` ### 2. Ingest & Search ```typescript import { Graphlit } from 'graphlit-client'; import { SearchTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); async function main() { // Ingest document const content = await graphlit.ingestUri( 'https://arxiv.org/pdf/1706.03762.pdf', 'Attention Paper', undefined, undefined, true, // Wait for processing ); console.log(`āœ… Document ready: ${content.ingestUri.id}`); // Hybrid search (vector + keyword) const results = await graphlit.queryContents({ search: 'transformer innovations', searchType: SearchTypes.Hybrid, }); console.log(`Found ${results.contents?.results?.length ?? 0} documents`); } main(); ``` ### 3. RAG Conversation ```typescript import { Graphlit } from 'graphlit-client'; const graphlit = new Graphlit(); async function main() { const content = await graphlit.ingestUri( 'https://arxiv.org/pdf/1706.03762.pdf', 'Attention Paper', undefined, undefined, true, ); const conversation = await graphlit.createConversation({ name: 'Q&A Session', filter: { contents: [{ id: content.ingestUri.id }] }, }); const answer = await graphlit.promptConversation( 'What are the key innovations?', conversation.createConversation.id, ); console.log(answer.promptConversation.message?.message); } main(); ``` {% hint style="success" %} **Ready for more?** ā†’ [Quickstart: Add streaming and tool calling](/getting-started/quickstart) {% endhint %} ## Production Ready Graphlit handles production scale out of the box: * **Multi-tenant**: Per-user data isolation within a single project (create users in Graphlit, scope SDK with userId) * **Scale**: Built to handle thousands of users and millions of documents per project * **Automatic sync**: Feed connectors poll on configurable schedules (30 seconds to hours) * **Proof**: [Zine](https://zine.ai) runs on Graphlit in production [Zine case study ā†’](/examples/zine-case-study) ## Connect Your Data 30+ feeds (Slack, Gmail, GitHub, S3, RSS, and more) with automatic sync - [view all ā†’](/platform/feeds) ## Next Steps **šŸš€ Start here**: [Quickstart: Your First Agent](/getting-started/quickstart) (7 minutes)\ Build a streaming agent with tool calling. Fastest way to see Graphlit in action. **Then explore**: * [AI Agents Tutorial](/tutorials/ai-agents) - Multi-agent patterns (15 min) * [Knowledge Graph](/tutorials/knowledge-graph) - Entity extraction (20 min) **Need help?** * [Discord Community](https://discord.gg/ygFmfjy3Qx) - Active developer community * [Ask Graphlit](https://ask.graphlit.dev) - AI code assistant * [60+ Examples](https://github.com/graphlit/graphlit-samples) - Working code *** Built by [Graphlit](https://www.graphlit.com) ā€¢ [Sign up free](https://portal.graphlit.dev) # Why Graphlit? Why developers choose Graphlit over DIY solutions and other memory platforms - TCO comparison, time savings, and competitive advantages **Graphlit is the data infrastructure layer for AI agents.** Whether you're building with Mastra, Agno, Vercel AI SDK, or custom code, Graphlit handles the hard parts: ingesting data from 30+ sources, processing audio/video, storing semantic memory, and providing retrieval - so you can focus on your agent's logic and UX. Building this infrastructure yourself means integrating 7+ services, maintaining sync pipelines, and staying current with AI models. Graphlit provides everything in one platform - and integrates with your existing agent frameworks via MCP. ## The Hidden Cost of DIY Most developers underestimate what it takes to build production-grade semantic memory. Here's what you're really signing up for: ### DIY Stack Requirements **Infrastructure Services** (you integrate and maintain): * Vector database (Pinecone, Weaviate, Qdrant) - $70-200/month * Document parser (Unstructured, LlamaParse) - $99-299/month * Audio transcription (Deepgram, AssemblyAI) - $50-200/month * Entity extraction (spaCy, custom NLP) - build yourself * Object storage (S3, Azure Blob) - $30-100/month * Search index (Elasticsearch) - $95-500/month * Embedding service (OpenAI, Cohere) - $20-100/month * Orchestration (LangChain, custom code) - maintain yourself **Development Time** (before you ship): *Note: Times shown are person-weeks of effort. With multiple developers or AI coding assistants (Cursor, Windsurf, etc.), calendar time can be shorter - but the complexity and coordination overhead remains.* *Basic RAG* (file upload + vector search): * Vector DB integration: 1-2 weeks * Document parsing pipeline: 2-3 weeks * Embedding generation: 1-2 weeks * Search API: 1-2 weeks * **Subtotal: 5-9 person-weeks (2-3 months calendar time with team)** *Production-Ready RAG* (multi-tenant, observability): * Multi-tenant architecture: 3-4 weeks * Logging & observability: 2-3 weeks * Security & encryption: 2-3 weeks * Usage tracking: 1-2 weeks * **Subtotal: 13-21 person-weeks (3-5 months calendar time with team)** *Graphlit-Equivalent* (30+ feeds, audio/video, workflows): * OAuth connectors (30+ sources @ 1-2 weeks each): 30-60 weeks * Automatic sync infrastructure: 3-4 weeks * Audio transcription + diarization: 2-3 weeks * Video processing: 3-4 weeks * Custom workflows engine: 4-6 weeks * Knowledge graph: 4-6 weeks * Publishing features: 2-3 weeks * **Subtotal: 48-86 person-weeks (12-20 months calendar time with team)** **Realistic Timeline** (with 2-person team + AI coding tools): * Basic RAG: **2-3 months** * Production-ready: **3-5 months** * Graphlit-equivalent: **12-20 months** (if you even attempt it) **Ongoing Maintenance** (every month): * Update dependencies: 4-8 hours * Tune vector search: 2-4 hours * Monitor performance: 4-8 hours * Debug data pipeline issues: 4-12 hours * Update to new models: 8-16 hours * Scale infrastructure: 4-8 hours * **Total: 26-48 hours/month** **Total First Year Cost** (Production-Ready RAG): * Infrastructure: **$8,000 - $20,000** * Development time (4 months @ $150/hr, 2 engineers): **$96,000** * Ongoing maintenance (35 hrs/month @ $150/hr): **$63,000** * **Grand Total: $167,000 - $179,000** **Total First Year Cost** (Graphlit-Equivalent): * You wouldn't do this. It would take 12-20 months and cost **$400,000+**. ### Graphlit Approach **One Platform**: ```typescript // Everything you need in one API const graphlit = new Graphlit(); // 5 minutes to production-ready semantic memory const feed = await graphlit.createFeed({ /* 30+ sources */ }); const content = await graphlit.ingestUri(/* audio, video, docs */); const results = await graphlit.queryContents({ /* hybrid search */ }); ``` **Actual Cost**: * Platform: **$0.10-0.08/credit** (volume discounts available) * Development time: **1 day to MVP** * Maintenance: **Zero** (we handle it) * Model updates: **Automatic** (GPT-5, Claude 4.5, etc.) **Savings vs Production-Ready RAG**: **$160,000+ in Year 1**\ **Savings vs Graphlit-Equivalent**: **$400,000+** (and 12-20 months faster to market) *** ## Time to Value Comparison ### Building a Slack Search Assistant **With Graphlit** (5 minutes): ```typescript // Step 1: Connect Slack (30 seconds) const feed = await graphlit.createFeed({ name: 'Team Slack', type: FeedTypes.Slack, slack: { type: FeedListingTypes.Past } }); const feedId = feed.createFeed.id; // āœ… OAuth automatic, messages syncing // Step 2: Search (30 seconds) const results = await graphlit.queryContents({ search: 'Q4 roadmap decisions', feeds: [{ id: feedId }] }); // āœ… Hybrid search across all messages ``` **DIY Stack** (2-3 weeks): * Week 1: Build Slack OAuth flow, handle token refresh * Week 1-2: Build polling infrastructure (handle rate limits, pagination) * Week 2: Parse messages, store in database * Week 2: Generate embeddings, index in vector DB * Week 3: Build search API, tune relevance * Week 3: Handle edge cases (threads, reactions, files) **Graphlit advantage**: **2-3 weeks saved**, production-ready from line 1 *** ### Audio Transcription with Speaker Identification **With Graphlit** (1 API call): ```typescript const audio = await graphlit.ingestUri( 'https://example.com/meeting.mp3', 'Team Meeting', workflow.createWorkflow.id, // Includes transcription + diarization undefined, true ); // āœ… Speaker #1, #2, #3 identified // āœ… Fully searchable transcript ``` **DIY Stack** (1 week): * Integrate Deepgram or AssemblyAI SDK * Handle audio format conversion * Implement diarization * Store and index transcripts * Build search interface **Graphlit advantage**: **1 week saved** per audio feature *** ### Multi-Source Search (Slack + Gmail + Google Drive) **With Graphlit** (10 minutes): ```typescript // Connect all sources const slackFeed = await graphlit.createFeed({ type: FeedTypes.Slack, ... }); const gmailFeed = await graphlit.createFeed({ type: FeedTypes.Email, ... }); // Google Drive is a Site feed (service type = GoogleDrive) const driveFeed = await graphlit.createFeed({ type: FeedTypes.Site, ... }); const slackFeedId = slackFeed.createFeed.id; const gmailFeedId = gmailFeed.createFeed.id; const driveFeedId = driveFeed.createFeed.id; // Search across all sources const results = await graphlit.queryContents({ search: 'Q4 budget approval', feeds: [ { id: slackFeedId }, { id: gmailFeedId }, { id: driveFeedId } ] }); // āœ… Unified search across 3 sources ``` **DIY Stack** (3-4 weeks): * Build OAuth for 3 services (1 week each) * Unify data schemas (1 week) * Build cross-source search (1 week) * Handle sync for all 3 (ongoing) **Graphlit advantage**: **3-4 weeks saved** *** ## What You Don't Have to Manage The "Zero Ops" advantage - here's what Graphlit handles so you don't have to: ### Infrastructure Management āŒ ``` You DON'T manage: āŒ Vector database configuration (indexes, sharding, replication) āŒ Embedding model selection (we benchmark and choose best) āŒ Chunking strategy optimization (we've tested 20+ approaches) āŒ Storage scaling (automatic as you grow) āŒ Search performance tuning (sub-second queries at scale) āŒ Backup and disaster recovery āŒ Security patches and updates āŒ Monitoring and alerting infrastructure ``` ### Staying Current with AI āŒ ``` You DON'T track: āŒ New LLM releases (GPT-5, Claude 4.5, Gemini 2.5) āŒ Better embedding models (we test and switch) āŒ Improved transcription services (Deepgram v4, etc.) āŒ New vision models (GPT-4V updates) āŒ Prompt engineering best practices āŒ Token optimization techniques ``` **With Graphlit**: Call the same API. Get the latest models automatically. Your agent framework (Mastra, Agno, etc.) just calls Graphlit via MCP - no updates needed. ```typescript // Today: Uses GPT-4 Turbo // Next month: Automatically uses GPT-5 (zero code changes) const conversation = await graphlit.createConversation({ name: 'Q&A' }); ``` ### Data Pipeline Maintenance āŒ ``` You DON'T build: āŒ OAuth connector for each service (30+ services = 30+ integrations) āŒ Polling infrastructure (rate limits, retries, exponential backoff) āŒ Data transformation (PDFs, audio, video, emails, Slack threads) āŒ Deduplication logic (content hashing, similarity detection) āŒ Error handling and retry logic āŒ Monitoring dashboards ``` *** ## Graphlit vs Memory-Only Platforms Platforms like **Mem0** and **Zep** provide memory storage but require YOU to build everything else. ### What They Provide * āœ… Vector storage * āœ… Memory retrieval APIs * āœ… (Zep) Temporal knowledge graph * āœ… (Mem0) Open-source flexibility ### What YOU Have to Build * āŒ **All data connectors** (Slack, Gmail, Google Drive, etc.) * āŒ **Automatic sync** infrastructure * āŒ **Audio transcription** pipeline * āŒ **Video processing** pipeline * āŒ **Document parsing** (PDFs, Word, etc.) * āŒ **OAuth flows** for every service * āŒ **Multi-format handling** (audio, video, images) * āŒ **Publishing capabilities** (audio generation, summaries) * āŒ **Content intelligence alerts** (notify on specific content) ### Example: Building Slack Search **With Mem0/Zep**: ```typescript // YOU build all of this (2-3 weeks): // 1. Slack OAuth integration const slackToken = await buildOAuthFlow(); // 3-5 days // 2. Polling infrastructure const poller = new SlackPoller(slackToken); // 2-3 days poller.onMessage(async (message) => { // 3. Parse and transform const parsed = parseSlackMessage(message); // 2 days // 4. Generate embeddings const embeddings = await openai.embeddings.create({ /* */ }); // 1 day // 5. Store in memory platform await mem0.add(parsed, embeddings); // 1 day }); // 6. Handle rate limits, retries, errors (ongoing) ``` **With Graphlit**: ```typescript // 5 minutes: const feed = await graphlit.createFeed({ type: FeedTypes.Slack, slack: { type: FeedListingTypes.Past } }); // Done. Everything else automatic. ``` **Verdict**: Mem0/Zep are excellent memory storage engines. Graphlit is a complete platform. If you're building production apps, you need the complete platform. *** ## Graphlit vs Limited Integration Platforms Platforms like **Supermemory** and **Hyperspell** have some data connectors but limited scope. ### Supermemory (3 OAuth Connectors) **What They Have**: * Google Drive, Notion, OneDrive connectors * Hybrid search (vector + keyword) * Knowledge graph **What They DON'T Have**: * āŒ **Only 3 connectors** (vs Graphlit's 30+) * āŒ **No Slack, Gmail, GitHub, Linear, Jira** (you build these) * āŒ **Claims audio support but rejects MP3 files** (tested) * āŒ **No video transcription** * āŒ **No audio transcription with diarization** * āŒ **No publishing** (audio generation, summaries, exports) * āŒ **No custom workflows** with vision models * āŒ **No content intelligence alerts** **Example**: Want to search your Slack + Gmail? * Supermemory: Build Slack OAuth yourself (1-2 weeks), build Gmail OAuth yourself (1-2 weeks) * Graphlit: 10 minutes for both ### Hyperspell (Similar Limitations) **What They Have**: * Slack, Gmail, Google Drive, Notion, Calendar connectors * Focus on privacy and compliance (SOC 2, GDPR) **What They DON'T Have**: * āŒ **Basic connectors only** (not OAuth feeds with auto-sync) * āŒ **No audio transcription** * āŒ **No video processing** * āŒ **No custom workflows** * āŒ **No publishing capabilities** * āŒ **Fixed pipeline** (can't customize extraction) **Verdict**: Supermemory and Hyperspell are great for basic document/message search. If you need audio, video, custom workflows, or 30+ data sources, you need Graphlit. *** ## Production-Ready from Day 1 Graphlit isn't just a memory layer - it's a production platform with enterprise features built-in. ### Multi-Tenant Architecture āœ… ```typescript // Day 1: Per-user data isolation const user = await graphlit.createUser({ identifier: 'user_123' }); const userId = user.createUser.id; // Scope all operations to this user const scopedGraphlit = new Graphlit({ userId }); // User A never sees User B's data const userContent = await scopedGraphlit.queryContents({ /* */ }); ``` **With competitors**: You build multi-tenancy yourself (2-4 weeks) ### Content Intelligence Alerts āœ… ```typescript // Day 1: Get notified when specific content arrives const alert = await graphlit.createAlert({ name: 'High-priority mentions', filter: { observations: [{ observable: { name: 'urgent' }}] }, integration: { type: IntegrationServiceTypes.Slack } }); // Sends Slack message when content with 'urgent' entity is ingested ``` **With competitors**: You build content filtering + webhook infrastructure (1-2 weeks) ### Usage Tracking & Billing āœ… ```typescript // Day 1: Track customer usage const correlationId = 'tenant_123'; // Your tenant correlation ID (optional) const usage = await graphlit.lookupProjectUsage( correlationId, undefined, // startDate undefined, // duration ); // Bill customers based on actual usage const credits = (usage.lookupUsage ?? []).reduce( (sum, record) => sum + Number(record?.credits ?? 0), 0, ); ``` **With competitors**: You build metering infrastructure (1-2 weeks) *** ## The Zine Proof Point Graphlit isn't just a platform - it's battle-tested in production. [**Zine**](https://zine.ai) is a production SaaS built on Graphlit: * Thousands of active users * 20+ OAuth data sources (Slack, Gmail, Calendar, Notion, Linear, etc.) * Millions of documents indexed * Real-time semantic search across all sources * Multi-tenant architecture with per-user isolation * Zero downtime since launch **Why this matters**: We built Graphlit to power our own SaaS. Every feature exists because we needed it in production. Every optimization exists because we felt the pain. **You get**: Production-proven infrastructure, not a research project. *** ## Developer Velocity at Scale As your application grows, Graphlit's advantages compound: ### Adding New Data Sources **Traditional approach** (1-2 weeks per source): * Research API documentation * Build OAuth integration * Handle rate limits and pagination * Parse and transform data * Store and index * Monitor and maintain **Graphlit approach** (5 minutes per source): ```typescript const jiraFeed = await graphlit.createFeed({ type: FeedTypes.Issue, ... }); const githubFeed = await graphlit.createFeed({ type: FeedTypes.Site, ... }); const notionFeed = await graphlit.createFeed({ type: FeedTypes.Notion, ... }); ``` **10 data sources**: * Traditional: 10-20 weeks * Graphlit: 50 minutes ### Updating to New Models **Traditional approach** (1-2 days): * Research new model (GPT-5, Claude 4.5) * Update code and parameters * Re-generate embeddings for existing content * Test and validate results * Deploy and monitor **Graphlit approach** (automatic): ```typescript // No code changes needed // New models available automatically // Existing content re-indexed transparently ``` ### Scaling to Production **Traditional approach** (2-4 weeks): * Set up observability (Datadog, New Relic) * Implement rate limiting * Add caching layer * Optimize database queries * Set up infrastructure alerting * Load testing and tuning **Graphlit approach** (built-in): * Automatic scaling * Sub-second queries at any scale * Usage dashboard included * Content intelligence alerts available * Battle-tested at Zine scale *** ## The Bottom Line ### Choose Graphlit If You Want: āœ… **Data infrastructure for your agents** - Works with Mastra, Agno, Vercel AI SDK (via MCP)\ āœ… **Ship fast** - Days to production, not months\ āœ… **Stay current** - Automatic model updates\ āœ… **Zero ops** - No infrastructure to manage\ āœ… **Production-ready** - Multi-tenant, content alerts, encryption built-in\ āœ… **Comprehensive** - 30+ feeds, audio/video, publishing\ āœ… **Proven** - Battle-tested at Zine's scale\ āœ… **Predictable costs** - Pay only for usage ### Build Your Own Data Layer If You Want: * To integrate 7+ services yourself (vector DB, storage, transcription, etc.) * To build OAuth connectors for 30+ data sources * To maintain sync infrastructure and data pipelines * To spend 3-20 months before shipping * To manage embedding models, scaling, and operations *** ## Start Building Today ```typescript npm install graphlit-client ``` **5-minute quickstart**: [Your First Agent](/getting-started/quickstart)\ **30+ data sources**: [Feeds](/platform/feeds)\ **Live help**: [Discord Community](https://discord.gg/ygFmfjy3Qx) *** ## Frequently Asked Questions **Q: What if I need on-premises deployment?**\ A: Graphlit is cloud-native by design (like Vercel, Netlify). We're exploring private Azure deployments for enterprise customers. This architecture enables automatic updates, zero maintenance, and superior reliability. **Q: Can I use my own vector database?**\ A: Graphlit manages vector storage internally for optimal performance. This "opinionated" approach means you get battle-tested configurations without research/tuning. We've benchmarked 12+ vector DBs - you get the best one automatically. **Q: How does pricing compare to building myself?**\ A: Starting at $0.10/credit (volume discounts to $0.08/credit, all-inclusive), you save $160,000+ in Year 1 building even basic production-ready RAG yourself. Building Graphlit-equivalent features would cost $400,000+ and take 12-20 months. See realistic TCO comparison above. **Q: What about data privacy and security?**\ A: Encryption at rest and in transit, multi-tenant isolation, SOC 2 compliance in progress. For sensitive workloads, we're exploring private Azure deployments where data stays in your tenant. **Q: Can I customize workflows and extraction?**\ A: Yes! Graphlit supports custom workflows with preparation stages (vision OCR) and extraction stages (entity extraction, summarization). You choose vision models (GPT-4V, Claude Vision, Gemini) and configure extraction rules. **Q: How do you compare to AI frameworks like Vercel AI SDK, Mastra, or Agno?**\ A: These are excellent frameworks (Vercel AI SDK for UI integration, Mastra for TypeScript agents, Agno for Python agents) - and you can use Graphlit WITH them! Via our MCP server, frameworks with MCP support can access Graphlit's 30+ feeds, audio/video processing, and semantic search as tools. The difference: frameworks are code libraries where you manage infrastructure; Graphlit is a managed platform handling data ingestion, sync, storage, and scaling. Use frameworks for custom UI/workflows, use Graphlit for the entire data pipeline - or combine both via MCP integration. *** *Last updated: January 2025* # Coding with AI Agents Use AI coding agents to work with Graphlit's documentation and SDK Graphlit provides AI-readable documentation through MCP integration, giving AI coding tools direct access to our docs, SDK references, and code examples: * **Agentic IDEs**: Cursor, Windsurf * **VS Code Extensions**: Cline * **CLI Tools**: Claude Code, Factory Droid, OpenAI Codex All can access Graphlit's documentation and SDK through our MCP servers. ## MCP Servers for Coding Assistance Graphlit provides two MCP servers to help AI agents assist you with coding: ### 1. Graphlit Documentation MCP Server Read-only access to Graphlit's documentation for understanding concepts and finding code examples. **Server details:** * **URL**: `https://docs.graphlit.dev/~gitbook/mcp` * **Type**: Documentation search * **Capabilities**: Search concepts, tutorials, API references, code examples ### 2. Ask Graphlit MCP Server AI-powered SDK code generation for Python, TypeScript, and .NET. **Server details:** * **URL**: `https://ask.graphlit.dev/mcp` * **Type**: Code generation * **Capabilities**: Generate working SDK code, translate between languages, suggest best practices ### Setting up MCP Servers #### Claude Desktop Add both servers to your Claude Desktop configuration (`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS): ```json { "mcpServers": { "graphlit-docs": { "url": "https://docs.graphlit.dev/~gitbook/mcp" }, "ask-graphlit": { "url": "https://ask.graphlit.dev/mcp" } } } ``` #### Cursor 1. Open Cursor Settings (`Cmd+,` on macOS, `Ctrl+,` on Windows/Linux) 2. Navigate to **Features** ā†’ **MCP Servers** 3. Click **Add MCP Server** twice to add both servers: * **Documentation Server**: * Name: Graphlit Documentation * URL: `https://docs.graphlit.dev/~gitbook/mcp` * **Code Generation Server**: * Name: Ask Graphlit * URL: `https://ask.graphlit.dev/mcp` 4. Click **Save** #### Windsurf / Cline / Other Editors Most MCP-compatible editors support URL-based MCP servers. Add both URLs above to your editor's MCP configuration. ### Using the MCP Servers #### Documentation Server Once configured, AI coding agents can automatically: * Search Graphlit concepts and features * Find SDK code examples (Python, TypeScript, .NET) * Access API references and use case guides * Retrieve troubleshooting information * Get production patterns from case studies **Example prompts**: * "How do I ingest a PDF with Graphlit?" * "Show me Graphlit's entity extraction workflow code" * "What are the differences between Graphlit's conversation types?" * "How does Graphlit handle multi-tenant applications?" #### Ask Graphlit Code Generation Server AI coding agents can generate SDK code in your preferred language: * Generate working code snippets (Python, TypeScript, .NET) * Translate examples between languages * Suggest best practices and patterns * Debug Graphlit integration code * Provide optimization recommendations **Example prompts**: * "Generate TypeScript code to ingest a PDF and extract entities" * "Convert this Python Graphlit code to .NET" * "Show me best practice for multi-tenant conversation setup" * "How do I optimize this Graphlit query for performance?" *** ## llms.txt Graphlit publishes standardized `llms.txt` files containing essential information optimized for AI coding assistants: * Core concepts and architecture * SDK usage patterns and examples * API operation summaries * Best practices and common patterns * Production deployment guidance ### Accessing llms.txt AI assistants can access Graphlit's llms.txt at: ``` https://docs.graphlit.dev/llms.txt ``` This file is automatically generated by GitBook and contains curated, AI-optimized content from our documentation. *** ## Ask Graphlit - Web App & Chatbot For generating Graphlit code without IDE setup, use the **Ask Graphlit** web app: **URL**: Ask Graphlit is an AI chatbot trained on Graphlit's complete documentation, SDK references, and 60+ sample applications. It can: * Generate working SDK code in Python, TypeScript, or .NET * Translate examples between languages * Debug Graphlit integration issues * Suggest best practices and optimization patterns * Answer questions about Graphlit concepts and features **Best for**: Quick code generation, learning Graphlit, debugging without IDE setup **Learn more**: [Ask Graphlit Documentation](/resources/ask-graphlit) *** ## Three Ways to Use AI for Coding | Method | Best For | Setup Required | | ---------------------------- | ---------------------------------------- | ---------------------- | | **Documentation MCP Server** | Understanding concepts, finding examples | Minimal (just URL) | | **Ask Graphlit Web App** | Quick code generation, no IDE needed | None (just visit site) | | **Ask Graphlit MCP Server** | Code generation inside your IDE | Minimal (just URL) | *** ## Best Practices ### Combining Documentation Search + Code Generation **Recommended workflow**: 1. **Learn** (Documentation MCP): Understand concepts and architecture 2. **Generate** (Ask Graphlit MCP): Generate SDK code in your language inside IDE 3. **Reference** (Documentation MCP): Look up details while coding 4. **Optimize** (Ask Graphlit MCP): Get suggestions for improvements ### Example Workflow **Scenario**: Build a Slack integration with entity extraction ``` Step 1 (Documentation MCP in Cursor): "How does Graphlit's Slack feed integration work?" ā†’ Returns feed documentation and OAuth setup guide Step 2 (Ask Graphlit MCP in Cursor): "Generate TypeScript code to create a Slack feed with entity extraction" ā†’ Returns complete working code with workflow configuration Step 3 (Documentation MCP in Cursor): "How do I deploy this to production with multi-tenant isolation?" ā†’ Returns production deployment patterns and user scoping guide Step 4 (Ask Graphlit MCP in Cursor): "Optimize this code for multiple users" ā†’ Generates code with userId scoping and best practices ``` *** ## What AI Assistants Can Access Through the MCP server and llms.txt, AI assistants have access to: āœ… **Getting Started guides** (Platform Overview, Quickstart)\ āœ… **SDK Setup** (Python, TypeScript, .NET installation)\ āœ… **Tutorials** (AI Agents, Knowledge Graph, Context Engineering)\ āœ… **Platform Concepts** (Key Concepts, Semantic Memory, Models, Connectors)\ āœ… **API Use Case Library** (100+ operation examples)\ āœ… **Production Guides** (Multi-tenant, scaling, deployment)\ āœ… **Case Studies** (Zine production architecture) āŒ **Not accessible**: Private project data, API keys, user credentials *** ## Need Help? If you have questions about using AI assistants with Graphlit: * **Discord**: [Join our community](https://discord.gg/ygFmfjy3Qx) * **Ask Graphlit**: [Generate code examples](https://ask.graphlit.dev) * **GitHub Discussions**: [Ask questions](https://github.com/graphlit/graphlit-samples/discussions) # Platform Overview Complete overview of Graphlit - what it is, why it exists, and what you can build with it Graphlit is the context layer for AI agents. We give developers complete infrastructure to build production AI applications with organizational knowledge ā€” entities, relationships, and temporal state. ## Quick Reference | Concept | Description | Learn More | | ---------------------- | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | | **Feeds** | **Automatic sync from 30+ sources** (Slack, Gmail, GitHub, S3, RSS, etc.) - Unique to Graphlit | [Feeds](/platform/feeds) | | **Content** | Documents, audio transcription, video analysis, web crawling | [Ingestion Guide](/api-guides/use-cases/content) | | **Advanced Filtering** | **Production-grade queries**: geo-spatial, image similarity, entity-based, temporal, boolean logic | [See Examples](#production-grade-metadata-filtering) | | **Workflows** | **Custom extraction pipelines** with vision models, OCR, entity extraction - Unique depth | [Workflows](/platform/key-concepts#workflows) | | **Conversations** | RAG with streaming, citations, and tool calling | [AI Agents Tutorial](/tutorials/ai-agents) | | **Knowledge Graph** | Schema.org entities + relationships + temporal context | [Knowledge Graph Tutorial](/tutorials/knowledge-graph) | | **Specifications** | Reusable model configurations (GPT-4, Claude, Gemini, Deepseek, etc.) | [AI Models](/platform/models) | | **Collections** | Flexible content grouping (virtual folders, topics, projects, users/teams) | [Key Concepts](/platform/key-concepts#collections) | *** {% hint style="success" %} **Start here roadmap:** 1. [Sign up for Graphlit](/account-setup-one-time/signup) 2. [Create your first project](/account-setup-one-time/create-project) 3. [Copy credentials and run `hello.ts`](/account-setup-one-time/credentials#verify-your-setup) 4. Continue to [Quickstart: Your First Agent](/getting-started/quickstart) {% endhint %} ## What is a Context Layer? A context layer gives AI agents the ability to understand entities, relationships, and temporal state - not just retrieve similar documents. **The key difference**: RAG retrieves text chunks by similarity. A context layer knows "Alice from Acme Corp mentioned pricing on Oct 15" and can answer "What did Alice say about pricing?" [Deep dive: Semantic Memory architecture ā†’](/platform/semantic-memory) *** ## What Makes Graphlit Different Graphlit provides a complete data platform for production AI applications - from ingestion to processing to retrieval. ### šŸ”Œ 30+ Data Connectors Connect to any data source with one API call. OAuth, API keys, or bot tokens - we handle authentication for you. **Communication:** Slack, Microsoft Teams, Discord\ **Email:** Gmail, Outlook\ **Project Management:** Jira, Linear, GitHub Issues, Trello\ **Documents:** Google Drive, OneDrive, SharePoint, Dropbox, Box, Notion\ **Cloud Storage:** AWS S3, Azure Blob, Google Cloud Storage\ **Meetings:** Fireflies.ai, Fathom, Attio Meetings, Salesforce ECI\ **CRM & Contacts:** Salesforce CRM, Attio CRM, Google Contacts, Microsoft Contacts\ **Research:** Parallel Research, Parallel Entity Discovery\ **Social:** Twitter, Reddit, YouTube\ **Calendars:** Google Calendar, Outlook Calendar\ **Support:** Zendesk, Intercom\ **Web:** RSS feeds, web crawling, site maps ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedListingTypes } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); // Create Slack feed with bot token const feed = await client.createFeed({ name: 'Team Slack', type: FeedTypes.Slack, slack: { type: FeedListingTypes.Past, token: process.env.SLACK_BOT_TOKEN, // Bot token from Slack app channel: 'general' // Channel name or ID } }); ``` **What this means:** Connect any data source with a single API call. Authentication (OAuth, API keys, bot tokens), sync scheduling, data parsing, and indexing all handled automatically. ### šŸŽ„ Multi-Format Processing **Audio:** Automatic transcription with speaker diarization (Speaker #1, #2, etc.) via Deepgram, AssemblyAI\ **Video:** Audio extraction + transcription (available today), frame analysis coming soon (TwelveLabs, Azure Video Indexer)\ **Documents:** OCR with vision models, layout preservation\ **Web:** Crawling, screenshots, search integration (Tavily, Exa)\ **Email:** Parse and index with attachments\ **Code:** Repository indexing with GitHub connector ```typescript import { Graphlit } from 'graphlit-client'; const client = new Graphlit(); // Transcribe audio with one call const audio = await client.ingestUri( 'https://example.com/meeting.mp3', 'Team Meeting' ); ``` **Result:** Automatic transcription with speaker diarization (Speaker #1, Speaker #2, etc.), searchable transcript indexed for retrieval. ### āš™ļø Custom Workflows **Most content doesn't need workflows** - Graphlit's intelligent defaults handle PDFs, audio, web pages automatically. **When you need workflows:** Entity extraction for knowledge graphs. ```typescript import { Graphlit } from 'graphlit-client'; import { EntityExtractionServiceTypes, ObservableTypes } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); // Build knowledge graph from your documents const workflow = await client.createWorkflow({ name: 'Extract Entities', extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, // Extract entities with LLM extractedTypes: [ ObservableTypes.Person, // People mentioned ObservableTypes.Organization, // Companies, teams ObservableTypes.Label // Topics, themes, tags ] } }] } }); ``` **Result:** Content is automatically prepared (PDFs, audio, web pages), then entities are extracted. Search by person ("all mentions of Alice"), organization ("documents about Acme Corp"), or label/topic. **For complex PDFs only:** Add preparation stage with vision models. See [Workflows documentation](/platform/workflows) for advanced options. ### šŸ”„ Automatic Sync Continuous polling from all connected sources (30 seconds to hours, configurable). After feed creation, data flows automatically - no manual polling, no webhooks to manage, no API rate limits to handle. ### šŸŽØ Publishing Capabilities **Audio Generation:** Text-to-speech with ElevenLabs\ **Summaries:** Automatic content summarization\ **Markdown Export:** Structured content extraction\ **Citations:** Entity-linked, contextualized references ```typescript import { Graphlit } from 'graphlit-client'; import { ContentPublishingFormats, ContentPublishingServiceTypes, FileTypes } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); // Publish markdown summaries of all document content const published = await client.publishContents( "Create a concise summary", { type: ContentPublishingServiceTypes.Text, format: ContentPublishingFormats.Markdown }, undefined, // summaryPrompt undefined, // summarySpecification undefined, // publishSpecification undefined, // name { fileTypes: [FileTypes.Document] } // filter: only documents ); ``` **What this enables:** Transform and republish content. Generate audio versions, create summaries, export structured data. Your knowledge base becomes a content creation engine. ### šŸ” Production-Grade Metadata Filtering Graphlit provides advanced filtering that would take weeks to build yourself - geo-spatial, image similarity, entity-based, temporal, and complex boolean queries all in one API: **Search by Location** (geo-spatial queries) ```typescript import { Graphlit } from 'graphlit-client'; const client = new Graphlit(); // Find all content within 10km of San Francisco const results = await client.queryContents({ search: 'restaurant reviews', location: { latitude: 37.7749, longitude: -122.4194, distance: 10000 } }); ``` **Search by Image** (visual similarity) ```typescript import { Graphlit } from 'graphlit-client'; import fs from 'fs'; const client = new Graphlit(); // Find similar images const imageBuffer = fs.readFileSync('./reference-image.jpg'); const base64Image = imageBuffer.toString('base64'); const similar = await client.queryContents({ imageData: base64Image, imageMimeType: 'image/jpeg', numberSimilar: 20 }); ``` **Search by Entity** (extracted people, orgs, places) ```typescript import { Graphlit } from 'graphlit-client'; const client = new Graphlit(); // Find all content mentioning specific people or organizations const mentions = await client.queryContents({ search: 'product launch', observations: [ { observable: { name: 'Kirk Marple' }}, { observable: { name: 'Graphlit' }} ] }); ``` **Complex Boolean Queries** (AND/OR logic) ```typescript import { Graphlit } from 'graphlit-client'; const client = new Graphlit(); // Find content created last 7 days mentioning "deal closure" const results = await client.queryContents({ search: 'deal closure', createdInLast: 'P7D' // ISO 8601 duration: 7 days }); ``` **What this means:** Filter by location (find content near you), by visual similarity (find images like this one), by entities (all mentions of a person/company), by time (last 24 hours, date ranges), or combine filters. This level of filtering is typically only found in enterprise search systems. ### šŸŽ¬ True Multimodal Processing Graphlit processes audio and video content - not just stores files, but actually extracts and indexes the content: **Audio Files (MP3, WAV, M4A, etc.)** ```typescript import { Graphlit } from 'graphlit-client'; const client = new Graphlit(); // Upload audio, get searchable transcript const audio = await client.ingestUri( 'https://example.com/podcast-episode.mp3', 'Podcast Episode' ); ``` **Result:** Searchable transcript with speaker diarization (Speaker #1, Speaker #2, etc.). **Video Files (MP4, MOV, etc.)** ```typescript import { Graphlit } from 'graphlit-client'; const client = new Graphlit(); // Upload video, extract and transcribe audio const video = await client.ingestUri( 'https://example.com/product-demo.mp4', 'Product Demo Video' ); ``` **Result:** Searchable transcript of audio track. Frame analysis coming soon (TwelveLabs, Azure Video Indexer). **What this means:** Upload media files and immediately search their content. Meeting recordings become searchable transcripts with speaker identification (Speaker #1, #2, etc.). Product videos' audio becomes fully searchable. No separate transcription services needed. *** ## Why Graphlit? Graphlit saves you 3-20 months of integration work and $160k-400k in Year 1 by providing complete data infrastructure for AI agents. [See detailed TCO and competitive comparison ā†’](/why-graphlit) *** ## AI Models Graphlit supports 100+ LLMs including GPT-5, Claude 4.5 Sonnet, Gemini 2.5 Pro, Deepseek Reasoner, and more. [Complete model reference and comparison ā†’](/platform/models) *** ## Data Connectors 30+ feeds including Slack, Gmail, GitHub, Notion, Linear, Jira, Google Drive, OneDrive, S3, RSS, and more. Automatic sync with OAuth, API keys, or public sources. [Browse all feeds and setup guides ā†’](/platform/feeds) *** ## MCP-Native Integration Bring Graphlit into Cursor, Windsurf, Claude Desktop, or VS Code - query your Slack, Gmail, Notion, and 30+ other sources directly from your IDE. Install: `npx -y graphlit-mcp-server` [Complete MCP setup guide ā†’](/mcp-integration/mcp-integration) *** ## What You Can Build **AI agents with memory** - Customer support, sales assistants, engineering agents with persistent context\ **Production SaaS apps** - Multi-tenant platforms ([Zine](https://www.zine.ai) runs on Graphlit with thousands of users)\ **Knowledge extraction** - Automatically extract entities, relationships, timelines from unstructured content [See tutorials ā†’](/getting-started/quickstart) | [Zine case study ā†’](/examples/zine-case-study) *** ## Developer Experience ### 60+ Working Examples Explore the sample gallery with working code you can run immediately: * **Google Colab notebooks** - Run in browser, no setup * **Next.js applications** - Deploy to Vercel * **Streamlit apps** - Python UI examples * **.NET console apps** - C# examples [Browse sample gallery ā†’](https://github.com/graphlit/graphlit-samples) ### Ask Graphlit Get instant code examples and answers from our AI code assistant. * Generate SDK code from descriptions * Get best practices * Find relevant examples * Troubleshoot issues [Use Ask Graphlit ā†’](/resources/ask-graphlit) *** ## Security & Scale ### Enterprise Security * **Encryption at rest** - All data encrypted using AES-256 * **Per-user data isolation** - Multi-tenant with user scoping * **Project-level access** - Managed via Developer Portal * **SOC 2** - Compliance coming soon * **API authentication** - JWT-based secure access ### Built for Scale * **Serverless architecture** - Auto-scaling infrastructure * **Global deployment** - Low latency worldwide * **Usage-based pricing** - Pay only for what you use * **No infrastructure** - We handle operations **Production proof:** Zine runs on Graphlit with thousands of users, automatic sync across 20+ data sources, and millions of documents. *** ## Pricing **Free to get started** - No credit card required. * **Free Tier**: 100 credits, 1GB storage, unlimited conversations * **Hobby**: $49/month + usage * **Starter**: $199/month + usage (10% off) * **Growth**: $999/month + usage (20% off) [View detailed pricing ā†’](https://www.graphlit.com/#pricing) *** ## Need Help? **Community & Support:** * [**Discord**](https://discord.gg/ygFmfjy3Qx) - Community support, fastest response time * [**Ask Graphlit**](/resources/ask-graphlit) - AI code assistant trained on Graphlit * [**GitHub Issues**](https://github.com/graphlit/graphlit-client-python/issues) - Report bugs and feature requests * **Email** - for direct support *** ## About Graphlit We're building the context layer infrastructure for AI applications and agents. **Our mission:** Give every developer the tools to build production AI apps and agents with organizational knowledge. **Products:** * **Graphlit** - Context layer for AI agents * **Zine** - Team memory built on Graphlit ([zine.ai](https://www.zine.ai)) **Connect:** [Website](https://www.graphlit.com) ā€¢ [Twitter](https://twitter.com/graphlit) ā€¢ [LinkedIn](https://www.linkedin.com/company/graphlit) ā€¢ [YouTube](https://www.youtube.com/@Graphlit) ā€¢ [Blog](https://www.graphlit.com/blog) # Quickstart: Your First Agent Build an AI agent with semantic memory in 7 minutes ā±ļø **Time**: 7 minutes\ šŸŽÆ **Level**: Beginner\ šŸ’» **SDK**: All SDKs (Step 4 streaming: TypeScript only) ## What You'll Build An AI agent that: * āœ… Ingests documents into semantic memory * āœ… Searches by meaning (not just keywords) * āœ… Answers questions with citations * āœ… Streams responses in real-time (TypeScript) * āœ… Calls tools to extend capabilities *** ## Prerequisites **You need**: Node.js 20+ ([download](https://nodejs.org/)) and a free Graphlit account. {% hint style="warning" %} **Don't have an account yet?** 1. [Sign up](/account-setup-one-time/signup) (30 seconds) 2. [Create project](/account-setup-one-time/create-project) (1 minute) 3. Copy your credentials from [API Settings](/account-setup-one-time/credentials) {% endhint %} ### Project setup Create a new project folder and initialize it: ```bash mkdir graphlit-quickstart && cd graphlit-quickstart npm init -y npm install graphlit-client dotenv tsx npm install openai # Needed for Steps 4-5 (streaming) ``` Create a `.env` file with your credentials from the [Developer Portal](https://portal.graphlit.dev): ```env GRAPHLIT_ORGANIZATION_ID=your_org_id GRAPHLIT_ENVIRONMENT_ID=your_env_id GRAPHLIT_JWT_SECRET=your_jwt_secret OPENAI_API_KEY=your_openai_key # Needed for Steps 4-5 (streaming) ``` {% hint style="info" %} The SDK automatically loads your `.env` file ā€” no import needed. Get your Graphlit credentials from your project's **API Settings** page in the [Developer Portal](https://portal.graphlit.dev). {% endhint %} ### Verify your setup Create `hello.ts` and run it: ```typescript import { Graphlit } from 'graphlit-client'; const graphlit = new Graphlit(); async function main() { const project = await graphlit.getProject(); console.log(`Connected to: ${project.project.name}`); } main(); ``` ```bash npx tsx hello.ts ``` **Expected output:** ``` Connected to: My Project ``` If you see your project name, you're ready. If you get an error, double-check the values in your `.env` file match what's shown in the Developer Portal. *** ## Step 1: Ingest Content Add a document to semantic memory. Save this as `step1.ts`: ```typescript import { Graphlit } from 'graphlit-client'; const graphlit = new Graphlit(); async function main() { const content = await graphlit.ingestUri( 'https://arxiv.org/pdf/1706.03762.pdf', 'Attention Paper', undefined, undefined, true, // Wait for processing to complete ); console.log(`āœ… Document ingested: ${content.ingestUri.id}`); } main(); ``` ```bash npx tsx step1.ts ``` **What happens**: Graphlit downloads the PDF, extracts text, generates embeddings, and stores in semantic memory. This may take 30-60 seconds. **Expected output:** ``` āœ… Document ingested: 01234567-89ab-cdef-0123-456789abcdef ``` {% hint style="success" %} **Python/.NET**: Get this code in your language instantly: * [Convert to Python](https://ask.graphlit.dev?prompt=Convert%20this%20TypeScript%20code%20to%20Python%3A%0A%0Aimport%20%7B%20Graphlit%20%7D%20from%20%27graphlit-client%27%3B%0A%0Aconst%20graphlit%20%3D%20new%20Graphlit%28%29%3B%0A%0Aasync%20function%20main%28%29%20%7B%0A%20%20const%20content%20%3D%20await%20graphlit.ingestUri%28%0A%20%20%20%20%27https%3A//arxiv.org/pdf/1706.03762.pdf%27%2C%0A%20%20%20%20%27Attention%20Paper%27%2C%0A%20%20%20%20undefined%2C%0A%20%20%20%20undefined%2C%0A%20%20%20%20true%0A%20%20%29%3B%0A%0A%20%20console.log%28%60%E2%9C%85%20Document%20ingested%3A%20%24%7Bcontent.ingestUri.id%7D%60%29%3B%0A%7D%0A%0Amain%28%29%3B) * [Convert to .NET](https://ask.graphlit.dev?prompt=Convert%20this%20TypeScript%20code%20to%20C%23/.NET%3A%0A%0Aimport%20%7B%20Graphlit%20%7D%20from%20%27graphlit-client%27%3B%0A%0Aconst%20graphlit%20%3D%20new%20Graphlit%28%29%3B%0A%0Aasync%20function%20main%28%29%20%7B%0A%20%20const%20content%20%3D%20await%20graphlit.ingestUri%28%0A%20%20%20%20%27https%3A//arxiv.org/pdf/1706.03762.pdf%27%2C%0A%20%20%20%20%27Attention%20Paper%27%2C%0A%20%20%20%20undefined%2C%0A%20%20%20%20undefined%2C%0A%20%20%20%20true%0A%20%20%29%3B%0A%0A%20%20console.log%28%60%E2%9C%85%20Document%20ingested%3A%20%24%7Bcontent.ingestUri.id%7D%60%29%3B%0A%7D%0A%0Amain%28%29%3B) {% endhint %} *** ## Step 2: Search Your Memory Query ingested content by meaning. Save this as `step2.ts`: ```typescript import { Graphlit } from 'graphlit-client'; const graphlit = new Graphlit(); async function main() { const results = await graphlit.queryContents({ search: 'transformer architecture innovations', }); console.log(`Found ${results.contents.results.length} documents:`); for (const item of results.contents.results) { console.log(`šŸ“„ ${item.name}`); } } main(); ``` ```bash npx tsx step2.ts ``` **Semantic search**: Finds documents by meaning, not just keyword matching. Try searching for "attention mechanism" and see it find the transformer paper. **Expected output:** ``` Found 1 documents: šŸ“„ Attention Paper ``` *** ## Step 3: RAG Conversation Ask questions about your content. Save this as `step3.ts`: ```typescript import { Graphlit } from 'graphlit-client'; const graphlit = new Graphlit(); async function main() { // Ingest the document (if you already ran Step 1, this returns the existing content) const content = await graphlit.ingestUri( 'https://arxiv.org/pdf/1706.03762.pdf', 'Attention Paper', undefined, undefined, true, ); // Create conversation scoped to this document const conversation = await graphlit.createConversation({ name: 'Q&A Session', filter: { contents: [{ id: content.ingestUri.id }] } }); // Ask questions const answer = await graphlit.promptConversation( 'What are the key innovations in this paper?', conversation.createConversation.id, ); console.log(answer.promptConversation.message?.message); } main(); ``` ```bash npx tsx step3.ts ``` **What happens**: Graphlit retrieves relevant sections, injects context into the LLM, and generates an answer with citations. **Expected output:** ``` The paper introduces the Transformer architecture, which relies entirely on self-attention mechanisms rather than recurrence or convolutions. Key innovations include multi-head attention and positional encodings. ``` {% hint style="success" %} **Get this code in your language**: * [Python version](https://ask.graphlit.dev?prompt=Convert%20this%20TypeScript%20to%20Python%3A%0A%0Aconst%20conversation%20%3D%20await%20graphlit.createConversation%28%7B%0A%20%20name%3A%20%27Q%26A%20Session%27%2C%0A%20%20filter%3A%20%7B%20contents%3A%20%5B%7B%20id%3A%20content.ingestUri.id%20%7D%5D%20%7D%0A%7D%29%3B%0A%0Aconst%20answer%20%3D%20await%20graphlit.promptConversation%28%0A%20%20%27What%20are%20the%20key%20innovations%3F%27%2C%0A%20%20conversation.createConversation.id%0A%29%3B) * [.NET version](https://ask.graphlit.dev?prompt=Convert%20this%20TypeScript%20to%20C%23/.NET%3A%0A%0Aconst%20conversation%20%3D%20await%20graphlit.createConversation%28%7B%0A%20%20name%3A%20%27Q%26A%20Session%27%2C%0A%20%20filter%3A%20%7B%20contents%3A%20%5B%7B%20id%3A%20content.ingestUri.id%20%7D%5D%20%7D%0A%7D%29%3B%0A%0Aconst%20answer%20%3D%20await%20graphlit.promptConversation%28%0A%20%20%27What%20are%20the%20key%20innovations%3F%27%2C%0A%20%20conversation.createConversation.id%0A%29%3B) {% endhint %} *** ## Step 4: Real-Time Streaming (TypeScript) {% hint style="info" %} **TypeScript SDK only**: Python and C# SDKs use synchronous `promptConversation()` from Step 3. Real-time streaming is TypeScript-specific. {% endhint %} {% hint style="warning" %} **Requires OpenAI API key.** If you haven't already, add `OPENAI_API_KEY=your_key` to your `.env` file. Get a key from [platform.openai.com/api-keys](https://platform.openai.com/api-keys). {% endhint %} Save this as `step4.ts`: ```typescript import { Graphlit } from 'graphlit-client'; import { OpenAI } from 'openai'; import { SpecificationTypes, ModelServiceTypes, OpenAiModels, } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Enable streaming with OpenAI client graphlit.setOpenAIClient(new OpenAI()); async function main() { const spec = await graphlit.createSpecification({ name: 'Assistant', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt4O_128K, temperature: 0.7 } }); await graphlit.streamAgent( 'Explain transformer attention in simple terms', (event) => { if (event.type === 'message_update') { process.stdout.write(event.message.message); if (!event.isStreaming) { console.log('\n[complete]'); } } }, undefined, { id: spec.createSpecification.id }, ); } main(); ``` ```bash npx tsx step4.ts ``` **What happens**: Tokens stream in real-time as the AI generates the response (like ChatGPT's typing effect). **Expected output:** ``` Transformer attention is a mechanism that allows the model to focus on different parts of the input when processing each token. Think of it like highlighting the most relevant words in a sentence when trying to understand each word's meaning. [complete] ``` *** ## Step 5: Add Tool Calling Give your agent functions to call. Save this as `step5.ts`: ```typescript import { Graphlit } from 'graphlit-client'; import { OpenAI } from 'openai'; import { SpecificationTypes, ModelServiceTypes, OpenAiModels, ToolDefinitionInput, } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); graphlit.setOpenAIClient(new OpenAI()); // Define tool const searchTool: ToolDefinitionInput = { name: 'search_memory', description: 'Search semantic memory for documents', schema: JSON.stringify({ type: 'object', properties: { query: { type: 'string', description: 'Search query' }, }, required: ['query'], }), }; // Tool implementation const toolHandlers = { search_memory: async (args: { query: string }) => { const results = await graphlit.queryContents({ search: args.query }); return results.contents.results.map((c) => c.name); }, }; async function main() { const spec = await graphlit.createSpecification({ name: 'Agent with Tools', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt4O_128K } }); await graphlit.streamAgent( 'Find documents about attention mechanisms', (event) => { if (event.type === 'tool_update' && event.status === 'completed') { console.log(`\nšŸ”§ Called ${event.toolCall.name}`); } else if (event.type === 'message_update') { process.stdout.write(event.message.message); if (!event.isStreaming) { console.log('\n[complete]'); } } }, undefined, { id: spec.createSpecification.id }, [searchTool], toolHandlers, ); } main(); ``` ```bash npx tsx step5.ts ``` **What happens**: The agent decides when to call your function, executes it, and uses the results in its response. {% hint style="success" %} **Tool calling works in all SDKs**: Python, TypeScript, and C# all support defining tools and handlers. {% endhint %} *** ## What You've Built In 7 minutes, you created an AI agent with:
CapabilityWhy It Matters
Semantic memoryIngest and search documents by meaning
RAG conversationsQ&A grounded in your content
Real-time streamingTypeScript token-by-token responses
Agentic behaviorAI that calls functions to accomplish tasks
### Data Flow Summary {% @mermaid/diagram content="graph LR A\[Ingest Content] -->|ingestUri| B\[Semantic Memory] C\[Create Specification] --> D\[Conversation] B --> D D -->|promptConversation| E\[Synchronous Response] D -->|streamAgent| F\[Streaming Response] F -->|Optional| G\[Tool Handlers]" %} 1. **Ingest Content** ā†’ Semantic memory indexes files, messages, and pages 2. **Create Specification** ā†’ Pick the LLM and parameters for the agent 3. **Create Conversation** ā†’ Optionally scope retrieval with filters 4. **promptConversation** (all SDKs) or **streamAgent** (TypeScript) ā†’ Get responses 5. **Tool Handlers** ā†’ Agent can call functions when needed *** ## Production Notes **Timeouts**: For very large files, `ingestUri(url, name, undefined, undefined, true)` may exceed default timeouts. Consider wrapping in `Promise.race` with a timeout or polling via `isContentDone`. **Logging**: Replace `console.log` with structured logging (Pino/Winston) in production services. **Secrets**: Keep `.env` out of version control; use platform secret stores in deployment. **Rate limits**: OpenAI streaming respects your account quotas. Handle `429` responses with retries. *** ## Next Steps ### Learn Advanced Patterns [**AI Agents with Memory**](/tutorials/ai-agents) - Multi-agent systems, advanced tool patterns (15 min) [**Knowledge Graph**](/tutorials/knowledge-graph) - Extract entities and relationships (20 min) [**MCP Integration**](/mcp-integration/mcp-integration) - Connect to your IDE (10 min) ### Explore Sample Applications [**šŸ““ 60+ Colab Notebooks**](https://github.com/graphlit/graphlit-samples/tree/main/python/Notebook%20Examples) - Run Python examples instantly * RAG & Conversations (15+ examples) * Ingestion & Preparation (6+ examples) * Knowledge Graph & Extraction (7+ examples) [**šŸš€ Next.js Apps**](https://github.com/graphlit/graphlit-samples/tree/main/nextjs) - Deploy-ready applications * Full-featured chat with streaming * Chat with knowledge graph visualization * Document extraction interface [**šŸ’» Streamlit Apps**](https://github.com/graphlit/graphlit-samples/tree/main/python/Streamlit) - Interactive Python UIs ### Add More Capabilities **Different AI Models:** ```typescript // Use Claude instead import { ModelServiceTypes, AnthropicModels } from 'graphlit-client/dist/generated/graphql-types'; serviceType: ModelServiceTypes.Anthropic, anthropic: { model: AnthropicModels.Claude_4_5Sonnet } ``` **Multiple Documents:** ```typescript // Upload multiple PDFs const urls = [ 'https://example.com/doc1.pdf', 'https://example.com/doc2.pdf', ]; const ids = []; for (const url of urls) { const content = await graphlit.ingestUri(url, undefined, undefined, undefined, true); ids.push(content.ingestUri.id); } // Create conversation with all documents const conversation = await graphlit.createConversation({ name: 'Multi-Document Chat', filter: { contents: ids.map(id => ({ id })) } }); ``` **Custom Tools:** ```typescript // Add a database query tool const dbTool: ToolDefinitionInput = { name: 'query_database', description: 'Query the customer database', schema: JSON.stringify({ type: 'object', properties: { query: { type: 'string', description: 'SQL query' }, }, required: ['query'], }), }; ``` *** ## Complete Examples **Full working code**: * [**TypeScript SDK README**](https://github.com/graphlit/graphlit-client-typescript#readme) - All examples tested and verified * [**Sample Repository**](https://github.com/graphlit/graphlit-samples) - 60+ working examples * [**Next.js Apps**](https://github.com/graphlit/graphlit-samples/tree/main/nextjs) - Full-stack applications *** ## Troubleshooting **"streamAgent is not a function" (Python/C#)** Use `prompt_conversation()` (Python) or `PromptConversation()` (C#). Streaming is TypeScript-only. See Step 3 for the universal pattern. **"OpenAI API key not found"** Only needed for TypeScript `streamAgent()` (Step 4). Add to `.env`: ```env OPENAI_API_KEY=your_key ``` Get your key from [platform.openai.com/api-keys](https://platform.openai.com/api-keys). **"Content not finished processing"** Use `isSynchronous: true` (fifth parameter) in `ingestUri()` to wait for completion: ```typescript await graphlit.ingestUri(url, name, undefined, undefined, true); ``` **"Module not found: dotenv"** Install dotenv: ```bash npm install dotenv ``` *** ## Need Help? [**Discord Community**](https://discord.gg/ygFmfjy3Qx) - Get help from the Graphlit team and community [**Ask Graphlit**](/resources/ask-graphlit) - AI code assistant for instant SDK code examples [**TypeScript SDK Docs**](https://github.com/graphlit/graphlit-client-typescript) - Complete API reference [**Sample Gallery**](https://github.com/graphlit/graphlit-samples) - Browse working examples # Python Install the Python SDK and start building AI applications with semantic memory. Build AI applications with Python using the Graphlit SDK. {% hint style="info" %} **New to Graphlit?** Complete the [Quickstart tutorial](/getting-started/quickstart) for a hands-on introduction. {% endhint %} *** ## Installation Install the Graphlit client with pip: ```bash pip install graphlit-client ``` **Requirements:** * Python 3.8 or higher * Graphlit account with [API credentials](/account-setup-one-time/create-project) *** ## Quick Start ```python import asyncio import os from graphlit import Graphlit from graphlit_api import * async def main(): # Reads from environment variables automatically graphlit = Graphlit() # Ingest content response = await graphlit.client.ingest_text( name="Product Requirements", text="Our AI agent needs persistent memory across sessions..." ) print(f"āœ… Memory created: {response.ingest_text.id}") asyncio.run(main()) ``` {% hint style="success" %} **That's it!** The SDK automatically reads `GRAPHLIT_ORGANIZATION_ID`, `GRAPHLIT_ENVIRONMENT_ID`, and `GRAPHLIT_JWT_SECRET` from your environment. {% endhint %} *** ## Configuration ### Environment Variables (Production) Create a `.env` file (never commit this): ```bash GRAPHLIT_ORGANIZATION_ID=your_actual_org_id GRAPHLIT_ENVIRONMENT_ID=your_actual_env_id GRAPHLIT_JWT_SECRET=your_actual_jwt_secret ``` Load it with python-dotenv: ```python import asyncio from dotenv import load_dotenv from graphlit import Graphlit load_dotenv() # Loads .env file async def main(): graphlit = Graphlit() # Reads from environment # Your code here asyncio.run(main()) ``` Install python-dotenv: ```bash pip install python-dotenv ``` {% hint style="warning" %} **Security:** Add `.env` to your `.gitignore` immediately. Use platform secrets (AWS Secrets Manager, etc.) in production deployments. {% endhint %} ### Alternative: Explicit Configuration Only use if you need to override environment variables: ```python from graphlit import Graphlit import os graphlit = Graphlit( organization_id=os.environ['GRAPHLIT_ORGANIZATION_ID'], environment_id=os.environ['GRAPHLIT_ENVIRONMENT_ID'], jwt_secret=os.environ['GRAPHLIT_JWT_SECRET'] ) ``` *** ## Common Patterns ### Ingest Content ```python # From URL response = await graphlit.client.ingest_uri( uri="https://example.com/document.pdf" ) # From text response = await graphlit.client.ingest_text( name="Meeting Notes", text="Discussion about Q4 planning..." ) ``` ### Search Memory ```python response = await graphlit.client.query_contents( filter=ContentFilter( search="quarterly planning" ) ) for content in response.contents.results: print(f"šŸ“„ {content.name}") ``` ### Chat with Context ```python # Create conversation conversation = await graphlit.client.create_conversation( conversation=ConversationInput( name="AI Assistant" ) ) # Ask questions response = await graphlit.client.prompt_conversation( prompt="What did we discuss about Q4 planning?", id=conversation.create_conversation.id ) print(response.prompt_conversation.message.message) ``` *** ## Next Steps **Quickstarts:** * [Quickstart: Your First Agent](/getting-started/quickstart) - Build a streaming agent in 7 minutes * [AI Agents](/tutorials/ai-agents) - Create agents with persistent memory * [Knowledge Graph](/tutorials/knowledge-graph) - Extract entities and relationships **Examples:** * [Python Notebooks](https://github.com/graphlit/graphlit-samples/tree/main/python/Notebook%20Examples) - 60+ working examples * [Streamlit Apps](https://github.com/graphlit/graphlit-samples/tree/main/python/Streamlit) - Full UI applications **Resources:** * [Python SDK on GitHub](https://github.com/graphlit/graphlit-client-python) * [Use Case Library](/api-guides/use-cases) - 100+ code examples * [Ask Graphlit](/resources/ask-graphlit) - AI code assistant * [Join Discord](https://discord.gg/ygFmfjy3Qx) - Get help from the community # TypeScript Install the TypeScript/Node.js SDK and start building AI applications with semantic memory. Build AI applications with TypeScript or JavaScript using the Graphlit SDK. {% hint style="info" %} **New to Graphlit?** Complete the [Quickstart tutorial](/getting-started/quickstart) for a hands-on introduction. {% endhint %} *** ## Installation Install the Graphlit client with npm or yarn: {% tabs %} {% tab title="npm" %} ```bash npm install graphlit-client ``` {% endtab %} {% tab title="yarn" %} ```bash yarn add graphlit-client ``` {% endtab %} {% tab title="pnpm" %} ```bash pnpm add graphlit-client ``` {% endtab %} {% endtabs %} **Requirements:** * Node.js 20 or higher (Docker: use `node:20-slim` or `node:22-slim`) * Graphlit account with [API credentials](/account-setup-one-time/create-project) *** ## Quick Start {% tabs %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; async function main() { const graphlit = new Graphlit(); const response = await graphlit.ingestText( 'Our AI agent needs persistent memory across sessions...', 'Product Requirements', ); console.log(`āœ… Memory created: ${response.ingestText.id}`); } main().catch((error) => { console.error(error); process.exit(1); }); ``` {% endtab %} {% tab title="JavaScript (ESM)" %} ```javascript import { Graphlit } from 'graphlit-client'; async function main() { const graphlit = new Graphlit(); const response = await graphlit.ingestText( 'Our AI agent needs persistent memory across sessions...', 'Product Requirements' ); console.log(`āœ… Memory created: ${response.ingestText.id}`); } main().catch((error) => { console.error(error); process.exit(1); }); ``` {% endtab %} {% tab title="JavaScript (CommonJS)" %} ```javascript require('dotenv/config'); const { Graphlit } = require('graphlit-client'); async function main() { const graphlit = new Graphlit(); const response = await graphlit.ingestText( 'Our AI agent needs persistent memory across sessions...', 'Product Requirements' ); console.log(`āœ… Memory created: ${response.ingestText.id}`); } main().catch((error) => { console.error(error); process.exit(1); }); ``` {% endtab %} {% endtabs %} {% hint style="success" %} **That's it!** You now have semantic memory for your AI application. {% endhint %} *** ## Configuration Options ### Environment Variables (Recommended) ```typescript import { Graphlit } from 'graphlit-client'; const graphlit = new Graphlit(); ``` {% hint style="warning" %} **Security:** Never commit credentials to git. Use environment variables or secrets management in production. {% endhint %} ### Alternative: Explicit Configuration Only use if you need to override environment variables: ```typescript import { Graphlit } from 'graphlit-client'; const graphlit = new Graphlit({ organizationId: process.env.GRAPHLIT_ORGANIZATION_ID, environmentId: process.env.GRAPHLIT_ENVIRONMENT_ID, jwtSecret: process.env.GRAPHLIT_JWT_SECRET, }); ``` *** ## Common Patterns ### Ingest Content ```typescript // From URL const pdf = await graphlit.ingestUri( 'https://example.com/document.pdf', 'Product Brief', undefined, // id undefined, // identifier true, // isSynchronous ); console.log(`šŸ“„ PDF ready: ${pdf.ingestUri.id}`); // From text const notes = await graphlit.ingestText( 'Discussion about Q4 planning...', 'Meeting Notes', ); console.log(`šŸ“ Notes ready: ${notes.ingestText.id}`); ``` ### Search Memory ```typescript const response = await graphlit.queryContents({ search: 'quarterly planning' }); for (const content of response.contents?.results ?? []) { console.log(`šŸ“„ ${content.name}`); } ``` ### Chat with Context ```typescript // Create conversation const conversation = await graphlit.createConversation({ name: 'AI Assistant', }); // Ask questions const answer = await graphlit.promptConversation( 'What did we discuss about Q4 planning?', conversation.createConversation.id, ); console.log(answer.promptConversation.message?.message); ``` *** ## Next Steps **Quickstarts:** * [Quickstart: Your First Agent](/getting-started/quickstart) - Build a streaming agent in 7 minutes * [AI Agents](/tutorials/ai-agents) - Create agents with persistent memory * [MCP Integration](/mcp-integration/mcp-integration) - Connect to your IDE **Examples:** * [Next.js Applications](https://github.com/graphlit/graphlit-samples/tree/main/nextjs) - Full-stack chat apps * [MCP Server](https://github.com/graphlit/graphlit-mcp-server) - Production MCP implementation **Resources:** * [TypeScript SDK on GitHub](https://github.com/graphlit/graphlit-client-typescript) * [Use Case Library](/api-guides/use-cases) - 100+ code examples * [Ask Graphlit](/resources/ask-graphlit) - AI code assistant * [Join Discord](https://discord.gg/ygFmfjy3Qx) - Get help from the community # .NET Install the .NET SDK and start building AI applications with semantic memory. Build AI applications with C# using the Graphlit SDK. {% hint style="info" %} **New to Graphlit?** Complete the [Quickstart tutorial](/getting-started/quickstart) for a hands-on introduction. {% endhint %} *** ## Installation Install the Graphlit client with NuGet: ```bash dotnet add package Graphlit ``` **Requirements:** * .NET 6.0 or higher (or .NET 8.0+) * Graphlit account with [API credentials](/account-setup-one-time/create-project) *** ## Quick Start ```csharp using System; using System.Net.Http; using System.Threading.Tasks; using Graphlit; class Program { static async Task Main(string[] args) { // Reads from environment variables using var httpClient = new HttpClient(); var graphlit = new Graphlit( httpClient, organizationId: Environment.GetEnvironmentVariable("GRAPHLIT_ORGANIZATION_ID"), environmentId: Environment.GetEnvironmentVariable("GRAPHLIT_ENVIRONMENT_ID"), jwtSecret: Environment.GetEnvironmentVariable("GRAPHLIT_JWT_SECRET") ); // Ingest content var response = await graphlit.IngestText.ExecuteAsync( name: "Product Requirements", text: "Our AI agent needs persistent memory across sessions..." ); response.EnsureNoErrors(); Console.WriteLine($"āœ… Memory created: {response.Data?.IngestText?.Id}"); } } ``` {% hint style="success" %} **That's it!** Set `GRAPHLIT_ORGANIZATION_ID`, `GRAPHLIT_ENVIRONMENT_ID`, and `GRAPHLIT_JWT_SECRET` in your environment and the SDK reads them automatically. {% endhint %} *** ## Configuration ### Environment Variables (Production) **Option 1: launchSettings.json** (Development) ```json { "profiles": { "MyApp": { "environmentVariables": { "GRAPHLIT_ORGANIZATION_ID": "your_actual_org_id", "GRAPHLIT_ENVIRONMENT_ID": "your_actual_env_id", "GRAPHLIT_JWT_SECRET": "your_actual_jwt_secret" } } } } ``` **Option 2: User Secrets** (Development) ```bash dotnet user-secrets init dotnet user-secrets set "GRAPHLIT_ORGANIZATION_ID" "your_actual_org_id" dotnet user-secrets set "GRAPHLIT_ENVIRONMENT_ID" "your_actual_env_id" dotnet user-secrets set "GRAPHLIT_JWT_SECRET" "your_actual_jwt_secret" ``` **Option 3: appsettings.json** (Never commit secrets) ```json { "Graphlit": { "OrganizationId": "", "EnvironmentId": "", "JwtSecret": "" } } ``` Load with IConfiguration: ```csharp using Microsoft.Extensions.Configuration; var config = new ConfigurationBuilder() .AddJsonFile("appsettings.json") .AddEnvironmentVariables() // Overrides appsettings .AddUserSecrets() // Development secrets .Build(); using var httpClient = new HttpClient(); var graphlit = new Graphlit( httpClient, organizationId: config["Graphlit:OrganizationId"], environmentId: config["Graphlit:EnvironmentId"], jwtSecret: config["Graphlit:JwtSecret"] ); ``` {% hint style="warning" %} **Security:** Use Azure Key Vault, AWS Secrets Manager, or environment variables in production. Never commit secrets to source control. Add `appsettings.*.json` with secrets to `.gitignore`. {% endhint %} *** ## Common Patterns ### Ingest Content ```csharp // From URL var response = await graphlit.IngestUri.ExecuteAsync( uri: "https://example.com/document.pdf" ); response.EnsureNoErrors(); // From text var textResponse = await graphlit.IngestText.ExecuteAsync( name: "Meeting Notes", text: "Discussion about Q4 planning..." ); textResponse.EnsureNoErrors(); ``` ### Search Memory ```csharp var input = new ContentFilter { Search = "quarterly planning" }; var response = await graphlit.QueryContents.ExecuteAsync(input); response.EnsureNoErrors(); foreach (var content in response.Data?.Contents?.Results ?? Array.Empty()) { Console.WriteLine($"šŸ“„ {content.Name}"); } ``` ### Chat with Context ```csharp // Create conversation var conversationInput = new ConversationInput { Name = "AI Assistant" }; var conversation = await graphlit.CreateConversation.ExecuteAsync(conversationInput); conversation.EnsureNoErrors(); // Ask questions var response = await graphlit.PromptConversation.ExecuteAsync( prompt: "What did we discuss about Q4 planning?", id: conversation.Data?.CreateConversation?.Id ); response.EnsureNoErrors(); Console.WriteLine(response.Data?.PromptConversation?.Message?.Message); ``` *** ## Next Steps **Quickstarts:** * [Quickstart: Your First Agent](/getting-started/quickstart) - Build a streaming agent in 7 minutes * [AI Agents](/tutorials/ai-agents) - Create agents with persistent memory **Examples:** * [.NET Examples](https://github.com/graphlit/graphlit-samples/tree/main/dotnet) - Console apps and samples * [Microsoft Agent Framework](/tutorials/ai-agents) - Integration patterns **Resources:** * [.NET SDK on GitHub](https://github.com/graphlit/graphlit-client-dotnet) * [Use Case Library](/api-guides/use-cases) - 100+ code examples * [Ask Graphlit](/resources/ask-graphlit) - AI code assistant * [Join Discord](https://discord.gg/ygFmfjy3Qx) - Get help from the community # AI Agents with Memory ā±ļø **Time to Complete:** 15 minutes\ šŸŽÆ **Level:** Intermediate\ šŸ’» **Language:** TypeScript ## What You'll Build An AI agent that: * āœ… Maintains conversation history across sessions * āœ… Searches your knowledge base semantically * āœ… Uses tools to accomplish tasks * āœ… Reasons across multiple data sources * āœ… Streams responses in real-time [**šŸ“ Tool calling examples**](https://github.com/graphlit/graphlit-samples/tree/main/nextjs/chat-graph) | [**šŸš€ Production example: Zine**](/examples/zine-case-study) *** ## Why Agents Need Memory Traditional chatbots forget everything between sessions. **AI agents with semantic memory**: * āœ… Remember past conversations and decisions * āœ… Search across all your company's knowledge * āœ… Use tools to take actions (not just answer questions) * āœ… Reason across multiple data sources * āœ… Share context with other agents **Real-world example**: [Zine](https://www.zine.ai) uses this pattern to let agents search Slack threads, meeting transcripts, and GitHub discussions simultaneously. *** ## Prerequisites * Complete the [Quickstart tutorial](/getting-started/quickstart) * Graphlit account with content ingested {% hint style="info" %} Need Python or .NET versions? Open **Ask Graphlit** from the Developer Portal sidebar (or visit [ask.graphlit.dev](https://ask.graphlit.dev)) for autogenerated samples in any SDK. {% endhint %} *** ## Pattern 1: Multi-Turn Conversations with Memory First, create an agent that keeps track of everything you've asked it: ```typescript import { Graphlit } from 'graphlit-client'; import { SpecificationTypes, ModelServiceTypes, OpenAiModels, } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); async function main() { console.log('āš™ļø Creating agent specification...'); const spec = await graphlit.createSpecification({ name: 'Sales Agent Spec', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt5_400K, temperature: 0.7, completionTokenLimit: 2000, }, systemPrompt: `You are a helpful sales agent. You have access to all customer conversations, product docs, and meeting notes. When asked about customers, search your knowledge base and provide specific details with context.`, }); console.log('šŸ§  Creating agent memory...'); const conversation = await graphlit.createConversation({ name: 'Sales Agent - Acme Corp', specification: { id: spec.createSpecification.id }, }); const conversationId = conversation.createConversation.id; console.log(`āœ… Agent created: ${conversationId}`); console.log('\nšŸ’¬ Turn 1: Asking about customer...'); const turn1 = await graphlit.promptConversation( "What do we know about Acme Corp's requirements?", conversationId, ); console.log(`Agent: ${turn1.promptConversation.message?.message}`); console.log('\nšŸ’¬ Turn 2: Follow-up question...'); const turn2 = await graphlit.promptConversation( 'What pricing concerns did they raise?', conversationId, ); console.log(`Agent: ${turn2.promptConversation.message?.message}`); console.log('\nšŸ’¬ Turn 3: Synthesis...'); const turn3 = await graphlit.promptConversation( "Based on everything we discussed, what's the best next step?", conversationId, ); console.log(`Agent: ${turn3.promptConversation.message?.message}`); console.log('\nšŸ“œ Conversation history:'); const history = await graphlit.getConversation(conversationId); history.conversation?.messages?.forEach((msg, index) => { if (!msg?.message) { return; } const role = msg.role === 'USER' ? 'User' : 'Agent'; console.log(` ${index + 1}. ${role}: ${msg.message}`); }); } main().catch((error) => { console.error(error); process.exit(1); }); ``` **What's happening:** * Agent automatically retrieves relevant context from its memory * Conversation state persists across turns (follow-up questions work) * You can inspect the full conversation history after each exchange *** ## Pattern 2: Agentic Tool Calling Let your agent call custom tools instead of just responding with text: ```typescript import { Graphlit } from 'graphlit-client'; import { SpecificationTypes, ModelServiceTypes, OpenAiModels, ToolDefinitionInput, } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); async function agentWithTools() { const tools: ToolDefinitionInput[] = [ { name: 'search_slack', description: 'Search Slack messages for information', schema: JSON.stringify({ type: 'object', properties: { query: { type: 'string', description: 'Search query' }, channel: { type: 'string', description: 'Slack channel' }, }, required: ['query'], }), }, { name: 'get_github_pr', description: 'Get details about a GitHub pull request', schema: JSON.stringify({ type: 'object', properties: { pr_number: { type: 'integer', description: 'PR number' }, }, required: ['pr_number'], }), }, ]; const spec = await graphlit.createSpecification({ name: 'Developer Assistant Spec', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt5_400K, temperature: 0.1, }, systemPrompt: `You are a developer assistant with access to your team's knowledge base. Use the available tools to search for relevant information, then synthesize findings to provide specific, actionable answers.`, }); const conversation = await graphlit.createConversation({ name: 'Multi-Tool Agent', specification: { id: spec.createSpecification.id }, tools, }); const response = await graphlit.promptConversation( 'Why did PR #247 fail CI? Check Slack discussions about it.', conversation.createConversation.id, ); const message = response.promptConversation.message; if (message?.toolCalls?.length) { console.log('šŸ› ļø Agent suggested tool usage:'); message.toolCalls.forEach((toolCall) => { if (!toolCall) { return; } console.log(` ${toolCall.name}(${toolCall.arguments})`); }); } console.log(`\nšŸ’¬ Agent: ${message?.message}`); } agentWithTools().catch((error) => { console.error(error); process.exit(1); }); ``` **What's happening:** * Agent sees the available tool definitions and selects the right one * Tool calls are recorded on the conversation so you can execute them * Responses combine tool outputs with natural language answers **Real-world example**: Zine uses this pattern for questions like ā€œWhy are checkout timeouts spiking?ā€ ā€” the agent queries Sentry, Slack, GitHub, and meeting notes via tools. *** ## Pattern 3: Multi-Agent Systems with Shared Knowledge Point multiple agents at the same semantic memory but give each its own persona: ```typescript import { Graphlit } from 'graphlit-client'; import { CollectionTypes, SpecificationTypes, ModelServiceTypes, OpenAiModels, RetrievalStrategyTypes, } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); async function multiAgentSystem() { const knowledgeBase = await graphlit.createCollection({ name: 'Company Knowledge Base', type: CollectionTypes.Collection, }); const collectionId = knowledgeBase.createCollection.id; const salesSpec = await graphlit.createSpecification({ name: 'Sales Agent Spec', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt5_400K, temperature: 0.7, }, systemPrompt: 'You are a sales agent. Focus on customer needs, pricing, and deal status.', retrievalStrategy: { type: RetrievalStrategyTypes.Content, contentLimit: 10, }, }); const engSpec = await graphlit.createSpecification({ name: 'Engineering Agent Spec', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt5_400K, temperature: 0.3, }, systemPrompt: 'You are an engineering agent. Focus on technical requirements, integrations, and implementation details.', retrievalStrategy: { type: RetrievalStrategyTypes.Content, contentLimit: 10, }, }); const salesAgent = await graphlit.createConversation({ name: 'Sales Agent', specification: { id: salesSpec.createSpecification.id }, filter: { collections: [{ id: collectionId }] }, }); const engAgent = await graphlit.createConversation({ name: 'Engineering Agent', specification: { id: engSpec.createSpecification.id }, filter: { collections: [{ id: collectionId }] }, }); const salesAnswer = await graphlit.promptConversation( "What are Acme Corp's budget constraints?", salesAgent.createConversation.id, ); const engineeringAnswer = await graphlit.promptConversation( 'What technical integrations does Acme Corp need?', engAgent.createConversation.id, ); console.log('šŸ’¼ Sales Agent:', salesAnswer.promptConversation.message?.message); console.log('āš™ļø Engineering Agent:', engineeringAnswer.promptConversation.message?.message); } multiAgentSystem().catch((error) => { console.error(error); process.exit(1); }); ``` **What's happening:** * Two agents share the same knowledge base but have different prompts and temperatures * Retrieval strategy keeps both agents grounded in the same set of documents * You can orchestrate the agents together (sales ā†’ engineering handoff) without duplicating memory *** ## Pattern 4: Streaming Responses For real-time user experiences, stream agent output as it happens: ```typescript import { Graphlit } from 'graphlit-client'; import { SpecificationTypes, ModelServiceTypes, OpenAiModels, } from 'graphlit-client/dist/generated/graphql-types'; import { OpenAI } from 'openai'; const graphlit = new Graphlit(); graphlit.setOpenAIClient(new OpenAI()); async function streamingAgent() { const spec = await graphlit.createSpecification({ name: 'Streaming Assistant', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt5_400K, temperature: 0.5, }, }); await graphlit.streamAgent( 'Explain our pricing model to enterprise customers.', (event) => { switch (event.type) { case 'conversation_started': console.log(`šŸŒŠ Streaming conversation ${event.conversationId}`); break; case 'message_update': process.stdout.write(event.message.message); break; case 'conversation_completed': console.log(`\nāœ… Complete! Tokens: ${event.usage?.tokens ?? 0}`); break; } }, undefined, { id: spec.createSpecification.id }, ); } streamingAgent().catch((error) => { console.error(error); process.exit(1); }); ``` **Use cases:** * Real-time chat experiences (Graphlit + Next.js chat app) * Live meeting transcription with incremental summaries * Progressive document generation or approvals *** ## Production Patterns ### Error Handling ```typescript try { const reply = await graphlit.promptConversation(userInput, conversationId); console.log(reply.promptConversation.message?.message); } catch (error) { console.error('āŒ Agent error:', error); console.log("I'm having trouble right now. Please try again."); } ``` ### Rate Limiting ```typescript // Simple delay between prompts to respect rate limits await new Promise((resolve) => setTimeout(resolve, 500)); ``` ### Context Management ```typescript // Update retrieval scope as the conversation evolves const followUpCollectionId = 'collection-id-for-the-follow-up'; await graphlit.updateConversation({ id: conversationId, filter: { collections: [{ id: followUpCollectionId }] }, }); // Or branch the thread to explore a new idea without losing history const branch = await graphlit.branchConversation(conversationId); console.log('āœØ Branched conversation:', branch.branchConversation?.id); ``` *** ## Real-World Examples ### 1. Customer Support Agent * Searches past tickets, product docs, and conversations * Maintains conversation history per customer * Uses tools to create tickets, update CRM ### 2. Engineering Agent (like Zine) * Searches Slack, GitHub, Jira simultaneously * Reasons across code, discussions, and meeting notes * Uses tools to fetch error logs, run queries ### 3. Sales Agent * Searches customer conversations, contracts, meeting notes * Tracks deal status and objections * Uses tools to update CRM, send follow-ups *** ## Next Steps * [**MCP Integration**](/mcp-integration/mcp-integration) - Connect your agents to your IDE * [**Knowledge Graph**](/tutorials/knowledge-graph) - Extract entities for richer context * [**Context Engineering**](/tutorials/context-engineering) - Optimize memory formation *** ## Full Example: Production Agent See the complete Next.js agent in [graphlit-samples](https://github.com/graphlit/graphlit-samples/tree/main/nextjs/chat-graph): * Tool calling with streaming UI * Shared collections and semantic memory queries * Production-ready configuration (environment variables, error handling) *** **Build agents that actually remember. Build with Graphlit.** # Knowledge Graph ā±ļø **Time to Complete:** 20 minutes\ šŸŽÆ **Level:** Intermediate\ šŸ’» **Language:** TypeScript ## What You'll Build A knowledge graph that automatically extracts: * āœ… Entities (people, organizations, places, events, products) * āœ… Relationships between entities * āœ… Queryable structure from unstructured content * āœ… Multi-hop reasoning across sources [**šŸ“ Knowledge graph example app**](https://github.com/graphlit/graphlit-samples/tree/main/nextjs/chat-graph) *** ## Why Knowledge Graphs Matter **Traditional search**: "Find documents mentioning Sarah"\ **Knowledge graph**: "Find all interactions between Sarah Chen at Acme Corp and our Sales team, including meetings, emails, and Slack conversations" **The difference:** * āœ… Entity recognition (Sarah Chen = person, Acme Corp = organization) * āœ… Relationship tracking (Sarah works at Acme, spoke with Sales) * āœ… Semantic understanding (group related mentions across sources) * āœ… Structured queries on unstructured data **Real-world example**: [Zine](https://www.zine.ai) uses knowledge graphs to answer "Who from Acme Corp have we talked to?" across Slack, email, meetings, and CRM notes. *** ## Prerequisites * Complete the [Quickstart tutorial](/getting-started/quickstart) * Graphlit project with API credentials configured in `.env` * `npm install graphlit-client dotenv` {% hint style="info" %} Need Python or .NET examples? Open **Ask Graphlit** in the Developer Portal (or visit [ask.graphlit.dev](https://ask.graphlit.dev)) for autogenerated samples tailored to your SDK. {% endhint %} *** ## Step 1: Extract Entities from Content Create a workflow, ingest content, and extract entities - all in one script: ```typescript import { Graphlit } from 'graphlit-client'; import { SpecificationTypes, ModelServiceTypes, OpenAiModels, FilePreparationServiceTypes, EntityExtractionServiceTypes, ObservableTypes, } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); const SAMPLE_NOTE = ` Met with Sarah Chen, CTO at Acme Corp, at their San Francisco office. She's evaluating our API for their 800-user deployment. Key attendees: - Sarah Chen (CTO, Acme Corp) - Mike Rodriguez (VP Engineering, Acme Corp) - Alex Kim (Solutions Engineer, our team) They need CRM integration and mentioned Salesforce as their current tool. Looking to deploy by Q4 2025. Budget approved for Enterprise tier. Follow-up: Technical demo scheduled for next Tuesday in Acme's office. `; async function main() { console.log('āš™ļø Creating extraction workflow...'); // 1ļøāƒ£ Configure the LLM that performs entity extraction const extractionSpec = await graphlit.createSpecification({ name: 'Knowledge Graph Extraction', type: SpecificationTypes.Extraction, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt5_400K, temperature: 0, // Deterministic for consistent extraction }, }); // 2ļøāƒ£ Build a workflow that prepares documents and extracts entities const workflow = await graphlit.createWorkflow({ name: 'Entity Extraction Workflow', preparation: { jobs: [ { connector: { type: FilePreparationServiceTypes.Document, }, }, ], }, extraction: { jobs: [ { connector: { type: EntityExtractionServiceTypes.ModelText, modelText: { specification: { id: extractionSpec.createSpecification.id }, tokenThreshold: 32, }, extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Place, ObservableTypes.Event, ObservableTypes.Product, ObservableTypes.Software, ObservableTypes.Repo, ], }, }, ], }, }); console.log('āœ… Workflow created:', workflow.createWorkflow.id); // 3ļøāƒ£ Ingest content with the workflow (extraction happens automatically) console.log('šŸ“„ Ingesting content...'); const content = await graphlit.ingestText( SAMPLE_NOTE, 'Sales Call Notes', undefined, undefined, undefined, undefined, true, // Wait for processing to finish { id: workflow.createWorkflow.id }, ); console.log('āœ… Content ready:', content.ingestText.id); // 4ļøāƒ£ List extracted entities console.log('\nšŸ” Extracted entities:'); const observables = await graphlit.queryObservables(); const results = observables.observables?.results ?? []; const format = (type: ObservableTypes) => results .filter((entry) => entry?.type === type) .map((entry) => entry?.observable.name) .filter(Boolean) as string[]; console.log('šŸ‘„ People:', format(ObservableTypes.Person)); console.log('šŸ¢ Organizations:', format(ObservableTypes.Organization)); console.log('šŸ“ Places:', format(ObservableTypes.Place)); console.log('šŸ› ļø Products/Software:', [ ...format(ObservableTypes.Product), ...format(ObservableTypes.Software), ]); return content.ingestText.id; } main().catch((error) => { console.error(error); process.exit(1); }); ``` **What happens:** * Creates extraction specification (GPT-5, temperature 0 for deterministic results) * Creates workflow with entity extraction * Ingests sample text with the workflow * Automatically extracts entities (people, organizations, places, etc.) * Lists all extracted entities Run: `npx tsx extract-entities.ts` *** ## Step 2: Explore Relationships in the Graph Query content and visualize the entity relationships: ```typescript import { Graphlit } from 'graphlit-client'; const graphlit = new Graphlit(); async function main() { // Query content by name (realistic production pattern) const contents = await graphlit.queryContents({ name: 'Sales Call Notes' }); const contentId = contents.contents?.results?.[0]?.id; if (!contentId) { console.error('Content not found. Run Step 1 first.'); process.exit(1); } console.log('šŸ•øļø Exploring knowledge graph...\n'); // Get the graph for this content const graph = await graphlit.queryContentsGraph({ contents: [{ id: contentId }], }); const nodes = (graph.contents?.graph?.nodes ?? []).filter( (node): node is NonNullable => Boolean(node), ); const edges = (graph.contents?.graph?.edges ?? []).filter( (edge): edge is NonNullable => Boolean(edge), ); const nodeById = new Map(nodes.map((node) => [node.id, node])); console.log('šŸ•øļø Relationships in this document:'); edges.forEach((edge) => { const from = nodeById.get(edge.from); const to = nodeById.get(edge.to); if (!from || !to) return; const relation = edge.relation ? ` ā€”${edge.relation}ā†’ ` : ' ā†’ '; console.log(`${from.name}${relation}${to.name}`); }); } main().catch((error) => { console.error(error); process.exit(1); }); ``` This renders the knowledge graph for a document. Each node represents a person, organization, or event; edges capture relationships Graphlit inferred (e.g., "Sarah Chen ā†’ worksFor ā†’ Acme Corp"). *** ## Step 3: Find Every Document Mentioning an Entity ```typescript import { Graphlit } from 'graphlit-client'; import { ObservableTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); async function findContentForPerson(personName: string) { const observables = await graphlit.queryObservables(); const matches = observables.observables?.results?.filter( (entry) => entry?.type === ObservableTypes.Person && entry.observable.name?.toLowerCase() === personName.toLowerCase(), ); if (!matches?.length) { console.log(`No entities found for ${personName}`); return; } const { id } = matches[0]!.observable; const related = await graphlit.queryContents({ observations: [ { type: ObservableTypes.Person, observable: { id }, }, ], }); const docs = related.contents?.results ?? []; console.log(` šŸ“š Documents mentioning ${personName}:`); docs.forEach((doc) => console.log(`- ${doc?.name} (${doc?.id})`)); } findContentForPerson('Sarah Chen').catch((error) => { console.error(error); process.exit(1); }); ``` Filtering by `observations` lets you answer questions like "Show me every asset where Sarah Chen appears" or "Find all notes referencing Salesforce". *** ## Production Patterns ### Batch Ingestion + Polling ```typescript // Query for your extraction workflow by name const workflows = await graphlit.queryWorkflows({ name: 'Entity Extraction Workflow' }); const workflowId = workflows.workflows?.results?.[0]?.id; // Ingest multiple files const batch = await graphlit.ingestBatch( ['https://example.com/questionnaire.pdf', 'https://example.com/demo-notes.docx'], { id: workflowId }, ); // Poll until processing completes await Promise.all( batch.ingestBatch.ids.map(async ({ id }) => { let done = false; while (!done) { const status = await graphlit.isContentDone(id); done = Boolean(status.isContentDone?.result); if (!done) await new Promise((r) => setTimeout(r, 1500)); } }), ); ``` ### Build Collections from Entities ```typescript const observables = await graphlit.queryObservables(); const organizations = observables.observables?.results?.filter( (entry) => entry?.type === ObservableTypes.Organization, ); if (organizations?.length) { const collection = await graphlit.createCollection({ name: `${organizations[0]!.observable.name} Knowledge Base`, }); const related = await graphlit.queryContents({ observations: [ { type: ObservableTypes.Organization, observable: { id: organizations[0]!.observable.id }, }, ], }); const ids = (related.contents?.results ?? []) .filter(Boolean) .map((item) => ({ id: item!.id })); if (ids.length) { await graphlit.addContentsToCollections(ids, [{ id: collection.createCollection.id }]); } } ``` ### Keep Graph Conversations Focused ```typescript // Query for the organization entity by name const observables = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Organization], name: 'Acme Corp' } }); const targetOrganizationId = observables.observables?.results?.[0]?.observable.id; // Create conversation filtered to that organization const conversation = await graphlit.createConversation({ name: 'Acme Corp CRM Review', filter: { observations: targetOrganizationId ? [ { type: ObservableTypes.Organization, observable: { id: targetOrganizationId }, }, ] : undefined, }, }); const response = await graphlit.promptConversation( 'Summarize our interactions with Acme Corp this quarter.', conversation.createConversation.id, ); console.log(response.promptConversation.message?.message); ``` *** ## Real-World Examples ### 1. Customer 360 * Aggregate Slack, email, and meeting notes per account * Surface key contacts + topics before every call * Feed a sales agent that remembers context across teams ### 2. Incident Response Watchtower * Track mentions of outages across Sentry, Slack, and PagerDuty * Connect incidents to affected services and owners * Push summaries to on-call staff in real time ### 3. Market Intelligence Radar * Monitor competitor names across research docs and news * Attach relationships between mentions, products, and regions * Drive alerts when new entities (people or products) appear *** ## Next Steps * [**Knowledge Graph Use Cases**](/api-guides/use-cases/knowledge-graph) ā€“ Deep-dive patterns for extraction, enrichment, and queries * [**Context Engineering**](/tutorials/context-engineering) ā€“ Feed knowledge graph insights into retrieval *** ## Full Example: Production Agent See the complete Next.js agent in [graphlit-samples](https://github.com/graphlit/graphlit-samples/tree/main/nextjs/chat-graph): * Visual knowledge-graph explorer with streaming chat * Entity filtering and relationship queries in the UI * Production-ready environment configuration and error handling *** **Build agents that understand your data model. Build with Graphlit.** # Context Engineering ā±ļø **Time to Complete:** 15 minutes\ šŸŽÆ **Level:** Intermediate\ šŸ’» **Language:** TypeScript ## What You'll Learn 2025 patterns for semantic memory: * āœ… Form memory intentionally (summaries, entities, embeddings) * āœ… Retrieve the right slice of context every time * āœ… Align conversations to domains, tenants, and entities * āœ… Control context windows and reranking without hand-editing prompts * āœ… Ship production-safe retrieval with the same configuration the Graphlit SDK uses in Zine [**šŸ“ Reference implementation**](https://github.com/graphlit/graphlit-samples/tree/main/nextjs/chat-graph) *** ## Prerequisites * Complete the [Quickstart tutorial](/getting-started/quickstart) * Graphlit credentials configured in `.env` (from Getting Started guide) * `npm install graphlit-client dotenv` {% hint style="info" %} Need Python or .NET samples? Open **Ask Graphlit** inside the Developer Portal (or visit [ask.graphlit.dev](https://ask.graphlit.dev)) and it will translate every call shown here into your SDK of choice. {% endhint %} *** ## Shared Setup ```typescript import { Graphlit } from 'graphlit-client'; import { ConversationSearchTypes, ContentTypes, EntityExtractionServiceTypes, FilePreparationServiceTypes, ModelServiceTypes, ObservableTypes, OpenAiModels, RetrievalStrategyTypes, RerankingModelServiceTypes, SearchTypes, SpecificationTypes, SummarizationTypes, } from 'graphlit-client/dist/generated/graphql-types'; export const graphlit = new Graphlit(); ``` *** ## Pattern 1 ā€“ Form the Right Memory Up Front Create a workflow that produces summaries, extracts entities, and respects your chunk budget before anything hits retrieval. ```typescript export async function createContextWorkflow() { const summarySpec = await graphlit.createSpecification({ name: 'Context Summaries', type: SpecificationTypes.Summarization, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt5Mini_400K, temperature: 0, }, }); const entitiesSpec = await graphlit.createSpecification({ name: 'Context Entities', type: SpecificationTypes.Extraction, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt5_400K, temperature: 0, }, }); const workflow = await graphlit.createWorkflow({ name: 'Context Engineering Workflow', preparation: { jobs: [ { connector: { type: FilePreparationServiceTypes.Document, document: { includeImages: true }, }, }, ], summarizations: [ { type: SummarizationTypes.Summary, tokens: 400, specification: { id: summarySpec.createSpecification.id }, }, ], }, extraction: { jobs: [ { connector: { type: EntityExtractionServiceTypes.ModelText, modelText: { specification: { id: entitiesSpec.createSpecification.id }, tokenThreshold: 48, }, extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Product, ObservableTypes.Event, ], }, }, ], }, }); console.log('Workflow ready:', workflow.createWorkflow.id); return workflow.createWorkflow.id; } ``` **Run once:** ```bash npx tsx scripts/create-context-workflow.ts ``` ### Ingest with the Workflow ```typescript export async function ingestNotebook(workflowId: string) { const response = await graphlit.ingestText( `Acme Corp escalation call with Sarah Chen (CTO) and Mike Rodriguez (VP Engineering). Action items: - Provide dedicated OAuth sandbox credentials. - Ship rate-limit dashboard before Q4 launch. Follow-up demo booked for next Tuesday.`, 'Acme escalation call', undefined, undefined, undefined, undefined, true, { id: workflowId }, ); console.log('Content ingested:', response.ingestText.id); return response.ingestText.id; } ``` Resulting content now ships with: * Summaries referenced by `content.summary` * Extracted observations for people, orgs, products, events * Images preserved for downstream vision models *** ## Pattern 2 ā€“ Retrieval That Matches the Question ### Hybrid vs Keyword vs Vector ```typescript export async function demoSearchModes(prompt: string) { const hybrid = await graphlit.queryContents({ search: prompt, searchType: SearchTypes.Hybrid, limit: 10, }); const keyword = await graphlit.queryContents({ search: prompt, searchType: SearchTypes.Keyword, limit: 10, }); const vector = await graphlit.queryContents({ search: prompt, searchType: SearchTypes.Vector, limit: 10, }); console.log({ hybridMatches: hybrid.contents?.results?.length ?? 0, keywordMatches: keyword.contents?.results?.length ?? 0, vectorMatches: vector.contents?.results?.length ?? 0, }); } ``` Use **Hybrid** by default, fall back to **Keyword** when you need exact IDs or error codes, and **Vector** for conceptual prompts. ### Filter to the Right Domain and Entity ```typescript export async function querySupportIncidents() { // First, find your collection by name (realistic production pattern) const collections = await graphlit.queryCollections({ filter: { name: 'Support Documents' } }); const supportCollectionId = collections.collections?.results?.[0]?.id; // Find your feed by name const feeds = await graphlit.queryFeeds({ filter: { name: 'Support Slack Channel' } }); const supportFeedId = feeds.feeds?.results?.[0]?.id; // Find specific entity (e.g., organization "Acme Corp") const observables = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Organization], name: 'Acme Corp' } }); const acmeOrgId = observables.observables?.results?.[0]?.id; // Now filter content using those IDs const thirtyDaysAgo = new Date(Date.now() - 1000 * 60 * 60 * 24 * 30).toISOString(); const today = new Date().toISOString(); const results = await graphlit.queryContents({ search: 'authentication timeout', searchType: SearchTypes.Hybrid, limit: 12, types: [ContentTypes.Message, ContentTypes.Issue], creationDateRange: { from: thirtyDaysAgo, to: today }, feeds: supportFeedId ? [{ id: supportFeedId }] : undefined, collections: supportCollectionId ? [{ id: supportCollectionId }] : undefined, observations: acmeOrgId ? [ { type: ObservableTypes.Organization, observable: { id: acmeOrgId }, }, ] : undefined, }); return results.contents?.results ?? []; } ``` Filtering on collections, feeds, or observations keeps retrieval scoped to the exact tenant, product line, or account team that matters to the current user. *** ## Pattern 3 ā€“ Specifications That Enforce Context Rules Build a conversation specification once, reuse it everywhere. ```typescript export async function createSupportSpec() { const specification = await graphlit.createSpecification({ name: 'Support Context (Section Retrieval)', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt5_400K, temperature: 0.15, chunkTokenLimit: 480, }, searchType: ConversationSearchTypes.Hybrid, retrievalStrategy: { type: RetrievalStrategyTypes.Section, contentLimit: 8, }, rerankingStrategy: { serviceType: RerankingModelServiceTypes.Cohere, threshold: 0.35, }, }); console.log('Specification ready:', specification.createSpecification.id); return specification.createSpecification.id; } ``` * **Section retrieval** keeps context slices coherent (page/segment level). * **Reranking** uses Cohere to score relevance before prompts see the content. * **chunkTokenLimit** ensures embeddings stay under the LLMā€™s budget. *** ## Pattern 4 ā€“ Conversations that Stay in Bounds ```typescript export async function startSupportConversation( specificationId: string, customerName: string, // e.g., "Acme Corp" ) { // In production: query for resources by name (or pass IDs from your database) const collections = await graphlit.queryCollections({ filter: { name: 'Support Documents' } }); const supportCollectionId = collections.collections?.results?.[0]?.id; const observables = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Organization], name: customerName } }); const customerId = observables.observables?.results?.[0]?.id; const conversation = await graphlit.createConversation({ name: `${customerName} Support Triage`, specification: { id: specificationId }, filter: { collections: supportCollectionId ? [{ id: supportCollectionId }] : undefined, observations: customerId ? [ { type: ObservableTypes.Organization, observable: { id: customerId }, }, ] : undefined, }, }); const response = await graphlit.promptConversation( `Summarize the last three authentication incidents for ${customerName} and list current blockers.`, conversation.createConversation.id, ); console.log(response.promptConversation.message?.message); return conversation.createConversation.id; } ``` The conversation never sees content outside the selected collection or organization. Tenants stay isolated without writing manual guardrails. *** ## Pattern 5 ā€“ Keep the Window Fresh ### Quickly Re-run with Updated Filters ```typescript export async function rehydrateConversation( conversationId: string, collectionId: string, ) { await graphlit.updateConversation({ id: conversationId, filter: { collections: [{ id: collectionId }], }, }); return graphlit.promptConversation( 'What changed since our last sync?', conversationId, undefined, undefined, undefined, undefined, undefined, 'Only report new incidents or updates since the previous answer.', ); } ``` ### Poll for Newly Processed Content Before Prompting ```typescript export async function waitForWorkflow(ids: string[]) { for (const id of ids) { let done = false; while (!done) { const status = await graphlit.isContentDone(id); done = Boolean(status.isContentDone?.result); if (!done) await new Promise((resolve) => setTimeout(resolve, 1500)); } } } ``` Ensuring ingestion is finished before prompting avoids stale context and reduces retries. *** ## Production Checklist * **Persist IDs**: Store specification, workflow, collection, and entity IDs in configuration so every service call reuses the exact same context rules. * **Log retrieval metadata**: Inspect `response.promptConversation.details?.sources` to validate which documents powered answers. * **Segment tenants early**: Collections or feeds per tenant keep context surfaces clean and simplify billing/quotas. * **Tune once**: Adjust retrieval/reranking directly on the specification instead of ad-hoc prompt engineering. * **Fallbacks**: Provide multiple specifications via `fallbacks` if different models handle niche cases better (e.g., legal vs. product queries). *** ## Next Steps * [**Knowledge Graph Tutorial**](/tutorials/knowledge-graph) ā€“ capture richer entities and relationships to feed your context filters. * Try the [Next.js chat-graph sample](https://github.com/graphlit/graphlit-samples/tree/main/nextjs/chat-graph) to see the same configuration running in a UI with streaming responses. *** Build context that respects your product and your customers. Build with Graphlit. # Zine Case Study How [Zine](https://www.zine.ai) demonstrates what's possible when you build on Graphlit. {% hint style="success" %} **Zine is proof Graphlit is production-ready.** Built 100% on Graphlit, Zine shows how to create team memory that actually works - connecting 20+ tools and making everything searchable with AI. {% endhint %} *** ## What is Zine? **The product:** Your team's memory, everywhere you work. **What it does:** * Connects Slack, Gmail, GitHub, Notion, Linear, Jira, Google Drive, and 20+ other tools * Makes everything searchable with semantic AI (not just keyword matching) * Provides MCP server so your IDE can access your team's knowledge * Automatic sync across all connected sources (configurable polling) **The tech stack:** Next.js frontend + Graphlit for everything else. {% hint style="info" %} **Philosophy:** Focus on user experience, not infrastructure. Graphlit handles ingestion, processing, search, and AI. Zine builds the product. {% endhint %} *** ## What Graphlit Enabled Building on Graphlit let Zine focus on user experience, not infrastructure.
What Zine builtUser experience
Next.js frontend
Product features
Deployment
What Graphlit provides30+ feeds
OCR & transcription
Embeddings & search
Entity extraction
Knowledge graph
Multi-tenant isolation
Webhook management
Automatic sync
{% hint style="success" %} **Result:** Ship features, not infrastructure. Zine went from idea to production in weeks, not months. {% endhint %} *** ## The Development Story: AI-Assisted Velocity Zine demonstrates what's possible when you combine Graphlit's platform with modern AI coding tools. **Timeline:** * **March:** First line of code * **6 weeks later:** Working MVP * **June (3 months):** Paying customers * **Today:** Production SaaS with multiple workspaces **How it was built:** * **One developer** (not even a frontend specialist) * AI coding tools: Claude Code, Cline, Factory Droid * Graphlit TypeScript SDK * "Vibe coding" approach - describe what you want, AI implements it {% hint style="info" %} **The 2025 way to build:** One developer + AI tools + great SDKs = what used to require a full team. You don't need to be a full-stack expert. The AI handles implementation, Graphlit handles infrastructure. {% endhint %} **What this enabled:** * Focus on product experience, not infrastructure plumbing * Iterate rapidly based on user feedback * Ship features in days, not weeks * Solo developer building what used to require a full engineering org
Traditional approach6-12 months to build infrastructure
Large engineering team
Stitching together 10+ services
Custom OAuth for each connector
Vector DB management
Scaling challenges
Graphlit + AI coding6 weeks to MVP
One developer (AI-assisted)
One SDK, one platform
30+ feeds work immediately
Production-ready from day one
Focus on user experience
{% hint style="success" %} **Key insight:** The right infrastructure abstractions unlock AI-assisted development. Graphlit's SDK is designed to be AI-coding-friendly. {% endhint %} *** ## The Problem Zine Solves ### Before Zine **Knowledge scattered everywhere:** * šŸ’¬ Slack conversations (decisions, discussions) * šŸ“§ Gmail threads (customer communications) * šŸ“… Meeting recordings (context, action items) * šŸ“ Notion docs (specs, wikis) * šŸ› Linear/Jira issues (product roadmap) * šŸ’» GitHub discussions (technical decisions) **The pain:** * "What did Sarah from Acme Corp say about pricing?" ā†’ Check 5 tools manually * "Why did we make this decision?" ā†’ Lost in Slack history * "What were the action items from last week's call?" ā†’ Scroll through recordings **Time wasted**: Hours per week per person searching for context. *** ### After Zine **One search across everything:** ``` Search: "Acme Corp pricing concerns" Results: šŸ“§ Email (Oct 12): Sarah initial inquiry šŸ’¬ Slack (Oct 13): Team discussion šŸ“… Meeting (Oct 15): Sales call transcript šŸ“ Notion (Oct 16): Deal notes šŸ› Linear (Oct 17): Pricing feature request ``` **Time saved**: Seconds to find anything. **Powered by Graphlit's semantic memory.** *** ## How Zine Works ### Multi-Source Ingestion **Zine creates Graphlit feeds** for each connected tool (Slack, Gmail, GitHub, etc.). **Example:** Connect Slack channel ā†’ Graphlit handles: * OAuth token refresh * Message polling * Thread preservation * File extraction * Deduplication **Result:** Connect once, syncs forever. No webhook infrastructure needed. *** ### Entity Extraction (Knowledge Graph) **Zine enables entity extraction in workflows.** **Graphlit automatically extracts:** * People (e.g., "Sarah Chen, CTO") * Organizations (e.g., "Acme Corp") * Places (e.g., "San Francisco") * Events (e.g., "Q4 Planning Meeting") * Products mentioned **Use case:** "Show me all interactions with Acme Corp" ā†’ Graphlit finds mentions across Slack, email, docs, meetings. *** ### Multi-Tenant Architecture **Pattern:** Each Zine workspace = separate Graphlit environment. **Benefits:** * Complete data isolation per customer * Independent scaling * Usage tracking per workspace * Easy data deletion *** ### Unified Search **Zine uses Graphlit's hybrid search** (vector + keyword) across all connected sources. **Filters available:** * By source (Slack, Gmail, docs, etc.) * By date range * By entities (people, organizations) * By content type **Graphlit handles:** * Vector embeddings * Search ranking * Result deduplication * Cross-source search *** ### AI Chat with Context **Zine creates Graphlit conversations** that automatically retrieve relevant context from the user's knowledge base. **Features:** * Streaming responses * Automatic context injection * Multi-turn memory * Source citations *** ### Meeting Intelligence **Zine ingests meeting recordings** via Graphlit audio processing. **Graphlit provides:** * Audio transcription (Deepgram, AssemblyAI) * Entity extraction from transcripts * Searchable meeting content * Summary generation *** ## What Builders Can Learn from Zine ### 1. AI Coding + Great SDKs = Velocity **The MVP was built in 6 weeks with AI coding tools** (Claude Code, Cline, Factory Droid). Paying customers in 3 months. **Why it worked:** * Graphlit SDK is AI-coding-friendly (clear patterns, TypeScript types) * Platform handles all the hard infrastructure problems * Focus 100% on user experience and product ### 2. Multi-Tenant Architecture Use separate Graphlit environments per customer for complete data isolation. ### 3. Focus on Your Unique Value Don't build OAuth management, feed polling, vector search, or transcription. Let Graphlit handle infrastructure. **Time saved:** Months of infrastructure work ā†’ weeks building product features. ### 4. Entity Extraction from Day One Enable entity extraction in workflows - it unlocks powerful queries like "Show me all interactions with Acme Corp." ### 5. Production Patterns * Retry logic for transient failures * Usage tracking per customer * Background jobs for long operations * Client caching for performance {% hint style="success" %} **Zine's lesson:** Build with platforms, not from scratch. Use AI coding to accelerate. Ship fast, iterate fast. {% endhint %} *** ## Zine's Tech Stack
FrontendNext.js
TypeScript
Tailwind CSS
BackendNext.js API routes
Graphlit TypeScript SDK
Clerk (auth & user metadata)
Redis (caching)
InfrastructureVercel (hosting)
Graphlit (content/AI)
{% hint style="success" %} **What Zine didn't build:** Vector databases, embedding pipelines, search infrastructure, 30+ feed connectors, OAuth management, audio transcription, entity extraction, knowledge graphs. **Result:** Focus 100% on user experience and product features. {% endhint %} *** ## Try Zine or Build Your Own **Want to see it in action?** * [Try Zine](https://www.zine.ai) - Free to start **Want to build your own in weeks, not months?** * [Get Started with Graphlit](/getting-started/quickstart) - Build your first agent in 7 minutes * [Use Case Library](/api-guides/use-cases) - 117+ code examples {% hint style="info" %} **AI-coding-friendly SDK:** Use Claude Code, Cline, Cursor, or Factory Droid with Graphlit's TypeScript, Python, or .NET SDKs. Clear patterns, comprehensive types, great docs. {% endhint %} *** **Zine went from idea to paying customers in 3 months. What will you build?** # Deep Research Learn to build autonomous AI research agents that perform multi-hop web research, entity extraction, and knowledge synthesis. ## Choose Your Framework Build the same deep research agent with your preferred orchestration framework: ### [Mastra (TypeScript)](/examples/deep-research/mastra) **Best for**: TypeScript developers, modern web apps ā±ļø **Time**: 30-40 minutes\ šŸŽÆ **Level**: Advanced **What you'll build**: * Entity-driven research using knowledge graphs * Pre-ingestion filtering with native reranking * Autonomous convergence detection * Multi-source synthesis with citations ### [Agno (Python)](/examples/deep-research/agno) **Best for**: Python developers, high-performance systems ā±ļø **Time**: 30-40 minutes\ šŸŽÆ **Level**: Advanced **Why Agno**: * 5000x faster than LangGraph * 50x less memory usage * Simpler code (just Python functions!) * Same algorithm, cleaner implementation *** ## What All Tutorials Cover Every tutorial teaches the same 5-phase research algorithm, implemented in different frameworks: ### Phase 1: Seed Acquisition Start from a URL or search query to establish initial knowledge base ### Phase 2: Entity-Driven Discovery Extract entities from your knowledge graph (Person, Organization, Category) ### Phase 3: Intelligent Expansion Search web for each entity, **filter before ingesting** (key innovation!) ### Phase 4: Convergence Detection Automatically detect when research has converged (novelty scoring) ### Phase 5: Multi-Source Synthesis Generate comprehensive reports from 100+ sources using summary-based RAG *** ## Key Innovations **1. Pre-Ingestion Filtering** * Analyze 50 sources, ingest only top 8 * Uses Graphlit's native reranker * Significantly faster with better quality **2. Diminishing Returns Detection** * Agent knows when to stop researching * Based on novelty scoring (% new sources in top 10) * No manual intervention needed **3. Summary-Based RAG** * Scales beyond traditional RAG (10-20 docs ā†’ 100+) * Operates on optimized summaries * Fast and accurate *** ## What Graphlit Provides All frameworks use the same Graphlit SDK: āœ… Automatic entity extraction (during ingestion)\ āœ… Knowledge graph (Schema.org/JSON-LD)\ āœ… Native reranker (enables pre-filtering)\ āœ… Exa search (built-in, no API key needed)\ āœ… Summary-based RAG (scalable synthesis)\ āœ… Multi-source citations **Time saved**: 12-14 weeks of infrastructure development *** ## Coming Soon More framework tutorials: * **LangGraph (Python)** - Graph-based state machines * **Vercel Workflow (TypeScript)** - Deterministic, durable orchestration *** ## Choose Your Tutorial **TypeScript developer?** ā†’ [Start with Mastra](/examples/deep-research/mastra) **Python developer?** ā†’ [Start with Agno](/examples/deep-research/agno) # Mastra (TypeScript) Build an autonomous AI research agent that performs multi-hop web research with entity extraction and intelligent filteringā€”like OpenAI's Deep Research ā±ļø **Time**: 30-40 minutes\ šŸŽÆ **Level**: Advanced\ šŸ’» **SDK**: TypeScript (Mastra framework) ## What You'll Learn In this tutorial, you'll build a production-ready research agent that: * āœ… Extracts entities from documents using Graphlit's knowledge graph * āœ… Performs multi-hop web research (searches for discovered entities) * āœ… Filters sources **before ingesting** using native reranking (key innovation!) * āœ… Detects convergence automatically (knows when to stop) * āœ… Synthesizes multi-source reports with citations (scales to 100+ sources) **What makes this production-ready**: Pre-ingestion filtering, autonomous stopping, and summary-based synthesis patterns used in real applications. *** ## What You'll Build An autonomous agent that takes a topic and: 1. **Ingests seed source** - Reads initial document or search results 2. **Discovers entities** - Extracts people, companies, concepts from your knowledge graph 3. **Researches each entity** - Searches Exa for 10 related sources per entity 4. **Filters intelligently** - Analyzes 50 sources, ingests only top 8 high-quality ones 5. **Detects convergence** - Stops when novelty score drops below 30% 6. **Synthesizes report** - Generates comprehensive markdown with proper citations **Example**: Start with Wikipedia on "RAG" ā†’ Extracts 15 entities ā†’ Searches 50 sources ā†’ Filters to 8 ā†’ Generates 2000-word report in \~45 seconds **šŸ”— Full code**: [GitHub](https://github.com/graphlit/graphlit-samples/tree/main/nextjs/mastra-deep-research) *** ## Prerequisites {% hint style="warning" %} **Required**: * Node.js 20+ * [Graphlit account](https://portal.graphlit.dev) + credentials * [OpenAI API key](https://platform.openai.com/api-keys) (for Mastra agent) * Basic TypeScript knowledge **Recommended** (helps understand concepts): * Complete [Quickstart](/getting-started/quickstart) (7 minutes) * Complete [Knowledge Graph tutorial](/tutorials/knowledge-graph) (20 minutes) {% endhint %} *** ## Why This Matters: What Graphlit Handles Before we dive into building, understand what Graphlit provides so you don't have to build it: ### Infrastructure (Weeks ā†’ Hours) * āœ… **File parsing** - PDFs, DOCX, audio, video (30+ formats) * āœ… **Vector database** - Managed Qdrant, auto-scaled * āœ… **Multi-tenant isolation** - Each user gets isolated environment * āœ… **GraphQL API** - Auto-generated, authenticated ### Intelligence (Months ā†’ API Calls) * āœ… **Automatic entity extraction** - LLM-powered workflow extracts Person, Organization, Category during ingestion * āœ… **Knowledge graph** - Built on Schema.org/JSON-LD standard, relationships auto-created * āœ… **Native reranker** - Fast, accurate relevance scoring (this enables our pre-filtering!) * āœ… **Exa search built-in** - No separate API key needed, semantic web search included * āœ… **Summary-based RAG** - Scales to 100+ documents via optimized summaries **Time savings**: Estimated 12-14 weeks of infrastructure development you skip. **Production proof**: This pattern is used in [Zine](/examples/zine-case-study), serving thousands of users with millions of documents. *** ## The Key Innovation: Pre-Ingestion Filtering Most research implementations blindly ingest everything they find. This creates noise and wastes processing. **The breakthrough**: Analyze sources **before** fully ingesting them. Here's the pattern: 1. Quick ingest to temporary collection (lightweight) 2. Use Graphlit's native reranker to score relevance 3. Filter out low-scoring sources (<0.5 relevance) 4. Only fully ingest top 5-8 sources 5. Delete temporary collection **Why this works**: Graphlit's native reranker is fast enough (\~2 seconds) to analyze 50 sources before deciding which to fully process. **Result**: Process 8 sources instead of 50. Faster, higher quality, better signal-to-noise. *** ## The 5-Phase Research Algorithm ### Phase 1: Seed Acquisition Two starting modes: **URL Mode** - Start from a specific source: ```bash pnpm start --url "https://arxiv.org/abs/2005.11401" ``` Best for: Research papers, documentation, whitepapers **Search Mode** - Discover seed sources automatically: ```bash pnpm start --search "retrieval augmented generation" --results 5 ``` Best for: Open-ended research, new topics ### Phase 2: Entity-Driven Discovery Instead of keyword-based research, let the knowledge graph drive discovery: * **Automatic extraction**: Entities extracted during ingestion (no separate step!) * **Types**: Person, Organization, Category (concepts/technical terms) * **Ranking**: By occurrence count and semantic importance * **Selection**: Top 5 become research seeds **Why entity-driven works**: A RAG paper mentions "vector databases" and "BERT"ā€”those naturally become your next research directions. Mimics human researcher behavior. ### Phase 3: Intelligent Expansion For each entity: 1. Search Exa for 10 related sources 2. **Pre-filter before ingesting** (the key innovation!) 3. Only ingest top 3-5 highest-quality sources The filtering workflow: ``` 50 sources found via Exa search ā†“ Quick ingest to temp collection ā†“ Rerank by relevance (native reranker) ā†“ Filter (keep score >0.5) ā†“ Full ingest top 5 only ā†“ Delete temp collection 8 sources ingested total ``` **Benefit**: Analyze 50, process 8. Significantly faster with better quality. ### Phase 4: Convergence Detection The agent automatically detects when to stop: 1. Rerank ALL content by relevance to original query 2. Calculate novelty: What % of newest sources rank in top 10? 3. Decision: * **Novelty >30%**: Continue, sources add value * **Novelty <30%**: Stop, diminishing returns **Why this matters**: No manual intervention needed. The agent knows when research has converged. ### Phase 5: Multi-Source Synthesis Graphlit's summary-based approach scales beyond traditional RAG: 1. **Auto-summarize** each source (25-50 key points + entities) 2. **Concatenate** summaries (efficient context usage) 3. **Synthesize** via LLM from summaries 4. **Citations** automatically included Traditional RAG hits limits at 10-20 docs. This approach handles 100+ sources. *** ## Implementation: Step-by-Step ### Step 1: Project Setup (5 min) ```bash mkdir deep-research && cd deep-research pnpm init -y pnpm add @mastra/core @ai-sdk/openai graphlit-client zod dotenv pnpm add -D typescript tsx pnpm add chalk ora boxen cli-table3 gradient-string ``` Create `.env`: ```env # From portal.graphlit.dev GRAPHLIT_ENVIRONMENT_ID=your_environment_id GRAPHLIT_ORGANIZATION_ID=your_organization_id GRAPHLIT_JWT_SECRET=your_jwt_secret # From platform.openai.com OPENAI_API_KEY=your_openai_key ``` Configure `tsconfig.json`: ```json { "compilerOptions": { "target": "ES2022", "module": "ESNext", "moduleResolution": "node", "outDir": "./dist", "strict": true, "esModuleInterop": true } } ``` ### Step 2: Singleton Graphlit Client (2 min) Create one shared Graphlit instance used by all tools. **File: `src/graphlit.ts`** ```typescript import { Graphlit } from 'graphlit-client'; export const graphlit = new Graphlit(); // Auto-reads env vars, handles auth, ready to use ``` **Why singleton**: Efficient, no redundant credential passing, production pattern. ### Step 3: Build the Key Tools (20 min) We'll build 10 Mastra tools. Here are the critical ones: #### Tool 1: Workflow with Entity Extraction This sets up automatic entity extraction during ingestion. **File: `src/tools/workflow-setup.ts`** ```typescript import { createTool } from '@mastra/core/tools'; import { z } from 'zod'; import { graphlit } from '../graphlit.js'; import { EntityExtractionServiceTypes, ObservableTypes, } from 'graphlit-client/dist/generated/graphql-types'; export const createWorkflowTool = createTool({ id: 'create-workflow', description: 'Create collection and workflow with entity extraction', inputSchema: z.object({ name: z.string(), }), outputSchema: z.object({ collectionId: z.string(), workflowId: z.string(), }), execute: async ({ context }) => { // Create collection for organizing content const collResp = await graphlit.createCollection({ name: context.input.name, }); // Create workflow with automatic entity extraction const wfResp = await graphlit.upsertWorkflow({ name: `${context.input.name} Workflow`, ingestion: { collections: [{ id: collResp.createCollection.id }], }, extraction: { jobs: [ { connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Category, ], extractedCount: 10, }, }, ], }, }); return { collectionId: collResp.createCollection.id, workflowId: wfResp.upsertWorkflow.id, }; }, }); ``` **Key insight**: Entity extraction happens automatically during ingestion. When you query later, entities are already in your knowledge graphā€”no separate extraction step needed. {% hint style="success" %} **Graphlit's Knowledge Graph**: Built on Schema.org/JSON-LD standards, ensuring interoperability and semantic richness beyond simple entity lists. {% endhint %} #### Tool 2: Pre-Ingestion Filtering (The Critical Innovation) This tool analyzes sources before fully ingesting them. **File: `src/tools/rerank.ts`** ```typescript import { createTool } from '@mastra/core/tools'; import { z } from 'zod'; import { graphlit } from '../graphlit.js'; export const filterSearchResultsTool = createTool({ id: 'filter-search-results', description: 'Filter search results BEFORE full ingestion using native reranker', inputSchema: z.object({ searchResults: z.array(z.object({ url: z.string(), title: z.string() })), query: z.string(), maxResults: z.number().default(5), minRelevanceScore: z.number().default(0.5), }), outputSchema: z.object({ filteredUrls: z.array(z.string()), skippedCount: z.number(), reasoning: z.string(), }), execute: async ({ context }) => { // Step 1: Create temporary resources for analysis const tempWorkflow = await graphlit.upsertWorkflow({ name: `Temp Filter ${Date.now()}`, }); const tempCollection = await graphlit.createCollection({ name: `Temp Filter ${Date.now()}`, }); // Step 2: Quick ingestion for analysis only const results = await Promise.all( context.input.searchResults.map(result => graphlit.ingestUri( result.url, result.title, undefined, undefined, true, { id: tempWorkflow.upsertWorkflow.id }, [{ id: tempCollection.createCollection.id }], ).catch(() => null) // Graceful failure ) ); const tempContentIds = results .filter(r => r?.ingestUri?.id) .map(r => r.ingestUri.id); if (tempContentIds.length === 0) { return { filteredUrls: [], skippedCount: context.input.searchResults.length, reasoning: 'Failed to ingest any results for analysis', }; } // Step 3: Use native reranker for relevance scoring const reranked = await graphlit.retrieveSources( context.input.query, { collections: [{ id: tempCollection.createCollection.id }] }, undefined, { type: 'VECTOR', limit: context.input.maxResults * 2 }, { type: 'RERANK' }, // Native reranker! ); // Step 4: Filter by relevance score const highQuality = (reranked.retrieveSources?.results ?? []) .filter(s => !s.score || s.score >= context.input.minRelevanceScore) .slice(0, context.input.maxResults); // Step 5: Map back to original URLs const filteredUrls = []; for (const source of highQuality) { const content = await graphlit.getContent(source.content?.id || ''); if (content.content?.uri) { filteredUrls.push(content.content.uri); } } // Step 6: Clean up temporary resources await graphlit.deleteCollection(tempCollection.createCollection.id); await graphlit.deleteWorkflow(tempWorkflow.upsertWorkflow.id); return { filteredUrls, skippedCount: context.input.searchResults.length - filteredUrls.length, reasoning: `Analyzed ${tempContentIds.length} sources. Kept top ${filteredUrls.length} with relevance >=${context.input.minRelevanceScore}. Skipped ${context.input.searchResults.length - filteredUrls.length} low-quality sources.`, }; }, }); ``` **Why this pattern works**: The native reranker is fast (\~2 seconds) and accurate. Temporary collection analysis is lightweight. This makes pre-filtering practical at scale. #### Tool 3: Diminishing Returns Detection Automatically detects when research has converged. **File: `src/tools/rerank.ts` (continued)** ```typescript export const detectDiminishingReturnsTool = createTool({ id: 'detect-diminishing-returns', description: 'Detect if new sources add value or just repeat existing knowledge', inputSchema: z.object({ collectionId: z.string(), recentContentIds: z.array(z.string()), query: z.string(), }), outputSchema: z.object({ isDiminishing: z.boolean(), noveltyScore: z.number(), recommendation: z.string(), }), execute: async ({ context }) => { // Rerank ALL content by relevance const allRanked = await graphlit.retrieveSources( context.input.query, { collections: [{ id: context.input.collectionId }] }, undefined, { type: 'VECTOR', limit: 50 }, { type: 'RERANK' }, ); const rankedIds = allRanked.retrieveSources?.results?.map(r => r.content?.id) ?? []; const topN = Math.min(10, rankedIds.length); const topRanked = rankedIds.slice(0, topN); // Count how many recent sources rank in top 10 const recentInTop = context.input.recentContentIds.filter(id => topRanked.includes(id) ).length; // Calculate novelty score const noveltyScore = recentInTop / Math.min(topN, context.input.recentContentIds.length); const isDiminishing = noveltyScore < 0.3; return { isDiminishing, noveltyScore, recommendation: isDiminishing ? `Stop - only ${recentInTop}/${topN} recent sources are highly relevant. Diminishing returns detected.` : `Continue - ${recentInTop}/${topN} recent sources rank in top results. Still adding value.`, }; }, }); ``` **The insight**: If new sources don't rank highly vs existing content, they're redundant. The agent stops automatically. {% hint style="info" %} **Full code for all 10 tools**: See the [GitHub repository](https://github.com/graphlit/graphlit-samples/tree/main/nextjs/mastra-deep-research/src/tools) for complete implementations of all tools including ingestion, entity extraction, web search, and report generation. {% endhint %} ### Step 4: Create the Mastra Agent (3 min) Bring all tools together with intelligent orchestration. **File: `src/agent.ts`** ```typescript import { Agent } from '@mastra/core/agent'; import { openai } from '@ai-sdk/openai'; import { createWorkflowTool, ingestDocumentTool, ingestBatchTool, extractEntitiesTool, selectTopEntitiesTool, searchWebTool, filterSearchResultsTool, detectDiminishingReturnsTool, generateReportTool, } from './tools/index.js'; export const deepResearchAgent = new Agent({ name: 'Deep Research Agent', instructions: `You are an autonomous research agent using semantic memory and knowledge graphs. Your workflow: 1. Create workflow + collection with entity extraction 2. Ingest seed URL or search results 3. Extract entities discovered in your knowledge graph 4. Select top 5 entities (focus on PERSON, ORGANIZATION, CATEGORY) 5. Search web for each entity (10 results via Exa) 6. Filter search results BEFORE ingesting (use filterSearchResults tool) 7. Batch ingest only filtered, high-quality sources 8. Check convergence (use detectDiminishingReturns - stop if novelty <30%) 9. Generate comprehensive report with citations Always filter before ingesting to ensure quality and efficiency.`, model: openai('gpt-4o'), tools: { createWorkflow: createWorkflowTool, ingestDocument: ingestDocumentTool, ingestBatch: ingestBatchTool, extractEntities: extractEntitiesTool, selectTopEntities: selectTopEntitiesTool, searchWeb: searchWebTool, filterSearchResults: filterSearchResultsTool, detectDiminishingReturns: detectDiminishingReturnsTool, generateReport: generateReportTool, }, }); ``` **Why agent pattern**: The LLM decides when to use each tool. Adaptive, resilient, production-ready for AI applications. ### Step 5: Build the CLI (5 min) Create a polished interface for running research. **File: `src/main.ts`** (abbreviated - see [full code](https://github.com/graphlit/graphlit-samples/blob/main/nextjs/mastra-deep-research/src/main.ts)) ```typescript #!/usr/bin/env node import { config } from 'dotenv'; import { deepResearchAgent } from './agent.js'; import chalk from 'chalk'; import ora from 'ora'; import boxen from 'boxen'; config(); async function main() { const args = process.argv.slice(2); const url = args[args.indexOf('--url') + 1]; const searchQuery = args[args.indexOf('--search') + 1]; const numResults = parseInt(args[args.indexOf('--results') + 1] || '5'); // Comprehensive validation (env vars, args, formats) // ... see full code for complete validation const spinner = ora('Starting research...').start(); const prompt = url ? `Research starting from: ${url}` : `Research: ${searchQuery} (top ${numResults} seeds)`; const response = await deepResearchAgent.generate(prompt, { onStepFinish: (step) => { spinner.succeed(chalk.green(`āœ… ${step.toolCalls?.[0]?.toolName}`)); spinner.start('Next...'); }, }); spinner.succeed(chalk.green('āœ… Complete!')); console.log('\n' + response.text); // Report to stdout } main(); ``` *** ## Running Your Agent **URL Mode:** ```bash pnpm start --url "https://en.wikipedia.org/wiki/RAG" > report.md ``` **Search Mode:** ```bash pnpm start --search "knowledge graph embeddings" --results 5 > report.md ``` **Expected output:** Terminal (progress): ``` ā•­ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā•® ā”‚ Deep Research Agent ā”‚ ā”‚ Powered by Mastra + Graphlit ā”‚ ā•°ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā•Æ šŸ” Research Query: "knowledge graph embeddings" āœ… searchWeb āœ… ingestBatch āœ… extractEntities āœ… filterSearchResults (kept 4/10) āœ… detectDiminishingReturns (novelty: 0.42 - continue) āœ… generateReport āœ… Complete! ``` Report (`report.md`): ```markdown # Research Report: Knowledge Graph Embeddings ## Executive Summary Knowledge graph embeddings represent entities and relations in continuous vector spaces... [2000 words synthesized from 8 sources] ## References 1. Smith et al. - TransE: Translating Embeddings for Knowledge Graphs https://papers.nips.cc/... 2. ... ``` *** ## Production Patterns ### Performance Optimizations **Parallel processing:** ```typescript // Search all entities concurrently await Promise.allSettled( entities.map(e => searchWebTool.execute({ query: e.name })) ); ``` **Synchronous ingestion:** ```typescript // No polling - content ready when call returns await graphlit.ingestUri(url, undefined, undefined, undefined, true); ``` **Pre-filtering:** ```typescript // Analyze 50, process 8 const filtered = await filterSearchResults(results, query, maxResults: 5); ``` ### Typical Session Metrics **Without filtering:** * Sources processed: \~50 * Processing time: 2-3 minutes * Quality: Significant noise **With filtering:** * Sources processed: \~8 * Processing time: 30-45 seconds * Quality: High signal-to-noise ratio *** ## Alternative Frameworks This tutorial uses Mastra (TypeScript). Graphlit works with other frameworks: **For Python developers:** * **Agno** - Ultra-fast Python agents * **LangGraph** - Graph-based state machines **For TypeScript developers:** * **Vercel AI SDK Workflow** - Deterministic orchestration All use the same Graphlit SDKā€”choose based on language preference. *** ## Next Steps ### Try It Out Clone and run: ```bash git clone https://github.com/graphlit/graphlit-samples.git cd graphlit-samples/nextjs/mastra-deep-research pnpm install cp .env.example .env # Add your credentials pnpm start --search "your query" ``` ### Extend It **Domain-specific entities:** * Medical: `ObservableTypes.MedicalCondition`, `Drug` * Legal: `ObservableTypes.LegalCase`, `Contract` * Business: `ObservableTypes.Product`, `Event` **Multi-pass research:** * Extract entities from Layer 2 results * Research 2-3 passes deep * Configurable depth limits **Real-time monitoring:** * Create Exa feeds for discovered entities * Auto-expand knowledge base daily ### Learn More **Related Tutorials:** * [Knowledge Graph](/tutorials/knowledge-graph) - Deep dive into entity extraction * [Context Engineering](/tutorials/context-engineering) - Advanced retrieval **Production Example:** * [Zine Case Study](/examples/zine-case-study) - Real-world implementation serving thousands of users **Graphlit Resources:** * [Semantic Memory Framework](/platform/semantic-memory) * [Key Concepts](/platform/key-concepts) * [Discord Community](https://discord.gg/ygFmfjy3Qx) **Mastra Resources:** * [Documentation](https://mastra.ai/docs) * [GitHub](https://github.com/mastra-ai/mastra) *** ## Summary You've learned how to build a production-ready autonomous research agent: **Key innovations:** 1. **Pre-ingestion filtering** - Native reranker analyzes sources before processing 2. **Diminishing returns detection** - Agent knows when to stop autonomously 3. **Summary-based synthesis** - Scales to 100+ sources via optimized summaries 4. **Entity-driven discovery** - Knowledge graph powers multi-hop reasoning **Architecture:** * Mastra handles orchestration and tool-calling * Graphlit provides semantic memory, knowledge graph, and intelligence * Clean separation of concerns, production-ready patterns **Time investment**: 30-40 minutes\ **Value delivered**: Weeks of infrastructure work eliminated, battle-tested patterns This approach works for competitive intelligence, market research, technical deep-dives, and any multi-source synthesis use case. *** *Complete implementation:* [*GitHub Repository*](https://github.com/graphlit/graphlit-samples/tree/main/nextjs/mastra-deep-research) # Agno (Python) Build an autonomous AI research agent in Python with Agno and Graphlitā€”5000x faster with simpler code ā±ļø **Time**: 30-40 minutes\ šŸŽÆ **Level**: Advanced\ šŸ’» **SDK**: Python (Agno framework) ## What You'll Learn In this tutorial, you'll build a production-ready research agent in Python that: * āœ… Extracts entities from documents using Graphlit's knowledge graph * āœ… Performs multi-hop web research (searches for discovered entities) * āœ… Filters sources **before ingesting** using native reranking * āœ… Detects convergence automatically (knows when to stop) * āœ… Synthesizes multi-source reports with citations **Why Agno**: 5000x faster than LangGraph, 50x less memory. Simple Python functions as toolsā€”no decorators, no complex schemas. *** ## What You'll Build Same autonomous research agent, Python implementation: 1. **Ingests seed source** - URL or search results 2. **Discovers entities** - From your knowledge graph 3. **Researches each entity** - Exa search, 10 sources per entity 4. **Filters intelligently** - Analyzes 50, ingests only top 8 5. **Detects convergence** - Stops at novelty score <30% 6. **Synthesizes report** - Markdown with citations **Example**: Wikipedia on "RAG" ā†’ 15 entities ā†’ 50 sources ā†’ 8 filtered ā†’ 2000-word report in \~45 seconds **šŸ”— Full code**: [GitHub](https://github.com/graphlit/graphlit-samples/tree/main/python/agno-deep-research) *** ## Prerequisites {% hint style="warning" %} **Required**: * Python 3.11+ * [Graphlit account](https://portal.graphlit.dev) + credentials * [OpenAI API key](https://platform.openai.com/api-keys) * Package manager: [uv](https://docs.astral.sh/uv/getting-started/installation/) (recommended, `curl -LsSf https://astral.sh/uv/install.sh | sh`) or pip **Recommended**: * Complete [Quickstart](/getting-started/quickstart) (7 minutes) * Complete [Knowledge Graph tutorial](/tutorials/knowledge-graph) (20 minutes) {% endhint %} *** ## Why Agno + Python? ### Agno's Advantages **Performance:** * **5000x faster** than LangGraph (\~2-3 microseconds per agent) * **50x less memory** (\~6.5KB per agent vs 325KB) **Simplicity:** * Tools are just Python functions (no decorators!) * No complex schemas (docstrings = tool descriptions) * Clean async/await syntax **Compare Tool Definition:** **Mastra (TypeScript):** ```typescript export const myTool = createTool({ id: 'my-tool', description: 'Tool description', inputSchema: z.object({ /* Zod schema */ }), outputSchema: z.object({ /* Zod schema */ }), execute: async ({ context }) => { /* ... */ }, }); ``` **Agno (Python):** ```python async def my_tool(param: str) -> dict: """Tool description.""" # ... implementation return {"result": "value"} ``` **460 lines of Python vs 750 lines of TypeScript** for the same functionality. *** ## Why This Matters: What Graphlit Handles Before we dive into building, understand what Graphlit provides so you don't have to build it: ### Infrastructure (Weeks ā†’ Hours) * āœ… **File parsing** - PDFs, DOCX, audio, video (30+ formats) * āœ… **Vector database** - Managed Qdrant, auto-scaled * āœ… **Multi-tenant isolation** - Each user gets isolated environment * āœ… **GraphQL API** - Auto-generated, authenticated ### Intelligence (Months ā†’ API Calls) * āœ… **Automatic entity extraction** - LLM-powered workflow extracts Person, Organization, Category during ingestion * āœ… **Knowledge graph** - Built on Schema.org/JSON-LD standard, relationships auto-created * āœ… **Native reranker** - Fast, accurate relevance scoring (enables our pre-filtering!) * āœ… **Exa search built-in** - No separate API key needed, semantic web search included * āœ… **Summary-based RAG** - Scales to 100+ documents via optimized summaries **Time savings**: Estimated 12-14 weeks of infrastructure development you skip. **Production proof**: This pattern is used in [Zine](/examples/zine-case-study), serving thousands of users with millions of documents. *** ## The Key Innovation: Pre-Ingestion Filtering Most research implementations blindly ingest everything they find. This creates noise and wastes processing. **The breakthrough**: Analyze sources **before** fully ingesting them. Here's the pattern: 1. Quick ingest to temporary collection (lightweight) 2. Use Graphlit's native reranker to score relevance 3. Filter out low-scoring sources (<0.5 relevance) 4. Only fully ingest top 5-8 sources 5. Delete temporary collection **Why this works**: Graphlit's native reranker is fast enough (\~2 seconds) to analyze 50 sources before deciding which to fully process. **Result**: Process 8 sources instead of 50. Faster, higher quality, better signal-to-noise. *** ## The 5-Phase Research Algorithm ### Phase 1: Seed Acquisition Two starting modes: **URL Mode** - Start from a specific source: ```bash uv run deep-research --url "https://arxiv.org/abs/2005.11401" ``` Best for: Research papers, documentation, whitepapers **Search Mode** - Discover seed sources automatically: ```bash uv run deep-research --search "retrieval augmented generation" --results 5 ``` Best for: Open-ended research, new topics ### Phase 2: Entity-Driven Discovery Instead of keyword-based research, let the knowledge graph drive discovery: * **Automatic extraction**: Entities extracted during ingestion (no separate step!) * **Types**: Person, Organization, Category (concepts/technical terms) * **Ranking**: By occurrence count and semantic importance * **Selection**: Top 5 become research seeds **Why entity-driven works**: A RAG paper mentions "vector databases" and "BERT"ā€”those naturally become your next research directions. Mimics human researcher behavior. ### Phase 3: Intelligent Expansion For each entity: 1. Search Exa for 10 related sources 2. **Pre-filter before ingesting** (the key innovation!) 3. Only ingest top 3-5 highest-quality sources The filtering workflow: ``` 50 sources found via Exa search ā†“ Quick ingest to temp collection ā†“ Rerank by relevance (native reranker) ā†“ Filter (keep score >0.5) ā†“ Full ingest top 5 only ā†“ Delete temp collection 8 sources ingested total ``` **Benefit**: Analyze 50, process 8. Significantly faster with better quality. ### Phase 4: Convergence Detection Automatically detect when research has plateaued: **Novelty scoring algorithm:** 1. After ingesting new sources, rerank ALL content by relevance to query 2. Check how many recent sources appear in top 10 3. Calculate novelty score: `recent_in_top_10 / total_recent` 4. If score <30%, diminishing returns detected ā†’ stop **Example:** * Ingested 5 new sources * Reranked all 25 total sources * Only 1 new source in top 10 * Novelty: 1/5 = 20% ā†’ **Stop researching** **Why this works**: If new sources don't rank highly vs existing content, they're redundant. Agent stops automatically, no manual intervention. ### Phase 5: Multi-Source Synthesis Traditional RAG struggles beyond 10-20 documents. We scale to 100+: **Summary-based RAG approach:** 1. Create conversation scoped to research collection 2. Use `publish_contents()` which operates on optimized summaries 3. LLM synthesizes across all sources simultaneously 4. Citations automatically included **Python implementation:** ```python # Create conversation conversation = await graphlit.client.create_conversation( input=ConversationInput( name="Research Report", collections=[EntityReferenceInput(id=collection_id)], ) ) # Generate report using publishContents (summary-based) response = await graphlit.client.publish_contents( publish_type=PublishTypes.MARKDOWN, prompt="Synthesize comprehensive report with citations", conversation=EntityReferenceInput(id=conversation.create_conversation.id), ) ``` **Why it scales**: Operates on summaries, not full content. Fast, accurate, handles 100+ sources. *** ## Implementation: Step-by-Step ### Step 1: Project Setup (3 min) **With uv (recommended - faster than pip):** Install uv if you haven't already: ```bash curl -LsSf https://astral.sh/uv/install.sh | sh ``` Create project: ```bash mkdir deep-research && cd deep-research uv init uv add agno graphlit-client python-dotenv rich openai ``` **Or with pip:** ```bash mkdir deep-research && cd deep-research python -m venv venv source venv/bin/activate pip install agno graphlit-client python-dotenv ``` Create `.env`: ```env # From portal.graphlit.dev GRAPHLIT_ENVIRONMENT_ID=your_id GRAPHLIT_ORGANIZATION_ID=your_org GRAPHLIT_JWT_SECRET=your_secret # From platform.openai.com OPENAI_API_KEY=your_key ``` ### Step 2: Singleton Graphlit Client (1 min) **File: `deep_research/graphlit_client.py`** ```python """Singleton Graphlit client.""" from graphlit import Graphlit from dotenv import load_dotenv # Load environment variables first load_dotenv() # One instance, auto-reads env vars graphlit = Graphlit() ``` **Why singleton**: Same pattern as Mastraā€”one shared instance, efficient. **Note**: We load `dotenv` here so environment variables are available when the module is imported. ### Step 3: Build Tools (15 min) Here's where Agno shinesā€”simple Python functions! **File: `deep_research/tools.py`** ```python """Research tools as simple Python functions. Agno advantage: No decorators, no schemas - just functions with docstrings! """ from .graphlit_client import graphlit from graphlit_api import * # Tool 1: Create Workflow async def create_workflow(name: str) -> dict: """Create collection and workflow with automatic entity extraction. Args: name: Name for the research collection Returns: Dict with collection_id and workflow_id """ # Create collection coll_response = await graphlit.client.create_collection( input=CollectionInput(name=name) ) collection_id = coll_response.create_collection.id # Create workflow with entity extraction wf_response = await graphlit.client.upsert_workflow( workflow=WorkflowInput( name=f"{name} Workflow", ingestion=IngestionWorkflowStageInput( collections=[EntityReferenceInput(id=collection_id)] ), extraction=ExtractionWorkflowStageInput( jobs=[ ExtractionWorkflowJobInput( connector=EntityExtractionConnectorInput( type=EntityExtractionServiceTypes.MODELTEXT, extracted_types=[ ObservableTypes.PERSON, ObservableTypes.ORGANIZATION, ObservableTypes.CATEGORY, ], extracted_count=10, ) ) ] ), ) ) return { "collection_id": collection_id, "workflow_id": wf_response.upsert_workflow.id, } # Tool 2: Ingest Document async def ingest_document(url: str, workflow_id: str, collection_id: str) -> dict: """Ingest single document with entity extraction. Args: url: URL to ingest workflow_id: Workflow for processing collection_id: Collection to add to Returns: Dict with content_id """ response = await graphlit.client.ingest_uri( uri=url, is_synchronous=True, # No polling! workflow=EntityReferenceInput(id=workflow_id), collections=[EntityReferenceInput(id=collection_id)], ) return {"content_id": response.ingest_uri.id} ``` **Compare to Mastra:** * No `createTool()` wrapper * No Zod schemas * Docstring = tool description (Agno reads it!) * Type hints = parameter validation * Clean async/await **Tool 3: Pre-Ingestion Filtering** (abbreviated - see [full code](https://github.com/graphlit/graphlit-samples/blob/main/python/agno-deep-research/deep_research/tools.py)): ```python async def filter_search_results( search_results: list[dict], query: str, max_results: int = 5, min_relevance_score: float = 0.5, ) -> dict: """Filter search results BEFORE full ingestion. This is the key innovation - analyze before processing! """ import time # Create temp resources temp_wf = await graphlit.client.upsert_workflow( workflow=WorkflowInput(name=f"Temp Filter {int(time.time())}") ) temp_coll = await graphlit.client.create_collection( input=CollectionInput(name=f"Temp Filter {int(time.time())}") ) # Quick ingestion for analysis import asyncio tasks = [/* parallel ingestion */] results = await asyncio.gather(*tasks, return_exceptions=True) # Rerank with native reranker reranked = await graphlit.client.retrieve_sources( prompt=query, filter=ContentCriteriaInput( collections=[EntityReferenceInput(id=temp_coll.create_collection.id)] ), search=SearchStrategyInput(type=SearchTypes.VECTOR), rerank=RerankStrategyInput(type=RerankTypes.RERANK), ) # Filter and clean up # ... (see full code) return {"filtered_urls": filtered_urls, "reasoning": "..."} ``` **Python advantages**: * `asyncio.gather()` for parallel operations (cleaner than `Promise.allSettled`) * List comprehensions for filtering * Clean exception handling with `return_exceptions=True` {% hint style="info" %} **Full tool implementations**: See [GitHub - tools.py](https://github.com/graphlit/graphlit-samples/blob/main/python/agno-deep-research/deep_research/tools.py) for all 9 tools (\~300 lines). {% endhint %} ### Step 4: Create Agno Agent (2 min) **File: `deep_research/agent.py`** ```python """Deep Research Agent using Agno.""" from agno.agent import Agent from agno.models.openai import OpenAIChat from . import tools research_agent = Agent( name="Deep Research Agent", model=OpenAIChat(id="gpt-4o"), instructions="""You are an autonomous research agent using semantic memory. Your workflow: 1. Create workflow + collection with entity extraction 2. Ingest seed URL or search results 3. Extract entities from knowledge graph 4. Select top 5 entities (PERSON, ORGANIZATION, CATEGORY) 5. Search web for each entity (10 results via Exa) 6. Filter search results BEFORE ingesting (use filter_search_results!) 7. Batch ingest only filtered sources 8. Check convergence (use detect_diminishing_returns - stop if <30%) 9. Generate comprehensive report Always filter before ingesting.""", # Agno: Just list functions - that's it! tools=[ tools.create_workflow, tools.ingest_document, tools.ingest_batch, tools.extract_entities, tools.select_top_entities, tools.search_web, tools.filter_search_results, tools.detect_diminishing_returns, tools.generate_report, ], markdown=True, ) ``` **Agno's simplicity**: No `createTool()`, no tool IDs, no schemas. Just functions. ### Step 5: Build CLI (3 min) **File: `deep_research/main.py`** (abbreviated - see [full code](https://github.com/graphlit/graphlit-samples/blob/main/python/agno-deep-research/deep_research/main.py)): ```python #!/usr/bin/env python3 import asyncio import sys from dotenv import load_dotenv from rich.console import Console from rich.panel import Panel from .agent import research_agent console = Console() async def main(): load_dotenv() # Parse args (same pattern as Mastra) url = None if "--url" not in sys.argv else sys.argv[sys.argv.index("--url") + 1] search_query = None if "--search" not in sys.argv else sys.argv[sys.argv.index("--search") + 1] # Comprehensive validation (env vars, args) # ... validation code ... # Polished header with rich console.print() console.print(Panel.fit( "[bold cyan]Deep Research Agent[/bold cyan]\n" "[dim]Powered by Agno + Graphlit[/dim]", border_style="cyan" )) console.print() console.print(f"[bold]šŸ” Research Query:[/bold] '{search_query}'\n") console.print("[bold green]šŸš€ Starting research...[/bold green]\n") # Run agent with streaming await research_agent.aprint_response(prompt, stream=True) console.print("\n\n[bold green]āœ… Research complete![/bold green]\n") if __name__ == "__main__": asyncio.run(main()) ``` **Python advantages**: * `asyncio.run()` handles event loop (simpler than Node.js setup) * `rich` library for beautiful CLI (like chalk + boxen + ora combined) * `aprint_response()` streams automatically with tool display * Clean, readable code *** ## Running Your Agent **With uv (recommended):** ```bash uv run deep-research --search "knowledge graphs" ``` **With pip:** ```bash pip install -e . deep-research --url "https://en.wikipedia.org/wiki/RAG" ``` **Save to file:** ```bash uv run deep-research --search "AI agents" > report.md ``` **Cleanup after** (deletes collection, workflow, and content): ```bash uv run deep-research --search "test query" --cleanup ``` **Note**: Without `--cleanup`, content remains in your Graphlit account for exploration in the portal. **Alternative commands (all equivalent):** ```bash python -m deep_research --search "query" # Works after install python -m deep_research.main --search "query" # Always works ``` **Expected output:** Terminal (progress): ``` ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā” ā”‚ Deep Research Agent ā”‚ ā”‚ Powered by Agno + Graphlit ā”‚ ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜ šŸ” Research Query: 'knowledge graphs' (Starting with top 5 sources) šŸš€ Starting research... [Tool: create_workflow] āœ“ Created collection and workflow [Tool: search_web] āœ“ Found 5 seed sources [Tool: ingest_batch] āœ“ Ingested 5 sources [Tool: extract_entities] āœ“ Extracted 12 entities [Tool: search_web] āœ“ Searched for 5 entities [Tool: filter_search_results] āœ“ Analyzed 50 sources. Kept 8 (relevance >=0.5) [Tool: ingest_batch] āœ“ Ingested 8 filtered sources [Tool: detect_diminishing_returns] āœ“ Novelty: 0.42 - Continue [Tool: generate_report] # Research Report: Knowledge Graphs ## Executive Summary ... āœ… Research complete! ``` *** ## Production Patterns ### Performance Optimizations **Parallel operations:** ```python import asyncio # Search all entities concurrently tasks = [search_web(entity["name"]) for entity in entities] results = await asyncio.gather(*tasks, return_exceptions=True) ``` **Synchronous ingestion:** ```python # No polling - content ready when call returns await graphlit.client.ingest_uri( uri=url, is_synchronous=True, # Blocks until processed workflow=EntityReferenceInput(id=workflow_id), ) ``` **Graceful error handling:** ```python # Some sources fail? Continue with successful ones results = await asyncio.gather(*tasks, return_exceptions=True) successful = [r for r in results if not isinstance(r, Exception)] ``` **Pre-filtering:** ```python # Analyze 50, ingest only 8 filtered = await filter_search_results( search_results=all_results, query=query, max_results=5, min_relevance_score=0.5 ) ``` ### Typical Session Metrics **Without filtering:** * Sources processed: \~50 * Processing time: 2-3 minutes * Quality: Significant noise **With filtering:** * Sources processed: \~8 * Processing time: 30-45 seconds * Quality: High signal-to-noise ratio **Agno Performance:** * Agent startup: <0.003ms (5000x faster than LangGraph) * Memory usage: \~6.5KB per agent (50x less) * Report generation: 5-10 seconds *** ## Agno vs Other Frameworks | Feature | Agno | Mastra | LangGraph | | ------------------- | --------------- | ------------ | --------------- | | **Language** | Python | TypeScript | Python | | **Speed** | 5000x faster | Fast | Baseline | | **Memory** | 50x less | Standard | Standard | | **Tool Definition** | Just functions | createTool() | @tool decorator | | **Schema Required** | No (docstrings) | Yes (Zod) | Yes (Pydantic) | | **Code Size** | \~460 lines | \~750 lines | \~800 lines | | **Learning Curve** | Easy | Medium | Hard | **Choose Agno when:** * āœ… You prefer Python * āœ… You want maximum performance * āœ… You want simpler code * āœ… You're building high-throughput systems *** ## Next Steps ### Try It Out ```bash git clone https://github.com/graphlit/graphlit-samples.git cd graphlit-samples/python/agno-deep-research uv sync cp .env.example .env # Add credentials uv run python -m deep_research.main --search "your query" ``` ### Extend It **Domain-specific entities:** Medical research: ```python extracted_types=[ ObservableTypes.MEDICALCONDITION, ObservableTypes.DRUG, ObservableTypes.PERSON, # Researchers ] ``` Legal research: ```python extracted_types=[ ObservableTypes.LEGALCASE, ObservableTypes.CONTRACT, ObservableTypes.ORGANIZATION, # Law firms ] ``` Business intelligence: ```python extracted_types=[ ObservableTypes.PRODUCT, ObservableTypes.EVENT, ObservableTypes.ORGANIZATION, # Companies ] ``` **Multi-pass research:** * Extract entities from Layer 2 results * Research 2-3 passes deep * Configurable depth limits **Real-time monitoring:** * Create Exa feeds for discovered entities * Auto-expand knowledge base daily **FastAPI server** (Agno built-in!): ```python from agno.agent import Agent agent = Agent(tools=[...]) agent.app # Built-in FastAPI server! ``` ### Learn More **Related Tutorials:** * [Mastra (TypeScript)](/examples/deep-research/mastra) - Same algorithm, TypeScript * [Knowledge Graph](/tutorials/knowledge-graph) - Entity extraction deep-dive **Production Example:** * [Zine Case Study](/examples/zine-case-study) - Real app serving thousands **Resources:** * [Agno Documentation](https://docs.agno.com) * [Graphlit Documentation](https://docs.graphlit.dev) * [Discord Community](https://discord.gg/ygFmfjy3Qx) *** ## Summary You've learned to build a production-ready autonomous research agent in Python: **Key innovations** (same as Mastra): 1. Pre-ingestion filtering with native reranker 2. Autonomous convergence detection 3. Summary-based RAG for scale 4. Entity-driven discovery **Agno advantages**: * 5000x faster execution * 50x less memory * Simpler code (460 vs 750 lines) * No complex schemas * Clean Python async/await **Time investment**: 30-40 minutes\ **Value delivered**: Production-ready patterns, weeks of infrastructure eliminated This approach works for competitive intelligence, market research, technical deep-dives, and any multi-source synthesis. *** *Complete implementation:* [*GitHub Repository*](https://github.com/graphlit/graphlit-samples/tree/main/python/agno-deep-research) # Key Concepts Core concepts for building AI agents with semantic memory using Graphlit Graphlit provides semantic memory for AI agents. Understanding the core concepts helps you build production AI applications that remember, understand, and reason about information over time. **On this page:** * [Data Model Overview](#data-model-overview) * [Content: The Foundation](#content-the-foundation) * [Feeds: Continuous Data Ingestion](#feeds-continuous-data-ingestion) * [Workflows: Memory Formation Pipeline](#workflows-memory-formation-pipeline) * [Conversations: Accessing Memory](#conversations-accessing-memory) * [Specifications: Controlling AI Behavior](#specifications-controlling-ai-behavior) * [Collections: Organizing Memory](#collections-organizing-memory) * [Knowledge Graph: Semantic Memory Layer](#knowledge-graph-semantic-memory-layer) * [Content Summarization](#content-summarization) * [Content Publishing](#content-publishing) * [Semantic Alerts](#semantic-alerts) *** ## Data Model Overview {% @mermaid/diagram content="graph TB A\[Content Sources] --> B\[Feeds] B --> C\[Workflows] C --> D\[Content] D --> E\[Knowledge Graph] E --> F\[Collections] D --> G\[Conversations] H\[Specifications] --> G ``` style E fill:#4CAF50,color:#fff style D fill:#2196F3,color:#fff" %} ``` Everything in Graphlit flows through this pipeline: sources ā†’ ingestion ā†’ processing ā†’ memory formation ā†’ retrieval. ### Common Confusions Clarified **"What's the difference between Content and Feed?"** * **Content** = Any document/file/text in Graphlit (the data itself) * **Feed** = A connection that continuously adds new content (the sync mechanism) **"Observable vs Entity - same thing?"** * **Entity** = A thing (person, company, place) * **Observable** = An entity + all places it appears across content * Think: Observable = Entity with observation history **"Specification vs Workflow - both configure things?"** * **Workflow** = How to process content (extraction, preparation) * **Specification** = Which AI model to use (GPT-5, Claude, etc.) **"Conversation vs Content - both have text?"** * **Content** = Your data (PDFs, emails, docs) * **Conversation** = Q\&A session about your content with AI **"When do I need a Workflow?"** * **Don't need**: Basic ingestion and search (default works) * **Need**: Extract entities, use vision models, custom processing **"When do I need a Specification?"** * **Don't need**: Default (latest OpenAI) is fine for most use cases * **Need**: Use different model (Claude, Gemini), custom prompts, token limits ### About IDs All Graphlit entities (content, collections, workflows, specifications, conversations, etc.) have unique identifiers: **Format**: GUIDs (Globally Unique Identifiers), also known as UUIDs\ **Example**: `550e8400-e29b-41d4-a716-446655440000` In code examples throughout this documentation, you'll see placeholder IDs like: * `'content-id'` * `'collection-id'` * `'workflow-id'` Replace these with actual GUID values returned from Graphlit API operations. *** ## Content: The Foundation ### What is Content? In semantic memory systems, knowledge exists in unstructured formats: * Documents (PDFs, Word, PowerPoint, Excel) * Audio (MP3, podcasts, meetings, calls) * Video (MP4, recordings, demos) * Web pages (HTML, markdown) * Messages (Slack, Teams, Discord) * Emails (Gmail, Outlook) * Issues (Jira, Linear, GitHub) * Social posts (Twitter, Reddit) When you ingest any of these into Graphlit, we create a **content object** that tracks: * Original source and metadata * Extracted text and structured data * Entities found (people, organizations, events) * Relationships to other content * Temporal context (when created, when ingested) ### Content with Context Each piece of content preserves its full context: * **Source metadata**: Where it came from, when it was created * **Temporal context**: When ingested, last modified * **Structural context**: Relationships to other content * **Semantic context**: Entities and facts extracted from it Some content types are **episodic-like** (specific events in time): * "This meeting recording from Oct 15, 2pm" * "This email sent from Sarah to Mike on Tuesday" * "This Slack message posted in #engineering yesterday" Other content is more **knowledge-based**: * "This documentation about our API" * "This web page explaining GraphQL" * "This PDF white paper on RAG" {% hint style="success" %} **Key insight:** Graphlit preserves the full context of each piece of content - not just text chunks, but metadata, relationships, and extracted knowledge. {% endhint %} [Understanding content types ā†’](/api-guides/use-cases/content/content-type-vs-file-type-explained) *** ## Feeds: Continuous Data Ingestion ### What are Feeds? **Feeds** are automated connectors that continuously ingest content from data sources. Instead of manually uploading each file, create a feed that monitors: * Cloud storage (S3, Azure Blob, Google Cloud, Dropbox, Box, OneDrive, SharePoint) * Communication tools (Slack, Teams, Discord, Twitter/X) * Email (Gmail, Outlook) * Issue trackers (Jira, Linear, GitHub) * Knowledge bases (Notion) * Content (RSS feeds, Reddit, podcasts) * Web (crawling, search, screenshots) ### Sync Modes **One-time sweep**: Ingest everything once * Good for: Initial knowledge base population * Example: "Import all existing SharePoint documents" **Recurring sync**: Check for new content periodically * Good for: Keeping memory up-to-date * Example: "Check Slack #engineering every 5 minutes" * Example: "Monitor Gmail inbox every hour" ### Real-World Pattern: Zine [Zine](https://www.zine.ai) uses 20+ feeds to continuously sync: * Slack channels * Gmail * Google Calendar * Notion pages * Linear issues * GitHub repos * Meeting recordings This creates a **living semantic memory** of everything your team does. [See feed examples ā†’](/api-guides/use-cases/feeds) *** ## Workflows: Memory Formation Pipeline ### What are Workflows? As content enters Graphlit, **workflows** control how raw data becomes semantic memory. This is the **memory formation cycle**: {% @mermaid/diagram content="flowchart LR A\[Ingestion] --> B\[Indexing] B --> C\[Preparation] C --> D\[Extraction] D --> E\[Enrichment] ``` style D fill:#ffe1e1" %} ``` ### Workflow Stages **1. Ingestion** * Filter what content to accept * Configure source-specific settings * Example: "Only ingest PDFs from /docs folder" [See ingestion examples ā†’](/api-guides/use-cases/content) **2. Indexing** * Extract metadata automatically * Document: author, creation date, title * Email: from/to, subject, timestamp * Audio: duration, speaker * Issue: reporter, assignee, status * Index for semantic search (embeddings) * Store raw content [See workflow examples ā†’](/api-guides/use-cases/workflows) **3. Preparation** * Extract text from various formats * Use vision models for PDFs (GPT-4 Vision, Claude Sonnet 3.5) * Transcribe audio (Deepgram, AssemblyAI, Whisper) * Parse HTML/markdown from web pages * Extract structured data [See preparation examples ā†’](/api-guides/use-cases/workflows) **4. Extraction** (Key to Semantic Memory) * **Entity extraction**: Identify people, organizations, places, events * **Relationship mapping**: Connect entities to each other * **Summarization**: Create concise representations * **Knowledge graph**: Build semantic memory layer This is where **raw content** becomes **semantic memory** (structured knowledge with entities and relationships). [See extraction examples ā†’](/api-guides/use-cases/workflows) **5. Enrichment** * Enrich entities with external data (Crunchbase, Wikipedia) * Add domain-specific knowledge * Link to existing entities [See workflow examples ā†’](/api-guides/use-cases/workflows) ### Example Workflow {% tabs %} {% tab title="Python" %} ```python from graphlit import Graphlit from graphlit_api import * graphlit = Graphlit() # Create workflow with vision model for OCR and entity extraction response = await graphlit.client.create_workflow( WorkflowInput( name="PDF with Vision", preparation=PreparationWorkflowStageInput( jobs=[ PreparationWorkflowJobInput( connector=FilePreparationConnectorInput( type=FilePreparationServiceTypes.MODEL_DOCUMENT ) ) ] ), extraction=ExtractionWorkflowStageInput( jobs=[ ExtractionWorkflowJobInput( connector=EntityExtractionConnectorInput( type=EntityExtractionServiceTypes.MODEL_TEXT ) ) ] ) ) ) workflow = response.create_workflow ``` {% endtab %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { FilePreparationServiceTypes, EntityExtractionServiceTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Create workflow with vision model for OCR and entity extraction const workflow = await graphlit.createWorkflow({ name: "PDF with Vision", preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.ModelDocument } }] }, extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText } }] } }); ``` {% endtab %} {% tab title=".NET" %} ```csharp using GraphlitClient; using System.Net.Http; using StrawberryShake; using var httpClient = new HttpClient(); var client = new Graphlit(httpClient); // Create workflow with vision model for OCR and entity extraction var input = new WorkflowInput( name: "PDF with Vision", preparation: new PreparationWorkflowStageInput( jobs: new[] { new PreparationWorkflowJobInput( connector: new FilePreparationConnectorInput( type: FilePreparationServiceTypes.ModelDocument ) ) } ), extraction: new ExtractionWorkflowStageInput( jobs: new[] { new ExtractionWorkflowJobInput( connector: new EntityExtractionConnectorInput( type: EntityExtractionServiceTypes.ModelText ) ) } ) ); var response = await client.CreateWorkflow.ExecuteAsync(input); response.EnsureNoErrors(); var workflow = response.Data?.CreateWorkflow; ``` {% endtab %} {% endtabs %} *** ## Conversations: Accessing Memory ### What are Conversations? **Conversations** let AI agents access your content and knowledge graph to answer questions, complete tasks, and reason about information. This isn't just "Retrieval Augmented Generation (RAG)" - it's semantic memory: * **Stateful**: Conversation history preserved * **Entity-aware**: Understands who/what you're asking about * **Context-aware**: Retrieves relevant memories * **Temporal**: Knows when things happened ### How It Works {% tabs %} {% tab title="Python" %} ```python # Create conversation conversation = await graphlit.client.create_conversation( name="Acme Corp Analysis" ) # Ask questions - memory retrieval automatic response = await graphlit.client.prompt_conversation( prompt="What are Acme Corp's main technical concerns?", id=conversation.create_conversation.id ) # Behind the scenes: # 1. Parses entities: "Acme Corp" (organization) # 2. Queries knowledge graph for related content # 3. Retrieves relevant content (emails, meetings, documents) # 4. Injects semantic memory (entities, relationships) # 5. Generates answer with citations ``` {% endtab %} {% tab title="TypeScript" %} ```typescript // Create conversation const conversation = await graphlit.createConversation({ name: "Acme Corp Analysis" }); // Ask questions - memory retrieval automatic const response = await graphlit.promptConversation({ prompt: "What are Acme Corp's main technical concerns?", id: conversation.createConversation.id }); // Behind the scenes: // 1. Parses entities: "Acme Corp" (organization) // 2. Queries knowledge graph for related content // 3. Retrieves relevant content (emails, meetings, documents) // 4. Injects semantic memory (entities, relationships) // 5. Generates answer with citations ``` {% endtab %} {% tab title=".NET" %} ```csharp // Create conversation var conversation = await graphlit.CreateConversation( name: "Acme Corp Analysis" ); // Ask questions - memory retrieval automatic var response = await graphlit.PromptConversation( prompt: "What are Acme Corp's main technical concerns?", id: conversation.CreateConversation.Id ); // Behind the scenes: // 1. Parses entities: "Acme Corp" (organization) // 2. Queries knowledge graph for related content // 3. Retrieves relevant content (emails, meetings, documents) // 4. Injects semantic memory (entities, relationships) // 5. Generates answer with citations ``` {% endtab %} {% endtabs %} ### Conversations as Working Memory While the conversation is active: * **Working memory**: Current conversation context (in LLM window) * **Long-term memory**: Content, entities, relationships (in knowledge graph) * **Retrieval**: Pull long-term memories into working memory as needed [See conversation examples ā†’](/api-guides/use-cases/conversations) *** ## Specifications: Configuring AI Models ### What are Specifications? **Specifications** configure how AI models process and generate information. {% hint style="info" %} Default: OpenAI GPT-4o (128k context) for conversations {% endhint %} ### What You Can Configure **Model Selection:** * OpenAI (GPT-5, GPT-4o, o4, GPT-4 Turbo) * Anthropic (Claude 4.5 Sonnet, Claude 4 Opus, Claude 3.5) * Google (Gemini 2.5 Pro, Gemini 2.0 Flash) * xAI (Grok 4, Grok 3) * Others (Groq, Mistral, Cohere, DeepSeek) [See all models ā†’](/platform/models) **Tool Calling:** * Define tools/functions the LLM can call * Enable agentic workflows * Connect to external APIs **Conversation Strategies:** * Windowed: Keep last N messages * Summarized: Summarize old messages * Full: Keep everything (until context limit) **Prompt Strategies:** * Rewriting: Improve user prompts * Planning: Break complex tasks into steps * RAG: Configure retrieval parameters ### Example {% tabs %} {% tab title="Python" %} ```python from graphlit import Graphlit from graphlit_api import * graphlit = Graphlit() # Create specification with Claude response = await graphlit.client.create_specification( SpecificationInput( name="Claude 4.5 for Analysis", type=SpecificationTypes.COMPLETION, serviceType=ModelServiceTypes.ANTHROPIC, anthropic=AnthropicModelPropertiesInput( model=AnthropicModels.CLAUDE_4_5_SONNET, temperature=0.2 ) ) ) spec = response.create_specification # Use in conversation response = await graphlit.client.create_conversation( ConversationInput( name="Technical Analysis", specification=EntityReferenceInput(id=spec.id) ) ) conversation = response.create_conversation ``` {% endtab %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { SpecificationTypes, ModelServiceTypes, AnthropicModels } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Create specification with Claude const specResponse = await graphlit.createSpecification({ name: "Claude 4.5 for Analysis", type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.Anthropic, anthropic: { model: AnthropicModels.Claude_4_5Sonnet, temperature: 0.2 } }); const spec = specResponse.createSpecification; // Use in conversation const convResponse = await graphlit.createConversation({ name: "Technical Analysis", specification: { id: spec.id } }); const conversation = convResponse.createConversation; ``` {% endtab %} {% tab title=".NET" %} ```csharp using GraphlitClient; using System.Net.Http; using StrawberryShake; using var httpClient = new HttpClient(); var client = new Graphlit(httpClient); // Create specification with Claude var specInput = new SpecificationInput( name: "Claude 4.5 for Analysis", type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.Anthropic, anthropic: new AnthropicModelPropertiesInput( model: AnthropicModels.Claude_4_5Sonnet, temperature: 0.2 ) ); var specResponse = await client.CreateSpecification.ExecuteAsync(specInput); specResponse.EnsureNoErrors(); var spec = specResponse.Data?.CreateSpecification; // Use in conversation var convInput = new ConversationInput( name: "Technical Analysis", specification: new EntityReferenceInput(id: spec.Id) ); var convResponse = await client.CreateConversation.ExecuteAsync(convInput); convResponse.EnsureNoErrors(); var conversation = convResponse.Data?.CreateConversation; ``` {% endtab %} {% endtabs %} [See specification examples ā†’](/api-guides/use-cases/specifications) *** ## Collections: Organizing Memory ### What are Collections? **Collections** group related content for organization and filtering. Think of them as: * Folders (but content can be in multiple collections) * Tags (but more structured) * Projects (grouping related work) ### Use Cases **By Topic:** * "Product Documentation" * "Customer Feedback" * "Engineering Discussions" **By Source:** * "Acme Corp Content" (all emails, meetings, docs) * "Q4 2024 Planning" * "Architecture Decisions" **By Workflow:** * "Needs Review" * "Published" * "Archived" ### Example {% tabs %} {% tab title="Python" %} ```python from graphlit import Graphlit from graphlit_api import * graphlit = Graphlit() # Create collection response = await graphlit.client.create_collection( CollectionInput( name="Acme Corp" ) ) collection = response.create_collection # Add content during ingestion response = await graphlit.client.ingest_uri( uri="https://example.com/acme-doc.pdf", collections=[EntityReferenceInput(id=collection.id)] ) content = response.ingest_uri # Query by collection response = await graphlit.client.query_contents( filter=ContentFilter( collections=[EntityReferenceFilter(id=collection.id)] ) ) results = response.contents.results ``` {% endtab %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; const graphlit = new Graphlit(); // Create collection const collResponse = await graphlit.createCollection({ name: "Acme Corp" }); const collection = collResponse.createCollection; // Add content during ingestion const ingestResponse = await graphlit.ingestUri( "https://example.com/acme-doc.pdf", undefined, undefined, undefined, false, undefined, [{ id: collection.id }] ); const content = ingestResponse.ingestUri; // Query by collection const queryResponse = await graphlit.queryContents({ collections: [{ id: collection.id }] }); const results = queryResponse.queryContents?.results; ``` {% endtab %} {% tab title=".NET" %} ```csharp using GraphlitClient; using System.Net.Http; using StrawberryShake; using var httpClient = new HttpClient(); var client = new Graphlit(httpClient); // Create collection var collInput = new CollectionInput(name: "Acme Corp"); var collResponse = await client.CreateCollection.ExecuteAsync(collInput); collResponse.EnsureNoErrors(); var collection = collResponse.Data?.CreateCollection; // Add content during ingestion var ingestResponse = await client.IngestUri.ExecuteAsync( uri: "https://example.com/acme-doc.pdf", collections: new[] { new EntityReferenceInput(id: collection.Id) } ); ingestResponse.EnsureNoErrors(); var content = ingestResponse.Data?.IngestUri; // Query by collection var filter = new ContentFilter( collections: new[] { new EntityReferenceFilter(id: collection.Id) } ); var queryResponse = await client.QueryContents.ExecuteAsync(filter); queryResponse.EnsureNoErrors(); var results = queryResponse.Data?.QueryContents?.Results; ``` {% endtab %} {% endtabs %} [See collection examples ā†’](/api-guides/use-cases/collections) *** ## Knowledge Graph: Semantic Memory Layer ### What is the Knowledge Graph? The **knowledge graph** is Graphlit's semantic memory - it stores entities and their relationships, not just documents. This is the key difference between Graphlit and simple RAG systems: * **RAG**: Stores documents, searches by similarity * **Semantic Memory**: Stores entities, searches by meaning and relationships ### Schema.org Foundation Graphlit uses **Schema.org** (JSON-LD) as the knowledge graph foundation: **Why Schema.org?** * Industry standard (Google, Microsoft use it) * Rich vocabulary (Person, Organization, Event, Place, Product, etc.) * Interoperable with other systems * Extensible **Example Entity:** ```json { "@context": "https://schema.org", "@type": "Person", "name": "Sarah Chen", "jobTitle": "CTO", "worksFor": { "@type": "Organization", "name": "Acme Corp" } } ``` ### Observations of Observable Entities **How the graph is built:** 1. **LLM reads content**: "Sarah Chen from Acme Corp mentioned pricing concerns" 2. **Identifies entities**: * Person: Sarah Chen * Organization: Acme Corp * Topic: "pricing concerns" 3. **Creates observations**: * Sarah mentioned in this document * Acme Corp mentioned in this document * Sarah works\_at Acme Corp (relationship) 4. **Links to source**: Observations point to specific content, pages, timestamps **This enables queries like:** * "Show me all content mentioning Sarah Chen" * "Who from Acme Corp have we talked to?" * "What technical issues did CTOs raise in Q4?" ### Observable Types Graphlit extracts these entity types: | Type | Example | Use Case | | ---------------- | -------------------------- | --------------------------- | | **Person** | Sarah Chen, Mike Rodriguez | Track people across sources | | **Organization** | Acme Corp, Google | Company mentions | | **Place** | San Francisco, HQ | Location context | | **Event** | Q4 Planning Meeting | Temporal events | | **Product** | Graphlit API, iPhone | Product mentions | | **Software** | PostgreSQL, Python | Tech stack | | **Repo** | github.com/org/repo | Code references | | **Label** | "bug", "feature-request" | Generic tags | | **Category** | PII classifications | Data categorization | ### Graph Relationships As more content is ingested, relationships become more valuable: **Example:** * Sarah Chen extracted from emails āœ“ * Sarah Chen extracted from Slack messages āœ“ * Sarah Chen extracted from SharePoint docs āœ“ **Query**: "Show me all content related to Sarah Chen" **Result**: Emails + Slack + SharePoint + any other mentions **Query**: "Show me collaboration between Sarah and Mike" **Result**: All content where both appear This is **auto-categorization** through entity recognition. ### GraphRAG: Enhanced Context Retrieval When you ask a question, Graphlit uses the knowledge graph for better context: **Traditional RAG:** 1. User asks: "What did we discuss about the recent earnings?" 2. Vector search for similar content 3. Return chunks 4. Hope it's relevant **GraphRAG (Graphlit):** 1. User asks: "What did we discuss about the recent earnings?" 2. Extract entities from query: "earnings" (topic) 3. Semantic search finds documents 4. Identify commonly observed entities: "CFO" person 5. **Also retrieve** content linked to CFO (Slack, emails, meetings) 6. Inject expanded context into LLM 7. Generate answer with full context **Result**: More relevant, complete answers. *** ## Content Repurposing ### Summarization Generate summaries of content using LLMs: **Built-in methods:** * Summary Paragraphs * Bullet Points * Headlines * Social Media Posts * Follow-up Questions **Custom prompts:** {% tabs %} {% tab title="Python" %} ```python from graphlit import Graphlit from graphlit_api import * graphlit = Graphlit() # Summarize all architecture content response = await graphlit.client.summarize_contents( summarizations=[ SummarizationStrategyInput( type=SummarizationTypes.CUSTOM, prompt="Create a technical summary for engineering team" ) ], filter=ContentFilter(search="architecture") ) summary = response.summarize_contents ``` {% endtab %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { SummarizationTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Summarize all architecture content const response = await graphlit.summarizeContents( [ { type: SummarizationTypes.Custom, prompt: "Create a technical summary for engineering team" } ], { search: "architecture" } ); const summary = response.summarizeContents; ``` {% endtab %} {% tab title=".NET" %} ```csharp using GraphlitClient; using System.Net.Http; using StrawberryShake; using var httpClient = new HttpClient(); var client = new Graphlit(httpClient); // Summarize all architecture content var summarizations = new[] { new SummarizationStrategyInput( type: SummarizationTypes.Custom, prompt: "Create a technical summary for engineering team" ) }; var filter = new ContentFilter(search: "architecture"); var response = await client.SummarizeContents.ExecuteAsync(summarizations, filter); response.EnsureNoErrors(); var summary = response.Data?.SummarizeContents; ``` {% endtab %} {% endtabs %} [See publishing examples ā†’](/api-guides/use-cases/content) *** ### Publishing Transform content into new formats: **Two-step process:** 1. **Summarization**: Each piece of content summarized individually 2. **Publishing**: Summaries combined with publishing prompt **Example:** {% tabs %} {% tab title="Python" %} ```python from graphlit import Graphlit from graphlit_api import * graphlit = Graphlit() # Publish blog post from Q4 collection response = await graphlit.client.publish_contents( publish_prompt="Write a blog post about our Q4 achievements", connector=ContentPublishingConnectorInput( type=ContentPublishingServiceTypes.TEXT, format=ContentPublishingFormats.MARKDOWN ), filter=ContentFilter( collections=[EntityReferenceFilter(id=q4_collection_id)] ) ) published = response.publish_contents ``` {% endtab %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { ContentPublishingFormats, ContentPublishingServiceTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Publish blog post from Q4 collection const response = await graphlit.publishContents( "Write a blog post about our Q4 achievements", { type: ContentPublishingServiceTypes.Text, format: ContentPublishingFormats.Markdown }, undefined, // summaryPrompt undefined, // summarySpecification undefined, // publishSpecification undefined, // name { collections: [{ id: q4_collection_id }] } ); const published = response.publishContents; ``` {% endtab %} {% tab title=".NET" %} ```csharp using GraphlitClient; using System.Net.Http; using StrawberryShake; using var httpClient = new HttpClient(); var client = new Graphlit(httpClient); // Publish blog post from Q4 collection var response = await client.PublishContents.ExecuteAsync( publishPrompt: "Write a blog post about our Q4 achievements", connector: new ContentPublishingConnectorInput( type: ContentPublishingServiceTypes.Text, format: ContentPublishingFormats.Markdown ), filter: new ContentFilter( collections: new[] { new EntityReferenceFilter(id: q4_collection_id) } ) ); response.EnsureNoErrors(); var published = response.Data?.PublishContents; ``` {% endtab %} {% endtabs %} **Or publish as audio:** * Use ElevenLabs text-to-speech * Generate AI podcasts * Create audio summaries [See publishing examples ā†’](/api-guides/use-cases/content) *** ### Alerts **Semantic alerts** are automated, recurring publications: **Use cases:** * Daily email summary of overnight messages * Weekly Slack post of key decisions * Hourly monitoring of customer feedback **Example:** {% tabs %} {% tab title="Python" %} ```python from graphlit import Graphlit from graphlit_api import * graphlit = Graphlit() # Alert: Summarize overnight emails every 24 hours response = await graphlit.client.create_alert( AlertInput( name="Overnight Email Summary", publish_prompt="Summarize key emails with action items", connector=ContentPublishingConnectorInput( type=ContentPublishingServiceTypes.TEXT, format=ContentPublishingFormats.MARKDOWN ), filter=ContentFilter( types=[ContentTypes.EMAIL], created_in_last="PT12H" # Last 12 hours (ISO 8601 duration) ), schedule_policy=AlertSchedulePolicyInput( recurrence_type=TimedPolicyRecurrenceTypes.REPEAT, repeat_interval="PT24H" ) ) ) alert = response.create_alert ``` {% endtab %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { ContentPublishingFormats, ContentPublishingServiceTypes, ContentTypes, TimedPolicyRecurrenceTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Alert: Summarize overnight emails every 24 hours const response = await graphlit.createAlert({ name: "Overnight Email Summary", publishPrompt: "Summarize key emails with action items", connector: { type: ContentPublishingServiceTypes.Text, format: ContentPublishingFormats.Markdown }, filter: { types: [ContentTypes.Email], createdInLast: "PT12H" // Last 12 hours (ISO 8601 duration) }, schedulePolicy: { recurrenceType: TimedPolicyRecurrenceTypes.Repeat, repeatInterval: "PT24H" } }); const alert = response.createAlert; ``` {% endtab %} {% tab title=".NET" %} ```csharp using GraphlitClient; using System.Net.Http; using StrawberryShake; using var httpClient = new HttpClient(); var client = new Graphlit(httpClient); // Alert: Summarize overnight emails every 24 hours var alertInput = new AlertInput( name: "Overnight Email Summary", publishPrompt: "Summarize key emails with action items", connector: new ContentPublishingConnectorInput( type: ContentPublishingServiceTypes.Text, format: ContentPublishingFormats.Markdown ), filter: new ContentFilter( types: new[] { ContentTypes.Email }, createdInLast: "PT12H" // Last 12 hours (ISO 8601 duration) ), schedulePolicy: new AlertSchedulePolicyInput( recurrenceType: TimedPolicyRecurrenceTypes.Repeat, repeatInterval: "PT24H" ) ); var response = await client.CreateAlert.ExecuteAsync(alertInput); response.EnsureNoErrors(); var alert = response.Data?.CreateAlert; ``` {% endtab %} {% endtabs %} [See alert examples ā†’](/api-guides/use-cases/alerts) *** ## Key Takeaways ### Memory, Not Just Documents Graphlit transforms unstructured content into structured memory: * **Content** = Long-term storage (documents, messages, recordings) * **Knowledge Graph** = Semantic memory (entities, facts, relationships) * **Conversations** = Working memory (active context with LLM) * **Episodic context** = Preserved for temporal content (emails, meetings, messages) ### Automated Formation Feeds + Workflows = continuous memory formation: * No manual data entry * Always up-to-date * Scales to millions of documents ### Entity-Centric Knowledge graph enables queries by meaning: * "Show me everything about Acme Corp" (not keyword "Acme") * "Who from enterprise customers raised concerns?" (multi-hop) * "Technical discussions in Q4" (temporal + semantic) ### Production-Ready Built for scale: * Multi-tenant isolation * Real-time ingestion * 30+ feeds * 20+ AI models *** ## Learn More **Understand the Concepts:** * [Semantic Memory](/platform/semantic-memory) - Deep dive on memory vs RAG * [Platform Overview](/getting-started/overview) - Complete platform capabilities * [Connectors](https://github.com/graphlit/graphlit-docs/blob/main/platform/connectors.md) - All data sources * [AI Models](/platform/models) - Model options **Build with Graphlit:** * [Quickstart: Your First Agent](/getting-started/quickstart) - Build a streaming agent in 7 minutes * [AI Agents](/tutorials/ai-agents) - Build agents with memory * [Knowledge Graph](/tutorials/knowledge-graph) - Extract entities **See It in Production:** * [Zine Case Study](/examples/zine-case-study) - Real-world patterns * [Sample Repository](https://github.com/graphlit/graphlit-samples) - 60+ examples *** **Give your AI semantic memory. Start with the core concepts. Build with Graphlit.** # Workflows Complete reference for Graphlit workflows - memory formation pipeline configuration Workflows define **how content is processed** as it flows through Graphlit's memory formation pipeline. This is the authoritative reference for all workflow configuration options, defaults, and decision guidance. **On this page:** * [Overview & Core Concepts](#overview--core-concepts) * [Default Behavior (No Workflow)](#default-behavior-no-workflow) * [When Do You Need a Workflow?](#when-do-you-need-a-workflow) * [Workflow Stages](#workflow-stages) * [Complete API Reference](#complete-api-reference) * [Production Patterns](#production-patterns) *** ## Overview & Core Concepts ### What Workflows Do Workflows control the **memory formation pipeline** - how raw content transforms into structured, searchable semantic memory: {% @mermaid/diagram content="graph LR A\[Raw Content] --> B\[Ingestion Filter] B --> C\[Indexing Config] C --> D\[Preparation] D --> E\[Extraction] E --> F\[Enrichment] F --> G\[Classification] G --> H\[Semantic Memory] ``` style B fill:#E91E63,color:#fff style C fill:#9C27B0,color:#fff style D fill:#2196F3,color:#fff style E fill:#4CAF50,color:#fff style F fill:#FF9800,color:#fff style G fill:#00BCD4,color:#fff" %} ``` **The Six Pipeline Stages (in execution order):** 1. **Ingestion** - Filter which content to accept (file types, paths) 2. **Indexing** - Configure embedding model and vector storage 3. **Preparation** - Extract text/markdown from files (PDFs, audio, images) 4. **Extraction** - Identify entities (people, organizations, topics) 5. **Enrichment** - Add external data (links, FHIR, Diffbot) 6. **Classification** - Categorize content using LLMs **Additional Configuration:** * **Storage** - Where files are stored (defaults to managed storage) * **Actions** - Post-processing webhooks and integrations ### The Workflow Object ```typescript interface WorkflowInput { name: string; // Required: Workflow name // Pipeline stages (in execution order): ingestion?: IngestionWorkflowStageInput; // Optional: Content filtering indexing?: IndexingWorkflowStageInput; // Optional: Custom indexing preparation?: PreparationWorkflowStageInput; // Optional: Text extraction extraction?: ExtractionWorkflowStageInput; // Optional: Entity extraction enrichment?: EnrichmentWorkflowStageInput; // Optional: External enrichment classification?: ClassificationWorkflowStageInput; // Optional: Content classification // Additional configuration: storage?: StorageWorkflowStageInput; // Optional: Storage settings actions?: WorkflowActionInput[]; // Optional: Post-processing actions } ``` **Key insight:** All stages are **optional**. Graphlit has intelligent defaults. *** ## Default Behavior (No Workflow) ### What Happens Without a Workflow ```typescript // Simple ingestion - NO workflow specified await graphlit.ingestUri('https://example.com/document.pdf'); ``` **Graphlit's Default Pipeline:** | Stage | Default Behavior | Speed | | --------------- | -------------------------------------------------------- | --------- | | **Preparation** | āœ… **Intelligent preparation** (see below) | āš” Fast | | **Extraction** | āŒ No entity extraction | āš” Instant | | **Enrichment** | āŒ No external enrichment | āš” Instant | | **Indexing** | āœ… **Project default embedding** (text-embedding-ada-002) | āš” Fast | ### Default Preparation: Intelligent Per-Format Processing Graphlit automatically selects the best preparation method based on content type: **PDFs & Office Documents:** * **Azure AI Document Intelligence (Layout model)** * āœ… Extracts text from PDFs, Word docs, PowerPoint * āœ… OCR for scanned documents * āœ… Basic table recognition * āœ… Layout analysis * āŒ Advanced table parsing * āŒ Image understanding (diagrams, charts) * āŒ Complex multi-column layouts **Audio & Video Files:** * **Deepgram Nova 2 Transcription** * āœ… Automatic transcription * āœ… High accuracy * āœ… Multiple language support * āŒ No speaker diarization (unless you add workflow) **Web Pages:** * **Built-in HTML Parser** * āœ… Smart HTML extraction * āœ… JavaScript rendering (by default) * āœ… Markdown conversion **Email, Text, Markdown:** * **Built-in Parsers** * āœ… Native format support * āœ… Metadata extraction **When default preparation is sufficient:** * Simple text-heavy PDFs (80%+ of documents) * Audio/video transcription without speaker identification * Most web pages * Office documents * Standard email/text content ### Default Indexing: Project Embedding Model **What it does:** * āœ… Creates vector embeddings for semantic search * āœ… Chunks content intelligently * āœ… Stores in vector database **Default model:** OpenAI `text-embedding-ada-002` (if not configured otherwise) *** ## When Do You Need a Workflow? ### Decision Matrix | Goal | Need Workflow? | Stage | Why | | ----------------------------------- | -------------- | ----------- | ------------------------------- | | **Extract text from simple PDF** | āŒ No | - | Default Azure AI is fine | | **Extract text from complex PDF** | āœ… Yes | Preparation | Use vision models (GPT-4o) | | **Handle images/diagrams in PDF** | āœ… Yes | Preparation | Vision models understand images | | **Transcribe audio/video** | āœ… Yes | Preparation | Use Deepgram or Assembly.AI | | **Extract entities (people, orgs)** | āœ… Yes | Extraction | No extraction by default | | **Build knowledge graph** | āœ… Yes | Extraction | Entity extraction required | | **Enrich with external data** | āœ… Yes | Enrichment | Add Diffbot, FHIR, etc. | | **Use custom embedding model** | āœ… Yes | Indexing | Override default embeddings | | **Filter content during ingestion** | āœ… Yes | Ingestion | Path/type filtering | ### Common Scenarios **Scenario 1: Simple Document Q\&A** ```typescript // NO WORKFLOW NEEDED āœ… await graphlit.ingestUri(pdfUrl); // Default preparation + indexing works great const answer = await graphlit.promptConversation({ prompt: 'What are the key points?' }); ``` **Scenario 2: Complex PDFs with Tables** ```typescript // WORKFLOW NEEDED āœ… const workflow = await graphlit.createWorkflow({ name: 'Vision Model Prep', preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.ModelDocument, modelDocument: { specification: { id: gpt4oSpecId } } } }] } }); await graphlit.ingestUri(pdfUrl, undefined, undefined, undefined, true, { id: workflow.createWorkflow.id }); ``` **Scenario 3: Knowledge Graph from Documents** ```typescript // WORKFLOW NEEDED āœ… const workflow = await graphlit.createWorkflow({ name: 'Extract Entities', extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, modelText: { specification: { id: claudeSpecId } } } }] } }); ``` *** ## Workflow Stages ### Preparation Stage **Purpose:** Extract text, markdown, and metadata from raw files. **When you don't need it:** Default Azure AI Document Intelligence handles most documents. **When you need it:** Complex PDFs, audio transcription, high-quality markdown extraction. #### Complete Configuration ```typescript interface PreparationWorkflowStageInput { jobs?: Array; // Preparation connectors summarizations?: Array; // Auto-summarization disableSmartCapture?: boolean; // Disable JS rendering for web pages enableUnblockedCapture?: boolean; // Use unblocked.com for Cloudflare bypass (10x cost) } interface PreparationWorkflowJobInput { connector: FilePreparationConnectorInput; // Required: Preparation method } interface FilePreparationConnectorInput { type: FilePreparationServiceTypes; // Required: Service type fileTypes?: Array; // Optional: Which file types to prepare // Service-specific properties: modelDocument?: ModelDocumentPreparationPropertiesInput; // Vision models deepgram?: DeepgramAudioPreparationPropertiesInput; // Deepgram transcription assemblyAi?: AssemblyAiAudioPreparationPropertiesInput; // Assembly.AI transcription document?: DocumentPreparationPropertiesInput; // Azure AI (explicit) mistral?: MistralDocumentPreparationPropertiesInput; // Mistral OCR reducto?: ReductoDocumentPreparationPropertiesInput; // Reducto email?: EmailPreparationPropertiesInput; // Email parsing page?: PagePreparationPropertiesInput; // Web page extraction } ``` #### FilePreparationServiceTypes (Recommended Order) | Type | Use Case | Speed | Quality | When to Use | | ----------------------------- | ----------------------------------------------------- | ------------ | -------------- | ----------------------------------------- | | `AZURE_DOCUMENT_INTELLIGENCE` | **Default for PDFs** - Most PDFs, Office docs | āš” Fast | ā­ā­ā­ Good | āœ… **Try this first** (automatic default) | | `REDUCTO` | **Specialized PDF extraction** - Better than default | āš” Fast | ā­ā­ā­ā­ Excellent | Try if default isn't good enough | | `MISTRAL_DOCUMENT` | Mistral OCR for documents | āš”āš” Very Fast | ā­ā­ā­ā­ Excellent | Alternative to Reducto | | `MODEL_DOCUMENT` | **Vision LLMs** - General-purpose, not tuned for docs | āš ļø Slower | ā­ā­ā­ā­ Very Good | Advanced: After trying defaults & Reducto | | `DEEPGRAM` | **Default for audio/video** - Transcription | āš” Fast | ā­ā­ā­ā­ Excellent | āœ… **Automatic default** | | `ASSEMBLY_AI` | **Audio/video** with speaker diarization | āš” Fast | ā­ā­ā­ā­ Excellent | Alternative to Deepgram | | `DOCUMENT` | Explicit Azure AI configuration | āš” Fast | ā­ā­ā­ Good | Rarely needed (use default) | | `EMAIL` | Email message parsing | āš” Instant | ā­ā­ā­ā­ā­ Perfect | āœ… **Automatic default** | | `PAGE` | Web page extraction | āš” Fast | ā­ā­ā­ā­ Excellent | āœ… **Automatic default** | #### MODEL\_DOCUMENT: Vision LLMs for Documents **Understanding the Options** Document preparation offers a spectrum of tools, each optimized for different needs and cost profiles: **Azure AI Document Intelligence (Default - $0)** * Automatic OCR and layout analysis * Fast, handles most PDFs and Office documents * Included in your Graphlit subscription **Reducto / Mistral Document ($)** * Specialized PDF extraction engines * Better at complex tables and multi-column layouts * Higher quality than default, but adds per-page cost **Vision LLMs (MODEL\_DOCUMENT - $$$)** * General-purpose vision models (GPT-4o, Claude, Gemini) * Understand content semantically, not just structurally * Can interpret diagrams, charts, and visual relationships * Highest cost (10x more than default), best flexibility **When to use vision LLMs:** * Specialized tools don't capture the visual meaning you need * Documents require semantic understanding of images/diagrams * Need custom prompting or model-specific behavior * Complex visual documents where structure alone isn't enough **Properties:** ```typescript interface ModelDocumentPreparationPropertiesInput { specification?: EntityReferenceInput; // Optional: LLM specification (GPT-4o, Claude, etc.) } ``` **Model Selection:** ```typescript // GPT-4o - Best balance (recommended) const gpt4oSpec = await graphlit.createSpecification({ type: SpecificationTypes.Preparation, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt4O_128K } }); // Claude Sonnet 3.7 - Best for complex documents const claudeSpec = await graphlit.createSpecification({ type: SpecificationTypes.Preparation, serviceType: ModelServiceTypes.Anthropic, anthropic: { model: AnthropicModels.Claude_3_7Sonnet } }); // Gemini 2.0 Flash - Fast and cheap const geminiSpec = await graphlit.createSpecification({ type: SpecificationTypes.Preparation, serviceType: ModelServiceTypes.Google, google: { model: GoogleModels.Gemini_2_0Flash } }); ``` **Cost vs. Capability:** * Vision LLMs cost \~10x more per page than specialized tools * Trade higher cost for semantic understanding and flexibility * Best for documents where visual meaning matters, not just text extraction **Example:** ```typescript const workflow = await graphlit.createWorkflow({ name: 'High-Quality PDF Extraction', preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.ModelDocument, fileTypes: [FileTypes.Document], // Documents (PDF, Word, etc.) modelDocument: { specification: { id: gpt4oSpecId } } } }] } }); ``` #### DEEPGRAM: Audio Transcription (Enhanced) **Default:** Deepgram Nova 2 is used automatically for audio/video files. **When to add workflow (enhance default):** * Enable speaker diarization (identify who's speaking) * Enable PII redaction * Use different Deepgram model * Configure language settings **Properties:** ```typescript interface DeepgramAudioPreparationPropertiesInput { key?: string; // Optional: Deepgram API key (uses project default if not provided) model?: DeepgramModels; // Optional: Transcription model (default: NOVA_2) language?: string; // Optional: BCP 47 language code (e.g., 'en', 'en-US') detectLanguage?: boolean; // Optional: Auto-detect language (default: false) enableSpeakerDiarization?: boolean; // Optional: Identify speakers (default: false) enableRedaction?: boolean; // Optional: Redact PII (default: false) } ``` **Models:** * `NOVA_2` - Best quality (recommended) * `NOVA_2_MEDICAL` - Medical terminology * `NOVA_2_FINANCE` - Financial terminology * `NOVA_2_CONVERSATIONAL_AI` - Real-time conversations * `NOVA_2_VOICEMAIL` - Voicemail transcription * `NOVA_2_VIDEO` - Video content * `NOVA_2_PHONE_CALL` - Phone calls **Example:** ```typescript const workflow = await graphlit.createWorkflow({ name: 'Audio Transcription', preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.Deepgram, fileTypes: [FileTypes.Audio, FileTypes.Video], deepgram: { model: DeepgramModels.Nova2, enableSpeakerDiarization: true, // Identify who's speaking language: 'en-US' } } }] } }); ``` #### ASSEMBLY\_AI: Alternative Audio Transcription **When to use (alternative to default Deepgram):** * Prefer Assembly.AI over Deepgram * Need their specific features * Already have Assembly.AI account/credits **Properties:** ```typescript interface AssemblyAiAudioPreparationPropertiesInput { key?: string; // Optional: Assembly.AI API key model?: AssemblyAiModels; // Optional: Model (default: BEST) language?: string; // Optional: BCP 47 language code detectLanguage?: boolean; // Optional: Auto-detect language enableSpeakerDiarization?: boolean; // Optional: Identify speakers enableRedaction?: boolean; // Optional: Redact PII } ``` **Models:** * `BEST` - Highest accuracy (default) * `NANO` - Fastest, lower cost **Example:** ```typescript const workflow = await graphlit.createWorkflow({ name: 'Meeting Transcription', preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.AssemblyAi, fileTypes: [FileTypes.Audio], assemblyAi: { model: AssemblyAiModels.Best, enableSpeakerDiarization: true, enableRedaction: true // Redact sensitive info } } }] } }); ``` #### Multi-Job Preparation **Use case:** Different file types need different preparation methods. **Example:** ```typescript const workflow = await graphlit.createWorkflow({ name: 'Multi-Format Processing', preparation: { jobs: [ { // Job 1: PDFs with vision model connector: { type: FilePreparationServiceTypes.ModelDocument, fileTypes: [FileTypes.Document], modelDocument: { specification: { id: gpt4oSpecId } } } }, { // Job 2: Audio with Deepgram connector: { type: FilePreparationServiceTypes.Deepgram, fileTypes: [FileTypes.Audio, FileTypes.Video], deepgram: { model: DeepgramModels.Nova2, enableSpeakerDiarization: true } } }, { // Job 3: Images with Azure AI (default for everything else) connector: { type: FilePreparationServiceTypes.AzureDocumentIntelligence, fileTypes: [FileTypes.Image] } } ] } }); ``` #### Web Page Capture Options **Properties (on PreparationWorkflowStageInput):** ```typescript disableSmartCapture?: boolean; // Disable JS rendering (faster, cheaper, but may miss content) enableUnblockedCapture?: boolean; // Use unblocked.com to bypass Cloudflare (10x cost!) ``` **Default:** `disableSmartCapture: false`, `enableUnblockedCapture: false` **When to adjust:** ```typescript // Static HTML sites (faster, cheaper) const workflow = await graphlit.createWorkflow({ name: 'Static HTML Crawl', preparation: { disableSmartCapture: true // Skip JS rendering } }); // Sites with Cloudflare protection const workflow = await graphlit.createWorkflow({ name: 'Cloudflare Bypass', preparation: { enableUnblockedCapture: true // 10x cost, but bypasses protection } }); ``` #### Auto-Summarization During Preparation **Properties:** ```typescript summarizations?: Array; interface SummarizationStrategyInput { type: SummarizationTypes; // Required: Summarization type specification?: EntityReferenceInput; // Optional: LLM for summarization tokens?: number; // Optional: Max summary length items?: number; // Optional: Number of items to summarize } ``` **SummarizationTypes:** * `Chapters` - Transcript chapters * `Headlines` - Extract headlines * `Questions` - Generate questions * `Posts` - Social media posts **Example:** ```typescript const workflow = await graphlit.createWorkflow({ name: 'Prep with Auto-Summary', preparation: { jobs: [{ /* preparation connector */ }], summarizations: [{ type: SummarizationTypes.Chapters, specification: { id: gpt4oSpecId }, tokens: 500 // Max 500 tokens per chapter summary }] } }); ``` *** ### Extraction Stage **Purpose:** Identify and extract entities (people, organizations, places, topics) to build knowledge graph. **Default:** āŒ **No extraction** - entities are NOT extracted unless you add this stage. **When you need it:** * Building knowledge graph * Entity-based search/filtering * Relationship discovery * Semantic understanding beyond keywords #### Complete Configuration ```typescript interface ExtractionWorkflowStageInput { jobs?: Array; // Extraction connectors } interface ExtractionWorkflowJobInput { connector: EntityExtractionConnectorInput; // Required: Extraction method } interface EntityExtractionConnectorInput { type: EntityExtractionServiceTypes; // Required: Service type extractedTypes?: Array; // Optional: Which entity types to extract customTypes?: Array; // Optional: Custom entity types extractedCount?: number; // Optional: Max entities per type (default: 100) fileTypes?: Array; // Optional: Which file types to extract from // Service-specific properties: modelText?: ModelTextExtractionPropertiesInput; // LLM extraction (recommended) modelImage?: ModelImageExtractionPropertiesInput; // Image entity extraction } ``` #### EntityExtractionServiceTypes | Type | Use Case | Quality | | -------------------------------- | -------------------------------------- | --------------- | | `MODEL_TEXT` | **Recommended** - LLM-based extraction | ā­ā­ā­ā­ā­ Excellent | | `AZURE_COGNITIVE_SERVICES_TEXT` | Azure Text Analytics | ā­ā­ā­ Good | | `MODEL_IMAGE` | Extract from images | ā­ā­ā­ā­ Excellent | | `AZURE_COGNITIVE_SERVICES_IMAGE` | Azure Vision | ā­ā­ā­ Good | #### MODEL\_TEXT: LLM Entity Extraction (Recommended) **Properties:** ```typescript interface ModelTextExtractionPropertiesInput { specification?: EntityReferenceInput; // Optional: LLM specification tokenThreshold?: number; // Optional: Min tokens to process (skip short sections) } ``` **Model Selection:** ```typescript // Claude Sonnet 3.7 - Best accuracy (recommended) const claudeSpec = await graphlit.createSpecification({ type: SpecificationTypes.Extraction, serviceType: ModelServiceTypes.Anthropic, anthropic: { model: AnthropicModels.Claude_3_7Sonnet } }); // GPT-4o - Good balance const gpt4oSpec = await graphlit.createSpecification({ type: SpecificationTypes.Extraction, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt4O_128K } }); // Claude Haiku - Fast and cheap const haikuSpec = await graphlit.createSpecification({ type: SpecificationTypes.Extraction, serviceType: ModelServiceTypes.Anthropic, anthropic: { model: AnthropicModels.Claude_3_5Haiku } }); ``` #### Observable Types (Standard Entities) **Complete list of built-in entity types:** ```typescript enum ObservableTypes { // People & Organizations PERSON // People, individuals, names ORGANIZATION // Companies, institutions, groups // Places PLACE // Locations, addresses, landmarks // Products & Services PRODUCT // Products, services, brands // Events & Time EVENT // Events, meetings, occurrences // Creative Works CREATIVE_WORK // Books, articles, movies, music // Concepts & Topics TOPIC // Abstract concepts, themes, subjects // Legal & Financial REGULATION // Laws, regulations, policies // Medical (when using FHIR) MEDICAL_CONDITION MEDICAL_PROCEDURE MEDICATION MEDICAL_TEST // And more... (check SDK for complete list) } ``` #### Custom Entity Types **Use case:** Domain-specific entities not covered by standard types. **Example:** ```typescript const workflow = await graphlit.createWorkflow({ name: 'Legal Document Extraction', extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, modelText: { specification: { id: claudeSpecId } }, // Extract standard entities extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization ], // AND custom legal entities customTypes: [ 'Contract', 'Clause', 'Obligation', 'Deadline', 'Payment Term', 'Jurisdiction', 'Termination Condition', 'Liability Limit' ] } }] } }); ``` #### Complete Example: Knowledge Graph Extraction ```typescript const workflow = await graphlit.createWorkflow({ name: 'Full Knowledge Graph', extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, modelText: { specification: { id: claudeSpecId }, tokenThreshold: 100 // Skip sections < 100 tokens }, extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Place, ObservableTypes.Product, ObservableTypes.Event, ObservableTypes.Label ], extractedCount: 200, // Max 200 entities per type contentTypes: [ContentTypes.File, ContentTypes.Page], fileTypes: [FileTypes.Document] } }] } }); ``` *** ### Enrichment Stage **Purpose:** Add external data to entities (links, FHIR medical data, Diffbot enrichment). **Default:** āŒ **No enrichment** - external data is NOT added unless you configure this. **When you need it:** * Medical applications (FHIR integration) * Web entity enrichment (Diffbot) * Link extraction from content #### Complete Configuration ```typescript interface EnrichmentWorkflowStageInput { jobs?: Array; // Enrichment connectors link?: LinkStrategyInput; // Link extraction configuration } interface EnrichmentWorkflowJobInput { connector: EntityEnrichmentConnectorInput; // Required: Enrichment method } interface EntityEnrichmentConnectorInput { type: EntityEnrichmentServiceTypes; // Required: Service type enrichedTypes?: Array; // Optional: Which entity types to enrich // Service-specific properties: diffbot?: DiffbotEnrichmentPropertiesInput; // Diffbot Knowledge Graph fhir?: FhirEnrichmentPropertiesInput; // FHIR medical data } ``` #### EntityEnrichmentServiceTypes | Type | Use Case | External API | | --------- | ------------------------- | ---------------------- | | `DIFFBOT` | Web entity enrichment | Diffbot API required | | `FHIR` | Medical entity enrichment | FHIR endpoint required | #### Link Extraction **Properties:** ```typescript interface LinkStrategyInput { allowedDomains?: Array; // Optional: Only extract links from these domains excludedDomains?: Array; // Optional: Don't extract links from these domains allowContentDomain?: boolean; // Optional: Allow links from same domain as content extractUri?: boolean; // Optional: Extract HTTP URLs from text (default: true) extractEmail?: boolean; // Optional: Extract email addresses (default: true) extractPhoneNumber?: boolean; // Optional: Extract phone numbers (default: false) } ``` **Example:** ```typescript const workflow = await graphlit.createWorkflow({ name: 'Link Extraction', enrichment: { link: { extractUri: true, extractEmail: true, extractPhoneNumber: true, allowedDomains: ['example.com', 'partner.com'], // Only these domains excludedDomains: ['spam.com', 'ads.com'] // Exclude these } } }); ``` #### FHIR Medical Enrichment **Use case:** Enrich medical entities with FHIR (Fast Healthcare Interoperability Resources) data. **Properties:** ```typescript interface FhirEnrichmentPropertiesInput { endpoint: URL; // Required: FHIR API endpoint } ``` **Example:** ```typescript const workflow = await graphlit.createWorkflow({ name: 'Medical Content with FHIR', extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, modelText: { specification: { id: claudeSpecId } }, extractedTypes: [ ObservableTypes.MedicalCondition, ObservableTypes.MedicalDrug, ObservableTypes.MedicalProcedure ] } }] }, enrichment: { jobs: [{ connector: { type: EntityEnrichmentServiceTypes.Fhir, fhir: { endpoint: 'https://fhir.example.org/api' }, enrichedTypes: [ ObservableTypes.MedicalCondition, ObservableTypes.MedicalDrug ] } }] } }); ``` #### Diffbot Enrichment **Use case:** Enrich entities with data from Diffbot Knowledge Graph. **Properties:** ```typescript interface DiffbotEnrichmentPropertiesInput { token: string; // Required: Diffbot API token } ``` **Example:** ```typescript const workflow = await graphlit.createWorkflow({ name: 'Entity Enrichment with Diffbot', extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, modelText: { specification: { id: claudeSpecId } }, extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Product ] } }] }, enrichment: { jobs: [{ connector: { type: EntityEnrichmentServiceTypes.Diffbot, diffbot: { token: process.env.DIFFBOT_TOKEN! }, enrichedTypes: [ ObservableTypes.Person, ObservableTypes.Organization ] } }] } }); ``` *** ### Indexing Stage **Purpose:** Apply optional indexing connectors (e.g., additional language indexing). **Default:** āœ… **Automatic indexing** is always applied. **Embedding models:** Configure embeddings at the project level (see [Specifications](/platform/specifications)). #### Complete Configuration ```typescript interface IndexingWorkflowStageInput { jobs?: Array; // Indexing connectors } interface IndexingWorkflowJobInput { connector: ContentIndexingConnectorInput; // Required: Indexing method } interface ContentIndexingConnectorInput { type?: ContentIndexingServiceTypes; // Optional: Indexing connector contentType?: ContentTypes; fileType?: FileTypes; } ``` **Most users don't need this stage** - the default indexing works well. Use this only if you need: * Different embedding model than project default * Specialized indexing configuration **Example:** ```typescript const workflow = await graphlit.createWorkflow({ name: 'Azure AI Language Indexing', indexing: { jobs: [{ connector: { type: ContentIndexingServiceTypes.AzureAiLanguage, contentType: ContentTypes.Page } }] } }); ``` *** ### Ingestion Stage **Purpose:** Filter which content gets ingested based on file type, path, or URL patterns. **Default:** āœ… **All content ingested** - no filtering. **When you need it:** * Web crawling (filter by URL path) * Selective file type ingestion * Exclude certain paths #### Complete Configuration ```typescript interface IngestionWorkflowStageInput { filter?: IngestionContentFilterInput; // Optional: Content filter } interface IngestionContentFilterInput { fileTypes?: Array; // Optional: Only ingest these file types fileExtensions?: Array; // Optional: Only ingest these extensions allowedPaths?: Array; // Optional: Regex patterns for allowed paths excludedPaths?: Array; // Optional: Regex patterns for excluded paths } ``` **Example: Web Crawling with Path Filters** ```typescript const workflow = await graphlit.createWorkflow({ name: 'Blog Posts Only', ingestion: { filter: { allowedPaths: ['^/blog/.*', '^/articles/.*'], // Only these paths excludedPaths: ['^/admin/.*', '^/internal/.*'], // Exclude these types: [ContentTypes.Page] // Only web pages } } }); ``` **Example: Selective File Types** ```typescript const workflow = await graphlit.createWorkflow({ name: 'Documents Only', ingestion: { filter: { fileTypes: [FileTypes.Document], fileExtensions: ['pdf', 'docx', 'txt'] } } }); ``` *** ### Storage Stage **Purpose:** Configure where and how content is stored. **Default:** āœ… **Managed storage** - Graphlit handles all storage. **When you need it:** Bring your own storage (Azure Blob, S3, Google Cloud Storage). #### Complete Configuration ```typescript interface StorageWorkflowStageInput { jobs?: Array; // Storage connectors } interface StorageWorkflowJobInput { connector: FileStorageConnectorInput; // Required: Storage method } interface FileStorageConnectorInput { type: FileStorageServiceTypes; // Required: Service type // Service-specific properties: azureBlob?: AzureBlobStoragePropertiesInput; s3?: S3StoragePropertiesInput; google?: GoogleStoragePropertiesInput; } ``` **Most users don't need this** - Graphlit's managed storage is recommended. *** ### Classification Stage **Purpose:** Classify content into categories using LLMs. **Default:** āŒ **No classification** - content is not categorized unless you add this. **When you need it:** * Content routing/filtering * Auto-categorization * Custom taxonomy #### Complete Configuration ```typescript interface ClassificationWorkflowStageInput { connector?: ContentClassificationConnectorInput; // Optional: Classification method } interface ContentClassificationConnectorInput { type: ContentClassificationServiceTypes; // Required: Service type // Service-specific properties: model?: ModelContentClassificationPropertiesInput; } interface ModelContentClassificationPropertiesInput { specification?: EntityReferenceInput; // Optional: LLM for classification rules?: Array; // Required: Classification rules } interface PromptClassificationRuleInput { label: string; // Required: Category label prompt: string; // Required: Classification criteria } ``` **Example:** ```typescript const workflow = await graphlit.createWorkflow({ name: 'Content Classification', classification: { connector: { type: ContentClassificationServiceTypes.Model, model: { specification: { id: gpt4oSpecId }, rules: [ { label: 'Technical', prompt: 'Content contains technical documentation, API references, or code examples' }, { label: 'Business', prompt: 'Content focuses on business strategy, marketing, or sales' }, { label: 'Support', prompt: 'Content is customer support documentation or FAQs' } ] } } } }); ``` *** ### Workflow Actions **Purpose:** Execute actions after content processing (webhooks, integrations). **Default:** āŒ **No actions** - nothing happens after processing unless configured. **When you need it:** * Webhook notifications * Integration triggers * Post-processing automation #### Complete Configuration ```typescript interface WorkflowActionInput { connector: IntegrationConnectorInput; // Required: Integration connector } interface IntegrationConnectorInput { type: IntegrationServiceTypes; // Required: Service type uri?: URL; // Optional: Webhook URL // Service-specific properties: slack?: SlackIntegrationPropertiesInput; teams?: MicrosoftTeamsIntegrationPropertiesInput; email?: EmailIntegrationPropertiesInput; } ``` **Example: Webhook on Completion** ```typescript const workflow = await graphlit.createWorkflow({ name: 'Workflow with Webhook', preparation: { /* ... */ }, actions: [{ connector: { type: IntegrationServiceTypes.WebHook, uri: 'https://api.example.com/webhook/content-processed' } }] }); ``` *** ## Complete API Reference ### WorkflowInput (Top-Level) ```typescript interface WorkflowInput { name: string; // Required preparation?: PreparationWorkflowStageInput; // Optional extraction?: ExtractionWorkflowStageInput; // Optional enrichment?: EnrichmentWorkflowStageInput; // Optional indexing?: IndexingWorkflowStageInput; // Optional ingestion?: IngestionWorkflowStageInput; // Optional storage?: StorageWorkflowStageInput; // Optional classification?: ClassificationWorkflowStageInput; // Optional actions?: Array; // Optional } ``` **All fields except `name` are optional.** Graphlit provides intelligent defaults for missing stages. *** ## Production Patterns ### Pattern 1: Multi-Tenant Workflows ```typescript // Different workflows for different customer tiers const workflows = { free: await graphlit.createWorkflow({ name: 'Free Tier', // Default preparation (Azure AI) // No extraction }), pro: await graphlit.createWorkflow({ name: 'Pro Tier', preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.ModelDocument, modelDocument: { specification: { id: geminiFlashSpecId } } // Cheaper vision model } }] } }), enterprise: await graphlit.createWorkflow({ name: 'Enterprise Tier', preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.ModelDocument, modelDocument: { specification: { id: gpt4oSpecId } } // Best quality } }] }, extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, modelText: { specification: { id: claudeSpecId } } } }] } }) }; // Use workflow based on user tier const workflowId = user.tier === 'enterprise' ? workflows.enterprise.createWorkflow.id : user.tier === 'pro' ? workflows.pro.createWorkflow.id : workflows.free.createWorkflow.id; await graphlit.ingestUri(uri, undefined, undefined, undefined, true, { id: workflowId }); ``` ### Pattern 2: Conditional Workflows (File Type Based) ```typescript // Different workflows for different file types async function ingestWithConditionalWorkflow(uri: string, fileType: FileTypes) { let workflowId: string | undefined; if (fileType === FileTypes.Document) { // Documents (PDF/Word) get vision model workflowId = pdfVisionWorkflowId; } else if (fileType === FileTypes.Audio || fileType === FileTypes.Video) { // Audio/video gets Deepgram workflowId = audioTranscriptionWorkflowId; } else { // Everything else uses default workflowId = undefined; } await graphlit.ingestUri(uri, undefined, undefined, undefined, true, workflowId ? { id: workflowId } : undefined); } ``` ### Pattern 3: Zine Production Pattern **What Zine uses** (from `/home/kirk/projects/zine`): ```typescript // Single workflow for all content const workflow = await graphlit.createWorkflow({ name: 'Zine Production Workflow', preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.ModelDocument, modelDocument: { specification: { id: gpt4oSpecId } } } }] }, extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, modelText: { specification: { id: claudeSpecId } }, extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Label ] } }] }, enrichment: { link: { extractUri: true, extractEmail: true } } }); // Applied to all feeds await graphlit.createFeed({ name: 'Slack Feed', type: FeedTypes.Slack, workflow: { id: workflow.createWorkflow.id }, // ... }); ``` **Key lessons from Zine:** * Single workflow for simplicity * Vision model preparation (complex Slack attachments) * Entity extraction for knowledge graph * Link extraction for context ### Pattern 4: Cost Optimization ```typescript // Use cheapest options that meet quality requirements const costOptimizedWorkflow = await graphlit.createWorkflow({ name: 'Cost Optimized', preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.ModelDocument, fileTypes: [FileTypes.Document], // Documents (PDF, etc.) modelDocument: { specification: { id: geminiFlashSpecId } } // Cheapest vision model } }], // Default Azure AI handles everything else (much cheaper) }, extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, fileTypes: [FileTypes.Document], // Only extract from documents modelText: { specification: { id: haikuSpecId }, // Cheapest LLM tokenThreshold: 500 // Skip short sections (saves money) }, extractedTypes: [ObservableTypes.Person, ObservableTypes.Organization], // Only essential types extractedCount: 50 // Limit entities per type } }] } }); ``` *** ## Summary **Key Takeaways:** 1. **Most workflows are optional** - Graphlit has intelligent defaults 2. **Default preparation is Azure AI Document Intelligence** - works for 80%+ of documents 3. **No extraction by default** - add extraction stage to build knowledge graph 4. **Vision models (MODEL\_DOCUMENT) are 10x more expensive** - only use for complex PDFs 5. **Audio requires explicit workflow** - use Deepgram or Assembly.AI 6. **All stages are composable** - mix and match as needed **When in doubt:** Start without a workflow, add stages only when you hit limitations. *** **Related Documentation:** * [Specifications ā†’](/platform/specifications) - Configure LLMs and embedding models * [Key Concepts ā†’](/platform/key-concepts) - High-level overview * [API Guides: Workflows ā†’](/api-guides/use-cases/workflows) - Code examples # Specifications Complete reference for Graphlit specifications - AI model configuration and behavior control Specifications control **which AI models** Graphlit uses and **how they behave**. This is the authoritative reference for all specification configuration options, defaults, model selection, and parameter tuning. **On this page:** * [Overview & Core Concepts](#overview--core-concepts) * [Default Behavior](#default-behavior) * [When Do You Need a Specification?](#when-do-you-need-a-specification) * [Specification Types](#specification-types) * [Model Service Providers](#model-service-providers) * [Complete API Reference](#complete-api-reference) * [Production Patterns](#production-patterns) *** ## Overview & Core Concepts ### What Specifications Do Specifications answer three fundamental questions: 1. **Which AI model?** (GPT-4o, Claude 4.5 Sonnet, Gemini 2.5 Flash, etc.) 2. **How should it behave?** (temperature, token limits, system prompts) 3. **How should it retrieve?** (RAG strategies, reranking, GraphRAG) {% @mermaid/diagram content="graph LR A\[User Question] --> B\[Specification] B --> C{Which Model?} C --> D\[GPT-4o] C --> E\[Claude 4.5 Sonnet] C --> F\[Gemini 2.5 Flash] D --> G\[With Parameters] E --> G F --> G G --> H\[AI Response] ``` style B fill:#2196F3,color:#fff style G fill:#4CAF50,color:#fff" %} ``` ### The Specification Object ```typescript interface SpecificationInput { name: string; // Required: Specification name type: SpecificationTypes; // Required: What this spec is for serviceType: ModelServiceTypes; // Required: AI provider // Provider-specific configuration (one of these): openAI?: OpenAiModelPropertiesInput; // OpenAI models anthropic?: AnthropicModelPropertiesInput; // Anthropic Claude google?: GoogleModelPropertiesInput; // Google Gemini groq?: GroqModelPropertiesInput; // Groq (ultra-fast) mistral?: MistralModelPropertiesInput; // Mistral models cohere?: CohereModelPropertiesInput; // Cohere models deepseek?: DeepseekModelPropertiesInput; // Deepseek models cerebras?: CerebrasModelPropertiesInput; // Cerebras (ultra-fast) bedrock?: BedrockModelPropertiesInput; // AWS Bedrock azureOpenAI?: AzureOpenAiModelPropertiesInput; // Azure OpenAI azureAI?: AzureAiModelPropertiesInput; // Azure AI replicate?: ReplicateModelPropertiesInput; // Replicate voyage?: VoyageModelPropertiesInput; // Voyage embeddings jina?: JinaModelPropertiesInput; // Jina embeddings xai?: XaiModelPropertiesInput; // xAI (Grok) // Advanced RAG configuration: retrievalStrategy?: RetrievalStrategyInput; // How to retrieve content rerankingStrategy?: RerankingStrategyInput; // How to rerank results graphStrategy?: GraphStrategyInput; // GraphRAG configuration revisionStrategy?: RevisionStrategyInput; // Self-revision // Customization: systemPrompt?: string; // Override system prompt customInstructions?: string; // Custom instructions customGuidance?: string; // Custom guidance searchType?: ConversationSearchTypes; // VECTOR, KEYWORD, HYBRID strategy?: ConversationStrategyInput; // Message history strategy } ``` **Key insight:** Most of this is optional. Graphlit has intelligent defaults. *** ## Default Behavior ### What Happens Without a Specification ```typescript // NO specification - uses project defaults const answer = await graphlit.promptConversation({ prompt: 'What are the key points?' }); ``` **Graphlit's Defaults:** | Use Case | Default Model | Default Type | | ------------------------ | ----------------------------------------------------- | -------------- | | **RAG Conversations** | Project default (usually GPT-4o or Claude 4.5 Sonnet) | Completion | | **Embeddings** | text-embedding-ada-002 | TextEmbedding | | **Entity Extraction** | No default (must configure workflow) | Extraction | | **Document Preparation** | No default (must configure workflow) | Preparation | | **Summarization** | Project default | Summarization | | **Classification** | No default (must configure workflow) | Classification | **Project defaults** are configured in the Developer Portal and apply to all conversations unless overridden. *** ## When Do You Need a Specification? ### Decision Matrix | Goal | Need Specification? | Specification Type | | --------------------------------------- | ------------------- | ---------------------------- | | **Basic RAG conversations** | āŒ No | Project default works | | **Use different model (Claude vs GPT)** | āœ… Yes | Completion | | **Adjust temperature/creativity** | āœ… Yes | Completion | | **Custom system prompts** | āœ… Yes | Completion | | **Better embeddings** | āœ… Yes | TextEmbedding | | **Change embedding dimensions** | āœ… Yes | TextEmbedding | | **Extract entities** | āœ… Yes | Extraction (in workflow) | | **Use vision for PDFs** | āœ… Yes | Preparation (in workflow) | | **Custom summarization** | āœ… Yes | Summarization | | **Classify content** | āœ… Yes | Classification (in workflow) | ### Common Scenarios **Scenario 1: Default RAG Works** ```typescript // NO specification needed āœ… const answer = await graphlit.promptConversation({ prompt: 'Explain the API' }); // Uses project default (GPT-4o or Claude 4.5 Sonnet) ``` **Scenario 2: Want Different Model** ```typescript // SPECIFICATION NEEDED āœ… const claudeSpec = await graphlit.createSpecification({ name: 'Claude 4.5 Sonnet', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.Anthropic, anthropic: { model: AnthropicModels.Claude_4_5Sonnet } }); const answer = await graphlit.promptConversation({ prompt: 'Explain the API', specification: { id: claudeSpec.createSpecification.id } }); ``` **Scenario 3: Fine-Tuned Behavior** ```typescript // SPECIFICATION NEEDED āœ… const customSpec = await graphlit.createSpecification({ name: 'Creative Writing', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt4O_128K, temperature: 0.9, // More creative completionTokenLimit: 4000 // Longer responses }, systemPrompt: 'You are a creative storyteller who writes in a poetic, engaging style.' }); ``` *** ## Specification Types ### Complete Type Reference ```typescript enum SpecificationTypes { COMPLETION // RAG conversations, chat, Q&A TEXT_EMBEDDING // Vector embeddings for semantic search EXTRACTION // Entity extraction (workflows) PREPARATION // Document preparation (workflows) SUMMARIZATION // Content summarization CLASSIFICATION // Content classification (workflows) IMAGE_EMBEDDING // Image embeddings (advanced) } ``` *** ## COMPLETION Specifications **Purpose:** Control LLM behavior for RAG conversations, chat, and Q\&A. **When you need it:** * Use different model than project default * Adjust creativity (temperature) * Limit response length (token limits) * Custom system prompts * Advanced RAG strategies **Where it's used:** * `promptConversation()` * `streamAgent()` * `promptAgent()` * `createConversation()` (set default for conversation) ### Model Selection Guide | Model | Best For | Speed | Context | Strengths | | ---------------------- | -------------------- | ------------- | ------- | ------------------------------------- | | **GPT-4o** | Balanced all-around | āš”āš” Fast | 128K | Best default, handles most tasks well | | **Claude 4.5 Sonnet** | Citation accuracy | āš” Moderate | 200K | Best for RAG, accurate citations | | **Claude 4.5 Opus** | Maximum quality | āš ļø Slower | 200K | Complex reasoning, highest capability | | **Gemini 2.5 Flash** | Speed + long docs | āš”āš”āš” Very Fast | 1M | Huge context, very fast | | **Gemini 2.5 Pro** | Reasoning + thinking | āš”āš” Fast | 1M | Extended thinking, strong reasoning | | **GPT-4o Mini** | Cost optimization | āš”āš”āš” Very Fast | 128K | Simple Q\&A, budget-conscious | | **Groq Llama 3.3** | Ultra-fast inference | āš”āš”āš”āš” Ultra | 128K | Real-time, latency-sensitive | | **Deepseek V3** | Quality + value | āš”āš” Fast | 64K | Strong performance, lower cost | | **Cerebras Llama 3.3** | Blazing speed | āš”āš”āš”āš” Ultra | 128K | Fastest inference available | | **OpenAI o1** | Deep reasoning | āš ļøāš ļø Slow | 128K | Math, code, complex problems | ### Complete Parameters #### OpenAI Configuration ```typescript interface OpenAiModelPropertiesInput { model: OpenAiModels; // Required: Which OpenAI model temperature?: number; // Optional: 0-2 (default: 0.5) probability?: number; // Optional: Top-p sampling 0-1 (default: 1) completionTokenLimit?: number; // Optional: Max response tokens chunkTokenLimit?: number; // Optional: Chunk size for embeddings (default: 600) reasoningEffort?: OpenAiReasoningEffortLevels; // Optional: For o1/o3 models (LOW, MEDIUM, HIGH) detailLevel?: OpenAiVisionDetailLevels; // Optional: For vision (LOW, HIGH, AUTO) // Bring your own key (optional): key?: string; // Your OpenAI API key endpoint?: URL; // Custom endpoint (for compatible APIs) modelName?: string; // Custom model name tokenLimit?: number; // Custom model token limit } ``` **Available OpenAI Models:** * `GPT4O_128K` - GPT-4o (Latest, recommended) * `GPT4O_MINI_128K` - GPT-4o Mini (Fast, cheap) * `GPT4O_CHAT_128K` - ChatGPT-4o * `O1` - o1 reasoning model * `O1_MINI` - o1-mini reasoning model * `O1_PREVIEW` - o1-preview * `O3_MINI` - o3-mini reasoning model **Example:** ```typescript const gpt4oSpec = await graphlit.createSpecification({ name: 'GPT-4o Production', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt4O_128K, temperature: 0.2, // Mostly factual completionTokenLimit: 3000 // ~2250 words max } }); ``` #### Anthropic Configuration ```typescript interface AnthropicModelPropertiesInput { model: AnthropicModels; // Required: Which Claude model temperature?: number; // Optional: 0-1 (default: 0.5) probability?: number; // Optional: Top-p sampling completionTokenLimit?: number; // Optional: Max response tokens (maxTokens in Claude API) chunkTokenLimit?: number; // Optional: Chunk size (default: 600) enableThinking?: boolean; // Optional: Extended thinking (Claude 3.7+) thinkingTokenLimit?: number; // Optional: Max thinking tokens // Bring your own key (optional): key?: string; // Your Anthropic API key modelName?: string; // Custom model name tokenLimit?: number; // Custom model token limit } ``` **Available Anthropic Models:** * `CLAUDE_4_5_SONNET` - Claude 4.5 Sonnet (Latest, best for RAG) * `CLAUDE_4_5_OPUS` - Claude 4.5 Opus (Highest quality) * `CLAUDE_4_5_HAIKU` - Claude 4.5 Haiku (Fast, cheap) * `CLAUDE_4_1_OPUS` - Claude 4.1 Opus * `CLAUDE_3_7_SONNET` - Claude 3.7 Sonnet (with thinking) * `CLAUDE_3_5_HAIKU` - Claude 3.5 Haiku **Example:** ```typescript const claudeSpec = await graphlit.createSpecification({ name: 'Claude 4.5 Sonnet with Thinking', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.Anthropic, anthropic: { model: AnthropicModels.Claude_4_5Sonnet, temperature: 0.1, // Very factual completionTokenLimit: 4000, enableThinking: true, // Better reasoning thinkingTokenLimit: 8000 // Allow up to 8K thinking tokens } }); ``` #### Google Configuration ```typescript interface GoogleModelPropertiesInput { model: GoogleModels; // Required: Which Gemini model temperature?: number; // Optional: 0-2 probability?: number; // Optional: Top-p sampling completionTokenLimit?: number; // Optional: Max response tokens chunkTokenLimit?: number; // Optional: Chunk size enableThinking?: boolean; // Optional: Extended thinking (Gemini 2.5+) thinkingTokenLimit?: number; // Optional: Max thinking tokens // Bring your own key (optional): key?: string; // Your Google API key modelName?: string; // Custom model name tokenLimit?: number; // Custom model token limit } ``` **Available Google Models:** * `GEMINI_2_5_FLASH` - Gemini 2.5 Flash (Fast, 1M context, thinking) * `GEMINI_2_5_PRO` - Gemini 2.5 Pro (Highest quality, thinking) * `GEMINI_2_0_FLASH` - Gemini 2.0 Flash (Fast, 1M context) * `GEMINI_1_5_PRO` - Gemini 1.5 Pro * `GEMINI_1_5_FLASH` - Gemini 1.5 Flash **Example:** ```typescript const geminiSpec = await graphlit.createSpecification({ name: 'Gemini 2.5 Flash', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.Google, google: { model: GoogleModels.Gemini_2_5Flash, temperature: 0.3, completionTokenLimit: 8000, enableThinking: true, thinkingTokenLimit: 10000 } }); ``` ### Parameter Deep Dive #### Temperature: Control Randomness ```typescript // Factual Q&A (deterministic) temperature: 0.1 // Very consistent, factual responses // Balanced (default) temperature: 0.5 // Good mix of accuracy and variety // Creative writing temperature: 0.9 // More random, creative responses // Maximum creativity (OpenAI only) temperature: 2.0 // Very random (rarely useful) ``` **Use cases:** * **0.0-0.2** - Technical documentation, factual Q\&A, code generation * **0.3-0.7** - General conversations, balanced responses * **0.8-1.0** - Creative writing, brainstorming, diverse outputs #### Probability (Top-P): Token Selection Controls which tokens the model considers: * `0.1` - Only top 10% most likely tokens (very focused) * `0.5` - Top 50% probable tokens (focused) * `0.9` - Top 90% probable tokens (diverse) * `1.0` - All tokens considered (default) **Relationship with Temperature:** * Low temperature + low probability = Very deterministic * High temperature + high probability = Very creative #### Completion Token Limit: Response Length ```typescript // Short answers (summaries, quick responses) completionTokenLimit: 500 // ~375 words // Medium answers (default) completionTokenLimit: 2000 // ~1500 words // Long-form content (articles, detailed explanations) completionTokenLimit: 4000 // ~3000 words // Very long (comprehensive documents) completionTokenLimit: 8000 // ~6000 words // Maximum output (model-dependent) completionTokenLimit: 16000 // GPT-4o/Claude max ``` **Important:** This limits OUTPUT only, not the context window. #### Advanced Parameters **Reasoning Effort (OpenAI o1/o3 models):** ```typescript openAI: { model: OpenAiModels.Gpt5Chat_400K, reasoningEffort: OpenAiReasoningEffortLevels.Low // Faster, simpler reasoning reasoningEffort: OpenAiReasoningEffortLevels.Medium // Balanced reasoningEffort: OpenAiReasoningEffortLevels.High // Deepest reasoning, slower } ``` **Extended Thinking (Claude 3.7+, Gemini 2.5+):** ```typescript // Claude 3.7 Sonnet with thinking anthropic: { model: AnthropicModels.Claude_3_7Sonnet, enableThinking: true, // Enable internal reasoning thinkingTokenLimit: 10000 // Max tokens for thinking process } // Gemini 2.5 with thinking google: { model: GoogleModels.Gemini_2_5Flash, enableThinking: true, thinkingTokenLimit: 8000 } ``` **Vision Detail Level (OpenAI):** ```typescript openAI: { model: OpenAiModels.Gpt4O_128K, detailLevel: OpenAiVisionDetailLevels.Low // Faster, less detailed image analysis detailLevel: OpenAiVisionDetailLevels.High // Slower, more detailed } ``` ### Complete Completion Example ```typescript const productionSpec = await graphlit.createSpecification({ name: 'Production RAG Spec', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.Anthropic, anthropic: { model: AnthropicModels.Claude_4_5Sonnet, temperature: 0.2, // Mostly factual probability: 0.9, // Focused but not too narrow completionTokenLimit: 3000, // Up to ~2250 words enableThinking: true, // Better reasoning thinkingTokenLimit: 5000 }, systemPrompt: 'You are a helpful AI assistant that provides accurate, well-cited answers. Always reference source documents.', // Advanced RAG configuration (covered later): retrievalStrategy: { maxCount: 20 // Retrieve up to 20 relevant chunks }, rerankingStrategy: { serviceType: RerankingModelServiceTypes.Cohere // Use Cohere reranking }, searchType: ConversationSearchTypes.Hybrid // Vector + keyword search }); ``` *** ### Using OpenAI-Compatible AI Gateways **Purpose:** Access multiple AI providers through a unified, OpenAI-compatible API with added benefits like observability, caching, and cost optimization. **Supported Gateways:** * **OpenRouter** - Access 200+ models from one API * **Vercel AI Gateway** - Enterprise observability and response caching **How it works:** AI gateways provide OpenAI-compatible endpoints, so you use `ModelServiceTypes.OpenAi` with custom `endpoint`, `key`, and `modelName` parameters. *** #### OpenRouter: 200+ Models via One API Access Claude, GPT, Gemini, Llama, Mistral, and 200+ other models through OpenRouter's unified API. **Configuration:** ```typescript const openRouterSpec = await graphlit.createSpecification({ name: 'Claude via OpenRouter', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Custom, // Use Custom for external endpoints endpoint: 'https://openrouter.ai/api/v1', key: process.env.OPENROUTER_API_KEY, modelName: 'anthropic/claude-4.5-sonnet', // Actual model used temperature: 0.2, completionTokenLimit: 4000 } }); ``` **Model naming:** Use `provider/model` format: * `anthropic/claude-4.5-sonnet` - Best for RAG ($3/$15 per M tokens) * `google/gemini-2.5-flash` - Fast, 1M context ($0.075/$0.30 per M tokens) * `openai/gpt-4o` - Balanced ($2.50/$10 per M tokens) * `meta-llama/llama-3.3-70b-instruct` - Open source ($0.59/$0.59 per M tokens) * `deepseek/deepseek-chat` - Ultra-cheap ($0.14/$0.28 per M tokens) **When to use OpenRouter:** * Need access to 200+ models without managing multiple API keys * Cost optimization (compare pricing across providers) * Want automatic fallbacks between providers * Access to open-source models (Llama, Qwen, Mixtral) * No vendor lock-in (switch models by changing one parameter) **Browse models:** *** #### Vercel AI Gateway: Enterprise Observability Enterprise AI gateway with response caching, observability, and multi-provider routing, integrated with the Vercel ecosystem. **Configuration:** ```typescript const vercelSpec = await graphlit.createSpecification({ name: 'Claude via Vercel Gateway', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Custom, // Use Custom for external endpoints endpoint: 'https://ai-gateway.vercel.sh/v1', key: process.env.VERCEL_AI_GATEWAY_KEY, // Or VERCEL_OIDC_TOKEN modelName: 'anthropic/claude-sonnet-4', temperature: 0.2, completionTokenLimit: 4000 } }); ``` **Model naming:** Use `provider/model` format: * `anthropic/claude-sonnet-4` - Claude 4.5 Sonnet * `openai/gpt-5` - Latest GPT model * `google/gemini-2.5-flash` - Gemini Flash * `openai/gpt-4.1-mini` - GPT-4 Mini **When to use Vercel AI Gateway:** * Need enterprise observability (request logs, analytics dashboard) * Want response caching to reduce costs (up to 90% savings on repeated queries) * Using Vercel ecosystem (automatic OIDC authentication) * Require multi-provider routing with automatic fallbacks * Need rate limiting and cost controls **Key features:** * **Automatic caching** - Repeated queries are cached for free * **Observability** - Full request/response logs, latency metrics, cost tracking * **Multi-provider routing** - Automatic fallbacks if primary provider fails * **Vercel integration** - Works seamlessly with Vercel deployments, Edge Functions **Learn more:** *** #### Gateway Comparison | Feature | OpenRouter | Vercel AI Gateway | | ------------- | -------------------------------- | --------------------------------- | | **Endpoint** | `openrouter.ai/api/v1` | `ai-gateway.vercel.sh/v1` | | **Models** | 200+ models | Major providers | | **Best For** | Model variety, cost optimization | Enterprise observability, caching | | **Caching** | No | Yes (automatic) | | **Analytics** | Basic | Advanced (Vercel dashboard) | | **Fallbacks** | Provider-level | Multi-provider routing | **āš ļø Important:** Always use `OpenAiModels.Custom` when configuring external gateways. The `modelName` field determines which model is actually used. **See also:** [Complete gateway examples and troubleshooting ā†’](/api-guides/use-cases/specifications/specification-create-openai-compatible) *** ## TEXT\_EMBEDDING Specifications **Purpose:** Configure vector embeddings for semantic search and RAG retrieval. **Default:** OpenAI `text-embedding-ada-002` (if not specified in project settings). **When you need it:** * Better embedding quality * Different embedding dimensions * Multi-language content * Cost optimization **āš ļø CRITICAL:** You **cannot change embeddings after content is ingested**. The embedding model used during ingestion is permanent for that content. Plan carefully! ### Embedding Model Selection | Model | Dimensions | Quality | Speed | Best For | | -------------------------- | ---------- | ------- | ------------ | ---------------------------- | | **text-embedding-3-large** | 3072 | ā­ā­ā­ā­ā­ | āš” Fast | Best quality (recommended) | | **text-embedding-3-small** | 1536 | ā­ā­ā­ā­ | āš”āš” Very Fast | Good balance, lower cost | | **text-embedding-ada-002** | 1536 | ā­ā­ā­ | āš”āš” Very Fast | Legacy default | | **Voyage Large 3** | 2048 | ā­ā­ā­ā­ā­ | āš” Fast | High quality alternative | | **Cohere Embed v3** | 1024 | ā­ā­ā­ā­ | āš”āš” Very Fast | Multi-language, good quality | | **Jina Embeddings v2** | 768 | ā­ā­ā­ | āš”āš” Very Fast | Free tier available | ### Configuration ```typescript interface EmbeddingSpecificationInput { name: string; type: SpecificationTypes.TextEmbedding; // Required serviceType: ModelServiceTypes; // Required: Which provider // Provider-specific: openAI?: { model: OpenAiModels }; // OpenAI embeddings voyage?: { model: VoyageModels }; // Voyage embeddings cohere?: { model: CohereModels }; // Cohere embeddings jina?: { model: JinaModels }; // Jina embeddings } ``` ### Examples **OpenAI text-embedding-3-large (Recommended):** ```typescript const embeddingSpec = await graphlit.createSpecification({ name: 'OpenAI Large Embeddings', type: SpecificationTypes.TextEmbedding, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Embedding_3Large // 3072 dimensions, best quality } }); // Use during ingestion await graphlit.ingestUri( uri, undefined, undefined, undefined, true, undefined, undefined, undefined, { id: embeddingSpec.createSpecification.id } // Apply to this content ); ``` **Voyage Large (Alternative):** ```typescript const voyageSpec = await graphlit.createSpecification({ name: 'Voyage Large Embeddings', type: SpecificationTypes.TextEmbedding, serviceType: ModelServiceTypes.Voyage, voyage: { model: VoyageModels.Voyage_3_0Large // 2048 dimensions } }); ``` **Cohere Multi-Language:** ```typescript const cohereSpec = await graphlit.createSpecification({ name: 'Cohere Multilingual', type: SpecificationTypes.TextEmbedding, serviceType: ModelServiceTypes.Cohere, cohere: { model: CohereModels.EmbedMultilingual_3_0 // Best for non-English } }); ``` ### āš ļø Cannot Change After Ingestion ```typescript // āŒ WRONG: Can't change embeddings after ingestion await graphlit.ingestUri(uri); // Uses default (ada-002) // Later... try to use different embeddings await graphlit.ingestUri( uri2, undefined, undefined, undefined, true, undefined, undefined, undefined, { id: largeEmbeddingSpecId } // Different embeddings! ); // Result: Mixed embeddings = poor search quality! // āœ… CORRECT: Choose embedding model FIRST, use consistently const embeddingSpec = await graphlit.createSpecification({ type: SpecificationTypes.TextEmbedding, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Embedding_3Large } }); // Use for ALL content await graphlit.ingestUri(uri1, ..., { id: embeddingSpec.createSpecification.id }); await graphlit.ingestUri(uri2, ..., { id: embeddingSpec.createSpecification.id }); await graphlit.ingestUri(uri3, ..., { id: embeddingSpec.createSpecification.id }); ``` *** ## EXTRACTION Specifications **Purpose:** Control LLM used for entity extraction in workflows. **Used in:** Extraction workflow stage (see [workflows.md](/platform/workflows)) **When you need it:** * Extract entities from content * Build knowledge graph * Custom entity types ### Model Selection | Model | Quality | Speed | Best For | | --------------------- | ------- | ------------- | -------------------------------------- | | **Claude 4.5 Sonnet** | ā­ā­ā­ā­ā­ | āš” Moderate | Best accuracy (recommended) | | **Claude 3.7 Sonnet** | ā­ā­ā­ā­ā­ | āš” Moderate | Extended thinking for complex entities | | **GPT-4o** | ā­ā­ā­ā­ | āš”āš” Fast | Good balance of speed/quality | | **Claude 4.5 Haiku** | ā­ā­ā­ | āš”āš”āš” Very Fast | Cost optimization | ### Configuration ```typescript const extractionSpec = await graphlit.createSpecification({ name: 'Claude Extraction', type: SpecificationTypes.Extraction, serviceType: ModelServiceTypes.Anthropic, anthropic: { model: AnthropicModels.Claude_4_5Sonnet } }); // Use in extraction workflow const workflow = await graphlit.createWorkflow({ name: 'Entity Extraction', extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, modelText: { specification: { id: extractionSpec.createSpecification.id } } } }] } }); ``` *** ## PREPARATION Specifications **Purpose:** Control vision model used for PDF/image preparation in workflows. **Used in:** Preparation workflow stage (see [workflows.md](/platform/workflows)) **When you need it:** * Complex PDFs with tables/images * Override default Azure AI Document Intelligence ### Model Selection | Model | Quality | Speed | Best For | | --------------------- | ------- | ------------- | -------------------------------- | | **GPT-4o** | ā­ā­ā­ā­ | āš”āš” Fast | Best balance (recommended) | | **Claude 4.5 Sonnet** | ā­ā­ā­ā­ā­ | āš” Moderate | Complex layouts, academic papers | | **Gemini 2.5 Flash** | ā­ā­ā­ā­ | āš”āš”āš” Very Fast | Fast, good quality, lower cost | ### Configuration ```typescript const preparationSpec = await graphlit.createSpecification({ name: 'GPT-4o for PDFs', type: SpecificationTypes.Preparation, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt4O_128K } }); // Use in preparation workflow const workflow = await graphlit.createWorkflow({ name: 'Vision Model Prep', preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.ModelDocument, modelDocument: { specification: { id: preparationSpec.createSpecification.id } } } }] } }); ``` *** ## Model Service Providers Complete reference for all 15 supported AI providers: ### OpenAI (`ModelServiceTypes.OpenAi`) **Best for:** General purpose, balanced quality/speed **Popular models:** GPT-4o, GPT-4o Mini, o1 **Context windows:** 128K (GPT-4o), 128K (o1) ### Anthropic (`ModelServiceTypes.Anthropic`) **Best for:** RAG with citations, extended thinking **Popular models:** Claude 4.5 Sonnet, Claude 4.5 Opus, Claude 3.7 Sonnet **Context windows:** 200K **Unique features:** Extended thinking, best citation accuracy ### Google (`ModelServiceTypes.Google`) **Best for:** Long documents, fast inference **Popular models:** Gemini 2.5 Flash, Gemini 2.5 Pro **Context windows:** 1M (1 million tokens!) **Unique features:** Massive context, extended thinking (2.5+) ### Groq (`ModelServiceTypes.Groq`) **Best for:** Ultra-fast inference, real-time applications **Popular models:** Llama 3.3 70B, Mixtral 8x7B **Context windows:** 128K **Unique features:** Fastest inference speed ### Mistral (`ModelServiceTypes.Mistral`) **Best for:** European data residency, cost-effective **Popular models:** Mistral Large, Mistral Small **Context windows:** 128K ### Cohere (`ModelServiceTypes.Cohere`) **Best for:** Multi-language embeddings, reranking **Popular models:** Command R+, Embed v3 **Unique features:** Best multi-language support, excellent reranking ### Deepseek (`ModelServiceTypes.Deepseek`) **Best for:** Cost optimization with good quality **Popular models:** Deepseek V3 **Context windows:** 64K ### Cerebras (`ModelServiceTypes.Cerebras`) **Best for:** Fastest inference available **Popular models:** Llama 3.3 70B **Unique features:** Blazing fast inference on custom chips ### Voyage (`ModelServiceTypes.Voyage`) **Best for:** High-quality embeddings **Popular models:** Voyage Large 3, Voyage 3 **Unique features:** Excellent embedding quality ### Jina (`ModelServiceTypes.Jina`) **Best for:** Free embeddings, budget projects **Popular models:** Jina Embeddings v2 **Unique features:** Free tier available ### xAI (`ModelServiceTypes.Xai`) **Best for:** Grok models, real-time data **Popular models:** Grok 2 **Unique features:** Real-time web data access ### Azure OpenAI (`ModelServiceTypes.AzureOpenAi`) **Best for:** Enterprise, Azure integration **Popular models:** Same as OpenAI (GPT-4o, etc.) **Unique features:** Enterprise SLAs, private deployment ### AWS Bedrock (`ModelServiceTypes.Bedrock`) **Best for:** AWS integration, multi-model **Popular models:** Claude, Llama, Mistral (via Bedrock) **Unique features:** Multiple models in one platform ### Replicate (`ModelServiceTypes.Replicate`) **Best for:** Open-source models, experimentation **Popular models:** Various open-source LLMs ### Azure AI (`ModelServiceTypes.AzureAi`) **Best for:** Azure-native AI services **Popular models:** Phi models *** ## Advanced RAG Configuration ### Retrieval Strategy **Purpose:** Control how content is retrieved for RAG. ```typescript interface RetrievalStrategyInput { maxCount?: number; // Max chunks to retrieve (default: 10) threshold?: number; // Relevance threshold 0-1 } ``` **Example:** ```typescript const spec = await graphlit.createSpecification({ type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.Anthropic, anthropic: { model: AnthropicModels.Claude_4_5Sonnet }, retrievalStrategy: { maxCount: 20, // Retrieve up to 20 chunks threshold: 0.7 // Only chunks with >0.7 relevance } }); ``` ### Reranking Strategy **Purpose:** Improve relevance of retrieved content using specialized reranking models. ```typescript interface RerankingStrategyInput { serviceType: RerankingModelServiceTypes; // COHERE, JINA threshold?: number; // Relevance threshold } ``` **Example:** ```typescript const spec = await graphlit.createSpecification({ type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt4O_128K }, rerankingStrategy: { serviceType: RerankingModelServiceTypes.Cohere, // Use Cohere reranking threshold: 0.5 } }); ``` **When to use reranking:** * Improved RAG accuracy (10-20% better) * Complex queries * Large content corpus * Trade-off: Slightly slower, small cost increase ### GraphRAG Strategy **Purpose:** Use knowledge graph entities to enhance RAG retrieval. ```typescript interface GraphStrategyInput { generateGraph?: boolean; // Generate knowledge graph } ``` **Example:** ```typescript const spec = await graphlit.createSpecification({ type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.Anthropic, anthropic: { model: AnthropicModels.Claude_4_5Sonnet }, graphStrategy: { generateGraph: true // Use entity graph for enhanced retrieval } }); ``` **When to use GraphRAG:** * Content with entity extraction workflow * Complex entity relationships matter * Trade-off: Better context, more complex ### Revision Strategy **Purpose:** Self-revision for improved answer quality. ```typescript interface RevisionStrategyInput { count?: number; // Number of revision passes (default: 1) } ``` **Example:** ```typescript const spec = await graphlit.createSpecification({ type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt4O_128K }, revisionStrategy: { count: 2 // Revise answer twice for better quality } }); ``` **Trade-off:** Better quality, but 2-3x slower and more expensive. ### Search Type **Purpose:** Control search algorithm for retrieval. ```typescript enum ConversationSearchTypes { VECTOR // Semantic search only (default) KEYWORD // Keyword search only HYBRID // Both vector + keyword (best) } ``` **Example:** ```typescript const spec = await graphlit.createSpecification({ type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.Anthropic, anthropic: { model: AnthropicModels.Claude_4_5Sonnet }, searchType: ConversationSearchTypes.Hybrid // Combine semantic + keyword }); ``` **When to use each:** * `VECTOR` - Conceptual understanding, semantic similarity * `KEYWORD` - Exact matches, specific terms * `HYBRID` - Best of both (recommended for most use cases) *** ## Production Patterns ### Pattern 1: Multi-Specification Strategy **Use case:** Different models for different use cases. ```typescript // High-accuracy for customer support const supportSpec = await graphlit.createSpecification({ name: 'Customer Support', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.Anthropic, anthropic: { model: AnthropicModels.Claude_4_5Sonnet, temperature: 0.1 // Very factual }, rerankingStrategy: { serviceType: RerankingModelServiceTypes.Cohere // Better accuracy } }); // Fast responses for internal queries const internalSpec = await graphlit.createSpecification({ name: 'Internal Queries', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.Groq, groq: { model: GroqModels.Llama_3_3_70B, // Ultra-fast temperature: 0.3 } }); // Route based on context const specId = isCustomerFacing ? supportSpec.id : internalSpec.id; ``` ### Pattern 2: Reusable Project Defaults ```typescript // Set up once during project initialization async function setupProjectSpecs() { const specs = { completion: await graphlit.createSpecification({ name: 'Default Completion', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.Anthropic, anthropic: { model: AnthropicModels.Claude_4_5Sonnet } }), embedding: await graphlit.createSpecification({ name: 'Default Embeddings', type: SpecificationTypes.TextEmbedding, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Embedding_3Large } }) }; // Store IDs in database/config await db.config.setMultiple({ default_completion_spec: specs.completion.createSpecification.id, default_embedding_spec: specs.embedding.createSpecification.id }); return specs; } // Use throughout application const completionSpecId = await db.config.get('default_completion_spec'); ``` ### Pattern 3: Zine Production Pattern **What Zine uses:** ```typescript // Single spec for all conversations const zineSpec = await graphlit.createSpecification({ name: 'Zine Production', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.Anthropic, anthropic: { model: AnthropicModels.Claude_4_5Sonnet, temperature: 0.2, completionTokenLimit: 3000 }, retrievalStrategy: { maxCount: 15 // Retrieve up to 15 relevant chunks }, searchType: ConversationSearchTypes.Hybrid, // Vector + keyword systemPrompt: 'You are Zine AI, a helpful assistant that provides accurate answers based on your synced data sources.' }); // Used for all user conversations const answer = await graphlit.streamAgent( userPrompt, eventHandler, conversationId, { id: zineSpec.createSpecification.id } ); ``` ### Pattern 4: Environment-Based Configuration ```typescript const specs = { development: await graphlit.createSpecification({ name: 'Dev Spec', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt4OMini_128K // Cheaper for dev } }), production: await graphlit.createSpecification({ name: 'Prod Spec', type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.Anthropic, anthropic: { model: AnthropicModels.Claude_4_5Sonnet // Best quality for prod } }) }; // Use based on environment const specId = process.env.NODE_ENV === 'production' ? specs.production.createSpecification.id : specs.development.createSpecification.id; ``` ### Pattern 5: A/B Testing Different Models ```typescript // Test model performance async function abTestModels(userPrompt: string, userId: string) { const variant = userId.charCodeAt(0) % 2; // Simple A/B split const specs = { a: gpt4oSpecId, // Variant A: GPT-4o b: claudeSpecId // Variant B: Claude 4.5 Sonnet }; const specId = variant === 0 ? specs.a : specs.b; const answer = await graphlit.promptConversation({ prompt: userPrompt, specification: { id: specId } }); // Log for analysis await analytics.track('conversation_model_test', { userId, variant: variant === 0 ? 'gpt4o' : 'claude', responseTime: answer.completionTime, tokenCount: answer.message.tokens }); return answer; } ``` *** ## Complete API Reference ### SpecificationInput (Top-Level) ```typescript interface SpecificationInput { // Required: name: string; type: SpecificationTypes; serviceType: ModelServiceTypes; // Provider configuration (one required based on serviceType): openAI?: OpenAiModelPropertiesInput; anthropic?: AnthropicModelPropertiesInput; google?: GoogleModelPropertiesInput; groq?: GroqModelPropertiesInput; mistral?: MistralModelPropertiesInput; cohere?: CohereModelPropertiesInput; deepseek?: DeepseekModelPropertiesInput; cerebras?: CerebrasModelPropertiesInput; bedrock?: BedrockModelPropertiesInput; azureOpenAI?: AzureOpenAiModelPropertiesInput; azureAI?: AzureAiModelPropertiesInput; replicate?: ReplicateModelPropertiesInput; voyage?: VoyageModelPropertiesInput; jina?: JinaModelPropertiesInput; xai?: XaiModelPropertiesInput; // Advanced RAG (all optional): retrievalStrategy?: RetrievalStrategyInput; rerankingStrategy?: RerankingStrategyInput; graphStrategy?: GraphStrategyInput; revisionStrategy?: RevisionStrategyInput; // Customization (all optional): systemPrompt?: string; customInstructions?: string; customGuidance?: string; searchType?: ConversationSearchTypes; strategy?: ConversationStrategyInput; } ``` *** ## Summary **Key Takeaways:** 1. **Project defaults usually work** - Only create specifications when you need different behavior 2. **Completion specs control RAG** - Model, temperature, token limits, system prompts 3. **Embedding specs are permanent** - Choose carefully before ingestion, can't change later 4. **Extraction/Preparation specs go in workflows** - Not used directly in conversations 5. **Advanced RAG features improve quality** - Reranking, GraphRAG, hybrid search 6. **15 model providers available** - OpenAI, Anthropic, Google, Groq, and more 7. **Temperature controls creativity** - Low (0.1) = factual, High (0.9) = creative **When in doubt:** Start with project defaults, add specifications only when you hit limitations. *** **Related Documentation:** * [Workflows ā†’](/platform/workflows) - Configure content processing pipeline * [Key Concepts ā†’](/platform/key-concepts) - High-level overview * [API Guides: Specifications ā†’](/api-guides/use-cases/specifications) - Code examples # AI Models Graphlit supports **15 AI model providers** with instant model switching and multi-model workflows. {% hint style="success" %} **Switch models instantly** - Update configuration, not your application. Access 100+ models from 15 providers including GPT-5, Claude 4.5, and Gemini 2.5 Pro. {% endhint %} *** ## Why Model Choice Matters The AI landscape evolves weekly. What you need:
Access latest modelsGPT-5, Claude 4.5 Sonnet, Gemini 2.5 Pro available immediately when released
Switch models instantlyUpdate configuration, test different models without rewriting code
Compare performanceA/B test models for your use case
Optimize cost per taskUse expensive models where they matter, cheap ones for simple tasks
Multi-model workflowsGPT-4o for chat, Claude for analysis, Cohere for embeddings
*** ## Supported Models (15 Providers) {% hint style="info" %} **How to specify models**: Use the model name string in your specification (e.g., `model: "GPT4_O"`). Model names are consistent across all SDKs - no need to import enums. **All models support:** Tool calling, streaming, system prompts, temperature control. Your data stays private - we don't train on it. {% endhint %} ### OpenAI GPT-5, GPT-4.1, GPT-4o series, and o-series reasoning models. Up to 1M+ token context windows. **Best for**: General purpose AI, complex reasoning, code generation, high-volume applications. *** ### Anthropic Claude 4.x and Claude 3.x series including Sonnet, Opus, and Haiku variants. Up to 200k token context. **Best for**: Analysis, writing, code generation, complex reasoning tasks. *** ### Google Gemini 2.5, 2.0, and 1.5 series. Up to 1M+ token context windows with multimodal capabilities. **Best for**: Long documents, video/image analysis, multimodal tasks. *** ### xAI (Grok) Grok 4, Grok 3, and Mini variants with real-time data capabilities. **Best for**: Real-time queries, Twitter/X integration, current events. *** ### Meta LLaMA LLaMA 4 and LLaMA 3.x series available through Groq, Cerebras, and AWS Bedrock. Open weights models. **Best for**: Cost-effective inference, on-premise deployment, high-volume applications. *** ### Deepseek Deepseek Reasoner and Chat models with strong reasoning and code generation capabilities. **Best for**: Cost-effective reasoning, code generation, Chinese language tasks. *** ### Mistral Mistral Large, Medium, Small, Mixtral, and Pixtral vision models. Includes text embeddings. **Best for**: European data residency, cost-effective alternatives, vision tasks. *** ### Cohere Command series models and multilingual embeddings optimized for retrieval and RAG. **Best for**: Enterprise RAG, multilingual embeddings, reranking. *** ### Groq Ultra-fast LLaMA model inference (500+ tokens/sec). LLaMA 4 and 3.x series. **Best for**: Real-time applications, streaming responses, high-volume inference. *** ### Cerebras Record-breaking inference speed (1800+ tokens/sec). LLaMA 4 and 3.x series. **Best for**: Fastest possible inference, streaming, real-time chat. *** ### AWS Bedrock Amazon Nova series and LLaMA models. AWS infrastructure integration. **Best for**: AWS deployments, compliance requirements, on-premise options. *** ### Jina Text and multimodal embeddings with 89-language support. Includes CLIP image embeddings. **Best for**: Multilingual embeddings, image-text search, rich media applications. *** ### Voyage High-quality text embeddings optimized for retrieval. Flexible output dimensions. **Best for**: Semantic search, RAG applications, document retrieval. *** ## Model Selection Guide ### By Use Case | Use Case | Recommended Models | Why | | -------------------- | --------------------------------------------- | ------------------------- | | **General Chat** | OpenAI GPT-4o, Anthropic Claude | Balanced cost/performance | | **Complex Analysis** | OpenAI GPT-5, Anthropic Claude, Google Gemini | Best reasoning | | **Code Generation** | Anthropic Claude, OpenAI, Deepseek | Strong at coding | | **Long Documents** | Google Gemini, OpenAI GPT-4.1 | 1M+ context | | **Fast Responses** | Groq, Cerebras, OpenAI Mini | Ultra-fast inference | | **Cost-Sensitive** | OpenAI Mini, LLaMA via Groq, Mistral | Budget-friendly | | **Reasoning** | OpenAI o-series, Deepseek | Math, logic, coding | | **Multimodal** | Google Gemini, OpenAI GPT-4o, Mistral Pixtral | Images + text | | **Real-time Data** | xAI Grok | Twitter integration | *** ### By Budget **Budget-Friendly** (< $0.50 per 1M tokens): OpenAI Mini, LLaMA via Groq/Cerebras, Mistral Small, Anthropic Haiku **Mid-Range** ($1-5 per 1M tokens): OpenAI GPT-4o, Anthropic Claude, Mistral Large, Google Gemini Flash **Premium** ($5-30 per 1M tokens): OpenAI GPT-5, Anthropic Claude 4.5, Google Gemini Pro, OpenAI o-series *** ## Switching Models Instantly Create specifications with different models, then switch by changing which specification you reference: ```typescript import { Graphlit } from 'graphlit-client'; import { SpecificationTypes, ModelServiceTypes, OpenAiModels, AnthropicModels, ConversationTypes } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); // Create multiple model specifications const gpt4Spec = await client.createSpecification({ name: "GPT-4o", type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt4O_128K, temperature: 0.7 } }); const claudeSpec = await client.createSpecification({ name: "Claude 3.5", type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.Anthropic, anthropic: { model: AnthropicModels.Claude_3_5Sonnet, temperature: 0.7 } }); // Create conversation with GPT-4o const conversation = await client.createConversation({ name: "My Agent", type: ConversationTypes.Content, specification: { id: claudeSpec.createSpecification.id } // ā† Use Claude }); // Switch to GPT-4o by updating await client.updateConversation({ id: conversation.createConversation.id, specification: { id: gpt4Spec.createSpecification.id } // ā† Now use GPT-4o }); ``` **Result**: Same conversation, different model - instant switch with zero code changes. *** ## Multi-Model Patterns ### Model Fallback Graphlit supports automatic fallback if the primary model fails: ```typescript // Create specifications const primarySpec = await client.createSpecification({ name: "Primary", type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.Anthropic, anthropic: { model: AnthropicModels.Claude_3_5Sonnet } }); const fallbackSpec = await client.createSpecification({ name: "Fallback", type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt4O_128K } }); // Use with fallbacks array const conversation = await client.createConversation({ name: "Resilient Agent", type: ConversationTypes.Content, specification: { id: primarySpec.createSpecification.id }, fallbacks: [{ id: fallbackSpec.createSpecification.id }] }); // Automatically uses fallback if primary fails ``` [See working examples](https://github.com/graphlit/graphlit-samples/tree/main/nextjs/chat) *** ## Specification Types & Where They're Used {% hint style="warning" %} **Critical**: Specification types must match where they're used. You can't use an Extraction spec in a conversation, or a Completion spec in a workflow extraction stage. {% endhint %} | Specification Type | Valid Context | Purpose | | ------------------ | --------------------------- | ----------------------------------------- | | **Completion** | Conversations | Chat, RAG, Q\&A with tool calling | | **Extraction** | Workflow extraction stages | Entity extraction, custom data extraction | | **Summarization** | Workflow extraction stages | Content summarization | | **Preparation** | Workflow preparation stages | Vision OCR, document processing | | **TextEmbedding** | Workflow indexing stages | Semantic search embeddings | ### Examples **Completion (for Conversations)**: ```typescript const spec = await client.createSpecification({ name: "Chat Model", type: SpecificationTypes.Completion, // ā† For conversations serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt4O_128K } }); await client.createConversation({ specification: { id: spec.createSpecification.id } // āœ… Valid }); ``` **Extraction (for Workflows)**: ```typescript const spec = await client.createSpecification({ name: "Entity Extraction", type: SpecificationTypes.Extraction, // ā† For workflow extraction serviceType: ModelServiceTypes.Anthropic, anthropic: { model: AnthropicModels.Claude_3_5Sonnet, temperature: 0.1 } }); await client.createWorkflow({ extraction: { jobs: [{ connector: { type: "MODEL_NAMED_ENTITY", specification: { id: spec.createSpecification.id } // āœ… Valid } }] } }); ``` *** ## Embeddings Models For semantic search and retrieval, use TextEmbedding specifications: ```typescript const embeddingSpec = await client.createSpecification({ name: "Cohere Embeddings", type: SpecificationTypes.TextEmbedding, serviceType: ModelServiceTypes.Cohere, cohere: { model: CohereModels.EmbedMultilingual_3_0 } }); // Use in workflow indexing stage await client.createWorkflow({ name: "Custom Embeddings", indexing: { jobs: [{ connector: { type: "EMBEDDING", specification: { id: embeddingSpec.createSpecification.id } } }] } }); ``` **Popular embedding models**: * OpenAI: `TextEmbedding_3Large`, `TextEmbedding_3Small` * Cohere: `EmbedMultilingualV3`, `EmbedEnglishV3` * Mistral: `MistralEmbed` [See embedding examples](https://github.com/graphlit/graphlit-samples) *** ## Cost Optimization 1. **Use cheaper models for simple tasks**: * GPT-4o Mini for search, simple Q\&A * LLaMA 3.1 8b for high-volume inference 2. **Use premium models for complex tasks**: * GPT-5, Claude 4.5 for analysis, writing * o3 for reasoning, coding 3. **Optimize token usage**: * Limit `maxTokens` in specifications * Use `limitResults` in retrieval strategies * Trim conversation history (`maxMessages`) 4. **Leverage fast inference**: * Groq, Cerebras for real-time (same cost, faster) 5. **Monitor usage**: * Track tokens per customer * Set budget alerts * A/B test cheaper alternatives *** ## Next Steps * [**Platform Overview**](/getting-started/overview) - See how models fit into the platform * [**AI Agents**](/tutorials/ai-agents) - Use models in agent workflows * [**Context Engineering**](/tutorials/context-engineering) - Optimize model inputs *** **Access 15 providers, 100+ models. Switch instantly. Build with confidence.** # Feeds Graphlit connects to **30+ data sources** with automatic sync. OAuth, API keys, or public sources. {% hint style="info" %} **API terminology**: In the Graphlit API, these are called **feeds** (you create them with `createFeed()`). **Feed** = A connection that continuously syncs data from a source (Slack, S3, RSS, etc.)\ **Connector** = Optional per-user auth credential storage (smaller, used with feeds) **Authentication**: Feeds support OAuth (Slack, Gmail), API keys (S3, Tavily), or public access (RSS, web crawls). {% endhint %} {% hint style="success" %} **Connect once, search forever** - Automatic sync across all your tools via configurable polling schedules (30 seconds to hours). No manual uploads. {% endhint %} *** ## How Feeds Work
No manual uploadsSet up once, content syncs automatically
Configurable pollingSchedule policies from 30 seconds to hours between checks
OAuth handledNo token management, Graphlit handles auth refresh
Incremental updatesOnly new/changed content syncs (not full re-ingest)
DeduplicationSame file in Slack and Drive = one copy in your knowledge base
### Listing modes (Past vs New) Many feeds support `Past` vs `New` listing modes: * `Past`: backfill existing items, then continue polling for new items * `New`: only ingest items created after the feed is created Some sources use specialized listing enums (e.g., `EmailListingTypes`, `CalendarListingTypes`) or date filters. ### Sync mode (Archive vs Mirror) Feeds support an optional `syncMode`: * `ARCHIVE`: preserve ingested content even if the source deletes it * `MIRROR`: synchronize with the source, including deletions *** ## All Feeds ### Cloud Storage

Amazon S3		Files from any S3 bucket
Access Key auth ā€¢ Automatic sync

Azure Blob Storage		Files from Azure containers
Connection String auth ā€¢ Automatic sync

Azure File Share		Files from Azure file shares
Connection String auth ā€¢ Automatic sync

Google Cloud Storage		Files from GCS buckets
Service Account auth ā€¢ Automatic sync
**Use cases:** Data lakes, backup archives, media libraries, log files **Example:** {% tabs %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedServiceTypes } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); const feed = await client.createFeed({ name: "Company Data Lake", type: FeedTypes.Site, site: { type: FeedServiceTypes.S3Blob, s3: { accessKey: process.env.AWS_ACCESS_KEY, secretAccessKey: process.env.AWS_SECRET_KEY, bucketName: "company-datalake", prefix: "documents/" // Optional: filter by prefix } } }); ``` {% endtab %} {% tab title="Python" %} ```python import os from graphlit import Graphlit from graphlit_api import FeedTypes, FeedServiceTypes graphlit = Graphlit() feed = await graphlit.client.create_feed( name="Company Data Lake", type=FeedTypes.SITE, site={ "type": FeedServiceTypes.S3_BLOB, "s3": { "accessKey": os.getenv("AWS_ACCESS_KEY"), "secretAccessKey": os.getenv("AWS_SECRET_KEY"), "bucketName": "company-datalake", "prefix": "documents/" # Optional: filter by prefix } } ) ``` {% endtab %} {% endtabs %} *** ### User Storage & Productivity

Microsoft SharePoint		Files and pages from SharePoint sites
OAuth (Microsoft) ā€¢ Automatic sync

Microsoft OneDrive		Personal and business files
OAuth (Microsoft) ā€¢ Automatic sync

Google Drive		Files, Docs, Sheets, Slides
OAuth (Google) ā€¢ Automatic sync

Dropbox		All file types
OAuth (Dropbox) ā€¢ Automatic sync

Box		Enterprise file storage
OAuth (Box) ā€¢ Automatic sync
**Use cases:** Company knowledge base, shared documentation, team collaboration, project resources **Example:** {% tabs %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedServiceTypes } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); const feed = await client.createFeed({ name: "Engineering Docs", type: FeedTypes.Site, site: { type: FeedServiceTypes.GoogleDrive, googleDrive: { refreshToken: process.env.GOOGLE_REFRESH_TOKEN, clientId: process.env.GOOGLE_CLIENT_ID, clientSecret: process.env.GOOGLE_CLIENT_SECRET // folderId: "..." // Optional: sync specific folder only } } }); ``` {% endtab %} {% tab title="Python" %} ```python import os from graphlit import Graphlit from graphlit_api import FeedTypes, FeedServiceTypes graphlit = Graphlit() feed = await graphlit.client.create_feed( name="Engineering Docs", type=FeedTypes.SITE, site={ "type": FeedServiceTypes.GOOGLE_DRIVE, "googleDrive": { "refreshToken": os.getenv("GOOGLE_REFRESH_TOKEN"), "clientId": os.getenv("GOOGLE_CLIENT_ID"), "clientSecret": os.getenv("GOOGLE_CLIENT_SECRET") # "folderId": "..." # Optional: sync specific folder only } } ) ``` {% endtab %} {% endtabs %} *** ### Communication & Messaging

Slack		Messages, threads, files from channels
OAuth (Bot Token) ā€¢ Automatic sync

Microsoft Teams		Messages from team channels
OAuth (Microsoft) ā€¢ Automatic sync

Discord		Messages, threads, files from servers
OAuth (Bot Token) ā€¢ Automatic sync

Twitter/X		Posts, media, conversations
OAuth (Twitter API) ā€¢ Automatic sync
**Use cases:** Team conversations, decision history, customer discussions, product feedback **Example:** {% tabs %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { FeedListingTypes, FeedTypes } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); const feed = await client.createFeed({ name: "Engineering Slack", type: FeedTypes.Slack, slack: { channel: "engineering", // Channel name or ID token: process.env.SLACK_BOT_TOKEN, type: FeedListingTypes.Past, includeAttachments: true } }); ``` {% endtab %} {% tab title="Python" %} ```python import os from graphlit import Graphlit from graphlit_api import FeedListingTypes, FeedTypes graphlit = Graphlit() feed = await graphlit.client.create_feed( name="Engineering Slack", type=FeedTypes.SLACK, slack={ "channel": "engineering", "token": os.getenv("SLACK_BOT_TOKEN"), "type": FeedListingTypes.PAST, "includeAttachments": True } ) ``` {% endtab %} {% endtabs %} *** ### Email

Gmail		Emails, attachments, labels, threads
OAuth (Google) ā€¢ Automatic sync

Microsoft Outlook		Emails, attachments, folders
OAuth (Microsoft) ā€¢ Automatic sync
**Use cases:** Customer communications, sales outreach, support context, contract negotiations **Example:** {% tabs %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedServiceTypes } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); const feed = await client.createFeed({ name: "Sales Inbox", type: FeedTypes.Email, email: { type: FeedServiceTypes.GoogleEmail, google: { refreshToken: process.env.GOOGLE_REFRESH_TOKEN, clientId: process.env.GOOGLE_CLIENT_ID, clientSecret: process.env.GOOGLE_CLIENT_SECRET, query: "from:*@acmecorp.com" } } }); ``` {% endtab %} {% tab title="Python" %} ```python import os from graphlit import Graphlit from graphlit_api import FeedTypes, FeedServiceTypes graphlit = Graphlit() feed = await graphlit.client.create_feed( name="Sales Inbox", type=FeedTypes.EMAIL, email={ "type": FeedServiceTypes.GOOGLE_EMAIL, "google": { "refreshToken": os.getenv("GOOGLE_REFRESH_TOKEN"), "clientId": os.getenv("GOOGLE_CLIENT_ID"), "clientSecret": os.getenv("GOOGLE_CLIENT_SECRET"), "query": "from:*@acmecorp.com" } } ) ``` {% endtab %} {% endtabs %} *** ### Issue Tracking & Project Management

Linear		Issues, comments, projects, roadmaps
OAuth (Linear API) ā€¢ Automatic sync

Jira		Issues, comments, boards, sprints
OAuth (Atlassian) ā€¢ Automatic sync

GitHub Issues		Repository issues and comments
OAuth (GitHub) ā€¢ Automatic sync

GitHub Commits		Repository commits and code changes
OAuth (GitHub) ā€¢ Automatic sync

GitHub Pull Requests		Pull requests and code reviews
OAuth (GitHub) ā€¢ Automatic sync

Trello		Cards, boards, checklists
OAuth (Trello API) ā€¢ Automatic sync
**Use cases:** Product roadmaps, bug tracking, feature requests, sprint planning **Example:** {% tabs %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedServiceTypes } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); const feed = await client.createFeed({ name: "Product Backlog", type: FeedTypes.Issue, issue: { type: FeedServiceTypes.Linear, linear: { apiKey: process.env.LINEAR_API_KEY, includeComments: true, includeClosed: false // teamId: "..." // Optional: sync specific team only } } }); ``` {% endtab %} {% tab title="Python" %} ```python import os from graphlit import Graphlit from graphlit_api import FeedTypes, FeedServiceTypes graphlit = Graphlit() feed = await graphlit.client.create_feed( name="Product Backlog", type=FeedTypes.ISSUE, issue={ "type": FeedServiceTypes.LINEAR, "linear": { "apiKey": os.getenv("LINEAR_API_KEY"), "includeComments": True, "includeClosed": False # "teamId": "..." # Optional: sync specific team only } } ) ``` {% endtab %} {% endtabs %} *** ### Knowledge Bases & Documentation | Connector | Content Types | Auth Type | Sync Mode | | --------------------- | ---------------- | ------------------ | --------- | | **Notion** | Pages, Databases | OAuth (Notion API) | Automatic | | **Intercom Articles** | Articles | OAuth (Intercom) | Automatic | | **Zendesk Articles** | Articles | OAuth (Zendesk) | Automatic | **Use cases:** * Internal wikis * Product documentation * Help center content * Company policies **Example:** {% tabs %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); const feed = await client.createFeed({ name: "Engineering Wiki", type: FeedTypes.Notion, notion: { apiKey: process.env.NOTION_API_KEY, includeSubpages: true // databaseId: "..." // Optional: sync specific database only } }); ``` {% endtab %} {% tab title="Python" %} ```python import os from graphlit import Graphlit from graphlit_api import FeedTypes graphlit = Graphlit() feed = await graphlit.client.create_feed( name="Engineering Wiki", type=FeedTypes.NOTION, notion={ "apiKey": os.getenv("NOTION_API_KEY"), "includeSubpages": True # "databaseId": "..." # Optional: sync specific database only } ) ``` {% endtab %} {% endtabs %} *** ### Support & Ticketing | Connector | Content Types | Auth Type | Sync Mode | | -------------------- | ---------------------- | ---------------- | --------- | | **Intercom Tickets** | Tickets, Conversations | OAuth (Intercom) | Automatic | | **Zendesk Tickets** | Tickets, Conversations | OAuth (Zendesk) | Automatic | **Use cases:** * Customer support history * Common issues * Product feedback * Feature requests *** ### Code Repositories | Connector | Content Types | Auth Type | Sync Mode | | ---------- | ------------------ | ------------------ | --------- | | **GitHub** | Code Files, README | OAuth (GitHub PAT) | Automatic | **Use cases:** * Code search * Documentation in repos * README indexing * API references **Example:** {% tabs %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedServiceTypes } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); const feed = await client.createFeed({ name: "Codebase", type: FeedTypes.Site, site: { type: FeedServiceTypes.GitHub, github: { personalAccessToken: process.env.GITHUB_TOKEN, repositoryOwner: "acmecorp", repositoryName: "backend-api" } } }); ``` {% endtab %} {% tab title="Python" %} ```python import os from graphlit import Graphlit from graphlit_api import FeedTypes, FeedServiceTypes graphlit = Graphlit() feed = await graphlit.client.create_feed( name="Codebase", type=FeedTypes.SITE, site={ "type": FeedServiceTypes.GIT_HUB, "github": { "personalAccessToken": os.getenv("GITHUB_TOKEN"), "repositoryOwner": "acmecorp", "repositoryName": "backend-api" } } ) ``` {% endtab %} {% endtabs %} *** ### Web & Content | Connector | Content Types | Auth Type | Sync Mode | | --------------- | ------------------- | ------------------ | --------------------- | | **Web Pages** | Pages, Files | None | On-demand / Scheduled | | **Web Search** | Search Results | None | On-demand | | **RSS Feeds** | Posts, Articles | None | Automatic | | **Podcast RSS** | Audio, Transcripts | None | Automatic | | **YouTube** | Audio (transcribed) | API Key (optional) | On-demand | | **Reddit** | Posts, Comments | API Key (optional) | Automatic | **Use cases:** * Competitive intelligence * News monitoring * Content aggregation * Market research **Example:** {% tabs %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, SearchServiceTypes, TimedPolicyRecurrenceTypes } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); // RSS feed const rssFeed = await client.createFeed({ name: "Competitor Blog", type: FeedTypes.Rss, rss: { uri: "https://competitor.com/blog/rss" } }); // Web search feed const searchFeed = await client.createFeed({ name: "AI News", type: FeedTypes.Search, search: { type: SearchServiceTypes.Tavily, text: "artificial intelligence semantic memory", readLimit: 10 }, schedulePolicy: { recurrenceType: TimedPolicyRecurrenceTypes.Repeat, repeatInterval: "PT1H" } }); ``` {% endtab %} {% tab title="Python" %} ```python from graphlit import Graphlit from graphlit_api import FeedTypes, SearchServiceTypes, TimedPolicyRecurrenceTypes from datetime import timedelta graphlit = Graphlit() # RSS feed rss_feed = await graphlit.client.create_feed( name="Competitor Blog", type=FeedTypes.RSS, rss={ "uri": "https://competitor.com/blog/rss" } ) # Web search feed search_feed = await graphlit.client.create_feed( name="AI News", type=FeedTypes.SEARCH, search={ "type": SearchServiceTypes.TAVILY, "text": "artificial intelligence semantic memory", "readLimit": 10 }, schedulePolicy={ "recurrenceType": TimedPolicyRecurrenceTypes.REPEAT, "repeatInterval": timedelta(hours=1) } ) ``` {% endtab %} {% endtabs %} *** ### Calendars | Connector | Content Types | Auth Type | Sync Mode | | ---------------------- | ------------- | ----------------- | --------- | | **Google Calendar** | Events | OAuth (Google) | Automatic | | **Microsoft Calendar** | Events | OAuth (Microsoft) | Automatic | **Use cases:** * Meeting recording triggers * Event-based workflows * Schedule context * Attendee tracking **Example:** {% tabs %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedServiceTypes } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); const feed = await client.createFeed({ name: "Team Calendar", type: FeedTypes.Calendar, calendar: { type: FeedServiceTypes.GoogleCalendar, google: { refreshToken: process.env.GOOGLE_REFRESH_TOKEN, clientId: process.env.GOOGLE_CLIENT_ID, clientSecret: process.env.GOOGLE_CLIENT_SECRET // calendarId: "primary" // Optional: defaults to primary calendar } } }); ``` {% endtab %} {% tab title="Python" %} ```python import os from graphlit import Graphlit from graphlit_api import FeedTypes, FeedServiceTypes graphlit = Graphlit() feed = await graphlit.client.create_feed( name="Team Calendar", type=FeedTypes.CALENDAR, calendar={ "type": FeedServiceTypes.GOOGLE_CALENDAR, "google": { "refreshToken": os.getenv("GOOGLE_REFRESH_TOKEN"), "clientId": os.getenv("GOOGLE_CLIENT_ID"), "clientSecret": os.getenv("GOOGLE_CLIENT_SECRET") # "calendarId": "primary" # Optional: defaults to primary calendar } } ) ``` {% endtab %} {% endtabs %} *** ### Meetings & Call Transcripts | Connector | Content Types | Auth Type | Sync Mode | | ------------------ | ------------------------------ | --------- | --------- | | **Fireflies.ai** | Meeting transcripts, summaries | API Key | Automatic | | **Fathom** | Meeting transcripts, summaries | API Key | Automatic | | **Attio Meetings** | Meeting transcripts | API Key | Automatic | **Use cases:** meeting intelligence, searchable call history, action items, follow-ups **Example:** {% tabs %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedServiceTypes } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); const feed = await client.createFeed({ name: 'Fireflies Transcripts', type: FeedTypes.Meeting, meeting: { type: FeedServiceTypes.Fireflies, fireflies: { apiKey: process.env.FIREFLIES_API_KEY } } }); ``` {% endtab %} {% tab title="Python" %} ```python import os from graphlit import Graphlit from graphlit_api import FeedTypes, FeedServiceTypes graphlit = Graphlit() feed = await graphlit.client.create_feed( name="Fireflies Transcripts", type=FeedTypes.MEETING, meeting={ "type": FeedServiceTypes.FIREFLIES, "fireflies": { "apiKey": os.getenv("FIREFLIES_API_KEY") } } ) ``` {% endtab %} {% endtabs %} *** ### CRM & Contacts | Connector | Content Types | Auth Type | Sync Mode | | ---------------------- | ------------- | ------------------ | --------- | | **Attio CRM** | CRM objects | API Key | Automatic | | **Google Contacts** | Contacts | OAuth (Google) | Automatic | | **Microsoft Contacts** | Contacts | OAuth (Microsoft) | Automatic | | **Salesforce CRM** | CRM objects | OAuth (Salesforce) | Automatic | **Use cases:** account context, contact enrichment, sales/support history, relationship intelligence *** ### Research & Entity Discovery | Connector | Content Types | Auth Type | Sync Mode | | ----------------------------- | ------------------------------ | --------- | --------- | | **Parallel Research** | Research reports, source links | API Key | Automatic | | **Parallel Entity Discovery** | Entity candidates + sources | API Key | Automatic | **Use cases:** competitive intel, market maps, account research, enrichment pipelines **Example:** {% tabs %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedServiceTypes } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); const feed = await client.createFeed({ name: 'Parallel Research', type: FeedTypes.Research, research: { type: FeedServiceTypes.Parallel, query: 'Who are the top competitors to Graphlit, and what are their differentiators?' } }); ``` {% endtab %} {% tab title="Python" %} ```python from graphlit import Graphlit from graphlit_api import FeedTypes, FeedServiceTypes graphlit = Graphlit() feed = await graphlit.client.create_feed( name="Parallel Research", type=FeedTypes.RESEARCH, research={ "type": FeedServiceTypes.PARALLEL, "query": "Who are the top competitors to Graphlit, and what are their differentiators?" } ) ``` {% endtab %} {% endtabs %} *** *** ## Feed Sync Modes ### Real-Time Sync (ARCHIVE) **Default mode**: Preserve everything, never delete {% tabs %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, TimedPolicyRecurrenceTypes } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); const feed = await client.createFeed({ name: "Slack Archive", type: FeedTypes.Slack, slack: { // ... slack configuration }, schedulePolicy: { recurrenceType: TimedPolicyRecurrenceTypes.Repeat, repeatInterval: "PT5M" // Check every 5 minutes (ISO 8601 duration) } }); ``` {% endtab %} {% tab title="Python" %} ```python from graphlit import Graphlit from graphlit_api import FeedTypes, TimedPolicyRecurrenceTypes from datetime import timedelta graphlit = Graphlit() feed = await graphlit.client.create_feed( name="Slack Archive", type=FeedTypes.SLACK, slack={ # ... slack configuration }, schedulePolicy={ "recurrenceType": TimedPolicyRecurrenceTypes.REPEAT, "repeatInterval": timedelta(minutes=5) } ) ``` {% endtab %} {% endtabs %} **Behavior:** * New content added * Updated content re-indexed * Deleted content at source ā†’ **preserved** in Graphlit * Use case: Compliance, audit logs, historical research *** ### Mirror Sync (MIRROR) **Mirror mode**: Keep in sync with source {% tabs %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedSyncMode } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); const feed = await client.createFeed({ name: "Google Drive Mirror", type: FeedTypes.Site, site: { // ... site configuration }, syncMode: FeedSyncMode.Mirror // Mirror mode }); ``` {% endtab %} {% tab title="Python" %} ```python from graphlit import Graphlit from graphlit_api import FeedTypes, FeedSyncMode graphlit = Graphlit() feed = await graphlit.client.create_feed( name="Google Drive Mirror", type=FeedTypes.SITE, site={ # ... site configuration }, syncMode=FeedSyncMode.MIRROR ) ``` {% endtab %} {% endtabs %} **Behavior:** * New content added * Updated content re-indexed * Deleted content at source ā†’ **deleted** in Graphlit * Use case: Live documentation, current state only *** ## OAuth Setup ### Graphlit Manages OAuth **You don't need to:** * āŒ Store refresh tokens securely * āŒ Handle token expiration * āŒ Implement refresh logic * āŒ Manage webhook subscriptions **Graphlit handles:** * āœ… Token storage (encrypted) * āœ… Automatic refresh * āœ… Webhook management * āœ… Error recovery ### Setup Process 1. **Get OAuth credentials** from provider (e.g., Google Cloud Console, Microsoft Azure Portal) 2. **Exchange for refresh token** (one-time) 3. **Pass to Graphlit** when creating feed 4. **Done** - Graphlit manages tokens forever **See**: [OAuth Setup Guide](/mcp-integration/mcp-integration#oauth-connectors-via-mcp) for detailed instructions per provider. *** ## Feed Filters ### Filter by Query (Gmail, Slack, etc.) {% tabs %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedServiceTypes } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); const feed = await client.createFeed({ name: "Customer Emails", type: FeedTypes.Email, email: { type: FeedServiceTypes.GoogleEmail, google: { query: "from:*@acmecorp.com OR to:*@acmecorp.com" } } }); ``` {% endtab %} {% tab title="Python" %} ```python from graphlit import Graphlit from graphlit_api import FeedTypes, FeedServiceTypes graphlit = Graphlit() feed = await graphlit.client.create_feed( name="Customer Emails", type=FeedTypes.EMAIL, email={ "type": FeedServiceTypes.GOOGLE_EMAIL, "google": { "query": "from:*@acmecorp.com OR to:*@acmecorp.com" } } ) ``` {% endtab %} {% endtabs %} ### Filter by Folder/Channel {% tabs %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedServiceTypes } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); const feed = await client.createFeed({ name: "Sales Folder", type: FeedTypes.Site, site: { type: FeedServiceTypes.OneDrive, oneDrive: { // folderId: "..." // Optional: sync specific folder only } } }); ``` {% endtab %} {% tab title="Python" %} ```python from graphlit import Graphlit from graphlit_api import FeedTypes, FeedServiceTypes graphlit = Graphlit() feed = await graphlit.client.create_feed( name="Sales Folder", type=FeedTypes.SITE, site={ "type": FeedServiceTypes.ONE_DRIVE, "oneDrive": { # "folderId": "..." # Optional: sync specific folder only } } ) ``` {% endtab %} {% endtabs %} ### Filter by Time (Query-Time) {% tabs %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { ContentTypes, SearchTypes } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); // Feeds generally don't support "oldest message" cutoffs; instead, ingest and then filter at query-time. const response = await client.queryContents({ types: [ContentTypes.Message], createdInLast: "PT2160H", // ~90 days searchType: SearchTypes.Hybrid, }); ``` {% endtab %} {% tab title="Python" %} ```python from graphlit import Graphlit from graphlit_api.enums import ContentTypes, SearchTypes from graphlit_api.input_types import ContentFilter graphlit = Graphlit() response = await graphlit.client.query_contents( filter=ContentFilter( types=[ContentTypes.MESSAGE], created_in_last="PT2160H", # ~90 days search_type=SearchTypes.HYBRID, ), ) ``` {% endtab %} {% endtabs %} *** ## Common Patterns ### Pattern 1: Multi-Source Customer View {% tabs %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; const client = new Graphlit(); // Connect all customer touchpoints const feeds = []; // Email conversations feeds.push(await client.createFeed({ /* Gmail config */ })); // Slack mentions feeds.push(await client.createFeed({ /* Slack config */ })); // Support tickets feeds.push(await client.createFeed({ /* Zendesk config */ })); // Meeting notes feeds.push(await client.createFeed({ /* Google Drive config */ })); // Now search across all sources const response = await client.queryContents({ search: "Acme Corp pricing concerns", }); // ā†‘ One query ā†’ all customer interactions ``` {% endtab %} {% tab title="Python" %} ```python from graphlit import Graphlit graphlit = Graphlit() # Connect all customer touchpoints feeds = [] # Email conversations feeds.append(await graphlit.client.create_feed(...)) # Gmail # Slack mentions feeds.append(await graphlit.client.create_feed(...)) # Slack # Support tickets feeds.append(await graphlit.client.create_feed(...)) # Zendesk # Meeting notes feeds.append(await graphlit.client.create_feed(...)) # Google Drive # Now search across all sources response = await graphlit.client.query_contents( filter={ "search": "Acme Corp pricing concerns" } ) # ā†‘ One query ā†’ all customer interactions ``` {% endtab %} {% endtabs %} *** ### Pattern 2: Developer Knowledge Base {% tabs %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; const client = new Graphlit(); // Connect engineering sources const feeds = [ // Code await client.createFeed({ /* GitHub backend-api */ }), await client.createFeed({ /* GitHub frontend-app */ }), // Discussions await client.createFeed({ /* Slack engineering */ }), await client.createFeed({ /* Slack architecture */ }), // Issues await client.createFeed({ /* Linear engineering-team */ }), // Docs await client.createFeed({ /* Notion engineering-wiki */ }), // Meetings await client.createFeed({ /* Google Drive meeting-notes */ }) ]; // Agent has full engineering context ``` {% endtab %} {% tab title="Python" %} ```python from graphlit import Graphlit graphlit = Graphlit() # Connect engineering sources feeds = [ # Code create_github_feed("backend-api"), create_github_feed("frontend-app"), # Discussions create_slack_feed("engineering"), create_slack_feed("architecture"), # Issues create_linear_feed("engineering-team"), # Docs create_notion_feed("engineering-wiki"), # Meetings create_google_drive_feed("meeting-notes") ] # Agent has full engineering context ``` {% endtab %} {% endtabs %} *** ### Pattern 3: Compliance Archive {% tabs %} {% tab title="TypeScript" %} ```typescript import { Graphlit } from 'graphlit-client'; import { FeedSyncMode } from 'graphlit-client/dist/generated/graphql-types'; const client = new Graphlit(); // Archive everything (never delete) const feeds = [ await client.createFeed({ syncMode: FeedSyncMode.Archive /* Gmail config */ }), await client.createFeed({ syncMode: FeedSyncMode.Archive /* Slack config */ }), await client.createFeed({ syncMode: FeedSyncMode.Archive /* Teams config */ }) ]; // All communications preserved // Searchable for compliance // Audit trail maintained ``` {% endtab %} {% tab title="Python" %} ```python from graphlit import Graphlit from graphlit_api import FeedSyncMode graphlit = Graphlit() # Archive everything (never delete) feeds = [ await graphlit.client.create_feed(syncMode=FeedSyncMode.ARCHIVE, ...), # Gmail await graphlit.client.create_feed(syncMode=FeedSyncMode.ARCHIVE, ...), # Slack await graphlit.client.create_feed(syncMode=FeedSyncMode.ARCHIVE, ...) # Teams ] # All communications preserved # Searchable for compliance # Audit trail maintained ``` {% endtab %} {% endtabs %} *** ## Real-World Example: Zine [Zine](https://www.zine.ai) uses 20+ Graphlit connectors: **Connected sources:** * Slack, Teams, Discord * Gmail, Outlook * Google Drive, OneDrive, Dropbox * Notion, Linear, Jira * GitHub Issues * Google Calendar, Microsoft Calendar * Meeting recordings (auto-attached to calendar events) **Result:** * One search across all tools * AI agents with full company context * Meeting intelligence * Customer interaction history * Developer knowledge base **All powered by Graphlit connectors.** *** ## Connector Roadmap **Coming soon** (upon request): * Confluence * ServiceNow * Azure Repos * Azure Boards **Want a connector?** [Request it in Discord](https://discord.gg/ygFmfjy3Qx) *** ## Next Steps * [**MCP Integration**](/mcp-integration/mcp-integration) - Use connectors via MCP * [**Platform Overview**](/getting-started/overview) - See how connectors fit into the platform *** **Connect once. Search forever.** # Semantic Memory Understanding semantic memory for AI - how Graphlit implements true memory systems beyond simple retrieval Semantic memory is the ability to understand, organize, and retrieve knowledge based on meaning and relationships - not just keywords or similarity scores. {% hint style="info" %} **Think of human memory:** You don't remember every word of every conversation. You remember **people, places, events, and how they relate**. You can recall "What did Sarah from Acme Corp say about pricing last month?" That's semantic memory. That's what Graphlit provides for AI. {% endhint %} *** ## Memory Types in AI AI agents need different types of memory, mirroring human cognition: {% @mermaid/diagram content="graph TD A\[AI Agent Memory] --> B\[Working Memory] A --> C\[Long-Term Memory] B --> B1\[Active Context
LLM Context Window] B --> B2\[Current Conversation] C --> D\[Episodic Memory] C --> E\[Semantic Memory] C --> F\[Procedural Memory] D --> D1\[Specific Events
Conversation History] E --> E1\[Facts & Relationships
Knowledge Graph] F --> F1\[How-to Knowledge
Workflows]" %} ### Memory Types in Graphlit | Type | What It Stores | Graphlit Implementation | Example Query | | -------------- | ------------------------------ | ---------------------------------------------------------------- | --------------------------------------- | | **Episodic** | Specific events, experiences | Conversation history, meeting transcripts, timestamps | "What was discussed in Oct 15 meeting?" | | **Semantic** | Facts, concepts, relationships | Entity extraction (Person, Organization, Event), knowledge graph | "Who works at Acme Corp?" | | **Procedural** | Skills, processes, how-to | Workflows, action sequences | "How do we deploy to production?" | | **Working** | Active context | LLM context window, current conversation | Current task focus | {% hint style="success" %} **Key insight:** Traditional RAG only uses working memory (context window). Graphlit provides all four memory types - a complete cognitive system. {% endhint %} *** ## Memory Formation Cycle Memory isn't just retrieval - it's a complete cognitive cycle: ### The Cycle Explained 1. **Ingestion**: Content enters (files, messages, meetings, web pages) 2. **Extraction**: LLM identifies entities, relationships, facts 3. **Consolidation**: Merge with existing knowledge, dedupe, organize 4. **Storage**: Persist in knowledge graph + vector store + object store 5. **Retrieval**: Query by meaning, entities, relationships, time 6. **Context Injection**: Relevant memories ā†’ LLM context window **RAG only does steps 5-6**. Semantic memory does the complete cycle. *** ## RAG vs Semantic Memory ### The Evolution | Era | Approach | What It Does | Limitation | | -------- | ------------------- | ----------------------- | --------------------- | | 2022 | Vector Search | Similarity matching | No understanding | | 2023 | RAG | Retrieve + generate | Still just similarity | | 2024 | GraphRAG | Graph + vectors | Complex to build | | **2025** | **Semantic Memory** | Meaning + relationships | Production-ready | ### What RAG Actually Is **Limitations**: * Each query independent (stateless) * No entity understanding * No relationship tracking * No temporal context * Just similarity scores **Example**: "What did Sarah say about pricing?" * RAG searches for vectors similar to query * Might return any mention of "Sarah" or "pricing" * No guarantee it's the right Sarah or relevant pricing discussion ### What Semantic Memory Is **Capabilities**: * Understands entities (Sarah Chen = person at Acme Corp) * Tracks relationships (Sarah works\_at Acme) * Temporal awareness (last month vs last year) * Stateful (knowledge builds over time) **Same query**: "What did Sarah say about pricing?" * Identifies "Sarah" as Person entity * Finds all content where Sarah is mentioned * Filters for pricing discussions * Returns ordered by time with full context **Result**: Actual useful answer. *** ## Knowledge Graph: The Semantic Memory Layer Knowledge graphs ARE the semantic memory - they store entities and their relationships. ### Schema.org Foundation Graphlit builds on **Schema.org** (JSON-LD), the industry standard: **Benefits**: * Standardized entity types (Person, Organization, Place, Event) * Interoperable with other systems * Rich vocabulary (Google, Microsoft use it) * Extensible **Example entity**: ```json { "@context": "https://schema.org", "@type": "Person", "name": "Sarah Chen", "jobTitle": "CTO", "worksFor": { "@type": "Organization", "name": "Acme Corp" } } ``` ### How the Graph is Built "**Observations of observable entities**" - LLM-driven extraction: **Process**: 1. LLM reads content: "Sarah Chen from Acme Corp mentioned pricing concerns" 2. Extracts entities: Person (Sarah), Organization (Acme Corp) 3. Creates relationships: Sarah works\_at Acme Corp 4. Links to source content 5. Knowledge graph evolves This enables multi-hop queries impossible with pure vector search: "What technical concerns have CTOs from enterprise customers raised in Q4?" *** ## How Graphlit Implements Semantic Memory ### Multi-Layer Architecture ### Temporal & Spatial Indexing **What Graphlit tracks**: * Content ingestion date * Content authorship date (from metadata) * Media timestamps ("At 5:23 in video") * Extracted event dates ("Meeting on Oct 15") * Geo-location (from metadata) **Query examples that work**: * "Content from Acme Corp in last 30 days" āœ… * "Meeting about pricing in Q4" āœ… * "At what point in video did they discuss architecture?" āœ… *** ## Intelligent Query Processing ### Two-Layer Architecture **Layer 1: Pre-Parsing** (natural language ā†’ structured filters) ```python import json from graphlit_api.input_types import ToolDefinitionInput # Natural language query query = "Show me Acme Corp pricing discussions from last week" # Use extractText() with a tool schema to convert natural language into a ContentFilter. tools = [ ToolDefinitionInput( name="content_filter", description="Return a JSON object compatible with Graphlit ContentFilter.", schema=json.dumps( { "type": "object", "properties": { "search": {"type": "string"}, "creationDateRange": { "type": "object", "properties": { "startDate": {"type": "string"}, "endDate": {"type": "string"}, }, }, }, } ), ) ] extracted = await graphlit.client.extract_text( prompt="Convert the query into a ContentFilter and return it via the content_filter tool.", text=query, tools=tools, ) filters = json.loads(extracted.extract_text[0].value) ``` **Layer 2: Hybrid Search** (vector + keyword) ```python # Execute with extracted filters results = await graphlit.client.query_contents(filter=filters) # Combines: # - Vector search: Semantic understanding # - Keyword search: Exact entity matching # - Date filtering: Temporal context ``` [See it in action: Filter Extraction Colab ā†’](https://github.com/graphlit/graphlit-samples) *** ## Accessing Semantic Memory ### Pattern 1: Manual RAG (Advanced) **Use `retrieveSources` for custom RAG pipelines:** ```python from graphlit_api.input_types import ContentFilter # Get LLM-optimized sources (reranked + context expanded) sources = await graphlit.client.retrieve_sources( prompt="Acme Corp pricing discussions from last week", filter=ContentFilter(search="Acme Corp pricing"), ) # Build your own prompt context = "\n\n".join([s.text for s in sources.retrieve_sources.results]) prompt = f"Based on this context:\n\n{context}\n\nAnswer: What are pricing concerns?" # Call your own LLM ``` **Features**: Reranking, context expansion (chunk ā†’ section/page), you control prompts *** ### Pattern 2: Automated RAG (Recommended) **Use `promptConversation` for turnkey Q\&A:** ```python # Let Graphlit handle everything response = await graphlit.client.prompt_conversation( prompt="What are Acme Corp's pricing concerns?", id=conversation_id ) # Behind the scenes: # 1. Uses retrieveSources internally # 2. Reranks for relevance # 3. Expands context # 4. Generates answer # 5. Returns with citations message = response.prompt_conversation.message print(message.message) for citation in message.citations or []: print(f"Source: {citation.content.name}") ``` **Most common pattern** - ideal for chatbots, Q\&A systems. *** ### Pattern 3: Agentic Workflows (TypeScript) **Use `promptAgent` or `streamAgent` for autonomous agents:** ```typescript // Agent decides when/how to retrieve await graphlit.streamAgent( 'Analyze Acme Corp interactions and summarize concerns', (event) => { // stream tokens + tool calls to your UI console.log(event); }, conversationId, ); // Agent autonomously: // - Calls retrieveSources as tool // - Extracts entities // - Synthesizes findings // - Streams response with UI events ``` **Like Zine**: Real-time streaming chat with tool transparency. [Learn more: AI Agents Quickstart ā†’](/tutorials/ai-agents) *** ### Pattern 4: Search UI (queryContents) **For building dashboards, not RAG:** ```python # Generic content search (not LLM-optimized) results = await graphlit.client.query_contents( filter=ContentFilter( search="Acme Corp", types=[ContentTypes.FILE, ContentTypes.EMAIL] ) ) # Display in UI for content in results.contents.results: print(f"{content.name} ({content.creation_date})") ``` *** ## Practical Example: Multi-Hop Query Query: **"What technical concerns have CTOs from enterprise customers raised in Q4?"** ### Memory Path: **Step 1: Parse entities** * "CTOs" ā†’ role filter * "enterprise customers" ā†’ organization type * "Q4" ā†’ date range **Step 2: Query knowledge graph** * Find all Person entities with jobTitle="CTO" * Filter to Organization entities with type="enterprise" * Get all content mentioning those people in Q4 date range **Step 3: Extract topics** * Search for "technical concerns" in filtered content * Rank by relevance and recency **Step 4: Return with context** * Who said it (entity) * When (timestamp) * Where (content source) * Full context **This is impossible with RAG alone** - requires entity understanding, relationships, and temporal awareness. *** ## The "iPhoto" Analogy Remember iPhoto? It changed how we organize photos. **Before iPhoto**: * Manual folders * Hard to find: "Photos of Mom at the beach" * No context **After iPhoto**: * People recognition: "Show me all photos of Mom" * Places: "Photos taken in Hawaii" * Events: "Christmas 2016" * Automatic, no manual tagging ### Graphlit Does This for Company Knowledge **Before Graphlit**: * Files scattered across tools (Slack, email, Notion, Drive) * Manual search in each tool * No relationships * Can't find "all interactions with Acme Corp" **After Graphlit**: * Semantic understanding: "Show me everything about Acme Corp" * Entity recognition: People, companies, events automatically * Unified search across all tools * Contextual: Knows relationships and time > **"Like iPhoto for your company's knowledge"** *** ## Use Cases Enabled ### Sales Agent (Episodic + Semantic Memory) ```python # Episodic: Past conversations conversations = await graphlit.client.query_contents(filter=ContentFilter( search="Acme Corp", types=[ContentTypes.MESSAGE, ContentTypes.EMAIL] )) # Semantic: Organization relationships entities = await graphlit.client.query_persons(filter=PersonFilter( search="Acme Corp" )) # Combined: Complete deal context # - All people contacted # - All conversations # - Key objections and requirements # - Timeline of engagement ``` *** ### Developer Agent (Episodic + Semantic Memory) ```python # Episodic: Specific PR discussion pr_context = await graphlit.client.query_contents(filter=ContentFilter( search="PR #247" )) # Semantic: Related architecture decisions architecture = await graphlit.client.query_contents(filter=ContentFilter( search="Redis caching architecture" )) # Combined: Root cause analysis # - PR discussion # - Related Slack threads # - Meeting where decision was made # - Who made it and why ``` *** ### Customer Intelligence (Episodic + Semantic) Combine conversation history (episodic) with entity relationships (semantic) to understand customer health: * Email sentiment over time * Support ticket trends * Product usage patterns * Meeting sentiment * Churn risk signals All connected through knowledge graph. *** ## Real-World Example: Zine [Zine](https://www.zine.ai) is built 100% on Graphlit's semantic memory. **What users ask**: > "What did Acme Corp say about pricing in our last 3 calls?" **What happens** (powered by Graphlit): 1. Identify "Acme Corp" as Organization entity 2. Find all Event entities (calls) with Acme Corp 3. Filter last 3 calls 4. Search meeting transcripts for "pricing" 5. Return quotes with timestamps and context **Result**: Answer in seconds, not hours of manual searching. [See Zine's architecture ā†’](/examples/zine-case-study) *** ## Why This Matters for Production AI ### Traditional Approach (RAG) ``` Ingest documents ā†“ Create embeddings ā†“ Vector search ā†“ Hope it works ``` **Problems**: * Tweak embeddings models endlessly * Results "meh" - similar but not relevant * Can't query by entity, time, or relationship * Agents forget context * Complex workflows don't work *** ### Semantic Memory Approach ``` Connect tools ā†“ Auto-extract entities & relationships ā†“ Build knowledge graph ā†“ Query by meaning ā†“ Agents remember context ``` **Benefits**: * Build features, not infrastructure * Results are relevant and contextual * Query by entity, time, relationship * Agents have true memory * Complex workflows just work *** ## The 2025 Shift ### From: * "Let's build a RAG system" * "What vector database should we use?" * "How do we chunk documents?" * "Which embedding model is best?" ### To: * "Let's give our AI semantic memory" * "Connect our tools to Graphlit" * "Query by entities and relationships" * "Build agents that remember" *** ## Key Principles 1. **Memory > Search** - Don't just search, remember 2. **Entities > Documents** - Track people, companies, events 3. **Context > Chunks** - Preserve relationships and time 4. **Meaning > Keywords** - Semantic understanding 5. **Agents > Chatbots** - Persistent memory across sessions *** ## Learn More **Build with Semantic Memory:** * [Quickstart: Your First Agent](/getting-started/quickstart) - Build a streaming agent in 7 minutes * [AI Agents](/tutorials/ai-agents) - Build agents with memory * [Knowledge Graph](/tutorials/knowledge-graph) - Extract entities * [Context Engineering](/tutorials/context-engineering) - Optimize memory **Understand the Platform:** * [Platform Overview](/getting-started/overview) - Complete platform capabilities * [Key Concepts](/platform/key-concepts) - Data model overview * [Connectors](https://github.com/graphlit/graphlit-docs/blob/main/platform/connectors.md) - 30+ integrations * [AI Models](/platform/models) - GPT-5, Claude 4.5, Gemini 2.5 **See It in Production:** * [Zine Case Study](/examples/zine-case-study) - Real-world patterns * [Sample Apps](https://github.com/graphlit/graphlit-samples) - 60+ examples *** **Give your AI semantic memory. Build with Graphlit.** # MCP Server Setup ā±ļø **Time to Complete:** 10 minutes\ šŸŽÆ **Level:** Beginner\ šŸ’» **Language:** TypeScript (MCP Server) ## What You'll Build * āœ… Connect Graphlit to your MCP-compatible IDE (Cursor, Windsurf, Claude Desktop, Cline) * āœ… Search your Graphlit project (Slack, Gmail, Notion, etc.) from your editor * āœ… Query conversations, collections, feeds, and workflows * āœ… Ingest new content or trigger workflows without leaving the IDE [**šŸ“ Graphlit MCP Server Repository**](https://github.com/graphlit/graphlit-mcp-server) *** ## Prerequisites * Node.js 20+ * One of the MCP-aware IDEs (Cursor, Windsurf, Claude Desktop, or VS Code with Cline) * Graphlit credentials in `.env` (from Getting Started guide): * `GRAPHLIT_ORGANIZATION_ID` * `GRAPHLIT_ENVIRONMENT_ID` * `GRAPHLIT_JWT_SECRET` * `npm install -g graphlit-mcp-server` (or use `npx` in configs below) {% hint style="info" %} **Need Graphlit code examples?** Check out [Ask Graphlit](/resources/ask-graphlit) - an AI assistant that generates Graphlit SDK code in Python, TypeScript, or .NET. This MCP integration is for accessing your **data** (Slack, Gmail, content), not for code help. {% endhint %} *** ## Setup The Graphlit MCP Server exposes your actual Graphlit project: contents, feeds, collections, workflows, conversations, and specifications. ### Environment Create `.env` with your Graphlit credentials (from Getting Started): ```bash GRAPHLIT_ORGANIZATION_ID=your_org_id_from_portal GRAPHLIT_ENVIRONMENT_ID=your_env_id_from_portal GRAPHLIT_JWT_SECRET=your_jwt_secret_from_portal ``` Install and test locally: ```bash npm install -g graphlit-mcp-server # or npx graphlit-mcp-server npx graphlit-mcp-server --test ``` ### IDE Configuration #### Cursor / Windsurf / Cline / Claude Desktop ```json { "mcpServers": { "graphlit": { "command": "npx", "args": ["graphlit-mcp-server"], "env": { "GRAPHLIT_ORGANIZATION_ID": "${GRAPHLIT_ORGANIZATION_ID}", "GRAPHLIT_ENVIRONMENT_ID": "${GRAPHLIT_ENVIRONMENT_ID}", "GRAPHLIT_JWT_SECRET": "${GRAPHLIT_JWT_SECRET}" } } } } ``` Restart your IDE. Graphlit will appear alongside other MCP tools. *** ## Key MCP Tools (from `graphlit-mcp-server`) The server registers 60+ tools via `registerTools`: | Category | Examples | | ------------- | -------------------------------------------------------------------------------------------------------- | | Retrieval | `queryContents`, `queryCollections`, `queryFeeds`, `queryConversations`, `retrieveRelevantSources` | | Conversations | `promptConversation` (streams via MCP channel) | | Ingestion | `ingestFile`, `ingestText`, `ingestWebPage`, `ingestEmail`, `ingestIssue`, `ingestMessage`, `ingestFeed` | | Publishing | `publishAudio`, `publishImage` | | Operations | `configureProject`, `createCollection`, `addContentsToCollection`, `deleteContent`, `isContentDone` | | Enumerations | `listSlackChannels`, `listNotionDatabases`, `listLinearProjects`, `listGoogleCalendars`, etc. | All enumerations, types, and arguments match the TypeScript SDK. For example: ```typescript import { FeedTypes, FeedListingTypes, EmailListingTypes, SearchServiceTypes, } from 'graphlit-client/dist/generated/graphql-types'; ``` Use these same enums in MCP prompts when specifying properties. *** ## Example Prompts (Local Server) ### Search Company Knowledge ``` @graphlit query_contents search="redis failover" searchType=HYBRID limit=5 ``` ### Scope by Collection + Entity ``` @graphlit query_contents search="pricing" searchType=HYBRID collections=["${GRAPHLIT_COLLECTION_SUPPORT}"] observations=[{"type":"ORGANIZATION","observable":{"id":"${GRAPHLIT_ORG_ACME}"}}] ``` ### Ingest a File ``` @graphlit ingest_file path="./runbooks/cache-playbook.pdf" workflow={"id":"${WORKFLOW_ID}"} isSynchronous=true ``` ### Prompt a Conversation ``` @graphlit prompt_conversation id="${CONVERSATION_ID}" prompt="Summarize the last three incidents for Acme Corp and list remaining blockers." ``` The MCP server streams token updates back to the IDE when using `promptConversation`. *** ## Integrating with Other MCP Servers Combine Graphlit with filesystem, GitHub, database or cloud MCP servers in one config: ```json { "mcpServers": { "graphlit": { ... }, "filesystem": { "command": "npx", "args": ["@modelcontextprotocol/server-filesystem", "/path/to/code"] }, "github": { "command": "npx", "args": ["@modelcontextprotocol/server-github"], "env": { "GITHUB_TOKEN": "${GITHUB_TOKEN}" } } } } ``` Now your AI can pull: * Local source files * Open PRs and discussions * Production incidents stored in Graphlit * Slack, email, meetings tied to an account *** ## Production Tips 1. **Separate projects per tenant/team**: set `GRAPHLIT_ENVIRONMENT_ID` per MCP entry. 2. **Persist identifiers**: store collection/workflow IDs in env vars to reuse across prompts. 3. **Use `isContentDone`** before prompting to avoid stale context on newly ingested data. 4. **Secure secrets**: reference OS-level environment variables (`${VAR}`) instead of hardcoding values. 5. **Log tool output**: Cursor and Claude show MCP logs; use them to audit retrieved sources. *** ## Troubleshooting * Run `npx graphlit-mcp-server --test` to validate credentials. * Confirm Node.js 20+ and that `graphlit-client` installs without errors. * OAuth-based tools (Slack, Gmail, GitHub) require tokens in the environment; follow instructions in the MCP server README. * If tools fail, inspect IDE MCP logs; most errors surface there (`stdout`/`stderr`). *** ## Next Steps * [**AI Agents**](/tutorials/ai-agents) ā€“ orchestrate tools from inside Graphlit agents. * [**Context Engineering**](/tutorials/context-engineering) ā€“ design memory and retrieval strategies. * [**Knowledge Graph**](/tutorials/knowledge-graph) ā€“ extract entities your MCP prompts can filter on. With MCP + Graphlit, your IDE has direct access to the same semantic memory your production agents use. Build smarter tooling without leaving the editor. # Ask Graphlit - AI Code Assistant AI code assistant trained on Graphlit - get instant working code examples for any task **Your AI pair programmer for Graphlit** - trained on all documentation, samples, and API patterns. Don't know the right operation name? Not sure about the parameters? Just ask in natural language and get working code. [**šŸš€ Open Ask Graphlit in the Developer Portal ā†’**](https://portal.graphlit.dev) Prefer the standalone experience? Visit [ask.graphlit.dev](https://ask.graphlit.dev). {% hint style="success" %} **No credit surprises**: Ask Graphlit is **unbilled** (it does not consume your project credits). {% endhint %} *** ## What is Ask Graphlit? Ask Graphlit is an AI chatbot that understands Graphlit's API and can generate working code in Python, TypeScript, or .NET based on your intent. Instead of searching through documentation or memorizing operation names, just describe what you want to do. *** ## Billing, Data Access, and Trust * āœ… **Unbilled**: Ask Graphlit does **not** consume your project credits. * āœ… **No project access**: Ask Graphlit does not have your Graphlit credentials and cannot access your content. * āœ… **Grounded answers**: Ask Graphlit is preloaded with the current GraphQL schema and documentation context. If you want an AI tool to search **your actual Graphlit data**, use the **Graphlit MCP Server** (local) from the [MCP integration guide](/mcp-integration/mcp-integration). ### The Discovery Problem **Traditional API documentation:** * You need to know operation names ("createWorkflow", "promptConversation") * You browse through exhaustive parameter lists * You read bottom-up (operations ā†’ use cases) **Ask Graphlit:** * You describe your intent ("extract entities from PDF") * You get working code immediately * You learn top-down (use case ā†’ operations) *** ## When to Use Ask Graphlit ### Perfect For **Learning the API:** * "How do I ingest a PDF?" * "Show me code to create a Slack feed" * "How do I extract entities from content?" **Finding the Right Operation:** * "What's the operation to search across multiple sources?" * "How do I create a conversation with custom model?" * "Show me streaming conversation code" **Getting Code Examples:** * "Give me Python code to ingest and search a PDF" * "Show TypeScript example for knowledge graph extraction" * "How do I use tool calling with conversations?" **Troubleshooting:** * "Why isn't my content finishing processing?" * "How do I wait for ingestion to complete?" * "What's the right way to handle async operations?" ### Not Ideal For * **Accessing your actual data** - Use [Graphlit MCP Server](/mcp-integration/mcp-integration) instead * **Real-time project integration** - Install MCP server locally * **Non-code questions** - Check [Getting Started](/getting-started/overview) or the [Quickstart tutorial](/getting-started/quickstart) *** ## SDK Parity (Python, TypeScript, .NET) All three SDKs are **code-generated from the same GraphQL schema**, so **core API operations and types are equivalent**. What differs: * **Naming conventions**: * TypeScript: `camelCase` * Python: `snake_case` * .NET: `PascalCase` * **TypeScript-only helpers**: the TypeScript SDK includes optional convenience wrappers (e.g., `streamAgent`, `promptAgent`) that build on top of the same underlying API operations. *** ## How to Use It ### Option 1: Developer Portal Chatbot Sign in to the [Graphlit Developer Portal](https://portal.graphlit.dev) and open **Ask Graphlit** from the sidebar. The chatbot lives alongside your projects so you can copy code straight into your app. Prefer the standalone site? Use [**ask.graphlit.dev**](https://ask.graphlit.dev). *** ### Option 2: In Your IDE (MCP) Integrate Ask Graphlit into Cursor, Windsurf, VS Code (Cline), or Claude Desktop. **Add this endpoint to your MCP configuration:** ``` https://ask.graphlit.dev/mcp ``` **Cursor example:** ```json { "mcpServers": { "ask-graphlit": { "url": "https://ask.graphlit.dev/mcp" } } } ``` Then use it like: ``` @ask-graphlit How do I ingest a PDF and extract entities? ``` [**Full MCP setup guide ā†’**](/mcp-integration/mcp-integration) *** ## Example Queries {% hint style="success" %} **Pro tip**: Add your preferred language to any query - "Show me **TypeScript** code to...", "Give me the **Python** version", or "How would I do this in **C#**?" {% endhint %} ### Ingestion ``` How do I ingest a PDF from a URL? Show me TypeScript code to create an RSS feed How do I upload a file and wait for it to finish processing? Give me Python code to ingest Slack messages ``` ### Searching & Retrieval ``` How do I search across all my content? Show me semantic search with filters in TypeScript How do I search within specific collections? What's the Python code to find content by entity? ``` ### Conversations & RAG ``` How do I create a conversation in TypeScript? Show me streaming conversation code Give me the C# version of tool calling with conversations What's the difference between promptConversation and streamAgent? ``` ### Knowledge Graph ``` How do I extract entities from content? Show me code to query people and organizations How do I build a knowledge graph from web search? What entities can I extract? ``` ### Workflows ``` How do I create a workflow? Show me workflow with entity extraction How do I use vision models for PDF extraction? What workflow stages are available? ``` ### Feeds & Connectors ``` How do I connect to Slack? Show me code to sync Google Drive How do I create a podcast RSS feed? What OAuth connectors are available? ``` *** ## What Ask Graphlit Knows Ask Graphlit is trained on: **Documentation:** * All quickstart tutorials * Platform concept pages * API patterns and best practices * MCP integration guides **Code Samples:** * 60+ Google Colab notebooks * Next.js application examples * Streamlit UI applications * Production patterns (from Zine) **API Coverage:** * All SDK operations (Python, TypeScript, .NET) * GraphQL schema and types * Parameter options and configurations * Common error patterns and solutions **Real-World Patterns:** * Multi-tenant architecture * Production deployment * Error handling * Async workflows * Tool calling with agents * Streaming responses *** ## Tips for Best Results ### Specify Your SDK/Language **Ask Graphlit supports all 3 SDKs** - just mention your preference: Instead of: > "How do I ingest content?" Try: > "Show me **TypeScript** code to ingest a PDF" "Give me the **Python** version" "Show this in **C#/.NET**" ### Describe Your Use Case Instead of: > "How do I use conversations?" Try: > "I want to chat with my ingested PDFs using GPT-4, show me the code" ### Ask for Complete Examples Instead of: > "What parameters does createWorkflow take?" Try: > "Give me a complete example of creating a workflow with entity extraction" ### Change Languages Mid-Conversation You can switch SDKs anytime: * "Show me this in TypeScript" * "Convert that to Python" * "Give me the .NET version" * "How would this look in C#?" *** ## Related Resources **Learn by Doing:** * [Quickstart: Your First Agent](/getting-started/quickstart) - Build a streaming agent in 7 minutes * [AI Agents Tutorial](/tutorials/ai-agents) - Build agents with memory * [Sample Gallery](https://github.com/graphlit/graphlit-samples) - 60+ working examples **Understand the Concepts:** * [Semantic Memory](/platform/semantic-memory) - How memory works * [Key Concepts](/platform/key-concepts) - Data model overview * [Platform Overview](/getting-started/overview) - Platform architecture **Get Help:** * [Discord Community](https://discord.gg/ygFmfjy3Qx) - Community support * [GitHub Issues](https://github.com/graphlit/graphlit-client-python/issues) - Report bugs * [MCP Integration](/mcp-integration/mcp-integration) - IDE setup *** ## Technical Details ### How It Works Ask Graphlit uses: * **RAG (Retrieval Augmented Generation)** - Searches documentation and samples * **Code generation models** - Generates syntactically correct SDK code * **Schema validation** - Ensures generated code matches current API * **Best practices** - Follows production patterns from real applications ### Privacy & Security * āœ… **No credentials required** - Ask Graphlit doesn't access your data * āœ… **No data stored** - Conversations aren't persisted * āœ… **Open source MCP server** - Inspect the code on [GitHub](https://github.com/graphlit/graphlit-mcp-server) For accessing your actual data, use the [Graphlit MCP Server](/mcp-integration/mcp-integration) with your credentials. *** ## Frequently Asked Questions ### Can Ask Graphlit access my Graphlit data? No - Ask Graphlit only helps you write code. It doesn't have your credentials. To search your actual data, use the [Graphlit MCP Server](/mcp-integration/mcp-integration) locally with your credentials. ### Does it work offline? No - Ask Graphlit is a cloud service. However, you can browse the [Sample Gallery](https://github.com/graphlit/graphlit-samples) offline. ### Can I use both Ask Graphlit and Graphlit MCP Server? Yes! Use them together: * **Ask Graphlit**: "How do I code this?" * **Graphlit MCP Server**: "Search my actual data" Both can be configured in your IDE simultaneously. ### Is it free? Ask Graphlit is **unbilled**. * **Standalone** (`ask.graphlit.dev`): no Graphlit account or credentials required. * **Developer Portal**: requires a Portal login, but Ask Graphlit itself does not consume credits. ### What if the code doesn't work? 1. Copy the code and try it 2. Check the error message 3. Ask a follow-up question with the error 4. Join [Discord](https://discord.gg/ygFmfjy3Qx) for help Ask Graphlit generates code based on current documentation, but edge cases may need adjustment. *** ## Example Session {% hint style="info" %} **Language flexibility**: Examples below use TypeScript (consistent with docs). Ask Graphlit supports **Python, TypeScript, and .NET** - just specify your preference in your question. {% endhint %} **You:** > I want to ingest a PDF from Google Drive and extract people and organizations. Show me the code. **Ask Graphlit:** ```typescript import { Graphlit } from 'graphlit-client'; import { EntityTypes, FeedTypes, ObservableTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); async function main() { // Step 1: Create workflow with entity extraction const workflow = await graphlit.createWorkflow({ name: 'Extract Entities', extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization ] } }] } }); // Step 2: Create Google Drive feed const feed = await graphlit.createFeed({ name: 'Google Drive Documents', type: FeedTypes.Site, site: { type: FeedServiceTypes.GoogleDrive, // Note: Set up OAuth credentials first }, workflow: { id: workflow.createWorkflow.id } }); console.log('Feed created:', feed.createFeed.id); } main(); ``` **You:** > How do I wait for the feed to finish processing before querying entities? **Ask Graphlit:** ```typescript async function waitForFeedCompletion(feedId: string) { while (true) { const feedResult = await graphlit.getFeed(feedId); const state = feedResult.feed?.state; if (state === 'COMPLETED') { console.log('Feed processing complete!'); break; } else if (state === 'FAILED') { console.log('Feed processing failed'); break; } await new Promise(resolve => setTimeout(resolve, 10000)); // Wait 10s } } // Use it await waitForFeedCompletion(feed.createFeed.id); // Now query extracted entities const observables = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Person, ObservableTypes.Organization] } }); const people = observables.observables?.results?.filter( obs => obs?.type === ObservableTypes.Person ); const orgs = observables.observables?.results?.filter( obs => obs?.type === ObservableTypes.Organization ); console.log('People:', people?.map(p => p?.observable.name)); console.log('Organizations:', orgs?.map(o => o?.observable.name)); ``` **You:** > Show me the Python version of this **Ask Graphlit:** ```python import asyncio from graphlit import Graphlit from graphlit_api.input_types import * graphlit = Graphlit() async def wait_for_feed_completion(feed_id): while True: feed_result = await graphlit.client.get_feed(id=feed_id) state = feed_result.feed.state if state == EntityState.COMPLETED: print("Feed processing complete!") break elif state == EntityState.FAILED: print("Feed processing failed") break await asyncio.sleep(10) # Use it await wait_for_feed_completion(feed.create_feed.id) # Query entities observables = await graphlit.client.query_observables( filter=ObservableFilter( types=[ObservableTypes.PERSON, ObservableTypes.ORGANIZATION] ) ) for obs in observables.query_observables.results: print(f"{obs.type}: {obs.observable.name}") ``` *** ## Get Started [**šŸš€ Launch Ask Graphlit in the Portal ā†’**](https://portal.graphlit.dev) Or integrate it into your IDE: * [MCP Integration Guide](/mcp-integration/mcp-integration) * [Cursor Setup](/mcp-integration/mcp-integration#cursor) * [VS Code Setup](/mcp-integration/mcp-integration#vs-code-cline) *** **Can't find what you need? Just ask!** # Sign Up Sign up for a free Graphlit developer account in under 2 minutes. Get started with Graphlit by creating your free developer account at [portal.graphlit.dev](https://portal.graphlit.dev). {% hint style="success" %} **šŸ†“ Free to start:** 1GB data included, no credit card required. {% endhint %} ***
{% stepper %} {% step %} #### Create Your Account Click **Sign up** (below the Continue button) if this is your first time using Graphlit. **Choose your sign-in method:** * GitHub account * Google account * Microsoft 365 account * Email + password {% hint style="info" %} **Tip:** Using OAuth (GitHub/Google/Microsoft) is faster - no password to remember. {% endhint %} {% endstep %} {% step %} #### Create Your Organization Once signed in, you'll create your **Organization** (typically your company name, or personal workspace). **Organization setup:** 1. Enter organization name 2. Upload profile image (optional - use your company logo) 3. Click **Create Organization** {% hint style="info" %} You can invite team members to your organization later from the settings page. {% endhint %}
{% endstep %} {% step %} #### Access the Portal Your organization is created! You're now logged into the Graphlit Developer Portal.
{% hint style="success" %} **You're ready!** Next, create your first project to get API credentials. {% endhint %} {% endstep %} {% endstepper %} *** ## Next Step [**Create a project ā†’**](/account-setup-one-time/create-project) - Set up your first project (1 minute) {% hint style="info" %} Already signed up? Jump ahead to [Copy credentials and run `hello.ts`](/account-setup-one-time/credentials) to verify your environment. {% endhint %} *** ## Need Help? * **Ask Graphlit**: Open the chatbot from the Developer Portal sidebar (or visit [ask.graphlit.dev](https://ask.graphlit.dev)) for instant code examples. * **Sample Apps**: Explore the [Graphlit samples repo](https://github.com/graphlit/graphlit-samples) for working projects. * **Discord Community**: Join [discord.gg/ygFmfjy3Qx](https://discord.gg/ygFmfjy3Qx) to get answers from the Graphlit team. # Create Project Create your first Graphlit project and get your API credentials. Projects in Graphlit are isolated workspaces for your applications. Each project has its own data and storage. {% hint style="info" %} **Think of projects like databases:** Each application gets its own project. Building two apps? Create two projects. {% endhint %} ***
{% stepper %} {% step %} #### Create a New Project From your organization dashboard, click **New Project**. **Project setup:** 1. Enter a project name (e.g., "My AI Agent", "Customer Support AI") 2. Click **Create Project** {% hint style="success" %} **Free tier by default:** No payment method required to get started. You get 1GB of data included. {% endhint %}
{% endstep %} {% step %} #### View Project Overview Your project is created! You'll see the project **Overview** page with connection information.
{% hint style="info" %} **Deployment region:** Projects currently deploy to US South Central (Azure). Multi-region support coming soon. {% endhint %} {% hint style="info" %} **Tip:** Click "Projects" at the top-left to return to your projects list. {% endhint %}
{% endstep %} {% step %} #### Get Your API Credentials Each project includes two environments: **Preview** (development) and **Production**. **Environments are fully isolated:** * Separate data and storage * Independent credentials (Environment ID, JWT Secret) * No resource sharing **Billing:** Projects are billed together (not per environment). Payment methods are managed at the organization level. **Copy your credentials:** Under **Connection Information**, you'll find: * **Organization ID** (shared across all projects) * **Environment ID** (Preview or Production) * **JWT Secret** (authentication key)
Click the copy icon to copy each value. {% hint style="warning" %} **Keep your JWT Secret secure:** Treat it like a password. Never commit it to git or share publicly. {% endhint %} {% endstep %} {% endstepper %} *** ## Next Step [**Get your API credentials ā†’**](/account-setup-one-time/credentials) - Configure your development environment (1 minute) {% hint style="success" %} Once credentials are copied, run the [`hello.ts`](/account-setup-one-time/credentials#verify-your-setup) check to confirm everything works before the Quickstart tutorial. {% endhint %} *** ## Need Help? * **Ask Graphlit**: Open the chatbot from the Developer Portal sidebar (or visit [ask.graphlit.dev](https://ask.graphlit.dev)) for SDK code snippets. * **Sample Apps**: See [graphlit-samples](https://github.com/graphlit/graphlit-samples) for end-to-end examples. * **Discord Community**: Join [discord.gg/ygFmfjy3Qx](https://discord.gg/ygFmfjy3Qx) to chat with the team. # Get Your Credentials Configure your API credentials for development After creating your project, you need three values to authenticate with the Graphlit API. *** ## Copy Your Credentials In the [Developer Portal](https://portal.graphlit.dev), navigate to your project and go to **API Settings**.

API Settings showing connection information

### Required Credentials (All SDKs) Copy these three values from the Developer Portal:
CredentialDescription
Organization IDYour organization identifier (shared across projects)
Environment IDYour environment identifier (Preview or Production)
JWT SecretAuthentication secret for API access
{% hint style="warning" %} **Security**: Never commit your JWT Secret to version control. Treat it like a password. {% endhint %} *** ## Configure Your Environment ### Required (All SDKs) Create a `.env` file in your project root: ```env GRAPHLIT_ORGANIZATION_ID=your_org_id GRAPHLIT_ENVIRONMENT_ID=your_env_id GRAPHLIT_JWT_SECRET=your_secret ``` The SDK will automatically read these environment variables. {% hint style="info" %} **TypeScript**: Install `dotenv` to load environment variables: `npm install dotenv` {% endhint %} ### Optional: TypeScript Streaming Only needed if using `streamAgent()` for real-time responses: ```env # Use the API key for your chosen LLM provider OPENAI_API_KEY=your_openai_key # OR ANTHROPIC_API_KEY=your_anthropic_key # OR GOOGLE_API_KEY=your_google_key # (etc.) ``` **Supported LLM clients**: `streamAgent()` works with OpenAI, Anthropic, Google (Gemini), Groq, Mistral, Cohere, Cerebras, Deepseek, xAI, and more. **Our examples use OpenAI**, but you can use any supported provider. Get your OpenAI key from [platform.openai.com/api-keys](https://platform.openai.com/api-keys). {% hint style="info" %} **Python/C# developers**: You don't need this. Use `promptConversation()` (synchronous) for all conversations. Real-time streaming is a TypeScript SDK-specific feature. {% endhint %} *** ## Verify Your Setup Test that your credentials work: ```typescript import { Graphlit } from 'graphlit-client'; const graphlit = new Graphlit(); async function main() { const project = await graphlit.getProject(); console.log(`āœ… Connected: ${project.project.name}`); } main(); ``` Run it with `npx tsx hello.ts` (or your build tool of choice). {% hint style="success" %} **Next step**: [Quickstart: Your First Agent](/getting-started/quickstart) - Build your first AI agent in 7 minutes. {% endhint %} *** ## Next Step [**Start building ā†’**](/getting-started/quickstart) - Create your first streaming AI agent *** ## Need Help? * **Ask Graphlit**: Launch the chatbot in the Developer Portal (or visit [ask.graphlit.dev](https://ask.graphlit.dev)) for instant SDK answers. * **Sample Apps**: Browse [graphlit-samples](https://github.com/graphlit/graphlit-samples) for end-to-end projects. * **Discord Community**: Join [discord.gg/ygFmfjy3Qx](https://discord.gg/ygFmfjy3Qx) for real-time help. * **Report Issues**: Open tickets at [graphlit-client-typescript](https://github.com/graphlit/graphlit-client-typescript/issues). # Use Case Library **Comprehensive code examples for every Graphlit API operation** - organized for maximum discoverability. *** ## šŸš€ New to Graphlit? **Start with the basics first**: [Quickstart: Your First Agent](/getting-started/quickstart) **Can't find what you need?** Open **Ask Graphlit** in the Developer Portal (or visit [ask.graphlit.dev](https://ask.graphlit.dev)) - AI assistant that writes code for you *** ## šŸŽÆ I Want To... (Find by Task) ### Getting Content In * [**Ingest a PDF or Word doc**](/api-guides/use-cases/content/content-ingest-uri-basic) - From any URL * [**Ingest text or notes**](/api-guides/use-cases/content/content-ingest-text) - Plain text content * [**Connect Slack**](/api-guides/use-cases/feeds/messaging/feed-create-slack) - Auto-sync messages * [**Connect Gmail**](https://github.com/graphlit/graphlit-samples/tree/main/python/Notebook%20Examples/Data%20Connectors) - Auto-sync emails * [**Connect Google Drive**](/api-guides/use-cases/feeds/cloud-storage/feed-create-google-drive) - Auto-sync files * [**See all 30+ feeds ā†’**](/api-guides/use-cases/feeds) ### Searching & Finding * [**Basic semantic search**](/api-guides/use-cases/content/content-search-vector-explained) - Search by meaning * [**Search with filters**](/api-guides/use-cases/content/content-search-with-filters) - Filter by date, type, etc. * [**Hybrid search (best results)**](/api-guides/use-cases/content/content-search-hybrid-deep-dive) - Combines vector + keyword * [**Find similar content**](/api-guides/use-cases/content/content-query-similar) - "More like this" ### AI Conversations * [**Basic Q\&A chat**](/api-guides/use-cases/conversations/conversation-create-and-prompt) - Ask questions about content * [**Streaming responses**](/api-guides/use-cases/conversations/conversation-stream-agent-real-time-ui) - Real-time UI with events * [**Get source citations**](/api-guides/use-cases/conversations/conversation-prompt-with-citations) - Answers with page numbers * [**Multi-turn conversations**](/api-guides/use-cases/conversations/conversation-multi-turn-with-context) - Memory across turns * [**Tool calling**](/api-guides/use-cases/conversations/conversation-prompt-agent) - Let AI use tools ### Extracting Knowledge * [**Extract entities from PDFs**](/api-guides/use-cases/knowledge-graph/knowledge-graph-from-pdf-documents) - People, companies, places * [**Extract from emails**](/api-guides/use-cases/knowledge-graph/knowledge-graph-from-emails) - Contacts and organizations * [**Extract from meetings**](/api-guides/use-cases/knowledge-graph/knowledge-graph-from-meetings) - Attendees and topics * [**Query knowledge graph**](/api-guides/use-cases/knowledge-graph/observable-query-entities) - Find entities * [**Find relationships**](/api-guides/use-cases/knowledge-graph/observable-relationship-queries) - Who works where ### Advanced * [**Configure processing (Workflows)**](/api-guides/use-cases/workflows/workflow-create-extraction) - Customize extraction * [**Choose AI models (Specifications)**](/api-guides/use-cases/specifications/specification-create-custom-model) - Pick the right LLM * [**Multi-tenant setup**](/api-guides/use-cases/production/production-multi-tenant-isolation) - Separate customer data *** ## šŸ“š Browse by API Entity (Advanced) **For developers who understand the Graphlit data model:** ### [Content Foundations](/api-guides/use-cases/content) (27 guides) Understanding content types, metadata, search, and the tri-store architecture. **Start here**: [Content Type vs File Type](/api-guides/use-cases/content/content-type-vs-file-type-explained) | [Hybrid Search](/api-guides/use-cases/content/content-search-hybrid-deep-dive) *** ### [Knowledge Graph](/api-guides/use-cases/knowledge-graph) (24 guides) Extract entities, build knowledge graphs, and query relationships. **Popular**: [PDF Entity Extraction](/api-guides/use-cases/knowledge-graph/knowledge-graph-from-pdf-documents) | [Email Entities](/api-guides/use-cases/knowledge-graph/knowledge-graph-from-emails) | [Observable Model](/api-guides/use-cases/knowledge-graph/observable-observation-model-explained) *** ### [Data Source Feeds](/api-guides/use-cases/feeds) (31 guides) Connect 25+ data sources: Slack, Gmail, GitHub, Google Drive, Jira, and more. **By Type**: [Messaging](/api-guides/use-cases/feeds/messaging) | [Cloud Storage](/api-guides/use-cases/feeds/cloud-storage) | [Project Management](/api-guides/use-cases/feeds/project-management) | [Social Media](/api-guides/use-cases/feeds/social-media) *** ### [Conversations & RAG](/api-guides/use-cases/conversations) (10 guides) Build AI agents with streaming responses, multi-turn context, and entity-filtered RAG. **Popular**: [Stream Agent Real-Time UI](/api-guides/use-cases/conversations/conversation-stream-agent-real-time-ui) | [Prompt with Citations](/api-guides/use-cases/conversations/conversation-prompt-with-citations) *** ### [Workflows](/api-guides/use-cases/workflows) (6 guides) Configure content processing pipelines with preparation and extraction stages. **Start here**: [Entity Extraction Workflow](/api-guides/use-cases/knowledge-graph/workflow-configure-entity-extraction) | [Complex Multi-Stage](/api-guides/use-cases/workflows/workflow-complex-all-stages) *** ### [Specifications](/api-guides/use-cases/specifications) (6 guides) Configure LLM models, embeddings, and custom specifications. **Popular**: [Custom Models](/api-guides/use-cases/specifications/specification-create-custom-model) | [Embedding Configuration](/api-guides/use-cases/specifications/specification-create-embedding) *** ### [Collections](/api-guides/use-cases/collections) (4 guides) Organize content into collections for scoped queries and management. **Start here**: [Create and Manage Collections](/api-guides/use-cases/collections/collection-create-and-manage) *** ### [Alerts](/api-guides/use-cases/alerts) (2 guides) Scheduled content publishing operations. **Note**: Alerts are periodic publishing operations, not monitoring alerts. *** ### [Views](/api-guides/use-cases/views) (2 guides) Organize and display content in customizable layouts. **Start here**: [Create Semantic Search View](/api-guides/use-cases/views/view-create) *** ### [Production Patterns](/api-guides/use-cases/production) (5 guides) Cost optimization, monitoring, multi-tenant isolation, and production best practices. **Popular**: [Cost Optimization](/api-guides/use-cases/production/cost-optimization-model-selection) | [Multi-Tenant Isolation](/api-guides/use-cases/production/production-multi-tenant-isolation) *** ## Most Popular Use Cases 1. [Knowledge Graph from PDFs](/api-guides/use-cases/knowledge-graph/knowledge-graph-from-pdf-documents) - Complete PDF ā†’ entities pipeline 2. [Streaming Conversation with Citations](/api-guides/use-cases/conversations/conversation-stream-agent-real-time-ui) - Real-time RAG with sources 3. [Create Slack Feed](/api-guides/use-cases/feeds/messaging/feed-create-slack) - Sync Slack messages with entity extraction 4. [Hybrid Search Deep Dive](/api-guides/use-cases/content/content-search-hybrid-deep-dive) - Understanding RRF algorithm 5. [Entity Extraction Workflow](/api-guides/use-cases/knowledge-graph/workflow-configure-entity-extraction) - Configure entity extraction *** ## Documentation Philosophy Every use case includes: * **TypeScript canonical examples** with Python/C# adaptations * **Complete, working code** ready to copy-paste * **Configuration options** and variations * **Developer hints** and gotchas * **Common issues & solutions** * **Production patterns** from real applications *** ## Quick Links **Getting Started**: [Quickstart: Your First Agent](/getting-started/quickstart) | [Knowledge Graph Tutorial](/tutorials/knowledge-graph) **Platform Concepts**: [Key Concepts](/platform/key-concepts) | [Semantic Memory](/platform/semantic-memory) **Sample Apps**: [GitHub Samples Repository](https://github.com/graphlit/graphlit-samples) *** **Total**: 117 verified use case guides | Updated: January 2025 | **All code verified against TypeScript SDK** # Content Foundations Understanding Graphlit's content model, tri-store architecture, and search capabilities. *** ## Core Concepts **Start Here**: * [Content Type vs File Type](/api-guides/use-cases/content/content-type-vs-file-type-explained) - Understanding the hierarchy * [Content Metadata Structure](/api-guides/use-cases/content/content-metadata-structure) - Auto-captured metadata by type * [Content Lifecycle States](/api-guides/use-cases/content/content-lifecycle-states) - Created, Enabled, Disabled, Archived *** ## Search & Retrieval ### Search Types * [Vector Search Explained](/api-guides/use-cases/content/content-search-vector-explained) - Semantic similarity * [Keyword Search Explained](/api-guides/use-cases/content/content-search-keyword-explained) - Full-text + BM25 * [Hybrid Search Deep Dive](/api-guides/use-cases/content/content-search-hybrid-deep-dive) - RRF algorithm (recommended) ### Query Patterns * [Advanced Search with Filters](/api-guides/use-cases/content/content-search-with-filters) - Complex multi-criteria queries * [Metadata Filtering Strategies](/api-guides/use-cases/content/content-metadata-filtering-strategies) - Entity + date + type filters * [Query Performance Patterns](/api-guides/use-cases/content/content-query-performance-patterns) - Pagination, caching, optimization *** ## Type-Specific Metadata * [Email Metadata Queries](/api-guides/use-cases/content/content-email-metadata-queries) - From/to, labels, threads * [Message Metadata Queries](/api-guides/use-cases/content/content-message-metadata-queries) - Slack/Teams channels, mentions * [Document Metadata Queries](/api-guides/use-cases/content/content-document-metadata-queries) - Page counts, authors, encryption *** ## CRUD Operations ### Ingestion * [Ingest URI (Basic)](/api-guides/use-cases/content/content-ingest-uri-basic) - Simple URL ingestion * [Ingest URI with Workflow](/api-guides/use-cases/content/content-ingest-uri-with-workflow) - With entity extraction * [Ingest Text](/api-guides/use-cases/content/content-ingest-text) - Plain text/markdown * [Ingest Event](/api-guides/use-cases/content/content-ingest-event) - Calendar events * [Ingest Encoded File](/api-guides/use-cases/content/content-ingest-encoded-file) - Base64 uploads ### Querying * [Query with Filters](/api-guides/use-cases/content/content-query-with-filters) - Advanced filtering * [Query Similar Content](/api-guides/use-cases/content/content-query-similar) - Find similar documents * [Is Done Polling](/api-guides/use-cases/content/content-is-done-polling) - Check processing status ### Management * [Get Content Details](/api-guides/use-cases/content/content-get) - Full content retrieval * [Update Metadata](/api-guides/use-cases/content/content-update-metadata) - Modify properties * [Delete Content](/api-guides/use-cases/content/content-delete) - Single deletion * [Batch Delete](/api-guides/use-cases/content/content-batch-delete) - Bulk removal ### Publishing * [Publish Summary](/api-guides/use-cases/content/content-publish-summary) - Generate summaries * [Publish Audio](/api-guides/use-cases/content/content-publish-audio) - Text-to-speech *** ## Related Tutorials * [Quickstart: Your First Agent](/getting-started/quickstart) - Build a streaming agent in 7 minutes * [Context Engineering](/tutorials/context-engineering) - Advanced retrieval patterns *** **29 guides** | [ā† Back to Use Cases](/api-guides/use-cases) # Batch Delete Content ## User Intent "How do I delete multiple content items at once? Show me batch deletion." ## Operation **SDK Method**: `deleteAllContents()` with filters\ **Use Case**: Bulk content removal *** ## Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; const graphlit = new Graphlit(); // Delete all content from specific feed await graphlit.deleteAllContents({ filter: { feeds: [{ id: 'feed-id-to-delete' }] }, isSynchronous: true }); // Delete content older than date await graphlit.deleteAllContents({ filter: { creationDateRange: { to: '2023-12-31' } } }); console.log('Batch deletion complete'); ``` *** # Delete ## Content: Delete ### User Intent "I want to delete specific content or bulk delete multiple content items" ### Operation * **SDK Method**: `graphlit.deleteContent()`, `graphlit.deleteContents()`, `graphlit.deleteAllContents()` * **GraphQL**: `deleteContent`, `deleteContents`, `deleteAllContents` mutations * **Entity Type**: Content * **Common Use Cases**: Remove outdated content, cleanup test data, bulk deletion by filter ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentTypes, EntityState, SearchTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Delete single content by ID const response = await graphlit.deleteContent('content-id-here'); if (response.deleteContent?.id) { console.log(`Deleted content: ${response.deleteContent.id}`); } // Delete multiple specific content items (by IDs) const contentIds = ['id-1', 'id-2', 'id-3']; const bulkResponse = await graphlit.deleteContents( contentIds, true // isSynchronous - wait for deletion to complete ); console.log(`Deleted ${contentIds.length} content items`); // Delete ALL content (use with caution!) const deleteAllResponse = await graphlit.deleteAllContents( true // isSynchronous ); console.log('All content deleted'); ``` ## Delete single content (snake\_case method) response = await graphlit.deleteContent(id="content-id-here") ## Delete multiple content items content\_ids = \["id-1", "id-2", "id-3"] bulk\_response = await graphlit.deleteContents( ids=content\_ids, is\_synchronous=True ) ## Delete all content delete\_all\_response = await graphlit.deleteAllContents( is\_synchronous=True ) ```` **C#**: ```csharp using Graphlit; var client = new Graphlit(); // Delete single content (PascalCase method) var response = await graphlit.DeleteContent("content-id-here"); // Delete multiple content items var contentIds = new[] { "id-1", "id-2", "id-3" }; var bulkResponse = await graphlit.DeleteContents( contentIds, isSynchronous: true ); // Delete all content var deleteAllResponse = await graphlit.DeleteAllContents( isSynchronous: true ); ```` ### Parameters #### deleteContent * **`id`** (string): Content ID to delete #### deleteContents * **`ids`** (string\[]): Array of content IDs to delete * **`isSynchronous`** (boolean): Wait for deletion to complete * **Default**: `false` * **Recommended**: `true` for immediate confirmation #### deleteAllContents * **`filter`** (ContentFilter): Optional filter to delete subset of content * If omitted, deletes ALL content in project * **`isSynchronous`** (boolean): Wait for deletion to complete * **Default**: `false` * **Recommended**: `true` ### Response ```typescript // deleteContent { deleteContent: { id: string; // ID of deleted content state: EntityState; // DELETED } } // deleteContents { deleteContents: { ids: string[]; // Array of deleted content IDs } } // deleteAllContents { deleteAllContents: { count: number; // Number of content items deleted } } ``` ### Developer Hints #### Deletion is Permanent **There is NO undo** for delete operations: ```typescript // Once deleted, content cannot be recovered await graphlit.deleteContent(contentId); // Content is gone forever - no trash bin, no restore ``` **Best Practice**: Query first, confirm IDs, then delete: ```typescript // 1. Query to see what will be deleted const toDelete = await graphlit.queryContents({ types: [ContentTypes.Text], states: [EntityState.Error] }); console.log(`Will delete ${toDelete.contents.results.length} error content items`); // 2. Confirm with user (in production apps) // if (userConfirmed) { ... } // 3. Delete const ids = toDelete.contents.results.map(c => c.id); await graphlit.deleteContents(ids, true); ``` #### Synchronous vs Asynchronous Deletion ```typescript // Asynchronous (default) - returns immediately await graphlit.deleteContent(id); // Deletion happens in background // Synchronous - waits for deletion to complete await graphlit.deleteContent(id); // Only returns when deletion is confirmed ``` **For bulk operations**, asynchronous can be faster, but you won't know if/when deletion completes. #### šŸšØ deleteAllContents with Filters ```typescript // WITHOUT filter - deletes EVERYTHING (dangerous!) await graphlit.deleteAllContents(true); // WITH filter - deletes only matching content (safer) await graphlit.deleteAllContents( { types: [ContentTypes.Text], states: [EntityState.Error] }, true ); // Only deletes text content in error state ``` #### āš” Performance Considerations For large deletions (>100 items): * Use `deleteAllContents` with filter (faster than individual deletes) * Use `deleteContents` with batches (more control than deleteAll) ```typescript // Slow: Individual deletes for (const id of contentIds) { await graphlit.deleteContent(id); // 1 API call per item } // Fast: Bulk delete await graphlit.deleteContents(contentIds, true); // 1 API call total ``` ### Variations #### 1. Delete Content by Filter (Conditional Deletion) Delete content matching specific criteria: ```typescript // Find error content first const errorContent = await graphlit.queryContents({ states: [EntityState.Error], limit: 100 }); console.log(`Found ${errorContent.contents.results.length} error content items`); // Delete using filter (more efficient than individual deletes) await graphlit.deleteAllContents( { states: [EntityState.Error] }, true ); console.log('Deleted all error content'); ``` #### 2. Delete Old Content (Cleanup) Remove content older than a certain date: ```typescript // Define cutoff date (e.g., 1 year ago) const oneYearAgo = new Date(); oneYearAgo.setFullYear(oneYearAgo.getFullYear() - 1); // Query old content const oldContent = await graphlit.queryContents({ creationDateRange: { from: new Date('2000-01-01'), // Very old date to: oneYearAgo } }); console.log(`Found ${oldContent.contents.results.length} old content items`); // Delete old content const ids = oldContent.contents.results.map(c => c.id); if (ids.length > 0) { await graphlit.deleteContents(ids, true); console.log(`Deleted ${ids.length} old content items`); } ``` #### 3. Delete Content from Specific Feed Remove all content from a feed before deleting the feed: ```typescript // Get feed const feeds = await graphlit.queryFeeds(); const feedToDelete = feeds.feeds.results.find(f => f.name === 'Old Feed'); if (feedToDelete) { // Delete all content from feed await graphlit.deleteAllContents( { feeds: [{ id: feedToDelete.id }] }, true ); console.log(`Deleted all content from feed: ${feedToDelete.name}`); // Now safe to delete the feed itself await graphlit.deleteFeed(feedToDelete.id); } ``` #### 4. Batch Delete with Progress Tracking Delete large number of items with progress updates: ```typescript async function batchDeleteWithProgress(contentIds: string[], batchSize: number = 50) { const batches = []; for (let i = 0; i < contentIds.length; i += batchSize) { batches.push(contentIds.slice(i, i + batchSize)); } console.log(`Deleting ${contentIds.length} items in ${batches.length} batches...`); for (let i = 0; i < batches.length; i++) { await graphlit.deleteContents(batches[i], true); console.log(`Progress: ${((i + 1) / batches.length * 100).toFixed(0)}% (${(i + 1) * batchSize}/${contentIds.length})`); } console.log('Deletion complete'); } // Usage const allContent = await graphlit.queryContents({ limit: 1000 }); const ids = allContent.contents.results.map(c => c.id); await batchDeleteWithProgress(ids); ``` #### 5. Safe Delete with Confirmation Implement confirmation in production: ```typescript async function safeDeleteContent(contentId: string) { // Get content details const content = await graphlit.getContent(contentId); console.log('About to delete:'); console.log(` Name: ${content.content.name}`); console.log(` Type: ${content.content.type}`); console.log(` Created: ${content.content.creationDate}`); // In real app, prompt user for confirmation const confirmed = true; // Replace with actual user confirmation if (confirmed) { await graphlit.deleteContent(contentId); console.log('āœ“ Content deleted'); } else { console.log('āœ— Deletion cancelled'); } } ``` #### 6. Delete Test Data Clean up after testing: ```typescript // Delete all content with 'test' in the name const testContent = await graphlit.queryContents({ search: 'test', searchType: SearchTypes.Keyword }); const testIds = testContent.contents.results .filter(c => c.name.toLowerCase().includes('test')) .map(c => c.id); if (testIds.length > 0) { console.log(`Deleting ${testIds.length} test content items...`); await graphlit.deleteContents(testIds, true); console.log('Test data cleaned up'); } ``` #### 7. Delete by Collection Remove all content in a specific collection: ```typescript // Get collection const collections = await graphlit.queryCollections({ name: 'Temporary Files' }); const collectionId = collections.collections.results[0]?.id; if (collectionId) { // Delete all content in collection await graphlit.deleteAllContents( { collections: [{ id: collectionId }] }, true ); console.log('Deleted all content in Temporary Files collection'); // Optionally delete the collection itself await graphlit.deleteCollection(collectionId); } ``` ### Common Issues **Issue**: `Content not found` when deleting\ **Solution**: Content may have already been deleted or ID is incorrect. Check with `getContent()` first. **Issue**: Deletion returns success but content still appears in queries\ **Solution**: If using `isSynchronous: false`, deletion happens in background. Wait a few seconds or use synchronous mode. **Issue**: Cannot delete content with "Content is locked" error\ **Solution**: Content may be part of an active feed or workflow. Disable/delete the feed first. **Issue**: `deleteAllContents` deletes too much\ **Solution**: Always use a filter when calling `deleteAllContents`: ```typescript // Dangerous - deletes everything await graphlit.deleteAllContents(); // Safe - deletes only filtered content await graphlit.deleteAllContents({ states: [EntityState.Error] }); ``` **Issue**: Bulk delete times out\ **Solution**: Delete in smaller batches (50-100 items per batch) or use `isSynchronous: false`. ### Production Example **Single deletion**: ```typescript await graphlit.deleteContent(contentId); console.log(`Successfully deleted content ${contentId}`); ``` **Bulk deletion**: ```typescript await graphlit.deleteContents(contentIds, true); console.log(`Deleted ${contentIds.length} content items`); ``` **Filtered deletion with count**: ```typescript const response = await graphlit.deleteAllContents( { states: [EntityState.Error] }, true ); console.log(`Deleted ${response.deleteAllContents.count} error content items`); ``` # Document Metadata Queries ## Content: Document Metadata Queries ### User Intent "How do I query documents by pages, author, file type, etc.?" ### Operation * **SDK Method**: `queryContents()` with document-specific patterns * **GraphQL**: `queryContents` query * **Entity Type**: Content (type: FILE, fileType: DOCUMENT) * **Common Use Cases**: Find PDFs, filter by page count, search by author, encrypted documents ### Document Metadata Structure Documents (PDFs, Word, Excel, PowerPoint) have metadata in the `document` field: ```typescript interface DocumentMetadata { title: string; subject: string; summary: string; author: string; lastModifiedBy: string; publisher: string; description: string; keywords: string[]; pageCount: number; worksheetCount: number; // Excel slideCount: number; // PowerPoint wordCount: number; lineCount: number; paragraphCount: number; isEncrypted: boolean; hasDigitalSignature: boolean; } ``` ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentTypes, FileTypes, SearchTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Query all documents const allDocs = await graphlit.queryContents({ types: [ContentTypes.File], fileTypes: [FileTypes.Document] }); // Search documents const searchDocs = await graphlit.queryContents({ search: "quarterly report", types: [ContentTypes.File], fileTypes: [FileTypes.Document] }); // Recent documents const recentDocs = await graphlit.queryContents({ types: [ContentTypes.File], fileTypes: [FileTypes.Document], createdInLast: 'P30D' }); console.log(`Found ${recentDocs.contents.results.length} recent documents`); // Access document metadata recentDocs.contents.results.forEach(doc => { if (doc.document) { console.log(`${doc.name}: ${doc.document.pageCount} pages`); console.log(` Author: ${doc.document.author || 'Unknown'}`); console.log(` Words: ${doc.document.wordCount}`); } }); ``` ### Query Patterns #### 1. Filter by Document Type ```typescript // PDFs only const pdfs = await graphlit.queryContents({ fileExtensions: ['pdf'] }); // Word documents const wordDocs = await graphlit.queryContents({ fileExtensions: ['docx', 'doc'] }); // Excel spreadsheets const excel = await graphlit.queryContents({ fileExtensions: ['xlsx', 'xls'] }); // PowerPoint presentations const powerpoint = await graphlit.queryContents({ fileExtensions: ['pptx', 'ppt'] }); console.log(`PDFs: ${pdfs.contents.results.length}`); console.log(`Word: ${wordDocs.contents.results.length}`); console.log(`Excel: ${excel.contents.results.length}`); console.log(`PowerPoint: ${powerpoint.contents.results.length}`); ``` #### 2. Filter by Page Count ```typescript // Get all documents const docs = await graphlit.queryContents({ types: [ContentTypes.File], fileTypes: [FileTypes.Document] }); // Filter by page count const shortDocs = docs.contents.results.filter(doc => doc.document && doc.document.pageCount < 10 ); const mediumDocs = docs.contents.results.filter(doc => doc.document && doc.document.pageCount >= 10 && doc.document.pageCount < 50 ); const longDocs = docs.contents.results.filter(doc => doc.document && doc.document.pageCount >= 50 ); console.log(`Short (<10 pages): ${shortDocs.length}`); console.log(`Medium (10-50 pages): ${mediumDocs.length}`); console.log(`Long (50+ pages): ${longDocs.length}`); // Find longest documents const sorted = docs.contents.results .filter(doc => doc.document?.pageCount) .sort((a, b) => (b.document?.pageCount || 0) - (a.document?.pageCount || 0)); console.log('\nTop 5 longest documents:'); sorted.slice(0, 5).forEach(doc => { console.log(` ${doc.name}: ${doc.document?.pageCount} pages`); }); ``` #### 3. Filter by Author ```typescript // Search by author name const byAuthor = await graphlit.queryContents({ search: "Kirk Marple", searchType: SearchTypes.Keyword, types: [ContentTypes.File], fileTypes: [FileTypes.Document] }); // Get all docs and filter by author const docs = await graphlit.queryContents({ types: [ContentTypes.File], fileTypes: [FileTypes.Document] }); const kirkDocs = docs.contents.results.filter(doc => doc.document?.author?.toLowerCase().includes('kirk') ); console.log(`Documents by Kirk: ${kirkDocs.length}`); // Count documents by author const byAuthorCount = new Map(); docs.contents.results.forEach(doc => { const author = doc.document?.author || 'Unknown'; byAuthorCount.set(author, (byAuthorCount.get(author) || 0) + 1); }); console.log('\nTop 10 authors:'); Array.from(byAuthorCount.entries()) .sort((a, b) => b[1] - a[1]) .slice(0, 10) .forEach(([author, count]) => { console.log(` ${author}: ${count} documents`); }); ``` #### 4. Filter by File Size ```typescript // Large documents (> 10MB) const largeDocs = await graphlit.queryContents({ types: [ContentTypes.File], fileTypes: [FileTypes.Document], fileSizeRange: { from: 10000000 // 10MB in bytes } }); // Medium documents (1-10MB) const mediumDocs = await graphlit.queryContents({ types: [ContentTypes.File], fileTypes: [FileTypes.Document], fileSizeRange: { from: 1000000, // 1MB to: 10000000 // 10MB } }); // Small documents (< 1MB) const smallDocs = await graphlit.queryContents({ types: [ContentTypes.File], fileTypes: [FileTypes.Document], fileSizeRange: { to: 1000000 // 1MB } }); console.log(`Large (>10MB): ${largeDocs.contents.results.length}`); console.log(`Medium (1-10MB): ${mediumDocs.contents.results.length}`); console.log(`Small (<1MB): ${smallDocs.contents.results.length}`); ``` #### 5. Excel-Specific Queries ```typescript // Get Excel files const excel = await graphlit.queryContents({ fileExtensions: ['xlsx', 'xls'] }); // Filter by worksheet count const multiSheet = excel.contents.results.filter(doc => doc.document && doc.document.worksheetCount > 1 ); console.log(`Excel files: ${excel.contents.results.length}`); console.log(`Multi-sheet workbooks: ${multiSheet.length}`); // Find largest workbooks const sorted = excel.contents.results .filter(doc => doc.document?.worksheetCount) .sort((a, b) => (b.document?.worksheetCount || 0) - (a.document?.worksheetCount || 0)); console.log('\nLargest workbooks:'); sorted.slice(0, 5).forEach(doc => { console.log(` ${doc.name}: ${doc.document?.worksheetCount} worksheets`); }); ``` #### 6. PowerPoint-Specific Queries ```typescript // Get PowerPoint files const ppt = await graphlit.queryContents({ fileExtensions: ['pptx', 'ppt'] }); // Filter by slide count const shortDecks = ppt.contents.results.filter(doc => doc.document && doc.document.slideCount < 20 ); const longDecks = ppt.contents.results.filter(doc => doc.document && doc.document.slideCount >= 50 ); console.log(`PowerPoint files: ${ppt.contents.results.length}`); console.log(`Short decks (<20 slides): ${shortDecks.length}`); console.log(`Long decks (50+ slides): ${longDecks.length}`); // Average slide count const avgSlides = ppt.contents.results .filter(doc => doc.document?.slideCount) .reduce((sum, doc) => sum + (doc.document?.slideCount || 0), 0) / ppt.contents.results.filter(doc => doc.document?.slideCount).length; console.log(`Average slides: ${avgSlides.toFixed(1)}`); ``` #### 7. Encrypted Documents ```typescript // Get all documents const docs = await graphlit.queryContents({ types: [ContentTypes.File], fileTypes: [FileTypes.Document] }); // Filter encrypted const encrypted = docs.contents.results.filter(doc => doc.document?.isEncrypted === true ); // Filter digitally signed const signed = docs.contents.results.filter(doc => doc.document?.hasDigitalSignature === true ); console.log(`Encrypted documents: ${encrypted.length}`); console.log(`Digitally signed: ${signed.length}`); // List encrypted docs if (encrypted.length > 0) { console.log('\nEncrypted documents:'); encrypted.forEach(doc => { console.log(` ${doc.name}`); }); } ``` #### 8. Content Analysis ```typescript // Get documents const docs = await graphlit.queryContents({ types: [ContentTypes.File], fileTypes: [FileTypes.Document] }); // Word count statistics const wordCounts = docs.contents.results .filter(doc => doc.document?.wordCount) .map(doc => doc.document?.wordCount || 0); const avgWords = wordCounts.reduce((a, b) => a + b, 0) / wordCounts.length; const maxWords = Math.max(...wordCounts); const minWords = Math.min(...wordCounts); console.log('Word count statistics:'); console.log(` Average: ${avgWords.toFixed(0)} words`); console.log(` Max: ${maxWords} words`); console.log(` Min: ${minWords} words`); // Find most content-rich documents const sorted = docs.contents.results .filter(doc => doc.document?.wordCount) .sort((a, b) => (b.document?.wordCount || 0) - (a.document?.wordCount || 0)); console.log('\nMost content-rich documents:'); sorted.slice(0, 5).forEach(doc => { console.log(` ${doc.name}: ${doc.document?.wordCount} words`); }); ``` ## Query documents docs = await graphlit.queryContents( filter=ContentFilterInput( types=\[ContentTypes.File], file\_types=\[FileTypes.Document] ) ) ## PDFs only pdfs = await graphlit.queryContents( filter=ContentFilterInput( file\_extensions=\['pdf'] ) ) ## Access metadata for doc in docs.contents.results: if doc.document: print(f"{doc.name}: {doc.document.page\_count} pages") print(f" Author: {doc.document.author}") print(f" Words: {doc.document.word\_count}") ```` **C#**: ```csharp using Graphlit; var client = new Graphlit(); // Query documents var docs = await graphlit.QueryContents(new ContentFilter { Filter = new ContentCriteria { Types = new[] { ContentTypes.File }, FileTypes = new[] { FileDocument } } }); // PDFs only var pdfs = await graphlit.QueryContents(new ContentFilter { Filter = new ContentCriteria { FileExtensions = new[] { "pdf" } } }); // Access metadata foreach (var doc in docs.Contents.Results) { if (doc.Document != null) { Console.WriteLine($"{doc.Name}: {doc.Document.PageCount} pages"); Console.WriteLine($" Author: {doc.Document.Author}"); Console.WriteLine($" Words: {doc.Document.WordCount}"); } } ```` ### Developer Hints #### Page Count is Automatic ```typescript // Page count automatically detected for: // - PDF files // - Word documents // - PowerPoint (slideCount instead) // - Excel (worksheetCount instead) if (doc.document?.pageCount) { console.log(`${doc.document.pageCount} pages`); } ``` #### Excel vs Word vs PowerPoint ```typescript // Excel: worksheetCount if (doc.document?.worksheetCount) { console.log(`Excel: ${doc.document.worksheetCount} worksheets`); } // PowerPoint: slideCount if (doc.document?.slideCount) { console.log(`PowerPoint: ${doc.document.slideCount} slides`); } // Word/PDF: pageCount if (doc.document?.pageCount) { console.log(`Document: ${doc.document.pageCount} pages`); } ``` #### Author from Document Properties ```typescript // Author comes from document properties // Set in Word/PDF metadata // May be null if not set const author = doc.document?.author || 'Unknown'; ``` ### Common Issues & Solutions **Issue**: Need to filter by exact page count **Solution**: Query all, filter client-side ```typescript const docs = await graphlit.queryContents({ types: [ContentTypes.File], fileTypes: [FileTypes.Document] }); const exactly10Pages = docs.contents.results.filter( d => d.document?.pageCount === 10 ); ``` **Issue**: Want PDFs only **Solution**: Use fileExtensions filter ```typescript const pdfs = await graphlit.queryContents({ fileExtensions: ['pdf'] }); ``` **Issue**: Need to count documents by file extension **Solution**: Query and aggregate ```typescript const docs = await graphlit.queryContents({ types: [ContentTypes.File], fileTypes: [FileTypes.Document] }); const byExtension = new Map(); docs.contents.results.forEach(doc => { const ext = doc.fileExtension || 'unknown'; byExtension.set(ext, (byExtension.get(ext) || 0) + 1); }); ``` ### Production Example ```typescript async function analyzeDocumentLibrary() { console.log('\n=== DOCUMENT LIBRARY ANALYSIS ===\n'); // Get all documents const docs = await graphlit.queryContents({ types: [ContentTypes.File], fileTypes: [FileTypes.Document] limit: 1000 }); console.log(`Total documents: ${docs.contents.results.length}`); // By file type const byType = new Map(); docs.contents.results.forEach(doc => { const ext = doc.fileExtension || 'unknown'; byType.set(ext, (byType.get(ext) || 0) + 1); }); console.log('\nDocument types:'); Array.from(byType.entries()) .sort((a, b) => b[1] - a[1]) .forEach(([ext, count]) => { console.log(` .${ext}: ${count}`); }); // Page statistics const withPages = docs.contents.results.filter(d => d.document?.pageCount); const totalPages = withPages.reduce((sum, d) => sum + (d.document?.pageCount || 0), 0); const avgPages = totalPages / withPages.length; console.log(`\nPage statistics:`); console.log(` Documents with pages: ${withPages.length}`); console.log(` Total pages: ${totalPages.toLocaleString()}`); console.log(` Average pages: ${avgPages.toFixed(1)}`); // Size statistics const totalSize = docs.contents.results.reduce((sum, d) => sum + (d.fileSize || 0), 0); const avgSize = totalSize / docs.contents.results.length; console.log(`\nSize statistics:`); console.log(` Total size: ${(totalSize / 1024 / 1024).toFixed(2)} MB`); console.log(` Average size: ${(avgSize / 1024 / 1024).toFixed(2)} MB`); // Security const encrypted = docs.contents.results.filter(d => d.document?.isEncrypted); const signed = docs.contents.results.filter(d => d.document?.hasDigitalSignature); console.log(`\nSecurity:`); console.log(` Encrypted: ${encrypted.length}`); console.log(` Digitally signed: ${signed.length}`); // Top authors const authors = new Map(); docs.contents.results.forEach(doc => { const author = doc.document?.author || 'Unknown'; authors.set(author, (authors.get(author) || 0) + 1); }); console.log(`\nTop 10 authors:`); Array.from(authors.entries()) .sort((a, b) => b[1] - a[1]) .slice(0, 10) .forEach(([author, count]) => { console.log(` ${author}: ${count} documents`); }); // Recent activity const last30Days = docs.contents.results.filter(doc => { const age = Date.now() - new Date(doc.creationDate).getTime(); return age < 30 * 24 * 60 * 60 * 1000; }); console.log(`\nRecent activity (last 30 days): ${last30Days.length} documents`); } await analyzeDocumentLibrary(); ``` # Email Metadata Queries ## Content: Email Metadata Queries ### User Intent "How do I query emails by sender, subject, labels, etc.?" ### Operation * **SDK Method**: `queryContents()` with email-specific patterns * **GraphQL**: `queryContents` query * **Entity Type**: Content (type: EMAIL) * **Common Use Cases**: Find emails by sender, filter by labels, search by subject, date range queries ### Email Metadata Structure Emails have rich metadata captured in the `email` field: ```typescript interface EmailMetadata { identifier: string; // Message-ID threadIdentifier: string; // Thread ID subject: string; from: PersonReference[]; to: PersonReference[]; cc: PersonReference[]; bcc: PersonReference[]; labels: string[]; // Gmail labels sensitivity: MailSensitivity; priority: MailPriority; importance: MailImportance; attachmentCount: number; unsubscribeUrl: string; publicationName: string; publicationUrl: string; } ``` ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentTypes, SearchTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Query all emails const allEmails = await graphlit.queryContents({ types: [ContentTypes.Email] }); // Search emails by keyword const searchEmails = await graphlit.queryContents({ search: "quarterly report", types: [ContentTypes.Email] }); // Recent emails const recentEmails = await graphlit.queryContents({ types: [ContentTypes.Email], createdInLast: 'P7D' // Last 7 days }); // Emails from specific sender (keyword search) const fromSender = await graphlit.queryContents({ search: "kirk@graphlit.com", searchType: SearchTypes.Keyword, types: [ContentTypes.Email] }); // Emails with attachments const withAttachments = await graphlit.queryContents({ types: [ContentTypes.Email] }); // Filter client-side for attachments const hasAttachments = withAttachments.contents.results.filter( email => email.email && email.email.attachmentCount > 0 ); console.log(`Found ${hasAttachments.length} emails with attachments`); ``` ### Query Patterns #### 1. Find by Sender ```typescript // Search by sender email const fromKirk = await graphlit.queryContents({ search: "kirk@graphlit.com", searchType: SearchTypes.Keyword, types: [ContentTypes.Email] }); // Search by sender name const fromName = await graphlit.queryContents({ search: "Kirk Marple", searchType: SearchTypes.Keyword, types: [ContentTypes.Email] }); // Access sender details fromKirk.contents.results.forEach(email => { if (email.email && email.email.from) { console.log(`From: ${email.email.from[0].name} <${email.email.from[0].email}>`); console.log(`Subject: ${email.email.subject}`); } }); ``` #### 2. Find by Subject ```typescript // Search in subject const subjectSearch = await graphlit.queryContents({ search: "quarterly earnings report", types: [ContentTypes.Email] }); // Exact subject match const exactSubject = await graphlit.queryContents({ search: '"Q4 2024 Earnings"', searchType: SearchTypes.Keyword, types: [ContentTypes.Email] }); ``` #### 3. Find by Recipient ```typescript // Search by recipient const toMaria = await graphlit.queryContents({ search: "maria@example.com", searchType: SearchTypes.Keyword, types: [ContentTypes.Email] }); // Check if specific person is in TO/CC/BCC const allEmails = await graphlit.queryContents({ types: [ContentTypes.Email] }); const toRecipient = allEmails.contents.results.filter(email => { if (!email.email) return false; const allRecipients = [ ...(email.email.to || []), ...(email.email.cc || []), ...(email.email.bcc || []) ]; return allRecipients.some(r => r.email === "maria@example.com"); }); console.log(`Found ${toRecipient.length} emails to maria@example.com`); ``` #### 4. Filter by Labels (Gmail) ```typescript // Get emails with specific label const emails = await graphlit.queryContents({ types: [ContentTypes.Email] }); // Filter by Gmail label const important = emails.contents.results.filter(email => email.email?.labels?.includes('IMPORTANT') ); const unread = emails.contents.results.filter(email => email.email?.labels?.includes('UNREAD') ); const starred = emails.contents.results.filter(email => email.email?.labels?.includes('STARRED') ); console.log(`Important: ${important.length}`); console.log(`Unread: ${unread.length}`); console.log(`Starred: ${starred.length}`); ``` #### 5. Date Range Queries ```typescript // Emails from last month const lastMonth = await graphlit.queryContents({ types: [ContentTypes.Email], createdInLast: 'P30D' }); // Emails in specific date range const dateRange = await graphlit.queryContents({ types: [ContentTypes.Email], creationDateRange: { from: '2024-01-01T00:00:00Z', to: '2024-12-31T23:59:59Z' } }); // Q4 2024 emails const q4 = await graphlit.queryContents({ types: [ContentTypes.Email], creationDateRange: { from: '2024-10-01', to: '2024-12-31' } }); ``` #### 6. Thread Queries ```typescript // Get all emails const emails = await graphlit.queryContents({ types: [ContentTypes.Email] limit: 100 }); // Group by thread const threads = new Map(); emails.contents.results.forEach(email => { if (email.email?.threadIdentifier) { const threadId = email.email.threadIdentifier; if (!threads.has(threadId)) { threads.set(threadId, []); } threads.get(threadId)?.push(email); } }); console.log(`Found ${threads.size} email threads`); // Find longest threads const sortedThreads = Array.from(threads.entries()) .sort((a, b) => b[1].length - a[1].length); console.log('\nTop 5 longest threads:'); sortedThreads.slice(0, 5).forEach(([threadId, emails]) => { console.log(`Thread ${threadId}: ${emails.length} emails`); console.log(` Subject: ${emails[0].email?.subject}`); }); ``` #### 7. With Attachments ```typescript // Query all emails const emails = await graphlit.queryContents({ types: [ContentTypes.Email] }); // Filter by attachment count const withAttachments = emails.contents.results.filter(email => email.email && email.email.attachmentCount > 0 ); const multipleAttachments = emails.contents.results.filter(email => email.email && email.email.attachmentCount > 1 ); console.log(`With attachments: ${withAttachments.length}`); console.log(`Multiple attachments: ${multipleAttachments.length}`); // List emails with most attachments const sorted = emails.contents.results .filter(email => email.email && email.email.attachmentCount > 0) .sort((a, b) => (b.email?.attachmentCount || 0) - (a.email?.attachmentCount || 0)); console.log('\nTop 5 emails by attachment count:'); sorted.slice(0, 5).forEach(email => { console.log(`${email.name}: ${email.email?.attachmentCount} attachments`); }); ``` #### 8. Newsletter Detection ```typescript // Find newsletters const emails = await graphlit.queryContents({ types: [ContentTypes.Email] }); const newsletters = emails.contents.results.filter(email => email.email?.publicationName || email.email?.unsubscribeUrl ); console.log(`Found ${newsletters.length} newsletters`); // Group by publication const byPublication = new Map(); newsletters.forEach(email => { const pub = email.email?.publicationName || 'Unknown'; byPublication.set(pub, (byPublication.get(pub) || 0) + 1); }); console.log('\nNewsletters by publication:'); Array.from(byPublication.entries()) .sort((a, b) => b[1] - a[1]) .forEach(([pub, count]) => { console.log(` ${pub}: ${count}`); }); ``` ### Sample Reference * `Graphlit_2024_09_07_Locate_Google_Emails_by_Person.ipynb` * `Graphlit_2024_12_09_Locate_Microsoft_Emails_by_Organization.ipynb` ## Query emails emails = await graphlit.queryContents( filter=ContentFilterInput( types=\[ContentTypes.Email] ) ) ## By sender from\_kirk = await graphlit.queryContents( search="", search\_type=SearchTypes.Keyword, filter=ContentFilterInput( types=\[ContentTypes.Email] ) ) ## Recent emails recent = await graphlit.queryContents( filter=ContentFilterInput( types=\[ContentTypes.Email], created\_in\_last='P7D' ) ) ## Access metadata for email in emails.contents.results: if email.email: print(f"From: {email.email.from\_\[0].email}") print(f"Subject: {email.email.subject}") ```` **C#**: ```csharp using Graphlit; var client = new Graphlit(); // Query emails var emails = await graphlit.QueryContents(new ContentFilter { Types = new[] { ContentTypes.Email } }); // By sender var fromKirk = await graphlit.QueryContents(new ContentFilter { Search = "kirk@graphlit.com", SearchType = SearchKeyword, Filter = new ContentCriteria { Types = new[] { ContentTypes.Email } } }); // Recent emails var recent = await graphlit.QueryContents(new ContentFilter { Filter = new ContentCriteria { Types = new[] { ContentTypes.Email }, CreatedInLast = "P7D" } }); // Access metadata foreach (var email in emails.Contents.Results) { if (email.Email != null) { Console.WriteLine($"From: {email.Email.From[0].Email}"); Console.WriteLine($"Subject: {email.Email.Subject}"); } } ```` ### Developer Hints #### Email Addresses are Searchable ```typescript // Keyword search works for email addresses const results = await graphlit.queryContents({ search: "kirk@graphlit.com", searchType: SearchTypes.Keyword, types: [ContentTypes.Email] }); ``` #### Labels are Gmail-Specific ```typescript // Gmail uses labels email.email?.labels // ["INBOX", "IMPORTANT", "UNREAD"] // Outlook uses folders (not in labels field) // Check feed type to determine source ``` #### Thread Detection ```typescript // Emails with same threadIdentifier are part of same conversation if (email.email?.threadIdentifier) { console.log(`Part of thread: ${email.email.threadIdentifier}`); } ``` ### Common Issues & Solutions **Issue**: Can't filter by specific label **Solution**: Query all emails, filter client-side ```typescript const all = await graphlit.queryContents({ types: [ContentTypes.Email] }); const important = all.contents.results.filter( e => e.email?.labels?.includes('IMPORTANT') ); ``` **Issue**: Want emails TO a specific person **Solution**: Search by email or filter client-side ```typescript // Keyword search await graphlit.queryContents({ search: "maria@example.com", types: [ContentTypes.Email] }); // Or filter after query const all = await graphlit.queryContents({ types: [ContentTypes.Email] }); const toMaria = all.contents.results.filter(e => e.email?.to?.some(r => r.email === "maria@example.com") ); ``` **Issue**: Need to count emails by sender **Solution**: Query and aggregate client-side ```typescript const emails = await graphlit.queryContents({ types: [ContentTypes.Email] }); const bySender = new Map(); emails.contents.results.forEach(email => { const sender = email.email?.from[0]?.email || 'unknown'; bySender.set(sender, (bySender.get(sender) || 0) + 1); }); ``` ### Production Example ```typescript async function analyzeEmails() { console.log('\n=== EMAIL ANALYSIS ===\n'); // Get all emails const emails = await graphlit.queryContents({ types: [ContentTypes.Email], createdInLast: 'P30D' // Last 30 days limit: 500 }); console.log(`Total emails: ${emails.contents.results.length}`); // Analyze senders const senders = new Map(); emails.contents.results.forEach(email => { if (email.email?.from[0]) { const sender = email.email.from[0].email; senders.set(sender, (senders.get(sender) || 0) + 1); } }); console.log(`\nTop 10 senders:`); Array.from(senders.entries()) .sort((a, b) => b[1] - a[1]) .slice(0, 10) .forEach(([sender, count]) => { console.log(` ${sender}: ${count} emails`); }); // Attachment statistics const withAttachments = emails.contents.results.filter( e => e.email && e.email.attachmentCount > 0 ); console.log(`\nEmails with attachments: ${withAttachments.length} (${(withAttachments.length / emails.contents.results.length * 100).toFixed(1)}%)`); // Thread analysis const threads = new Map(); emails.contents.results.forEach(email => { if (email.email?.threadIdentifier) { const tid = email.email.threadIdentifier; if (!threads.has(tid)) threads.set(tid, []); threads.get(tid)?.push(email); } }); console.log(`\nThreads: ${threads.size}`); const threadLengths = Array.from(threads.values()).map(t => t.length); const avgThreadLength = threadLengths.reduce((a, b) => a + b, 0) / threadLengths.length; console.log(`Average thread length: ${avgThreadLength.toFixed(1)} emails`); // Labels (Gmail) const allLabels = new Set(); emails.contents.results.forEach(email => { email.email?.labels?.forEach(label => allLabels.add(label)); }); console.log(`\nUnique labels: ${allLabels.size}`); console.log(`Labels: ${Array.from(allLabels).join(', ')}`); // Newsletters const newsletters = emails.contents.results.filter( e => e.email?.publicationName ); console.log(`\nNewsletters: ${newsletters.length}`); } await analyzeEmails(); ``` # Get Complete Content Details # Get Content Details ## User Intent "I want to retrieve full details for a specific content item" ## Operation * **SDK Method**: `graphlit.getContent()` * **GraphQL**: `getContent` query * **Entity Type**: Content * **Common Use Cases**: View content details, access extracted markdown, check metadata ## TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentTypes, EntityState, FileTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Content IDs are GUIDs (e.g., '550e8400-e29b-41d4-a716-446655440000') const contentId = 'content-id-here'; // Get full content details const content = await graphlit.getContent(contentId); console.log(`\nContent: ${content.content.name}`); console.log(`Type: ${content.content.type}`); console.log(`State: ${content.content.state}`); console.log(`Created: ${content.content.creationDate}`); // Access extracted text if (content.content.markdown) { console.log(`\nExtracted text (first 500 chars):`); console.log(content.content.markdown.substring(0, 500)); } // Check file details if (content.content.fileType) { console.log(`\nFile Type: ${content.content.fileType}`); console.log(`File Size: ${content.content.fileSize} bytes`); } // Access URI if (content.content.uri) { console.log(`URI: ${content.content.uri}`); } ``` ## Parameters * **`id`** (string): Content ID ## Response ```typescript { content: { id: string; name: string; type: ContentTypes; state: EntityState; creationDate: Date; markdown?: string; summary?: string; uri?: string; fileType?: FileTypes; fileSize?: number; // Many more fields... } } ``` ## Developer Hints ### Access Extracted Content ```typescript const content = await graphlit.getContent(contentId); // Full extracted markdown console.log(content.content.markdown); // Auto-generated summary console.log(content.content.summary); // Custom summary (if set) console.log(content.content.customSummary); ``` ### Check Processing State ```typescript const content = await graphlit.getContent(contentId); if (content.content.state === EntityState.Finished) { console.log(' Processing complete'); } else { console.log('ā³ Still processing'); } ``` ## Variations ### 1. Basic Content Retrieval ```typescript const content = await graphlit.getContent(contentId); console.log(content.content.name); ``` ### 2. Get Extracted Text ```typescript const content = await graphlit.getContent(contentId); const markdown = content.content.markdown; console.log(markdown); ``` ### 3. Check File Details ```typescript const content = await graphlit.getContent(contentId); console.log(`File: ${content.content.fileType}`); console.log(`Size: ${content.content.fileSize} bytes`); ``` # Ingest Encoded File ## User Intent "I want to upload a file directly from memory/buffer without using a URL" ## Operation * **SDK Method**: `graphlit.ingestEncodedFile()` * **GraphQL**: `ingestEncodedFile` mutation * **Entity Type**: Content * **Common Use Cases**: File uploads from web forms, email attachments, programmatically generated files, binary data ## TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentState, FileTypes } from 'graphlit-client/dist/generated/graphql-types'; import { readFileSync } from 'fs'; const graphlit = new Graphlit(); // Read file from disk const fileBuffer = readFileSync('/path/to/document.pdf'); const base64Data = fileBuffer.toString('base64'); // Ingest encoded file const response = await graphlit.ingestEncodedFile( 'document.pdf', base64Data, 'application/pdf', undefined, undefined, undefined, undefined, true, { id: workflowId }, [{ id: collectionId }], undefined, 'upload-demo' ); const contentId = response.ingestEncodedFile.id; console.log(`File ingested: ${contentId}`); // Retrieve the content const content = await graphlit.getContent(contentId); console.log(`File type: ${content.content.fileType}`); console.log(`Markdown extracted: ${content.content.markdown?.substring(0, 100)}...`); ``` ## Parameters ### Required * **`name`** (string): Filename (including extension) * Used to determine file type * Should include proper extension (.pdf, .docx, .jpg, etc.) * **`data`** (string): Base64-encoded file data * Binary file content encoded as base64 string * No size limit in API, but consider network constraints * **`mimeType`** (string): MIME type of the file * Examples: `application/pdf`, `image/jpeg`, `text/plain`, `application/vnd.openxmlformats-officedocument.wordprocessingml.document` (DOCX) * Must match the actual file type ### Optional * **`fileCreationDate`** (DateTime): Original file creation date * **`fileModifiedDate`** (DateTime): Original file modification date * **`id`** (string): Custom ID for the content * **`identifier`** (string): Custom identifier for deduplication * **`isSynchronous`** (boolean): Wait for processing to complete * **Default**: `false` * **Recommended**: `true` for immediate access to extracted content * **`workflow`** (EntityReferenceInput): Workflow for extraction/preparation * **`collections`** (EntityReferenceInput\[]): Collections to add content to * **`observations`** (ObservationReferenceInput\[]): Observations to link * **`correlationId`** (string): For tracking in production systems ## Response ```typescript { ingestEncodedFile: { id: string; // Content ID name: string; // Filename you provided state: ContentState; // FINISHED (if synchronous) type: ContentFILE; // Always FILE fileType: FileTypes; // PDF, DOCX, IMAGE, AUDIO, VIDEO, etc. mimeType: string; // MIME type you provided markdown?: string; // Extracted text (for documents) originalData?: string; // Base64 data (if stored) } } ``` ## Developer Hints ### ingestEncodedFile vs ingestUri | Aspect | ingestEncodedFile | ingestUri | | -------------- | ------------------------------- | ------------------------------ | | **Source** | File in memory/buffer | URL or file path | | **Encoding** | Requires base64 encoding | No encoding needed | | **Use Case** | File uploads, email attachments | Web scraping, public URLs | | **Network** | Uploads file data to Graphlit | Graphlit downloads from URL | | **Size Limit** | Network/timeout constraints | More efficient for large files | ### When to Use ingestEncodedFile Use `ingestEncodedFile` when: * Handling file uploads from users (web forms, mobile apps) * Processing email attachments * Working with programmatically generated files * Files are in memory/buffer * No public URL available Use `ingestUri` when: * File is at a public URL * File is very large (>100MB) * Want Graphlit to handle download ### Base64 Encoding Guide ```typescript // Node.js (filesystem) import { readFileSync } from 'fs'; const buffer = readFileSync('file.pdf'); const base64 = buffer.toString('base64'); // Browser (File input) const file = input.files?.[0]; const base64 = await new Promise((resolve) => { const reader = new FileReader(); reader.onload = () => resolve((reader.result as string).split(',')[1]); reader.readAsDataURL(file); }); ``` ### MIME Type Reference Common MIME types: * **PDF**: `application/pdf` * **Word**: `application/vnd.openxmlformats-officedocument.wordprocessingml.document` * **Excel**: `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet` * **PowerPoint**: `application/vnd.openxmlformats-officedocument.presentationml.presentation` * **JPEG**: `image/jpeg` * **PNG**: `image/png` * **MP3**: `audio/mpeg` * **MP4**: `video/mp4` * **Plain Text**: `text/plain` ## Variations ### 1. Browser File Upload Handle file uploads in web applications: ```typescript // React/Next.js component async function handleFileUpload(event: React.ChangeEvent) { const file = event.target.files?.[0]; if (!file) return; // Convert to base64 const base64 = await new Promise((resolve) => { const reader = new FileReader(); reader.onload = () => { const result = reader.result as string; // Remove data URL prefix (data:mime;base64,) const base64Data = result.split(',')[1]; resolve(base64Data); }; reader.readAsDataURL(file); }); // Ingest file const response = await graphlit.ingestEncodedFile( file.name, base64, file.type, undefined, undefined, true ); console.log(`File uploaded: ${response.ingestEncodedFile.id}`); } ``` ### 2. Email Attachment Processing Ingest email attachments: ```typescript // Process email with attachments interface EmailAttachment { filename: string; mimeType: string; data: Buffer; } async function processEmailAttachments(attachments: EmailAttachment[]) { const contentIds: string[] = []; for (const attachment of attachments) { const base64Data = attachment.data.toString('base64'); const response = await graphlit.ingestEncodedFile( attachment.filename, attachment.mimeType, base64Data, undefined, undefined, false // Async for bulk processing ); contentIds.push(response.ingestEncodedFile.id); } return contentIds; } ``` ### 3. Ingesting with Workflow Apply extraction during upload: ```typescript // Create workflow for document extraction const workflowInput: WorkflowInput = { name: 'Document Extraction', preparation: { jobs: [ { connector: { type: FilePreparationServiceTypes.ModelDocument, modelDocument: { includeImages: true // Better extraction for scanned PDFs }, fileTypes: [FileTypes.Document] } } ] } }; const workflowResponse = await graphlit.createWorkflow(workflowInput); // Read and encode file const fileBuffer = fs.readFileSync('contract.pdf'); const base64Data = fileBuffer.toString('base64'); // Ingest with workflow const response = await graphlit.ingestEncodedFile( 'contract.pdf', 'application/pdf', base64Data, { id: workflowResponse.createWorkflow.id }, // Apply workflow undefined, true // Wait for extraction to complete ); // Access extracted content const content = await graphlit.getContent(response.ingestEncodedFile.id); console.log(`Extracted text: ${content.content.markdown}`); ``` ### 4. Batch File Upload Upload multiple files efficiently: ```typescript async function batchUploadFiles(filePaths: string[]) { const uploadPromises = filePaths.map(async (filePath) => { const fileBuffer = fs.readFileSync(filePath); const base64Data = fileBuffer.toString('base64'); const fileName = filePath.split('/').pop() || 'unknown'; // Detect MIME type (simplified) const ext = fileName.split('.').pop()?.toLowerCase(); const mimeType = getMimeType(ext || ''); return graphlit.ingestEncodedFile( fileName, mimeType, base64Data, undefined, undefined, false // Async for parallel uploads ); }); const responses = await Promise.all(uploadPromises); return responses.map(r => r.ingestEncodedFile.id); } function getMimeType(extension: string): string { const mimeTypes: Record = { 'pdf': 'application/pdf', 'docx': 'application/vnd.openxmlformats-officedocument.wordprocessingml.document', 'jpg': 'image/jpeg', 'jpeg': 'image/jpeg', 'png': 'image/png', 'txt': 'text/plain' }; return mimeTypes[extension] || 'application/octet-stream'; } ``` ### 5. Ingesting Programmatically Generated Files Upload files created in code: ```typescript // Generate a report and ingest it import PDFDocument from 'pdfkit'; async function generateAndIngestReport() { const doc = new PDFDocument(); const chunks: Buffer[] = []; doc.on('data', (chunk) => chunks.push(chunk)); doc.on('end', async () => { const pdfBuffer = Buffer.concat(chunks); const base64Data = pdfBuffer.toString('base64'); const response = await graphlit.ingestEncodedFile( 'monthly-report.pdf', 'application/pdf', base64Data, undefined, undefined, true ); console.log(`Report ingested: ${response.ingestEncodedFile.id}`); }); // Generate PDF content doc.fontSize(20).text('Monthly Report', 100, 100); doc.fontSize(12).text('Data and analysis...', 100, 150); doc.end(); } ``` ## Common Issues **Issue**: `Invalid base64 data` error\ **Solution**: Ensure data is properly base64 encoded. Remove any data URL prefixes (`data:mime;base64,`). **Issue**: `Unsupported MIME type`\ **Solution**: Check MIME type spelling. Use exact MIME type strings from reference list above. **Issue**: File ingested but no text extracted\ **Solution**: Ensure file is not corrupted. For scanned PDFs, use a workflow with `useVision: true`. **Issue**: Large file upload times out\ **Solution**: For files >50MB, consider using `ingestUri` with a temporary signed URL instead, or split into chunks. **Issue**: Filename has no extension\ **Solution**: Add proper extension to `name` parameter. Graphlit uses extension to determine file type. ## Production Example **Email attachment ingestion**: ```typescript const response = await graphlit.ingestEncodedFile( email.subject || 'Email Attachment', 'message/rfc822', // Email MIME type base64EncodedEmail, undefined, undefined, true ); ``` **File upload API endpoint pattern**: ```typescript // Server-side file upload handler await graphlit.ingestEncodedFile( fileName, mimeType, base64Data, // From multipart form upload workflow ? { id: workflow } : undefined, collections?.map((id) => ({ id })), isSynchronous ); ``` # Ingest Event ## Content: Ingest Event ### User Intent "I want to ingest time-series events or episodic memories into Graphlit for temporal search and recall" ### Operation * **SDK Method**: `graphlit.ingestEvent()` * **GraphQL**: `ingestEvent` mutation * **Entity Type**: Content (Event subtype) * **Common Use Cases**: User activity logs, calendar events, app events, journal entries, timeline data, audit logs ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentState, ObservableTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Ingest an event with timestamp const response = await graphlit.ingestEvent( 'User completed onboarding tutorial', // markdown (event description) 'Onboarding Completed', // name 'New user finished the onboarding flow', // description (optional) new Date('2025-01-15T10:30:00Z') // eventDate (when it occurred) ); const eventId = response.ingestEvent.id; console.log(`Event ingested: ${eventId}`); // Retrieve the event const content = await graphlit.getContent(eventId); console.log(`Event text: ${content.content.markdown}`); console.log(`Event date: ${content.content.eventDate}`); ``` ## Ingest event (snake\_case method) response = await graphlit.ingestEvent( markdown="User completed onboarding tutorial", name="Onboarding Completed", description="New user finished the onboarding flow", event\_date=datetime(2025, 1, 15, 10, 30, 0) ) event\_id = response.ingest\_event.id if response.ingest\_event else None ```` **C#**: ```csharp using Graphlit; using System; var client = new Graphlit(); // Ingest event (PascalCase method) var response = await graphlit.IngestEvent( markdown: "User completed onboarding tutorial", name: "Onboarding Completed", description: "New user finished the onboarding flow", eventDate: new DateTime(2025, 1, 15, 10, 30, 0, DateTimeKind.Utc) ); var eventId = response.IngestEvent?.Id; ```` ### Parameters #### Required * **`markdown`** (string): Event description/content * Should be descriptive and searchable * Can include markdown formatting * This becomes the event's main content * **`name`** (string): Display name for the event (optional but recommended) #### Optional * **`description`** (string): Additional description of the event * Provides context beyond the markdown content * **`eventDate`** (Date/DateTime): When the event occurred * **Critical**: This is the temporal anchor for the event * Used for chronological ordering and time-based queries * Must be ISO 8601 format or Date object * If omitted, uses current time * **`id`** (string): Custom ID for the event * **`identifier`** (string): Custom identifier for deduplication * **`collections`** (EntityReferenceInput\[]): Collections to organize events * **`correlationId`** (string): For tracking in production systems ### Response ```typescript { ingestEvent: { id: string; // Event ID name: string; // Name you provided state: ContentState; // FINISHED type: ContentEVENT; // Always EVENT markdown: string; // The event content description?: string; // Description if provided eventDate: Date; // Event timestamp } } ``` ### Developer Hints #### Event vs Text vs Memory: Key Differences | Aspect | ingestEvent | ingestText | ingestMemory | | ----------------- | ------------------------- | ----------------- | ----------------- | | **Purpose** | Time-series events | General content | Semantic memories | | **Timestamp** | eventDate (specific time) | creationDate only | creationDate only | | **Use Case** | Calendar events, logs | Documents, notes | AI agent memory | | **Query Pattern** | Time-based filtering | Full-text search | Semantic search | | **Content Type** | `EVENT` | `TEXT` | `MEMORY` | #### When to Use ingestEvent Use `ingestEvent` when: * Events have a specific timestamp (when it happened matters) * Building a timeline or activity log * Tracking calendar events or milestones * Creating audit trails * Need temporal queries (e.g., "what happened last week?") * Event occurred at a specific point in time Use `ingestText` when: * Content is timeless (e.g., documentation) * Timestamp is not meaningful * Just storing reference information Use `ingestMemory` when: * Storing semantic memories for AI agents * Building conversational context * Don't need specific event timestamps #### šŸ• Understanding Timestamps ```typescript // Current time const now = new Date(); await graphlit.ingestMemory('Event just happened', 'Current Event', now, undefined, undefined, true); // Past event const pastDate = new Date('2025-01-01T12:00:00Z'); await graphlit.ingestMemory('New Year celebration', 'New Year', pastDate, undefined, undefined, true); // Future event (scheduled) const futureDate = new Date('2025-12-31T23:59:59Z'); await graphlit.ingestMemory('Year-end review scheduled', 'Future Event', futureDate, undefined, undefined, true); ``` **Important**: Timestamps enable time-range queries when searching memories. ### Variations #### 1. Ingesting User Activity Log Track user actions for personalization: ```typescript // Create a collection for user activities const collectionResponse = await graphlit.createCollection({ name: 'User Activity Log' }); const collectionId = collectionResponse.createCollection.id; // Log multiple activities const activities = [ { text: 'User logged in', name: 'Login', timestamp: new Date('2025-01-15T09:00:00Z') }, { text: 'User viewed product page: Widget A', name: 'Page View', timestamp: new Date('2025-01-15T09:05:00Z') }, { text: 'User added Widget A to cart', name: 'Add to Cart', timestamp: new Date('2025-01-15T09:10:00Z') }, { text: 'User completed purchase: $99.99', name: 'Purchase', timestamp: new Date('2025-01-15T09:15:00Z') } ]; for (const activity of activities) { await graphlit.ingestMemory( activity.text, activity.name, activity.timestamp, undefined, [{ id: collectionId }], true ); } // Now you can query activities by time range const filter = { collections: [collectionId], dateRange: { from: new Date('2025-01-15T09:00:00Z'), to: new Date('2025-01-15T09:30:00Z') } }; const results = await graphlit.queryContents(filter); console.log(`Found ${results.contents.results.length} activities in time range`); ``` #### 2. Ingesting with Entity Extraction Extract structured data from memory text: ```typescript // Create extraction workflow const workflowInput: WorkflowInput = { name: 'Extract Memory Entities', extraction: { jobs: [ { connector: { type: EntityExtractionServiceTypes.ModelText, modelText: { extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Place, ObservableTypes.Event ] } } } ] } }; const workflowResponse = await graphlit.createWorkflow(workflowInput); // Ingest memory with extraction const response = await graphlit.ingestMemory( 'Had lunch with Sarah Johnson at Cafe Roma to discuss the partnership with TechCorp', 'Business Lunch', new Date('2025-01-15T12:00:00Z'), { id: workflowResponse.createWorkflow.id }, undefined, true // Must be synchronous for workflow ); // Retrieve with extracted entities const content = await graphlit.getContent(response.ingestMemory.id); // Will extract: Person (Sarah Johnson), Place (Cafe Roma), Organization (TechCorp) console.log(`Extracted ${content.content.observations?.length || 0} entities from memory`); ``` #### 3. Building Conversation History Store conversation turns as memories: ```typescript const conversationId = 'user-123'; const collectionResponse = await graphlit.createCollection({ name: `Conversation History - ${conversationId}` }); const collectionId = collectionResponse.createCollection.id; // User message await graphlit.ingestMemory( 'User asked: What are the best practices for API rate limiting?', `${conversationId} - User Message`, new Date(), undefined, [{ id: collectionId }], true ); // Assistant response await graphlit.ingestMemory( 'Assistant responded: Here are the key strategies for API rate limiting: 1. Token bucket algorithm...', `${conversationId} - Assistant Message`, new Date(), undefined, [{ id: collectionId }], true ); // Later, retrieve full conversation history const filter = { collections: [collectionId], orderBy: 'CREATION_DATE', orderDirection: 'ASC' }; const history = await graphlit.queryContents(filter); // Chronologically ordered conversation ``` #### 4. Importing Historical Data Bulk import past events: ```typescript // Import calendar events from external system const calendarEvents = [ { description: 'Team standup meeting', date: '2025-01-10T10:00:00Z' }, { description: 'Client presentation', date: '2025-01-12T14:00:00Z' }, { description: 'Code review session', date: '2025-01-14T15:30:00Z' } ]; for (const event of calendarEvents) { await graphlit.ingestMemory( event.description, 'Calendar Event', new Date(event.date), undefined, undefined, false // Asynchronous for bulk import ); } console.log(`Imported ${calendarEvents.length} historical events`); ``` ### Common Issues **Issue**: Memories not appearing in chronological order\ **Solution**: Ensure timestamps are correctly set. Use `orderBy: 'CREATION_DATE'` when querying. **Issue**: `timestamp` parameter not accepted\ **Solution**: Ensure you're using a Date object (JavaScript) or datetime (Python), not a string. Or use ISO 8601 string format. **Issue**: Cannot filter memories by date range\ **Solution**: When querying, use `dateRange` filter with `from` and `to` timestamps. **Issue**: Workflow not extracting entities from memories\ **Solution**: Ensure workflow `extraction.jobs[].connector.type` is set to `EntityExtractionServiceTypes.ModelText` for text-based memory extraction. ### Production Example **Server-side memory ingestion**: ```typescript const response = await graphlit.ingestMemory( text, name, timestamp, workflow ? { id: workflow } : undefined, collections?.map((id) => ({ id })), isSynchronous ); ``` **Activity logging pattern**: ```typescript // Log user activity with current timestamp await graphlit.ingestMemory( `User ${userId} completed action: ${actionName}`, `Activity: ${actionName}`, new Date(), // Current time undefined, [{ id: activityCollectionId }], true ); ``` # Ingest Text ## Content: Ingest Text ### User Intent "I want to ingest plain text directly into Graphlit without a file or URL" ### Operation * **SDK Method**: `graphlit.ingestText()` * **GraphQL**: `ingestText` mutation * **Entity Type**: Content * **Common Use Cases**: User-generated notes, chat messages, API responses, scraped text, clipboard content ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentState, ObservableTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Basic text ingestion (synchronous) const response = await graphlit.ingestText( 'This is my note about the project meeting. We discussed Q1 goals and timeline.', // text 'Meeting Notes - January 2025', // name undefined, // textType (optional) undefined, // uri (optional - for reference) undefined, // id (optional) undefined, // identifier (optional) true, // isSynchronous undefined, // workflow (optional) undefined // collections (optional) ); const contentId = response.ingestText.id; console.log(`Text ingested: ${contentId}`); // Retrieve the content const content = await graphlit.getContent(contentId); console.log(`Content markdown: ${content.content.markdown}`); ``` ## Ingest text (snake\_case method) ## Note: Python SDK uses named parameters, order doesn't matter response = await graphlit.ingestText( text="This is my note about the project meeting.", name="Meeting Notes - January 2025", is\_synchronous=True ) content\_id = response.ingest\_text.id if response.ingest\_text else None ```` **C#**: ```csharp using Graphlit; var client = new Graphlit(); // Ingest text (PascalCase method) var response = await graphlit.IngestText( text: "This is my note about the project meeting.", name: "Meeting Notes - January 2025", isSynchronous: true ); var contentId = response.IngestText?.Id; ```` ### Parameters #### Required * **`text`** (string): The text content to ingest * Can be plain text or markdown * No size limit specified, but keep reasonable for performance #### Optional * **`name`** (string): Display name for the content * **`textType`** (TextTypes): Type hint for the text * `PLAIN` - Plain text (default) * `MARKDOWN` - Markdown formatted text * **`uri`** (string): Optional reference URI for the text source * **`id`** (string): Custom ID for the content * **`identifier`** (string): Custom identifier for deduplication * **`isSynchronous`** (boolean): Wait for ingestion to complete * **Default**: `false` (asynchronous) * **Recommended**: `true` for immediate use * **`workflow`** (EntityReferenceInput): Workflow to apply during ingestion * **`collections`** (EntityReferenceInput\[]): Collections to add content to * **`observations`** (ObservationReferenceInput\[]): Observations to link * **`correlationId`** (string): For tracking in production systems ### Response ```typescript { ingestText: { id: string; // Content ID name: string; // Name you provided state: ContentState; // FINISHED (if synchronous) type: ContentTEXT; // Always TEXT markdown: string; // The ingested text textType?: TextTypes; // PLAIN or MARKDOWN uri?: string; // Reference URI if provided } } ``` ### Developer Hints #### Key Differences from ingestUri 1. **No file parsing needed** - Text is used directly, no extraction step 2. **Immediate availability** - Even with `isSynchronous: false`, text is usually available immediately 3. **No file metadata** - No fileType, mimeType, or file-specific fields 4. **Markdown support** - Can ingest pre-formatted markdown #### When to Use ingestText vs ingestUri Use `ingestText` when: * You already have the text in memory * Text comes from user input, API, or database * No file parsing needed * You want to store snippets, notes, or short content Use `ingestUri` when: * Content is in a file (PDF, DOCX, etc.) * Need file parsing/extraction * Content is at a URL #### Understanding isSynchronous ```typescript // Asynchronous (default) - returns immediately const response = await graphlit.ingestText(text, name, undefined, undefined, undefined, undefined, false); // Content ID available immediately, processing happens in background // Synchronous - waits for processing const response = await graphlit.ingestText(text, name, undefined, undefined, undefined, undefined, true); // Returns when content.state === 'FINISHED' ``` **For ingestText, the difference is minimal** since text doesn't require heavy processing. However, if you're using a workflow (e.g., entity extraction), synchronous mode ensures the workflow completes before returning. ### Variations #### 1. Ingesting with Collections Organize text into collections during ingestion: ```typescript // Create a collection first const collectionResponse = await graphlit.createCollection({ name: 'Meeting Notes' }); const collectionId = collectionResponse.createCollection.id; // Ingest text into collection const response = await graphlit.ingestText( 'Discussion about Q1 product roadmap and feature priorities.', 'Product Planning Meeting', undefined, // textType undefined, // uri undefined, // workflow [{ id: collectionId }], // collections true ); ``` #### 2. Ingesting Markdown Preserve markdown formatting: ```typescript const markdownText = ` # Project Update ## Completed Tasks - Feature A implementation - Bug fixes in module B ## Next Steps - Code review - Deployment planning `; const response = await graphlit.ingestText( markdownText, 'Weekly Project Update', TextTypes.Markdown, // Specify markdown type undefined, undefined, undefined, true ); // The markdown structure is preserved const content = await graphlit.getContent(response.ingestText.id); console.log(content.content.markdown); // Includes formatting ``` #### 3. Ingesting with Entity Extraction Apply a workflow to extract entities from text: ```typescript // Create extraction workflow const workflowInput: WorkflowInput = { name: 'Extract People and Orgs', extraction: { jobs: [ { connector: { type: EntityExtractionServiceTypes.ModelText, modelText: { extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization ] } } } ] } }; const workflowResponse = await graphlit.createWorkflow(workflowInput); // Ingest text with extraction const response = await graphlit.ingestText( 'John Smith from Acme Corp discussed the partnership with Jane Doe from TechCo.', 'Partnership Discussion', undefined, undefined, { id: workflowResponse.createWorkflow.id }, // workflow undefined, true // Must be synchronous to wait for extraction ); // Retrieve with extracted entities const content = await graphlit.getContent(response.ingestText.id); console.log(`Extracted ${content.content.observations?.length || 0} entities`); // Will include Person entities for "John Smith" and "Jane Doe" // Will include Organization entities for "Acme Corp" and "TechCo" ``` #### 4. Ingesting with Reference URI Track the source of text: ```typescript // Text scraped from a web API const scrapedText = 'Product description from external API...'; const response = await graphlit.ingestText( scrapedText, 'Product Description - Widget A', TextTypes.Plain, 'https://api.example.com/products/widget-a', // Reference URI undefined, undefined, true ); // The URI is stored for reference const content = await graphlit.getContent(response.ingestText.id); console.log(`Source: ${content.content.uri}`); ``` ### Common Issues **Issue**: Text ingested but not appearing in search\ **Solution**: Ensure embeddings are generated. Check project configuration for embedding model. If using asynchronous mode, wait a few seconds for embedding generation. **Issue**: Special characters or formatting lost\ **Solution**: Use `textType: TextTypes.Markdown` if your text includes markdown formatting. Plain text is default. **Issue**: Workflow not executing\ **Solution**: Must use `isSynchronous: true` when applying workflows. Asynchronous mode may return before workflow completes. **Issue**: Content state is `AWAITING_EXTRACTION`\ **Solution**: Wait for processing to complete if using asynchronous mode. Or switch to synchronous mode. ### Production Example **From Public Samples**: ```typescript // Server-side text ingestion with all options const response = await graphlit.ingestText( text, name, textType as TextTypes | undefined, uri, workflow ? { id: workflow } : undefined, collections?.map((id) => ({ id })), isSynchronous ); ``` **Re-ingesting updated text**: ```typescript // Update content by re-ingesting with same name const response = await graphlit.ingestText( editedText, originalName, TextTypes.Plain, undefined, undefined, undefined, true ); ``` # Ingest URI (Basic) ## Content: Ingest URI (Basic) ### User Intent "I want to ingest a document, web page, or file from a URL into Graphlit" ### Operation * **SDK Method**: `graphlit.ingestUri()` * **GraphQL**: `ingestUri` mutation * **Entity Type**: Content * **Common Use Cases**: PDF ingestion, web page extraction, audio/video transcription, image processing ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentState, ContentTypes, FileTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Basic ingestion (asynchronous - returns immediately) const response = await graphlit.ingestUri( 'https://example.com/document.pdf' ); const contentId = response.ingestUri.id; console.log(`Content ingestion started: ${contentId}`); // Synchronous ingestion (waits for completion) const syncResponse = await graphlit.ingestUri( 'https://example.com/document.pdf', undefined, // workflow (optional) undefined, // collections (optional) true // isSynchronous ); const completedContentId = syncResponse.ingestUri.id; console.log(`Content ingested and processed: ${completedContentId}`); // Retrieve the ingested content const content = await graphlit.getContent(completedContentId); console.log(`Content name: ${content.content.name}`); console.log(`Content type: ${content.content.type}`); ``` ## Synchronous ingestion (snake\_case method names) response = await graphlit.ingestUri( uri="", is\_synchronous=True ) content\_id = response.ingest\_uri.id if response.ingest\_uri else None ```` **C#**: ```csharp using Graphlit; var client = new Graphlit(); // Synchronous ingestion (PascalCase method names) var response = await graphlit.IngestUri( uri: "https://example.com/document.pdf", isSynchronous: true ); var contentId = response.IngestUri?.Id; ```` ### Parameters #### Required * **`uri`** (string): URL of the content to ingest * Supports: HTTP/HTTPS URLs * File types: PDF, DOCX, images, audio, video, web pages, etc. #### Optional * **`workflow`** (EntityReferenceInput): Workflow ID for custom extraction/preparation * **`collections`** (EntityReferenceInput\[]): Collections to assign content to * **`isSynchronous`** (boolean): Wait for ingestion to complete (default: false) * **`correlationId`** (string): For tracking ingestion in production systems ### Response ```typescript { ingestUri: { id: string; // Content ID name: string; // Extracted filename state: ContentState; // AWAITING_EXTRACTION, FINISHED, ERROR type: ContentTypes; // FILE, PAGE, EMAIL, etc. fileType: FileTypes; // PDF, DOCX, IMAGE, AUDIO, VIDEO mimeType: string; // MIME type of the content uri: string; // Original URI markdown?: string; // Extracted text (if available) } } ``` ### Variations #### 1. Asynchronous Ingestion with Polling (Production Pattern) For high-volume ingestion, use asynchronous mode and poll for completion: ```typescript // Start ingestion (returns immediately) const response = await graphlit.ingestUri( 'https://example.com/large-video.mp4', undefined, // name (optional) undefined, // id (optional) undefined, // identifier (optional) false // isSynchronous - async mode ); const contentId = response.ingestUri.id; // Poll for completion using isContentDone let isDone = false; while (!isDone) { const status = await graphlit.isContentDone(contentId); isDone = status.isContentDone.result || false; if (!isDone) { await new Promise(resolve => setTimeout(resolve, 5000)); // Wait 5 seconds console.log('Still processing...'); } } console.log('Content processing complete!'); // Now fetch the fully processed content const content = await graphlit.getContent(contentId); console.log(`Processed: ${content.content.name}`); ``` #### 2. Ingestion with Collections Organize content during ingestion: ```typescript // Create or reference a collection const collectionResponse = await graphlit.createCollection({ name: 'Product Documentation' }); // Ingest into collection const response = await graphlit.ingestUri( 'https://example.com/user-guide.pdf', undefined, // workflow [{ id: collectionResponse.createCollection.id }], // collections true // isSynchronous ); ``` #### 3. Ingestion with Custom Workflow Apply extraction or preparation during ingestion: ```typescript // Reference a workflow (e.g., for entity extraction) const response = await graphlit.ingestUri( 'https://example.com/contract.pdf', { id: 'workflow-id-here' }, // workflow undefined, // collections true // isSynchronous ); // Content will be processed through the workflow const content = await graphlit.getContent(response.ingestUri.id); console.log(`Entities extracted: ${content.content.observations?.length || 0}`); ``` ### Common Issues **Issue**: `Error: Failed to download content from URI`\ **Solution**: Ensure the URL is publicly accessible or provide authentication via workflow configuration. **Issue**: `Content state is ERROR`\ **Solution**: Check `content.error` for details. Common causes: * Unsupported file format * File too large (check project limits) * Corrupt file * Network timeout **Issue**: Synchronous ingestion timing out\ **Solution**: For large files (>100MB), use asynchronous mode and poll for completion instead. ### Production Example **Server-side ingestion with all options**: ```typescript const response = await graphlit.ingestUri( uri, name, undefined, // id undefined, // identifier isSynchronous, workflow ? { id: workflow } : undefined, collections?.map(id => ({ id })) ); ``` **Conditional workflow application**: ```typescript // Apply different workflows based on file type // Assumes you have created workflows beforehand: // const docWorkflow = await graphlit.createWorkflow({ name: "Document Processing", extraction: {...} }); // const documentWorkflowId = docWorkflow.createWorkflow.id; const isDocument = uri.endsWith('.pdf') || uri.endsWith('.docx'); const workflowId = isDocument ? documentWorkflowId : undefined; const response = await graphlit.ingestUri( uri, undefined, // name (auto-generated) undefined, // id undefined, // identifier true, // isSynchronous workflowId ? { id: workflowId } : undefined ); ``` # Ingest URI with Workflow ## User Intent "I want to apply custom extraction, preparation, or processing to content during ingestion" ## Operation * **SDK Method**: `graphlit.ingestUri()` with workflow parameter * **GraphQL**: `ingestUri` mutation with workflow reference * **Entity Type**: Content + Workflow * **Common Use Cases**: Entity extraction, vision-based PDF parsing, audio transcription with custom models ## TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { EntityExtractionServiceTypes, FilePreparationServiceTypes, WorkflowActionServiceTypes, DeepgramModels, ObservableTypes, WorkflowInput, ContentState, FileTypes, } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // 1. Create a workflow that extracts people & organizations const workflowInput: WorkflowInput = { name: 'Entity Extraction Workflow', extraction: { jobs: [ { connector: { type: EntityExtractionServiceTypes.ModelText, modelText: { extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization, ], }, }, }, ], }, }; const workflowResponse = await graphlit.createWorkflow(workflowInput); const workflowId = workflowResponse.createWorkflow.id; // 2. Ingest content with that workflow enabled const ingestResponse = await graphlit.ingestUri( 'https://example.com/contract.pdf', 'Vendor Contract', { id: workflowId }, true, // wait until extraction completes ); // 3. Retrieve entities extracted during ingestion const content = await graphlit.getContent(ingestResponse.ingestUri.id); const entities = content.content.observations ?? []; console.log(`Extracted ${entities.length} entities`); console.log(entities.slice(0, 5).map((obs) => `${obs.observable?.type}: ${obs.observable?.name}`)); ``` ## Parameters ### Required * **`uri`** (string): URL of the content to ingest * **`workflow`** (EntityReferenceInput): Workflow ID to apply ### Optional * **`collections`** (EntityReferenceInput\[]): Collections to assign content to * **`isSynchronous`** (boolean): Wait for workflow completion (recommended: true) ## Response ```typescript { ingestUri: { id: string; state: ContentState; // FINISHED when workflow completes observations?: Observable[]; // Extracted entities (if extraction workflow) markdown?: string; // Extracted text (if preparation workflow) metadata?: { // Custom metadata from workflow actions [key: string]: any; } } } ``` ## Variations ### 1. Vision-Based PDF Extraction Use vision models for better PDF text extraction: ```typescript // Create preparation workflow with vision model const visionWorkflow: WorkflowInput = { name: 'Vision PDF Extraction', preparation: { jobs: [ { connector: { type: FilePreparationServiceTypes.ModelDocument, modelDocument: { includeImages: true // Enable vision-based extraction }, fileTypes: [FileTypes.Document] } } ] } }; const workflowResponse = await graphlit.createWorkflow(visionWorkflow); // Ingest PDF with vision extraction const response = await graphlit.ingestUri( 'https://example.com/scanned-document.pdf', { id: workflowResponse.createWorkflow.id }, undefined, true ); // Better markdown extraction from scanned/image-based PDFs const content = await graphlit.getContent(response.ingestUri.id); console.log(content.content.markdown); ``` ### 2. Audio Transcription Workflow Transcribe audio/video with custom settings: ```typescript // Create preparation workflow for audio const audioWorkflow: WorkflowInput = { name: 'Audio Transcription', preparation: { jobs: [ { connector: { type: FilePreparationServiceTypes.Deepgram, deepgram: { model: DeepgramModels.Nova2 }, fileTypes: [FileTypes.Audio, FileTypes.Video] } } ] } }; const workflowResponse = await graphlit.createWorkflow(audioWorkflow); // Ingest audio with transcription const response = await graphlit.ingestUri( 'https://example.com/podcast-episode.mp3', { id: workflowResponse.createWorkflow.id }, undefined, true ); // Access transcript const content = await graphlit.getContent(response.ingestUri.id); console.log(content.content.markdown); // Full transcript ``` ### 3. Combined Preparation + Extraction Chain preparation and extraction in one workflow: ```typescript const combinedWorkflow: WorkflowInput = { name: 'Prepare and Extract', preparation: { jobs: [ { connector: { type: FilePreparationServiceTypes.ModelDocument, fileTypes: [FileTypes.Document] } } ] }, extraction: { jobs: [ { connector: { type: EntityExtractionServiceTypes.ModelText, modelText: { extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Event ] } } } ] } }; const workflowResponse = await graphlit.createWorkflow(combinedWorkflow); // Content will be prepared (text extraction), then entities extracted const response = await graphlit.ingestUri( 'https://example.com/meeting-notes.pdf', { id: workflowResponse.createWorkflow.id }, undefined, true ); ``` ### 4. Workflow with Custom Actions Execute custom code during workflow: ```typescript const actionWorkflow: WorkflowInput = { name: 'Custom Action Workflow', actions: [ { connector: { type: WorkflowActionServiceTypes.Webhook, uri: 'https://your-api.com/webhook', // Custom action called during workflow } } ] }; ``` ## Common Issues **Issue**: `Workflow not found`\ **Solution**: Ensure workflow ID is valid and belongs to your project. Create workflow first. **Issue**: Workflow takes too long / times out\ **Solution**: Use asynchronous ingestion for large files: ```typescript const response = await graphlit.ingestUri(uri, { id: workflowId }, undefined, false); // Poll for completion ``` **Issue**: Entities not extracted\ **Solution**: Check workflow `extraction.jobs[].connector.extractionTypes` matches content type. **Issue**: Vision extraction not working\ **Solution**: Ensure `useVision: true` in preparation workflow and content is PDF or image. # Poll for Completion ## Content: Poll for Completion ### User Intent "I want to know when content processing has finished" ### Operation * **SDK Method**: `graphlit.isContentDone()` * **GraphQL**: `isContentDone` query * **Entity Type**: Content * **Common Use Cases**: Wait for async ingestion, check processing status ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { Types } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // After async ingestion const contentResponse = await graphlit.ingestUri( 'https://example.com/document.pdf', undefined, undefined, undefined, false // isSynchronous = false (async) ); const contentId = contentResponse.ingestUri.id; console.log('Content ingestion started, polling for completion...'); // Poll for completion let isDone = false; let attempts = 0; const maxAttempts = 60; // 10 minutes max while (!isDone && attempts < maxAttempts) { const status = await graphlit.isContentDone(contentId); isDone = status.isContentDone.result || false; if (!isDone) { attempts++; console.log(`Still processing... (${attempts}/${maxAttempts})`); await new Promise(resolve => setTimeout(resolve, 10000)); // Wait 10 seconds } } if (isDone) { console.log(' Content processing complete!'); // Now safe to access content const content = await graphlit.getContent(contentId); console.log(`Extracted ${content.content.markdown?.length || 0} chars`); } else { console.log('ā° Timeout - still processing'); } ``` ## After async ingestion content\_response = await graphlit.ingestUri( uri="", is\_synchronous=False ) content\_id = content\_response.ingest\_uri.id ## Poll for completion (snake\_case) is\_done = False attempts = 0 max\_attempts = 60 while not is\_done and attempts < max\_attempts: status = await graphlit.isContentDone(content\_id) is\_done = status.is\_content\_done.result if status.is\_content\_done else False ``` if not is_done: attempts += 1 print(f"Still processing... ({attempts}/{max_attempts})") await asyncio.sleep(10) ``` if is\_done: print(" Content processing complete!") ```` **C#**: ```csharp using Graphlit; using System.Threading.Tasks; var client = new Graphlit(); // After async ingestion var contentResponse = await graphlit.IngestUri( uri: "https://example.com/document.pdf", isSynchronous: false ); var contentId = contentResponse.IngestUri.Id; // Poll for completion (PascalCase) bool isDone = false; int attempts = 0; int maxAttempts = 60; while (!isDone && attempts < maxAttempts) { var status = await graphlit.IsContentDone(contentId); isDone = status.IsContentDone?.Result ?? false; if (!isDone) { attempts++; Console.WriteLine($"Still processing... ({attempts}/{maxAttempts})"); await Task.Delay(10000); // Wait 10 seconds } } if (isDone) { Console.WriteLine(" Content processing complete!"); } ```` ### Parameters * **`id`** (string): Content ID to check ### Response ```typescript { isContentDone: { result: boolean; // true = complete, false = still processing } } ``` ### Developer Hints #### Async vs Sync Ingestion ```typescript // Synchronous - waits automatically const content = await graphlit.ingestUri(uri, undefined, undefined, undefined, true); // Content ready immediately // Asynchronous - need to poll const content = await graphlit.ingestUri(uri, undefined, undefined, undefined, false); // Poll with isContentDone() ``` #### Polling Helper Function ```typescript async function waitForContent( contentId: string, timeoutMinutes: number = 10 ): Promise { const maxAttempts = timeoutMinutes * 6; // 6 checks per minute let attempts = 0; while (attempts < maxAttempts) { const status = await graphlit.isContentDone(contentId); if (status.isContentDone.result) { return true; } attempts++; await new Promise(resolve => setTimeout(resolve, 10000)); } return false; // Timeout } // Usage const done = await waitForContent(contentId, 10); if (done) { console.log('Ready!'); } ``` # Lifecycle States ## Content: Lifecycle States ### User Intent "What do content states mean? When should I use each state?" ### Operation * **SDK Method**: `updateContent()` with `state` parameter * **GraphQL Field**: `content.state` (EntityState enum) * **Entity Type**: Content * **Common Use Cases**: Content management, soft delete, archival, workflow states ### Content States Overview Content progresses through states during its lifecycle. Understanding states is critical for content management and querying. ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { EntityState } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Check content state const content = await graphlit.getContent('content-id'); console.log(`State: ${content.content.state}`); // Update content state await graphlit.updateContent({ id: 'content-id', state: EntityState.Disabled // Hide from queries }); // Query only enabled content const activeContent = await graphlit.queryContents({ states: [EntityState.Enabled] }); // Query disabled content (hidden) const hiddenContent = await graphlit.queryContents({ states: [EntityState.Disabled] }); // Query all states (including disabled) const allContent = await graphlit.queryContents({ states: [ EntityState.Enabled, EntityState.Disabled, EntityState.Created ] }); ``` ### Entity States ```typescript enum EntityState { CREATED = 'CREATED', // Just ingested, not yet processed ENABLED = 'ENABLED', // Fully processed, searchable, active DISABLED = 'DISABLED', // Hidden from default queries (soft delete) DELETED = 'DELETED', // Permanently deleted (rarely seen) ARCHIVED = 'ARCHIVED' // Preserved but not actively used } ``` #### CREATED * **When**: Immediately after ingestion * **Duration**: Brief (during workflow processing) * **Searchable**: No (processing not complete) * **Use Case**: Content is being prepared/extracted **Example**: ```typescript const content = await graphlit.ingestUri( 'https://example.com/document.pdf', undefined, undefined, undefined, false, // Returns immediately { id: 'workflow-id' } ); console.log(content.ingestUri.state); // CREATED // Wait for processing let isDone = false; while (!isDone) { const status = await graphlit.isContentDone(content.ingestUri.id); isDone = status.isContentDone.result; await new Promise(resolve => setTimeout(resolve, 1000)); } // Check state after processing const processed = await graphlit.getContent(content.ingestUri.id); console.log(processed.content.state); // ENABLED ``` #### ENABLED * **When**: After successful workflow processing * **Searchable**: Yes (default state for queries) * **Use Case**: Active, searchable content **Example**: ```typescript // Default queries only return ENABLED content const results = await graphlit.queryContents({ search: "graphlit" }); // All results will have state = ENABLED results.contents.results.forEach(content => { console.log(content.state); // ENABLED }); ``` #### DISABLED * **When**: Manually disabled by user * **Searchable**: No (unless explicitly queried) * **Use Case**: Soft delete, hide temporarily, compliance hold **Example**: ```typescript // Disable content (soft delete) await graphlit.updateContent({ id: 'content-id', state: EntityState.Disabled }); // Default queries won't return it const results = await graphlit.queryContents({ search: "query" }); // Content with DISABLED state won't appear // Must explicitly query disabled content const disabled = await graphlit.queryContents({ states: [EntityState.Disabled] }); ``` #### DELETED * **When**: Permanently deleted via deleteContent() * **Searchable**: No (content is gone) * **Use Case**: Permanent removal **Example**: ```typescript // Permanent delete await graphlit.deleteContent('content-id'); // Content no longer exists // Can't query or retrieve ``` #### ARCHIVED * **When**: Manually archived by user * **Searchable**: No (unless explicitly queried) * **Use Case**: Long-term retention, compliance, not actively used **Example**: ```typescript // Archive content await graphlit.updateContent({ id: 'content-id', state: EntityState.Archived }); // Query archived content const archived = await graphlit.queryContents({ states: [EntityState.Archived] }); ``` ### State Transitions ``` Ingest ā†’ CREATED ā†“ Workflow Processing ā†“ ENABLED ā†ā†’ DISABLED (soft delete/restore) ā†“ ARCHIVED (long-term storage) ā†“ DELETED (permanent removal) ``` **Valid Transitions**: * `CREATED` ā†’ `ENABLED` (automatic after processing) * `ENABLED` ā†’ `DISABLED` (soft delete) * `DISABLED` ā†’ `ENABLED` (restore) * `ENABLED` ā†’ `ARCHIVED` (archive) * `ARCHIVED` ā†’ `ENABLED` (restore from archive) * Any ā†’ `DELETED` (permanent delete) ## Check state content = await graphlit.getContent('content-id') print(f"State: {content.content.state}") ## Update state (snake\_case) await graphlit.updateContent( id='content-id', state=EntityState.DISABLED ) ## Query by state results = await graphlit.queryContents( filter=ContentFilterInput( states=\[EntityState.ENABLED] ) ) ```` **C#**: ```csharp using Graphlit; var client = new Graphlit(); // Check state var content = await graphlit.GetContent("content-id"); Console.WriteLine($"State: {content.Content.State}"); // Update state (PascalCase) await graphlit.UpdateContent(new ContentUpdateInput { Id = "content-id", State = EntityState.Disabled }); // Query by state var results = await graphlit.QueryContents(new ContentFilter { States = new[] { EntityState.Enabled } }); ```` ### Developer Hints #### Default Query Behavior ```typescript // This query... const results = await graphlit.queryContents({ search: "query" }); // ...is equivalent to: const results = await graphlit.queryContents({ search: "query", states: [EntityState.Enabled] // Default! }); // DISABLED, ARCHIVED content won't appear unless explicitly queried ``` #### Soft Delete vs Hard Delete ```typescript // Soft delete (reversible) await graphlit.updateContent({ id: 'content-id', state: EntityState.Disabled }); // Can restore later await graphlit.updateContent({ id: 'content-id', state: EntityState.Enabled }); // Hard delete (permanent) await graphlit.deleteContent('content-id'); // Content is gone forever ``` #### Querying Multiple States ```typescript // Query both enabled and archived const results = await graphlit.queryContents({ states: [ EntityState.Enabled, EntityState.Archived ] }); // Query everything except deleted const everything = await graphlit.queryContents({ states: [ EntityState.Created, EntityState.Enabled, EntityState.Disabled, EntityState.Archived ] }); ``` ### Variations #### 1. Soft Delete Content ```typescript // Hide from queries but preserve await graphlit.updateContent({ id: 'content-id', state: EntityState.Disabled }); ``` #### 2. Restore Soft-Deleted Content ```typescript // Make searchable again await graphlit.updateContent({ id: 'content-id', state: EntityState.Enabled }); ``` #### 3. Archive Old Content ```typescript // Archive content older than 1 year const oneYearAgo = new Date(); oneYearAgo.setFullYear(oneYearAgo.getFullYear() - 1); const oldContent = await graphlit.queryContents({ creationDateRange: { to: oneYearAgo.toISOString() }, states: [EntityState.Enabled] }); // Archive each for (const content of oldContent.contents.results) { await graphlit.updateContent({ id: content.id, state: EntityState.Archived }); console.log(`Archived: ${content.name}`); } ``` #### 4. Query Only Active Content ```typescript // Explicit enabled filter (best practice) const active = await graphlit.queryContents({ states: [EntityState.Enabled] }); ``` #### 5. View Disabled Content (Admin View) ```typescript // Show hidden content const hidden = await graphlit.queryContents({ states: [EntityState.Disabled] }); console.log(`${hidden.contents.results.length} hidden items`); ``` #### 6. Bulk State Changes ```typescript // Disable multiple content items const contentIds = ['id1', 'id2', 'id3']; for (const id of contentIds) { await graphlit.updateContent({ id, state: EntityState.Disabled }); } // Or use batch operations await Promise.all( contentIds.map(id => graphlit.updateContent({ id, state: EntityState.Disabled }) ) ); ``` #### 7. Compliance Archival ```typescript // Archive but preserve for compliance await graphlit.updateContent({ id: 'content-id', state: EntityState.Archived, metadata: { archivedDate: new Date().toISOString(), archivedReason: 'Compliance retention', retainUntil: '2030-01-01' } }); ``` ### Common Issues & Solutions **Issue**: Content not appearing in queries **Solution**: Check if content is disabled or archived ```typescript // Check content state const content = await graphlit.getContent('content-id'); console.log(`State: ${content.content.state}`); if (content.content.state !== EntityState.Enabled) { console.log('Content is not enabled'); } ``` **Issue**: Accidentally deleted content **Solution**: Use soft delete (DISABLED) instead ```typescript // DON'T - Permanent delete await graphlit.deleteContent('content-id'); // DO - Soft delete await graphlit.updateContent({ id: 'content-id', state: EntityState.Disabled }); ``` **Issue**: Archived content still appearing **Solution**: Default queries only return ENABLED ```typescript // This won't return archived content const results = await graphlit.queryContents({}); // Must explicitly exclude if needed const onlyActive = await graphlit.queryContents({ states: [EntityState.Enabled] }); ``` **Issue**: Content stuck in CREATED state **Solution**: Check workflow processing status ```typescript const content = await graphlit.getContent('content-id'); if (content.content.state === EntityState.Created) { // Check if processing is done const status = await graphlit.isContentDone('content-id'); if (!status.isContentDone.result) { console.log('Still processing...'); } else if (content.content.error) { console.log(`Error: ${content.content.error}`); } } ``` ### Production Example ```typescript async function manageContentLifecycle(contentId: string) { const content = await graphlit.getContent(contentId); console.log(`\n=== CONTENT LIFECYCLE ===`); console.log(`ID: ${content.content.id}`); console.log(`Name: ${content.content.name}`); console.log(`Current State: ${content.content.state}`); console.log(`Created: ${content.content.creationDate}`); // State-based actions switch (content.content.state) { case EntityState.Created: console.log('\nā³ Processing...'); const isDone = await graphlit.isContentDone(contentId); if (isDone.isContentDone.result) { console.log('āœ“ Processing complete'); } else { console.log('Still processing workflow'); } break; case EntityState.Enabled: console.log('\nāœ“ Active and searchable'); // Check if old enough to archive const age = Date.now() - new Date(content.content.creationDate).getTime(); const daysOld = age / (1000 * 60 * 60 * 24); if (daysOld > 365) { console.log(`šŸ“¦ Content is ${Math.floor(daysOld)} days old - consider archiving`); } break; case EntityState.Disabled: console.log('\nšŸ”’ Hidden (soft deleted)'); console.log('Can be restored with state: ENABLED'); break; case EntityState.Archived: console.log('\nšŸ“¦ Archived'); console.log('Preserved for compliance, not actively used'); break; } return content.content.state; } // Usage await manageContentLifecycle('content-id'); ``` # Message Metadata Queries ## Content: Message Metadata Queries ### User Intent "How do I query Slack/Teams/Discord messages by channel, author, mentions, etc.?" ### Operation * **SDK Method**: `queryContents()` with message-specific patterns * **GraphQL**: `queryContents` query * **Entity Type**: Content (type: MESSAGE) * **Common Use Cases**: Find messages by channel, author queries, mentions detection, link analysis ### Message Metadata Structure Messages (from Slack, Teams, Discord) have metadata in the `message` field: ```typescript interface MessageMetadata { identifier: string; // Message ID conversationIdentifier: string; // Thread/conversation ID channelIdentifier: string; // Channel ID channelName: string; // Channel name author: PersonReference; // Message author mentions: PersonReference[]; // @mentioned users attachmentCount: number; links: string[]; // URLs in message } ``` ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentTypes, SearchTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Query all messages const allMessages = await graphlit.queryContents({ types: [ContentTypes.Message] }); // Search messages by keyword const searchMessages = await graphlit.queryContents({ search: "product roadmap", types: [ContentTypes.Message] }); // Recent messages const recentMessages = await graphlit.queryContents({ types: [ContentTypes.Message], createdInLast: 'P7D' // Last 7 days }); // Messages from specific feed (e.g., Slack workspace) const slackMessages = await graphlit.queryContents({ types: [ContentTypes.Message], feeds: [{ id: 'slack-feed-id' }] }); console.log(`Found ${slackMessages.contents.results.length} messages`); // Access message metadata slackMessages.contents.results.forEach(msg => { if (msg.message) { console.log(`Channel: ${msg.message.channelName}`); console.log(`Author: ${msg.message.author?.name}`); console.log(`Message: ${msg.markdown?.substring(0, 100)}...`); } }); ``` ### Query Patterns #### 1. Filter by Channel ```typescript // Get messages from all channels const messages = await graphlit.queryContents({ types: [ContentTypes.Message] limit: 500 }); // Filter by channel name const engineering = messages.contents.results.filter(msg => msg.message?.channelName === 'engineering' ); const product = messages.contents.results.filter(msg => msg.message?.channelName === 'product' ); console.log(`Engineering messages: ${engineering.length}`); console.log(`Product messages: ${product.length}`); // List all channels const channels = new Set( messages.contents.results .map(m => m.message?.channelName) .filter(Boolean) ); console.log(`Channels: ${Array.from(channels).join(', ')}`); ``` #### 2. Find by Author ```typescript // Search by author name const byAuthor = await graphlit.queryContents({ search: "Kirk Marple", searchType: SearchTypes.Keyword, types: [ContentTypes.Message] }); // Filter by author email const messages = await graphlit.queryContents({ types: [ContentTypes.Message] }); const kirkMessages = messages.contents.results.filter(msg => msg.message?.author?.email === "kirk@graphlit.com" ); console.log(`Kirk's messages: ${kirkMessages.length}`); // Count messages by author const byAuthorCount = new Map(); messages.contents.results.forEach(msg => { if (msg.message?.author) { const author = msg.message.author.name || msg.message.author.email; byAuthorCount.set(author, (byAuthorCount.get(author) || 0) + 1); } }); console.log('\nTop 10 authors:'); Array.from(byAuthorCount.entries()) .sort((a, b) => b[1] - a[1]) .slice(0, 10) .forEach(([author, count]) => { console.log(` ${author}: ${count} messages`); }); ``` #### 3. Find Mentions ```typescript // Get all messages const messages = await graphlit.queryContents({ types: [ContentTypes.Message] }); // Find messages mentioning specific user const mentioningKirk = messages.contents.results.filter(msg => msg.message?.mentions?.some(m => m.email === "kirk@graphlit.com") ); console.log(`Messages mentioning Kirk: ${mentioningKirk.length}`); // Find messages with any mentions const withMentions = messages.contents.results.filter(msg => msg.message?.mentions && msg.message.mentions.length > 0 ); console.log(`Messages with mentions: ${withMentions.length}`); // Count mentions per person const mentionCounts = new Map(); messages.contents.results.forEach(msg => { msg.message?.mentions?.forEach(mention => { const name = mention.name || mention.email; mentionCounts.set(name, (mentionCounts.get(name) || 0) + 1); }); }); console.log('\nMost mentioned people:'); Array.from(mentionCounts.entries()) .sort((a, b) => b[1] - a[1]) .slice(0, 10) .forEach(([person, count]) => { console.log(` ${person}: ${count} mentions`); }); ``` #### 4. Thread/Conversation Queries ```typescript // Get all messages const messages = await graphlit.queryContents({ types: [ContentTypes.Message] }); // Group by conversation/thread const conversations = new Map(); messages.contents.results.forEach(msg => { if (msg.message?.conversationIdentifier) { const convId = msg.message.conversationIdentifier; if (!conversations.has(convId)) { conversations.set(convId, []); } conversations.get(convId)?.push(msg); } }); console.log(`Found ${conversations.size} conversations`); // Find longest threads const sortedConversations = Array.from(conversations.entries()) .sort((a, b) => b[1].length - a[1].length); console.log('\nTop 5 longest threads:'); sortedConversations.slice(0, 5).forEach(([convId, messages]) => { const channel = messages[0].message?.channelName; const firstMessage = messages[0].markdown?.substring(0, 50); console.log(`${messages.length} messages in #${channel}: "${firstMessage}..."`); }); // Find replies (where identifier !== conversationIdentifier) const replies = messages.contents.results.filter(msg => msg.message?.identifier !== msg.message?.conversationIdentifier ); console.log(`\nReplies: ${replies.length} out of ${messages.contents.results.length} messages`); ``` #### 5. Messages with Links ```typescript // Get messages const messages = await graphlit.queryContents({ types: [ContentTypes.Message] }); // Filter messages with links const withLinks = messages.contents.results.filter(msg => msg.message?.links && msg.message.links.length > 0 ); console.log(`Messages with links: ${withLinks.length}`); // Extract all shared links const allLinks = new Set(); messages.contents.results.forEach(msg => { msg.message?.links?.forEach(link => allLinks.add(link)); }); console.log(`Unique links shared: ${allLinks.size}`); // Most shared domains const domains = new Map(); allLinks.forEach(link => { try { const url = new URL(link); domains.set(url.hostname, (domains.get(url.hostname) || 0) + 1); } catch (e) { // Invalid URL } }); console.log('\nMost shared domains:'); Array.from(domains.entries()) .sort((a, b) => b[1] - a[1]) .slice(0, 10) .forEach(([domain, count]) => { console.log(` ${domain}: ${count} links`); }); ``` #### 6. Messages with Attachments ```typescript // Get messages const messages = await graphlit.queryContents({ types: [ContentTypes.Message] }); // Filter by attachments const withAttachments = messages.contents.results.filter(msg => msg.message && msg.message.attachmentCount > 0 ); console.log(`Messages with attachments: ${withAttachments.length}`); // Distribution const attachmentDist = new Map(); withAttachments.forEach(msg => { const count = msg.message?.attachmentCount || 0; attachmentDist.set(count, (attachmentDist.get(count) || 0) + 1); }); console.log('\nAttachment distribution:'); Array.from(attachmentDist.entries()) .sort((a, b) => a[0] - b[0]) .forEach(([count, messages]) => { console.log(` ${count} attachment(s): ${messages} messages`); }); ``` #### 7. Channel Activity Analysis ```typescript // Get messages const messages = await graphlit.queryContents({ types: [ContentTypes.Message], createdInLast: 'P30D' // Last 30 days limit: 1000 }); // Messages per channel const byChannel = new Map(); messages.contents.results.forEach(msg => { const channel = msg.message?.channelName || 'unknown'; byChannel.set(channel, (byChannel.get(channel) || 0) + 1); }); console.log('Channel activity (last 30 days):'); Array.from(byChannel.entries()) .sort((a, b) => b[1] - a[1]) .forEach(([channel, count]) => { console.log(` #${channel}: ${count} messages`); }); // Active hours const byHour = new Array(24).fill(0); messages.contents.results.forEach(msg => { const hour = new Date(msg.creationDate).getHours(); byHour[hour]++; }); console.log('\nMost active hours:'); byHour .map((count, hour) => ({ hour, count })) .sort((a, b) => b.count - a.count) .slice(0, 5) .forEach(({ hour, count }) => { console.log(` ${hour}:00 - ${count} messages`); }); ``` #### 8. Collaboration Patterns ```typescript // Get messages const messages = await graphlit.queryContents({ types: [ContentTypes.Message] }); // Build interaction matrix (who mentions whom) const interactions = new Map>(); messages.contents.results.forEach(msg => { const author = msg.message?.author?.email; if (!author) return; msg.message?.mentions?.forEach(mention => { if (!interactions.has(author)) { interactions.set(author, new Map()); } const authorInteractions = interactions.get(author)!; const mentionEmail = mention.email; authorInteractions.set(mentionEmail, (authorInteractions.get(mentionEmail) || 0) + 1); }); }); // Find strongest collaborations console.log('Strongest collaborations:'); const collaborations: Array<{ from: string; to: string; count: number }> = []; interactions.forEach((targets, source) => { targets.forEach((count, target) => { collaborations.push({ from: source, to: target, count }); }); }); collaborations .sort((a, b) => b.count - a.count) .slice(0, 10) .forEach(({ from, to, count }) => { console.log(` ${from} ā†’ ${to}: ${count} mentions`); }); ``` ## Query messages messages = await graphlit.queryContents( filter=ContentFilterInput( types=\[ContentTypes.Message] ) ) ## From specific feed slack\_messages = await graphlit.queryContents( filter=ContentFilterInput( types=\[ContentTypes.Message], feeds=\[EntityReferenceInput(id='slack-feed-id')] ) ) ## Access metadata for msg in messages.contents.results: if msg.message: print(f"Channel: {msg.message.channel\_name}") print(f"Author: {msg.message.author.name}") if msg.message.mentions: print(f"Mentions: {len(msg.message.mentions)}") ```` **C#**: ```csharp using Graphlit; var client = new Graphlit(); // Query messages var messages = await graphlit.QueryContents(new ContentFilter { Filter = new ContentCriteria { Types = new[] { ContentTypes.Message } } }); // From specific feed var slackMessages = await graphlit.QueryContents(new ContentFilter { Filter = new ContentCriteria { Types = new[] { ContentTypes.Message }, Feeds = new[] { new EntityReference { Id = "slack-feed-id" } } } }); // Access metadata foreach (var msg in messages.Contents.Results) { if (msg.Message != null) { Console.WriteLine($"Channel: {msg.Message.ChannelName}"); Console.WriteLine($"Author: {msg.Message.Author.Name}"); if (msg.Message.Mentions != null) { Console.WriteLine($"Mentions: {msg.Message.Mentions.Length}"); } } } ```` ### Developer Hints #### Channel Names are Searchable ```typescript // Search for channel name const results = await graphlit.queryContents({ search: "engineering", types: [ContentTypes.Message] }); // Or filter after query const all = await graphlit.queryContents({ types: [ContentTypes.Message] }); const engineering = all.contents.results.filter( m => m.message?.channelName === 'engineering' ); ``` #### Mentions Array ```typescript // Mentions is array of PersonReference message.mentions.forEach(mention => { console.log(`@${mention.name} (${mention.email})`); }); ``` #### Thread Detection ```typescript // Message is thread reply if: if (message.identifier !== message.conversationIdentifier) { console.log('This is a reply in a thread'); } ``` ### Common Issues & Solutions **Issue**: Can't filter by specific channel in query **Solution**: Query all messages, filter client-side ```typescript const all = await graphlit.queryContents({ types: [ContentTypes.Message] }); const engineering = all.contents.results.filter( m => m.message?.channelName === 'engineering' ); ``` **Issue**: Want to find all messages from one user to another **Solution**: Filter by author and mentions ```typescript const messages = await graphlit.queryContents({ types: [ContentTypes.Message] }); const kirkToMaria = messages.contents.results.filter(msg => msg.message?.author?.email === 'kirk@graphlit.com' && msg.message?.mentions?.some(m => m.email === 'maria@example.com') ); ``` **Issue**: Need to count messages per channel **Solution**: Query and aggregate ```typescript const messages = await graphlit.queryContents({ types: [ContentTypes.Message] }); const byChannel = new Map(); messages.contents.results.forEach(msg => { const channel = msg.message?.channelName || 'unknown'; byChannel.set(channel, (byChannel.get(channel) || 0) + 1); }); ``` ### Production Example ```typescript async function analyzeTeamCommunication() { console.log('\n=== TEAM COMMUNICATION ANALYSIS ===\n'); // Get recent messages const messages = await graphlit.queryContents({ types: [ContentTypes.Message], createdInLast: 'P30D' limit: 1000 }); console.log(`Total messages (last 30 days): ${messages.contents.results.length}`); // Channel breakdown const channels = new Map(); messages.contents.results.forEach(msg => { const ch = msg.message?.channelName || 'unknown'; channels.set(ch, (channels.get(ch) || 0) + 1); }); console.log(`\nActive channels: ${channels.size}`); console.log('Top 10 channels:'); Array.from(channels.entries()) .sort((a, b) => b[1] - a[1]) .slice(0, 10) .forEach(([channel, count]) => { console.log(` #${channel}: ${count} messages`); }); // Author activity const authors = new Map(); messages.contents.results.forEach(msg => { const author = msg.message?.author?.name || 'unknown'; authors.set(author, (authors.get(author) || 0) + 1); }); console.log(`\nActive authors: ${authors.size}`); console.log('Top 10 authors:'); Array.from(authors.entries()) .sort((a, b) => b[1] - a[1]) .slice(0, 10) .forEach(([author, count]) => { console.log(` ${author}: ${count} messages`); }); // Mention statistics const mentioned = new Map(); let totalMentions = 0; messages.contents.results.forEach(msg => { msg.message?.mentions?.forEach(mention => { const name = mention.name || mention.email; mentioned.set(name, (mentioned.get(name) || 0) + 1); totalMentions++; }); }); console.log(`\nTotal mentions: ${totalMentions}`); console.log('Most mentioned:'); Array.from(mentioned.entries()) .sort((a, b) => b[1] - a[1]) .slice(0, 10) .forEach(([person, count]) => { console.log(` ${person}: ${count} mentions`); }); // Collaboration score const collaborations = new Map>(); messages.contents.results.forEach(msg => { const author = msg.message?.author?.email; if (author) { if (!collaborations.has(author)) { collaborations.set(author, new Set()); } msg.message?.mentions?.forEach(m => { collaborations.get(author)!.add(m.email); }); } }); console.log(`\n Collaboration Matrix:`); Array.from(collaborations.entries()) .sort((a, b) => b[1].size - a[1].size) .slice(0, 5) .forEach(([author, collaborators]) => { console.log(` ${author}: works with ${collaborators.size} people`); }); } await analyzeTeamCommunication(); ``` # Metadata Filtering Strategies ## Content: Metadata Filtering Strategies ### User Intent "How do I filter content by metadata? What filters are available?" ### Operation * **SDK Method**: `queryContents()` with `filter` parameter * **GraphQL**: `queryContents` query with ContentFilter * **Entity Type**: Content * **Common Use Cases**: Filtered search, date range queries, type filtering, collection filtering, entity-based queries ### Metadata Filtering Overview Graphlit provides powerful metadata filtering to narrow search results based on indexed properties. Filters can be combined with search queries or used alone. ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentTypes, EntityState, FileTypes, ObservableTypes, SearchTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Basic metadata filtering (no search query) const filtered = await graphlit.queryContents({ types: [ContentTypes.Email], creationDateRange: { from: '2024-01-01T00:00:00Z', to: '2024-12-31T23:59:59Z' } }); // Combine search with filters const searchAndFilter = await graphlit.queryContents({ search: "machine learning", types: [ContentTypes.File], fileTypes: [FileTypes.Document], collections: [{ id: 'research-papers' }] }); // Complex filter with multiple criteria const complex = await graphlit.queryContents({ types: [ContentTypes.Email, ContentTypes.Message], creationDateRange: { from: '2024-01-01' }, feeds: [{ id: 'slack-feed-id' }, { id: 'gmail-feed-id' }], collections: [{ id: 'team-docs' }], states: [EntityState.Enabled] }); console.log(`Found ${complex.contents.results.length} results`); ``` ### Available Filters #### 1. Content Type Filter ```typescript // Filter by content type const emails = await graphlit.queryContents({ types: [ContentTypes.Email] }); // Multiple types (OR logic) const communications = await graphlit.queryContents({ types: [ ContentTypes.Email, ContentTypes.Message, ContentTypes.Post ] }); ``` #### 2. File Type Filter ```typescript // Filter by file type (for FILE content only) const documents = await graphlit.queryContents({ types: [ContentTypes.File], fileTypes: [FileTypes.Document] }); // Multiple file types const media = await graphlit.queryContents({ fileTypes: [ FileTypes.Image, FileTypes.Audio, FileTypes.Video ] }); ``` #### 3. Date Range Filters ```typescript // Creation date range const thisYear = await graphlit.queryContents({ creationDateRange: { from: '2024-01-01T00:00:00Z', to: '2024-12-31T23:59:59Z' } }); // Original date (document creation date) const originalDate = await graphlit.queryContents({ dateRange: { from: '2023-01-01' } }); // Relative date (last N days) - ISO 8601 duration const lastMonth = await graphlit.queryContents({ inLast: 'P30D' // Last 30 days }); const lastWeek = await graphlit.queryContents({ createdInLast: 'P7D' // Created in last 7 days }); ``` #### 4. Collection Filter ```typescript // Single collection const inCollection = await graphlit.queryContents({ collections: [{ id: 'collection-id' }] }); // Multiple collections (OR logic) const multiCollection = await graphlit.queryContents({ collections: [ { id: 'engineering-docs' }, { id: 'product-docs' }, { id: 'design-docs' } ] }); // Content WITHOUT any collection const noCollection = await graphlit.queryContents({ hasCollections: false }); // Content WITH any collection const hasCollection = await graphlit.queryContents({ hasCollections: true }); ``` #### 5. Feed Filter ```typescript // Content from specific feed const fromFeed = await graphlit.queryContents({ feeds: [{ id: 'slack-feed-id' }] }); // Multiple feeds const multiFeeds = await graphlit.queryContents({ feeds: [ { id: 'slack-feed-id' }, { id: 'gmail-feed-id' }, { id: 'github-feed-id' } ] }); // Content WITHOUT any feed (manually ingested) const noFeed = await graphlit.queryContents({ hasFeeds: false }); ``` #### 6. Workflow Filter ```typescript // Content processed by specific workflow const byWorkflow = await graphlit.queryContents({ workflows: [{ id: 'extraction-workflow-id' }] }); // Multiple workflows const multiWorkflows = await graphlit.queryContents({ workflows: [ { id: 'extraction-workflow' }, { id: 'preparation-workflow' } ] }); // Content WITHOUT any workflow const noWorkflow = await graphlit.queryContents({ hasWorkflows: false }); ``` #### 7. State Filter ```typescript // Only enabled content (default) const enabled = await graphlit.queryContents({ states: [EntityState.Enabled] }); // Include disabled (hidden) content const all = await graphlit.queryContents({ states: [ EntityState.Enabled, EntityState.Disabled ] }); // Only archived content const archived = await graphlit.queryContents({ states: [EntityState.Archived] }); ``` #### 8. File Size Filter ```typescript // File size range (bytes) const largeDocs = await graphlit.queryContents({ fileSizeRange: { from: 10000000, // 10MB to: 100000000 // 100MB } }); // Files larger than X const huge = await graphlit.queryContents({ fileSizeRange: { from: 100000000 // > 100MB } }); ``` #### 9. MIME Type Filter ```typescript // Specific MIME type const pdfs = await graphlit.queryContents({ formats: ['application/pdf'] }); // Multiple MIME types const images = await graphlit.queryContents({ formats: [ 'image/jpeg', 'image/png', 'image/gif' ] }); ``` #### 10. File Extension Filter ```typescript // Specific extensions const codeFiles = await graphlit.queryContents({ fileExtensions: ['ts', 'tsx', 'js', 'jsx'] }); const docs = await graphlit.queryContents({ fileExtensions: ['pdf', 'docx', 'xlsx', 'pptx'] }); ``` #### 11. Entity Filter (Observations) ```typescript // Content mentioning specific person const withPerson = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Person, observable: { id: 'person-id' } }] }); // Multiple entities (AND logic - content must mention ALL) const multiEntity = await graphlit.queryContents({ observations: [ { type: ObservableTypes.Person, observable: { id: 'person-1' } }, { type: ObservableTypes.Organization, observable: { id: 'org-1' } } ] }); // Content with any observations const hasEntities = await graphlit.queryContents({ hasObservations: true }); // Content without observations const noEntities = await graphlit.queryContents({ hasObservations: false }); ``` #### 12. Similarity Filter ```typescript // Find similar content const similar = await graphlit.queryContents({ similarContents: [{ id: 'content-id' }] limit: 10 }); ``` ### Boolean Logic (OR / AND) #### OR Logic Within Filter ```typescript // Multiple values in same filter = OR const emailsOrMessages = await graphlit.queryContents({ types: [ ContentTypes.Email, // OR ContentTypes.Message ] }); ``` #### AND Logic Across Filters ```typescript // Multiple filters = AND const emailsFromSlack = await graphlit.queryContents({ types: [ContentTypes.Message], // AND feeds: [{ id: 'slack-feed' }], // AND creationDateRange: { from: '2024-01-01' } // AND }); ``` #### Complex OR/AND (Advanced) ```typescript // Complex boolean logic const complex = await graphlit.queryContents({ // Base criteria (AND) types: [ContentTypes.Email], creationDateRange: { from: '2024-01-01' }, // OR clause or: [ { feeds: [{ id: 'gmail-feed' }] }, { feeds: [{ id: 'outlook-feed' }] } ], // AND clause and: [ { collections: [{ id: 'important' }] } ] }); // This finds: Emails from 2024 AND // (from Gmail OR Outlook) AND // in 'important' collection ``` ### Performance Considerations #### Fast Filters (Indexed) ```typescript // These are fast (indexed): const fast = await graphlit.queryContents({ types: [ContentTypes.Email], // Indexed fileTypes: [FileTypes.Document], // Indexed creationDateRange: { from: '2024-01-01' }, // Indexed feeds: [{ id: 'feed-id' }], // Indexed collections: [{ id: 'collection-id' }], // Indexed workflows: [{ id: 'workflow-id' }], // Indexed states: [EntityState.Enabled], // Indexed fileExtensions: ['pdf'], // Indexed formats: ['application/pdf'] // Indexed }); ``` #### Slower Filters (Requires Graph Query) ```typescript // Entity filters query graph database (slower) const slower = await graphlit.queryContents({ observations: [{ // Graph query type: ObservableTypes.Person, observable: { id: 'person-id' } }] }); // Still fast enough for production, just slower than pure index queries ``` ## Metadata filtering (snake\_case) filtered = await graphlit.queryContents( filter=ContentFilterInput( types=\[ContentTypes.Email], creation\_date\_range=DateRangeInput( from\_=datetime(2024, 1, 1).isoformat() ), feeds=\[EntityReferenceInput(id='feed-id')] ) ) ## Complex filter complex = await graphlit.queryContents( filter=ContentFilterInput( types=\[ContentTypes.File], file\_types=\[FileTypes.Document], file\_size\_range=Int64RangeInput( from\_=10000000 # 10MB ), collections=\[EntityReferenceInput(id='collection-id')] ) ) ```` **C#**: ```csharp using Graphlit; var client = new Graphlit(); // Metadata filtering (PascalCase) var filtered = await graphlit.QueryContents(new ContentFilter { Types = new[] { ContentTypes.Email }, CreationDateRange = new DateRange { From = new DateTime(2024, 1, 1) }, Feeds = new[] { new EntityReference { Id = "feed-id" } } }); // Complex filter var complex = await graphlit.QueryContents(new ContentFilter { Types = new[] { ContentTypes.File }, FileTypes = new[] { FileDocument }, FileSizeRange = new Int64Range { From = 10000000 // 10MB }, Collections = new[] { new EntityReference { Id = "collection-id" } } }); ```` ### Developer Hints #### Combine Filters for Precision ```typescript // āœ“ Narrow down results const precise = await graphlit.queryContents({ search: "quarterly report", types: [ContentTypes.File], fileTypes: [FileTypes.Document], creationDateRange: { from: '2024-01-01', to: '2024-03-31' }, collections: [{ id: 'finance-docs' }] }); ``` #### ISO 8601 Duration Format ```typescript // Relative dates use ISO 8601 durations 'P7D' // Last 7 days 'P30D' // Last 30 days 'P1M' // Last 1 month 'P3M' // Last 3 months 'P1Y' // Last 1 year 'PT24H' // Last 24 hours ``` #### Empty Array vs Null ```typescript // No filter (all content) await graphlit.queryContents({}); // Explicit empty array (same as no filter) await graphlit.queryContents({ types: [] // Same as not providing types }); // Single type await graphlit.queryContents({ types: [ContentTypes.Email] }); ``` ### Variations #### 1. Filter by Type and Date ```typescript const emailsThisYear = await graphlit.queryContents({ types: [ContentTypes.Email], creationDateRange: { from: '2024-01-01' } }); ``` #### 2. Filter by Multiple Collections ```typescript const docs = await graphlit.queryContents({ collections: [ { id: 'engineering' }, { id: 'product' }, { id: 'design' } ] }); ``` #### 3. Large Documents Only ```typescript const large = await graphlit.queryContents({ types: [ContentTypes.File], fileTypes: [FileTypes.Document], fileSizeRange: { from: 50000000 // > 50MB } }); ``` #### 4. Recent Content from Specific Feed ```typescript const recent = await graphlit.queryContents({ feeds: [{ id: 'slack-feed' }], createdInLast: 'P7D' // Last 7 days }); ``` #### 5. Content with Entities ```typescript const withEntities = await graphlit.queryContents({ hasObservations: true }); ``` #### 6. PDFs in Collection ```typescript const pdfs = await graphlit.queryContents({ fileExtensions: ['pdf'], collections: [{ id: 'research-papers' }] }); ``` #### 7. Search + Complex Filter ```typescript const searchFiltered = await graphlit.queryContents({ search: "machine learning", searchType: SearchTypes.Hybrid, types: [ContentTypes.File], fileTypes: [FileTypes.Document], creationDateRange: { from: '2024-01-01' }, collections: [{ id: 'research' }], fileSizeRange: { from: 1000000, // > 1MB to: 100000000 // < 100MB } }); ``` ### Common Issues & Solutions **Issue**: No results with multiple filters **Solution**: Check if filters are too restrictive ```typescript // Too restrictive (might return 0 results) await graphlit.queryContents({ types: [ContentTypes.Email], fileTypes: [FileTypes.Document] // Emails don't have fileType! }); // Correct await graphlit.queryContents({ types: [ContentTypes.Email], }); ``` **Issue**: Date filter not working **Solution**: Use correct ISO 8601 format ```typescript // Wrong format creationDateRange: { from: '01/01/2024' } // Correct format creationDateRange: { from: '2024-01-01T00:00:00Z' } // Also correct (date only) creationDateRange: { from: '2024-01-01' } ``` **Issue**: Want content from Feed A OR Feed B **Solution**: Multiple feeds in array = OR ```typescript // āœ“ This is OR logic await graphlit.queryContents({ feeds: [ { id: 'feed-a' }, { id: 'feed-b' }, ], }); ``` ### Production Example ```typescript async function advancedFilter() { console.log(`\n=== ADVANCED FILTERING ===`); // Build filter programmatically const filter: any = { states: [EntityState.Enabled] }; // Add date filter (last 90 days) filter.createdInLast = 'P90D'; // Add type filter filter.types = [ ContentTypes.Email, ContentTypes.Message, ContentTypes.File ]; // Add collection filter if specified const collectionId = 'team-docs'; if (collectionId) { filter.collections = [{ id: collectionId }]; } // Execute query const results = await graphlit.queryContents({ search: "project status", ...filter, }); console.log(`\nFilters applied:`); console.log(` - States: ${filter.states.join(', ')}`); console.log(` - Created: Last 90 days`); console.log(` - Types: ${filter.types.join(', ')}`); if (filter.collections) { console.log(` - Collection: ${filter.collections[0].id}`); } console.log(`\nResults: ${results.contents.results.length}`); // Group by type const byType = results.contents.results.reduce((acc, content) => { acc[content.type] = (acc[content.type] || 0) + 1; return acc; }, {} as Record); console.log('\nBy Type:'); Object.entries(byType).forEach(([type, count]) => { console.log(` ${type}: ${count}`); }); } await advancedFilter(); ``` # Metadata Structure by Content Type ## Content: Metadata Structure by Content Type ### User Intent "What metadata is captured for each content type? How do I access it?" ### Operation * **Concept**: Content metadata fields * **GraphQL Fields**: Type-specific metadata objects * **Entity Type**: Content * **Common Use Cases**: Accessing email details, message info, document properties, image metadata, event details ### Metadata Fields Overview Each ContentType has a corresponding metadata field with type-specific properties automatically captured during ingestion. ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentTypes, FileTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); const content = await graphlit.getContent('content-id'); // Access metadata based on content type if (content.content.type === ContentTypes.Email && content.content.email) { const email = content.content.email; console.log(`Subject: ${email.subject}`); console.log(`From: ${email.from[0].email}`); console.log(`To: ${email.to?.map(p => p.email).join(', ')}`); console.log(`Labels: ${email.labels?.join(', ')}`); console.log(`Attachments: ${email.attachmentCount}`); } if (content.content.type === ContentTypes.Message && content.content.message) { const msg = content.content.message; console.log(`Channel: ${msg.channelName}`); console.log(`Author: ${msg.author?.name} (${msg.author?.email})`); console.log(`Mentions: ${msg.mentions?.map(p => p.name).join(', ')}`); } if (content.content.fileType === FileTypes.Document && content.content.document) { const doc = content.content.document; console.log(`Pages: ${doc.pageCount}`); console.log(`Author: ${doc.author}`); console.log(`Words: ${doc.wordCount}`); console.log(`Encrypted: ${doc.isEncrypted}`); } ``` ### Email Metadata (ContentTypes.Email) **Field**: `content.email` (EmailMetadata) **Properties**: ```typescript interface EmailMetadata { identifier: string; // Message-ID threadIdentifier: string; // In-Reply-To / References subject: string; from: PersonReference[]; // Sender to: PersonReference[]; // Recipients cc: PersonReference[]; // CC recipients bcc: PersonReference[]; // BCC recipients labels: string[]; // Gmail labels or folder names sensitivity: MailSensitivity; // NORMAL, PERSONAL, PRIVATE, CONFIDENTIAL priority: MailPriority; // LOW, NORMAL, HIGH importance: MailImportance; // LOW, NORMAL, HIGH attachmentCount: number; unsubscribeUrl: string; // Newsletter unsubscribe publicationName: string; // Newsletter name publicationUrl: string; // Newsletter URL } interface PersonReference { name: string; email: string; givenName: string; familyName: string; } ``` **Example Access**: ```typescript const email = content.content.email; // Sender details console.log(`From: ${email.from[0].name} <${email.from[0].email}>`); // All recipients const allRecipients = [ ...email.to || [], ...email.cc || [], ...email.bcc || [] ]; console.log(`Total recipients: ${allRecipients.length}`); // Thread detection if (email.threadIdentifier) { console.log(`Part of thread: ${email.threadIdentifier}`); } // Newsletter detection if (email.publicationName) { console.log(`Newsletter: ${email.publicationName}`); } ``` ### Message Metadata (ContentTypes.Message) **Field**: `content.message` (MessageMetadata) **Properties**: ```typescript interface MessageMetadata { identifier: string; // Message ID conversationIdentifier: string; // Thread/conversation ID channelIdentifier: string; // Channel ID channelName: string; // Channel name (e.g., "engineering") author: PersonReference; // Message author mentions: PersonReference[]; // @mentioned users attachmentCount: number; links: string[]; // URLs in message } ``` **Example Access**: ```typescript const msg = content.content.message; // Author info console.log(`Posted by: ${msg.author.name} in #${msg.channelName}`); // Mentions if (msg.mentions && msg.mentions.length > 0) { console.log(`Mentioned: ${msg.mentions.map(p => '@' + p.name).join(' ')}`); } // Links shared if (msg.links && msg.links.length > 0) { console.log(`Links: ${msg.links.length}`); } // Thread detection if (msg.conversationIdentifier !== msg.identifier) { console.log(`Reply in conversation: ${msg.conversationIdentifier}`); } ``` ### Document Metadata (FileDocument) **Field**: `content.document` (DocumentMetadata) **Properties**: ```typescript interface DocumentMetadata { title: string; subject: string; summary: string; author: string; lastModifiedBy: string; publisher: string; description: string; keywords: string[]; pageCount: number; worksheetCount: number; // Excel slideCount: number; // PowerPoint wordCount: number; lineCount: number; paragraphCount: number; isEncrypted: boolean; hasDigitalSignature: boolean; } ``` **Example Access**: ```typescript const doc = content.content.document; // Document stats console.log(`${doc.pageCount} pages, ${doc.wordCount} words`); console.log(`Author: ${doc.author}`); // Excel-specific if (doc.worksheetCount) { console.log(`Excel workbook: ${doc.worksheetCount} worksheets`); } // PowerPoint-specific if (doc.slideCount) { console.log(`Presentation: ${doc.slideCount} slides`); } // Security if (doc.isEncrypted) { console.log(` Encrypted document`); } if (doc.hasDigitalSignature) { console.log(`āœ“ Digitally signed`); } ``` ### Image Metadata (FileImage) **Field**: `content.image` (ImageMetadata) **Properties**: ```typescript interface ImageMetadata { width: number; height: number; resolutionX: number; // DPI horizontal resolutionY: number; // DPI vertical bitsPerComponent: number; components: number; projectionType: ImageProjectionTypes; // EQUIRECTANGULAR, etc. orientation: OrientationTypes; description: string; // Camera EXIF data make: string; // Camera manufacturer model: string; // Camera model software: string; // Editing software lens: string; focalLength: number; // mm exposureTime: string; // e.g., "1/125" fNumber: string; // e.g., "f/2.8" iso: string; // e.g., "ISO 400" // GPS data heading: number; // Compass direction pitch: number; // Angle } ``` **Example Access**: ```typescript const img = content.content.image; // Basic info console.log(`${img.width}x${img.height} pixels`); console.log(`Resolution: ${img.resolutionX} DPI`); // Camera info if (img.make && img.model) { console.log(`Camera: ${img.make} ${img.model}`); console.log(`Lens: ${img.lens}`); console.log(`Settings: ${img.exposureTime} at ${img.fNumber}, ${img.iso}`); } // GPS location if (content.content.location) { console.log(`Location: ${content.content.location.latitude}, ${content.content.location.longitude}`); } ``` ### Audio Metadata (FileAudio) **Field**: `content.audio` (AudioMetadata) **Properties**: ```typescript interface AudioMetadata { title: string; description: string; author: string; keywords: string[]; // Podcast-specific series: string; // Podcast series name episode: string; // Episode number episodeType: string; // FULL, TRAILER, BONUS season: string; // Season number publisher: string; copyright: string; genre: string; // Technical bitrate: number; // bits per second channels: number; // 1=mono, 2=stereo sampleRate: number; // Hz bitsPerSample: number; duration: string; // ISO 8601 duration } ``` **Example Access**: ```typescript const audio = content.content.audio; // Podcast info if (audio.series) { console.log(`Podcast: ${audio.series}`); console.log(`Episode: ${audio.episode} (${audio.episodeType})`); if (audio.season) console.log(`Season: ${audio.season}`); } // Audio quality console.log(`${audio.bitrate / 1000}kbps, ${audio.sampleRate}Hz`); console.log(`${audio.channels === 1 ? 'Mono' : 'Stereo'}`); // Duration import { Duration } from 'luxon'; const dur = Duration.fromISO(audio.duration); console.log(`Length: ${dur.toFormat('mm:ss')}`); ``` ### Video Metadata (FileVideo) **Field**: `content.video` (VideoMetadata) **Properties**: ```typescript interface VideoMetadata { width: number; height: number; duration: string; // ISO 8601 duration title: string; description: string; keywords: string[]; author: string; // Camera/device info make: string; model: string; software: string; } ``` **Example Access**: ```typescript const video = content.content.video; console.log(`${video.width}x${video.height}`); console.log(`Duration: ${video.duration}`); console.log(`Title: ${video.title}`); if (video.make) { console.log(`Recorded on: ${video.make} ${video.model}`); } ``` ### Event Metadata (ContentEvent) **Field**: `content.event` (EventMetadata) **Properties**: ```typescript interface EventMetadata { eventIdentifier: string; calendarIdentifier: string; subject: string; startDateTime: Date; endDateTime: Date; isAllDay: boolean; timezone: string; status: CalendarEventStatus; // CONFIRMED, TENTATIVE, CANCELLED visibility: CalendarEventVisibility; // PUBLIC, PRIVATE, CONFIDENTIAL meetingLink: string; categories: string[]; organizer: CalendarAttendee; attendees: CalendarAttendee[]; reminders: CalendarReminder[]; // Recurring events recurringEventIdentifier: string; isRecurring: boolean; recurrence: CalendarRecurrence; } interface CalendarAttendee { name: string; email: string; isOptional: boolean; isOrganizer: boolean; responseStatus: CalendarAttendeeResponseStatus; // ACCEPTED, DECLINED, TENTATIVE, NEEDS_ACTION } ``` **Example Access**: ```typescript const event = content.content.event; console.log(`Event: ${event.subject}`); console.log(`When: ${event.startDateTime} - ${event.endDateTime}`); console.log(`Timezone: ${event.timezone}`); // Organizer console.log(`Organizer: ${event.organizer.name} <${event.organizer.email}>`); // Attendees console.log(`Attendees (${event.attendees?.length || 0}):`); event.attendees?.forEach(att => { const status = att.responseStatus; const icon = att.isOptional ? '(optional)' : '(required)'; console.log(` ${att.name}: ${status} ${icon}`); }); // Meeting link if (event.meetingLink) { console.log(`Join: ${event.meetingLink}`); } // Recurring if (event.isRecurring) { console.log(`Recurring: ${event.recurrence.pattern} every ${event.recurrence.interval}`); } ``` ### Issue Metadata (ContentTypes.Issue) **Field**: `content.issue` (IssueMetadata) **Properties**: ```typescript interface IssueMetadata { identifier: string; // Issue ID/key (e.g., "PROJ-123") title: string; project: string; // Project name team: string; // Team name (Linear) status: string; // Status name priority: string; // Priority level type: string; // Issue type (Bug, Feature, etc.) labels: string[]; } ``` **Example Access**: ```typescript const issue = content.content.issue; console.log(`${issue.identifier}: ${issue.title}`); console.log(`Project: ${issue.project}`); console.log(`Status: ${issue.status}`); console.log(`Priority: ${issue.priority}`); console.log(`Type: ${issue.type}`); if (issue.labels && issue.labels.length > 0) { console.log(`Labels: ${issue.labels.join(', ')}`); } ``` ### Post Metadata (ContentPost) **Field**: `content.post` (PostMetadata) **Properties**: ```typescript interface PostMetadata { identifier: string; // Post ID title: string; author: PersonReference; upvotes: number; downvotes: number; commentCount: number; links: string[]; } ``` **Example Access**: ```typescript const post = content.content.post; console.log(`${post.title}`); console.log(`By: ${post.author.name}`); console.log(`Score: ${post.upvotes - post.downvotes} (${post.upvotes}ā†‘ ${post.downvotes}ā†“)`); console.log(`Comments: ${post.commentCount}`); if (post.links && post.links.length > 0) { console.log(`Links: ${post.links.join(', ')}`); } ``` ## Access email metadata (snake\_case fields) if content.content.type == "EMAIL" and content.content.email: email = content.content.email print(f"Subject: {email.subject}") print(f"From: {email.from\_\[0].email}") # Note: from\_ (reserved word) print(f"Attachments: {email.attachment\_count}") ## Access message metadata if content.content.type == "MESSAGE" and content.content.message: msg = content.content.message print(f"Channel: {msg.channel\_name}") print(f"Author: {msg.author.name}") ## Access document metadata if content.content.file\_type == "DOCUMENT" and content.content.document: doc = content.content.document print(f"Pages: {doc.page\_count}") print(f"Words: {doc.word\_count}") ```` **C#**: ```csharp using Graphlit; var client = new Graphlit(); var content = await graphlit.GetContent("content-id"); // Access email metadata (PascalCase) if (content.Content.Type == ContentTypes.Email && content.Content.Email != null) { var email = content.Content.Email; Console.WriteLine($"Subject: {email.Subject}"); Console.WriteLine($"From: {email.From[0].Email}"); Console.WriteLine($"Attachments: {email.AttachmentCount}"); } // Access message metadata if (content.Content.Type == ContentTypes.Message && content.Content.Message != null) { var msg = content.Content.Message; Console.WriteLine($"Channel: {msg.ChannelName}"); Console.WriteLine($"Author: {msg.Author.Name}"); } ```` ### Developer Hints #### Always Check Type First ```typescript // WRONG - Might be null console.log(content.content.email.subject); // CORRECT - Check type first if (content.content.type === ContentTypes.Email && content.content.email) { console.log(content.content.email.subject); } ``` #### Null-Safe Access ```typescript // Use optional chaining for nested properties console.log(`Attachments: ${content.content.email?.attachmentCount || 0}`); console.log(`Mentions: ${content.content.message?.mentions?.length || 0}`); ``` #### PersonReference Pattern ```typescript // PersonReference appears in multiple metadata types function formatPerson(person: PersonReference): string { if (person.name && person.email) { return `${person.name} <${person.email}>`; } return person.name || person.email || 'Unknown'; } // Use for email, message mentions, event attendees, etc. console.log(formatPerson(content.content.email.from[0])); ``` ### Common Issues & Solutions **Issue**: Metadata field is undefined **Solution**: Check content type matches expected type ```typescript // Check before accessing if (content.content.type === ContentTypes.Email && content.content.email) { // Safe to access email metadata } ``` **Issue**: Arrays are null instead of empty **Solution**: Use nullish coalescing ```typescript const labels = content.content.email?.labels || []; const mentions = content.content.message?.mentions || []; ``` **Issue**: Date fields are strings not Date objects **Solution**: Parse ISO 8601 strings ```typescript const created = new Date(content.content.creationDate); const eventStart = new Date(content.content.event.startDateTime); ``` ### Production Example ```typescript async function analyzeContent(contentId: string) { const content = await graphlit.getContent(contentId); console.log(`\n=== CONTENT ANALYSIS ===`); console.log(`ID: ${content.content.id}`); console.log(`Name: ${content.content.name}`); console.log(`Type: ${content.content.type}`); console.log(`Created: ${content.content.creationDate}`); // Type-specific metadata switch (content.content.type) { case ContentTypes.Email: if (content.content.email) { console.log(`\n Email Metadata:`); console.log(` From: ${content.content.email.from[0].email}`); console.log(` Subject: ${content.content.email.subject}`); console.log(` Recipients: ${content.content.email.to?.length || 0}`); console.log(` Labels: ${content.content.email.labels?.join(', ') || 'none'}`); } break; case ContentTypes.Message: if (content.content.message) { console.log(`\nšŸ’¬ Message Metadata:`); console.log(` Channel: ${content.content.message.channelName}`); console.log(` Author: ${content.content.message.author?.name}`); console.log(` Mentions: ${content.content.message.mentions?.length || 0}`); } break; case ContentTypes.File: if (content.content.fileType === FileTypes.Document && content.content.document) { console.log(`\n Document Metadata:`); console.log(` Pages: ${content.content.document.pageCount}`); console.log(` Words: ${content.content.document.wordCount}`); console.log(` Author: ${content.content.document.author || 'Unknown'}`); } else if (content.content.fileType === FileTypes.Image && content.content.image) { console.log(`\nšŸ–¼ Image Metadata:`); console.log(` Size: ${content.content.image.width}x${content.content.image.height}`); console.log(` Camera: ${content.content.image.make} ${content.content.image.model}`); } break; } // Common properties console.log(`\nFile: ${content.content.fileName || 'N/A'}`); console.log(`Size: ${content.content.fileSize || 0} bytes`); console.log(`MIME: ${content.content.mimeType || 'N/A'}`); } ``` # Publish Content as Audio ## User Intent "How do I convert text to speech? Show me audio generation." ## Operation **SDK Method**: `publishText()`\ **Use Case**: Text-to-speech conversion with ElevenLabs or OpenAI *** ## Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentPublishingServiceTypes, ContentPublishingFormats, ElevenLabsModels, TextTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Generate audio from text const result = await graphlit.publishText( 'Hello, this is a text-to-speech demo using Graphlit.', // text TextTypes.Plain, // textType { type: ContentPublishingServiceTypes.ElevenLabsAudio, format: ContentPublishingFormats.Mp3, elevenLabs: { model: ElevenLabsModels.FlashV2_5, voice: 'HqW11As4VRPkApNPkAZp' // ElevenLabs voice ID } }, // connector 'Generated Audio', // name (optional) undefined, // workflow (optional) true // isSynchronous ); const contentId = result.publishText?.contents?.[0]?.id; // Retrieve the content to get audio URL const content = await graphlit.getContent(contentId!); console.log('Audio URL:', content.content?.uri); ``` *** ## Voice Options **ElevenLabs**: Use voice IDs from ElevenLabs (higher quality)\ **OpenAI TTS**: Use OpenAI voice names (alloy, echo, fable, onyx, nova, shimmer) *** # Publish Content Summary ## User Intent "How do I generate summaries of documents? Show me content summarization." ## Operation **SDK Method**: `summarizeContents()`\ **Use Case**: Automatic document summarization with multiple strategies *** ## Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { SummarizationTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Summarize content by ID const result = await graphlit.summarizeContents( [ { type: SummarizationTypes.Bullets, tokens: 500, items: 10 } ], { id: 'content-id' // Filter to specific content } ); const summary = result.summarizeContents?.[0]; console.log('Summary:', summary?.items?.[0]?.text); ``` *** ## Summarization Types **BULLETS**: Bullet point summary\ **CHAPTERS**: Chapter-by-chapter breakdown\ **HEADLINES**: Key headlines\ **QUESTIONS**: Q\&A format\ **TOPICS**: Topic extraction *** # Query Performance Patterns ## Content: Query Performance Patterns ### User Intent "How do I optimize content queries for production?" ### Operation * **SDK Method**: `queryContents()` * **GraphQL**: `queryContents` query * **Entity Type**: Content * **Common Use Cases**: Performance optimization, pagination, large result sets, production patterns ### Performance Overview Query performance depends on query type, filters used, result size, and indexing. Understanding these patterns helps optimize for production. ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentTypes, ObservableTypes, SearchTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Efficient pagination const page1 = await graphlit.queryContents({ search: "query", limit: 50, offset: 0 }); const page2 = await graphlit.queryContents({ search: "query", limit: 50, offset: 50 }); // Use indexed filters for speed const fast = await graphlit.queryContents({ search: "machine learning", types: [ContentTypes.File], // Indexed creationDateRange: { from: '2024-01-01' }, // Indexed collections: [{ id: 'collection-id' }] // Indexed limit: 20 // Reasonable limit }); // Narrow scope before querying const scoped = await graphlit.queryContents({ types: [ContentTypes.Email], feeds: [{ id: 'specific-feed' }], // Narrow to one feed createdInLast: 'P7D' // Recent only }); ``` ### Query Performance Characteristics #### Fast Queries (<50ms) ```typescript // 1. Keyword search with indexed filters const fast1 = await graphlit.queryContents({ search: "PROJ-1234", searchType: SearchTypes.Keyword, types: [ContentTypes.File], creationDateRange: { from: '2024-01-01' } limit: 20 }); // 2. Metadata-only filter (no search) const fast2 = await graphlit.queryContents({ types: [ContentTypes.Email], feeds: [{ id: 'feed-id' }], createdInLast: 'P7D' }); // 3. Get by ID (single result) const fast3 = await graphlit.getContent('content-id'); ``` #### Medium Queries (50-200ms) ```typescript // 1. Vector search const medium1 = await graphlit.queryContents({ search: "machine learning applications", searchType: SearchTypes.Vector, limit: 20 }); // 2. Hybrid search (default) const medium2 = await graphlit.queryContents({ search: "AI research", searchType: SearchTypes.Hybrid, limit: 20 }); // 3. Large result sets const medium3 = await graphlit.queryContents({ types: [ContentTypes.File] limit: 100 }); ``` #### Slow Queries (200ms+) ```typescript // 1. Entity filters (graph queries) const slow1 = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Person, observable: { id: 'person-id' } }] }); // 2. No filters (full scan) const slow2 = await graphlit.queryContents({ limit: 1000 // Large limit }); // 3. Complex multi-entity filters const slow3 = await graphlit.queryContents({ observations: [ { type: ObservableTypes.Person, observable: { id: 'p1' } }, { type: ObservableTypes.Person, observable: { id: 'p2' } } ] }); ``` ### Pagination Best Practices #### Offset-Based Pagination ```typescript // Page size const pageSize = 50; // Get page function async function getPage(pageNumber: number) { return await graphlit.queryContents({ search: "query", limit: pageSize, offset: pageNumber * pageSize }); } // Usage const page1 = await getPage(0); // First page const page2 = await getPage(1); // Second page const page3 = await getPage(2); // Third page ``` #### Progressive Loading ```typescript // Load results progressively async function loadAllResults(query: string) { const pageSize = 50; let offset = 0; let allResults: any[] = []; let hasMore = true; while (hasMore) { const page = await graphlit.queryContents({ search: query, limit: pageSize, offset: offset }); allResults = allResults.concat(page.contents.results); offset += pageSize; // Stop if we got fewer results than page size hasMore = page.contents.results.length === pageSize; console.log(`Loaded ${allResults.length} results so far...`); } return allResults; } ``` #### Limit Best Practices ```typescript // āœ“ Good limits limit: 10 // UI: Quick preview limit: 20 // UI: Search results page limit: 50 // API: Reasonable batch limit: 100 // API: Large batch (use with caution) // āœ— Avoid limit: 1000 // Too large, slow queries limit: 10000 // Will timeout ``` ### Caching Strategies #### Client-Side Result Caching ```typescript // Simple in-memory cache const queryCache = new Map(); async function cachedQuery(query: string, ttlMs: number = 60000) { const cacheKey = `query:${query}`; const cached = queryCache.get(cacheKey); if (cached && Date.now() - cached.timestamp < ttlMs) { console.log('Cache hit'); return cached.results; } console.log('Cache miss'); const results = await graphlit.queryContents({ search: query }); queryCache.set(cacheKey, { results, timestamp: Date.now() }); return results; } // Usage const results = await cachedQuery("machine learning", 60000); // 1 minute TTL ``` #### Query Fingerprinting ```typescript // Cache by query fingerprint function getQueryFingerprint(params: any): string { return JSON.stringify({ search: params.search, filter: params.filter, searchType: params.searchType, limit: params.limit, offset: params.offset }); } const cache = new Map(); async function cachedQueryByFingerprint(params: any, ttlMs: number = 60000) { const fingerprint = getQueryFingerprint(params); const cached = cache.get(fingerprint); if (cached && Date.now() - cached.timestamp < ttlMs) { return cached.results; } const results = await graphlit.queryContents(params); cache.set(fingerprint, { results, timestamp: Date.now() }); return results; } ``` ### Indexed vs Non-Indexed Fields #### Indexed (Fast) ```typescript // These filters are indexed and fast: const fast = await graphlit.queryContents({ types: [...], // āœ“ Indexed fileTypes: [...], // āœ“ Indexed creationDateRange: {...}, // āœ“ Indexed dateRange: {...}, // āœ“ Indexed feeds: [...], // āœ“ Indexed collections: [...], // āœ“ Indexed workflows: [...], // āœ“ Indexed states: [...], // āœ“ Indexed fileExtensions: [...], // āœ“ Indexed formats: [...], // āœ“ Indexed fileSizeRange: {...} // āœ“ Indexed }); ``` #### Non-Indexed (Slower) ```typescript // Entity filters require graph database lookup: const slower = await graphlit.queryContents({ observations: [...] // āœ— Graph query (slower) }); // Custom metadata not indexed: // Can't filter directly on custom metadata fields // Must query all and filter client-side ``` ### Optimization Patterns #### Pattern 1: Filter Before Search ```typescript // āœ“ Good: Narrow scope with filters first const optimized = await graphlit.queryContents({ search: "query", collections: [{ id: 'small-collection' }], // Reduce search space createdInLast: 'P30D' // Recent only }); // āœ— Less optimal: Search everything const slow = await graphlit.queryContents({ search: "query" // No filters = searches all content }); ``` #### Pattern 2: Use Appropriate Search Type ```typescript // For exact matching: use keyword (fastest) const exact = await graphlit.queryContents({ search: "PROJ-1234", searchType: SearchTypes.Keyword // Fast }); // For concepts: use vector const semantic = await graphlit.queryContents({ search: "machine learning applications", searchType: SearchTypes.Vector }); // For general queries: use hybrid (default, balanced) const balanced = await graphlit.queryContents({ search: "AI research papers" // Hybrid is default }); ``` #### Pattern 3: Batch Operations ```typescript // Get multiple content items efficiently async function batchGetContent(ids: string[]) { // Parallel requests const results = await Promise.all( ids.map(id => graphlit.getContent(id)) ); return results; } // Better than sequential // for (const id of ids) { // await graphlit.getContent(id); // Slow! // } ``` #### Pattern 4: Count vs Results ```typescript // If you only need count, don't fetch full results const results = await graphlit.queryContents({ types: [ContentTypes.Email] limit: 1 // Minimal fetch }); const count = results.contents.results.length; // Use count for display, not actual results ``` ## Pagination page1 = await graphlit.queryContents( search="query", limit=50, offset=0 ) page2 = await graphlit.queryContents( search="query", limit=50, offset=50 ) ## Optimized query fast = await graphlit.queryContents( search="machine learning", filter=ContentFilterInput( types=\[ContentTypes.File], creation\_date\_range=DateRangeInput( from\_='2024-01-01' ) ), limit=20 ) ```` **C#**: ```csharp using Graphlit; var client = new Graphlit(); // Pagination var page1 = await graphlit.QueryContents(new ContentFilter { Search = "query", Limit = 50, Offset = 0 }); var page2 = await graphlit.QueryContents(new ContentFilter { Search = "query", Limit = 50, Offset = 50 }); // Optimized query var fast = await graphlit.QueryContents(new ContentFilter { Search = "machine learning", Filter = new ContentCriteria { Types = new[] { ContentTypes.File }, CreationDateRange = new DateRange { From = "2024-01-01" } }, Limit = 20 }); ```` ### Developer Hints #### Measure Query Performance ```typescript async function measureQuery(queryFn: () => Promise) { const start = Date.now(); const results = await queryFn(); const elapsed = Date.now() - start; console.log(`Query time: ${elapsed}ms`); console.log(`Results: ${results.contents.results.length}`); console.log(`Ms per result: ${(elapsed / results.contents.results.length).toFixed(2)}`); return results; } // Usage await measureQuery(() => graphlit.queryContents({ search: "query" })); ``` #### Parallelize Independent Queries ```typescript // āœ“ Parallel (fast) const [emails, messages, files] = await Promise.all([ graphlit.queryContents({ types: [ContentTypes.Email] }), graphlit.queryContents({ types: [ContentTypes.Message] }), graphlit.queryContents({ types: [ContentTypes.File] }) ]); // āœ— Sequential (slow) const emails = await graphlit.queryContents({ types: [ContentTypes.Email] }); const messages = await graphlit.queryContents({ types: [ContentTypes.Message] }); const files = await graphlit.queryContents({ types: [ContentTypes.File] }); ``` #### Use Reasonable Limits ```typescript // āœ“ Good: Reasonable page size const results = await graphlit.queryContents({ search: "query", limit: 50 // Good balance }); // āœ— Bad: Too large const huge = await graphlit.queryContents({ search: "query", limit: 1000 // Will be slow }); ``` ### Common Issues & Solutions **Issue**: Queries timing out **Solution**: Reduce scope with filters and limits ```typescript // āœ— Too broad await graphlit.queryContents({ limit: 1000 }); // āœ“ Narrowed await graphlit.queryContents({ collections: [{ id: 'collection-id' }], createdInLast: 'P7D' limit: 50 }); ``` **Issue**: Slow entity-based queries **Solution**: Expected behavior, optimize where possible ```typescript // Entity queries are slower (graph lookup) // Optimize by combining with other filters const optimized = await graphlit.queryContents({ types: [ContentTypes.Email], // Narrow first createdInLast: 'P30D', // Then narrow more observations: [{ // Then entity filter type: ObservableTypes.Person, observable: { id: 'person-id' } }] }); ``` **Issue**: Need all results but hitting limits **Solution**: Use pagination ```typescript async function getAllResults(filter: any) { const pageSize = 50; let allResults: any[] = []; let offset = 0; let hasMore = true; while (hasMore) { const page = await graphlit.queryContents({ filter, limit: pageSize, offset }); allResults.push(...page.contents.results); hasMore = page.contents.results.length === pageSize; offset += pageSize; } return allResults; } ``` ### Production Example ```typescript class ContentQueryService { private cache = new Map(); constructor(private client: Graphlit) {} // Optimized query with caching async query(params: { search?: string; filter?: any; limit?: number; offset?: number; useCache?: boolean; cacheTTL?: number; }) { const { search, filter, limit = 20, offset = 0, useCache = true, cacheTTL = 60000 } = params; // Generate cache key const cacheKey = JSON.stringify({ search, filter, limit, offset }); // Check cache if (useCache) { const cached = this.cache.get(cacheKey); if (cached && Date.now() - cached.timestamp < cacheTTL) { console.log('āœ“ Cache hit'); return cached.data; } } // Measure performance const start = Date.now(); // Execute query const results = await this.graphlit.queryContents({ search, filter, limit, offset }); const elapsed = Date.now() - start; console.log(`Query: ${elapsed}ms, Results: ${results.contents.results.length}`); // Cache results if (useCache) { this.cache.set(cacheKey, { data: results, timestamp: Date.now() }); } return results; } // Paginated query async queryPaginated(params: { search?: string; filter?: any; pageSize?: number; }) { const { search, filter, pageSize = 50 } = params; let page = 0; return { async next() { const results = await this.query({ search, filter, limit: pageSize, offset: page * pageSize }); page++; return { results: results.contents.results, hasMore: results.contents.results.length === pageSize }; } }; } // Clear cache clearCache() { this.cache.clear(); } } // Usage const service = new ContentQueryService(client); // Cached query const results = await service.query({ search: "machine learning", filter: { types: [ContentTypes.File] }, useCache: true, cacheTTL: 300000 // 5 minutes }); // Paginated query const paginator = await service.queryPaginated({ search: "AI research", pageSize: 50 }); let page1 = await paginator.next(); let page2 = await paginator.next(); ``` # Query Similar Content ## Content: Query Similar Content ### User Intent "I want to find content similar to a specific document" ### Operation * **SDK Method**: `graphlit.queryContents()` with content reference * **GraphQL**: `queryContents` query with content filter * **Entity Type**: Content * **Common Use Cases**: Find related documents, similarity search, content recommendations ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { FileTypes, SearchTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Find content similar to a specific document const sourceContentId = 'content-id-here'; const similarContent = await graphlit.queryContents({ contents: [{ id: sourceContentId }], searchType: SearchTypes.Vector, limit: 10 }); console.log(`Found ${similarContent.contents.results.length} similar documents:\n`); similarContent.contents.results.forEach((content, index) => { console.log(`${index + 1}. ${content.name}`); if (content.summary) { console.log(` ${content.summary.substring(0, 100)}...`); } }); ``` ## Find similar content (snake\_case) similar\_content = await graphlit.queryContents( filter=ContentFilterInput( contents=\[EntityReferenceFilterInput(id=source\_content\_id)], search\_type=SearchTypes.Vector, limit=10 ) ) print(f"Found {len(similar\_content.contents.results)} similar documents") for idx, content in enumerate(similar\_content.contents.results, 1): print(f"{idx}. {content.name}") ```` **C#**: ```csharp using Graphlit; var client = new Graphlit(); var sourceContentId = "content-id-here"; // Find similar content (PascalCase) var similarContent = await graphlit.QueryContents(new ContentFilter { Contents = new[] { new EntityReferenceFilter { Id = sourceContentId } }, SearchType = SearchVector, Limit = 10 }); Console.WriteLine($"Found {similarContent.Contents.Results.Count} similar documents"); foreach (var (content, index) in similarContent.Contents.Results.Select((c, i) => (c, i))) { Console.WriteLine($"{index + 1}. {content.Name}"); } ```` ### Parameters #### ContentFilter * **`contents`** (EntityReferenceFilter\[]): Source content for similarity * **`searchType`** (SearchTypes): Must be `VECTOR` for similarity * **`limit`** (int): Max results to return (default: 100) * **`collections`** (EntityReferenceFilter\[]): Filter by collection (optional) ### Response ```typescript { contents: { results: Content[]; // Similar content, ordered by similarity } } ``` ### Developer Hints #### Vector Search for Similarity **Important**: Use `searchType: Vector` for semantic similarity. ```typescript // CORRECT - Vector search for similarity const similar = await graphlit.queryContents({ contents: [{ id: sourceContentId }], searchType: SearchTypes.Vector }); // WRONG - Keyword search won't find similar content const wrong = await graphlit.queryContents({ contents: [{ id: sourceContentId }], searchType: SearchTypes.Keyword }); ``` #### Exclude Source Document ```typescript // Find similar but exclude the source const similar = await graphlit.queryContents({ contents: [{ id: sourceContentId }], searchType: SearchTypes.Vector, limit: 11 // Get 11 results }); // Filter out source document const filtered = similar.contents.results.filter( c => c.id !== sourceContentId ).slice(0, 10); console.log(`${filtered.length} similar documents (excluding source)`); ``` #### Filter Similar Content ```typescript // Find similar PDFs only const similarPdfs = await graphlit.queryContents({ contents: [{ id: sourceContentId }], searchType: SearchTypes.Vector, fileTypes: [FileTypes.Pdf], limit: 10 }); // Find similar in specific collection const similarInCollection = await graphlit.queryContents({ contents: [{ id: sourceContentId }], searchType: SearchTypes.Vector, collections: [{ id: collectionId }], limit: 10 }); ``` ### Variations #### 1. Basic Similarity Search Find top 10 similar documents: ```typescript const similar = await graphlit.queryContents({ contents: [{ id: sourceContentId }], searchType: SearchTypes.Vector, limit: 10 }); ``` #### 2. Similar Content in Collection Limit to specific collection: ```typescript const similar = await graphlit.queryContents({ contents: [{ id: sourceContentId }], searchType: SearchTypes.Vector, collections: [{ id: collectionId }], limit: 10 }); ``` #### 3. Similar by File Type Find similar files of same type: ```typescript const similar = await graphlit.queryContents({ contents: [{ id: sourceContentId }], searchType: SearchTypes.Vector, fileTypes: [FileTypes.Pdf], limit: 10 }); ``` #### 4. Related Documents Widget Build "Related Articles" feature: ```typescript async function getRelatedDocuments(contentId: string, limit: number = 5) { const results = await graphlit.queryContents({ contents: [{ id: contentId }], searchType: SearchTypes.Vector, limit: limit + 1 }); // Exclude source return results.contents.results .filter(c => c.id !== contentId) .slice(0, limit); } // Usage const related = await getRelatedDocuments('article-123', 5); console.log('Related Articles:'); related.forEach(doc => console.log(`- ${doc.name}`)); ``` #### 5. Duplicate Detection Find near-duplicates: ```typescript const similar = await graphlit.queryContents({ contents: [{ id: sourceContentId }], searchType: SearchTypes.Vector, limit: 5 }); // Check for high similarity (potential duplicates) similar.contents.results.forEach(content => { if (content.id !== sourceContentId) { console.log(`Potential duplicate: ${content.name}`); } }); ``` #### 6. Content Clustering Group similar content: ```typescript async function findContentClusters(contentIds: string[]) { const clusters: Record = {}; for (const id of contentIds) { const similar = await graphlit.queryContents({ contents: [{ id }], searchType: SearchTypes.Vector, limit: 5 }); clusters[id] = similar.contents.results .map(c => c.id) .filter(cid => cid !== id); } return clusters; } ``` ### Common Issues **Issue**: Source document appears in results\ **Solution**: Filter out source document from results manually. **Issue**: No similar content found\ **Solution**: Ensure content has been embedded. Check embedding specification was used during ingestion. **Issue**: Results not semantically similar\ **Solution**: Verify `searchType: Vector` is set. Use better embedding model (text-embedding-3-large). ### Production Example **Related content recommendation**: ```typescript async function getRecommendations(articleId: string) { const similar = await graphlit.queryContents({ contents: [{ id: articleId }], searchType: SearchTypes.Vector, limit: 6 }); const recommendations = similar.contents.results .filter(c => c.id !== articleId) .slice(0, 5); return recommendations.map(doc => ({ id: doc.id, title: doc.name, summary: doc.summary?.substring(0, 150), uri: doc.uri })); } const recs = await getRecommendations('article-123'); console.log('You might also like:', recs); ``` # Query with Filters ## User Intent "I want to search and filter my ingested content by various criteria" ## Operation * **SDK Method**: `graphlit.queryContents()` * **GraphQL**: `queryContents` query * **Entity Type**: Content * **Common Use Cases**: Semantic search, filter by type/date/collection, faceted search, similarity search ## TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentTypes, EntityState, FileTypes, ObservableTypes, SearchTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Basic search - returns all content const response = await graphlit.queryContents({}); console.log(`Found ${response.contents.results.length} content items`); // Text search with semantic similarity const searchResponse = await graphlit.queryContents({ search: 'machine learning best practices', searchType: SearchTypes.Hybrid, // Combines keyword + vector search limit: 10 }); console.log(`Search found ${searchResponse.contents.results.length} results`); // Filter by content type const pdfResponse = await graphlit.queryContents({ types: [ContentTypes.File], fileTypes: [FileTypes.Document], fileExtensions: ['pdf'], limit: 20 }); console.log(`Found ${pdfResponse.contents.results.length} PDF files`); ``` ## Parameters ### ContentFilter Options **Search**: * **`search`** (string): Search query text * Uses semantic similarity (embeddings) * Supports natural language queries * **`searchType`** (SearchTypes): Search strategy * `KEYWORD` - Traditional keyword matching * `VECTOR` - Semantic/embedding search * `HYBRID` - Combined keyword + vector (recommended) **Content Type Filters**: * **`types`** (ContentTypes\[]): Filter by content type * `FILE`, `PAGE`, `TEXT`, `MEMORY`, `EMAIL`, `MESSAGE`, `POST`, `ISSUE`, `EVENT` * **`fileTypes`** (FileTypes\[]): Filter by file type (when type = FILE) * `PDF`, `DOCX`, `IMAGE`, `AUDIO`, `VIDEO`, `MARKDOWN`, etc. * **`textTypes`** (TextTypes\[]): Filter text content * `PLAIN`, `MARKDOWN` **Temporal Filters**: * **`creationDateRange`** (DateRangeFilter): Filter by creation date * `from` (Date): Start date * `to` (Date): End date * **`modifiedDateRange`** (DateRangeFilter): Filter by modification date **Organization Filters**: * **`collections`** (EntityReferenceFilter\[]): Filter by collection membership * **`feeds`** (EntityReferenceFilter\[]): Filter by source feed * **`workflows`** (EntityReferenceFilter\[]): Filter by workflow used **State Filters**: * **`states`** (EntityState\[]): Filter by processing state * `Enabled`, `Disabled`, `Finished`, `Errored` **Advanced Filters**: * **`similarContents`** (EntityReferenceInput\[]): Find similar content * **`observations`** (ObservationFilter): Filter by extracted entities **Pagination & Sorting**: * **`offset`** (number): Skip first N results (default: 0) * **`limit`** (number): Max results to return (default: 10, max: 100) * **`orderBy`** (OrderByTypes): Sort field * `RELEVANCE` - By search relevance score (when search is used) * `CREATION_DATE` - By creation date * `NAME` - Alphabetically by name * **`direction`** (OrderDirectionTypes): Sort direction * `Ascending` - Ascending * `Descending` - Descending ## Response ```typescript { contents: { results: Content[]; // Array of content items // Each content has: // - id, name, state, type, fileType, mimeType // - markdown (extracted text) // - uri (source URL) // - creationDate, modifiedDate // - collections, feed, workflow // - observations (extracted entities) // - relevance (search score, 0.0-1.0) } } ``` ## Developer Hints ### Search Type Selection **When to use each search type**: | Search Type | Best For | How It Works | | ----------- | -------------------------------------------- | ------------------------- | | `KEYWORD` | Exact term matching, technical terms | Traditional text matching | | `VECTOR` | Semantic/conceptual search, natural language | Embedding similarity | | `HYBRID` | Most searches (recommended) | Combines both approaches | ```typescript // Use KEYWORD for exact terms const keywordResults = await graphlit.queryContents({ search: 'GPT-4', searchType: SearchTypes.Keyword }); // Use VECTOR for conceptual search const vectorResults = await graphlit.queryContents({ search: 'articles about artificial intelligence', searchType: SearchTypes.Vector }); // Use HYBRID for best results (default) const hybridResults = await graphlit.queryContents({ search: 'machine learning tutorials', searchType: SearchTypes.Hybrid }); ``` ### OrderBy Behavior ```typescript // When using search, ALWAYS use orderBy: RELEVANCE const searchResults = await graphlit.queryContents({ search: 'product documentation', orderBy: OrderByTypes.Relevance // Sorts by search score }); // When NOT searching, use CREATION_DATE or NAME const recentContent = await graphlit.queryContents({ orderBy: OrderByTypes.CreationDate, direction: OrderDirectionTypes.Descending, // Newest first limit: 10 }); ``` ### Understanding Relevance Scores ```typescript // Results include relevance scores when searching const results = await graphlit.queryContents({ search: 'project requirements', searchType: SearchTypes.Hybrid }); results.contents.results.forEach(content => { console.log(`${content.name}: ${content.relevance?.toFixed(2)} relevance`); // Relevance is 0.0 to 1.0 (higher = more relevant) }); ``` ### Pagination Best Practices ```typescript // Fetch first page let offset = 0; const limit = 20; const page1 = await graphlit.queryContents({ search: 'meeting notes', offset: 0, limit: limit }); // Fetch second page const page2 = await graphlit.queryContents({ search: 'meeting notes', offset: limit, // offset = 20 limit: limit }); // Continue until results.length < limit ``` ## Variations ### 1. Filter by Date Range Find content created in a specific time period: ```typescript const lastWeek = new Date(); lastWeek.setDate(lastWeek.getDate() - 7); const response = await graphlit.queryContents({ creationDateRange: { from: lastWeek, to: new Date() }, orderBy: OrderByTypes.CreationDate, direction: OrderDirectionTypes.Descending }); console.log(`Content created in last week: ${response.contents.results.length}`); ``` ### 2. Filter by Collection Search within specific collections: ```typescript // Get collection ID first const collections = await graphlit.queryCollections({ name: 'Product Docs' }); const collectionId = collections.collections.results[0]?.id; if (collectionId) { const response = await graphlit.queryContents({ collections: [{ id: collectionId }], search: 'API reference', searchType: SearchTypes.Hybrid }); console.log(`Found ${response.contents.results.length} results in Product Docs`); } ``` ### 3. Filter by Multiple Content Types Search across specific content types: ```typescript const response = await graphlit.queryContents({ types: [ ContentTypes.Email, ContentTypes.Message, ContentTypes.Post ], search: 'project update', searchType: SearchTypes.Hybrid, limit: 50 }); console.log(`Found communications about project update: ${response.contents.results.length}`); ``` ### 4. Similarity Search Find content similar to existing content: ```typescript // First, get a content ID const originalContent = await graphlit.queryContents({ search: 'machine learning tutorial', limit: 1 }); const contentId = originalContent.contents.results[0]?.id; if (contentId) { // Find similar content const similarResults = await graphlit.queryContents({ similarContents: [{ id: contentId }], searchType: SearchTypes.Vector, // Must use Vector or Hybrid limit: 10 }); console.log(`Found ${similarResults.contents.results.length} similar documents`); } ``` ### 5. Filter by State Find content in specific processing states: ```typescript // Find content that failed processing const errorContent = await graphlit.queryContents({ states: [EntityState.Errored], orderBy: OrderByTypes.CreationDate, direction: OrderDirectionTypes.Descending }); console.log(`Content with errors: ${errorContent.contents.results.length}`); // Find content still processing const processingContent = await graphlit.queryContents({ states: [EntityState.Created] }); console.log(`Content awaiting processing: ${processingContent.contents.results.length}`); ``` ### 6. Filter by Feed Source Find content from specific feeds: ```typescript // Get feed ID const feeds = await graphlit.queryFeeds(); const slackFeedId = feeds.feeds.results.find(f => f.name.includes('Slack'))?.id; if (slackFeedId) { const response = await graphlit.queryContents({ feeds: [{ id: slackFeedId }], creationDateRange: { from: new Date('2025-01-01'), to: new Date() } }); console.log(`Slack messages in 2025: ${response.contents.results.length}`); } ``` ### 7. Complex Filter Combination Combine multiple filters: ```typescript const response = await graphlit.queryContents({ // Text search search: 'quarterly results', searchType: SearchTypes.Hybrid, // Content type types: [ContentTypes.File], fileTypes: [FileTypes.Document], fileExtensions: ['pdf', 'docx'], // Date range creationDateRange: { from: new Date('2024-01-01'), to: new Date('2024-12-31') }, // Pagination offset: 0, limit: 50, // Sorting orderBy: OrderByTypes.Relevance }); console.log(`Found ${response.contents.results.length} relevant documents`); ``` ### 8. Filter by Extracted Entities Find content mentioning specific entities: ```typescript // Find content mentioning a person const response = await graphlit.queryContents({ observations: { observable: { types: [ObservableTypes.Person], states: [EntityState.Enabled] }, name: 'John Smith' }, limit: 100 }); console.log(`Content mentioning John Smith: ${response.contents.results.length}`); ``` ## Common Issues **Issue**: Search returns no results even though content exists\ **Solution**: Check that content has finished processing (`state: Finished`). Embeddings must be generated for vector search. **Issue**: Relevance scores seem low (< 0.5)\ **Solution**: This is normal. Relevance is relative. Use `HYBRID` search type for better results. **Issue**: `orderBy: RELEVANCE` but results not sorted by relevance\ **Solution**: Relevance ordering only works when `search` parameter is provided. Without search, use `CREATION_DATE` or `NAME`. **Issue**: Getting duplicate results across pages\ **Solution**: Ensure consistent `orderBy` and `direction` across pagination requests. Content may shift if new items are added during pagination. **Issue**: Pagination returns fewer results than `limit`\ **Solution**: You've reached the end of results. This is expected behavior. ## Production Example **Parallel query with count**: ```typescript // Query results and total count in parallel const [response, countResponse] = await Promise.all([ graphlit.queryContents(filter), graphlit.countContents({ ...filter, offset: 0, limit: 1000000 // Count all matching }) ]); const results = response.contents?.results || []; const totalCount = countResponse.countContents?.count || undefined; ``` # Hybrid Search Deep Dive ## Content: Hybrid Search Deep Dive ### User Intent "What is hybrid search and why is it the default?" ### Operation * **SDK Method**: `queryContents()` with `searchType: SearchTypes.Hybrid` (default) * **GraphQL**: `queryContents` query * **Common Use Cases**: Production search, best results, general-purpose queries ### What is Hybrid Search? Hybrid search combines **vector search** (semantic) and **keyword search** (exact matching) using Reciprocal Rank Fusion (RRF) to get the best of both worlds. **Why it's the default**: It handles diverse query types better than either approach alone, with minimal downside. ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentTypes, EntityState, FileTypes, ObservableTypes, SearchTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Hybrid search (default - can omit searchType) const results = await graphlit.queryContents({ search: "machine learning applications in healthcare" }); // Explicit hybrid search const explicitHybrid = await graphlit.queryContents({ search: "machine learning applications in healthcare", searchType: SearchTypes.Hybrid // Default value }); console.log(`Found ${results.contents.results.length} results`); results.contents.results.forEach((content, index) => { console.log(`\n${index + 1}. ${content.name}`); console.log(` Relevance: ${(content.relevance * 100).toFixed(1)}%`); console.log(` Type: ${content.type}`); }); ``` ### How Hybrid Search Works #### The RRF Algorithm **RRF = Reciprocal Rank Fusion** ``` For each result: RRF_score = Ī£ (1 / (k + rank_i)) Where: k = 60 (constant) rank_i = position in result list i ``` **Example**: ```typescript // Query: "machine learning" // Vector search results (semantic): 1. "ML Applications" (rank 1) 2. "AI Algorithms" (rank 2) 3. "Deep Learning Guide" (rank 3) // Keyword search results (exact match): 1. "Machine Learning Basics" (rank 1) 2. "ML Applications" (rank 2) // Also in vector! 3. "Learn Machine Learning" (rank 3) // RRF scoring: "ML Applications": Vector: 1/(60+1) = 0.0164 Keyword: 1/(60+2) = 0.0161 Combined: 0.0325 (highest!) "Machine Learning Basics": Vector: not in top results = 0 Keyword: 1/(60+1) = 0.0164 Combined: 0.0164 "AI Algorithms": Vector: 1/(60+2) = 0.0161 Keyword: not in top results = 0 Combined: 0.0161 // Final ranking: 1. "ML Applications" (0.0325) - appears in BOTH 2. "Machine Learning Basics" (0.0164) 3. "AI Algorithms" (0.0161) ``` #### Pipeline ``` User Query: "machine learning" ā†“ Split into TWO parallel searches: ā”œā”€ Vector Search (semantic) ā”‚ ā†“ ā”‚ Query ā†’ Embedding ā”‚ ā†“ ā”‚ Cosine similarity ā”‚ ā†“ ā”‚ Ranked results A ā”‚ ā””ā”€ Keyword Search (exact) ā†“ Token matching ā†“ BM25 ranking ā†“ Ranked results B ā†“ RRF Fusion (merge A + B) ā†“ Final ranked results ``` ### Why Hybrid is Best #### 1. Handles Diverse Queries ```typescript // Conceptual query (vector helps) await graphlit.queryContents({ search: "reducing carbon emissions" // Finds: "climate change mitigation", "lowering CO2", etc. }); // Exact phrase (keyword helps) await graphlit.queryContents({ search: "Project Alpha" // Finds: Exact "Project Alpha" mentions }); // Mixed query (both help) await graphlit.queryContents({ search: "Kirk Marple discussing AI safety" // Keyword: "Kirk Marple" (exact name) // Vector: "AI safety" concepts }); ``` #### 2. Better Precision ```typescript // Vector alone might be too broad // Keyword alone might miss synonyms // Hybrid: Precise + comprehensive const hybrid = await graphlit.queryContents({ search: "natural language processing" }); // Results include: // āœ“ "NLP" (keyword matches abbreviation) // āœ“ "text understanding" (vector finds concept) // āœ“ "natural language processing" (both rank highest) ``` #### 3. Robust to Query Types ```typescript // Works well for: // - Short queries: "AI" // - Long queries: "How does machine learning improve healthcare outcomes?" // - Names: "Kirk Marple" // - Concepts: "context layer" // - Mixed: "Kirk's context layer for AI agents" // Single search type that handles everything ``` ### Comparison: Vector vs Keyword vs Hybrid ```typescript const query = "AI safety research"; // Vector search (semantic) const vector = await graphlit.queryContents({ search: query, searchType: SearchTypes.Vector }); // Finds: "artificial intelligence safety", "AI alignment", // "machine learning ethics", "safe AI systems" // Misses: Exact phrase "AI safety research" might rank lower // Keyword search (exact) const keyword = await graphlit.queryContents({ search: query, searchType: SearchTypes.Keyword }); // Finds: "AI safety research", "AI safety", "research on AI" // Misses: "artificial intelligence safety", "ML safety" // Hybrid search (both) const hybrid = await graphlit.queryContents({ search: query, searchType: SearchTypes.Hybrid }); // Finds: All of the above // Ranks: Exact "AI safety research" highest (appears in both) // Then semantic matches and keyword matches ``` ### Sample Results Comparison **Query**: "machine learning tutorial" | Rank | Vector | Keyword | Hybrid | | ---- | ----------------------- | ---------------------------- | ----------------------------- | | 1 | "Deep Learning Guide" | "Machine Learning Tutorial" | "Machine Learning Tutorial" āœ“ | | 2 | "Neural Network Basics" | "ML Tutorial 2024" | "Deep Learning Guide" | | 3 | "AI Fundamentals" | "Tutorial: Machine Learning" | "ML Tutorial 2024" | | 4 | "ML Concepts" | "Machine Learning Intro" | "Neural Network Basics" | | 5 | "Understanding AI" | "Learn ML" | "Tutorial: Machine Learning" | **Winner**: Hybrid (exact match ranks #1, semantically similar also included) ## Hybrid search (default) results = await graphlit.queryContents( search="machine learning applications" ) ## Explicit hybrid hybrid = await graphlit.queryContents( search="machine learning applications", search\_type=SearchTypes.Hybrid ) for content in results.contents.results: print(f"{content.name} - {content.relevance:.3f}") ```` **C#**: ```csharp using Graphlit; var client = new Graphlit(); // Hybrid search (default) var results = await graphlit.QueryContents(new ContentFilter { Search = "machine learning applications" }); // Explicit hybrid var hybrid = await graphlit.QueryContents(new ContentFilter { Search = "machine learning applications", SearchType = SearchHybrid }); foreach (var content in results.Contents.Results) { Console.WriteLine($"{content.Name} - {content.Relevance:F3}"); } ```` ### Developer Hints #### Default for Good Reason ```typescript // These are equivalent: const results1 = await graphlit.queryContents({ search: "query" }); const results2 = await graphlit.queryContents({ search: "query", searchType: SearchTypes.Hybrid }); // Hybrid is default because it works best for 90% of queries ``` #### No Tuning Parameters ```typescript // RRF algorithm is parameter-free // k=60 is hardcoded (industry standard) // No knobs to turn // Just works // This is a FEATURE not a limitation // Prevents over-optimization and parameter tuning hell ``` #### When NOT to Use Hybrid ```typescript // Rare cases where you want only one approach: // 1. Only semantic matching (ignore exact terms) const onlySemantic = await graphlit.queryContents({ search: "climate change", searchType: SearchTypes.Vector }); // 2. Only exact matching (ignore semantics) const onlyExact = await graphlit.queryContents({ search: "PROJ-1234", searchType: SearchTypes.Keyword }); // But for 90%+ of queries: use Hybrid (default) ``` #### Performance ```typescript // Hybrid is slightly slower than pure keyword // (runs both searches) // But only ~10-20ms difference // And quality improvement is worth it const start = Date.now(); const results = await graphlit.queryContents({ search: "query", searchType: SearchTypes.Hybrid }); console.log(`Time: ${Date.now() - start}ms`); // Typically: 50-100ms (vs 20-50ms for keyword only) ``` ### Variations #### 1. Basic Hybrid Search (Default) ```typescript const results = await graphlit.queryContents({ search: "AI applications in healthcare" }); ``` #### 2. Hybrid with Filters ```typescript const filtered = await graphlit.queryContents({ search: "machine learning", types: [ContentTypes.File], fileTypes: [FileTypes.Document], creationDateRange: { from: '2024-01-01' } }); ``` #### 3. Hybrid with Collection Filter ```typescript const inCollection = await graphlit.queryContents({ search: "product roadmap", collections: [ { id: 'engineering-docs' }, { id: 'product-docs' } ] }); ``` #### 4. Hybrid Search Pagination ```typescript // Page 1 const page1 = await graphlit.queryContents({ search: "query", limit: 20, offset: 0 }); // Page 2 const page2 = await graphlit.queryContents({ search: "query", limit: 20, offset: 20 }); ``` #### 5. Compare Hybrid vs Pure Approaches ```typescript const query = "machine learning"; const [hybrid, vector, keyword] = await Promise.all([ graphlit.queryContents({ search: query, searchType: SearchTypes.Hybrid }), graphlit.queryContents({ search: query, searchType: SearchTypes.Vector }), graphlit.queryContents({ search: query, searchType: SearchTypes.Keyword }) ]); console.log('Hybrid results:', hybrid.contents.results.length); console.log('Vector results:', vector.contents.results.length); console.log('Keyword results:', keyword.contents.results.length); // Compare top result console.log('\nTop result by search type:'); console.log('Hybrid:', hybrid.contents.results[0]?.name); console.log('Vector:', vector.contents.results[0]?.name); console.log('Keyword:', keyword.contents.results[0]?.name); ``` #### 6. Hybrid with Entity Filter ```typescript // Combines all three: vector, keyword, and graph const entitySearch = await graphlit.queryContents({ search: "project status", observations: [{ type: ObservableTypes.Person, observable: { id: 'person-id' } }] }); ``` ### Common Issues & Solutions **Issue**: Results not relevant enough **Solution**: Hybrid is usually best, but check query quality ```typescript // Too vague await graphlit.queryContents({ search: "docs" }); // More specific await graphlit.queryContents({ search: "API documentation for authentication" }); // Add filters await graphlit.queryContents({ search: "authentication", collections: [{ id: 'api-docs' }] }); ``` **Issue**: Want pure semantic search **Solution**: Override with Vector search type ```typescript const semantic = await graphlit.queryContents({ search: "climate change solutions", searchType: SearchTypes.Vector // Override hybrid }); ``` **Issue**: Want pure exact matching **Solution**: Override with Keyword search type ```typescript const exact = await graphlit.queryContents({ search: "PROJ-1234", searchType: SearchTypes.Keyword // Override hybrid }); ``` **Issue**: Queries slower than expected **Solution**: Hybrid runs both searches (small overhead acceptable) ```typescript // If speed critical and only need exact matching: const fast = await graphlit.queryContents({ search: "query", searchType: SearchTypes.Keyword // Faster }); // But for best results: stick with hybrid (default) ``` ### Production Example ```typescript async function productionSearch(query: string) { console.log(`\n=== PRODUCTION SEARCH ===`); console.log(`Query: "${query}"`); console.log(`Using: Hybrid search (RRF)`); const startTime = Date.now(); // Hybrid search with sensible defaults const results = await graphlit.queryContents({ search: query, // searchType: SearchTypes.Hybrid (default, can omit) limit: 20, states: [EntityState.Enabled] // Only active content }); const elapsed = Date.now() - startTime; console.log(`\n Results: ${results.contents.results.length} in ${elapsed}ms`); // Analyze relevance distribution const relevanceGroups = { excellent: results.contents.results.filter(c => c.relevance >= 0.8).length, good: results.contents.results.filter(c => c.relevance >= 0.6 && c.relevance < 0.8).length, fair: results.contents.results.filter(c => c.relevance >= 0.4 && c.relevance < 0.6).length, poor: results.contents.results.filter(c => c.relevance < 0.4).length }; console.log('\nšŸ“ˆ Relevance Distribution:'); console.log(` Excellent (ā‰„80%): ${relevanceGroups.excellent}`); console.log(` Good (60-80%): ${relevanceGroups.good}`); console.log(` Fair (40-60%): ${relevanceGroups.fair}`); console.log(` Poor (<40%): ${relevanceGroups.poor}`); // Group by content type const byType = results.contents.results.reduce((acc, content) => { acc[content.type] = (acc[content.type] || 0) + 1; return acc; }, {} as Record); console.log('\n Results by Type:'); Object.entries(byType).forEach(([type, count]) => { console.log(` ${type}: ${count}`); }); // Top 5 results console.log('\nšŸ† Top 5 Results:'); results.contents.results.slice(0, 5).forEach((content, index) => { console.log(`\n${index + 1}. ${content.name}`); console.log(` Relevance: ${(content.relevance * 100).toFixed(1)}%`); console.log(` Type: ${content.type}`); console.log(` Created: ${new Date(content.creationDate).toLocaleDateString()}`); }); // Performance analysis console.log(`\nāš” Performance:`); console.log(` Query time: ${elapsed}ms`); console.log(` Avg per result: ${(elapsed / results.contents.results.length).toFixed(2)}ms`); return results; } // Usage await productionSearch("machine learning applications"); await productionSearch("Kirk Marple AI research"); await productionSearch("PROJ-1234 status report"); ``` ### Sample Reference `Graphlit_2024_09_13_Compare_RAG_strategies.ipynb` - Compares search strategies including hybrid # Keyword Search Explained ## Content: Keyword Search Explained ### User Intent "How does keyword/full-text search work in Graphlit?" ### Operation * **SDK Method**: `queryContents()` with `searchType: SearchKeyword` * **GraphQL**: `queryContents` query * **Common Use Cases**: Exact phrase matching, names, IDs, codes, fast lookups ### How Keyword Search Works Keyword search uses traditional full-text indexing with token-based matching. It's fast, precise, and ideal for exact phrases, names, and identifiers. ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentTypes, SearchTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Pure keyword search const results = await graphlit.queryContents({ search: "Project Alpha status report", searchType: SearchTypes.Keyword, limit: 10 }); console.log(`Found ${results.contents.results.length} keyword matches`); results.contents.results.forEach((content, index) => { console.log(`\n${index + 1}. ${content.name}`); console.log(` Relevance: ${content.relevance}`); console.log(` Type: ${content.type}`); }); // Search for exact phrase const exactPhrase = await graphlit.queryContents({ search: '"quarterly earnings report"', // Quotes = exact phrase searchType: SearchTypes.Keyword }); // Search for ID or code const byId = await graphlit.queryContents({ search: "PROJ-1234", searchType: SearchTypes.Keyword }); // Search for email address const byEmail = await graphlit.queryContents({ search: "kirk@graphlit.com", searchType: SearchTypes.Keyword }); // Search for specific name const byName = await graphlit.queryContents({ search: "Kirk Marple", searchType: SearchTypes.Keyword }); ``` ### Keyword Search Features #### 1. Token-Based Matching ```typescript // Search tokens: "machine", "learning", "tutorial" const results = await graphlit.queryContents({ search: "machine learning tutorial", searchType: SearchTypes.Keyword }); // Matches documents containing these tokens // Order doesn't matter (by default) // Stemming applied: "learning" matches "learn", "learned", "learns" ``` #### 2. Exact Phrase Search ```typescript // Use quotes for exact phrase const exact = await graphlit.queryContents({ search: '"machine learning tutorial"', // Exact order required searchType: SearchTypes.Keyword }); // Only matches documents with this exact phrase ``` #### 3. Boolean Operators (if supported) ```typescript // AND operator (both terms required) const andSearch = await graphlit.queryContents({ search: "machine AND learning", searchType: SearchTypes.Keyword }); // OR operator (either term) const orSearch = await graphlit.queryContents({ search: "machine OR learning", searchType: SearchTypes.Keyword }); // NOT operator (exclude term) const notSearch = await graphlit.queryContents({ search: "machine NOT learning", searchType: SearchTypes.Keyword }); ``` #### 4. Case-Insensitive ```typescript // All equivalent: await graphlit.queryContents({ search: "graphlit", searchType: SearchTypes.Keyword }); await graphlit.queryContents({ search: "Graphlit", searchType: SearchTypes.Keyword }); await graphlit.queryContents({ search: "GRAPHLIT", searchType: SearchTypes.Keyword }); // All match "Graphlit", "graphlit", "GRAPHLIT" ``` #### 5. Stemming ```typescript // Search for "running" const results = await graphlit.queryContents({ search: "running", searchType: SearchTypes.Keyword }); // Also matches: "run", "runs", "ran", "runner" // English stemming rules applied ``` ### When to Use Keyword Search \*\* Good For\*\*: * Exact phrases ("Project Alpha v2.3") * Names ("Kirk Marple", "Graphlit") * Email addresses ("") * IDs and codes ("PROJ-1234", "ORDER-5678") * URLs ("") * Specific terminology * Fast lookups \*\* Not Good For\*\*: * Conceptual queries (use vector search) * Synonyms and paraphrases (use vector search) * Semantic similarity (use vector search) * "Find similar" queries (use vector search) ### Performance Characteristics ```typescript // Keyword search is FAST: // - Inverted index lookup // - Token-based (no vector computation) // - Sub-100ms queries typical // - Scales to billions of documents const startTime = Date.now(); const results = await graphlit.queryContents({ search: "PROJ-1234", searchType: SearchTypes.Keyword }); console.log(`Query time: ${Date.now() - startTime}ms`); // Typically: 20-50ms ``` ### BM25 Ranking **Algorithm**: BM25 (Best Matching 25) * Standard information retrieval algorithm * Considers term frequency and document length * More sophisticated than simple TF-IDF **Relevance Score**: ```typescript const results = await graphlit.queryContents({ search: "machine learning", searchType: SearchTypes.Keyword }); results.contents.results.forEach(content => { console.log(`${content.name}: ${content.relevance}`); // Higher score = more relevant // Multiple occurrences of terms increase score // Shorter documents rank higher (BM25 length normalization) }); ``` ## Keyword search (snake\_case) results = await graphlit.client.query\_contents( search="Project Alpha", search\_type=SearchTypes.Keyword, limit=10 ) for content in results.contents.results: print(f"{content.name} - Relevance: {content.relevance}") ## Exact phrase exact = await graphlit.client.query\_contents( search='"quarterly report"', search\_type=SearchTypes.Keyword ) ## Search by ID by\_id = await graphlit.client.query\_contents( search="PROJ-1234", search\_type=SearchTypes.Keyword ) ```` **C#**: ```csharp using Graphlit; var client = new Graphlit(); // Keyword search (PascalCase) var results = await graphlit.QueryContents(new ContentFilter { Search = "Project Alpha", SearchType = SearchKeyword, Limit = 10 }); foreach (var content in results.Contents.Results) { Console.WriteLine($"{content.Name} - Relevance: {content.Relevance}"); } // Exact phrase var exact = await graphlit.QueryContents(new ContentFilter { Search = "\"quarterly report\"", SearchType = SearchKeyword }); // Search by ID var byId = await graphlit.QueryContents(new ContentFilter { Search = "PROJ-1234", SearchType = SearchKeyword }); ```` ### Developer Hints #### Exact Phrase vs Token Search ```typescript // Token search (default): const tokens = await graphlit.queryContents({ search: "machine learning", // No quotes searchType: SearchTypes.Keyword }); // Matches: "machine learning", "learning about machines", "machine for learning" // Exact phrase: const exact = await graphlit.queryContents({ search: '"machine learning"', // With quotes searchType: SearchTypes.Keyword }); // Matches: Only "machine learning" in that exact order ``` #### Special Characters ```typescript // Email addresses work as-is const email = await graphlit.queryContents({ search: "kirk@graphlit.com", searchType: SearchTypes.Keyword }); // URLs work (may need quotes for special chars) const url = await graphlit.queryContents({ search: "https://graphlit.com", searchType: SearchTypes.Keyword }); // IDs with hyphens work const id = await graphlit.queryContents({ search: "PROJ-1234", searchType: SearchTypes.Keyword }); ``` #### Combining with Filters ```typescript // Keyword search + metadata filters const filtered = await graphlit.queryContents({ search: "Project Alpha", searchType: SearchTypes.Keyword, types: [ContentTypes.Email], creationDateRange: { from: '2024-01-01' } }); // Fast and precise ``` ### Variations #### 1. Basic Keyword Search ```typescript const results = await graphlit.queryContents({ search: "Graphlit platform", searchType: SearchTypes.Keyword }); ``` #### 2. Exact Phrase Search ```typescript const exact = await graphlit.queryContents({ search: '"context layer for AI"', searchType: SearchTypes.Keyword }); ``` #### 3. Search by ID ```typescript const byId = await graphlit.queryContents({ search: "PROJ-1234", searchType: SearchTypes.Keyword }); ``` #### 4. Search by Email ```typescript const byEmail = await graphlit.queryContents({ search: "kirk@graphlit.com", searchType: SearchTypes.Keyword }); ``` #### 5. Search with Content Type Filter ```typescript const emailsOnly = await graphlit.queryContents({ search: "Project Alpha", searchType: SearchTypes.Keyword, types: [ContentTypes.Email] }); ``` #### 6. Search in Specific Collection ```typescript const inCollection = await graphlit.queryContents({ search: "meeting notes", searchType: SearchTypes.Keyword, collections: [{ id: 'team-docs-collection' }] }); ``` #### 7. Multi-Term Search ```typescript // All terms required (implicit AND) const multiTerm = await graphlit.queryContents({ search: "quarterly report Q4 2024", searchType: SearchTypes.Keyword }); ``` ### Common Issues & Solutions **Issue**: No results for partial words **Solution**: Keyword search doesn't do prefix matching by default ```typescript // Won't match "Graphlit" await graphlit.queryContents({ search: "Graph", searchType: SearchTypes.Keyword }); // Use full word await graphlit.queryContents({ search: "Graphlit", searchType: SearchTypes.Keyword }); // Or use vector search for fuzzy matching await graphlit.queryContents({ search: "Graph", searchType: SearchTypes.Vector }); ``` **Issue**: Too many results **Solution**: Use exact phrase or add filters ```typescript // Too broad await graphlit.queryContents({ search: "report", searchType: SearchTypes.Keyword }); // More specific await graphlit.queryContents({ search: '"quarterly earnings report"', searchType: SearchTypes.Keyword }); // Add date filter await graphlit.queryContents({ search: "report", searchType: SearchTypes.Keyword, creationDateRange: { from: '2024-01-01' } }); ``` **Issue**: Want semantic + keyword **Solution**: Use hybrid search (combines both) ```typescript // āœ“ Best of both worlds await graphlit.queryContents({ search: "Project Alpha status", searchType: SearchTypes.Hybrid // Recommended }); ``` **Issue**: Special characters breaking search **Solution**: Use quotes for exact phrase ```typescript // āœ“ Escape with quotes await graphlit.queryContents({ search: '"C++ programming guide"', searchType: SearchTypes.Keyword }); ``` ### Production Example ```typescript async function keywordSearch(query: string) { console.log(`\n=== KEYWORD SEARCH ===`); console.log(`Query: "${query}"`); const startTime = Date.now(); const results = await graphlit.queryContents({ search: query, searchType: SearchTypes.Keyword, limit: 20 }); const elapsed = Date.now() - startTime; console.log(`\nFound ${results.contents.results.length} results in ${elapsed}ms`); // Group by content type const byType = new Map(); results.contents.results.forEach(content => { byType.set(content.type, (byType.get(content.type) || 0) + 1); }); console.log(`\n Results by Type:`); byType.forEach((count, type) => { console.log(` ${type}: ${count}`); }); // Top results console.log(`\nšŸ” Top 5 Results:`); results.contents.results.slice(0, 5).forEach((content, index) => { console.log(`\n${index + 1}. ${content.name}`); console.log(` Type: ${content.type}`); console.log(` Relevance: ${(content.relevance * 100).toFixed(1)}%`); console.log(` Created: ${new Date(content.creationDate).toLocaleDateString()}`); // Show snippet if available if (content.markdown) { const snippet = content.markdown .split('\n') .find(line => line.toLowerCase().includes(query.toLowerCase())); if (snippet) { console.log(` Snippet: "${snippet.trim().substring(0, 100)}..."`); } } }); // Performance analysis console.log(`\nāš” Performance:`); console.log(` Query time: ${elapsed}ms`); console.log(` Results per ms: ${(results.contents.results.length / elapsed).toFixed(2)}`); } // Usage examples await keywordSearch("Project Alpha"); await keywordSearch("kirk@graphlit.com"); await keywordSearch("PROJ-1234"); await keywordSearch('"quarterly earnings report"'); ``` # Vector Search Explained ## Content: Vector Search Explained ### User Intent "How does semantic/vector search work in Graphlit?" ### Operation * **SDK Method**: `queryContents()` with `searchType: SearchVector` * **GraphQL**: `queryContents` query * **Common Use Cases**: Semantic similarity, concept search, "find similar documents" ### How Vector Search Works Vector search finds content based on semantic meaning, not just keyword matches. Content is converted to high-dimensional vectors (embeddings), and similarity is measured by distance between vectors. ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentTypes, FileTypes, ModelServiceTypes, SearchTypes, SpecificationTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Pure vector search const results = await graphlit.queryContents({ search: "machine learning research papers", searchType: SearchTypes.Vector, limit: 10 }); console.log(`Found ${results.contents.results.length} semantically similar results`); results.contents.results.forEach((content, index) => { console.log(`\n${index + 1}. ${content.name}`); console.log(` Relevance: ${content.relevance}`); console.log(` Type: ${content.type}`); // Show matching text chunk if (content.pages && content.pages.length > 0) { const topChunk = content.pages[0].chunks?.[0]; if (topChunk) { console.log(` Match: "${topChunk.text.substring(0, 100)}..."`); } } }); // Find similar documents to a specific one const similar = await graphlit.queryContents({ similarContents: [{ id: 'content-id' }], limit: 5 }); console.log(`\nDocuments similar to content-id:`); similar.contents.results.forEach(content => { console.log(`- ${content.name} (relevance: ${content.relevance})`); }); ``` ### The Vector Search Pipeline #### 1. Ingestion Time (Embedding Creation) ```typescript // When content is ingested: const content = await graphlit.ingestUri( 'https://example.com/document.pdf', undefined, undefined, undefined, undefined, { id: 'workflow-id' } ); // Behind the scenes: // 1. Document is chunked (default: 512 tokens per chunk) // 2. Each chunk ā†’ embedding model (e.g., text-embedding-3-large) // 3. Model returns 3072-dimensional vector per chunk // 4. Vectors stored in Azure AI Search // 5. Ready for similarity search ``` #### 2. Query Time (Semantic Search) ```typescript // User searches: const results = await graphlit.queryContents({ search: "climate change impact on agriculture", searchType: SearchTypes.Vector }); // Behind the scenes: // 1. Query text ā†’ embedding model (same model as ingestion) // 2. Get query vector (3072 dimensions) // 3. Cosine similarity against all content vectors // 4. Return top-K most similar chunks // 5. Aggregate by parent content // 6. Rank and return results ``` ### When to Use Vector Search \*\* Good For\*\*: * Conceptual queries ("AI safety concerns") * Semantic similarity ("find similar documents") * Cross-language concepts * Synonyms and paraphrases * Question answering * Vague or exploratory queries \*\* Not Good For\*\*: * Exact phrases ("Project Alpha v2.3") * Names and IDs ("PROJ-1234") * Codes and identifiers * Very short queries (< 3 words) * Spelling-sensitive searches ### Examples: Vector vs Keyword ```typescript // Vector search finds semantic meaning const vector = await graphlit.queryContents({ search: "reducing carbon emissions", searchType: SearchTypes.Vector }); // Matches: "lowering CO2 output", "decreasing greenhouse gases", // "climate change mitigation", "sustainability efforts" // Keyword search finds exact tokens const keyword = await graphlit.queryContents({ search: "reducing carbon emissions", searchType: SearchTypes.Keyword }); // Matches: Only documents with words "reducing", "carbon", "emissions" ``` ### Embedding Models **Current Default**: OpenAI `text-embedding-3-large` * **Dimensions**: 3072 * **Quality**: Highest * **Cost**: Higher * **Use**: Best results, production recommended **Alternative**: OpenAI `text-embedding-3-small` * **Dimensions**: 1536 * **Quality**: Good * **Cost**: Lower * **Use**: Cost-sensitive applications **Legacy**: OpenAI `text-embedding-ada-002` * **Dimensions**: 1536 * **Quality**: Baseline * **Cost**: Lower * **Use**: Not recommended for new projects **Configure Embedding Model**: ```typescript // Create embedding specification const embeddingSpec = await graphlit.createSpecification({ name: "High Quality Embeddings", type: SpecificationTypes.TextEmbedding, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Embedding_3Large, chunkTokenLimit: 512 // Tokens per chunk } }); // Assign as the project default text embedding strategy await graphlit.updateProject({ embeddings: { textSpecification: { id: embeddingSpec.createSpecification.id }, }, }); ``` ### Similarity Scoring **Relevance Score** (`content.relevance`): * Range: 0.0 (no match) to 1.0 (perfect match) * Based on cosine similarity * Higher = more semantically similar * Typically use threshold (e.g., > 0.7) ```typescript const results = await graphlit.queryContents({ search: "query", searchType: SearchTypes.Vector }); // Filter by relevance const highRelevance = results.contents.results.filter( content => content.relevance > 0.7 ); console.log(`High relevance matches: ${highRelevance.length}`); ``` ## Vector search (snake\_case) results = await graphlit.client.query\_contents( search="machine learning research", search\_type=SearchTypes.Vector, limit=10 ) for content in results.contents.results: print(f"{content.name} - Relevance: {content.relevance}") ## Find similar documents similar = await graphlit.client.query\_contents( filter=ContentFilterInput( similar\_contents=\[ EntityReferenceInput(id='content-id') ] ) ) ```` **C#**: ```csharp using Graphlit; var client = new Graphlit(); // Vector search (PascalCase) var results = await graphlit.QueryContents(new ContentFilter { Search = "machine learning research", SearchType = SearchVector, Limit = 10 }); foreach (var content in results.Contents.Results) { Console.WriteLine($"{content.Name} - Relevance: {content.Relevance}"); } // Find similar documents var similar = await graphlit.QueryContents(new ContentFilter { SimilarContents = new[] { new EntityReference { Id = "content-id" } } }); ```` ### Developer Hints #### Vector Search is Expensive ```typescript // Each query: // 1. Embeds query text (LLM API call) // 2. Compares against millions of vectors // 3. Aggregates and ranks results // Cost factors: // - Embedding API calls // - Vector index size // - Query frequency // Optimization: // - Cache query embeddings for common queries // - Use hybrid search (better results, similar cost) // - Limit result count ``` #### Chunk Size Matters ```typescript // Small chunks (256 tokens): // - More precise matching // - More chunks = more embeddings = higher cost // - Better for specific queries // Large chunks (1024 tokens): // - More context per chunk // - Fewer chunks = lower cost // - Better for broad queries // Default (512 tokens): // - Balanced approach // - Recommended for most use cases ``` #### Query Quality Tips ```typescript // āœ“ Good queries (natural language) "How does machine learning improve healthcare?" "Impact of remote work on productivity" "Best practices for API security" // āœ— Poor queries (too short/vague) "AI" "docs" "help" // āœ“ Better versions "Artificial intelligence applications" "Documentation about features" "Help with authentication setup" ``` ### Variations #### 1. Basic Vector Search ```typescript const results = await graphlit.queryContents({ search: "quantum computing applications", searchType: SearchTypes.Vector }); ``` #### 2. Vector Search with Filters ```typescript // Combine semantic search with metadata filters const filtered = await graphlit.queryContents({ search: "machine learning", searchType: SearchTypes.Vector, types: [ContentTypes.File], fileTypes: [FileTypes.Document], creationDateRange: { from: '2024-01-01' } }); ``` #### 3. Find Similar Documents ```typescript // "More like this" functionality const similar = await graphlit.queryContents({ similarContents: [{ id: 'original-content-id' }] limit: 10 }); ``` #### 4. Vector Search with Relevance Threshold ```typescript const results = await graphlit.queryContents({ search: "query", searchType: SearchTypes.Vector, limit: 50 }); // Client-side filtering by relevance const relevant = results.contents.results.filter( content => content.relevance >= 0.75 ); ``` #### 5. Multi-Collection Vector Search ```typescript // Search across specific collections const results = await graphlit.queryContents({ search: "product roadmap", searchType: SearchTypes.Vector, collections: [ { id: 'engineering-docs' }, { id: 'product-docs' } ] }); ``` #### 6. Vector Search Pagination ```typescript // First page const page1 = await graphlit.queryContents({ search: "query", searchType: SearchTypes.Vector, limit: 20, offset: 0 }); // Second page const page2 = await graphlit.queryContents({ search: "query", searchType: SearchTypes.Vector, limit: 20, offset: 20 }); ``` ### Common Issues & Solutions **Issue**: No results returned **Solution**: Query might be too specific or embeddings not created ```typescript // Check if content has embeddings const content = await graphlit.getContent('content-id'); if (!content.content.pages || content.content.pages.length === 0) { console.log('Content not embedded yet'); } // Try broader query const results = await graphlit.queryContents({ search: "machine learning", // Broader searchType: SearchTypes.Vector }); ``` **Issue**: Results not semantically relevant **Solution**: Try hybrid search or keyword search ```typescript // Hybrid often works better const results = await graphlit.queryContents({ search: "query", searchType: SearchTypes.Hybrid // Default and recommended }); ``` **Issue**: Want to change embedding model for existing content **Solution**: Must re-ingest content with new specification ```typescript // Create new specification const newSpec = await graphlit.createSpecification({ type: SpecificationTypes.TextEmbedding, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Embedding_3Small // Smaller, cheaper } }); await graphlit.updateProject({ embeddings: { textSpecification: { id: newSpec.createSpecification.id }, }, }); // Re-ingest content with new spec // (embeddings are immutable once created) ``` **Issue**: Slow query performance **Solution**: Vector search is computationally expensive ```typescript // Optimizations: // 1. Use filters to reduce search space const results = await graphlit.queryContents({ search: "query", searchType: SearchTypes.Vector, collections: [{ id: 'small-collection' }] // Narrow scope }); // 2. Reduce limit const results = await graphlit.queryContents({ search: "query", searchType: SearchTypes.Vector, limit: 10 // Fewer results = faster }); // 3. Consider hybrid search (often faster) const results = await graphlit.queryContents({ search: "query", searchType: SearchTypes.Hybrid }); ``` ### Production Example ```typescript async function semanticSearch(query: string) { console.log(`\n=== SEMANTIC SEARCH ===`); console.log(`Query: "${query}"`); const startTime = Date.now(); const results = await graphlit.queryContents({ search: query, searchType: SearchTypes.Vector, limit: 10 }); const elapsed = Date.now() - startTime; console.log(`\nFound ${results.contents.results.length} results in ${elapsed}ms`); results.contents.results.forEach((content, index) => { console.log(`\n${index + 1}. ${content.name}`); console.log(` Relevance: ${(content.relevance * 100).toFixed(1)}%`); console.log(` Type: ${content.type}`); console.log(` Created: ${new Date(content.creationDate).toLocaleDateString()}`); // Show best matching chunk if (content.pages && content.pages.length > 0) { const topPage = content.pages.sort((a, b) => (b.relevance || 0) - (a.relevance || 0) )[0]; if (topPage.chunks && topPage.chunks.length > 0) { const topChunk = topPage.chunks.sort((a, b) => (b.relevance || 0) - (a.relevance || 0) )[0]; console.log(` Matching text: "${topChunk.text.substring(0, 150)}..."`); } } }); // Relevance distribution const highRelevance = results.contents.results.filter(c => c.relevance >= 0.8).length; const mediumRelevance = results.contents.results.filter(c => c.relevance >= 0.6 && c.relevance < 0.8).length; const lowRelevance = results.contents.results.filter(c => c.relevance < 0.6).length; console.log(`\n Relevance Distribution:`); console.log(` High (ā‰„80%): ${highRelevance}`); console.log(` Medium (60-80%): ${mediumRelevance}`); console.log(` Low (<60%): ${lowRelevance}`); } // Usage await semanticSearch("impact of artificial intelligence on healthcare"); ``` # Advanced Content Search with Filters ## User Intent "How do I combine semantic search with metadata filters? Show me complex search queries." ## Operation **SDK Method**: `queryContents()` with comprehensive filters\ **Use Case**: Precision search with multiple criteria *** ## Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentTypes, SearchTypes, FileTypes, ObservableTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); const results = await graphlit.queryContents({ search: "machine learning optimization", searchType: SearchTypes.Hybrid, // Vector + keyword types: [ContentTypes.File], fileTypes: [FileTypes.Document], creationDateRange: { from: '2024-01-01', to: '2024-12-31' }, collections: [{ id: 'research-papers-id' }], feeds: [{ id: 'arxiv-feed-id' }], observations: [{ type: ObservableTypes.Person, observable: { id: 'author-id' } }], offset: 0, limit: 20 }); console.log(`Found ${results.contents.results.length} results`); ``` *** ## Filter Combinations **Content Type + Date**: Recent PDFs\ **Entity + Collection**: Person in project\ **Search + Feed**: Keywords in source\ **Multiple Types**: Flexible queries *** # Understanding ContentType vs FileType ## User Intent "What's the difference between contentType and fileType? When do I use each?" ## Operation * **Concept**: Content classification hierarchy * **GraphQL Fields**: `content.type` (ContentType), `content.fileType` (FileType) * **Entity Type**: Content * **Common Use Cases**: Filtering content, UI display logic, workflow routing, understanding content structure ## Key Concept: The Hierarchy **ContentType is PRIMARY** - Semantic classification of what the content represents: * `EMAIL` - Email messages * `MESSAGE` - Chat messages (Slack, Teams, Discord) * `PAGE` - Web pages * `FILE` - Files (documents, images, audio, video, code) * `POST` - Social posts (Reddit, RSS) * `EVENT` - Calendar events * `ISSUE` - Issue tracker items (Jira, Linear, GitHub) * `TEXT` - Plain text, markdown, HTML * `MEMORY` - Agent or user memory **FileType is SECONDARY** - Physical format (ONLY when contentType = FILE): * `DOCUMENT` - PDF, Word, Excel, PowerPoint * `IMAGE` - JPEG, PNG, GIF, TIFF * `AUDIO` - MP3, WAV, podcast files * `VIDEO` - MP4, MOV, AVI * `CODE` - Source code files * `DATA` - JSON, XML, CSV * `PACKAGE` - ZIP, TAR, archives * `ANIMATION`, `DRAWING`, `GEOMETRY`, `POINT_CLOUD`, `SHAPE`, etc. ## TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentTypes, FileTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Query by ContentType const emails = await graphlit.queryContents({ types: [ContentTypes.Email], }); emails.contents.results.forEach(content => { console.log(`Type: ${content.type}`); // EMAIL console.log(`FileType: ${content.fileType}`); // null (emails don't have fileType) // Access email-specific metadata if (content.type === ContentTypes.Email && content.email) { console.log(`From: ${content.email.from[0].email}`); console.log(`Subject: ${content.email.subject}`); } }); // Query by FileType (for files) const pdfs = await graphlit.queryContents({ types: [ContentTypes.File], fileTypes: [FileTypes.Document], }); pdfs.contents.results.forEach(content => { console.log(`Type: ${content.type}`); // FILE console.log(`FileType: ${content.fileType}`); // DOCUMENT // Access document-specific metadata if (content.document) { console.log(`Pages: ${content.document.pageCount}`); console.log(`Author: ${content.document.author}`); } }); // Query all images (regardless of source) const images = await graphlit.queryContents({ fileTypes: [FileTypes.Image], }); images.contents.results.forEach(content => { console.log(`Type: ${content.type}`); // Could be FILE, MESSAGE (if image in message), etc. console.log(`FileType: ${content.fileType}`); // IMAGE if (content.image) { console.log(`Dimensions: ${content.image.width}x${content.image.height}`); } }); ``` ## ContentType ā†’ Metadata Field Mapping **CRITICAL**: Each ContentType has a corresponding metadata field on the content object: | ContentType | Metadata Field | Type | When Set | | ----------------- | ---------------------------------- | ---------------- | ---------------------- | | `EMAIL` | `content.email` | EmailMetadata | Always for emails | | `MESSAGE` | `content.message` | MessageMetadata | Always for messages | | `EVENT` | `content.event` | EventMetadata | Always for events | | `ISSUE` | `content.issue` | IssueMetadata | Always for issues | | `POST` | `content.post` | PostMetadata | Always for posts | | `FILE` (Document) | `content.document` | DocumentMetadata | When fileType=DOCUMENT | | `FILE` (Image) | `content.image` | ImageMetadata | When fileType=IMAGE | | `FILE` (Audio) | `content.audio` | AudioMetadata | When fileType=AUDIO | | `FILE` (Video) | `content.video` | VideoMetadata | When fileType=VIDEO | | `FILE` (Package) | `content.package` | PackageMetadata | When fileType=PACKAGE | | `PAGE` | `content.html`, `content.markdown` | Strings | Always for pages | **Access Pattern**: ```typescript const content = await graphlit.getContent('content-id'); // Always check type first switch (content.content.type) { case ContentTypes.Email: // Safe to access content.email console.log(`Subject: ${content.content.email.subject}`); break; case ContentTypes.Message: // Safe to access content.message console.log(`Channel: ${content.content.message.channelName}`); break; case ContentTypes.File: // Check fileType for specific metadata if (content.content.fileType === FileTypes.Document) { console.log(`Pages: ${content.content.document.pageCount}`); } else if (content.content.fileType === FileTypes.Image) { console.log(`Size: ${content.content.image.width}x${content.content.image.height}`); } break; case ContentTypes.Event: // Safe to access content.event console.log(`Start: ${content.content.event.startDateTime}`); break; } ``` ## Developer Hints ### FileType Only Exists for Files ```typescript // WRONG - Emails don't have fileType const emails = await graphlit.queryContents({ types: [ContentTypes.Email], fileTypes: [FileTypes.Document] // This makes no sense! }); // CORRECT - FileType only for FILE content const pdfs = await graphlit.queryContents({ types: [ContentTypes.File], fileTypes: [FileTypes.Document], }); // ALSO CORRECT - FileType alone (implicit FILE contentType) const images = await graphlit.queryContents({ fileTypes: [FileTypes.Image], }); ``` ### Automatic Classification ```typescript // ContentType is automatically determined at ingestion: // - Email from Gmail feed ā†’ ContentTypes.Email // - Slack message ā†’ ContentTypes.Message // - PDF file ā†’ ContentTypes.File with FileDocument // - Web page ā†’ ContentTypes.Page // - GitHub issue ā†’ ContentTypes.Issue // You CANNOT override contentType // It's determined by the source ``` ### Query Flexibility ```typescript // Query by ContentType only const allEmails = await graphlit.queryContents({ types: [ContentTypes.Email], }); // Query by FileType only (searches all FILE content) const allImages = await graphlit.queryContents({ fileTypes: [FileTypes.Image], }); // Query by both (most specific) const pdfFiles = await graphlit.queryContents({ types: [ContentTypes.File], fileTypes: [FileTypes.Document], }); // Query multiple types (OR logic) const communications = await graphlit.queryContents({ types: [ContentTypes.Email, ContentTypes.Message], }); ``` ## Variations ### 1. Query All Communication Content ```typescript // Emails + Messages + Posts const communications = await graphlit.queryContents({ types: [ ContentTypes.Email, ContentTypes.Message, ContentTypes.Post, ], }); ``` ### 2. Query All Media Files ```typescript // Images + Audio + Video const media = await graphlit.queryContents({ types: [ContentTypes.File], fileTypes: [ FileTypes.Image, FileTypes.Audio, FileTypes.Video, ], }); ``` ### 3. Query Documents Only (No Other Files) ```typescript const documents = await graphlit.queryContents({ types: [ContentTypes.File], fileTypes: [FileTypes.Document], }); // This excludes images, audio, video, code, etc. ``` ### 4. Query Calendar Events ```typescript const events = await graphlit.queryContents({ types: [ContentTypes.Event], }); events.contents.results.forEach(content => { if (content.event) { console.log(`Event: ${content.event.subject}`); console.log(`When: ${content.event.startDateTime}`); console.log(`Attendees: ${content.event.attendees?.length || 0}`); } }); ``` ### 5. Query Issues from Project Management Tools ```typescript const issues = await graphlit.queryContents({ types: [ContentTypes.Issue], }); issues.contents.results.forEach(content => { if (content.issue) { console.log(`Issue: ${content.issue.title}`); console.log(`Status: ${content.issue.status}`); console.log(`Priority: ${content.issue.priority}`); } }); ``` ### 6. UI Display Logic by Type ```typescript const content = await graphlit.getContent('content-id'); // Render UI based on content type function renderContent(content: Content) { switch (content.type) { case ContentTypes.Email: return ; case ContentTypes.Message: return ; case ContentTypes.File: if (content.fileType === FileTypes.Document) { return ; } else if (content.fileType === FileTypes.Image) { return ; } break; case ContentTypes.Page: return ; case ContentTypes.Event: return ; } } ``` ### 7. Workflow Routing by Type ```typescript // Create different workflows for different content types const documentWorkflow = await graphlit.createWorkflow({ name: "Document Extraction", preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.Document } }] }, extraction: { /* ... */ } }); const audioWorkflow = await graphlit.createWorkflow({ name: "Audio Transcription", preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.Deepgram, fileTypes: [FileTypes.Audio, FileTypes.Video], deepgram: { model: DeepgramModels.Nova2 } } }] } }); // Route content to appropriate workflow const content = await graphlit.getContent('content-id'); if (content.content.fileType === FileTypes.Document) { // Re-process with document workflow } else if (content.content.fileType === FileTypes.Audio) { // Re-process with audio workflow } ``` ## Common Issues & Solutions **Issue**: Querying for emails by fileType returns no results ```typescript // WRONG const emails = await graphlit.queryContents({ types: [ContentTypes.Email], fileTypes: [FileTypes.Document] // Emails don't have fileType! }); ``` **Solution**: Only use fileType with FILE content ```typescript // CORRECT const emails = await graphlit.queryContents({ types: [ContentTypes.Email], }); ``` **Issue**: Metadata field is null even though content exists ```typescript const content = await graphlit.getContent('content-id'); console.log(content.content.email); // undefined ``` **Solution**: Check content type first ```typescript if (content.content.type === ContentTypes.Email) { // Now safe to access email metadata console.log(content.content.email.subject); } else { console.log(`Content is ${content.content.type}, not EMAIL`); } ``` **Issue**: Want to query "all files" but fileType filter is too specific ```typescript // TOO SPECIFIC - Only gets documents const files = await graphlit.queryContents({ fileTypes: [FileTypes.Document], }); ``` **Solution**: Use contentType=FILE without fileType filter ```typescript // CORRECT - Gets all files const allFiles = await graphlit.queryContents({ types: [ContentTypes.File], }); ``` **Issue**: TypeScript type errors when accessing metadata ```typescript // TypeScript error: Property 'email' does not exist console.log(content.content.email.subject); ``` **Solution**: Use type guards ```typescript if (content.content.type === ContentTypes.Email && content.content.email) { // TypeScript knows email exists console.log(content.content.email.subject); } ``` ## Production Example **Real-world pattern: Content type routing in UI**: ```typescript async function displayContent(contentId: string) { const content = await graphlit.getContent(contentId); console.log(`=== CONTENT DISPLAY ===`); console.log(`Type: ${content.content.type}`); console.log(`Name: ${content.content.name}`); switch (content.content.type) { case ContentTypes.Email: console.log(`\n EMAIL`); console.log(`From: ${content.content.email.from[0].email}`); console.log(`Subject: ${content.content.email.subject}`); console.log(`Date: ${content.content.creationDate}`); console.log(`Attachments: ${content.content.email.attachmentCount || 0}`); break; case ContentTypes.Message: console.log(`\nšŸ’¬ MESSAGE`); console.log(`Channel: ${content.content.message.channelName}`); console.log(`Author: ${content.content.message.author?.name}`); console.log(`Mentions: ${content.content.message.mentions?.length || 0}`); break; case ContentTypes.File: console.log(`\n FILE`); console.log(`FileType: ${content.content.fileType}`); console.log(`Size: ${content.content.fileSize} bytes`); if (content.content.fileType === FileTypes.Document) { console.log(`Pages: ${content.content.document?.pageCount}`); console.log(`Author: ${content.content.document?.author}`); } else if (content.content.fileType === FileTypes.Image) { console.log(`Dimensions: ${content.content.image?.width}x${content.content.image?.height}`); } break; case ContentTypes.Page: console.log(`\nšŸŒ WEB PAGE`); console.log(`URL: ${content.content.uri}`); break; case ContentTypes.Event: console.log(`\nšŸ“… EVENT`); console.log(`Subject: ${content.content.event?.subject}`); console.log(`Start: ${content.content.event?.startDateTime}`); console.log(`Attendees: ${content.content.event?.attendees?.length || 0}`); break; case ContentTypes.Issue: console.log(`\nšŸŽ« ISSUE`); console.log(`Title: ${content.content.issue?.title}`); console.log(`Status: ${content.content.issue?.status}`); console.log(`Priority: ${content.content.issue?.priority}`); break; } // Show extracted entities regardless of type if (content.content.observations?.length > 0) { console.log(`\nšŸ· ENTITIES (${content.content.observations.length}):`); content.content.observations.forEach(obs => { console.log(` ${obs.type}: ${obs.observable.name}`); }); } } // Usage await displayContent('email-content-id'); await displayContent('slack-message-id'); await displayContent('pdf-document-id'); ``` # Update Content Metadata ## User Intent "How do I update content metadata? Show me content updates." ## Operation **SDK Method**: `updateContent()`\ **Use Case**: Modify content properties *** ## Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; const graphlit = new Graphlit(); // Update content await graphlit.updateContent({ id: 'content-id', name: "Updated Document Name", customMetadata: { category: "financial-reports", quarter: "Q4-2024", reviewed: true } }); console.log('Content updated'); ``` *** ## Updatable Fields **name**: Display name\ **customMetadata**: App-specific JSON\ **collections**: Add/remove from collections\ **state**: Enable/disable/archive *** # Knowledge Graph Extract entities, build knowledge graphs, and query relationships from your content. *** ## Core Concepts * [Observable/Observation Model](/api-guides/use-cases/knowledge-graph/observable-observation-model-explained) - **Start here** - Entity architecture * [Entity Types Comprehensive](/api-guides/use-cases/knowledge-graph/observable-entity-types-comprehensive) - All 20+ types (12 medical included) * [Confidence and Occurrences](/api-guides/use-cases/knowledge-graph/observable-confidence-and-occurrences) - Timestamps, pages, bounding boxes * [Workflow Extraction Explained](/api-guides/use-cases/knowledge-graph/workflow-extraction-how-it-works) - How LLMs extract entities * [Configure Entity Extraction](/api-guides/use-cases/knowledge-graph/workflow-configure-entity-extraction) - Workflow setup *** ## End-to-End Examples Build complete knowledge graphs from various sources: * [From PDF Documents](/api-guides/use-cases/knowledge-graph/knowledge-graph-from-pdf-documents) - Most popular - Complete pipeline * [From Emails](/api-guides/use-cases/knowledge-graph/knowledge-graph-from-emails) - Gmail/Outlook ā†’ contact networks * [From Slack Messages](/api-guides/use-cases/knowledge-graph/knowledge-graph-from-slack-messages) - Team collaboration graphs * [From GitHub Repositories](/api-guides/use-cases/knowledge-graph/knowledge-graph-from-github) - Contributors, repos, dependencies * [From Meeting Recordings](/api-guides/use-cases/knowledge-graph/knowledge-graph-from-meetings) - Audio ā†’ transcript ā†’ entities * [From Medical Content](/api-guides/use-cases/knowledge-graph/knowledge-graph-medical-content) - Clinical entity extraction (12 types) *** ## Graph Queries * [Relationship Queries](/api-guides/use-cases/knowledge-graph/observable-relationship-queries) - Person ā†’ Organization patterns * [Advanced Graph Patterns](/api-guides/use-cases/knowledge-graph/observable-query-graph-patterns) - Subgraphs, centrality, paths * [Entity Deduplication](/api-guides/use-cases/knowledge-graph/observable-entity-deduplication) - How Graphlit handles duplicates * [Knowledge Graph Enrichment](/api-guides/use-cases/knowledge-graph/knowledge-graph-enrichment) - External data augmentation *** ## Content + KG Integration * [Filter Content by Entities](/api-guides/use-cases/knowledge-graph/content-filter-by-entities) - Find all content mentioning entity * [Entity Co-Occurrence](/api-guides/use-cases/knowledge-graph/content-entity-co-occurrence) - Network analysis * [KG-Guided Search](/api-guides/use-cases/knowledge-graph/knowledge-graph-guided-search) - Entity expansion * [Entity Timeline](/api-guides/use-cases/knowledge-graph/observable-content-timeline) - First/last mentions * [Export with Entities](/api-guides/use-cases/knowledge-graph/content-export-with-entities) - JSON/CSV export *** ## Related Tutorials * [Knowledge Graph Quickstart](/tutorials/knowledge-graph) - 20-minute complete tutorial *** **24 guides** | [ā† Back to Use Cases](/api-guides/use-cases) # Analyze Entity Co-Occurrence ## User Intent "How do I find content where multiple entities appear together? Show me entity co-occurrence patterns." ## Operation **SDK Methods**: `queryContents()` with multiple entity filters\ **GraphQL**: Multi-entity content queries\ **Use Case**: Relationship discovery through co-occurrence ## Prerequisites * Content with extracted entities * Multiple entities to analyze * Understanding of entity relationships *** ## Complete Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { ObservableTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Find entities const kirk = await graphlit.queryObservables({ search: "Kirk Marple", filter: { types: [ObservableTypes.Person] } }); const maria = await graphlit.queryObservables({ search: "Maria Garcia", filter: { types: [ObservableTypes.Person] } }); // Find co-occurrence const cooccurrence = await graphlit.queryContents({ observations: [ { observable: { id: kirk.observables.results[0].observable.id } }, { observable: { id: maria.observables.results[0].observable.id } } ] }); console.log(`Kirk and Maria mentioned together in ${cooccurrence.contents.results.length} documents`); // Build co-occurrence matrix const people = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Person] } }); const matrix = new Map>(); for (let i = 0; i < people.observables.results.length; i++) { for (let j = i + 1; j < people.observables.results.length; j++) { const personA = people.observables.results[i]; const personB = people.observables.results[j]; const together = await graphlit.queryContents({ observations: [ { observable: { id: personA.observable.id } }, { observable: { id: personB.observable.id } } ] }); const count = together.contents.results.length; if (count > 0) { if (!matrix.has(personA.observable.name)) { matrix.set(personA.observable.name, new Map()); } matrix.get(personA.observable.name)!.set(personB.observable.name, count); } } } // Display top co-occurrences console.log('\nTop co-occurrences:'); const flat: Array<{ a: string; b: string; count: number }> = []; matrix.forEach((bMap, a) => { bMap.forEach((count, b) => { flat.push({ a, b, count }); }); }); flat.sort((x, y) => y.count - x.count) .slice(0, 10) .forEach(({ a, b, count }) => { console.log(` ${a} ā†” ${b}: ${count} times`); }); ``` *** ## Key Patterns ### 1. Pairwise Co-occurrence Two specific entities: ```typescript observations: [ { observable: { id: entityA } }, { observable: { id: entityB } } ] ``` ### 2. Entity + Type Specific entity with any of type: ```typescript observations: [ { observable: { id: personId } }, { type: ObservableTypes.Organization } ] ``` ### 3. Co-occurrence Strength Frequency indicates relationship strength *** ## Use Cases **Team Collaboration Analysis**: Who works with whom\ **Influence Mapping**: Entity co-mention networks\ **Topic Association**: Products + Organizations\ **Relationship Discovery**: Find hidden connections *** ## Developer Hints * Co-occurrence implies relationship * Frequency = strength * Useful for network analysis * Export for graph visualization * Can be computationally expensive (cache results) *** # Export Content with Entity Annotations ## User Intent "How do I export content with entity annotations? Show me export formats with entity metadata." ## Operation **SDK Methods**: `getContent()`, `queryContents()` for export\ **Format**: JSON export with entity data\ **Use Case**: Data export with entity annotations ## Prerequisites * Content with extracted entities * Export destination (file, database) * Understanding of entity structure *** ## Complete Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import * as fs from 'fs'; const graphlit = new Graphlit(); // Get content with entities const content = await graphlit.getContent('content-id'); // Prepare export data const exportData = { id: content.content.id, name: content.content.name, type: content.content.type, creationDate: content.content.creationDate, text: content.content.markdown, entities: content.content.observations?.map(obs => ({ type: obs.type, name: obs.observable.name, id: obs.observable.id, occurrences: obs.occurrences?.map(occ => ({ confidence: occ.confidence, pageIndex: occ.pageIndex, boundingBox: occ.boundingBox, startTime: occ.startTime, endTime: occ.endTime })) })) || [] }; // Export as JSON fs.writeFileSync('export.json', JSON.stringify(exportData, null, 2)); console.log('Exported with entity annotations'); // Export as CSV (flattened) const csvRows = [ ['Entity Type', 'Entity Name', 'Confidence', 'Page', 'Document'].join(',') ]; content.content.observations?.forEach(obs => { obs.occurrences?.forEach(occ => { csvRows.push([ obs.type, obs.observable.name, occ.confidence.toFixed(2), occ.pageIndex?.toString() || '', content.content.name ].join(',')); }); }); fs.writeFileSync('export.csv', csvRows.join('\n')); console.log('Exported as CSV'); ``` *** ## Export Formats ### JSON (Full Fidelity) Complete entity data with occurrences ### CSV (Tabular) Flattened for spreadsheet analysis ### XML (Structured) Hierarchical entity annotations ### Database (Structured Storage) PostgreSQL, MongoDB for querying *** ## Use Cases **Data Analysis**: Export for external tools\ **Backup**: Archive with entity data\ **Integration**: Feed to other systems\ **Reporting**: Generate entity reports\ **Training Data**: ML model training *** ## Developer Hints * Include entity IDs for re-linking * Preserve occurrence data (timestamps, pages) * Export in batches for large datasets * Compress for large exports * Include metadata (dates, sources) *** # Filter Content by Entities ## User Intent "How do I find all content mentioning a specific person or organization? Show me entity-based content filtering." ## Operation **SDK Method**: `queryContents()` with `observations` filter\ **GraphQL Query**: `queryContents`\ **Use Case**: Entity-driven content discovery ## Prerequisites * Content with extracted entities * Knowledge of entity IDs or names * Understanding of Observable model *** ## Complete Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { ObservableTypes, ContentTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Example 1: Find all content mentioning a person async function findContentByPerson(personName: string) { // Step 1: Find the person entity const people = await graphlit.queryObservables({ search: personName, filter: { types: [ObservableTypes.Person] } }); if (people.observables.results.length === 0) { console.log(`No entity found for: ${personName}`); return; } const person = people.observables.results[0]; console.log(`Found entity: ${person.observable.name} (${person.observable.id})\n`); // Step 2: Find all content mentioning this person const content = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Person, observable: { id: person.observable.id } }] }); console.log(`Content mentioning ${person.observable.name}: ${content.contents.results.length}`); // Group by type const byType = new Map(); content.contents.results.forEach(item => { const type = item.type || 'UNKNOWN'; byType.set(type, (byType.get(type) || 0) + 1); }); console.log('\nBy content type:'); byType.forEach((count, type) => { console.log(` ${type}: ${count}`); }); return content.contents.results; } // Example 2: Find content mentioning multiple entities (AND logic) async function findContentWithMultipleEntities( personName: string, orgName: string ) { // Find entities const person = await graphlit.queryObservables({ search: personName, filter: { types: [ObservableTypes.Person] } }); const org = await graphlit.queryObservables({ search: orgName, filter: { types: [ObservableTypes.Organization] } }); if (person.observables.results.length === 0 || org.observables.results.length === 0) { console.log('One or both entities not found'); return; } // Find content mentioning BOTH const content = await graphlit.queryContents({ observations: [ { type: ObservableTypes.Person, observable: { id: person.observables.results[0].observable.id } }, { type: ObservableTypes.Organization, observable: { id: org.observables.results[0].observable.id } } ] }); console.log(`Content mentioning both ${personName} and ${orgName}: ${content.contents.results.length}`); return content.contents.results; } // Run examples await findContentByPerson("Kirk Marple"); console.log('\n---\n'); await findContentWithMultipleEntities("Kirk Marple", "Graphlit"); ``` *** ## Key Patterns ### Pattern 1: Single Entity Filter Find all content mentioning one entity: ```typescript const content = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Person, observable: { id: entityId } }] }); ``` ### Pattern 2: Multiple Entities (AND) Content mentioning ALL specified entities: ```typescript const content = await graphlit.queryContents({ observations: [ { observable: { id: entityId1 } }, { observable: { id: entityId2 } } ] }); ``` ### Pattern 3: Entity Type Filter All content with any entity of type: ```typescript const content = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Organization // No specific observable ID = any organization }] }); ``` ### Pattern 4: Combined Filters Entity + content type + date: ```typescript const content = await graphlit.queryContents({ types: [ContentTypes.Email], // Only emails observations: [{ type: ObservableTypes.Person, observable: { id: personId } }], creationDateRange: { from: '2024-01-01' } }); ``` *** ## Use Cases **1. Entity Timeline**: Track mentions over time\ **2. Cross-Source Discovery**: Find entity across email, Slack, docs\ **3. Relationship Mapping**: Content connecting two entities\ **4. Entity-Specific Search**: Scope search to entity context\ **5. Content Audit**: What references this person/org? *** ## Developer Hints * Entity filters query graph database (slower than pure content queries) * Combine with content type filters for speed * Use entity search first to get IDs * Multiple entity filters = AND logic (all must match) * Great for focused discovery *** # Enrich Knowledge Graph with External Data ## User Intent "How do I add additional information to extracted entities? Can I enrich person entities with LinkedIn data or organizations with company information?" ## Operation **Concept**: Entity enrichment strategies\ **SDK Methods**: `queryObservables()`, external API calls, data augmentation\ **Entity**: Enriching Observable properties with external data ## Prerequisites * Knowledge graph with extracted entities * External data sources (APIs, databases) * Understanding of Observable properties *** ## Enrichment Strategies ### 1. External API Enrichment **Person Enrichment with LinkedIn**: ```typescript import { Graphlit } from 'graphlit-client'; import { ObservableTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Get person entity const people = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Person] } }); // Enrich with LinkedIn data (example) for (const person of people.observables.results.slice(0, 5)) { const email = person.observable.properties?.email; if (email) { // Call external LinkedIn API (pseudocode) const linkedInData = await fetchLinkedInData(email); // Store enriched data const enrichedPerson = { ...person.observable, properties: { ...person.observable.properties, linkedInUrl: linkedInData.profileUrl, currentTitle: linkedInData.currentPosition?.title, currentCompany: linkedInData.currentPosition?.company, skills: linkedInData.skills, enrichedAt: new Date().toISOString() } }; console.log(`Enriched ${person.observable.name}:`); console.log(` Title: ${enrichedPerson.properties.currentTitle}`); console.log(` Company: ${enrichedPerson.properties.currentCompany}`); } } async function fetchLinkedInData(email: string): Promise { // External API call (example) // In production, use actual LinkedIn API or similar service return { profileUrl: `https://linkedin.com/in/${email.split('@')[0]}`, currentPosition: { title: "CEO", company: "Graphlit" }, skills: ["AI", "Knowledge Graphs", "Semantic Memory"] }; } ``` **Organization Enrichment with Clearbit**: ```typescript // Enrich organization data const orgs = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Organization] } }); for (const org of orgs.observables.results.slice(0, 5)) { // Get domain from properties or infer const domain = org.observable.properties?.url || `${org.observable.name.toLowerCase()}.com`; // Call Clearbit API (example) const companyData = await fetchClearbitData(domain); const enrichedOrg = { ...org.observable, properties: { ...org.observable.properties, description: companyData.description, industry: companyData.category.industry, employees: companyData.metrics?.employees, founded: companyData.foundedYear, location: companyData.geo?.city, logo: companyData.logo } }; console.log(`Enriched ${org.observable.name}:`); console.log(` Industry: ${enrichedOrg.properties.industry}`); console.log(` Employees: ${enrichedOrg.properties.employees}`); } async function fetchClearbitData(domain: string): Promise { // External API call return { description: "Company description", category: { industry: "Technology" }, metrics: { employees: 50 }, foundedYear: 2023, geo: { city: "Seattle" }, logo: "https://logo.clearbit.com/" + domain }; } ``` ### 2. Workflow-Based Enrichment **Enrichment Stage in Workflow** (future feature): ```typescript // Future: Enrichment stage in workflows const workflow = await graphlit.createWorkflow({ name: "Extract and Enrich", extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ObservableTypes.Person, ObservableTypes.Organization] } }] }, enrichment: { // Future feature jobs: [{ connector: { type: EnrichmentServiceExternal, config: { // Automatic enrichment configuration } } }] } }); ``` ### 3. Internal Data Enrichment **Aggregate from Multiple Sources**: ```typescript // Enrich entity with data from multiple content sources async function enrichFromContent(entityId: string): Promise { // Find all content mentioning entity const content = await graphlit.queryContents({ observations: [{ observable: { id: entityId } }] }); // Aggregate properties from observations const aggregated = { mentionCount: content.contents.results.length, sources: new Set(), firstMention: null as Date | null, lastMention: null as Date | null, avgConfidence: 0 }; let totalConfidence = 0; let confidenceCount = 0; content.contents.results.forEach(item => { // Track sources if (item.feedId) { aggregated.sources.add(item.feedId); } // Track dates const date = new Date(item.creationDate); if (!aggregated.firstMention || date < aggregated.firstMention) { aggregated.firstMention = date; } if (!aggregated.lastMention || date > aggregated.lastMention) { aggregated.lastMention = date; } // Calculate avg confidence item.observations?.forEach(obs => { if (obs.observable.id === entityId) { obs.occurrences?.forEach(occ => { totalConfidence += occ.confidence; confidenceCount++; }); } }); }); aggregated.avgConfidence = confidenceCount > 0 ? totalConfidence / confidenceCount : 0; return aggregated; } const enrichedData = await enrichFromContent('entity-id'); console.log('Entity enriched with internal data:'); console.log(` Mentions: ${enrichedData.mentionCount}`); console.log(` Sources: ${enrichedData.sources.size}`); console.log(` Avg confidence: ${enrichedData.avgConfidence.toFixed(2)}`); ``` ### 4. Geographic Enrichment **Place Entity Geocoding**: ```typescript async function enrichPlaces(): Promise { const places = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Place] } }); for (const place of places.observables.results) { // Geocode using external service const geocoded = await geocodePlace(place.observable.name); const enriched = { ...place.observable, properties: { ...place.observable.properties, latitude: geocoded.lat, longitude: geocoded.lng, country: geocoded.country, region: geocoded.region, population: geocoded.population } }; console.log(`Enriched ${place.observable.name}:`); console.log(` Coordinates: ${geocoded.lat}, ${geocoded.lng}`); } } async function geocodePlace(name: string): Promise { // Call Google Maps API or similar return { lat: 47.6062, lng: -122.3321, country: "United States", region: "Washington", population: 750000 }; } ``` *** ## Enrichment Patterns ### Pattern 1: Batch Enrichment Process all entities of type: ```typescript async function batchEnrich( entityType: ObservableTypes, enrichFunc: (entity: Observable) => Promise ): Promise { const entities = await graphlit.queryObservables({ filter: { types: [entityType] } }); console.log(`Enriching ${entities.observables.results.length} ${entityType} entities...`); for (const entity of entities.observables.results) { try { const enrichedData = await enrichFunc(entity); console.log(`āœ“ Enriched ${entity.observable.name}`); // Store enrichedData in your application database } catch (error) { console.error(`āœ— Failed to enrich ${entity.observable.name}:`, error); } } } // Enrich all people await batchEnrich(ObservableTypes.Person, async (person) => { return await fetchLinkedInData(person.observable.properties?.email); }); ``` ### Pattern 2: On-Demand Enrichment Enrich when entity is accessed: ```typescript const enrichmentCache = new Map(); async function getEnrichedEntity(entityId: string): Promise { // Check cache if (enrichmentCache.has(entityId)) { return enrichmentCache.get(entityId); } // Fetch entity const entities = await graphlit.queryObservables({ filter: { ids: [entityId] } }); if (entities.observables.results.length === 0) { return null; } const entity = entities.observables.results[0]; // Enrich const enrichedData = await enrichEntity(entity); // Cache enrichmentCache.set(entityId, enrichedData); return enrichedData; } ``` ### Pattern 3: Periodic Refresh Update enrichment data regularly: ```typescript async function scheduleEnrichmentRefresh(): Promise { // Run every 24 hours setInterval(async () => { console.log('Refreshing entity enrichment...'); const entities = await graphlit.queryObservables({}); for (const entity of entities.observables.results) { // Re-enrich entity // Update enrichment timestamp } console.log('Enrichment refresh complete'); }, 24 * 60 * 60 * 1000); // 24 hours } ``` *** ## Storage Considerations **Where to Store Enriched Data**: 1. **Application Database**: Store alongside entity IDs 2. **Cache**: Redis/Memcached for fast access 3. **File System**: JSON files for simple cases 4. **Custom Properties** (if supported): Extend Observable properties **Example Storage**: ```typescript // PostgreSQL schema example /* CREATE TABLE entity_enrichment ( entity_id VARCHAR(255) PRIMARY KEY, entity_type VARCHAR(50), enriched_data JSONB, enriched_at TIMESTAMP, source VARCHAR(100) ); */ // Store enriched data async function storeEnrichment( entityId: string, entityType: string, data: any, source: string ): Promise { // Store in your database // await db.query('INSERT INTO entity_enrichment ...'); } ``` *** ## Common External Data Sources ### Person Enrichment * **LinkedIn**: Professional data * **Clearbit**: Contact information * **FullContact**: Social profiles * **Hunter.io**: Email verification ### Organization Enrichment * **Clearbit**: Company data * **Crunchbase**: Funding, valuation * **Google Places**: Location, reviews * **D\&B**: Business intelligence ### Place Enrichment * **Google Maps**: Geocoding, details * **OpenStreetMap**: Geographic data * **GeoNames**: Place information *** ## Developer Hints * Enrichment is external to Graphlit (store in your app) * Use entity IDs to link enrichment data * Cache enriched data to avoid repeated API calls * Respect external API rate limits * Track enrichment timestamps * Handle API failures gracefully * Future: Native enrichment workflows *** # Build Knowledge Graph from Emails ## Use Case: Build Knowledge Graph from Emails ### User Intent "How do I extract entities from my Gmail or Outlook emails to build a knowledge graph? Show me how to connect contacts, organizations, and build relationship networks from email data." ### Operation **SDK Methods**: `createWorkflow()`, `createFeed()`, `isFeedDone()`, `queryContents()`, `queryObservables()`\ **GraphQL**: Feed creation + entity extraction + relationship queries\ **Entity**: Email Feed ā†’ Email Content ā†’ Observations ā†’ Observables (Contact Graph) ### Prerequisites * Graphlit project with API credentials * Gmail or Microsoft 365 account * OAuth tokens for email access (via Graphlit Developer Portal) * Understanding of feed and workflow concepts *** ### Complete Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentTypes, EntityState, FeedServiceTypes, ObservableTypes } from 'graphlit-client/dist/generated/graphql-types'; import { FeedTypes, FeedServiceTypes, ExtractionServiceTypes, ObservableTypes, ContentTypes, EntityState } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); console.log('=== Building Knowledge Graph from Emails ===\n'); // Step 1: Create extraction workflow console.log('Step 1: Creating entity extraction workflow...'); const workflow = await graphlit.createWorkflow({ name: "Email Entity Extraction", extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.Person, // Senders, recipients, mentions ObservableTypes.Organization, // Companies from domains/signatures ObservableTypes.Event, // Meeting mentions, deadlines ObservableTypes.Product, // Products/services discussed ObservableTypes.Place // Locations mentioned ] } }] } }); console.log(`āœ“ Workflow: ${workflow.createWorkflow.id}\n`); // Step 2: Create Gmail feed with OAuth console.log('Step 2: Creating Gmail feed...'); const feed = await graphlit.createFeed({ name: "My Gmail", type: FeedEmail, email: { type: FeedServiceGmail, token: process.env.GOOGLE_OAUTH_TOKEN!, // From Developer Portal readLimit: 100, // Number of emails to sync includeAttachments: true // Sync attachments too }, workflow: { id: workflow.createWorkflow.id } }); console.log(`āœ“ Feed: ${feed.createFeed.id}\n`); // Step 3: Wait for email sync console.log('Step 3: Syncing emails...'); let isDone = false; while (!isDone) { const status = await graphlit.isFeedDone(feed.createFeed.id); isDone = status.isFeedDone.result; if (!isDone) { console.log(' Syncing... (checking again in 5s)'); await new Promise(resolve => setTimeout(resolve, 5000)); } } console.log('āœ“ Sync complete\n'); // Step 4: Query synced emails console.log('Step 4: Querying synced emails...'); const emails = await graphlit.queryContents({ types: [ContentTypes.Email], feeds: [{ id: feed.createFeed.id }] }); console.log(`āœ“ Synced ${emails.contents.results.length} emails\n`); // Step 5: Analyze email metadata console.log('Step 5: Analyzing email senders...\n'); const senders = new Map(); emails.contents.results.forEach(email => { if (email.email?.from) { email.email.from.forEach(sender => { const email_addr = sender.email || 'unknown'; senders.set(email_addr, (senders.get(email_addr) || 0) + 1); }); } }); console.log('Top email senders:'); Array.from(senders.entries()) .sort((a, b) => b[1] - a[1]) .slice(0, 5) .forEach(([email, count]) => { console.log(` ${email}: ${count} emails`); }); console.log(); // Step 6: Query extracted entities console.log('Step 6: Querying knowledge graph...\n'); // Get all people from emails const people = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Person], states: [EntityState.Enabled] } }); console.log(`People extracted: ${people.observables.results.length}`); // Get all organizations const orgs = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Organization], states: [EntityState.Enabled] } }); console.log(`Organizations extracted: ${orgs.observables.results.length}\n`); // Step 7: Build contact network console.log('Step 7: Building contact network...\n'); // Email threads create person-to-person relationships const contactNetwork = new Map>(); emails.contents.results.forEach(email => { const from = email.email?.from?.[0]?.email; const toList = email.email?.to?.map(t => t.email) || []; const ccList = email.email?.cc?.map(c => c.email) || []; const recipients = [...toList, ...ccList].filter(e => e); if (from && recipients.length > 0) { if (!contactNetwork.has(from)) { contactNetwork.set(from, new Set()); } recipients.forEach(recipient => { contactNetwork.get(from)!.add(recipient); }); } }); console.log('Top email relationships:'); Array.from(contactNetwork.entries()) .map(([from, to]) => ({ from, count: to.size })) .sort((a, b) => b.count - a.count) .slice(0, 5) .forEach(({ from, count }) => { console.log(` ${from} ā†’ ${count} contacts`); }); console.log('\nāœ“ Knowledge graph complete!'); ``` *** ## Run asyncio.run(build\_kg\_from\_emails()) ```` ### C# ```csharp using Graphlit; using Graphlit.Api.Input; var graphlit = new Graphlit(); Console.WriteLine("=== Building Knowledge Graph from Emails ===\n"); // Step 1: Create workflow Console.WriteLine("Step 1: Creating entity extraction workflow..."); var workflow = await graphlit.CreateWorkflow( name: "Email Entity Extraction", extraction: new WorkflowExtractionInput { Jobs = new[] { new WorkflowExtractionJobInput { Connector = new ExtractionConnectorInput { Type = ExtractionServiceModelText, ExtractedTypes = new[] { ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Event, ObservableTypes.Product } } } } } ); Console.WriteLine($"āœ“ Workflow: {workflow.CreateWorkflow.Id}\n"); // Step 2: Create Gmail feed Console.WriteLine("Step 2: Creating Gmail feed..."); var feed = await graphlit.CreateFeed( name: "My Gmail", type: FeedEmail, email: new EmailFeedInput { Type = FeedServiceGmail, Token = Environment.GetEnvironmentVariable("GOOGLE_OAUTH_TOKEN"), ReadLimit = 100, IncludeAttachments = true }, workflow: new EntityReferenceInput { Id = workflow.CreateWorkflow.Id } ); Console.WriteLine($"āœ“ Feed: {feed.CreateFeed.Id}\n"); // (Continue with remaining steps...) ```` *** ### Step-by-Step Explanation #### Step 1: Create Entity Extraction Workflow **Email-Specific Entity Types**: * **Person**: Senders, recipients, people mentioned in body, signatures * **Organization**: Companies from email domains, mentioned in text, signatures * **Event**: Meetings, deadlines, calendar invites mentioned * **Product**: Products/services discussed in emails * **Place**: Locations mentioned (meeting locations, offices) **Why Text Extraction**: * Emails are primarily text-based * No visual analysis needed (unlike PDFs) * Fast and cost-effective * Handles HTML email bodies #### Step 2: Configure Email Feed **Gmail Feed Configuration**: ```typescript feed: { type: FeedEmail, email: { type: FeedServiceGmail, token: googleOAuthToken, // From Developer Portal OAuth readLimit: 100, // How many emails to sync includeAttachments: true, // Sync attachments as separate content labels: ['INBOX', 'SENT'] // Optional: specific labels } } ``` **Microsoft Outlook Feed**: ```typescript feed: { type: FeedEmail, email: { type: FeedServiceOutlook, token: microsoftOAuthToken, // Microsoft OAuth token readLimit: 100, includeAttachments: true, folderNames: ['Inbox', 'Sent Items'] // Optional: specific folders } } ``` **OAuth Token Setup**: 1. Go to Graphlit Developer Portal 2. Navigate to Connectors ā†’ Email 3. Authorize Gmail or Outlook 4. Copy OAuth token 5. Use in feed creation #### Step 3: Sync and Wait for Processing **Sync Timeline**: * 100 emails: 1-2 minutes * 1,000 emails: 10-15 minutes * 10,000 emails: 1-2 hours **Polling Strategy**: ```typescript const pollInterval = 5000; // 5 seconds const maxWait = 600000; // 10 minutes max const startTime = Date.now(); while (!isDone && (Date.now() - startTime < maxWait)) { const status = await graphlit.isFeedDone(feedId); isDone = status.isFeedDone.result; if (!isDone) { await new Promise(resolve => setTimeout(resolve, pollInterval)); } } ``` #### Step 4: Query Email Content **Email Metadata Structure**: ```typescript email: { from: [{ name: "Kirk Marple", email: "kirk@graphlit.com" }], to: [{ name: "John Doe", email: "john@example.com" }], cc: [{ name: "Jane Smith", email: "jane@example.com" }], bcc: [], // Usually empty (privacy) subject: "Q4 Planning Meeting", labels: ["INBOX", "IMPORTANT"], // Gmail labels identifier: "", threadIdentifier: "", sensitivity: "Normal", priority: "High", attachmentCount: 2 } ``` #### Step 5: Extract Entity Observations **Email Body Extraction**: ```typescript const emailContent = await graphlit.getContent(emailId); // Entities from email body emailContent.content.observations?.forEach(obs => { console.log(`${obs.type}: ${obs.observable.name}`); // No page numbers (emails aren't paginated) // High confidence for explicit mentions }); ``` **Signature Extraction**: Email signatures are rich sources of Person/Organization data: ``` Kirk Marple CEO, Graphlit kirk@graphlit.com https://graphlit.com ``` Extracts: Person("Kirk Marple"), Organization("Graphlit") #### Step 6: Build Contact Network **Email Threads Create Relationships**: * `from` ā†’ `to`/`cc`: Direct communication * Frequency indicates relationship strength * Thread IDs group related emails **Network Analysis**: ```typescript // Who communicates with whom const relationships = new Map>(); emails.contents.results.forEach(email => { const from = email.email?.from?.[0]?.email; const recipients = [ ...(email.email?.to?.map(t => t.email) || []), ...(email.email?.cc?.map(c => c.email) || []) ]; if (from && recipients.length > 0) { if (!relationships.has(from)) { relationships.set(from, new Map()); } recipients.forEach(to => { const recipientMap = relationships.get(from)!; recipientMap.set(to, (recipientMap.get(to) || 0) + 1); }); } }); ``` #### Step 7: Query Knowledge Graph **Cross-Feed Entity Queries**: Entities from emails become part of global knowledge graph: ```typescript // Find all content mentioning a person (emails + other sources) const kirkContent = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Person, observable: { id: kirkPersonId } }] }); // Includes: emails, Slack messages, documents, etc. ``` *** ### Configuration Options #### Limiting Email Sync Scope **By Count**: ```typescript email: { readLimit: 500 // Most recent 500 emails } ``` **By Date Range**: ```typescript email: { readLimit: 1000, // Only recent emails (Graphlit handles recency automatically) } ``` **By Labels/Folders**: ```typescript // Gmail email: { type: FeedServiceGmail, labels: ['INBOX', 'IMPORTANT', 'Sent'] // Specific labels only } // Outlook email: { type: FeedServiceOutlook, folderNames: ['Inbox', 'Sent Items', 'Archive'] } ``` #### Handling Attachments **Include Attachments**: ```typescript email: { includeAttachments: true // PDFs, images, etc. become separate content } ``` Attachments are processed through workflow: * PDFs ā†’ extraction ā†’ entities * Images ā†’ vision analysis ā†’ entities * Documents ā†’ text extraction ā†’ entities **Exclude Attachments** (faster): ```typescript email: { includeAttachments: false // Email body only } ``` *** ### Variations #### Variation 1: Organization Email Domain Mapping Extract organizations from email domains: ```typescript function extractOrgFromDomain(email: string): string | null { const domain = email.split('@')[1]; if (!domain) return null; // Map common domains const orgMap: Record = { 'gmail.com': null, // Personal email 'outlook.com': null, // Personal email 'graphlit.com': 'Graphlit', 'microsoft.com': 'Microsoft', // ... add more }; return orgMap[domain] || domain.replace(/\.(com|org|net|io)$/, ''); } // Build org roster from emails const emailsByOrg = new Map>(); emails.contents.results.forEach(email => { email.email?.from?.forEach(sender => { const org = extractOrgFromDomain(sender.email || ''); if (org) { if (!emailsByOrg.has(org)) { emailsByOrg.set(org, new Set()); } emailsByOrg.get(org)!.add(sender.email || ''); } }); }); console.log('Emails by organization:'); emailsByOrg.forEach((emails, org) => { console.log(` ${org}: ${emails.size} contacts`); }); ``` #### Variation 2: Email Thread Analysis Analyze conversation threads: ```typescript // Group emails by thread const threads = new Map>(); emails.contents.results.forEach(email => { const threadId = email.email?.threadIdentifier || email.id; if (!threads.has(threadId)) { threads.set(threadId, []); } threads.get(threadId)!.push(email); }); // Find longest threads const longThreads = Array.from(threads.entries()) .sort((a, b) => b[1].length - a[1].length) .slice(0, 5); console.log('Longest email threads:'); longThreads.forEach(([threadId, emails]) => { const subject = emails[0].email?.subject; console.log(` "${subject}": ${emails.length} emails`); }); ``` #### Variation 3: Contact Frequency Ranking Rank contacts by interaction frequency: ```typescript interface ContactStats { email: string; name?: string; emailsReceived: number; emailsSent: number; total: number; } const myEmail = 'my@email.com'; // Your email address const contactStats = new Map(); emails.contents.results.forEach(email => { const from = email.email?.from?.[0]; const toList = email.email?.to || []; const ccList = email.email?.cc || []; if (from?.email === myEmail) { // Email I sent [...toList, ...ccList].forEach(recipient => { if (!contactStats.has(recipient.email!)) { contactStats.set(recipient.email!, { email: recipient.email!, name: recipient.name, emailsReceived: 0, emailsSent: 0, total: 0 }); } const stats = contactStats.get(recipient.email!)!; stats.emailsSent++; stats.total++; }); } else if (from?.email) { // Email I received if (!contactStats.has(from.email)) { contactStats.set(from.email, { email: from.email, name: from.name, emailsReceived: 0, emailsSent: 0, total: 0 }); } const stats = contactStats.get(from.email)!; stats.emailsReceived++; stats.total++; } }); // Top contacts const topContacts = Array.from(contactStats.values()) .sort((a, b) => b.total - a.total) .slice(0, 10); console.log('Top contacts:'); topContacts.forEach((contact, i) => { console.log(`${i + 1}. ${contact.name || contact.email}`); console.log(` Received: ${contact.emailsReceived}, Sent: ${contact.emailsSent}`); }); ``` #### Variation 4: Entity-Enhanced Email Search Search emails by entity: ```typescript // Find all emails mentioning Graphlit const graphlitOrg = await graphlit.queryObservables({ search: "Graphlit", filter: { types: [ObservableTypes.Organization] } }); const graphlitEmails = await graphlit.queryContents({ types: [ContentTypes.Email], observations: [{ type: ObservableTypes.Organization, observable: { id: graphlitOrg.observables.results[0].observable.id } }] }); console.log(`Emails mentioning Graphlit: ${graphlitEmails.contents.results.length}`); // Who sent these emails? const senders = new Set(); graphlitEmails.contents.results.forEach(email => { email.email?.from?.forEach(sender => { if (sender.email) senders.add(sender.email); }); }); console.log('Senders:', Array.from(senders)); ``` #### Variation 5: Cross-Source Entity Linking Link email entities with other sources: ```typescript // Find person across email + Slack + documents const person = await graphlit.queryObservables({ search: "Kirk Marple", filter: { types: [ObservableTypes.Person] } }); const allMentions = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Person, observable: { id: person.observables.results[0].observable.id } }] }); // Group by content type const byType = allMentions.contents.results.reduce((groups, content) => { const type = content.type || 'UNKNOWN'; if (!groups[type]) groups[type] = []; groups[type].push(content); return groups; }, {} as Record); console.log('Kirk Marple mentions:'); Object.entries(byType).forEach(([type, contents]) => { console.log(` ${type}: ${contents.length} items`); }); ``` *** ### Common Issues & Solutions #### Issue: OAuth Token Expired **Problem**: Feed sync fails with authorization error. **Solution**: Refresh OAuth token in Developer Portal: 1. Go to Developer Portal ā†’ Connectors 2. Re-authorize Gmail/Outlook 3. Copy new token 4. Update feed or create new feed ```typescript // Can't update token on existing feed - create new feed const newFeed = await graphlit.createFeed({ name: "Gmail (Updated)", type: FeedEmail, email: { type: FeedServiceGmail, token: newOAuthToken // Fresh token } }); ``` #### Issue: Duplicate Entities from Sender/Recipient and Body **Problem**: Same person appears as sender AND extracted from body. **Explanation**: This is expected and valuable: * Email metadata (from/to/cc) captured automatically * Body extraction finds additional context * Multiple mentions increase confidence **Not a Problem**: Graphlit deduplicates to single Observable. #### Issue: Too Many Low-Confidence Entities **Problem**: Email extraction finds many uncertain entities. **Solution**: Filter by confidence threshold: ```typescript const highConfidence = email.observations ?.filter(obs => obs.occurrences?.some(occ => occ.confidence >= 0.75)) || []; ``` Emails can have ambiguous mentions ("John said...") with low confidence. #### Issue: Missing Email Body Entities **Problem**: Only sender/recipient captured, no body extraction. **Causes**: 1. Workflow not configured with extraction stage 2. Email is HTML-only with no text 3. Extraction failed for some emails **Solution**: Verify workflow has extraction: ```typescript // Check workflow configuration const workflowDetails = await graphlit.getWorkflow(workflowId); console.log('Extraction jobs:', workflowDetails.workflow.extraction?.jobs); ``` *** ### Developer Hints #### OAuth Token Management * Tokens expire after 1 hour (short-lived) * Refresh tokens valid for 6 months (Gmail) or indefinitely (Outlook) * Use Developer Portal for token management * Production apps should handle token refresh automatically #### Email Sync Best Practices 1. **Start small**: Test with readLimit: 100 first 2. **Incremental sync**: Graphlit tracks what's synced 3. **Monitor quota**: Gmail API has rate limits 4. **Handle failures**: Email sync can be interrupted 5. **Attachments optional**: Skip for faster sync #### Entity Quality from Emails * **High confidence**: Senders/recipients, signatures * **Medium confidence**: Explicit mentions in body * **Low confidence**: Implicit references, pronouns * **Filter threshold**: >=0.7 recommended for emails #### Performance Considerations * Email sync is incremental (doesn't re-sync) * 100 emails = \~1-2 minutes processing * Attachments increase processing time significantly * Entity extraction adds 10-30% overhead #### Privacy and Security * OAuth tokens have user-level permissions * Graphlit never stores raw OAuth refresh tokens * Email content encrypted at rest * Multi-tenant isolation ensures data privacy *** # Build Knowledge Graph from GitHub Repositories ## User Intent "How do I extract entities from GitHub repositories and issues to build a knowledge graph? Show me how to analyze code, contributors, dependencies, and project relationships." ## Operation **SDK Methods**: `createWorkflow()`, `createFeed()` (Site, Issue, Commit, or PullRequest feeds), `isFeedDone()`, `queryContents()`, `queryObservables()`\ **GraphQL**: GitHub feed creation + entity extraction + contributor/project graphs\ **Entity**: GitHub Feed ā†’ Content (Files/Issues/Commits/PRs) ā†’ Observations ā†’ Observables (Developer/Project Graph) ## Prerequisites * Graphlit project with API credentials * GitHub personal access token (via Graphlit Developer Portal) * GitHub repository access * Understanding of feed and workflow concepts *** ## Complete Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentTypes, EntityExtractionServiceTypes, EntityState, FeedServiceTypes, FeedTypes, ObservableTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); console.log('=== Building Knowledge Graph from GitHub ===\n'); // Step 1: Create extraction workflow console.log('Step 1: Creating entity extraction workflow...'); const workflow = await graphlit.createWorkflow({ name: "GitHub Entity Extraction", extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.Repo, ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Software, ObservableTypes.Category, ObservableTypes.Label ] } }] } }); console.log(`āœ“ Workflow: ${workflow.createWorkflow.id}\n`); // Step 2: Create GitHub repository feed console.log('Step 2: Creating GitHub repository feed...'); const repoFeed = await graphlit.createFeed({ name: "Graphlit Samples Repo", type: FeedTypes.Site, site: { type: FeedServiceTypes.GitHub, github: { repositoryOwner: 'graphlit', repositoryName: 'graphlit-samples', personalAccessToken: process.env.GITHUB_TOKEN! }, allowedPaths: ['README.md', 'docs/**', 'python/**', 'nextjs/**'], excludedPaths: ['**/node_modules/**', '**/dist/**'] }, workflow: { id: workflow.createWorkflow.id } }); console.log(`āœ“ Repo Feed: ${repoFeed.createFeed.id}\n`); // Step 3: Create GitHub issues feed console.log('Step 3: Creating GitHub issues feed...'); const issuesFeed = await graphlit.createFeed({ name: "Graphlit Issues", type: FeedTypes.Issue, issue: { type: FeedServiceTypes.GitHubIssues, github: { repositoryOwner: 'graphlit', repositoryName: 'graphlit-samples', personalAccessToken: process.env.GITHUB_TOKEN! }, readLimit: 100 }, workflow: { id: workflow.createWorkflow.id } }); console.log(`āœ“ Issues Feed: ${issuesFeed.createFeed.id}\n`); // Step 4: Wait for sync console.log('Step 4: Syncing repository...'); let repoDone = false; let issuesDone = false; while (!repoDone || !issuesDone) { if (!repoDone) { const repoStatus = await graphlit.isFeedDone(repoFeed.createFeed.id); repoDone = repoStatus.isFeedDone.result; } if (!issuesDone) { const issuesStatus = await graphlit.isFeedDone(issuesFeed.createFeed.id); issuesDone = issuesStatus.isFeedDone.result; } if (!repoDone || !issuesDone) { console.log(' Syncing... (checking again in 5s)'); await new Promise(resolve => setTimeout(resolve, 5000)); } } console.log('āœ“ Sync complete\n'); // Step 5: Query repository content console.log('Step 5: Querying repository files...'); const repoFiles = await graphlit.queryContents({ types: [ContentTypes.File], feeds: [{ id: repoFeed.createFeed.id }] }); console.log(`āœ“ Synced ${repoFiles.contents.results.length} files\n`); // Step 6: Query issues console.log('Step 6: Querying issues...'); const issues = await graphlit.queryContents({ types: [ContentTypes.Issue], feeds: [{ id: issuesFeed.createFeed.id }] }); console.log(`āœ“ Synced ${issues.contents.results.length} issues\n`); // Step 7: Extract repository entities console.log('Step 7: Analyzing repository entities...\n'); // Get all Repo entities const repos = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Repo] } }); console.log(`Repositories: ${repos.observables.results.length}`); // Get contributors (Person entities) const people = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Person] } }); console.log(`Contributors: ${people.observables.results.length}`); // Get dependencies (Software entities) const software = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Software] } }); console.log(`Software/Dependencies: ${software.observables.results.length}\n`); // Step 8: Analyze issue labels console.log('Step 8: Analyzing issue labels...\n'); const labelCounts = new Map(); issues.contents.results.forEach(issue => { (issue.issue?.labels || []).filter(Boolean).forEach(label => { labelCounts.set(label!, (labelCounts.get(label!) || 0) + 1); }); }); console.log('Most common issue labels:'); Array.from(labelCounts.entries()) .sort((a, b) => b[1] - a[1]) .slice(0, 5) .forEach(([label, count]) => { console.log(` ${label}: ${count} issues`); }); // Step 9: Build contributor network console.log('\nStep 9: Building contributor network...\n'); const contributors = new Map(); // Count files by contributor (from observations) repoFiles.contents.results.forEach(file => { file.observations ?.filter(Boolean) .filter(obs => obs?.type === ObservableTypes.Person) .forEach(obs => { const name = obs!.observable.name; if (!contributors.has(name)) contributors.set(name, { files: 0, issues: 0, total: 0 }); contributors.get(name)!.files++; contributors.get(name)!.total++; }); }); // Count issues by contributor (from observations) issues.contents.results.forEach(issue => { issue.observations ?.filter(Boolean) .filter(obs => obs?.type === ObservableTypes.Person) .forEach(obs => { const name = obs!.observable.name; if (!contributors.has(name)) contributors.set(name, { files: 0, issues: 0, total: 0 }); contributors.get(name)!.issues++; contributors.get(name)!.total++; }); }); console.log('Top contributors:'); Array.from(contributors.entries()) .sort((a, b) => b[1].total - a[1].total) .slice(0, 5) .forEach(([name, stats]) => { console.log(` ${name}: ${stats.files} files, ${stats.issues} issues`); }); console.log('\nāœ“ Repository analysis complete!'); ``` *** ## Step-by-Step Explanation ### Step 1: Create Entity Extraction Workflow **GitHub-Specific Entity Types**: * **Repo**: Repository as entity (name, owner, URL, description) * **Person**: Contributors, commit authors, issue creators, reviewers * **Organization**: Repository owner if organization, mentioned companies * **Software**: Dependencies (from package.json, requirements.txt, etc.) * **Category**: Topics, tags, project themes * **Label**: Issue/PR labels **Why Text Extraction**: * Code files, README, docs are text-based * Fast and cost-effective * Handles markdown, code, JSON, YAML ### Step 2: Configure GitHub Repository Feed **Repository Feed Options**: ```typescript site: { type: FeedServiceTypes.GitHub, github: { repositoryOwner: 'graphlit', repositoryName: 'graphlit-samples', personalAccessToken: githubToken }, // Optional: scope what you ingest (recommended) allowedPaths: ['README.md', 'docs/**'], excludedPaths: ['**/node_modules/**', '**/dist/**'] } ``` **What Gets Synced**: * README.md * Repository files matching your `allowedPaths`/`excludedPaths` * Documentation files * Configuration files (package.json, requirements.txt, etc.) ### Step 3: Configure GitHub Issues Feed **Issues Feed Options**: ```typescript issue: { type: FeedServiceTypes.GitHubIssues, github: { repositoryOwner: 'graphlit', repositoryName: 'graphlit-samples', personalAccessToken: githubToken }, readLimit: 100 } ``` **What Gets Synced**: * Issue title and body * Labels * Issue number/identifier * Created/updated dates If you want to focus on specific labels/states, filter after ingestion using `queryContents()`. *** {% hint style="info" %} **Additional GitHub Feed Types**: You can also create feeds for: * **GitHub Commits** (`FeedTypes.Commit`) - Sync commit history, code changes, and developer activity * **GitHub Pull Requests** (`FeedTypes.PullRequest`) - Sync pull requests, reviews, and merge history These feed types are useful for analyzing code review patterns, tracking developer contributions, and understanding project evolution over time. See the [GitHub Commits](/api-guides/use-cases/feeds/project-management/feed-create-github-commits) and [GitHub Pull Requests](/api-guides/use-cases/feeds/project-management/feed-create-github-pull-requests) feed guides for details. {% endhint %} *** ### Step 4: GitHub Token Setup **Creating GitHub Token**: 1. GitHub ā†’ Settings ā†’ Developer settings ā†’ Personal access tokens 2. Generate new token (classic) 3. Select scopes: * `repo` (for private repos) or `public_repo` (for public only) * `read:org` (if accessing org repos) 4. Copy token 5. Use in Graphlit feed creation **OR via Graphlit Developer Portal**: 1. Go to Developer Portal ā†’ Connectors ā†’ Version Control 2. Authorize GitHub 3. Copy OAuth token ### Step 5: Analyze Repository Files **File Content Structure**: ```typescript const file = await graphlit.getContent(fileId); console.log(`File: ${file.content.name}`); console.log(`Type: ${file.content.fileType}`); console.log(`Path: ${file.content.uri}`); // Extracted entities from file file.content.observations?.forEach(obs => { console.log(`${obs.type}: ${obs.observable.name}`); }); ``` **README Analysis**: * Rich source of Repo, Person, Organization entities * Contributors listed * Dependencies mentioned * Project description **package.json/requirements.txt Analysis**: * Dependencies as Software entities * Version information * Project metadata ### Step 6: Analyze GitHub Issues **Issue Metadata**: ```typescript issue: { identifier: "42", // Issue number title: "Add feature X", project: "graphlit-samples", status: "Open", priority: "High", labels: ["feature", "enhancement"], author: { name: "Kirk Marple", email: "kirk@graphlit.com" } } ``` **Entity Extraction from Issues**: * **Person**: Issue author, mentioned contributors (@username) * **Software**: Tools/libraries mentioned * **Category**: Feature areas, components * **Label**: Issue labels as Label entities ### Step 7: Build Contributor Graph **Contributors from Multiple Sources**: 1. **File authors**: Extracted from README, commit mentions 2. **Issue creators**: From issue author field 3. **Code comments**: Developers mentioned in code 4. **Documentation**: Authors in docs ```typescript // Deduplicate contributors const uniqueContributors = new Map(); // Combine from all sources allContent.forEach(content => { content.observations ?.filter(obs => obs.type === ObservableTypes.Person) .forEach(obs => { if (!uniqueContributors.has(obs.observable.id)) { uniqueContributors.set(obs.observable.id, { observableId: obs.observable.id, name: obs.observable.name, email: obs.observable.properties?.email, contributions: 0 }); } uniqueContributors.get(obs.observable.id)!.contributions++; }); }); ``` ### Step 8: Dependency Analysis **Extract Software Dependencies**: ```typescript // Get all Software entities const dependencies = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Software] } }); // Find which files reference each dependency for (const dep of dependencies.observables.results) { const references = await graphlit.queryContents({ feeds: [{ id: repoFeed.createFeed.id }], observations: [{ type: ObservableTypes.Software, observable: { id: dep.observable.id } }] }); console.log(`${dep.observable.name}: ${references.contents.results.length} files`); } ``` *** ## Configuration Options ### Scope the repository sync Use `allowedPaths`/`excludedPaths` on the `site` feed to control repository size and cost: ```typescript site: { allowedPaths: ['README.md', 'docs/**', 'src/**'], excludedPaths: ['**/node_modules/**', '**/dist/**', '**/*.lock'] } ``` ### Limit issue backfill size Use `readLimit` on the `issue` feed: ```typescript issue: { readLimit: 500 } ``` If you want to focus on a specific subset (e.g., only certain labels), filter after ingestion using `queryContents()`. *** ## Variations ### Variation 1: Multi-Repository Analysis Analyze multiple repositories in an organization: ```typescript const repos = [ { owner: 'graphlit', name: 'graphlit-client-typescript' }, { owner: 'graphlit', name: 'graphlit-client-python' }, { owner: 'graphlit', name: 'graphlit-client-dotnet' } ]; const githubToken = process.env.GITHUB_TOKEN!; const feeds = await Promise.all( repos.map(repo => graphlit.createFeed({ name: `${repo.owner}/${repo.name}`, type: FeedTypes.Site, site: { type: FeedServiceTypes.GitHub, github: { repositoryOwner: repo.owner, repositoryName: repo.name, personalAccessToken: githubToken } }, workflow: { id: workflowId } }) ) ); // Wait for all to sync const waitForAll = async () => { let allDone = false; while (!allDone) { const statuses = await Promise.all( feeds.map(f => graphlit.isFeedDone(f.createFeed.id)) ); allDone = statuses.every(s => s.isFeedDone.result); if (!allDone) { await new Promise(resolve => setTimeout(resolve, 5000)); } } }; await waitForAll(); // Analyze cross-repo entities const allRepos = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Repo] } }); console.log(`Total repositories: ${allRepos.observables.results.length}`); ``` ### Variation 2: Dependency Graph Visualization Map software dependencies: ```typescript // Extract all Software entities and their relationships const dependencies = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Software] } }); interface DependencyNode { name: string; usedBy: string[]; // Files/repos using this dependency version?: string; } const depGraph = new Map(); for (const dep of dependencies.observables.results) { const usages = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Software, observable: { id: dep.observable.id } }] }); depGraph.set(dep.observable.name, { name: dep.observable.name, usedBy: usages.contents.results.map(c => c.name), version: dep.observable.properties?.version }); } // Find most common dependencies const topDeps = Array.from(depGraph.values()) .sort((a, b) => b.usedBy.length - a.usedBy.length) .slice(0, 10); console.log('Most used dependencies:'); topDeps.forEach(dep => { console.log(` ${dep.name}: ${dep.usedBy.length} files`); }); ``` ### Variation 3: Issue Classification by Entities Categorize issues by extracted entities: ```typescript // Group issues by entity types const issuesByEntity = new Map>(); issues.contents.results.forEach(issue => { issue.observations?.forEach(obs => { const key = `${obs.type}: ${obs.observable.name}`; if (!issuesByEntity.has(key)) { issuesByEntity.set(key, []); } issuesByEntity.get(key)!.push(issue); }); }); // Find entities with most issues const entityIssueCount = Array.from(issuesByEntity.entries()) .map(([entity, issues]) => ({ entity, count: issues.length })) .sort((a, b) => b.count - a.count); console.log('Entities with most related issues:'); entityIssueCount.slice(0, 10).forEach(item => { console.log(` ${item.entity}: ${item.count} issues`); }); ``` ### Variation 4: Contributor Activity Timeline Track contributor activity over time: ```typescript interface ContributorActivity { name: string; firstContribution: Date; lastContribution: Date; contributions: Array<{ date: Date; type: 'file' | 'issue' }>; } const activity = new Map(); // Track file contributions repoFiles.contents.results.forEach(file => { const date = new Date(file.creationDate); file.observations ?.filter(obs => obs.type === ObservableTypes.Person) .forEach(obs => { const name = obs.observable.name; if (!activity.has(name)) { activity.set(name, { name, firstContribution: date, lastContribution: date, contributions: [] }); } const contrib = activity.get(name)!; contrib.contributions.push({ date, type: 'file' }); if (date < contrib.firstContribution) contrib.firstContribution = date; if (date > contrib.lastContribution) contrib.lastContribution = date; }); }); // Track issue contributions issues.contents.results.forEach(issue => { const author = issue.issue?.author?.name; const date = new Date(issue.creationDate); if (author) { if (!activity.has(author)) { activity.set(author, { name: author, firstContribution: date, lastContribution: date, contributions: [] }); } const contrib = activity.get(author)!; contrib.contributions.push({ date, type: 'issue' }); if (date < contrib.firstContribution) contrib.firstContribution = date; if (date > contrib.lastContribution) contrib.lastContribution = date; } }); // Find most active contributors (by recent activity) const recent = Array.from(activity.values()) .sort((a, b) => b.lastContribution.getTime() - a.lastContribution.getTime()) .slice(0, 10); console.log('Most recently active contributors:'); recent.forEach(contrib => { console.log(` ${contrib.name}: ${contrib.contributions.length} contributions`); console.log(` First: ${contrib.firstContribution.toLocaleDateString()}`); console.log(` Last: ${contrib.lastContribution.toLocaleDateString()}`); }); ``` ### Variation 5: Cross-Repository Entity Linking Find entities that appear across multiple repositories: ```typescript // After syncing multiple repos, find cross-repo entities const allPeople = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Person] } }); for (const person of allPeople.observables.results) { // Find all content (across repos) mentioning this person const mentions = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Person, observable: { id: person.observable.id } }] }); // Group by feed (repository) const reposMentioned = new Set( mentions.contents.results.map(c => c.feed?.id).filter(Boolean) ); if (reposMentioned.size > 1) { console.log(`${person.observable.name}: appears in ${reposMentioned.size} repos`); } } ``` *** ## Common Issues & Solutions ### Issue: Large Repository, Slow Sync **Problem**: Repository with 1000s of files takes hours to sync. **Solutions**: 1. **Scope paths**: Only sync the folders you need 2. **Exclude build output**: Skip `node_modules`, `dist`, etc. 3. **Keep readLimit conservative**: Backfill in smaller chunks ```typescript site: { allowedPaths: ['README.md', 'docs/**', 'src/**'], excludedPaths: ['**/node_modules/**', '**/dist/**', '**/build/**'], readLimit: 250 } ``` ### Issue: GitHub API Rate Limiting **Problem**: Sync fails with rate limit error. **Explanation**: GitHub API has rate limits (5000 requests/hour for authenticated). **Solutions**: 1. Use authenticated token (higher limits) 2. Sync fewer repositories simultaneously 3. Increase polling interval 4. Wait for rate limit reset ### Issue: Missing Dependencies from package.json **Problem**: Software entities not extracted from package files. **Cause**: Need to sync configuration files explicitly. **Solution**: Include config file types: ```typescript allowedPaths: ['**/package.json', '**/requirements.txt', '**/*.toml', '**/*.lock'] ``` ### Issue: No Contributor Entities **Problem**: No Person entities extracted from repository. **Causes**: 1. README doesn't list contributors 2. Code comments don't mention developers 3. Only files were synced; issues/PRs were not ingested **Solution**: Sync more sources (like issues) and ensure your extraction types include `Person`: ```typescript issue: { type: FeedServiceTypes.GitHubIssues, github: { repositoryOwner: 'graphlit', repositoryName: 'graphlit-samples', personalAccessToken: process.env.GITHUB_TOKEN! } } ``` *** ## Developer Hints ### GitHub Token Best Practices * Use fine-grained tokens (new GitHub feature) when possible * Minimum scope: `repo` for private, `public_repo` for public * Rotate tokens regularly * Don't commit tokens to code (use env variables) * Monitor token usage in GitHub settings ### File Type Recommendations **Documentation Analysis**: ```typescript allowedPaths: ['README.md', '**/*.md', '**/*.rst', '**/*.txt', '**/*.adoc'] ``` **Full Code Analysis**: ```typescript allowedPaths: ['**/*.py', '**/*.js', '**/*.ts', '**/*.java', '**/*.go', '**/*.rs', '**/*.cpp', '**/*.c', '**/*.h'] ``` **Configuration + Dependencies**: ```typescript allowedPaths: ['**/package.json', '**/*.yaml', '**/*.yml', '**/*.toml', '**/*.lock', '**/*.txt'] ``` ### Performance Optimization * Start with README + package files only * Add more file types incrementally * Sync issues separately (can be slow) * Use multiple feeds for large org * Cache entity queries ### Entity Quality by Source * **High confidence**: README (explicit mentions), package.json (dependencies) * **Medium confidence**: Code comments, documentation * **Low confidence**: Implicit mentions in code *** ## Production Patterns ### Pattern from Graphlit Samples `Graphlit_2024_09_29_Explore_GitHub_Repo.ipynb`: * Syncs public GitHub repository * Extracts Repo, Person, Software entities * Analyzes dependencies from package.json * Builds contributor network * Exports entity graph for visualization `Graphlit_2025_03_17_Classify_GitHub_Issues.ipynb`: * Syncs GitHub issues * Extracts entities from issue descriptions * Classifies issues by entity types * Groups related issues * Priority ranking by entity importance ### Open Source Intelligence Use Cases * **Dependency tracking**: Which projects use which libraries * **Contributor analysis**: Developer activity, collaboration * **Project relationships**: Shared contributors, common dependencies * **Technology adoption**: What tools/frameworks gaining traction * **Security analysis**: Vulnerable dependency detection *** # Build Knowledge Graph from Meeting Recordings ## User Intent "How do I extract entities from meeting recordings (audio/video)? Show me how to transcribe meetings and analyze participants, topics, action items, and decisions." ## Operation **SDK Methods**: `createWorkflow()`, `ingestUri()`, `isContentDone()`, `getContent()`, `queryObservables()`\ **GraphQL**: Audio/video ingestion + transcription + entity extraction\ **Entity**: Audio/Video ā†’ Transcription ā†’ Text ā†’ Observations ā†’ Observables (Meeting Graph) ## Prerequisites * Graphlit project with API credentials * Meeting recordings (MP3, MP4, WAV, or other audio/video formats) * Understanding of workflow configuration * Transcription service access (Deepgram, AssemblyAI, or Whisper) *** ## Complete Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { ObservableTypes } from 'graphlit-client/dist/generated/graphql-types'; import { FilePreparationServiceTypes, AudioTranscriptionServiceTypes, ExtractionServiceTypes, ObservableTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); console.log('=== Building Knowledge Graph from Meeting ===\n'); // Step 1: Create transcription + extraction workflow console.log('Step 1: Creating workflow...'); const workflow = await graphlit.createWorkflow({ name: "Meeting Entity Extraction", preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.Deepgram, audioTranscription: { model: DeepgramModels.Nova2 // Fast, accurate } } }] }, extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.Person, // Participants, mentioned people ObservableTypes.Organization, // Companies discussed ObservableTypes.Product, // Products, services mentioned ObservableTypes.Event, // Action items, deadlines, follow-ups ObservableTypes.Place, // Locations mentioned ObservableTypes.Category // Topics, projects, themes ] } }] } }); console.log(`āœ“ Workflow: ${workflow.createWorkflow.id}\n`); // Step 2: Ingest meeting recording console.log('Step 2: Ingesting meeting recording...'); const meeting = await graphlit.ingestUri( 'https://example.com/meetings/q4-planning.mp4', "Q4 Planning Meeting", undefined, undefined, undefined, { id: workflow.createWorkflow.id } ); console.log(`āœ“ Ingested: ${meeting.ingestUri.id}\n`); // Step 3: Wait for transcription + extraction console.log('Step 3: Transcribing and extracting entities...'); console.log('(This may take several minutes for long recordings)\n'); let isDone = false; let lastStatus = ''; while (!isDone) { const status = await graphlit.isContentDone(meeting.ingestUri.id); isDone = status.isContentDone.result; if (!isDone) { const newStatus = ' Processing...'; if (newStatus !== lastStatus) { console.log(newStatus); lastStatus = newStatus; } await new Promise(resolve => setTimeout(resolve, 5000)); } } console.log('āœ“ Processing complete\n'); // Step 4: Get meeting details console.log('Step 4: Retrieving meeting transcript and entities...'); const meetingDetails = await graphlit.getContent(meeting.ingestUri.id); const content = meetingDetails.content; console.log(`āœ“ Meeting: ${content.name}`); console.log(` Duration: ${content.audio?.duration || 0} seconds`); console.log(` Entities: ${content.observations?.length || 0}\n`); // Step 5: Display transcript excerpt console.log('Step 5: Transcript excerpt...\n'); const transcript = content.markdown || content.text || ''; const excerpt = transcript.substring(0, 500); console.log(excerpt); console.log(transcript.length > 500 ? '...\n' : '\n'); // Step 6: Analyze extracted entities console.log('Step 6: Analyzing entities...\n'); // Group by type const byType = new Map>(); content.observations?.forEach(obs => { if (!byType.has(obs.type)) { byType.set(obs.type, new Set()); } byType.get(obs.type)!.add(obs.observable.name); }); byType.forEach((entities, type) => { console.log(`${type} (${entities.size}):`); Array.from(entities).slice(0, 5).forEach(name => { console.log(` - ${name}`); }); if (entities.size > 5) { console.log(` ... and ${entities.size - 5} more`); } console.log(); }); // Step 7: Analyze entity timestamps console.log('Step 7: Entity mentions with timestamps...\n'); const people = content.observations?.filter(obs => obs.type === ObservableTypes.Person ) || []; people.slice(0, 3).forEach(person => { console.log(`${person.observable.name}:`); person.occurrences?.slice(0, 3).forEach(occ => { if (occ.startTime !== undefined && occ.endTime !== undefined) { const minutes = Math.floor(occ.startTime / 60); const seconds = Math.floor(occ.startTime % 60); console.log(` ${minutes}:${seconds.toString().padStart(2, '0')} - Confidence: ${occ.confidence.toFixed(2)}`); } }); console.log(); }); // Step 8: Extract action items (Events) console.log('Step 8: Action items and deadlines...\n'); const events = content.observations?.filter(obs => obs.type === ObservableTypes.Event ) || []; if (events.length > 0) { console.log('Identified action items:'); events.forEach(event => { console.log(` - ${event.observable.name}`); // Show when mentioned const firstMention = event.occurrences?.[0]; if (firstMention?.startTime !== undefined) { const min = Math.floor(firstMention.startTime / 60); const sec = Math.floor(firstMention.startTime % 60); console.log(` Mentioned at: ${min}:${sec.toString().padStart(2, '0')}`); } }); } else { console.log('No action items identified'); } console.log('\nāœ“ Meeting analysis complete!'); ``` *** ## Step-by-Step Explanation ### Step 1: Create Transcription Workflow **Audio Preparation**: ```typescript preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.Deepgram, deepgram: { model: DeepgramModels.Nova2 } } }] } ``` **Transcription Service Options**: * **Deepgram**: Fast, accurate, cost-effective (recommended) * **AssemblyAI**: Good quality, speaker diarization support * **Whisper**: OpenAI's model, very accurate but slower **What Transcription Produces**: * Full text transcript * Timestamps per word/segment * Speaker diarization (who said what) * Confidence scores per segment ### Step 2: Supported Audio/Video Formats **Audio Formats**: * MP3, WAV, M4A, AAC, FLAC, OGG * Supported bitrates: 8kbps - 320kbps * Sample rates: 8kHz - 48kHz **Video Formats**: * MP4, MOV, AVI, MKV, WEBM * Audio track extracted automatically * Video analysis not performed (audio only) **Ingestion Sources**: ```typescript // From URL ingestUri('https://example.com/meeting.mp3') // From local file const audio = fs.readFileSync('./meeting.mp3'); const base64 = audio.toString('base64'); ingestEncodedFile({ name: 'meeting.mp3', data: base64, mimeType: 'audio/mpeg' }) // From cloud storage (via feed) createFeed({ type: FeedTypes.Site, site: { type: FeedServiceTypes.AzureFile, // Azure Blob Storage config } }) ``` ### Step 3: Processing Timeline **Transcription Time** (approximate): * 10-minute meeting: 1-2 minutes * 30-minute meeting: 3-5 minutes * 1-hour meeting: 5-10 minutes * 2-hour meeting: 10-20 minutes **Factors Affecting Speed**: * Audio quality (clean audio faster) * Number of speakers (more speakers slower) * Background noise (noisy audio slower) * File size and bitrate ### Step 4: Transcript Structure **Markdown Format**: ```markdown # Meeting Transcript ## Segment 1 (00:00 - 00:15) Speaker 1: Welcome everyone to the Q4 planning meeting... ## Segment 2 (00:15 - 00:45) Speaker 2: Thanks Kirk. I wanted to discuss the product roadmap... ## Segment 3 (00:45 - 01:20) Speaker 1: Great points. Let's talk about the Graphlit launch timeline... ``` **Accessing Transcript**: ```typescript const content = await graphlit.getContent(meetingId); // Full transcript const transcript = content.content.markdown || content.content.text; // Audio metadata const duration = content.content.audio?.duration; // seconds const channels = content.content.audio?.channels; const bitrate = content.content.audio?.bitrate; ``` ### Step 5: Entity Extraction from Transcript **Person Entities**: * Participants (from speaker labels) * People mentioned in discussion * Names in action items **Organization Entities**: * Companies discussed * Partners, clients, competitors * Departments, teams **Event Entities**: * Action items ("Send proposal by Friday") * Deadlines ("Launch date: October 15") * Follow-up meetings ("Schedule review call") **Product/Software Entities**: * Tools discussed * Products mentioned * Features planned **Category Entities**: * Topics, themes * Projects, initiatives * Meeting subjects ### Step 6: Timestamp Analysis **Occurrence Timestamps**: ```typescript occurrence: { startTime: 125.3, // Seconds from recording start endTime: 127.8, // Seconds from recording start confidence: 0.92 // Extraction confidence } ``` **Use Cases**: * Jump to specific entity mentions in playback * Create entity timeline visualization * Find when action items were assigned * Track discussion flow by entity **Format Timestamps for Display**: ```typescript function formatTime(seconds: number): string { const mins = Math.floor(seconds / 60); const secs = Math.floor(seconds % 60); return `${mins}:${secs.toString().padStart(2, '0')}`; } obs.occurrences?.forEach(occ => { if (occ.startTime !== undefined) { console.log(`${formatTime(occ.startTime)} - ${obs.observable.name}`); } }); ``` ### Step 7: Speaker Diarization **Identifying Speakers**: ```typescript // Deepgram and AssemblyAI provide speaker labels const transcript = content.content.markdown; // Parse speaker segments const speakerPattern = /Speaker (\d+):/g; const speakers = new Set(); let match; while ((match = speakerPattern.exec(transcript)) !== null) { speakers.add(match[1]); } console.log(`Number of speakers: ${speakers.size}`); ``` **Linking Speakers to Person Entities**: ```typescript // Cross-reference speaker labels with extracted Person entities const people = content.content.observations ?.filter(obs => obs.type === ObservableTypes.Person) || []; console.log('Participants:'); people.forEach(person => { console.log(` ${person.observable.name}`); // Match to speaker label if possible }); ``` *** ## Configuration Options ### Choosing Transcription Service **Deepgram (Recommended)**: ```typescript deepgram: { model: DeepgramModels.Nova2 } ``` * **Pros**: Fast, accurate, cost-effective, good speaker diarization * **Cons**: Requires internet connection * **Best for**: Most use cases, production **AssemblyAI**: ```typescript type: FilePreparationServiceTypes.AssemblyAi, assemblyAi: { model: AssemblyAiModels.Best } ``` * **Pros**: Very accurate, excellent speaker diarization * **Cons**: Slower, more expensive * **Best for**: High-quality transcription needs **Whisper (via Deepgram)**: ```typescript type: FilePreparationServiceTypes.Deepgram, deepgram: { model: DeepgramModels.WhisperLarge } ``` * **Pros**: Very accurate, multilingual support * **Cons**: Slower than Nova models * **Best for**: Non-English meetings, maximum accuracy ### Audio Quality Preprocessing **For Noisy Audio**: ```typescript preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.Deepgram, deepgram: { model: DeepgramModels.Nova2 }, // Preprocessing options (if supported) } }] } ``` **Tips for Better Transcription**: 1. Use high-quality recording equipment 2. Minimize background noise 3. Single speaker per microphone when possible 4. 16kHz+ sample rate recommended 5. Avoid heavy audio compression *** ## Variations ### Variation 1: Multi-Meeting Analysis Analyze a series of recurring meetings: ```typescript const meetingUrls = [ 'https://example.com/meetings/week1.mp4', 'https://example.com/meetings/week2.mp4', 'https://example.com/meetings/week3.mp4' ]; // Ingest all meetings const meetings = await Promise.all( meetingUrls.map(uri => graphlit.ingestUri(uri, undefined, undefined, undefined, undefined, { id: workflowId }) ) ); // Wait for all to process const waitForAll = async () => { const ids = meetings.map(m => m.ingestUri.id); let allDone = false; while (!allDone) { const statuses = await Promise.all( ids.map(id => graphlit.isContentDone(id)) ); allDone = statuses.every(s => s.isContentDone.result); if (!allDone) { await new Promise(resolve => setTimeout(resolve, 5000)); } } }; await waitForAll(); // Analyze trends over time const allEntities = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Person, ObservableTypes.Event] } }); console.log(`Total entities across ${meetings.length} meetings: ${allEntities.observables.results.length}`); ``` ### Variation 2: Action Item Tracker Extract and track action items: ```typescript interface ActionItem { description: string; assignee?: string; deadline?: string; meetingDate: Date; timestamp: number; } function extractActionItems(content: Content): ActionItem[] { const events = content.observations ?.filter(obs => obs.type === ObservableTypes.Event) || []; const actionItems: ActionItem[] = []; events.forEach(event => { // Look for action-like events const desc = event.observable.name.toLowerCase(); const isAction = desc.includes('send') || desc.includes('schedule') || desc.includes('prepare') || desc.includes('follow up') || desc.includes('review'); if (isAction) { actionItems.push({ description: event.observable.name, meetingDate: new Date(content.creationDate), timestamp: event.occurrences?.[0]?.startTime || 0 }); } }); return actionItems; } const actions = extractActionItems(meetingDetails.content); console.log(`Action items: ${actions.length}`); actions.forEach(action => { console.log(` - ${action.description}`); console.log(` At: ${formatTime(action.timestamp)}`); }); ``` ### Variation 3: Meeting Sentiment & Topic Analysis Analyze discussion topics and participant contributions: ```typescript interface MeetingInsights { duration: number; participantCount: number; topicsDiscussed: string[]; mostMentionedEntity: string; actionItemCount: number; } function analyzeMeeting(content: Content): MeetingInsights { const people = new Set( content.observations ?.filter(obs => obs.type === ObservableTypes.Person) .map(obs => obs.observable.name) || [] ); const categories = content.observations ?.filter(obs => obs.type === ObservableTypes.Category) .map(obs => obs.observable.name) || []; const events = content.observations ?.filter(obs => obs.type === ObservableTypes.Event) || []; // Find most mentioned entity const entityCounts = new Map(); content.observations?.forEach(obs => { const count = obs.occurrences?.length || 0; entityCounts.set(obs.observable.name, count); }); const mostMentioned = Array.from(entityCounts.entries()) .sort((a, b) => b[1] - a[1])[0]; return { duration: content.audio?.duration || 0, participantCount: people.size, topicsDiscussed: categories, mostMentionedEntity: mostMentioned?.[0] || 'None', actionItemCount: events.length }; } const insights = analyzeMeeting(meetingDetails.content); console.log('Meeting Insights:'); console.log(` Duration: ${Math.floor(insights.duration / 60)} minutes`); console.log(` Participants: ${insights.participantCount}`); console.log(` Topics: ${insights.topicsDiscussed.join(', ')}`); console.log(` Most discussed: ${insights.mostMentionedEntity}`); console.log(` Action items: ${insights.actionItemCount}`); ``` ### Variation 4: Searchable Meeting Archive Build searchable meeting repository: ```typescript // Ingest entire meeting archive const archive = await graphlit.createFeed({ name: "Meeting Archive", type: FeedTypes.Site, site: { type: FeedServiceTypes.AzureFile, // Point to Azure Blob container with recordings }, workflow: { id: workflowId } }); // Wait for all meetings to process await graphlit.isFeedDone(archive.createFeed.id); // Search meetings by entity const searchForPerson = async (personName: string) => { const personEntity = await graphlit.queryObservables({ search: personName, filter: { types: [ObservableTypes.Person] } }); if (personEntity.observables.results.length > 0) { const meetings = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Person, observable: { id: personEntity.observables.results[0].observable.id } }] }); return meetings.contents.results; } return []; }; // Find all meetings Kirk participated in const kirkMeetings = await searchForPerson("Kirk Marple"); console.log(`Kirk mentioned in ${kirkMeetings.length} meetings`); ``` ### Variation 5: Meeting Summary Generation Generate AI summaries with entity context: ```typescript // After transcription + extraction, generate summary const conversation = await graphlit.createConversation({ name: "Meeting Summary" }); // Provide meeting content as context const summary = await graphlit.promptConversation({ prompt: "Summarize this meeting, highlighting key decisions, action items, and participants.", id: conversation.createConversation.id, filter: { contents: [{ id: meeting.ingestUri.id }] } }); console.log('Meeting Summary:'); console.log(summary.message.message); // Extract structured data from summary const structuredPrompt = await graphlit.promptConversation({ prompt: `Extract from this meeting: 1. Key decisions made 2. Action items with assignees 3. Follow-up topics 4. Next steps Format as JSON.`, id: conversation.createConversation.id, filter: { contents: [{ id: meeting.ingestUri.id }] } }); console.log('\nStructured Summary:'); console.log(structuredPrompt.message.message); ``` *** ## Common Issues & Solutions ### Issue: Poor Transcription Quality **Problem**: Transcript has many errors, missing words. **Causes & Solutions**: 1. **Low audio quality**: Use higher bitrate recordings (128kbps+) 2. **Background noise**: Record in quiet environment, use noise cancellation 3. **Multiple speakers**: Use individual microphones when possible 4. **Heavy accents**: Try Whisper model (better multilingual support) 5. **Poor microphone**: Invest in quality recording equipment ```typescript // Try Whisper for difficult audio type: FilePreparationServiceTypes.Deepgram, deepgram: { model: DeepgramModels.WhisperLarge // Better multilingual support } ``` ### Issue: Processing Takes Too Long **Problem**: 1-hour meeting takes 30+ minutes to process. **Explanation**: Normal for certain conditions. **Timeline Expectations**: * Deepgram: \~10% of audio duration (6 min for 1-hour) * AssemblyAI: \~15% of audio duration (9 min for 1-hour) * Whisper: \~20-30% of audio duration (12-18 min for 1-hour) **Optimization**: * Use Deepgram for speed * Process shorter segments * Upload during off-peak hours ### Issue: No Speaker Diarization **Problem**: All speakers labeled as "Speaker 1". **Causes**: 1. Single audio channel (mono) 2. Poor speaker separation 3. Overlapping speech **Solution**: Use stereo recording with separate channels per speaker, or accept single speaker label. ### Issue: Missing Action Items **Problem**: No Event entities extracted for obvious action items. **Explanation**: Action items are implicit, not always explicitly stated. **Solution**: Use LLM to extract action items from transcript: ```typescript // After transcription, use RAG to extract actions const conversation = await graphlit.createConversation({ name: "Extract Actions" }); const actions = await graphlit.promptConversation({ prompt: "List all action items, deadlines, and follow-ups mentioned in this meeting. Format as a bullet list with assignees if mentioned.", id: conversation.createConversation.id, filter: { contents: [{ id: meetingId }] } }); console.log(actions.message.message); ``` *** ## Developer Hints ### Transcription Service Selection * **Deepgram**: Best default choice (speed + accuracy + cost) * **AssemblyAI**: When speaker diarization critical * **Whisper (Deepgram)**: Non-English meetings, multilingual support ### Audio Format Best Practices * **Bitrate**: 128kbps minimum, 256kbps recommended * **Sample rate**: 16kHz minimum, 44.1kHz recommended * **Channels**: Stereo preferred for multi-speaker * **Format**: WAV/FLAC for quality, MP3 for size ### Cost Optimization * Deepgram cheapest per minute * Compress large video files (audio track only needed) * Batch process during off-peak hours * Cache transcripts (don't re-transcribe) ### Meeting Entity Quality * **High confidence**: Participant names, company names * **Medium confidence**: Action items, deadlines * **Low confidence**: Implicit mentions, pronouns * **Filter threshold**: >=0.6 for meetings (lower than documents) ### Performance Tips * Process in background (don't block UI) * Show progress estimates (based on duration) * Cache transcripts for quick re-query * Parallel process multiple meetings * Use webhooks for completion notification *** ## Production Patterns ### Pattern from Meeting Intelligence Apps * Zoom/Meet recordings ā†’ automatic transcription * Entity extraction: participants, action items, topics * Searchable archive by person, topic, or date * Action item tracking dashboard * Meeting summary emails ### Enterprise Use Cases * **Sales calls**: Extract prospects, products, objections * **Customer support**: Track issues, customers, solutions * **Board meetings**: Decisions, financial mentions, strategic initiatives * **Team standups**: Tasks, blockers, sprint planning * **Training sessions**: Topics covered, questions, feedback *** # Build Knowledge Graph from PDF Documents ## Use Case: Build Knowledge Graph from PDF Documents ### User Intent "How do I extract entities from PDF documents to build a knowledge graph? Show me a complete workflow from PDF ingestion to querying entities." ### Operation **SDK Methods**: `createWorkflow()`, `ingestUri()`, `isContentDone()`, `getContent()`, `queryObservables()`\ **GraphQL**: Complete workflow + ingestion + entity querying\ **Entity**: PDF ā†’ Content ā†’ Observations ā†’ Observables (Knowledge Graph) ### Prerequisites * Graphlit project with API credentials * PDF documents to process (local files or URLs) * Understanding of entity types * Basic knowledge of workflows *** ### Complete Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { FilePreparationServiceTypes, EntityExtractionServiceTypes, ObservableTypes, EntityState } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); console.log('=== Building Knowledge Graph from PDF ===\n'); // Step 1: Create extraction workflow console.log('Step 1: Creating extraction workflow...'); const workflow = await graphlit.createWorkflow({ name: "PDF Entity Extraction", preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.ModelDocument // PDF, Word, Excel, etc. } }] }, extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Place, ObservableTypes.Event ] } }] } }); console.log(`āœ“ Created workflow: ${workflow.createWorkflow.id}\n`); // Step 2: Ingest PDF console.log('Step 2: Ingesting PDF document...'); const content = await graphlit.ingestUri('https://arxiv.org/pdf/2301.00001.pdf', "Research Paper", undefined, undefined, undefined, { id: workflow.createWorkflow.id }); console.log(`āœ“ Ingested: ${content.ingestUri.id}\n`); // Step 3: Wait for processing console.log('Step 3: Waiting for entity extraction...'); let isDone = false; while (!isDone) { const status = await graphlit.isContentDone(content.ingestUri.id); isDone = status.isContentDone.result; if (!isDone) { console.log(' Processing... (checking again in 2s)'); await new Promise(resolve => setTimeout(resolve, 2000)); } } console.log('āœ“ Extraction complete\n'); // Step 4: Retrieve content with entities console.log('Step 4: Retrieving extracted entities...'); const contentDetails = await graphlit.getContent(content.ingestUri.id); const observations = contentDetails.content.observations || []; console.log(`āœ“ Found ${observations.length} entity observations\n`); // Step 5: Analyze entities by type console.log('Step 5: Analyzing entities...\n'); const byType = new Map>(); observations.forEach(obs => { if (!byType.has(obs.type)) { byType.set(obs.type, new Set()); } byType.get(obs.type)!.add(obs.observable.name); }); byType.forEach((entities, type) => { console.log(`${type} (${entities.size} unique):`); Array.from(entities).slice(0, 5).forEach(name => { console.log(` - ${name}`); }); if (entities.size > 5) { console.log(` ... and ${entities.size - 5} more`); } console.log(); }); // Step 6: Query knowledge graph console.log('Step 6: Querying knowledge graph...\n'); // Get all unique people const people = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Person], states: [EntityState.Enabled] } }); console.log(`Total people in knowledge graph: ${people.observables.results.length}`); // Get all organizations const orgs = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Organization], states: [EntityState.Enabled] } }); console.log(`Total organizations in knowledge graph: ${orgs.observables.results.length}`); // Step 7: Find entity relationships console.log('\nStep 7: Analyzing entity co-occurrences...\n'); const cooccurrences: Array<{ person: string; organization: string; count: number }> = []; observations .filter(obs => obs.type === ObservableTypes.Person) .forEach(personObs => { observations .filter(obs => obs.type === ObservableTypes.Organization) .forEach(orgObs => { // Check if they appear on same pages const personPages = new Set( personObs.occurrences?.map(occ => occ.pageIndex) || [] ); const orgPages = new Set( orgObs.occurrences?.map(occ => occ.pageIndex) || [] ); const sharedPages = Array.from(personPages).filter(p => orgPages.has(p)); if (sharedPages.length > 0) { cooccurrences.push({ person: personObs.observable.name, organization: orgObs.observable.name, count: sharedPages.length }); } }); }); console.log('Top person-organization relationships:'); cooccurrences .sort((a, b) => b.count - a.count) .slice(0, 5) .forEach(({ person, organization, count }) => { console.log(` ${person} ā†” ${organization} (${count} pages)`); }); console.log('\nāœ“ Knowledge graph analysis complete!'); ``` *** ## Run asyncio.run(build\_kg\_from\_pdf()) ```` ### C# ```csharp using Graphlit; using Graphlit.Api.Input; var graphlit = new Graphlit(); Console.WriteLine("=== Building Knowledge Graph from PDF ===\n"); // Step 1: Create workflow Console.WriteLine("Step 1: Creating extraction workflow..."); var workflow = await graphlit.CreateWorkflow( name: "PDF Entity Extraction", preparation: new WorkflowPreparationInput { Jobs = new[] { new WorkflowPreparationJobInput { Connector = new FilePreparationConnectorInput { Type = FilePreparationServiceDocument } } } }, extraction: new WorkflowExtractionInput { Jobs = new[] { new WorkflowExtractionJobInput { Connector = new ExtractionConnectorInput { Type = ExtractionServiceModelDocument, ExtractedTypes = new[] { ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Place, ObservableTypes.Event } } } } } ); Console.WriteLine($"āœ“ Created workflow: {workflow.CreateWorkflow.Id}\n"); // Step 2: Ingest PDF Console.WriteLine("Step 2: Ingesting PDF document..."); var content = await graphlit.IngestUri( name: "Research Paper", uri: "https://arxiv.org/pdf/2301.00001.pdf", workflow: new EntityReferenceInput { Id = workflow.CreateWorkflow.Id } ); Console.WriteLine($"āœ“ Ingested: {content.IngestUri.Id}\n"); // (Continue with remaining steps...) ```` *** ### Step-by-Step Explanation #### Step 1: Create Extraction Workflow **Document Preparation**: * `FilePreparationServiceDocument` handles PDFs, Word, Excel, PowerPoint * Extracts text, tables, images * Preserves page structure and layout * Handles encrypted PDFs (if password provided) **Vision-Based Extraction**: * `ExtractionServiceModelDocument` uses vision models * Analyzes visual layout (charts, diagrams, tables) * Better for scanned PDFs * Extracts from images within PDFs **Entity Type Selection**: * Choose types relevant to your domain * More types = longer processing time * Start with Person, Organization, Place, Event #### Step 2: Ingest PDF Document **Ingestion Options**: ```typescript // From URL await graphlit.ingestUri('https://example.com/document.pdf', undefined, undefined, undefined, undefined, { id: workflowId }); // From local file (base64 encoded) const fileBuffer = fs.readFileSync('./document.pdf'); const base64 = fileBuffer.toString('base64'); await graphlit.ingestEncodedFile({ name: 'document.pdf', data: base64, mimeType: 'application/pdf', workflow: { id: workflowId } }); // From cloud storage (via feed) const feed = await graphlit.createFeed({ type: FeedTypes.Site, site: { type: FeedServiceTypes.AzureFile, // ... Azure Blob Storage config }, workflow: { id: workflowId } }); ``` #### Step 3: Poll for Completion **Processing Timeline**: * Small PDF (<10 pages): 30-60 seconds * Medium PDF (10-50 pages): 1-3 minutes * Large PDF (50-200 pages): 3-10 minutes * Very large PDF (200+ pages): 10-30 minutes **Polling Strategy**: ```typescript // Efficient polling with backoff let retries = 0; const maxRetries = 60; // 2 minutes max const delay = 2000; // 2 seconds while (retries < maxRetries) { const status = await graphlit.isContentDone(contentId); if (status.isContentDone.result) break; await new Promise(resolve => setTimeout(resolve, delay)); retries++; } ``` #### Step 4: Retrieve Extracted Entities **Full Content Details**: ```typescript const content = await graphlit.getContent(contentId); // Access entity observations const observations = content.content.observations || []; // Access content metadata console.log(`Pages: ${content.content.document?.pageCount}`); console.log(`File size: ${content.content.fileSize}`); console.log(`Created: ${content.content.creationDate}`); ``` #### Step 5: Analyze Entities **Group by Type**: ```typescript const entityGroups = observations.reduce((groups, obs) => { if (!groups[obs.type]) { groups[obs.type] = []; } groups[obs.type].push(obs.observable); return groups; }, {} as Record); ``` **Deduplicate**: ```typescript const uniqueEntities = new Map(); observations.forEach(obs => { uniqueEntities.set(obs.observable.id, obs.observable); }); ``` #### Step 6: Query Knowledge Graph After entities are extracted, they're available globally: ```typescript // All people across all content const allPeople = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Person] } }); // Search for specific person const kirkEntities = await graphlit.queryObservables({ search: "Kirk Marple", filter: { types: [ObservableTypes.Person] } }); ``` #### Step 7: Analyze Relationships **Co-occurrence Analysis**: * Entities on same page likely related * Frequency indicates relationship strength * Build relationship graph from co-occurrences **Cross-document Relationships**: ```typescript // Find content mentioning both entities const relatedContent = await graphlit.queryContents({ observations: [ { type: ObservableTypes.Person, observable: { id: personId } }, { type: ObservableTypes.Organization, observable: { id: orgId } } ] }); ``` *** ### Configuration Options #### Choosing Text vs Vision Extraction **Use Text Extraction (`ModelText`) When**: * PDFs are text-based (born-digital) * No important visual elements * Want faster/cheaper processing * Content is primarily textual **Use Vision Extraction (`ModelDocument`) When**: * PDFs are scanned documents * Contains important charts/diagrams * Mixed text and visual content * OCR quality is poor with text extraction #### Model Selection for Quality vs Speed ```typescript // High quality (slower, more expensive) const specGPT4 = await graphlit.createSpecification({ name: "GPT-4 Extraction", type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAIModels.Gpt4, temperature: 0.1 } }); // Balanced (recommended) const specGPT4o = await graphlit.createSpecification({ name: "GPT-4o Extraction", type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAIModels.Gpt4o, temperature: 0.1 } }); // Fast and cost-effective const specGemini = await graphlit.createSpecification({ name: "Gemini Extraction", type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.Google, gemini: { model: GeminiModels.Gemini15Pro, temperature: 0.1 } }); ``` *** ### Variations #### Variation 1: Legal Contract Analysis Extract parties, dates, obligations from legal documents: ```typescript const legalWorkflow = await graphlit.createWorkflow({ name: "Legal Contract Extraction", extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelDocument, extractedTypes: [ ObservableTypes.Person, // Parties ObservableTypes.Organization, // Companies ObservableTypes.Place, // Jurisdictions ObservableTypes.Event // Effective dates, deadlines ] } }] } }); // Analyze extracted obligations const contractContent = await graphlit.getContent(contentId); const events = contractContent.content.observations ?.filter(obs => obs.type === ObservableTypes.Event) || []; console.log('Contract deadlines:'); events.forEach(event => { console.log(` - ${event.observable.name}`); event.occurrences?.forEach(occ => { console.log(` Page ${(occ.pageIndex || 0) + 1}, Confidence: ${occ.confidence}`); }); }); ``` #### Variation 2: Research Paper Citation Network Build academic citation graphs: ```typescript const researchWorkflow = await graphlit.createWorkflow({ name: "Research Paper Extraction", extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.Person, // Authors ObservableTypes.Organization, // Institutions ObservableTypes.Category // Topics, keywords ] } }] } }); // Build author collaboration network const papers = await graphlit.queryContents({ workflows: [{ id: researchWorkflow.createWorkflow.id }] }); const collaborations = new Map>(); papers.contents.results.forEach(paper => { const authors = paper.observations ?.filter(obs => obs.type === ObservableTypes.Person) .map(obs => obs.observable.name) || []; // Record co-authorships for (let i = 0; i < authors.length; i++) { for (let j = i + 1; j < authors.length; j++) { const key = [authors[i], authors[j]].sort().join(' & '); if (!collaborations.has(key)) { collaborations.set(key, new Set()); } collaborations.get(key)!.add(paper.name); } } }); console.log('Top collaborations:'); Array.from(collaborations.entries()) .sort((a, b) => b[1].size - a[1].size) .slice(0, 10) .forEach(([authors, papers]) => { console.log(` ${authors}: ${papers.size} papers`); }); ``` #### Variation 3: Invoice/Receipt Processing Extract vendors, amounts, dates from financial documents: ```typescript const invoiceWorkflow = await graphlit.createWorkflow({ name: "Invoice Extraction", extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelDocument, // Vision for logos/stamps extractedTypes: [ ObservableTypes.Organization, // Vendor, customer ObservableTypes.Place, // Billing/shipping address ObservableTypes.Event, // Invoice date, due date ObservableTypes.Product // Line items ] } }] } }); // Extract invoice metadata const invoice = await graphlit.getContent(invoiceId); const vendor = invoice.content.observations ?.find(obs => obs.type === ObservableTypes.Organization); console.log(`Vendor: ${vendor?.observable.name}`); ``` #### Variation 4: Medical Records Analysis Extract medical entities from clinical documents: ```typescript const medicalWorkflow = await graphlit.createWorkflow({ name: "Medical Records Extraction", extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.Person, // Patients, doctors ObservableTypes.MedicalCondition, // Diagnoses ObservableTypes.MedicalDrug, // Medications ObservableTypes.MedicalProcedure, // Treatments ObservableTypes.MedicalTest // Lab tests ] } }] } }); // Analyze patient record const record = await graphlit.getContent(recordId); const conditions = record.content.observations ?.filter(obs => obs.type === ObservableTypes.MedicalCondition) || []; const drugs = record.content.observations ?.filter(obs => obs.type === ObservableTypes.MedicalDrug) || []; console.log('Diagnoses:', conditions.map(c => c.observable.name)); console.log('Medications:', drugs.map(d => d.observable.name)); ``` #### Variation 5: Batch PDF Processing Process multiple PDFs efficiently: ```typescript const pdfUrls = [ 'https://example.com/doc1.pdf', 'https://example.com/doc2.pdf', 'https://example.com/doc3.pdf', // ... more PDFs ]; // Ingest all PDFs const contentIds = await Promise.all( pdfUrls.map(uri => graphlit.ingestUri({ uri, workflow: { id: workflowId } }) .then(result => result.ingestUri.id) ) ); console.log(`Ingested ${contentIds.length} PDFs`); // Wait for all to complete const waitForAll = async () => { let allDone = false; while (!allDone) { const statuses = await Promise.all( contentIds.map(id => graphlit.isContentDone(id)) ); allDone = statuses.every(s => s.isContentDone.result); if (!allDone) { console.log('Processing...'); await new Promise(resolve => setTimeout(resolve, 5000)); } } }; await waitForAll(); console.log('All PDFs processed'); // Query all extracted entities const allEntities = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Person, ObservableTypes.Organization] } }); console.log(`Total entities extracted: ${allEntities.observables.results.length}`); ``` *** ### Common Issues & Solutions #### Issue: No Entities Extracted from Scanned PDF **Problem**: PDF is scanned image, text extraction fails. **Solution**: Use vision model + proper preparation: ```typescript preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.ModelDocument, document: { includeOCR: true // Enable OCR for scanned docs } } }] }, extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelDocument // Vision model } }] } ``` #### Issue: Encrypted PDF Won't Process **Problem**: PDF is password-protected. **Solution**: Provide password in preparation: ```typescript preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.ModelDocument, document: { password: 'document-password-here' } } }] } ``` #### Issue: Missing Entities from Images/Charts **Problem**: Text-based extraction misses visual elements. **Solution**: Use vision model extraction: ```typescript extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelDocument, // Analyzes images extractedTypes: [/* ... */] } }] } ``` #### Issue: Processing Takes Too Long **Problem**: Large PDF processing exceeds timeout. **Solutions**: 1. Split large PDFs into smaller chunks 2. Use faster model (GPT-4o instead of GPT-4) 3. Reduce number of entity types 4. Process in background, poll asynchronously ```typescript // Async processing pattern const content = await graphlit.ingestUri(pdfUrl, undefined, undefined, undefined, undefined, { id: workflowId }); // Don't block - return ID immediately console.log(`Processing started: ${content.ingestUri.id}`); // Poll in background job or webhook ``` *** ### Developer Hints #### PDF Processing Best Practices 1. **Check file size first**: >50MB PDFs may need special handling 2. **Test with sample page**: Validate extraction quality before batch 3. **Use appropriate model**: Vision for scanned, text for born-digital 4. **Monitor confidence scores**: Filter entities with confidence <0.7 5. **Handle failures gracefully**: PDFs can be corrupt or malformed #### Vision Model Selection * **GPT-4o Vision**: Best balance (recommended) * **Claude 3.5 Sonnet**: Good for complex layouts * **GPT-4 Vision**: Highest quality but slower/expensive #### Cost Optimization * Text extraction much cheaper than vision * GPT-4o significantly cheaper than GPT-4 * Extract only needed entity types * Batch processing for volume discounts #### Performance Optimization * Parallel ingestion up to 10 PDFs simultaneously * Poll every 2-5 seconds (not more frequently) * Cache entity results to avoid re-querying * Use collections to organize large sets of PDFs *** ### Production Patterns #### Pattern from Graphlit Samples `Graphlit_2024_09_13_Extract_People_Organizations_from_ArXiv_Papers.ipynb`: * Ingests ArXiv research papers (PDFs) * Extracts Person (authors), Organization (institutions) * Filters by confidence >=0.7 * Builds citation network from entities * Exports to CSV for analysis #### Pattern from Legal Tech * Process contracts in batch (100s of PDFs) * Extract parties, dates, obligations * Build contract database with entity index * Enable search by party or jurisdiction * Alert on approaching deadlines *** # Build Knowledge Graph from Slack Messages ## User Intent "How do I extract entities from Slack messages to build a knowledge graph?" ## What this does 1. Create an entity-extraction workflow. 2. Create a Slack feed (one feed per channel). 3. Wait for the initial sync. 4. Query message content and inspect extracted observations (entities). ## TypeScript (End-to-End Example) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentTypes, EntityExtractionServiceTypes, FeedListingTypes, FeedTypes, ObservableTypes, } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // 1) Create a workflow that extracts entities from content const workflow = await graphlit.createWorkflow({ name: 'Extract Entities from Slack Messages', extraction: { jobs: [ { connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Label], }, }, ], }, }); // 2) Create a Slack feed (one channel per feed) const feed = await graphlit.createFeed({ name: 'Slack #engineering', type: FeedTypes.Slack, slack: { type: FeedListingTypes.Past, channel: 'engineering', token: process.env.SLACK_TOKEN!, readLimit: 1000, includeAttachments: true, }, workflow: { id: workflow.createWorkflow.id }, }); // 3) Wait for initial sync while (true) { const status = await graphlit.isFeedDone(feed.createFeed.id); if (status.isFeedDone.result) break; await new Promise((r) => setTimeout(r, 5000)); } // 4) Query messages ingested by this feed const messages = await graphlit.queryContents({ types: [ContentTypes.Message], feeds: [{ id: feed.createFeed.id }], }); // 5) Observations (entities) are attached to content for (const c of messages.contents.results.slice(0, 5)) { const people = (c.observations ?? []).filter((o) => o.type === ObservableTypes.Person); console.log(c.name, people.map((p) => p.observable.name)); } ``` ## Notes * Slack feeds are scoped to a single `channel` name. If you want multiple channels, create multiple feeds. * Thread/reply behavior is handled by the connector; there is no separate boolean field in the current Slack feed input shape. # Knowledge Graph-Guided Search ## User Intent "How can the knowledge graph improve my search results? Show me entity expansion and graph-aware retrieval." ## Operation **SDK Methods**: `queryObservables()` + `queryContents()` combined\ **Concept**: Use entity relationships to enhance search\ **Use Case**: Graph-enhanced information retrieval ## Prerequisites * Knowledge graph with entities * Search queries * Understanding of entity relationships *** ## Key Concepts ### 1. Entity Expansion Expand search terms using entity variants: ```typescript // User searches "Kirk" // System expands to: // - "Kirk Marple" // - "K. Marple" // - kirk@graphlit.com ``` ### 2. Relationship-Aware Search Use entity relationships to broaden results: ```typescript // Search "Graphlit team" // 1. Find Graphlit (Organization) // 2. Find all People at Graphlit // 3. Return content mentioning any team member ``` ### 3. Entity Disambiguation Use context to identify correct entity: ```typescript // "Apple" could be company or fruit // Knowledge graph helps disambiguate ``` *** ## Complete Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { ObservableTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); async function graphGuidedSearch(query: string) { console.log(`Searching: "${query}"\n`); // Step 1: Find entities matching query const entities = await graphlit.queryObservables({ search: query }); console.log(`Found ${entities.observables.results.length} matching entities\n`); // Step 2: For each entity, find related entities const allEntityIds = new Set(); entities.observables.results.forEach(e => allEntityIds.add(e.observable.id)); for (const entity of entities.observables.results.slice(0, 3)) { const related = await graphlit.queryContents({ observations: [{ observable: { id: entity.observable.id } }] }); // Extract related entities related.contents.results.forEach(content => { content.observations?.forEach(obs => { allEntityIds.add(obs.observable.id); }); }); } console.log(`Expanded to ${allEntityIds.size} entities via relationships\n`); // Step 3: Search content mentioning any of these entities const expandedResults = await graphlit.queryContents({ search: query // Semantic search // Plus entity filter (if needed) }); console.log(`Results: ${expandedResults.contents.results.length}`); return expandedResults.contents.results; } await graphGuidedSearch("Graphlit"); ``` *** ## Benefits **Better Recall**: Find variations and related mentions\ **Entity Resolution**: "Kirk" expands to "Kirk Marple"\ **Context Awareness**: Relationships provide context\ **Disambiguation**: Choose correct entity meaning\ **Richer Results**: Include related entities *** ## Patterns ### Pattern 1: Team Search "Show me Company X team" ā†’ Find all people at Company X ### Pattern 2: Topic Expansion "AI research" ā†’ Find research + related papers + authors ### Pattern 3: Temporal Context Entity mentions over time with relationship context *** ## Developer Hints * Entity resolution improves recall * Graph relationships add context * Combine with semantic search * Better than pure keyword search * Cache entity expansions *** # Extract Medical Entities from Clinical Content ## User Intent "How do I extract medical entities (conditions, drugs, procedures, tests) from clinical documents and research papers? Show me how to build medical knowledge graphs for healthcare applications." ## Operation **SDK Methods**: `createWorkflow()`, `ingestUri()`, `isContentDone()`, `getContent()`, `queryObservables()`\ **GraphQL**: Medical content ingestion + extraction of 12 medical entity types\ **Entity**: Medical Content ā†’ Observations ā†’ Medical Observables (Clinical Knowledge Graph) ## Prerequisites * Graphlit project with API credentials * Medical/clinical documents (PDFs, research papers, clinical notes) * Understanding of medical entity types * Appropriate data privacy/HIPAA compliance measures *** ## Complete Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { ModelServiceTypes, ObservableTypes, SpecificationTypes } from 'graphlit-client/dist/generated/graphql-types'; import { FilePreparationServiceTypes, ExtractionServiceTypes, ObservableTypes, ModelServiceTypes, OpenAIModels } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); console.log('=== Building Medical Knowledge Graph ===\n'); // Step 1: Create high-quality medical extraction workflow console.log('Step 1: Creating medical entity extraction workflow...'); // Use GPT-4 for medical accuracy const spec = await graphlit.createSpecification({ name: "GPT-4 Medical Extraction", type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAIModels.Gpt4, // Best quality for medical temperature: 0.1 // Low temperature for consistency } }); const workflow = await graphlit.createWorkflow({ name: "Medical Entity Extraction", preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.ModelDocument // PDFs, Word, etc. } }] }, extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ // All 12 medical entity types ObservableTypes.MedicalCondition, // Diseases, symptoms, diagnoses ObservableTypes.MedicalDrug, // Medications, pharmaceuticals ObservableMedicalDrugClass, // Drug categories (antibiotics, etc.) ObservableTypes.MedicalProcedure, // Surgeries, treatments ObservableTypes.MedicalTest, // Lab tests, diagnostics ObservableTypes.MedicalStudy, // Clinical trials, research ObservableMedicalDevice, // Medical equipment, implants ObservableMedicalTherapy, // Therapies, treatments ObservableMedicalGuideline, // Clinical guidelines, protocols ObservableMedicalIndication, // Reasons for treatment ObservableMedicalContraindication, // Reasons to avoid treatment // Also extract non-medical entities for context ObservableTypes.Person, // Patients, doctors, researchers ObservableTypes.Organization // Hospitals, pharma companies ] } }] }, specification: { id: spec.createSpecification.id } }); console.log(`āœ“ Workflow: ${workflow.createWorkflow.id}\n`); // Step 2: Ingest clinical research paper console.log('Step 2: Ingesting clinical research paper...'); const paper = await graphlit.ingestUri('https://example.com/papers/clinical-trial.pdf', "Clinical Trial: Drug X for Condition Y", undefined, undefined, undefined, { id: workflow.createWorkflow.id }); console.log(`āœ“ Ingested: ${paper.ingestUri.id}\n`); // Step 3: Wait for extraction console.log('Step 3: Extracting medical entities...'); let isDone = false; while (!isDone) { const status = await graphlit.isContentDone(paper.ingestUri.id); isDone = status.isContentDone.result; if (!isDone) { console.log(' Processing...'); await new Promise(resolve => setTimeout(resolve, 3000)); } } console.log('āœ“ Extraction complete\n'); // Step 4: Retrieve extracted entities console.log('Step 4: Retrieving medical entities...'); const paperDetails = await graphlit.getContent(paper.ingestUri.id); const content = paperDetails.content; console.log(`āœ“ Document: ${content.name}`); console.log(` Pages: ${content.document?.pageCount}`); console.log(` Total entities: ${content.observations?.length || 0}\n`); // Step 5: Analyze by medical entity type console.log('Step 5: Analyzing medical entities...\n'); const medicalTypes = [ ObservableTypes.MedicalCondition, ObservableTypes.MedicalDrug, ObservableMedicalDrugClass, ObservableTypes.MedicalProcedure, ObservableTypes.MedicalTest, ObservableTypes.MedicalStudy, ObservableMedicalDevice, ObservableMedicalTherapy ]; medicalforEach(type => { const entities = content.observations?.filter(obs => obs.type === type) || []; const unique = new Set(entities.map(e => e.observable.name)); if (unique.size > 0) { console.log(`${type} (${unique.size}):`); Array.from(unique).slice(0, 5).forEach(name => { console.log(` - ${name}`); }); if (unique.size > 5) { console.log(` ... and ${unique.size - 5} more`); } console.log(); } }); // Step 6: Build drug-condition relationships console.log('Step 6: Analyzing drug-condition relationships...\n'); const drugs = content.observations?.filter(obs => obs.type === ObservableTypes.MedicalDrug ) || []; const conditions = content.observations?.filter(obs => obs.type === ObservableTypes.MedicalCondition ) || []; // Co-occurrence analysis const relationships: Array<{ drug: string; condition: string; confidence: number }> = []; drugs.forEach(drug => { conditions.forEach(condition => { // Check if they appear on same pages const drugPages = new Set(drug.occurrences?.map(occ => occ.pageIndex)); const condPages = new Set(condition.occurrences?.map(occ => occ.pageIndex)); const sharedPages = Array.from(drugPages).filter(p => condPages.has(p)); if (sharedPages.length > 0) { // Calculate average confidence const avgConf = ( (drug.occurrences?.reduce((sum, occ) => sum + occ.confidence, 0) || 0) / (drug.occurrences?.length || 1) + (condition.occurrences?.reduce((sum, occ) => sum + occ.confidence, 0) || 0) / (condition.occurrences?.length || 1) ) / 2; relationships.push({ drug: drug.observable.name, condition: condition.observable.name, confidence: avgConf }); } }); }); console.log('Drug-Condition relationships:'); relationships .sort((a, b) => b.confidence - a.confidence) .slice(0, 5) .forEach(({ drug, condition, confidence }) => { console.log(` ${drug} ā†” ${condition} (confidence: ${confidence.toFixed(2)})`); }); // Step 7: Query medical knowledge graph console.log('\nStep 7: Querying medical knowledge graph...\n'); // Get all conditions across all documents const allConditions = await graphlit.queryObservables({ filter: { types: [ObservableTypes.MedicalCondition] } }); console.log(`Total conditions in knowledge graph: ${allConditions.observables.results.length}`); // Get all drugs const allDrugs = await graphlit.queryObservables({ filter: { types: [ObservableTypes.MedicalDrug] } }); console.log(`Total drugs in knowledge graph: ${allDrugs.observables.results.length}`); console.log('\nāœ“ Medical knowledge graph complete!'); ``` *** ## Step-by-Step Explanation ### Step 1: Understanding Medical Entity Types Graphlit supports **12 medical entity types** (all fully supported, not beta): **Core Clinical Entities**: 1. **MedicalCondition**: * Diseases, symptoms, diagnoses * Examples: "Type 2 diabetes", "hypertension", "chest pain", "COVID-19" * Schema.org: `@type: "MedicalCondition"` 2. **MedicalDrug**: * Specific medications, pharmaceuticals * Examples: "metformin", "lisinopril", "aspirin", "Pfizer-BioNTech vaccine" * Schema.org: `@type: "Drug"` 3. **MedicalDrugClass**: * Categories of drugs * Examples: "antibiotics", "beta-blockers", "statins", "ACE inhibitors" * Schema.org: `@type: "DrugClass"` 4. **MedicalProcedure**: * Surgeries, treatments, interventions * Examples: "coronary artery bypass", "hip replacement", "chemotherapy" * Schema.org: `@type: "MedicalProcedure"` 5. **MedicalTest**: * Diagnostic tests, lab tests * Examples: "HbA1c test", "MRI scan", "blood pressure measurement" * Schema.org: `@type: "MedicalTest"` **Advanced Medical Entities**: 6. **MedicalStudy**: * Clinical trials, research studies * Examples: "Phase III trial", "randomized controlled trial", "cohort study" * Schema.org: `@type: "MedicalStudy"` 7. **MedicalDevice**: * Medical equipment, implants * Examples: "pacemaker", "insulin pump", "surgical robot", "stent" * Schema.org: `@type: "MedicalDevice"` 8. **MedicalTherapy**: * Therapies, treatment approaches * Examples: "physical therapy", "radiation therapy", "cognitive behavioral therapy" * Schema.org: `@type: "MedicalTherapy"` 9. **MedicalGuideline**: * Clinical guidelines, protocols * Examples: "WHO guidelines", "treatment protocol", "diagnostic criteria" * Schema.org: `@type: "MedicalGuideline"` 10. **MedicalIndication**: * Reasons for treatment * Examples: "indicated for hypertension", "approved for diabetes management" * Schema.org: `@type: "MedicalIndication"` 11. **MedicalContraindication**: * Reasons to avoid treatment * Examples: "contraindicated in pregnancy", "not for use with kidney disease" * Schema.org: `@type: "MedicalContraindication"` 12. **MedicalRiskFactor** (if supported): * Risk factors for conditions * Examples: "smoking", "obesity", "family history" ### Step 2: Model Selection for Medical Content **GPT-4 (Recommended for Medical)**: * Highest accuracy for medical terminology * Best understanding of clinical context * Lower false positive rate * More expensive but worth it for healthcare **GPT-4o**: * Good balance for less critical medical content * Faster processing * Lower cost * Acceptable for research papers, general medical content **Claude 3.5 Sonnet**: * Good alternative to GPT-4 * Strong medical knowledge * Handles long clinical documents well **NOT Recommended**: * Gemini: Less accurate for medical terminology * GPT-3.5: Too many medical errors ### Step 3: Clinical Document Types **Research Papers**: ```typescript // PubMed, ArXiv medical papers extractedTypes: [ ObservableTypes.MedicalCondition, ObservableTypes.MedicalDrug, ObservableTypes.MedicalStudy, ObservableTypes.MedicalProcedure, ObservableTypes.Person, // Authors, researchers ObservableTypes.Organization // Institutions ] ``` **Clinical Notes** (HIPAA considerations): ```typescript // Patient records, clinical summaries extractedTypes: [ ObservableTypes.MedicalCondition, // Diagnoses ObservableTypes.MedicalDrug, // Medications ObservableTypes.MedicalProcedure, // Treatments ObservableTypes.MedicalTest // Lab results // NOTE: Do NOT extract Person for patient privacy ] ``` **Drug Information Sheets**: ```typescript // Prescribing information, package inserts extractedTypes: [ ObservableTypes.MedicalDrug, ObservableMedicalDrugClass, ObservableMedicalIndication, ObservableMedicalContraindication, ObservableTypes.MedicalCondition // What it treats ] ``` **Clinical Guidelines**: ```typescript // Treatment protocols, best practices extractedTypes: [ ObservableMedicalGuideline, ObservableTypes.MedicalProcedure, ObservableTypes.MedicalTest, ObservableTypes.MedicalCondition ] ``` ### Step 4: Medical Entity Relationships **Drug-Condition Relationships**: * Co-occurrence on same pages * "Drug X is indicated for Condition Y" * "Patients with Condition Y treated with Drug X" **Procedure-Condition Relationships**: * "Procedure X performed for Condition Y" * Diagnostic procedures for conditions **Drug-Drug Interactions**: * Contraindications between drugs * Combination therapies **Test-Condition Relationships**: * Diagnostic tests for conditions * Monitoring tests for treated conditions ### Step 5: Confidence Scoring for Medical Entities **High Confidence (>=0.9)**: * Explicit medical terminology * Standard nomenclature (ICD, SNOMED CT terms) * Clear clinical context **Medium Confidence (0.7-0.9)**: * Common medical terms * Some ambiguity in context * Abbreviations with context **Low Confidence (<0.7)**: * Ambiguous terms * Incomplete information * Uncertain context **Recommended Threshold**: **>=0.75 for medical applications** (higher than general content) *** ## Configuration Options ### Precision vs Recall Tradeoff **High Precision** (fewer false positives): ```typescript // Use GPT-4, high confidence threshold specification: { model: OpenAIModels.Gpt4, temperature: 0.05 // Very low temperature } // Filter results const highConfidence = observations.filter(obs => obs.occurrences?.every(occ => occ.confidence >= 0.85) ); ``` **High Recall** (fewer false negatives): ```typescript // Extract all possible entities, filter later extractedTypes: [ // All 12 medical types ...allMedicalTypes ] // Lower confidence threshold const allEntities = observations.filter(obs => obs.occurrences?.some(occ => occ.confidence >= 0.6) ); ``` ### Domain-Specific Extraction **Cardiology**: ```typescript extractedTypes: [ ObservableTypes.MedicalCondition, // Heart diseases ObservableTypes.MedicalProcedure, // Cardiac procedures ObservableMedicalDevice, // Pacemakers, stents ObservableTypes.MedicalDrug, // Cardiac medications ObservableTypes.MedicalTest // ECG, stress tests ] ``` **Oncology**: ```typescript extractedTypes: [ ObservableTypes.MedicalCondition, // Cancer types ObservableMedicalTherapy, // Chemotherapy, radiation ObservableTypes.MedicalDrug, // Cancer drugs ObservableTypes.MedicalStudy, // Clinical trials ObservableTypes.MedicalProcedure // Surgeries, biopsies ] ``` **Pharmacology**: ```typescript extractedTypes: [ ObservableTypes.MedicalDrug, ObservableMedicalDrugClass, ObservableMedicalIndication, ObservableMedicalContraindication, ObservableTypes.MedicalCondition ] ``` *** ## Variations ### Variation 1: Drug Information Database Build comprehensive drug knowledge base: ```typescript // Ingest drug information sheets const drugDocs = [ 'https://example.com/drugs/metformin-info.pdf', 'https://example.com/drugs/lisinopril-info.pdf', // ... more drugs ]; const drugWorkflow = await graphlit.createWorkflow({ name: "Drug Information Extraction", extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.MedicalDrug, ObservableMedicalDrugClass, ObservableMedicalIndication, ObservableMedicalContraindication, ObservableTypes.MedicalCondition ] } }] } }); // Ingest all drug docs await Promise.all( drugDocs.map(uri => graphlit.ingestUri({ uri, workflow: { id: drugWorkflow.createWorkflow.id } }) ) ); // Query drug database const metformin = await graphlit.queryObservables({ search: "metformin", filter: { types: [ObservableTypes.MedicalDrug] } }); // Find what conditions it treats const conditions = await graphlit.queryContents({ observations: [ { type: ObservableTypes.MedicalDrug, observable: { id: metformin.observables.results[0].observable.id } }, { type: ObservableMedicalIndication, observable: { /* any indication */ } } ] }); ``` ### Variation 2: Clinical Trial Analysis Analyze clinical trial results: ```typescript const trialWorkflow = await graphlit.createWorkflow({ name: "Clinical Trial Extraction", extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.MedicalStudy, ObservableTypes.MedicalDrug, ObservableTypes.MedicalCondition, ObservableTypes.MedicalProcedure, ObservableTypes.Person, // Principal investigators ObservableTypes.Organization // Sponsors ] } }] } }); // Ingest clinical trial paper const trial = await graphlit.ingestUri('https://clinicaltrials.gov/study/NCT12345678/document.pdf', undefined, undefined, undefined, undefined, { id: trialWorkflow.createWorkflow.id }); // Wait and analyze const trialDetails = await graphlit.getContent(trial.ingestUri.id); // Extract trial metadata const studyType = trialDetails.content.observations ?.find(obs => obs.type === ObservableTypes.MedicalStudy); const drugTested = trialDetails.content.observations ?.find(obs => obs.type === ObservableTypes.MedicalDrug); const conditionTreated = trialDetails.content.observations ?.find(obs => obs.type === ObservableTypes.MedicalCondition); console.log(`Study: ${studyType?.observable.name}`); console.log(`Drug: ${drugTested?.observable.name}`); console.log(`Condition: ${conditionTreated?.observable.name}`); ``` ### Variation 3: Adverse Event Monitoring Track drug side effects and adverse events: ```typescript // Process adverse event reports const adverseWorkflow = await graphlit.createWorkflow({ name: "Adverse Event Extraction", extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.MedicalDrug, ObservableTypes.MedicalCondition, // Side effects ObservableMedicalContraindication ] } }] } }); // Ingest multiple adverse event reports // ... (similar to above) // Query for drug-side effect relationships const drugId = 'drug-observable-id'; const adverseEvents = await graphlit.queryContents({ observations: [{ type: ObservableTypes.MedicalDrug, observable: { id: drugId } }] }); // Extract side effects co-occurring with drug const sideEffects = new Map(); adverseEvents.contents.results.forEach(report => { report.observations ?.filter(obs => obs.type === ObservableTypes.MedicalCondition) .forEach(obs => { sideEffects.set( obs.observable.name, (sideEffects.get(obs.observable.name) || 0) + 1 ); }); }); console.log('Common side effects:'); Array.from(sideEffects.entries()) .sort((a, b) => b[1] - a[1]) .slice(0, 10) .forEach(([effect, count]) => { console.log(` ${effect}: ${count} reports`); }); ``` ### Variation 4: Medical Literature Review Build knowledge base from research papers: ```typescript // Process PubMed papers on specific topic const reviewWorkflow = await graphlit.createWorkflow({ name: "Literature Review Extraction", extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.MedicalCondition, ObservableTypes.MedicalDrug, ObservableTypes.MedicalProcedure, ObservableTypes.MedicalStudy, ObservableTypes.Person, // Authors ObservableTypes.Organization // Institutions ] } }] } }); // Ingest collection of papers const papers = [ 'https://pubmed.ncbi.nlm.nih.gov/paper1.pdf', 'https://pubmed.ncbi.nlm.nih.gov/paper2.pdf', // ... more papers ]; await Promise.all( papers.map(uri => graphlit.ingestUri({ uri, workflow: { id: reviewWorkflow.createWorkflow.id } }) ) ); // Analyze trends const allConditions = await graphlit.queryObservables({ filter: { types: [ObservableTypes.MedicalCondition] } }); // Find most researched conditions const researchCounts = new Map(); for (const condition of allConditions.observables.results) { const papers = await graphlit.queryContents({ observations: [{ type: ObservableTypes.MedicalCondition, observable: { id: condition.observable.id } }] }); researchCounts.set(condition.observable.name, papers.contents.results.length); } console.log('Most researched conditions:'); Array.from(researchCounts.entries()) .sort((a, b) => b[1] - a[1]) .slice(0, 10) .forEach(([condition, count]) => { console.log(` ${condition}: ${count} papers`); }); ``` ### Variation 5: Treatment Protocol Assistant RAG-based clinical decision support: ```typescript // After ingesting clinical guidelines and protocols const conversation = await graphlit.createConversation({ name: "Treatment Protocol Assistant" }); // Query for treatment recommendations const response = await graphlit.promptConversation({ prompt: "What is the recommended treatment protocol for a patient with Type 2 diabetes and hypertension?", id: conversation.createConversation.id // RAG will search across all ingested guidelines }); console.log('Treatment Recommendation:'); console.log(response.message.message); // Extract structured treatment plan const structured = await graphlit.promptConversation({ prompt: "Based on the guidelines, provide a structured treatment plan with: 1) First-line medications, 2) Monitoring tests, 3) Lifestyle modifications, 4) Follow-up schedule. Format as JSON.", id: conversation.createConversation.id }); console.log('\nStructured Plan:'); console.log(structured.message.message); ``` *** ## Common Issues & Solutions ### Issue: Medical Abbreviations Not Recognized **Problem**: "HTN", "DM", "CHF" not extracted as conditions. **Solution**: Medical abbreviations may have low confidence. Either: 1. Use lower confidence threshold (>=0.6) 2. Expand abbreviations in preprocessing 3. Train on medical-specific model (future feature) ### Issue: False Positives on Common Terms **Problem**: "Cold" extracted as MedicalCondition when discussing weather. **Solution**: Context-aware filtering: ```typescript // Check surrounding context or confidence const validConditions = conditions.filter(cond => cond.occurrences?.some(occ => occ.confidence >= 0.8) ); ``` ### Issue: Missing Drug-Condition Relationships **Problem**: Drug and condition mentioned but not linked. **Solution**: Use co-occurrence analysis (same page) or RAG queries: ```typescript // Find relationships via RAG const relationship = await graphlit.promptConversation({ prompt: "What conditions is Drug X used to treat according to this document?", filter: { contents: [{ id: documentId }] } }); ``` ### Issue: HIPAA Compliance Concerns **Problem**: Patient names being extracted from clinical notes. **Solution**: Don't extract Person entities from patient records: ```typescript extractedTypes: [ ObservableTypes.MedicalCondition, ObservableTypes.MedicalDrug, ObservableTypes.MedicalProcedure // DO NOT include ObservableTypes.Person for patient records ] ``` Also implement proper data handling: * Encrypt data at rest * Access controls * Audit logging * BAA with Graphlit (if processing PHI) *** ## Developer Hints ### Medical Entity Quality by Source * **High quality**: Published research papers, drug information sheets * **Medium quality**: Clinical guidelines, review articles * **Variable quality**: Clinical notes (abbreviations, typos) ### Model Recommendations by Use Case * **Clinical decision support**: GPT-4 (highest accuracy required) * **Research literature review**: GPT-4o (good balance) * **General medical knowledge**: Claude 3.5 Sonnet ### Confidence Thresholds * **Regulatory/clinical use**: >=0.85 * **Research/analysis**: >=0.75 * **Exploratory/discovery**: >=0.65 ### HIPAA and Privacy * Graphlit is HIPAA-compliant when properly configured * Sign BAA (Business Associate Agreement) * Use encryption, access controls * Don't extract identifiable patient information * Consider de-identification before ingestion ### Performance Optimization * Medical extraction is slower (complex terminology) * Expect 20-30% longer processing than general content * Batch process overnight for large volumes * Cache commonly queried entities *** ## Production Patterns ### Healthcare Use Cases * **Clinical decision support**: Query guidelines by condition * **Drug information lookup**: Interactive drug database * **Adverse event monitoring**: Track side effects across reports * **Literature review**: Automated systematic reviews * **Treatment protocol matching**: Match patients to protocols * **Medical education**: Interactive medical knowledge base ### Compliance Considerations * PHI (Protected Health Information) requires HIPAA compliance * De-identify data when possible * Implement access controls * Audit all queries * Regular security assessments * Data retention policies *** # Understanding Confidence Scores and Occurrences ## Use Case: Understanding Confidence Scores and Occurrences ### User Intent "What are confidence scores and occurrences in entity extraction? How do I use them to validate and filter entities?" ### Operation **SDK Method**: Access via `content.observations[].occurrences[]`\ **GraphQL Query**: `getContent` with observations\ **Entity**: Observation occurrence data ### Prerequisites * Content with extracted entities (workflow with extraction stage) * Understanding of observation model * Graphlit project with API credentials *** ### Complete Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; const graphlit = new Graphlit(); // Get content with entity observations const contentResponse = await graphlit.getContent('content-id-here'); const content = contentResponse.content; console.log(`\nAnalyzing entities in: ${content.name}\n`); // Iterate through all observations content.observations?.forEach(observation => { console.log(`\n${observation.type}: ${observation.observable.name}`); console.log(`Entity ID: ${observation.observable.id}`); console.log(`Total occurrences: ${observation.occurrences?.length || 0}\n`); // Analyze each occurrence observation.occurrences?.forEach((occurrence, index) => { console.log(` Occurrence #${index + 1}:`); console.log(` Confidence: ${occurrence.confidence.toFixed(3)}`); // Location context (varies by content type) if (occurrence.pageIndex !== undefined) { console.log(` Page: ${occurrence.pageIndex}`); } if (occurrence.boundingBox) { console.log(` Location: (${occurrence.boundingBox.left}, ${occurrence.boundingBox.top})`); console.log(` Size: ${occurrence.boundingBox.width} x ${occurrence.boundingBox.height}`); } if (occurrence.startTime !== undefined) { console.log(` Time: ${occurrence.startTime}s - ${occurrence.endTime}s`); } console.log(); }); // Calculate average confidence const avgConfidence = observation.occurrences!.reduce( (sum, occ) => sum + occ.confidence, 0 ) / observation.occurrences!.length; console.log(` Average confidence: ${avgConfidence.toFixed(3)}`); }); // Filter high-confidence entities const highConfidenceEntities = content.observations?.filter(obs => obs.occurrences?.some(occ => occ.confidence >= 0.8) ); console.log(`\nHigh-confidence entities (>=0.8): ${highConfidenceEntities?.length || 0}`); ``` *** ## Key differences: snake\_case methods content\_response = await graphlit.getContent(id="content-id-here") content = content\_response.content print(f"\nAnalyzing entities in: {content.name}\n") ## Iterate through observations for observation in content.observations or \[]: print(f"\n{observation.type}: {observation.observable.name}") print(f"Entity ID: {observation.observable.id}") print(f"Total occurrences: {len(observation.occurrences or \[])}\n") ``` # Analyze occurrences for idx, occurrence in enumerate(observation.occurrences or []): print(f" Occurrence #{idx + 1}:") print(f" Confidence: {occurrence.confidence:.3f}") if occurrence.page_index is not None: print(f" Page: {occurrence.page_index}") if occurrence.bounding_box: box = occurrence.bounding_box print(f" Location: ({box.left}, {box.top})") print(f" Size: {box.width} x {box.height}") if occurrence.start_time is not None: print(f" Time: {occurrence.start_time}s - {occurrence.end_time}s") print() # Average confidence avg_conf = sum(occ.confidence for occ in observation.occurrences) / len(observation.occurrences) print(f" Average confidence: {avg_conf:.3f}") ``` ## Filter high-confidence high\_conf = \[ obs for obs in content.observations or \[] if any(occ.confidence >= 0.8 for occ in obs.occurrences or \[]) ] print(f"\nHigh-confidence entities (>=0.8): {len(high\_conf)}") ```` ### C# ```csharp using Graphlit; var graphlit = new Graphlit(); // Key differences: PascalCase methods var contentResponse = await graphlit.GetContent(id: "content-id-here"); var content = contentResponse.Content; Console.WriteLine($"\nAnalyzing entities in: {content.Name}\n"); // Iterate through observations foreach (var observation in content.Observations ?? new List()) { Console.WriteLine($"\n{observation.Type}: {observation.Observable.Name}"); Console.WriteLine($"Entity ID: {observation.Observable.Id}"); Console.WriteLine($"Total occurrences: {observation.Occurrences?.Count ?? 0}\n"); // Analyze occurrences var occurrences = observation.Occurrences ?? new List(); for (int i = 0; i < occurrences.Count; i++) { var occurrence = occurrences[i]; Console.WriteLine($" Occurrence #{i + 1}:"); Console.WriteLine($" Confidence: {occurrence.Confidence:F3}"); if (occurrence.PageIndex.HasValue) Console.WriteLine($" Page: {occurrence.PageIndex}"); if (occurrence.BoundingBox != null) { var box = occurrence.BoundingBox; Console.WriteLine($" Location: ({box.Left}, {box.Top})"); Console.WriteLine($" Size: {box.Width} x {box.Height}"); } if (occurrence.StartTime.HasValue) Console.WriteLine($" Time: {occurrence.StartTime}s - {occurrence.EndTime}s"); Console.WriteLine(); } // Average confidence var avgConf = occurrences.Average(occ => occ.Confidence); Console.WriteLine($" Average confidence: {avgConf:F3}"); } ```` *** ### Step-by-Step Explanation #### Step 1: Understanding Confidence Scores **What is Confidence?** * Score from 0.0 (uncertain) to 1.0 (very certain) * Provided by LLM during entity extraction * Indicates extraction quality/reliability * Per-occurrence (same entity can have different confidences) **Confidence Ranges**: * **0.9 - 1.0**: Very high confidence (explicit mentions, clear context) * **0.7 - 0.9**: High confidence (standard mentions, good context) * **0.5 - 0.7**: Medium confidence (implicit mentions, unclear context) * **0.3 - 0.5**: Low confidence (ambiguous mentions, weak context) * **0.0 - 0.3**: Very low confidence (likely false positives) #### Step 2: Occurrence Context by Content Type **For Documents (PDF, Word, etc.)**: ```typescript occurrence: { confidence: 0.92, pageIndex: 5, // Zero-based page number boundingBox: { left: 100.5, // X coordinate (pixels or points) top: 250.3, // Y coordinate width: 150.2, // Width height: 20.5 // Height } } ``` **For Audio/Video (Transcripts)**: ```typescript occurrence: { confidence: 0.88, startTime: 125.3, // Seconds from start endTime: 127.8, // Seconds from start transcript: "Kirk Marple" // Optional: exact text } ``` **For Images**: ```typescript occurrence: { confidence: 0.85, boundingBox: { left: 50, // Pixel coordinates top: 100, width: 200, height: 150 } } ``` **For Text/Messages (Emails, Slack)**: ```typescript occurrence: { confidence: 0.95, // No spatial context - just presence in text } ``` #### Step 3: Multiple Occurrences Same entity mentioned multiple times = multiple occurrences: ```typescript // Example: "Kirk Marple" appears on pages 1, 5, and 12 observation: { type: ObservableTypes.Person, observable: { id: "obs-123", name: "Kirk Marple" }, occurrences: [ { confidence: 0.95, pageIndex: 0 }, // Page 1 (zero-based) { confidence: 0.88, pageIndex: 4 }, // Page 5 { confidence: 0.92, pageIndex: 11 } // Page 12 ] } ``` #### Step 4: Filtering by Confidence **High Precision (Few False Positives)**: ```typescript const highPrecision = content.observations?.filter(obs => obs.occurrences?.every(occ => occ.confidence >= 0.8) // ALL occurrences high ); ``` **High Recall (Few False Negatives)**: ```typescript const highRecall = content.observations?.filter(obs => obs.occurrences?.some(occ => occ.confidence >= 0.5) // ANY occurrence medium+ ); ``` **Balanced**: ```typescript const balanced = content.observations?.filter(obs => obs.occurrences?.some(occ => occ.confidence >= 0.7) // ANY occurrence high ); ``` *** ### Configuration Options #### Setting Confidence Thresholds **By Use Case**: **Legal/Compliance (High Precision)**: ```typescript const threshold = 0.85; // Very conservative const entities = content.observations?.filter(obs => obs.occurrences?.every(occ => occ.confidence >= threshold) ); ``` **Research/Discovery (High Recall)**: ```typescript const threshold = 0.5; // More permissive const entities = content.observations?.filter(obs => obs.occurrences?.some(occ => occ.confidence >= threshold) ); ``` **Production (Balanced)**: ```typescript const threshold = 0.7; // Recommended default const entities = content.observations?.filter(obs => obs.occurrences?.some(occ => occ.confidence >= threshold) ); ``` #### Analyzing Confidence Distribution ```typescript // Group by confidence range function analyzeConfidence(observations: Observation[]) { const distribution = { veryHigh: 0, // 0.9 - 1.0 high: 0, // 0.7 - 0.9 medium: 0, // 0.5 - 0.7 low: 0, // 0.3 - 0.5 veryLow: 0 // 0.0 - 0.3 }; observations.forEach(obs => { obs.occurrences?.forEach(occ => { if (occ.confidence >= 0.9) distribution.veryHigh++; else if (occ.confidence >= 0.7) distribution.high++; else if (occ.confidence >= 0.5) distribution.medium++; else if (occ.confidence >= 0.3) distribution.low++; else distribution.veryLow++; }); }); return distribution; } const dist = analyzeConfidence(content.observations || []); console.log('Confidence distribution:', dist); ``` *** ### Variations #### Variation 1: Find Entities by Page Number Locate entities on specific document pages: ```typescript function findEntitiesOnPage(observations: Observation[], pageNum: number) { const pageIndex = pageNum - 1; // Convert to zero-based return observations .map(obs => ({ entity: obs.observable, type: obs.type, occurrences: obs.occurrences?.filter(occ => occ.pageIndex === pageIndex ) || [] })) .filter(item => item.occurrences.length > 0); } const page5Entities = findEntitiesOnPage(content.observations || [], 5); console.log(`Entities on page 5: ${page5Entities.length}`); ``` #### Variation 2: Find Entities in Time Range (Audio/Video) Extract entities mentioned during specific timeframe: ```typescript function findEntitiesInTimeRange( observations: Observation[], startSec: number, endSec: number ) { return observations .map(obs => ({ entity: obs.observable, type: obs.type, occurrences: obs.occurrences?.filter(occ => occ.startTime !== undefined && occ.startTime >= startSec && occ.startTime <= endSec ) || [] })) .filter(item => item.occurrences.length > 0); } // Find entities mentioned between 5:00 and 10:00 const segment = findEntitiesInTimeRange(content.observations || [], 300, 600); console.log(`Entities in time range: ${segment.length}`); ``` #### Variation 3: Visual Entity Locator (PDFs with Bounding Boxes) Find entity positions for highlighting: ```typescript function getEntityLocations(observations: Observation[]) { const locations: Array<{ entity: string; type: string; page: number; box: { x: number; y: number; width: number; height: number }; confidence: number; }> = []; observations.forEach(obs => { obs.occurrences?.forEach(occ => { if (occ.boundingBox && occ.pageIndex !== undefined) { locations.push({ entity: obs.observable.name, type: obs.type, page: occ.pageIndex + 1, // Convert to 1-based box: { x: occ.boundingBox.left, y: occ.boundingBox.top, width: occ.boundingBox.width, height: occ.boundingBox.height }, confidence: occ.confidence }); } }); }); return locations; } const locations = getEntityLocations(content.observations || []); // Use for PDF highlighting UI ``` #### Variation 4: Confidence-Weighted Entity Ranking Rank entities by total confidence across all occurrences: ```typescript function rankEntitiesByConfidence(observations: Observation[]) { return observations .map(obs => { const totalConf = obs.occurrences?.reduce( (sum, occ) => sum + occ.confidence, 0 ) || 0; const avgConf = totalConf / (obs.occurrences?.length || 1); const occCount = obs.occurrences?.length || 0; return { entity: obs.observable, type: obs.type, averageConfidence: avgConf, occurrenceCount: occCount, totalConfidence: totalConf, score: totalConf * occCount // Weighted by frequency }; }) .sort((a, b) => b.score - a.score); } const ranked = rankEntitiesByConfidence(content.observations || []); console.log('Top entities by confidence:'); ranked.slice(0, 10).forEach((item, i) => { console.log(`${i + 1}. ${item.entity.name} (${item.type})`); console.log(` Avg conf: ${item.averageConfidence.toFixed(3)}, Count: ${item.occurrenceCount}`); }); ``` #### Variation 5: Occurrence Clustering (Find Dense Entity Regions) Find pages/sections with high entity density: ```typescript function findEntityClusters(observations: Observation[]) { const pageMap = new Map(); // page ā†’ entity count observations.forEach(obs => { obs.occurrences?.forEach(occ => { if (occ.pageIndex !== undefined) { pageMap.set( occ.pageIndex, (pageMap.get(occ.pageIndex) || 0) + 1 ); } }); }); // Sort pages by entity density return Array.from(pageMap.entries()) .map(([page, count]) => ({ page: page + 1, entityCount: count })) .sort((a, b) => b.entityCount - a.entityCount); } const clusters = findEntityClusters(content.observations || []); console.log('Pages with most entities:'); clusters.slice(0, 5).forEach(cluster => { console.log(` Page ${cluster.page}: ${cluster.entityCount} entities`); }); ``` *** ### Common Issues & Solutions #### Issue: All Confidences Are Low **Problem**: Most entities have confidence <0.5. **Solutions**: 1. **Upgrade model**: Use GPT-4 instead of Gemini for better quality 2. **Improve preparation**: Better OCR, cleaner text extraction 3. **Use vision models**: For scanned documents or images 4. **Check content quality**: Low-quality scans produce low confidence ```typescript // Solution: Use better model const spec = await graphlit.createSpecification({ name: "High-Quality Extraction", type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAIModels.Gpt4, // Better quality temperature: 0.1 } }); ``` #### Issue: Same Entity, Varying Confidence **Problem**: Multiple occurrences of same entity have wildly different confidence. **Explanation**: This is normal and indicates: * Some mentions are explicit ("Kirk Marple, CEO of Graphlit") * Some mentions are implicit ("Kirk said...") * Context quality varies throughout document **Solution**: Use average or maximum confidence: ```typescript // Use max confidence across occurrences const maxConf = Math.max(...observation.occurrences.map(occ => occ.confidence)); // Or use average const avgConf = observation.occurrences.reduce( (sum, occ) => sum + occ.confidence, 0 ) / observation.occurrences.length; ``` #### Issue: Bounding Boxes Missing **Problem**: Occurrences don't have bounding box coordinates. **Causes**: 1. Content type doesn't support spatial info (emails, text) 2. Used text extraction instead of vision extraction 3. Document doesn't have layout information **Solution**: Use vision model for PDFs: ```typescript extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelDocument, // Vision model extractedTypes: [/* ... */] } }] } ``` #### Issue: No Page Numbers in Occurrences **Problem**: Page index is undefined for document content. **Causes**: 1. Content isn't page-based (email, message, web page) 2. PDF preparation didn't preserve page structure 3. Used wrong preparation type **Solution**: Ensure proper document preparation: ```typescript preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.ModelDocument // Preserves pages } }] } ``` *** ### Developer Hints #### Confidence Interpretation by Model * **GPT-4**: Generally conservative, confidence >0.7 is very reliable * **GPT-4o**: Well-calibrated, confidence >0.75 recommended * **Claude 3.5**: Slightly optimistic, confidence >0.8 for high precision * **Gemini**: More variable, confidence >0.7 minimum recommended #### When to Use Occurrence Data * **Page index**: PDF highlighting, citation validation, page-specific queries * **Bounding boxes**: Visual annotation, entity location UI, layout analysis * **Timestamps**: Video playback navigation, meeting segment analysis * **Confidence**: Quality filtering, precision/recall tuning, validation #### Performance Considerations * Occurrence data increases response size * Filter occurrences client-side for specific pages/times * Don't fetch occurrence details if not needed * Cache occurrence analysis results #### Validation Strategies 1. **Manual review**: Check high-confidence entities first 2. **Cross-reference**: Verify entities across multiple content items 3. **Confidence distribution**: Expect most >0.7 for good extraction 4. **Frequency analysis**: Common entities should appear multiple times 5. **Context checking**: Use page/time context for validation *** # Entity Mention Timeline ## User Intent "When was this entity first/last mentioned? Show me a timeline of entity mentions." ## Operation **SDK Method**: `queryContents()` with entity filter and date sorting\ **GraphQL**: Timeline queries\ **Use Case**: Track entity mentions over time ## Prerequisites * Content with entities and dates * Entity to track * Understanding of date filtering *** ## Complete Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { ObservableTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Find entity const entity = await graphlit.queryObservables({ search: "Graphlit", filter: { types: [ObservableTypes.Organization] } }); const entityId = entity.observables.results[0]?.observable.id; // Get mentions sorted by date const mentions = await graphlit.queryContents({ observations: [{ observable: { id: entityId } }] orderBy: { creationDate: 'ASCENDING' } }); // First and last mentions const first = mentions.contents.results[0]; const last = mentions.contents.results[mentions.contents.results.length - 1]; console.log(`First mention: ${new Date(first.creationDate).toLocaleDateString()}`); console.log(`Last mention: ${new Date(last.creationDate).toLocaleDateString()}`); // Group by month const byMonth = new Map(); mentions.contents.results.forEach(content => { const month = content.creationDate.substring(0, 7); // YYYY-MM byMonth.set(month, (byMonth.get(month) || 0) + 1); }); console.log('\nMentions by month:'); Array.from(byMonth.entries()) .sort((a, b) => a[0].localeCompare(b[0])) .forEach(([month, count]) => { console.log(` ${month}: ${count}`); }); ``` *** ## Key Patterns ### 1. First/Last Mention ```typescript orderBy: { creationDate: 'ASCENDING' } // or 'DESCENDING' ``` ### 2. Time Range ```typescript filter: { creationDateRange: { from: '2024-01-01', to: '2024-12-31' } } ``` ### 3. Activity Trends Group by period and count mentions *** ## Use Cases **Entity Lifecycle**: Track when entity appears\ **Trending Analysis**: Rising/falling mentions\ **Event Detection**: Spikes in mentions\ **Historical Research**: Entity history\ **Activity Monitoring**: Real-time tracking *** ## Developer Hints * Use creationDate for timelines * Aggregate for trends * Visualize with charts * Alert on spikes * Cache timeline data *** # Understanding Entity Deduplication ## User Intent "How does Graphlit handle duplicate entities? Why do I sometimes see 'Kirk Marple' and 'Kirk' as separate entities?" ## Operation **Concept**: Entity resolution and deduplication\ **SDK Methods**: `queryObservables()` for finding potential duplicates\ **Entity**: Observable deduplication behavior ## Prerequisites * Knowledge graph with entities * Understanding of Observable model * Multiple content sources with entity mentions *** ## How Deduplication Works ### Automatic Deduplication **At Creation Time**: ```typescript // When entities are extracted, Graphlit attempts to deduplicate // "Kirk Marple" mentioned in 10 documents ā†’ 1 Observable with 10 Observations ``` **Deduplication Strategies**: 1. **Exact Name Match**: "Kirk Marple" = "Kirk Marple" 2. **Email Matching** (for Person): always same person 3. **URL Matching** (for Organization): graphlit.com domain 4. **Normalization**: Case-insensitive, whitespace trimming ### Race Conditions **Problem**: Parallel processing can create duplicates ```typescript // Document 1 and Document 2 processed simultaneously // Both mention "Kirk Marple" for first time // May create 2 separate Observables before deduplication runs ``` **When This Happens**: * Multiple feeds syncing in parallel * Batch ingestion of many documents * High-frequency entity creation **Future Improvement**: More robust entity resolution is roadmap item *** ## Finding Duplicates ### Query Similar Entities ```typescript import { Graphlit } from 'graphlit-client'; const graphlit = new Graphlit(); // Find all "Kirk" variants const kirkEntities = await graphlit.queryObservables({ search: "Kirk", filter: { types: [ObservableTypes.Person] } }); console.log(`Found ${kirkEntities.observables.results.length} entities matching "Kirk"`); kirkEntities.observables.results.forEach(entity => { console.log(` - ${entity.observable.name} (ID: ${entity.observable.id})`); console.log(` Email: ${entity.observable.properties?.email || 'N/A'}`); }); ``` ### Identify Potential Duplicates ```typescript function findPotentialDuplicates(entities: Observable[]): Map { const groups = new Map(); entities.forEach(entity => { const normalized = entity.observable.name.toLowerCase().trim(); // Group by normalized name if (!groups.has(normalized)) { groups.set(normalized, []); } groups.get(normalized)!.push(entity); }); // Return only groups with duplicates return new Map( Array.from(groups.entries()).filter(([_, group]) => group.length > 1) ); } const allPeople = await graphlit.queryObservables({ filter: { types: [ObservableTypes.Person] } }); const duplicates = findPotentialDuplicates(allPeople.observables.results); console.log(`Found ${duplicates.size} potential duplicate groups`); duplicates.forEach((group, name) => { console.log(`\n${name}:`); group.forEach(entity => { console.log(` - ID: ${entity.observable.id}`); }); }); ``` *** ## Working with Duplicates ### Query All Variants ```typescript // Find content mentioning ANY variant of entity const kirkVariants = await graphlit.queryObservables({ search: "Kirk", filter: { types: [ObservableTypes.Person] } }); const allKirkContent = await Promise.all( kirkVariants.observables.results.map(variant => graphlit.queryContents({ observations: [{ type: ObservableTypes.Person, observable: { id: variant.observable.id } }] }) ) ); const totalMentions = allKirkContent.reduce( (sum, result) => sum + result.contents.results.length, 0 ); console.log(`Total content mentioning Kirk: ${totalMentions}`); ``` ### Disambiguate by Properties ```typescript // Use email or other properties to identify correct entity const kirkWithEmail = kirkVariants.observables.results.find( entity => entity.observable.properties?.email === 'kirk@graphlit.com' ); if (kirkWithEmail) { console.log(`Canonical Kirk Marple: ${kirkWithEmail.observable.id}`); } ``` *** ## Best Practices ### 1. Use Unique Identifiers When available, use email (Person) or URL (Organization): ```typescript // Search by email for Person entities const person = await graphlit.queryObservables({ search: "kirk@graphlit.com", filter: { types: [ObservableTypes.Person] } }); ``` ### 2. Aggregate Across Variants Combine mentions from all duplicate entities: ```typescript const allVariants = await graphlit.queryObservables({ search: "Kirk Marple OR Kirk", filter: { types: [ObservableTypes.Person] } }); // Aggregate data from all variants ``` ### 3. Normalize in UI Display normalized names in UI: ```typescript function normalizeEntityName(name: string): string { // "kirk marple" and "Kirk Marple" ā†’ "Kirk Marple" return name.split(' ') .map(word => word.charAt(0).toUpperCase() + word.slice(1).toLowerCase()) .join(' '); } ``` *** ## Future Improvements **Roadmap Items** (not yet available): * Manual entity merging API * More sophisticated entity resolution * Cross-source entity linking * Entity resolution confidence scores **Current Limitations**: * No API to manually merge duplicates * Some race conditions create duplicates * Name variants not always linked **Workarounds**: * Query all variants and aggregate * Use unique identifiers (email, URL) * Filter by properties to disambiguate *** ## Developer Hints * Duplicates are rare but possible * Most common after parallel batch ingestion * Email/URL properties help disambiguation * Query by identifier, not just name * Future releases will improve resolution * Not a critical issue for most use cases *** # Comprehensive Entity Types Reference ## Observable: Comprehensive Entity Types Reference ### User Intent "What entity types can Graphlit extract? What properties does each type have?" ### Operation * **Concept**: Observable entity types (Schema.org-based) * **GraphQL Enum**: ObservableTypes * **Entity Types**: 20+ types supported * **Common Use Cases**: Understanding available entities, choosing extraction types, entity properties ### All Observable Types Graphlit supports 20+ entity types, all based on Schema.org standards for interoperability. ### Core Entity Types #### Person **Type**: `ObservableTypes.Person`\ **Schema.org**: `@type: "Person"`\ **Use For**: People, authors, contacts, team members, contributors **Properties**: ```typescript { name: string; // Full name email: string; // Email address givenName: string; // First name familyName: string; // Last name jobTitle: string; // Job title/role affiliation: string; // Company/organization telephone: string; // Phone number address: string; // Physical address } ``` **Examples**: * "Kirk Marple, CEO of Graphlit" * "Maria Garcia, Software Engineer at Microsoft" * Contact information in emails #### Organization **Type**: `ObservableTypes.Organization`\ **Schema.org**: `@type: "Organization"`\ **Use For**: Companies, teams, departments, organizations **Properties**: ```typescript { name: string; // Organization name description: string; // Description url: string; // Website URL foundingDate: string; // When founded location: string; // Headquarters location } ``` **Examples**: * "Graphlit, a context layer for AI agents" * "Microsoft Corporation" * "Stanford University" #### Place **Type**: `ObservableTypes.Place`\ **Schema.org**: `@type: "Place"`\ **Use For**: Locations, offices, venues, geographic entities **Properties**: ```typescript { name: string; // Place name address: { streetAddress: string; city: string; region: string; // State/province country: string; postalCode: string; }; geo: { latitude: number; longitude: number; }; h3: string; // H3 geospatial index } ``` **Examples**: * "Seattle, Washington" * "123 Main St, San Francisco, CA 94105" * "Golden Gate Park" #### Event **Type**: `ObservableTypes.Event`\ **Schema.org**: `@type: "Event"`\ **Use For**: Meetings, conferences, appointments, events **Properties**: ```typescript { name: string; // Event name description: string; // Event description startDate: string; // Start date/time endDate: string; // End date/time location: string; // Where it takes place attendees: string[]; // Who attends organizer: string; // Who organizes } ``` **Examples**: * "AWS re:Invent 2024" * "Q4 Planning Meeting" * "Product Launch Event" #### Product **Type**: `ObservableTypes.Product`\ **Schema.org**: `@type: "Product"`\ **Use For**: Products, tools, services **Properties**: ```typescript { name: string; // Product name brand: string; // Brand/manufacturer price: number; // Price description: string; // Product description manufacturer: string; // Who makes it } ``` **Examples**: * "iPhone 15 Pro" * "Tesla Model 3" * "Adobe Photoshop" #### Repo **Type**: `ObservableTypes.Repo`\ **Schema.org**: `@type: "SoftwareSourceCode"`\ **Use For**: GitHub/GitLab repositories, source code repos **Properties**: ```typescript { name: string; // Repository name url: string; // Repository URL owner: string; // Owner/organization description: string; // Repository description } ``` **Examples**: * "graphlit/graphlit-samples" * "facebook/react" * "microsoft/vscode" #### Software **Type**: `ObservableTypes.Software`\ **Schema.org**: `@type: "SoftwareApplication"`\ **Use For**: Software products, applications, tools **Properties**: ```typescript { name: string; // Software name version: string; // Version number description: string; // What it does manufacturer: string; // Who makes it } ``` **Examples**: * "Python 3.11" * "Docker" * "PostgreSQL 15" ### Classification Types #### Category **Type**: `ObservableTypes.Category`\ **Schema.org**: `@type: "Thing"` with category\ **Use For**: Topics, classifications, tags, subjects **Properties**: ```typescript { name: string; // Category name description: string; // Category description } ``` **Examples**: * "Machine Learning" * "Product Development" * "Financial Reports" #### Label **Type**: `ObservableLabel`\ **Schema.org**: `@type: "Thing"` with label\ **Use For**: Tags, labels, keywords **Properties**: ```typescript { name: string; // Label name color: string; // Label color (optional) description: string; // Label description } ``` **Examples**: * "urgent" * "bug" * "high-priority" ### Medical Entity Types (12 types) **All fully supported** - not beta #### MedicalCondition **Type**: `ObservableTypes.MedicalCondition`\ **Schema.org**: `@type: "MedicalCondition"`\ **Use For**: Diseases, symptoms, diagnoses **Examples**: * "Type 2 Diabetes" * "Hypertension" * "COVID-19" #### MedicalDrug **Type**: `ObservableTypes.MedicalDrug`\ **Schema.org**: `@type: "Drug"`\ **Use For**: Medications, pharmaceuticals **Examples**: * "Aspirin" * "Metformin" * "Lisinopril" #### MedicalDrugClass **Type**: `ObservableMedicalDrugClass`\ **Schema.org**: `@type: "DrugClass"`\ **Use For**: Drug categories **Examples**: * "Antibiotics" * "Beta blockers" * "Statins" #### MedicalProcedure **Type**: `ObservableTypes.MedicalProcedure`\ **Schema.org**: `@type: "MedicalProcedure"`\ **Use For**: Surgeries, medical procedures **Examples**: * "Appendectomy" * "MRI Scan" * "Blood Transfusion" #### MedicalTest **Type**: `ObservableTypes.MedicalTest`\ **Schema.org**: `@type: "MedicalTest"`\ **Use For**: Diagnostic tests, lab tests **Examples**: * "Complete Blood Count" * "X-Ray" * "Glucose Tolerance Test" #### MedicalStudy **Type**: `ObservableTypes.MedicalStudy`\ **Schema.org**: `@type: "MedicalStudy"`\ **Use For**: Clinical trials, research studies **Examples**: * "Phase III Clinical Trial" * "Longitudinal Study on Heart Disease" #### MedicalDevice **Type**: `ObservableMedicalDevice`\ **Schema.org**: `@type: "MedicalDevice"`\ **Use For**: Medical equipment, devices **Examples**: * "Pacemaker" * "Insulin Pump" * "Stethoscope" #### MedicalTherapy **Type**: `ObservableMedicalTherapy`\ **Schema.org**: `@type: "MedicalTherapy"`\ **Use For**: Treatments, therapies **Examples**: * "Physical Therapy" * "Radiation Therapy" * "Cognitive Behavioral Therapy" #### MedicalGuideline **Type**: `ObservableMedicalGuideline`\ **Schema.org**: `@type: "MedicalGuideline"`\ **Use For**: Clinical guidelines, protocols **Examples**: * "AHA Heart Disease Guidelines" * "CDC Vaccination Schedule" #### MedicalIndication **Type**: `ObservableMedicalIndication`\ **Schema.org**: `@type: "MedicalIndication"`\ **Use For**: Reasons for treatment **Examples**: * "High Blood Pressure" * "Bacterial Infection" #### MedicalContraindication **Type**: `ObservableMedicalContraindication`\ **Schema.org**: `@type: "MedicalContraindication"`\ **Use For**: Reasons to avoid treatment **Examples**: * "Pregnancy" * "Allergy to Penicillin" ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ObservableTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Extract all core types const workflow = await graphlit.createWorkflow({ name: "Comprehensive Extraction", preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.Document } }] }, extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ // Core types ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Place, ObservableTypes.Event, ObservableTypes.Product, ObservableTypes.Repo, ObservableTypes.Software, // Classification ObservableTypes.Category, ObservableTypes.Label ] } }] } }); // Extract medical types const medicalWorkflow = await graphlit.createWorkflow({ name: "Medical Extraction", extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.MedicalCondition, ObservableTypes.MedicalDrug, ObservableTypes.MedicalProcedure, ObservableTypes.MedicalTest, ObservableTypes.MedicalStudy ] } }] } }); ``` ### Use Cases by Domain #### Business Documents ```typescript extractedTypes: [ ObservableTypes.Person, // Employees, contacts ObservableTypes.Organization, // Companies, partners ObservableTypes.Place, // Offices, locations ObservableTypes.Event, // Meetings, conferences ObservableTypes.Product // Products discussed ] ``` #### Legal Documents ```typescript extractedTypes: [ ObservableTypes.Person, // Parties, attorneys ObservableTypes.Organization, // Companies involved ObservableTypes.Place, // Jurisdiction, addresses ObservableTypes.Event // Important dates ] ``` #### Technical Documentation ```typescript extractedTypes: [ ObservableTypes.Software, // Technologies mentioned ObservableTypes.Product, // Products/tools ObservableTypes.Repo, // Code repositories ObservableTypes.Organization // Companies/projects ] ``` #### Medical Literature ```typescript extractedTypes: [ ObservableTypes.MedicalCondition, ObservableTypes.MedicalDrug, ObservableTypes.MedicalProcedure, ObservableTypes.MedicalTest, ObservableTypes.MedicalStudy, ObservableTypes.Person, // Researchers, authors ObservableTypes.Organization // Research institutions ] ``` #### News Articles ```typescript extractedTypes: [ ObservableTypes.Person, // People mentioned ObservableTypes.Organization, // Companies, orgs ObservableTypes.Place, // Locations ObservableTypes.Event, // News events ObservableTypes.Category // Topics, subjects ] ``` #### GitHub/Code ```typescript extractedTypes: [ ObservableTypes.Repo, // Repositories ObservableTypes.Person, // Contributors ObservableTypes.Organization, // Companies, teams ObservableTypes.Software, // Technologies ObservableTypes.Label // Issue labels ] ``` ## All entity types available as enums extraction\_types = \[ ObservableTypes.PERSON, ObservableTypes.ORGANIZATION, ObservableTypes.PLACE, ObservableTypes.EVENT, ObservableTypes.PRODUCT, ObservableTypes.REPO, ObservableTypes.SOFTWARE, ObservableTypes.CATEGORY, ObservableTypes.LABEL, # Medical types ObservableTypes.MEDICAL\_CONDITION, ObservableTypes.MEDICAL\_DRUG, # ... etc ] ```` **C#**: ```csharp using Graphlit; // All entity types available as enum var extractionTypes = new[] { ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Place, ObservableTypes.Event, ObservableTypes.Product, ObservableTypes.Repo, ObservableTypes.Software, ObservableTypes.Category, ObservableLabel, // Medical types ObservableTypes.MedicalCondition, ObservableTypes.MedicalDrug, // ... etc }; ```` ### Developer Hints #### Choose Relevant Types ```typescript // Don't extract everything extractedTypes: [ /* all 20+ types */ ] // Slow, expensive, noisy // āœ“ Extract what you need extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization ] // Fast, focused, relevant ``` #### Schema.org Compliance ```typescript // All types map to Schema.org // Enables: // - JSON-LD export // - Standard property names // - Interoperability with other tools // - SEO-friendly structured data ``` #### Medical Types Fully Supported ```typescript // Not beta - production ready // All 12 medical types work extractedTypes: [ ObservableTypes.MedicalCondition, ObservableTypes.MedicalDrug, ObservableTypes.MedicalProcedure, // ... all 12 types available ] ``` ### Common Issues & Solutions **Issue**: Too many entities extracted (noise) **Solution**: Narrow down entity types ```typescript // Too broad extractedTypes: [ /* all types */ ] // āœ“ Focused extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization ] ``` **Issue**: Missing entities **Solution**: Check if type is included ```typescript // If you want products, must include: extractedTypes: [ ObservableTypes.Product // Don't forget to add ] ``` **Issue**: Want custom entity types **Solution**: Currently limited to Schema.org types ```typescript // Cannot create custom types // Use closest Schema.org type // Or use Category/Label for custom classifications ``` ### Production Example ```typescript async function demonstrateEntityTypes() { console.log('\n=== ENTITY TYPE DEMONSTRATION ===\n'); // Create workflows for different domains // 1. Business workflow const businessWorkflow = await graphlit.createWorkflow({ name: "Business Entity Extraction", extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Place, ObservableTypes.Event, ObservableTypes.Product ] } }] } }); console.log('āœ“ Created business workflow'); // 2. Technical workflow const techWorkflow = await graphlit.createWorkflow({ name: "Technical Entity Extraction", extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.Software, ObservableTypes.Repo, ObservableTypes.Product, ObservableTypes.Organization, ObservableTypes.Person ] } }] } }); console.log('āœ“ Created technical workflow'); // 3. Medical workflow const medicalWorkflow = await graphlit.createWorkflow({ name: "Medical Entity Extraction", extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.MedicalCondition, ObservableTypes.MedicalDrug, ObservableTypes.MedicalProcedure, ObservableTypes.MedicalTest, ObservableTypes.MedicalStudy, ObservableTypes.Person, ObservableTypes.Organization ] } }] } }); console.log('āœ“ Created medical workflow'); console.log('\nEntity type counts by workflow:'); console.log(` Business: 5 types`); console.log(` Technical: 5 types`); console.log(` Medical: 7 types`); console.log('\nAll available types:'); const allTypes = [ 'Person', 'Organization', 'Place', 'Event', 'Product', 'Repo', 'Software', 'Category', 'Label', 'MedicalCondition', 'MedicalDrug', 'MedicalDrugClass', 'MedicalProcedure', 'MedicalTest', 'MedicalStudy', 'MedicalDevice', 'MedicalTherapy', 'MedicalGuideline', 'MedicalIndication', 'MedicalContraindication' ]; console.log(` Total: ${alllength} types`); console.log(` ${alljoin(', ')}`); } await demonstrateEntityTypes(); ``` # Get Observable Entity Details ## User Intent "How do I retrieve entity details? Show me observable retrieval." ## Operation **SDK Method**: `queryObservables()`\ **Use Case**: Fetch entity information *** ## Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { ObservableTypes, } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Get observable details const result = await graphlit.queryObservables({ observables: [{ id: 'observable-id' }] }); const entity = result.observables?.results?.[0]; if (!entity) { console.log('Observable not found'); process.exit(0); } console.log('Entity Details:'); console.log(`Name: ${entity.observable.name}`); console.log(`Type: ${entity.type}`); console.log(`ID: ${entity.observable.id}`); // Properties vary by type if (entity.observable.properties) { console.log('\nProperties:'); console.log(JSON.stringify(entity.observable.properties, null, 2)); } // Count mentions const mentions = await graphlit.queryContents({ observations: [{ observable: { id: entity.observable.id } }] }); console.log(`\nMentioned in ${mentions.contents.results.length} documents`); ``` *** # Get Entity Details ## Observable: Get Entity Details ### User Intent "I want to retrieve full details for a specific entity" ### Operation * **SDK Method**: `graphlit.queryObservables()` * **GraphQL**: `observables` query * **Entity Type**: Observable * **Common Use Cases**: View entity summary details, diagnose extraction, check occurrence counts ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentFilterInput, ObservableTypes, } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); const observableId = 'observable-id-here'; const response = await graphlit.queryObservables({ contents: [{ id: 'content-id-here' }], observables: [{ id: observableId }], }); const observable = response.observables?.results?.[0]; if (!observable) { console.log('Observable not found'); process.exit(0); } console.log(`Entity: ${observable.observable.name}`); console.log(`Type: ${observable.type}`); if (observable.observable.description) { console.log(`Description: ${observable.observable.description}`); } ``` ## Query observable (snake\_case) result = await graphlit.client.query\_observables( filter={ "observables": \[{"id": observable\_id}], "contents": \[{"id": content\_id}] } ) observable = (result.observables.results or \[None])\[0] if observable: print(f"Entity: {observable.observable.name}") print(f"Type: {observable.type}") else: print("Observable not found") ```` **C#**: ```csharp using Graphlit; var client = new Graphlit(); var observableId = "observable-id-here"; var result = await graphlit.QueryObservables(new ContentFilter { Observables = new[] { new EntityReferenceFilter { Id = observableId } }, Contents = new[] { new EntityReferenceFilter { Id = contentId } } }); var observable = result.Observables?.Results?.FirstOrDefault(); if (observable is null) { Console.WriteLine("Observable not found"); return; } Console.WriteLine($"Entity: {observable.Observable?.Name}"); Console.WriteLine($"Type: {observable.Type}"); ```` ### Parameters * **`filter.observables`** (`EntityReferenceFilterInput[]`): One or more observable references (IDs, URIs, names) * **`filter.contents`** (`EntityReferenceFilterInput[]`, optional): Scope to specific content items * **`filter.types`** (`ObservableTypes[]`, optional): Restrict to certain entity types (Person, Organization, etc.) * **`filter.search`** (string, optional): Keyword search across observable names ### Response ```typescript { observables: { results: Array<{ type: ObservableTypes; observable: { id: string; name?: string; description?: string; }; }>; }; } ``` # Understanding the Observable/Observation Model ## Observable: Understanding the Observable/Observation Model ### User Intent "What's the difference between observables and observations? How does the entity model work?" ### Operation * **Concept**: Entity data model * **GraphQL Types**: Observable, Observation * **Entity Types**: Observable (entity), Observation (mention) * **Common Use Cases**: Understanding entities, entity relationships, provenance tracking ### The Model Explained **Observable** = The entity itself (e.g., Person "Kirk Marple" with unique ID)\ **Observation** = A specific mention/occurrence of that entity in content **Relationship**: Content ā†’ Many Observations ā†’ Many Observables ### Why This Architecture? #### 1. Deduplication "Kirk Marple" mentioned 100 times across documents ā†’ **1 Observable**, **100 Observations** #### 2. Confidence Scoring Each observation has its own confidence level (0.0-1.0) #### 3. Provenance Track exactly where each entity was found (page number, bounding box, timestamp) #### 4. Context Each observation includes location context (page, coordinates, time) ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ObservableTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Get content with observations const content = await graphlit.getContent('content-id'); console.log(`Content: ${content.content.name}`); console.log(`Observations: ${content.content.observations?.length || 0}`); // Iterate through observations content.content.observations?.forEach((observation, index) => { console.log(`\n${index + 1}. Observation:`); console.log(` Type: ${observation.type}`); console.log(` Entity: ${observation.observable.name}`); console.log(` Entity ID: ${observation.observable.id}`); console.log(` Observation ID: ${observation.id}`); // Occurrences (where/when mentioned) observation.occurrences?.forEach(occurrence => { console.log(` Occurrence:`); console.log(` Confidence: ${occurrence.confidence}`); console.log(` Page: ${occurrence.pageIndex}`); if (occurrence.boundingBox) { console.log(` Location: (${occurrence.boundingBox.left}, ${occurrence.boundingBox.top})`); } }); }); // Get observable (entity) details const observableResult = await graphlit.queryObservables({ observables: [ { id: content.content.observations?.[0]?.observable.id ?? '' } ] }); const observable = observableResult.observables?.results?.[0]?.observable; if (observable) { console.log(`\nObservable Details:`); console.log(` ID: ${observable.id}`); console.log(` Name: ${observable.name}`); console.log(` Type: ${observableResult.observables?.results?.[0]?.type}`); } ``` ### Data Flow ``` Content Ingestion ā†“ Workflow Processing (Extraction Stage) ā†“ LLM Extracts Entities from Text ā†“ For Each Extracted Entity: ā”œā”€ Create Observation (linked to content) ā”‚ ā”œā”€ Type (PERSON, ORGANIZATION, etc.) ā”‚ ā”œā”€ Confidence score ā”‚ ā”œā”€ Occurrence details (page, location, time) ā”‚ ā””ā”€ Text context ā†“ Entity Resolution (Deduplication) ā”œā”€ Check if entity already exists ā”œā”€ Match by name, properties, etc. ā””ā”€ Create new Observable OR link to existing ā†“ Observable Created/Updated ā”œā”€ Unique ID ā”œā”€ Canonical name ā”œā”€ Type ā”œā”€ Properties ā””ā”€ Links to all Observations ``` ### Key Differences #### Observable (Entity) ```typescript // Observable represents THE ENTITY { id: "obs-12345", // Unique entity ID name: "Kirk Marple", // Canonical name type: ObservableTypes.Person, // Entity type properties: { // Entity properties email: "kirk@graphlit.com", jobTitle: "CEO", affiliation: "Graphlit" }, // Links to ALL observations of this entity } ``` **Characteristics**: * One per unique entity * Deduplicated automatically * Has canonical properties * Persistent across content #### Observation (Mention) ```typescript // Observation represents A SPECIFIC MENTION { id: "observation-67890", // Unique observation ID type: ObservableTypes.Person, // Entity type observable: { // The entity being mentioned id: "obs-12345", name: "Kirk Marple" }, occurrences: [{ // Where mentioned confidence: 0.95, // How confident pageIndex: 3, // Which page boundingBox: { ... }, // Where on page type: OccurrenceLocation // Type of occurrence }], // Linked to specific content } ``` **Characteristics**: * One per mention in content * Linked to specific content * Has location context * Has confidence score * Multiple per observable ### Example: Same Entity, Multiple Observations ```typescript // Document 1 mentions "Kirk Marple" on page 3 // Document 2 mentions "Kirk Marple" on page 1 and page 5 // Document 3 mentions "Kirk" on page 2 // Results in: // - 1 Observable (id: obs-12345, name: "Kirk Marple") // - 4 Observations: // - Observation 1: Document 1, page 3, confidence 0.95 // - Observation 2: Document 2, page 1, confidence 0.98 // - Observation 3: Document 2, page 5, confidence 0.92 // - Observation 4: Document 3, page 2, confidence 0.85 (matched to "Kirk Marple") // Query to find all content mentioning Kirk Marple: const content = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Person, observable: { id: 'obs-12345' } }] }); // Returns: Document 1, Document 2, Document 3 ``` ### Graph Structure ``` Observable (Kirk Marple) ā†“ Observation 1 ā†’ Content A (page 3) Observation 2 ā†’ Content B (page 1) Observation 3 ā†’ Content B (page 5) Observation 4 ā†’ Content C (page 2) Observable (Graphlit) ā†“ Observation 5 ā†’ Content A (page 3) // Same content as Kirk Observation 6 ā†’ Content D (page 1) // This creates relationships: // - Kirk Marple ā†” Graphlit (co-occur in Content A) // - Kirk Marple appears in 3 documents // - Graphlit appears in 2 documents ``` ### Querying Patterns #### Get Content with Observations ```typescript const content = await graphlit.getContent('content-id'); // Check if has observations if (content.content.observations && content.content.observations.length > 0) { console.log(`Found ${content.content.observations.length} entity observations`); // Group by type const byType = new Map(); content.content.observations.forEach(obs => { byType.set(obs.type, (byType.get(obs.type) || 0) + 1); }); console.log('Entities by type:'); byType.forEach((count, type) => { console.log(` ${type}: ${count}`); }); } ``` #### Find All Content Mentioning Entity ```typescript // Find all content mentioning specific person const personContent = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Person, observable: { id: 'person-id' } }] }); console.log(`Found ${personContent.contents.results.length} documents mentioning this person`); // Each result has observations array showing WHERE in that document personContent.contents.results.forEach(content => { console.log(`\n${content.name}:`); content.observations?.forEach(obs => { obs.occurrences?.forEach(occ => { console.log(` - Page ${occ.pageIndex}, confidence: ${occ.confidence}`); }); }); }); ``` #### Get Observable Details ```typescript const observables = await graphlit.queryObservables({ observables: [{ id: 'observable-id' }] }); const observable = observables.observables?.results?.[0]; if (observable) { console.log(`Entity: ${observable.observable.name}`); console.log(`Type: ${observable.type}`); if (observable.type === ObservableTypes.Person) { console.log(`Email: ${observable.observable.properties?.email}`); console.log(`Job Title: ${observable.observable.properties?.jobTitle}`); } if (observable.type === ObservableTypes.Organization) { console.log(`URL: ${observable.observable.properties?.url}`); console.log(`Description: ${observable.observable.properties?.description}`); } } ``` ### Entity Resolution (Deduplication) #### Automatic at Creation Time ```typescript // When extraction finds "Kirk Marple" in multiple documents: // 1. First mention: Creates new Observable (obs-12345) // 2. Second mention: Matches to existing Observable (obs-12345) // 3. Result: 1 Observable, 2 Observations // Matching considers: // - Name similarity ("Kirk Marple" = "K. Marple") // - Email addresses (unique identifier for Person) // - URLs (unique identifier for Organization) // - Context and properties ``` #### Race Conditions **Note**: Parallel ingestion can create duplicates due to race conditions. This is a known limitation with future improvements planned. ```typescript // If two documents processed simultaneously: // - Both might create separate Observables for "Kirk Marple" // - Result: 2 Observables instead of 1 (duplicate) // - Future releases will improve entity resolution ``` ## Get content with observations content = await graphlit.getContent('content-id') print(f"Content: {content.content.name}") print(f"Observations: {len(content.content.observations or \[])}") ## Iterate observations for obs in content.content.observations or \[]: print(f"\nEntity: {obs.observable.name}") print(f"Type: {obs.type}") print(f"Entity ID: {obs.observable.id}") ``` # Occurrences for occ in obs.occurrences or []: print(f" Page: {occ.page_index}") print(f" Confidence: {occ.confidence}") ``` ## Get observable result = await graphlit.client.query\_observables( filter={"observables": \[{"id": "observable-id"}]} ) observable = (result.observables.results or \[None])\[0] if observable: print(f"Observable: {observable.observable.name}") ```` **C#**: ```csharp using Graphlit; var client = new Graphlit(); // Get content with observations var content = await graphlit.GetContent("content-id"); Console.WriteLine($"Content: {content.Content.Name}"); Console.WriteLine($"Observations: {content.Content.Observations?.Length ?? 0}"); // Iterate observations foreach (var obs in content.Content.Observations ?? Array.Empty()) { Console.WriteLine($"\nEntity: {obs.Observable.Name}"); Console.WriteLine($"Type: {obs.Type}"); Console.WriteLine($"Entity ID: {obs.Observable.Id}"); // Occurrences foreach (var occ in obs.Occurrences ?? Array.Empty()) { Console.WriteLine($" Page: {occ.PageIndex}"); Console.WriteLine($" Confidence: {occ.Confidence}"); } } // Get observable var observable = await graphlit.GetObservable("observable-id"); Console.WriteLine($"Observable: {observable.Observable.Name}"); ```` ### Developer Hints #### One Observable, Many Observations ```typescript // Think of it like: // Observable = The person "Kirk Marple" (unique entity) // Observations = All the times Kirk is mentioned (mentions) // Query by Observable ID to find ALL mentions: const allMentions = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Person, observable: { id: 'kirk-observable-id' } }] }); ``` #### Confidence Thresholds ```typescript // Filter low-confidence observations const content = await graphlit.getContent('content-id'); const highConfidence = content.content.observations?.filter(obs => obs.occurrences?.some(occ => occ.confidence >= 0.8) ); console.log(`High confidence entities: ${highConfidence?.length}`); ``` #### Observation IDs vs Observable IDs ```typescript // Observation ID: Unique to this mention observation.id // "observation-67890" // Observable ID: The entity being mentioned observation.observable.id // "obs-12345" // Use Observable ID to find all mentions across content ``` ### Common Issues & Solutions **Issue**: Same person appearing as multiple entities **Solution**: Entity resolution happens automatically, but race conditions can create duplicates ```typescript // This is a known limitation // Future releases will improve entity resolution // Currently, parallel ingestion can create duplicates ``` **Issue**: Want to find all mentions of an entity **Solution**: Query by Observable ID ```typescript const allMentions = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Person, observable: { id: 'observable-id' } }] }); ``` **Issue**: Need to access entity properties **Solution**: Use getObservable, not just the observation ```typescript // Observation only has id and name const obs = content.content.observations[0]; console.log(obs.observable.name); // āœ“ console.log(obs.observable.properties); // āœ— Not available // Get full observable for properties const observable = await graphlit.getObservable(obs.observable.id); console.log(observable.observable.properties); // āœ“ Full properties ``` ### Production Example ```typescript async function analyzeEntityMentions(contentId: string) { console.log('\n=== ENTITY MENTION ANALYSIS ===\n'); // Get content with observations const content = await graphlit.getContent(contentId); console.log(`Content: ${content.content.name}`); console.log(`Total observations: ${content.content.observations?.length || 0}`); if (!content.content.observations || content.content.observations.length === 0) { console.log('No entities extracted'); return; } // Group by type const byType = new Map(); content.content.observations.forEach(obs => { if (!byType.has(obs.type)) { byType.set(obs.type, []); } byType.get(obs.type)?.push(obs); }); console.log('\nEntities by type:'); byType.forEach((observations, type) => { console.log(` ${type}: ${observations.length}`); }); // Analyze each entity type for (const [type, observations] of byType.entries()) { console.log(`\n${type} entities:`); // Deduplicate by observable ID const uniqueObservables = new Map(); observations.forEach(obs => { if (!uniqueObservables.has(obs.observable.id)) { uniqueObservables.set(obs.observable.id, { id: obs.observable.id, name: obs.observable.name, mentions: [] }); } uniqueObservables.get(obs.observable.id)?.mentions.push(obs); }); console.log(` Unique entities: ${uniqueObservables.size}`); console.log(` Total mentions: ${observations.length}`); // Show entities with multiple mentions const multipleMentions = Array.from(uniqueObservables.values()) .filter(e => e.mentions.length > 1) .sort((a, b) => b.mentions.length - a.mentions.length); if (multipleMentions.length > 0) { console.log(` Entities with multiple mentions: ${multipleMentions.length}`); console.log(' Top mentioned:'); multipleMentions.slice(0, 5).forEach(entity => { console.log(` ${entity.name}: ${entity.mentions.length} mentions`); // Show pages where mentioned const pages = entity.mentions .flatMap((m: any) => m.occurrences || []) .map((o: any) => o.pageIndex) .filter(Boolean); console.log(` Pages: ${Array.from(new Set(pages)).sort((a, b) => a - b).join(', ')}`); }); } } // Confidence analysis const allOccurrences = content.content.observations .flatMap(obs => obs.occurrences || []); if (allOccurrences.length > 0) { const avgConfidence = allOccurrences .reduce((sum, occ) => sum + (occ.confidence || 0), 0) / allOccurrences.length; const highConfidence = allOccurrences.filter(occ => occ.confidence >= 0.8).length; const mediumConfidence = allOccurrences.filter(occ => occ.confidence >= 0.6 && occ.confidence < 0.8).length; const lowConfidence = allOccurrences.filter(occ => occ.confidence < 0.6).length; console.log(`\nConfidence Distribution:`); console.log(` High (ā‰„80%): ${highConfidence}`); console.log(` Medium (60-80%): ${mediumConfidence}`); console.log(` Low (<60%): ${lowConfidence}`); console.log(` Average: ${(avgConfidence * 100).toFixed(1)}%`); } } await analyzeEntityMentions('content-id'); ``` # Query Entities ## Observable: Query Entities ### User Intent "I want to query extracted entities (people, organizations, topics) from my knowledge graph" ### Operation * **SDK Method**: `graphlit.queryObservables()` * **GraphQL**: `queryObservables` query * **Entity Type**: Observable * **Common Use Cases**: Knowledge graph queries, entity search, relationship discovery, semantic network exploration ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ObservableTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Query all people and organizations const observablesResponse = await graphlit.queryObservables({ observableTypes: [ ObservableTypes.Person, ObservableTypes.Organization ] }); console.log(`Found ${observablesResponse.observables.results.length} entities`); observablesResponse.observables.results.forEach(entity => { console.log(`${entity.type}: ${entity.name}`); if (entity.description) { console.log(` Description: ${entity.description}`); } }); // Search for specific entity const searchResponse = await graphlit.queryObservables({ observableTypes: [ObservableTypes.Person], search: 'John Smith' }); // Query entities from specific content const contentEntities = await graphlit.queryObservables({ contents: [{ id: contentId }] }); console.log(`Entities in content: ${contentEntities.observables.results.length}`); ``` ## Query entities (snake\_case) response = await graphlit.queryObservables( filter=ObservableFilterInput( observable\_types=\[ ObservableTypes.PERSON, ObservableTypes.ORGANIZATION ] ) ) for entity in response.observables.results: print(f"{entity.type}: {entity.name}") ## Search for specific entity search\_response = await graphlit.queryObservables( filter=ObservableFilterInput( observable\_types=\[ObservableTypes.PERSON], search="John Smith" ) ) ```` **C#**: ```csharp using Graphlit; var client = new Graphlit(); // Query entities (PascalCase) var response = await graphlit.QueryObservables(new ObservableFilter { ObservableTypes = new[] { ObservableTypes.Person, ObservableTypes.Organization } }); foreach (var entity in response.Observables.Results) { Console.WriteLine($"{entity.Type}: {entity.Name}"); } // Search for specific entity var searchResponse = await graphlit.QueryObservables(new ObservableFilter { ObservableTypes = new[] { ObservableTypes.Person }, Search = "John Smith" }); ```` ### Parameters #### ObservableFilter (Optional) * **`observableTypes`** (ObservableTypes\[]): Types to query * `PERSON` - People, individuals * `ORGANIZATION` - Companies, institutions * `PLACE` - Locations, addresses * `PRODUCT` - Products, services * `EVENT` - Events, occurrences * `TOPIC` - Concepts, themes * Custom types (if defined in extraction workflow) * **`search`** (string): Search query for entity names * **`contents`** (EntityReferenceFilter\[]): Filter by source content * **`limit`** (int): Max results (default: 100) * **`offset`** (int): Pagination offset ### Response ```typescript { observables: { results: Observable[]; // Array of entities } } interface Observable { id: string; // Entity ID name: string; // Entity name type: ObservableType; // Entity type description?: string; // Entity description uri?: string; // Related URI occurrenceCount: number; // Times mentioned contents?: Content[]; // Source content } ``` ### Developer Hints #### Entities Must Be Extracted First **Important**: You must use an extraction workflow during content ingestion to create entities. ```typescript // 1. Create extraction workflow const workflow = await graphlit.createWorkflow({ name: 'Entity Extraction', extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, modelText: { specification: { id: extractionSpecId } } } }] } }); // 2. Ingest with workflow await graphlit.ingestUri( uri, undefined, undefined, undefined, true, { id: workflow.createWorkflow.id } ); // 3. Now query entities const entities = await graphlit.queryObservables(); ``` #### Filter by Entity Type ```typescript // Query specific types only const people = await graphlit.queryObservables({ observableTypes: [ObservableTypes.Person] }); const orgs = await graphlit.queryObservables({ observableTypes: [ObservableTypes.Organization] }); // Multiple types const peopleAndOrgs = await graphlit.queryObservables({ observableTypes: [ ObservableTypes.Person, ObservableTypes.Organization ] }); ``` #### Search for Entities ```typescript // Search by name const results = await graphlit.queryObservables({ search: 'Microsoft', observableTypes: [ObservableTypes.Organization] }); // Fuzzy matching works const fuzzy = await graphlit.queryObservables({ search: 'micro', // Finds "Microsoft", "Microservices", etc. }); ``` #### Entity Occurrence Count ```typescript // Get most frequently mentioned entities const entities = await graphlit.queryObservables({ observableTypes: [ObservableTypes.Person], limit: 10 }); // Sorted by occurrence count (most mentioned first) entities.observables.results.forEach(entity => { console.log(`${entity.name}: mentioned ${entity.occurrenceCount} times`); }); ``` ### Variations #### 1. Query All Entities Get all extracted entities: ```typescript const allEntities = await graphlit.queryObservables(); console.log(`Total entities: ${allEntities.observables.results.length}`); ``` #### 2. Query People Only Find all people: ```typescript const people = await graphlit.queryObservables({ observableTypes: [ObservableTypes.Person] }); people.observables.results.forEach(person => { console.log(person.name); }); ``` #### 3. Search for Specific Entity Find specific entity by name: ```typescript const microsoft = await graphlit.queryObservables({ search: 'Microsoft', observableTypes: [ObservableTypes.Organization] }); if (microsoft.observables.results.length > 0) { const entity = microsoft.observables.results[0]; console.log(`Found: ${entity.name}`); console.log(`Mentioned: ${entity.occurrenceCount} times`); } ``` #### 4. Query Entities from Specific Content Get entities from specific document: ```typescript const contentEntities = await graphlit.queryObservables({ contents: [{ id: contentId }] }); console.log('Entities in document:'); contentEntities.observables.results.forEach(entity => { console.log(`- ${entity.type}: ${entity.name}`); }); ``` #### 5. Top Mentioned Entities Get most frequently mentioned: ```typescript const topEntities = await graphlit.queryObservables({ observableTypes: [ ObservableTypes.Person, ObservableTypes.Organization ], limit: 20 // Top 20 }); console.log('Top mentioned entities:'); topEntities.observables.results.forEach((entity, index) => { console.log(`${index + 1}. ${entity.name} (${entity.occurrenceCount} mentions)`); }); ``` #### 6. Custom Entity Types Query custom extraction types: ```typescript // If you extracted custom types like "Contract", "Regulation" const contracts = await graphlit.queryObservables({ observableTypes: ['Contract'] // Custom type }); contracts.observables.results.forEach(contract => { console.log(`Contract: ${contract.name}`); }); ``` ### Common Issues **Issue**: No entities returned\ **Solution**: Ensure content was ingested with extraction workflow. Check workflow has entity extraction configured. **Issue**: Wrong entity types extracted\ **Solution**: Specify `observableTypes` in extraction workflow. Use `customTypes` for domain-specific entities. **Issue**: Entity names are incomplete\ **Solution**: Use better extraction model (Claude Sonnet 3.7, GPT-4o). Check source content quality. **Issue**: Too many irrelevant entities\ **Solution**: Filter by `observableTypes`. Use `search` parameter to narrow results. **Issue**: Entity occurrence count seems wrong\ **Solution**: Count reflects mentions across all content. Entity may be mentioned multiple times per document. ### Production Example **Entity discovery pipeline**: ```typescript // 1. Create extraction workflow const workflow = await graphlit.createWorkflow({ name: 'Entity Extraction', extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, modelText: { specification: { id: extractionSpecId } } } }] } }); // 2. Ingest content with extraction await graphlit.ingestUri( 'https://company-docs.com/page.html', undefined, undefined, undefined, true, { id: workflow.createWorkflow.id } ); // 3. Query extracted entities const entities = await graphlit.queryObservables({ observableTypes: [ ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Product ] }); // 4. Display results console.log(`\nExtracted ${entities.observables.results.length} entities:`); entities.observables.results.forEach(entity => { console.log(`\n${entity.type}: ${entity.name}`); console.log(` Mentions: ${entity.occurrenceCount}`); if (entity.description) { console.log(` Description: ${entity.description}`); } }); ``` **Entity search interface**: ```typescript // Search for entities by user query async function searchEntities(query: string, types?: ObservableTypes[]) { const results = await graphlit.queryObservables({ search: query, observableTypes: types, limit: 50 }); return results.observables.results.map(entity => ({ id: entity.id, name: entity.name, type: entity.type, mentions: entity.occurrenceCount, description: entity.description })); } // Usage const people = await searchEntities('John', [ObservableTypes.Person]); const companies = await searchEntities('Tech', [ObservableTypes.Organization]); ``` # Advanced Knowledge Graph Query Patterns ## User Intent "What are advanced patterns for querying my knowledge graph? Show me graph traversal, subgraph extraction, and complex relationship queries." ## Operation **SDK Methods**: `queryObservables()`, `queryContents()`, `queryGraph()` (if available)\ **GraphQL**: Complex graph queries with filters\ **Entity**: Advanced observable query patterns ## Prerequisites * Knowledge graph with extracted entities * Understanding of relationship queries * Familiarity with graph concepts *** ## Complete Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { ObservableTypes, EntityState } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); console.log('=== Advanced Graph Query Patterns ===\n'); // Pattern 1: Subgraph Extraction (N-hop neighborhood) console.log('Pattern 1: Extract 2-Hop Subgraph\n'); async function extractSubgraph( entityId: string, depth: number = 2 ): Promise<{ nodes: any[]; edges: any[] }> { const nodes = new Map(); const edges: any[] = []; const visited = new Set(); async function traverse(id: string, currentDepth: number) { if (currentDepth > depth || visited.has(id)) return; visited.add(id); const content = await graphlit.queryContents({ observations: [{ observable: { id } }] }); content.contents.results.forEach(item => { item.observations?.forEach(obs => { nodes.set(obs.observable.id, obs.observable); if (obs.observable.id !== id) { edges.push({ from: id, to: obs.observable.id }); } if (currentDepth < depth) { traverse(obs.observable.id, currentDepth + 1); } }); }); } await traverse(entityId, 0); return { nodes: Array.from(nodes.values()), edges }; } const subgraph = await extractSubgraph('entity-id-here', 2); console.log(`Subgraph: ${subgraph.nodes.length} nodes, ${subgraph.edges.length} edges\n`); // Pattern 2: Centrality Calculation (most connected entities) console.log('Pattern 2: Entity Centrality\n'); async function calculateCentrality( entityType: ObservableTypes ): Promise> { const entities = await graphlit.queryObservables({ filter: { types: [entityType] } }); const centrality: Array<{ name: string; degree: number }> = []; for (const entity of entities.observables.results.slice(0, 20)) { // Limit for demo const content = await graphlit.queryContents({ observations: [{ observable: { id: entity.observable.id } }] }); // Count unique connected entities const connected = new Set(); content.contents.results.forEach(item => { item.observations?.forEach(obs => { if (obs.observable.id !== entity.observable.id) { connected.add(obs.observable.id); } }); }); centrality.push({ name: entity.observable.name, degree: connected.size }); } return centrality.sort((a, b) => b.degree - a.degree); } const topEntities = await calculateCentrality(ObservableTypes.Person); console.log('Most connected people:'); topEntities.slice(0, 5).forEach((e, i) => { console.log(`${i + 1}. ${e.name}: ${e.degree} connections`); }); console.log('\nāœ“ Advanced graph queries complete!'); ``` *** ## Key Patterns ### 1. Subgraph Extraction Extract neighborhood around entity: * N-hop traversal * Collect nodes and edges * Export for visualization ### 2. Path Finding Find shortest path between entities: * Breadth-first search * Limited depth * Return path as array ### 3. Community Detection Find clusters of related entities: * Co-occurrence analysis * Connected components * Group by relationships ### 4. Centrality Metrics Rank entities by importance: * Degree centrality (connection count) * Betweenness centrality (bridge entities) * PageRank-style scoring ### 5. Temporal Patterns Analyze graph changes over time: * First/last entity mentions * Relationship formation timeline * Entity lifecycle tracking *** ## Common Patterns **Ego Network**: Extract all entities connected to one entity\ **Star Schema**: Find entities of type B connected to entity A\ **Triangle Closure**: Find mutual connections (A-B, B-C, A-C)\ **Weak Links**: Find rarely co-occurring entities\ **Strong Links**: Find frequently co-occurring entities *** ## Developer Hints * Cache content queries to avoid repeated API calls * Limit graph traversal depth (max 3 hops) * Use pagination for large result sets * Parallelize independent queries * Export to graph visualization tools (D3.js, Cytoscape) *** # Query Knowledge Graph ## Observable: Query Knowledge Graph ### User Intent "I want to explore entity relationships and visualize my knowledge graph" ### Operation * **SDK Method**: `graphlit.queryContentsGraph()` * **GraphQL**: `queryContentsGraph` query * **Entity Type**: Content/Observable (graph relationships) * **Common Use Cases**: Visualize knowledge graphs, explore entity relationships, discover connections, graph analytics ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ObservableTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Query knowledge graph for specific content const graphResponse = await graphlit.queryContentsGraph({ contents: [{ id: contentId }] }); console.log('Knowledge Graph:'); console.log(`Nodes: ${graphResponse.graph.nodes?.length || 0}`); console.log(`Edges: ${graphResponse.graph.edges?.length || 0}`); // Access nodes (entities) graphResponse.graph.nodes?.forEach(node => { console.log(`\n${node.type}: ${node.name}`); if (node.description) { console.log(` Description: ${node.description}`); } }); // Access edges (relationships) graphResponse.graph.edges?.forEach(edge => { const from = graphResponse.graph.nodes?.find(n => n.id === edge.from); const to = graphResponse.graph.nodes?.find(n => n.id === edge.to); console.log(`${from?.name} ā†’ ${edge.type} ā†’ ${to?.name}`); }); // Query graph across all content const fullGraph = await graphlit.queryContentsGraph(); console.log(`\nFull knowledge graph:`); console.log(`Total entities: ${fullGraph.graph.nodes?.length || 0}`); console.log(`Total relationships: ${fullGraph.graph.edges?.length || 0}`); ``` ## Query knowledge graph (snake\_case) graph\_response = await graphlit.queryContentsGraph( filter=ContentFilterInput( contents=\[EntityReferenceFilterInput(id=content\_id)] ) ) print(f"Nodes: {len(graph\_response.graph.nodes or \[])}") print(f"Edges: {len(graph\_response.graph.edges or \[])}") ## Access nodes for node in (graph\_response.graph.nodes or \[]): print(f"{node.type}: {node.name}") ## Access edges for edge in (graph\_response.graph.edges or \[]): print(f"Relationship: {edge.from\_name} ā†’ {edge.type} ā†’ {edge.to\_name}") ```` **C#**: ```csharp using Graphlit; var client = new Graphlit(); // Query knowledge graph (PascalCase) var graphResponse = await graphlit.QueryContentsGraph(new ContentFilter { Contents = new[] { new EntityReferenceFilter { Id = contentId } } }); Console.WriteLine($"Nodes: {graphResponse.Graph.Nodes?.Count ?? 0}"); Console.WriteLine($"Edges: {graphResponse.Graph.Edges?.Count ?? 0}"); // Access nodes foreach (var node in graphResponse.Graph.Nodes ?? new List()) { Console.WriteLine($"{node.Type}: {node.Name}"); } // Access edges foreach (var edge in graphResponse.Graph.Edges ?? new List()) { Console.WriteLine($"{edge.From} ā†’ {edge.Type} ā†’ {edge.To}"); } ```` ### Parameters #### ContentFilter (Optional) * **`contents`** (EntityReferenceFilter\[]): Filter by specific content * Query graph for specific documents * **`collections`** (EntityReferenceFilter\[]): Filter by collection * **`observableTypes`** (ObservableTypes\[]): Filter by entity types * `PERSON`, `ORGANIZATION`, `PLACE`, etc. ### Response ```typescript { graph: { nodes: GraphNode[]; // Entities edges: GraphEdge[]; // Relationships } } interface GraphNode { id: string; // Entity ID name: string; // Entity name type: ObservableType; // Entity type description?: string; // Entity description } interface GraphEdge { from: string; // Source entity ID to: string; // Target entity ID type: string; // Relationship type weight?: number; // Relationship strength } ``` ### Developer Hints #### Graph Must Be Built First **Important**: Knowledge graph requires content with extraction workflow. ```typescript // 1. Create extraction workflow const workflow = await graphlit.createWorkflow({ name: 'Entity Extraction', extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, modelText: { specification: { id: extractionSpecId } } } }] } }); // 2. Ingest with extraction await graphlit.ingestUri( uri, undefined, undefined, undefined, true, { id: workflow.createWorkflow.id } ); // 3. Now query graph const graph = await graphlit.queryContentsGraph(); ``` #### Filter by Content ```typescript // Graph for specific document const docGraph = await graphlit.queryContentsGraph({ contents: [{ id: contentId }] }); // Graph for collection const collectionGraph = await graphlit.queryContentsGraph({ collections: [{ id: collectionId }] }); // Entire knowledge graph const fullGraph = await graphlit.queryContentsGraph(); ``` #### Analyze Relationships ```typescript const graph = await graphlit.queryContentsGraph(); // Find most connected entities const connectionCount = new Map(); graph.graph.edges?.forEach(edge => { connectionCount.set(edge.from, (connectionCount.get(edge.from) || 0) + 1); connectionCount.set(edge.to, (connectionCount.get(edge.to) || 0) + 1); }); // Sort by connections const sorted = Array.from(connectionCount.entries()) .sort((a, b) => b[1] - a[1]) .slice(0, 10); console.log('Top 10 most connected entities:'); sorted.forEach(([nodeId, count]) => { const node = graph.graph.nodes?.find(n => n.id === nodeId); console.log(`${node?.name}: ${count} connections`); }); ``` #### Visualize Graph ```typescript // Export to visualization format (e.g., vis.js, d3.js) function exportGraphForVisualization(graph: any) { const nodes = graph.graph.nodes?.map(node => ({ id: node.id, label: node.name, title: node.description, group: node.type })); const edges = graph.graph.edges?.map(edge => ({ from: edge.from, to: edge.to, label: edge.type, value: edge.weight })); return { nodes, edges }; } const graph = await graphlit.queryContentsGraph(); const vizData = exportGraphForVisualization(graph); // Use with visualization library console.log(JSON.stringify(vizData, null, 2)); ``` ### Variations #### 1. Query Full Knowledge Graph Get entire graph: ```typescript const graph = await graphlit.queryContentsGraph(); console.log(`Knowledge Graph:`); console.log(`- Entities: ${graph.graph.nodes?.length || 0}`); console.log(`- Relationships: ${graph.graph.edges?.length || 0}`); ``` #### 2. Graph for Specific Document Single document graph: ```typescript const docGraph = await graphlit.queryContentsGraph({ contents: [{ id: contentId }] }); console.log(`Document entities: ${docGraph.graph.nodes?.length || 0}`); ``` #### 3. Graph by Entity Type Filter by entity types: ```typescript const peopleGraph = await graphlit.queryContentsGraph({ observableTypes: [ ObservableTypes.Person, ObservableTypes.Organization ] }); console.log('People and Organizations network'); ``` #### 4. Collection Knowledge Graph Graph for collection: ```typescript const collectionGraph = await graphlit.queryContentsGraph({ collections: [{ id: collectionId }] }); console.log(`Collection graph:`); console.log(`- Entities: ${collectionGraph.graph.nodes?.length || 0}`); console.log(`- Relationships: ${collectionGraph.graph.edges?.length || 0}`); ``` #### 5. Find Entity Connections Explore specific entity relationships: ```typescript const graph = await graphlit.queryContentsGraph(); // Find entity by name const targetEntity = graph.graph.nodes?.find( n => n.name.includes('Microsoft') ); if (targetEntity) { // Find all connections const connections = graph.graph.edges?.filter( e => e.from === targetEntity.id || e.to === targetEntity.id ); console.log(`${targetEntity.name} has ${connections?.length || 0} connections`); connections?.forEach(edge => { const other = graph.graph.nodes?.find( n => n.id === (edge.from === targetEntity.id ? edge.to : edge.from) ); console.log(`- ${edge.type}: ${other?.name}`); }); } ``` #### 6. Graph Analytics Analyze graph structure: ```typescript const graph = await graphlit.queryContentsGraph(); // Count by entity type const typeCounts = graph.graph.nodes?.reduce((acc, node) => { acc[node.type] = (acc[node.type] || 0) + 1; return acc; }, {} as Record); console.log('Entity Distribution:'); Object.entries(typeCounts || {}).forEach(([type, count]) => { console.log(` ${type}: ${count}`); }); // Relationship type distribution const edgeTypes = graph.graph.edges?.reduce((acc, edge) => { acc[edge.type] = (acc[edge.type] || 0) + 1; return acc; }, {} as Record); console.log('\nRelationship Types:'); Object.entries(edgeTypes || {}).forEach(([type, count]) => { console.log(` ${type}: ${count}`); }); ``` ### Common Issues **Issue**: Empty graph returned\ **Solution**: Ensure content was ingested with extraction workflow. Check entities were actually extracted. **Issue**: No relationships/edges\ **Solution**: Relationships are automatically inferred from co-occurrence and context. More content = more relationships. **Issue**: Graph too large to visualize\ **Solution**: Filter by specific content or collections. Use entity type filters to reduce scope. **Issue**: Missing expected entities\ **Solution**: Check extraction workflow configuration. Try better extraction model (Claude Sonnet 3.7). ### Production Example **Knowledge graph visualization pipeline**: ```typescript // 1. Query knowledge graph const graph = await graphlit.queryContentsGraph({ collections: [{ id: collectionId }] }); console.log('=== KNOWLEDGE GRAPH ANALYSIS ===\n'); // 2. Analyze graph structure const nodeCount = graph.graph.nodes?.length || 0; const edgeCount = graph.graph.edges?.length || 0; console.log(`Total Entities: ${nodeCount}`); console.log(`Total Relationships: ${edgeCount}`); console.log(`Average Connections: ${(edgeCount / nodeCount * 2).toFixed(2)}\n`); // 3. Find central entities (most connected) const connectionCount = new Map(); graph.graph.edges?.forEach(edge => { connectionCount.set(edge.from, (connectionCount.get(edge.from) || 0) + 1); connectionCount.set(edge.to, (connectionCount.get(edge.to) || 0) + 1); }); const topEntities = Array.from(connectionCount.entries()) .sort((a, b) => b[1] - a[1]) .slice(0, 10); console.log('Top 10 Central Entities:'); topEntities.forEach(([nodeId, count]) => { const node = graph.graph.nodes?.find(n => n.id === nodeId); console.log(` ${count} connections: ${node?.name} (${node?.type})`); }); // 4. Export for visualization const vizData = { nodes: graph.graph.nodes?.map(node => ({ id: node.id, label: node.name, title: node.description || node.name, group: node.type, value: connectionCount.get(node.id) || 1 })), edges: graph.graph.edges?.map(edge => ({ from: edge.from, to: edge.to, label: edge.type, value: edge.weight || 1 })) }; // Save for visualization tool console.log('\nGraph exported for visualization'); // Use with vis.js, d3.js, Cytoscape.js, etc. ``` **Entity relationship explorer**: ```typescript // Find relationships for specific entity async function exploreEntityRelationships(entityName: string) { const graph = await graphlit.queryContentsGraph(); // Find entity const entity = graph.graph.nodes?.find( n => n.name.toLowerCase().includes(entityName.toLowerCase()) ); if (!entity) { console.log(`Entity "${entityName}" not found`); return; } console.log(`\n=== ${entity.name} (${entity.type}) ===`); if (entity.description) { console.log(`Description: ${entity.description}`); } // Find all relationships const outgoing = graph.graph.edges?.filter(e => e.from === entity.id) || []; const incoming = graph.graph.edges?.filter(e => e.to === entity.id) || []; console.log(`\nOutgoing Relationships (${outgoing.length}):`); outgoing.forEach(edge => { const target = graph.graph.nodes?.find(n => n.id === edge.to); console.log(` ${edge.type} ā†’ ${target?.name} (${target?.type})`); }); console.log(`\nIncoming Relationships (${incoming.length}):`); incoming.forEach(edge => { const source = graph.graph.nodes?.find(n => n.id === edge.from); console.log(` ${source?.name} (${source?.type}) ā†’ ${edge.type}`); }); } // Usage await exploreEntityRelationships('Microsoft'); await exploreEntityRelationships('John Smith'); ``` # Query Entity Relationships in Knowledge Graph ## User Intent "How do I query relationships between entities? Show me how to find all people at an organization, products by a company, or events at a location." ## Operation **SDK Methods**: `queryObservables()` with relationship filters, `queryContents()` with entity filters\ **GraphQL Query**: `queryObservables`, `queryContents`\ **Entity**: Observable relationships and co-occurrence patterns ## Prerequisites * Graphlit project with extracted entities (knowledge graph built) * Understanding of Observable/Observation model * Content with entity extraction completed *** ## Complete Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { ObservableTypes, EntityState } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); console.log('=== Querying Entity Relationships ===\n'); // Example 1: Find all people at Graphlit console.log('Example 1: Person ā†’ Organization\n'); // First, find the Graphlit organization entity const graphlitOrg = await graphlit.queryObservables({ search: "Graphlit", filter: { types: [ObservableTypes.Organization], states: [EntityState.Enabled] } }); if (graphlitOrg.observables.results.length > 0) { const orgId = graphlitOrg.observables.results[0].observable.id; console.log(`Found organization: ${graphlitOrg.observables.results[0].observable.name}\n`); // Find all content mentioning Graphlit const graphlitContent = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Organization, observable: { id: orgId } }] }); // Extract all people mentioned in that content const peopleAtGraphlit = new Map(); graphlitContent.contents.results.forEach(content => { content.observations ?.filter(obs => obs.type === ObservableTypes.Person) .forEach(obs => { peopleAtGraphlit.set(obs.observable.id, { id: obs.observable.id, name: obs.observable.name }); }); }); console.log(`People at Graphlit: ${peopleAtGraphlit.size}`); Array.from(peopleAtGraphlit.values()).slice(0, 5).forEach(person => { console.log(` - ${person.name}`); }); if (peopleAtGraphlit.size > 5) { console.log(` ... and ${peopleAtGraphlit.size - 5} more`); } } console.log('\n---\n'); // Example 2: Find all events at a location console.log('Example 2: Event ā†’ Place\n'); // Find a place entity const seattle = await graphlit.queryObservables({ search: "Seattle", filter: { types: [ObservableTypes.Place], states: [EntityState.Enabled] } }); if (seattle.observables.results.length > 0) { const placeId = seattle.observables.results[0].observable.id; console.log(`Found place: ${seattle.observables.results[0].observable.name}\n`); // Find content mentioning both events and this place const contentWithPlace = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Place, observable: { id: placeId } }] }); // Extract events from that content const eventsAtPlace = new Set(); contentWithPlace.contents.results.forEach(content => { content.observations ?.filter(obs => obs.type === ObservableTypes.Event) .forEach(obs => { eventsAtPlace.add(obs.observable.name); }); }); console.log(`Events in Seattle: ${eventsAtPlace.size}`); Array.from(eventsAtPlace).slice(0, 5).forEach(event => { console.log(` - ${event}`); }); } console.log('\n---\n'); // Example 3: Find products by organization console.log('Example 3: Product ā†’ Organization\n'); // Find Microsoft const microsoft = await graphlit.queryObservables({ search: "Microsoft", filter: { types: [ObservableTypes.Organization], states: [EntityState.Enabled] } }); if (microsoft.observables.results.length > 0) { const msftId = microsoft.observables.results[0].observable.id; console.log(`Found organization: ${microsoft.observables.results[0].observable.name}\n`); // Find content mentioning Microsoft const msftContent = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Organization, observable: { id: msftId } }] }); // Extract products const msftProducts = new Map(); msftContent.contents.results.forEach(content => { content.observations ?.filter(obs => obs.type === ObservableTypes.Product) .forEach(obs => { msftProducts.set( obs.observable.name, (msftProducts.get(obs.observable.name) || 0) + 1 ); }); }); console.log(`Products by Microsoft: ${msftProducts.size}`); Array.from(msftProducts.entries()) .sort((a, b) => b[1] - a[1]) .slice(0, 5) .forEach(([product, count]) => { console.log(` - ${product} (mentioned ${count} times)`); }); } console.log('\n---\n'); // Example 4: Multi-hop relationship (Person ā†’ Organization ā†’ Event) console.log('Example 4: Multi-hop Relationship (Person ā†’ Org ā†’ Event)\n'); // Find a person const person = await graphlit.queryObservables({ search: "Kirk Marple", filter: { types: [ObservableTypes.Person], states: [EntityState.Enabled] } }); if (person.observables.results.length > 0) { const personId = person.observables.results[0].observable.id; console.log(`Person: ${person.observables.results[0].observable.name}\n`); // Step 1: Find organizations this person is associated with const personContent = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Person, observable: { id: personId } }] }); const relatedOrgs = new Set(); personContent.contents.results.forEach(content => { content.observations ?.filter(obs => obs.type === ObservableTypes.Organization) .forEach(obs => { relatedOrgs.add(obs.observable.id); }); }); console.log(`Associated organizations: ${relatedOrgs.size}`); // Step 2: For each organization, find events const allEvents = new Set(); for (const orgId of Array.from(relatedOrgs).slice(0, 3)) { // Limit for demo const orgEvents = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Organization, observable: { id: orgId } }] }); orgEvents.contents.results.forEach(content => { content.observations ?.filter(obs => obs.type === ObservableTypes.Event) .forEach(obs => { allEvents.add(obs.observable.name); }); }); } console.log(`Events related to person's organizations: ${allEvents.size}`); Array.from(allEvents).slice(0, 5).forEach(event => { console.log(` - ${event}`); }); } console.log('\nāœ“ Relationship queries complete!'); ``` *** ## Step-by-Step Explanation ### Step 1: Understanding Relationship Types **Direct Relationships** (inferred from co-occurrence): * **Person ā†’ Organization**: People work at organizations * **Event ā†’ Place**: Events happen at locations * **Product ā†’ Organization**: Companies make products * **Person ā†’ Event**: People attend/organize events * **Software ā†’ Organization**: Companies develop software **Implicit Relationships** (from content context): * Entities mentioned together in same document * Entities on same page (PDF documents) * Entities in same conversation thread * Entities in same time window (audio/video) ### Step 2: Query Pattern 1 - Find Related Entities **Pattern**: Entity A ā†’ Entity B ```typescript // 1. Find entity A const entityA = await graphlit.queryObservables({ search: "Entity A Name", filter: { types: [ObservableTypeA] } }); const entityAId = entityA.observables.results[0].observable.id; // 2. Find content mentioning entity A const content = await graphlit.queryContents({ observations: [{ type: ObservableTypeA, observable: { id: entityAId } }] }); // 3. Extract entity B from that content const relatedEntitiesB = new Set(); content.contents.results.forEach(item => { item.observations ?.filter(obs => obs.type === ObservableTypeB) .forEach(obs => { relatedEntitiesB.add(obs.observable); }); }); ``` ### Step 3: Query Pattern 2 - Co-Occurrence Filtering **Pattern**: Find content with BOTH entities ```typescript // Find content mentioning both Person X and Organization Y const cooccurrence = await graphlit.queryContents({ observations: [ { type: ObservableTypes.Person, observable: { id: personId } }, { type: ObservableTypes.Organization, observable: { id: orgId } } ] }); // This returns only content where BOTH entities appear console.log(`Co-occurrence count: ${cooccurrence.contents.results.length}`); ``` ### Step 4: Query Pattern 3 - Multi-Hop Relationships **Pattern**: Entity A ā†’ Entity B ā†’ Entity C ```typescript // Example: Person ā†’ Organization ā†’ Event // "Find events at companies where Person X works" // Step 1: Person ā†’ Organizations const personContent = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Person, observable: { id: personId } }] }); const orgs = new Set(); personContent.contents.results.forEach(content => { content.observations ?.filter(obs => obs.type === ObservableTypes.Organization) .forEach(obs => orgs.add(obs.observable.id)); }); // Step 2: Organizations ā†’ Events const allEvents = new Set(); for (const orgId of orgs) { const orgContent = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Organization, observable: { id: orgId } }] }); orgContent.contents.results.forEach(content => { content.observations ?.filter(obs => obs.type === ObservableTypes.Event) .forEach(obs => allEvents.add(obs.observable.name)); }); } ``` ### Step 5: Relationship Strength (Frequency) **Pattern**: Count co-occurrences to measure relationship strength ```typescript interface Relationship { entityA: string; entityB: string; strength: number; // Number of co-occurrences } function calculateRelationshipStrength( entityAId: string, entityBId: string ): number { const cooccurrence = await graphlit.queryContents({ observations: [ { type: ObservableTypes.Person, observable: { id: entityAId } }, { type: ObservableTypes.Organization, observable: { id: entityBId } } ] }); return cooccurrence.contents.results.length; } // Find strongest Person-Organization relationships const relationships: Relationship[] = []; for (const person of people) { for (const org of organizations) { const strength = await calculateRelationshipStrength(person.id, org.id); if (strength > 0) { relationships.push({ entityA: person.name, entityB: org.name, strength }); } } } relationships.sort((a, b) => b.strength - a.strength); ``` *** ## Configuration Options ### Filtering by Content Type ```typescript // Only find relationships in emails const emailRelationships = await graphlit.queryContents({ types: [ContentTypes.Email], observations: [{ type: ObservableTypes.Person, observable: { id: personId } }] }); // Only in Slack messages const slackRelationships = await graphlit.queryContents({ types: [ContentTypes.Message], observations: [{ type: ObservableTypes.Person, observable: { id: personId } }] }); ``` ### Time-Based Relationship Queries ```typescript // Find relationships in last 30 days const recentRelationships = await graphlit.queryContents({ creationDateRange: { from: new Date(Date.now() - 30 * 24 * 60 * 60 * 1000).toISOString() }, observations: [{ type: ObservableTypes.Person, observable: { id: personId } }] }); ``` ### Confidence-Based Filtering ```typescript // Only high-confidence relationships const content = await graphlit.queryContents({ observations: [{ type: ObservableTypes.Person, observable: { id: personId } }] }); // Filter by confidence client-side const highConfidence = content.contents.results.filter(item => item.observations?.some(obs => obs.observable.id === orgId && obs.occurrences?.some(occ => occ.confidence >= 0.8) ) ); ``` *** ## Variations ### Variation 1: Build Network Graph Create network visualization data: ```typescript interface NetworkNode { id: string; name: string; type: string; } interface NetworkEdge { source: string; target: string; weight: number; } async function buildNetworkGraph( startEntityId: string, startEntityType: ObservableTypes, depth: number = 2 ): Promise<{ nodes: NetworkNode[]; edges: NetworkEdge[] }> { const nodes = new Map(); const edges: NetworkEdge[] = []; const visited = new Set(); async function traverse(entityId: string, entityType: ObservableTypes, currentDepth: number) { if (currentDepth > depth || visited.has(entityId)) return; visited.add(entityId); // Get content mentioning this entity const content = await graphlit.queryContents({ observations: [{ type: entityType, observable: { id: entityId } }] }); // Extract all other entities from that content content.contents.results.forEach(item => { item.observations?.forEach(obs => { // Add node if (!nodes.has(obs.observable.id)) { nodes.set(obs.observable.id, { id: obs.observable.id, name: obs.observable.name, type: obs.type }); } // Add edge if (obs.observable.id !== entityId) { edges.push({ source: entityId, target: obs.observable.id, weight: 1 }); } // Recursively traverse if (currentDepth < depth) { traverse(obs.observable.id, obs.type, currentDepth + 1); } }); }); } await traverse(startEntityId, startEntityType, 0); return { nodes: Array.from(nodes.values()), edges }; } // Build 2-hop network from person const network = await buildNetworkGraph(personId, ObservableTypes.Person, 2); console.log(`Network: ${network.nodes.length} nodes, ${network.edges.length} edges`); // Export for visualization (D3.js, Cytoscape, etc.) fs.writeFileSync('network.json', JSON.stringify(network, null, 2)); ``` ### Variation 2: Relationship Timeline Track when relationships formed: ```typescript interface RelationshipTimeline { entityA: string; entityB: string; firstMention: Date; lastMention: Date; mentions: Array<{ date: Date; contentId: string }>; } async function buildRelationshipTimeline( entityAId: string, entityBId: string ): Promise { const content = await graphlit.queryContents({ observations: [ { observable: { id: entityAId } }, { observable: { id: entityBId } } ] orderBy: { creationDate: 'ASCENDING' } }); const mentions = content.contents.results.map(item => ({ date: new Date(item.creationDate), contentId: item.id })); return { entityA: entityAId, entityB: entityBId, firstMention: mentions[0]?.date, lastMention: mentions[mentions.length - 1]?.date, mentions }; } const timeline = await buildRelationshipTimeline(personId, orgId); console.log(`First mentioned together: ${timeline.firstMention.toLocaleDateString()}`); console.log(`Last mentioned together: ${timeline.lastMention.toLocaleDateString()}`); console.log(`Total mentions: ${timeline.mentions.length}`); ``` ### Variation 3: Entity Influence Score Rank entities by relationship count: ```typescript interface InfluenceScore { entityId: string; entityName: string; relationshipCount: number; uniqueConnections: number; } async function calculateInfluence( entityType: ObservableTypes ): Promise { // Get all entities of type const entities = await graphlit.queryObservables({ filter: { types: [entityType] } }); const scores: InfluenceScore[] = []; for (const entity of entities.observables.results) { // Find all content mentioning this entity const content = await graphlit.queryContents({ observations: [{ type: entityType, observable: { id: entity.observable.id } }] }); // Count unique connected entities const connections = new Set(); content.contents.results.forEach(item => { item.observations?.forEach(obs => { if (obs.observable.id !== entity.observable.id) { connections.add(obs.observable.id); } }); }); scores.push({ entityId: entity.observable.id, entityName: entity.observable.name, relationshipCount: content.contents.results.length, uniqueConnections: connections.size }); } return scores.sort((a, b) => b.uniqueConnections - a.uniqueConnections); } // Find most connected people const topPeople = await calculateInfluence(ObservableTypes.Person); console.log('Most connected people:'); topPeople.slice(0, 10).forEach((score, i) => { console.log(`${i + 1}. ${score.entityName}: ${score.uniqueConnections} connections`); }); ``` ### Variation 4: Relationship Path Finding Find shortest path between two entities: ```typescript async function findPath( startId: string, endId: string, maxDepth: number = 5 ): Promise { const queue: Array<{ id: string; path: string[] }> = [ { id: startId, path: [startId] } ]; const visited = new Set(); while (queue.length > 0) { const { id, path } = queue.shift()!; if (id === endId) { return path; // Found! } if (path.length > maxDepth || visited.has(id)) { continue; } visited.add(id); // Find connected entities const content = await graphlit.queryContents({ observations: [{ observable: { id } }] }); const connected = new Set(); content.contents.results.forEach(item => { item.observations?.forEach(obs => { if (obs.observable.id !== id && !visited.has(obs.observable.id)) { connected.add(obs.observable.id); } }); }); // Add to queue for (const connectedId of connected) { queue.push({ id: connectedId, path: [...path, connectedId] }); } } return null; // No path found } // Find path between two people const path = await findPath(personAId, personBId, 5); if (path) { console.log(`Path found (${path.length - 1} hops):`); console.log(path.join(' ā†’ ')); } else { console.log('No path found'); } ``` ### Variation 5: Relationship Export for Analysis Export relationship data for external analysis: ```typescript interface RelationshipExport { nodes: Array<{ id: string; name: string; type: string; properties: any }>; edges: Array<{ source: string; target: string; type: string; weight: number }>; } async function exportRelationships(): Promise { const nodes: RelationshipExport['nodes'] = []; const edges: RelationshipExport['edges'] = []; // Get all observables const allObservables = await graphlit.queryObservables({}); // Add nodes allObservables.observables.results.forEach(obs => { nodes.push({ id: obs.observable.id, name: obs.observable.name, type: obs.type, properties: obs.observable.properties || {} }); }); // Find edges (co-occurrences) const processed = new Set(); for (const obsA of allObservables.observables.results) { const content = await graphlit.queryContents({ observations: [{ observable: { id: obsA.observable.id } }] }); content.contents.results.forEach(item => { item.observations?.forEach(obsB => { const key = [obsA.observable.id, obsB.observable.id].sort().join('-'); if (obsA.observable.id !== obsB.observable.id && !processed.has(key)) { processed.add(key); edges.push({ source: obsA.observable.id, target: obsB.observable.id, type: 'CO_OCCURS', weight: 1 }); } }); }); } return { nodes, edges }; } // Export to file const export_data = await exportRelationships(); fs.writeFileSync('relationships.json', JSON.stringify(export_data, null, 2)); console.log(`Exported ${export_data.nodes.length} nodes and ${export_data.edges.length} edges`); ``` *** ## Common Issues & Solutions ### Issue: Too Many API Calls for Large Graphs **Problem**: Multi-hop queries make hundreds of API calls. **Solution**: Batch queries and cache results: ```typescript // Cache entity content const contentCache = new Map(); async function getCachedContent(entityId: string) { if (!contentCache.has(entityId)) { const content = await graphlit.queryContents({ observations: [{ observable: { id: entityId } }] }); contentCache.set(entityId, content); } return contentCache.get(entityId)!; } ``` ### Issue: No Relationships Found **Problem**: Query returns no related entities. **Causes**: 1. Entities from different content sources 2. Entities never co-occur 3. Filters too restrictive **Solution**: Broaden query: ```typescript // Remove date/type filters const content = await graphlit.queryContents({ observations: [{ observable: { id: entityId } }] // No other filters }); ``` ### Issue: Duplicate Relationships **Problem**: Same relationship counted multiple times. **Solution**: Deduplicate by entity ID: ```typescript const unique = new Map(); entities.forEach(entity => { unique.set(entity.id, entity); }); const deduplicated = Array.from(unique.values()); ``` ### Issue: Slow Multi-Hop Queries **Problem**: Deep relationship traversal very slow. **Solution**: Limit depth and parallelize: ```typescript // Limit to 2-3 hops max const maxDepth = 2; // Parallelize queries const results = await Promise.all( entityIds.map(id => queryRelationships(id)) ); ``` *** ## Developer Hints ### Relationship Query Performance * Direct relationships (1-hop): Fast (<1s) * 2-hop relationships: Moderate (1-5s) * 3+ hop relationships: Slow (5-30s) * Cache aggressively for large graphs ### Relationship Types by Content * **Emails**: Strong Person-Person, Person-Organization * **Slack**: Person-Person, Organization-Product * **Documents**: All types, especially Product-Organization * **GitHub**: Person-Repo, Software-Organization ### Co-Occurrence Reliability * Same page (PDF): Very strong relationship indicator * Same document: Strong relationship * Same thread (Slack/email): Strong relationship * Different documents: Weaker relationship ### Graph Traversal Strategies * **Breadth-first**: Find shortest paths * **Depth-first**: Explore deep relationships * **Limited depth**: Prevent explosion (max 3 hops) * **Type-specific**: Only traverse certain entity types *** # Configure Workflow for Entity Extraction ## Use Case: Configure Workflow for Entity Extraction ### User Intent "How do I configure a workflow to extract entities from my content? What entity types should I choose and which models work best?" ### Operation **SDK Method**: `createWorkflow()` with `extraction` stage\ **GraphQL Mutation**: `createWorkflow`\ **Entity**: Workflow with extraction configuration ### Prerequisites * Graphlit project with API credentials * Understanding of entity types (Person, Organization, etc.) * Content to process (documents, emails, messages, etc.) *** ### Complete Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { FilePreparationServiceTypes, EntityExtractionServiceTypes, ObservableTypes, } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Create workflow with entity extraction const workflow = await graphlit.createWorkflow({ name: "Entity Extraction Workflow", preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.Document } }] }, extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Place, ObservableTypes.Event ] } }] } }); console.log(`Created workflow: ${workflow.createWorkflow.id}`); console.log(`Extracting: Person, Organization, Place, Event`); // Use workflow with content ingestion const content = await graphlit.ingestUri( 'https://example.com/document.pdf', 'Entity Extraction Doc', undefined, undefined, true, { id: workflow.createWorkflow.id } ); console.log(`Ingesting content with entity extraction...`); ``` *** ## Key differences: snake\_case methods, enum values workflow\_response = await graphlit.createWorkflow( name="Entity Extraction Workflow", preparation={ "jobs": \[{ "connector": { "type": FilePreparationServiceTEXT } }] }, extraction={ "jobs": \[{ "connector": { "type": ExtractionServiceMODEL\_TEXT, "extractedTypes": \[ ObservablePERSON, ObservableORGANIZATION, ObservablePLACE, ObservableEVENT ] } }] } ) print(f"Created workflow: {workflow\_response.create\_workflow\.id}") ```` ### C# ```csharp using Graphlit; using Graphlit.Api.Input; var graphlit = new Graphlit(); // Key differences: PascalCase methods var workflow = await graphlit.CreateWorkflow( name: "Entity Extraction Workflow", preparation: new WorkflowPreparationInput { Jobs = new[] { new WorkflowPreparationJobInput { Connector = new FilePreparationConnectorInput { Type = FilePreparationServiceText } } } }, extraction: new WorkflowExtractionInput { Jobs = new[] { new WorkflowExtractionJobInput { Connector = new ExtractionConnectorInput { Type = ExtractionServiceModelText, ExtractedTypes = new[] { ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Place, ObservableTypes.Event } } } } } ); Console.WriteLine($"Created workflow: {workflow.CreateWorkflow.Id}"); ```` *** ### Step-by-Step Explanation #### Step 1: Choose Extraction Type Graphlit supports two extraction connector types: **`ExtractionServiceModelText`**: * For text-based content (documents, emails, messages) * Uses LLM to analyze prepared text * Fast and cost-effective * Best for: PDFs with text, emails, Slack messages, web pages **`ExtractionServiceModelDocument`**: * For visual document analysis * Uses vision models (GPT-4o Vision, Claude 3.5 Sonnet) * Analyzes images, charts, diagrams, scanned documents * Best for: PDFs with images, scanned documents, presentation slides #### Step 2: Select Entity Types Choose entity types based on your domain: **Business Documents**: ```typescript extractedTypes: [ ObservableTypes.Person, // People, contacts ObservableTypes.Organization, // Companies, departments ObservableTypes.Place, // Locations, offices ObservableTypes.Event, // Meetings, deadlines ObservableTypes.Product // Products, services ] ``` **Medical/Clinical Content**: ```typescript extractedTypes: [ ObservableTypes.MedicalCondition, ObservableTypes.MedicalDrug, ObservableTypes.MedicalProcedure, ObservableTypes.MedicalTest, ObservableTypes.MedicalStudy ] ``` **Technical Documentation**: ```typescript extractedTypes: [ ObservableTypes.Software, // Software products ObservableTypes.Repo, // Code repositories ObservableTypes.Organization, // Tech companies ObservableTypes.Person, // Developers, authors ObservableTypes.Category // Topics, tags ] ``` #### Step 3: Add Preparation Stage Preparation extracts text before entity extraction: ```typescript preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.Document, // PDFs, Word, etc. // FilePreparationServiceTypes.Text for plain text // FilePreparationServiceTypes.Audio for transcription } }] } ``` #### Step 4: Configure Model (Optional) Specify which LLM model to use via specification: ```typescript const spec = await graphlit.createSpecification({ name: "GPT-4 Extraction", type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt4O_128K, temperature: 0.1 // Low temperature for consistent extraction } }); const workflow = await graphlit.createWorkflow({ name: "High-Quality Extraction", extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [/* ... */] } }] }, specification: { id: spec.createSpecification.id } }); ``` *** ### Configuration Options #### Changing Extraction Models **GPT-4o (Default - Recommended)**: * Fast and accurate * Good balance of quality and cost * Handles 20+ entity types * Best for production **GPT-4**: * Highest quality * More expensive * Slower processing * Best for critical accuracy **Claude 3.5 Sonnet**: * Very good quality * Fast processing * Good for long documents * Alternative to GPT-4o **Gemini 1.5 Pro**: * Cost-effective * Good quality * Fast processing * Budget-friendly option #### Vision Model Extraction (for PDFs with Images) ```typescript const workflow = await graphlit.createWorkflow({ name: "Vision-Based Extraction", preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.Document } }] }, extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelDocument, // Vision model extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Product ] } }] } }); ``` #### Multiple Extraction Jobs Extract different types with different models: ```typescript extraction: { jobs: [ { // Fast extraction for basic types connector: { type: EntityEntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization ] } }, { // Vision extraction for complex types connector: { type: EntityEntityExtractionServiceTypes.ModelDocument, extractedTypes: [ ObservableTypes.Product, ObservableTypes.Software ] } } ] } ``` *** ### Variations #### Variation 1: Minimal Extraction (Fast) Extract only core entity types for speed: ```typescript const workflow = await graphlit.createWorkflow({ name: "Fast Extraction", extraction: { jobs: [{ connector: { type: EntityEntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization ] } }] } }); ``` #### Variation 2: Comprehensive Extraction Extract all relevant entity types: ```typescript const workflow = await graphlit.createWorkflow({ name: "Comprehensive Extraction", extraction: { jobs: [{ connector: { type: EntityEntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Place, ObservableTypes.Event, ObservableTypes.Product, ObservableTypes.Software, ObservableTypes.Category, ObservableLabel ] } }] } }); ``` #### Variation 3: Medical Content Extraction Extract medical entities: ```typescript const workflow = await graphlit.createWorkflow({ name: "Medical Extraction", extraction: { jobs: [{ connector: { type: EntityEntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.MedicalCondition, ObservableTypes.MedicalDrug, ObservableMedicalDrugClass, ObservableTypes.MedicalProcedure, ObservableTypes.MedicalTest, ObservableTypes.MedicalStudy, ObservableMedicalDevice, ObservableMedicalTherapy ] } }] } }); ``` #### Variation 4: Audio/Video Transcription + Extraction Extract entities from meeting recordings: ```typescript const workflow = await graphlit.createWorkflow({ name: "Meeting Entity Extraction", preparation: { jobs: [{ connector: { type: FilePreparationServiceAudio, audioTranscription: { model: AudioTranscriptionServiceDeepgram } } }] }, extraction: { jobs: [{ connector: { type: EntityEntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Event, ObservableTypes.Product ] } }] } }); ``` #### Variation 5: GitHub Repository Analysis Extract technical entities from code repositories: ```typescript const workflow = await graphlit.createWorkflow({ name: "GitHub Extraction", extraction: { jobs: [{ connector: { type: EntityEntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.Repo, ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Software, ObservableTypes.Category ] } }] } }); ``` *** ### Common Issues & Solutions #### Issue: No Entities Extracted **Problem**: Workflow completes but no observations found. **Solutions**: 1. Check content has text (not just images without OCR) 2. Verify extraction stage is configured 3. Ensure entity types are appropriate for content 4. Check confidence threshold isn't too high 5. Try vision model for scanned documents ```typescript // Debug: Check if extraction stage exists const workflowDetails = await graphlit.getWorkflow(workflow.id); console.log('Extraction jobs:', workflowDetails.workflow.extraction?.jobs); // Try vision model if text extraction fails extraction: { jobs: [{ connector: { type: EntityEntityExtractionServiceTypes.ModelDocument // Use vision } }] } ``` #### Issue: Too Many Low-Quality Entities **Problem**: Many entities with low confidence scores. **Solutions**: 1. Use better model (GPT-4 instead of Gemini) 2. Filter by confidence threshold (>0.7) 3. Reduce number of entity types 4. Improve content quality (OCR accuracy) ```typescript // Filter low-confidence entities when querying content.observations .filter(obs => obs.occurrences.some(occ => occ.confidence >= 0.7)) .forEach(obs => console.log(obs.observable.name)); ``` #### Issue: Extraction Too Slow **Problem**: Processing takes too long. **Solutions**: 1. Reduce number of entity types 2. Use faster model (GPT-4o instead of GPT-4) 3. Split large documents 4. Process in batches ```typescript // Optimize: Extract only critical types extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization // Remove less critical types for speed ] ``` #### Issue: Wrong Entity Types Extracted **Problem**: Entities classified incorrectly. **Solutions**: 1. Use more specific entity types 2. Provide better preparation (clean text) 3. Use vision models for visual documents 4. Combine multiple extraction jobs ```typescript // Example: Separate medical and general extraction extraction: { jobs: [ { connector: { type: EntityEntityExtractionServiceTypes.ModelText, extractedTypes: [ObservableTypes.Person, ObservableTypes.Organization] } }, { connector: { type: EntityEntityExtractionServiceTypes.ModelText, extractedTypes: [ObservableTypes.MedicalCondition, ObservableTypes.MedicalDrug] } } ] } ``` *** ### Developer Hints #### Model Selection Guidelines * **Production default**: GPT-4o (fast, accurate, cost-effective) * **Highest quality**: GPT-4 (use for critical applications) * **Long documents**: Claude 3.5 Sonnet (128K context) * **Budget-friendly**: Gemini 1.5 Pro * **Visual content**: GPT-4o Vision or Claude 3.5 Sonnet #### Entity Type Selection Strategy 1. Start with core types (Person, Organization) 2. Add domain-specific types (Medical\*, Software, Repo) 3. Test extraction quality 4. Add more types incrementally 5. Monitor processing time and cost #### Performance Considerations * More entity types = longer processing time * Vision models slower than text models * GPT-4 slower but more accurate than GPT-4o * Batch processing for large volumes * Consider cost per extraction job #### Confidence Threshold Recommendations * **High precision needed**: confidence >= 0.8 * **Balanced**: confidence >= 0.7 (recommended) * **High recall needed**: confidence >= 0.5 * **Research/exploration**: confidence >= 0.3 #### Cost Optimization 1. Use text extraction when possible (cheaper than vision) 2. Choose appropriate model (GPT-4o vs GPT-4) 3. Extract only needed entity types 4. Batch process for volume discounts 5. Cache extracted entities *** # How Entity Extraction Works ## Workflow: How Entity Extraction Works ### User Intent "How does entity extraction actually work? What happens during the extraction stage?" ### Operation * **SDK Method**: `createWorkflow()` with `extraction` stage * **GraphQL**: Workflow with extraction configuration * **Entity Type**: Workflow * **Common Use Cases**: Understanding extraction pipeline, configuring extraction, choosing models ### Extraction Pipeline Overview Entity extraction is an LLM-based process that analyzes text and identifies structured entities (people, organizations, places, etc.). ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ModelServiceTypes, ObservableTypes, SpecificationTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Create workflow with extraction const workflow = await graphlit.createWorkflow({ name: "Document Entity Extraction", preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.Document } }] }, extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Place, ObservableTypes.Event ] } }] } }); // Ingest with extraction workflow const content = await graphlit.ingestUri( 'https://example.com/document.pdf', undefined, undefined, undefined, true, { id: workflow.createWorkflow.id } ); // Check extracted entities const result = await graphlit.getContent(content.ingestUri.id); console.log(`Extracted ${result.content.observations?.length || 0} entity observations`); result.content.observations?.forEach(obs => { console.log(`${obs.type}: ${obs.observable.name}`); console.log(` Confidence: ${obs.occurrences?.[0]?.confidence}`); }); ``` ### The Extraction Pipeline #### Step-by-Step Process ``` 1. Content Ingestion ā†“ 2. Preparation Stage - Text extraction (PDF, Word, etc.) - OCR (scanned documents) - Audio transcription - Text chunking ā†“ 3. Extraction Stage (THIS IS WHERE IT HAPPENS) - Send text to LLM (GPT-4, Claude, etc.) - LLM analyzes text for entities - LLM returns structured JSON with entities - Each entity has: type, name, properties, confidence ā†“ 4. Observation Creation - Create Observation records - Link to content - Store occurrence details (page, location, confidence) ā†“ 5. Entity Resolution - Check if entity already exists (by name, email, url, etc.) - Create new Observable OR link to existing - Deduplicate entities ā†“ 6. Graph Storage - Store in graph database - Create entity nodes - Create observation edges - Link to content ā†“ 7. Content State ā†’ ENABLED ``` ### LLM-Based Extraction #### What the LLM Does ```typescript // Behind the scenes, LLM receives prompt like: const prompt = ` Extract entities from the following text. Return a JSON array of entities with: - type (PERSON, ORGANIZATION, PLACE, EVENT, etc.) - name - properties (email, jobTitle, url, etc.) - confidence (0.0 to 1.0) Text: """ Kirk Marple is the CEO of Graphlit, a context layer for AI agents based in Seattle. The company was founded in 2023 and raised $2.5M in funding. """ Expected output: [ { "type": "PERSON", "name": "Kirk Marple", "properties": { "jobTitle": "CEO", "affiliation": "Graphlit" }, "confidence": 0.95 }, { "type": "ORGANIZATION", "name": "Graphlit", "properties": { "description": "context layer for AI agents", "foundingDate": "2023" }, "confidence": 0.98 }, { "type": "PLACE", "name": "Seattle", "confidence": 0.92 } ] `; // LLM processes and returns structured JSON // Graphlit parses and creates Observations ``` #### Model Selection ```typescript // Specify model via specification const gpt4Spec = await graphlit.createSpecification({ name: "GPT-4 Extraction", type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt4Turbo_128K, // High quality temperature: 0.0 // Deterministic } }); const workflow = await graphlit.createWorkflow({ name: "High Quality Extraction", extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ObservableTypes.Person, ObservableTypes.Organization] } }] }, specification: { id: gpt4Spec.createSpecification.id } }); ``` ### Vision-Based Extraction For PDFs with images, charts, diagrams: ```typescript const visionWorkflow = await graphlit.createWorkflow({ name: "PDF Vision Extraction", preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.Document, extractImages: true, // Extract images from PDF ocrImages: true // OCR on images } }] }, extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelImage, extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization ] } }] } }); // Vision models can extract from: // - Charts and diagrams // - Organizational charts // - Scanned documents // - Images with text // - Infographics ``` ### Extraction Models Comparison #### GPT-4 (OpenAI) * **Quality**: Highest * **Speed**: Moderate * **Cost**: High * **Use**: Production, high-value content #### GPT-4o (OpenAI) * **Quality**: High * **Speed**: Fast * **Cost**: Moderate * **Use**: Balanced production workloads #### Claude 3.5 Sonnet (Anthropic) * **Quality**: High * **Speed**: Fast * **Cost**: Moderate * **Use**: Alternative to GPT-4o, good quality #### Gemini Pro (Google) * **Quality**: Good * **Speed**: Fast * **Cost**: Lower * **Use**: Cost-sensitive applications ```typescript // Configure model const spec = await graphlit.createSpecification({ name: "Model Spec", type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, // or Anthropic, Google openAI: { model: OpenAiModels.Gpt4Turbo_128K } }); ``` ### Prompt Engineering for Extraction #### Default Prompts Graphlit uses optimized prompts for each entity type: ```typescript // Person extraction prompt (conceptual): // "Extract people mentioned in the text. Include: // - Full name // - Email address (if mentioned) // - Job title (if mentioned) // - Affiliation/company (if mentioned) // - Provide confidence score" // Organization extraction prompt: // "Extract organizations mentioned. Include: // - Full organization name // - URL (if mentioned) // - Description (if available) // - Provide confidence score" ``` #### Custom Prompts (Advanced) Future feature: Custom extraction prompts for domain-specific needs ### Confidence Scoring #### How Confidence is Calculated LLM provides confidence based on: * Context clarity * Explicit mentions vs inferences * Ambiguity in text * Supporting evidence ```typescript // High confidence (0.9-1.0): // "Kirk Marple is the CEO..." // Clear, explicit, unambiguous // Medium confidence (0.7-0.9): // "Kirk mentioned that..." // Implicit context, less clear // Low confidence (0.5-0.7): // "The CEO said..." // Pronoun reference, ambiguous // Very low confidence (<0.5): // "He suggested..." // Multiple possible referents ``` #### Using Confidence Thresholds ```typescript // Filter by confidence const content = await graphlit.getContent('content-id'); const highConfidence = content.content.observations?.filter(obs => obs.occurrences?.some(occ => (occ.confidence || 0) >= 0.8) ); console.log(`High confidence entities: ${highConfidence?.length}`); ``` ### When Extraction Runs #### During Workflow Processing ```typescript // Extraction runs AFTER preparation const workflow = await graphlit.createWorkflow({ preparation: { /* Extract text first */ }, extraction: { /* Then extract entities */ } }); // Timeline: // 1. Ingest content ā†’ State: CREATED // 2. Preparation runs ā†’ Extract text/OCR/transcribe // 3. Extraction runs ā†’ LLM analyzes text // 4. Observations created // 5. State: ENABLED ``` #### Multiple Extraction Jobs ```typescript // Run multiple extraction jobs in parallel const workflow = await graphlit.createWorkflow({ name: "Multi-Model Extraction", extraction: { jobs: [ { // Text-based extraction connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ObservableTypes.Person, ObservableTypes.Organization] } }, { // Vision-based extraction (runs in parallel) connector: { type: EntityExtractionServiceTypes.ModelImage, extractedTypes: [ObservableTypes.Person, ObservableTypes.Organization] } } ] } }); ``` ## Create extraction workflow workflow = await graphlit.createWorkflow( name="Entity Extraction", preparation=input\_types.PreparationWorkflowStageInput( jobs=\[ input\_types.PreparationWorkflowJobInput( connector=input\_types.FilePreparationConnectorInput( type=enums.FilePreparationServiceDOCUMENT ) ) ] ), extraction=input\_types.ExtractionWorkflowStageInput( jobs=\[ input\_types.ExtractionWorkflowJobInput( connector=input\_types.EntityExtractionConnectorInput( type=enums.ExtractionServiceMODEL\_TEXT, extracted\_types=\[ enums.ObservablePERSON, enums.ObservableORGANIZATION ] ) ) ] ) ) ## Ingest with extraction content = await graphlit.ingestUri( uri='', workflow=input\_types.EntityReferenceInput(id=workflow\.create\_workflow\.id), is\_synchronous=True ) ## Check entities result = await graphlit.getContent(content.ingest\_uri.id) print(f"Extracted {len(result.content.observations or \[])} entities") ```` **C#**: ```csharp using Graphlit; var client = new Graphlit(); // Create extraction workflow var workflow = await graphlit.CreateWorkflow(new WorkflowInput { Name = "Entity Extraction", Preparation = new PreparationWorkflowStage { Jobs = new[] { new PreparationWorkflowJob { Connector = new FilePreparationConnector { Type = FilePreparationServiceDocument } } } }, Extraction = new ExtractionWorkflowStage { Jobs = new[] { new ExtractionWorkflowJob { Connector = new EntityExtractionConnector { Type = ExtractionServiceModelText, ExtractedTypes = new[] { ObservableTypes.Person, ObservableTypes.Organization } } } } } }); // Ingest with extraction var content = await graphlit.IngestUri(new IngestUriInput { Uri = "https://example.com/doc.pdf", Workflow = new EntityReference { Id = workflow.CreateWorkflow.Id }, IsSynchronous = true }); // Check entities var result = await graphlit.GetContent(content.IngestUri.Id); Console.WriteLine($"Extracted {result.Content.Observations?.Length ?? 0} entities"); ```` ### Developer Hints #### Extraction Requires Preparation ```typescript // Won't work - extraction needs text const workflow = await graphlit.createWorkflow({ extraction: { /* ... */ } // Missing preparation stage! }); // āœ“ Correct - prepare first const workflow = await graphlit.createWorkflow({ preparation: { /* Extract text */ }, extraction: { /* Then extract entities */ } }); ``` #### More Entity Types = Slower + More Expensive ```typescript // Fast + cheap (2 types) extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization ] // Slower + more expensive (10 types) extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Place, ObservableTypes.Event, ObservableTypes.Product, // ... more types ] // Choose types relevant to your domain ``` #### Vision Models for Complex PDFs ```typescript // Use ModelDocument (vision) for: // - Scanned documents // - PDFs with charts/diagrams // - Organizational charts // - Infographics // Use ModelText for: // - Plain text documents // - Word documents // - Clean PDFs // - Transcribed audio ``` ### Common Issues & Solutions **Issue**: No entities extracted **Solution**: Check if workflow has extraction stage and preparation completed ```typescript // Check content state const content = await graphlit.getContent('content-id'); console.log(`State: ${content.content.state}`); // Check if workflow has extraction const workflow = await graphlit.getWorkflow('workflow-id'); console.log(`Has extraction: ${!!workflow.workflow.extraction}`); ``` **Issue**: Low confidence scores **Solution**: Text may be ambiguous or context unclear ```typescript // Use higher quality model const betterSpec = await graphlit.createSpecification({ type: SpecificationTypes.Completion, openAI: { model: OpenAiModels.Gpt4Turbo_128K // Better than GPT-3.5 } }); // Apply threshold const highConfidence = observations.filter( obs => obs.occurrences?.[0]?.confidence >= 0.8 ); ``` **Issue**: Too many false positives **Solution**: Increase confidence threshold or narrow entity types ```typescript // Only extract specific types extractedTypes: [ ObservableTypes.Person // Just people, not everything ] // Filter by confidence const reliable = observations.filter( obs => obs.occurrences?.[0]?.confidence >= 0.85 ); ``` ### Production Example ```typescript async function createProductionExtractionWorkflow() { console.log('Creating production extraction workflow...\n'); // Create high-quality specification const spec = await graphlit.createSpecification({ name: "Production Extraction", type: SpecificationTypes.Completion, serviceType: ModelServiceTypes.OpenAi, openAI: { model: OpenAiModels.Gpt4Turbo_128K, temperature: 0.0 // Deterministic } }); console.log(`āœ“ Created specification: ${spec.createSpecification.id}`); // Create workflow with both text and vision extraction const workflow = await graphlit.createWorkflow({ name: "Production Entity Extraction", preparation: { jobs: [{ connector: { type: FilePreparationServiceTypes.Document, extractImages: true, ocrImages: true } }] }, extraction: { jobs: [ { // Text extraction connector: { type: EntityExtractionServiceTypes.ModelText, extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Place, ObservableTypes.Event, ObservableTypes.Product ] } }, { // Vision extraction (for images/charts) connector: { type: EntityExtractionServiceTypes.ModelImage, extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization ] } } ] }, specification: { id: spec.createSpecification.id } }); console.log(`āœ“ Created workflow: ${workflow.createWorkflow.id}\n`); // Test with sample document console.log('Testing extraction...'); const content = await graphlit.ingestUri( 'https://example.com/sample-document.pdf', undefined, undefined, undefined, true, { id: workflow.createWorkflow.id } ); // Get results const result = await graphlit.getContent(content.ingestUri.id); console.log(`\n=== EXTRACTION RESULTS ===`); console.log(`Document: ${result.content.name}`); console.log(`Total observations: ${result.content.observations?.length || 0}`); // Group by type const byType = new Map(); result.content.observations?.forEach(obs => { byType.set(obs.type, (byType.get(obs.type) || 0) + 1); }); console.log('\nEntities by type:'); byType.forEach((count, type) => { console.log(` ${type}: ${count}`); }); // Confidence analysis const confidences = result.content.observations ?.flatMap(obs => obs.occurrences || []) .map(occ => occ.confidence || 0) || []; if (confidences.length > 0) { const avg = confidences.reduce((a, b) => a + b, 0) / confidences.length; const high = confidences.filter(c => c >= 0.8).length; const med = confidences.filter(c => c >= 0.6 && c < 0.8).length; const low = confidences.filter(c => c < 0.6).length; console.log('\nConfidence distribution:'); console.log(` High (ā‰„80%): ${high}`); console.log(` Medium (60-80%): ${med}`); console.log(` Low (<60%): ${low}`); console.log(` Average: ${(avg * 100).toFixed(1)}%`); } return workflow.createWorkflow.id; } await createProductionExtractionWorkflow(); ``` # Data Source Feeds Connect 30+ data sources to automatically sync and process content. *** ## Feed Types ### [Messaging](/api-guides/use-cases/feeds/messaging) (7 feeds) Team collaboration, email, and support * [Slack](/api-guides/use-cases/feeds/messaging/feed-create-slack) * [Microsoft Teams](/api-guides/use-cases/feeds/messaging/feed-create-microsoft-teams) * [Discord](/api-guides/use-cases/feeds/messaging/feed-create-discord) * [Gmail](/api-guides/use-cases/feeds/messaging/feed-create-gmail) * [Outlook Email](/api-guides/use-cases/feeds/messaging/feed-create-outlook-email) * [Intercom Tickets](/api-guides/use-cases/feeds/messaging/feed-create-intercom) * [Intercom Articles](/api-guides/use-cases/feeds/messaging/feed-create-intercom-articles) ### [Cloud Storage](/api-guides/use-cases/feeds/cloud-storage) (9 feeds) Document repositories and file storage * [Google Drive](/api-guides/use-cases/feeds/cloud-storage/feed-create-google-drive) * [OneDrive](/api-guides/use-cases/feeds/cloud-storage/feed-create-onedrive) * [SharePoint](/api-guides/use-cases/feeds/cloud-storage/feed-create-sharepoint) * [Dropbox](/api-guides/use-cases/feeds/cloud-storage/feed-create-dropbox) * [Box](/api-guides/use-cases/feeds/cloud-storage/feed-create-box) * [AWS S3](/api-guides/use-cases/feeds/cloud-storage/feed-create-aws-s3) * [Azure Blob Storage](/api-guides/use-cases/feeds/cloud-storage/feed-create-azure-storage) * [Google Cloud Storage](/api-guides/use-cases/feeds/cloud-storage/feed-create-google-cloud-storage) * [GitHub Repository](/api-guides/use-cases/feeds/cloud-storage/feed-create-github) ### [Project Management](/api-guides/use-cases/feeds/project-management) (9 feeds) Issue tracking, support, and documentation * [Jira](/api-guides/use-cases/feeds/project-management/feed-create-jira) * [Linear](/api-guides/use-cases/feeds/project-management/feed-create-linear) * [Notion](/api-guides/use-cases/feeds/project-management/feed-create-notion) * [GitHub Issues](/api-guides/use-cases/feeds/project-management/feed-create-github-issues) * [GitHub Commits](/api-guides/use-cases/feeds/project-management/feed-create-github-commits) * [GitHub Pull Requests](/api-guides/use-cases/feeds/project-management/feed-create-github-pull-requests) * [Trello](/api-guides/use-cases/feeds/project-management/feed-create-trello) * [Zendesk Tickets](https://github.com/graphlit/graphlit-docs/blob/main/api-guides/use-cases/feeds/project-management/feed-create-zendesk.md) * [Zendesk Articles](/api-guides/use-cases/feeds/project-management/feed-create-zendesk-articles) ### [Social Media](/api-guides/use-cases/feeds/social-media) (3 feeds) Public social platforms * [Reddit](/api-guides/use-cases/feeds/social-media/feed-create-reddit) * [Twitter/X](/api-guides/use-cases/feeds/social-media/feed-create-twitter) * [YouTube](/api-guides/use-cases/feeds/social-media/feed-create-youtube) ### [Calendars](/api-guides/use-cases/feeds/calendars) (2 feeds) Event and meeting sync * [Google Calendar](/api-guides/use-cases/feeds/calendars/feed-create-google-calendar) * [Outlook Calendar](/api-guides/use-cases/feeds/calendars/feed-create-outlook-calendar) ### [Meetings](https://github.com/graphlit/graphlit-docs/blob/main/api-guides/use-cases/feeds/meetings/README.md) (3 feeds) Meeting transcripts * [Fireflies.ai](https://github.com/graphlit/graphlit-docs/blob/main/api-guides/use-cases/feeds/meetings/feed-create-fireflies.md) * [Fathom](https://github.com/graphlit/graphlit-docs/blob/main/api-guides/use-cases/feeds/meetings/feed-create-fathom.md) * [Attio Meetings](https://github.com/graphlit/graphlit-docs/blob/main/api-guides/use-cases/feeds/meetings/feed-create-attio-meetings.md) ### [CRM & Contacts](https://github.com/graphlit/graphlit-docs/blob/main/api-guides/use-cases/feeds/crm/README.md) (4 feeds) Contact and CRM sync * [Attio CRM](https://github.com/graphlit/graphlit-docs/blob/main/api-guides/use-cases/feeds/crm/feed-create-attio-crm.md) * [Google Contacts](https://github.com/graphlit/graphlit-docs/blob/main/api-guides/use-cases/feeds/crm/feed-create-google-contacts.md) * [Microsoft Contacts](https://github.com/graphlit/graphlit-docs/blob/main/api-guides/use-cases/feeds/crm/feed-create-microsoft-contacts.md) * [Salesforce CRM](https://github.com/graphlit/graphlit-docs/blob/main/api-guides/use-cases/feeds/crm/feed-create-salesforce-crm.md) ### [Web Sources](/api-guides/use-cases/feeds/web) (5 feeds) Web content, search, and podcasts * [RSS Feeds](/api-guides/use-cases/feeds/web/feed-create-rss) * [Web Crawling](/api-guides/use-cases/feeds/web/feed-create-web-crawl) * [Tavily Search](/api-guides/use-cases/feeds/web/feed-create-tavily-search) * [Exa Search](/api-guides/use-cases/feeds/web/feed-create-exa-search) * [Podscan Podcast Search](/api-guides/use-cases/feeds/web/feed-create-podscan-search) ### [Research](https://github.com/graphlit/graphlit-docs/blob/main/api-guides/use-cases/feeds/research/README.md) (1 feed) Recurring web research * [Parallel Research](https://github.com/graphlit/graphlit-docs/blob/main/api-guides/use-cases/feeds/research/feed-create-parallel-research.md) ### [Entity Discovery](https://github.com/graphlit/graphlit-docs/blob/main/api-guides/use-cases/feeds/entity-discovery/README.md) (1 feed) Entity discovery from a natural-language query * [Parallel Entity Discovery](https://github.com/graphlit/graphlit-docs/blob/main/api-guides/use-cases/feeds/entity-discovery/feed-create-parallel-entity-discovery.md) *** ## Feed Operations * [Query Feeds](/api-guides/use-cases/feeds/feed-query) - List and filter feeds * [Update Feed](/api-guides/use-cases/feeds/feed-update) - Modify configuration * [Delete Feed](/api-guides/use-cases/feeds/feed-delete) - Remove feed * [Feed Polling](/api-guides/use-cases/feeds/feed-is-done-polling) - Check sync status * [Get Feed Details](/api-guides/use-cases/feeds/feed-get-details) - Full feed info *** ## Common Patterns **OAuth Setup**: Most feeds require OAuth tokens from Developer Portal\ **Incremental Sync**: Feeds track what's synced, only pull new items\ **With Workflows**: Attach extraction workflows for entity extraction\ **Real-time**: Configure sync intervals (15min - 24hr) *** **44 feed guides** | [ā† Back to Use Cases](/api-guides/use-cases) # Delete Feed ## User Intent "How do I delete a feed? Show me feed cleanup." ## Operation **SDK Method**: `deleteFeed()`\ **Use Case**: Remove feed and stop syncing *** ## Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; const graphlit = new Graphlit(); // Delete feed await graphlit.deleteFeed('feed-id'); console.log('Feed deleted'); // Note: Content from feed is NOT automatically deleted // Delete content separately if needed await graphlit.deleteAllContents({ filter: { feeds: [{ id: 'feed-id' }], }, }); console.log('Feed content deleted'); ``` *** ## Important **Feed deletion stops syncing** but doesn't delete content\ **Delete content separately** if needed\ **Cannot undo** deletion *** # Get Feed Details ## User Intent "I want to retrieve full configuration details for a specific feed" ## Operation * **SDK Method**: `graphlit.getFeed()` * **GraphQL**: `getFeed` query * **Entity Type**: Feed * **Common Use Cases**: View feed configuration, check OAuth tokens, inspect sync settings ## TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { FeedServiceTypes, FeedTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); const feedId = 'feed-id-here'; // Get full feed details const feed = await graphlit.getFeed(feedId); console.log(`\nFeed: ${feed.feed.name}`); console.log(`Type: ${feed.feed.type}`); console.log(`State: ${feed.feed.state}`); console.log(`Created: ${feed.feed.creationDate}`); // Check type-specific configuration switch (feed.feed.type) { case FeedTypes.Slack: { if (!feed.feed.slack) break; console.log(`\nSlack Configuration:`); console.log(` Channel: ${feed.feed.slack.channel}`); console.log(` Listing: ${feed.feed.slack.type}`); console.log(` Read Limit: ${feed.feed.slack.readLimit}`); console.log(` Include Attachments: ${feed.feed.slack.includeAttachments}`); break; } case FeedTypes.Web: { if (!feed.feed.web) break; console.log(`\nWeb Configuration:`); console.log(` Start URI: ${feed.feed.web.uri}`); console.log(` Read Limit: ${feed.feed.web.readLimit}`); console.log(` Include Files: ${feed.feed.web.includeFiles}`); console.log(` Allowed Domains: ${feed.feed.web.allowedDomains?.join(', ')}`); break; } case FeedTypes.Rss: { if (!feed.feed.rss) break; console.log(`\nRSS Configuration:`); console.log(` URI: ${feed.feed.rss.uri}`); console.log(` Read Limit: ${feed.feed.rss.readLimit}`); break; } case FeedTypes.MicrosoftTeams: { if (!feed.feed.microsoftTeams) break; console.log(`\nMicrosoft Teams Configuration:`); console.log(` Team ID: ${feed.feed.microsoftTeams.teamId}`); console.log(` Channel ID: ${feed.feed.microsoftTeams.channelId}`); console.log(` Listing: ${feed.feed.microsoftTeams.type}`); console.log(` Read Limit: ${feed.feed.microsoftTeams.readLimit}`); console.log(` Include Attachments: ${feed.feed.microsoftTeams.includeAttachments}`); break; } case FeedTypes.Site: { if (!feed.feed.site) break; if (feed.feed.site.type === FeedServiceTypes.GoogleDrive) { console.log(`\nGoogle Drive Configuration:`); console.log(` Folder ID: ${feed.feed.site.googleDrive?.folderId ?? 'Drive root'}`); console.log(` Files: ${feed.feed.site.googleDrive?.files?.join(', ') ?? 'None'}`); console.log(` Read Limit: ${feed.feed.site.readLimit}`); } break; } default: { console.log('\nNo specialized configuration for this feed type.'); } } ``` ## Python ```python from graphlit import Graphlit graphlit = Graphlit() feed_id = "feed-id-here" feed = await graphlit.client.get_feed(id=feed_id) print(f"Feed: {feed.feed.name}") print(f"Type: {feed.feed.type}") print(f"State: {feed.feed.state}") if feed.feed.type == "SLACK" and feed.feed.slack: print(f"Channel: {feed.feed.slack.channel}") print(f"Listing: {feed.feed.slack.type}") print(f"ReadLimit: {feed.feed.slack.read_limit}") print(f"IncludeAttachments: {feed.feed.slack.include_attachments}") ``` ## .NET ```csharp using GraphlitClient; using System.Net.Http; using StrawberryShake; using var httpClient = new HttpClient(); var graphlit = new Graphlit(httpClient); var feedId = "feed-id-here"; var response = await graphlit.Client.GetFeed.ExecuteAsync(feedId); response.EnsureNoErrors(); var feed = response.Data?.Feed; Console.WriteLine($"Feed: {feed?.Name}"); Console.WriteLine($"Type: {feed?.Type}"); Console.WriteLine($"State: {feed?.State}"); if (feed?.Type == FeedTypes.Slack && feed.Slack != null) { Console.WriteLine($"Channel: {feed.Slack.Channel}"); Console.WriteLine($"Listing: {feed.Slack.Type}"); Console.WriteLine($"ReadLimit: {feed.Slack.ReadLimit}"); Console.WriteLine($"IncludeAttachments: {feed.Slack.IncludeAttachments}"); } ``` ## Parameters * **`id`** (string): Feed ID ## Response ```typescript { feed: { id: string; name: string; type: FeedTypes; state: EntityState; creationDate: Date; slack?: SlackFeedProperties; google?: GoogleFeedProperties; rss?: RSSFeedProperties; web?: WebFeedProperties; } } ``` ## Developer Hints ### Inspect Configuration ```typescript import { FeedServiceTypes, FeedTypes } from 'graphlit-client/dist/generated/graphql-types'; const feed = await graphlit.getFeed(feedId); // Check what type of feed switch (feed.feed.type) { case FeedTypes.Slack: console.log('Slack feed'); console.log('Channel:', feed.feed.slack?.channel); break; case FeedTypes.Web: console.log('Web crawl feed'); console.log('Starting URL:', feed.feed.web?.uri); break; case FeedTypes.Rss: console.log('RSS feed'); console.log('Feed URL:', feed.feed.rss?.uri); break; case FeedTypes.Site: if (feed.feed.site?.type === FeedServiceTypes.GoogleDrive) { console.log('Google Drive feed'); console.log('Folder ID:', feed.feed.site.googleDrive?.folderId); console.log('Files:', feed.feed.site.googleDrive?.files); } break; } ``` ### Check OAuth Token Status ```typescript const feed = await graphlit.getFeed(feedId); // OAuth tokens are not returned for security // If feed is syncing, token is valid // If feed has errors, token may be expired if (feed.feed.state === EntityState.Enabled) { console.log(' Feed is active'); } else { console.log(' Feed is disabled'); } ``` ## Variations ### 1. Basic Feed Retrieval ```typescript const feed = await graphlit.getFeed(feedId); console.log(feed.feed.name); ``` ### 2. Check Configuration ```typescript const feed = await graphlit.getFeed(feedId); console.log('Configuration:'); console.log(` Type: ${feed.feed.type}`); console.log(` State: ${feed.feed.state}`); ``` ### 3. Verify Sync Settings ```typescript const feed = await graphlit.getFeed(feedId); if (feed.feed.web) { console.log('Web crawl settings:'); console.log(` Max pages: ${feed.feed.web.readLimit}`); console.log(` Include files: ${feed.feed.web.includeFiles}`); } ``` ## Production Example **Feed health check**: ```typescript async function checkFeedHealth(feedId: string) { const feed = await graphlit.getFeed(feedId); console.log(`=== FEED HEALTH: ${feed.feed.name} ===`); console.log(`Type: ${feed.feed.type}`); console.log(`State: ${feed.feed.state}`); console.log(`Created: ${new Date(feed.feed.creationDate).toLocaleDateString()}`); // Check if syncing const isDone = await graphlit.isFeedDone(feedId); console.log(`Sync Status: ${isDone.isFeedDone.result ? 'Complete' : 'In Progress'}`); // Count synced content const contents = await graphlit.queryContents({ feeds: [{ id: feedId }] }); console.log(`Content Items: ${contents.contents.results.length}`); // Health indicator const isHealthy = feed.feed.state === EntityState.Enabled && contents.contents.results.length > 0; console.log(`Health: ${isHealthy ? ' Healthy' : ' Issues Detected'}`); } await checkFeedHealth(feedId); ``` # Poll for Completion ## User Intent "I want to know when a feed has finished its initial sync" ## Operation * **SDK Method**: `graphlit.isFeedDone()` * **GraphQL**: `isFeedDone` query * **Entity Type**: Feed * **Common Use Cases**: Wait for initial feed sync, verify feed completion before querying content ## TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; const graphlit = new Graphlit(); // After creating a feed const feedResponse = await graphlit.createFeed(feedInput); const feedId = feedResponse.createFeed.id; console.log(`Feed created: ${feedId}`); // Poll for completion const maxAttempts = 60; // 10 minutes max (60 * 10 seconds) for (let attempts = 1; attempts <= maxAttempts; attempts++) { const status = await graphlit.isFeedDone(feedId); if (status.isFeedDone.result) { console.log('Feed sync complete!'); const contents = await graphlit.queryContents({ feeds: [{ id: feedId }], }); console.log(`Synced ${contents.contents.results.length} items`); return; } console.log(`Still syncing... (${attempts}/${maxAttempts})`); await new Promise((resolve) => setTimeout(resolve, 10_000)); } console.log('Feed sync timeout - still processing'); ``` **Python**: ```python feed_response = await graphlit.create_feed(feed_input) feed_id = feed_response.createFeed.id is_done = False attempts = 0 max_attempts = 60 while not is_done and attempts < max_attempts: status = await graphlit.isFeedDone(feed_id) is_done = status.isFeedDone.result if status.isFeedDone else False if not is_done: attempts += 1 print(f"Still syncing... ({attempts}/{max_attempts})") await asyncio.sleep(10) if is_done: print("Feed sync complete!") ``` **C#**: ```csharp using Graphlit; using System.Threading.Tasks; var graphlit = new Graphlit(); var feedResponse = await graphlit.CreateFeed(feedInput); var feedId = feedResponse.CreateFeed.Id; // Poll for completion (PascalCase) bool isDone = false; int attempts = 0; int maxAttempts = 60; while (!isDone && attempts < maxAttempts) { var status = await graphlit.IsFeedDone(feedId); isDone = status.IsFeedDone?.Result ?? false; if (!isDone) { attempts++; Console.WriteLine($"Still syncing... ({attempts}/{maxAttempts})"); await Task.Delay(10000); // Wait 10 seconds } } if (isDone) { Console.WriteLine("Feed sync complete!"); } ``` ## Parameters ### Required * **`id`** (string): Feed ID to check ## Response ```typescript { isFeedDone: { result: boolean; // true = sync complete, false = still syncing } } ``` ## Developer Hints ### Only for Initial Sync `isFeedDone()` indicates **initial sync completion**, not ongoing monitoring: ```typescript // After feed creation const feed = await graphlit.createFeed(feedInput); // isFeedDone checks initial sync await waitUntilFeedDone(feed.createFeed.id); // After initial sync, feed continues to monitor for new content // You don't need to call isFeedDone again ``` **Important**: Once initial sync completes, the feed continuously monitors for new content automatically. ### Recommended Polling Interval ```typescript // Too frequent (wasteful) await new Promise(resolve => setTimeout(resolve, 1000)); // Every 1 second // Good balance await new Promise(resolve => setTimeout(resolve, 10000)); // Every 10 seconds // For very large feeds await new Promise(resolve => setTimeout(resolve, 30000)); // Every 30 seconds ``` **Why 10 seconds?**: Balance between responsiveness and API efficiency. Initial feed syncs typically take 1-10 minutes depending on content volume. ### šŸ• Timeout Considerations ```typescript // Typical sync times by feed type const timeouts = { rss: 2 minutes, // Small: 10-100 items slack: 5 minutes, // Medium: 100s-1000s of messages googleDrive: 10 minutes, // Large: Many files web: 15 minutes // Very large: Deep crawls }; // Adjust maxAttempts based on feed type const maxAttempts = feedType === 'rss' ? 12 : 60; // 2 min vs 10 min ``` ### Helper Function Pattern ```typescript async function waitForFeedCompletion( feedId: string, timeoutMinutes: number = 10, pollIntervalSeconds: number = 10 ): Promise { const maxAttempts = (timeoutMinutes * 60) / pollIntervalSeconds; let attempts = 0; while (attempts < maxAttempts) { const status = await graphlit.isFeedDone(feedId); if (status.isFeedDone.result) { return true; // Success } attempts++; await new Promise(resolve => setTimeout(resolve, pollIntervalSeconds * 1000)); } return false; // Timeout } // Usage const completed = await waitForFeedCompletion(feedId, 10, 10); if (completed) { console.log('Ready to query content'); } else { console.log('Timeout - feed still processing'); } ``` ## Variations ### 1. Basic Polling with Progress Simple polling with progress updates: ```typescript async function pollFeedWithProgress(feedId: string) { console.log('Waiting for feed sync to complete...'); let isDone = false; let attempts = 0; while (!isDone && attempts < 60) { const status = await graphlit.isFeedDone(feedId); isDone = status.isFeedDone.result || false; if (!isDone) { attempts++; const elapsed = attempts * 10; // seconds console.log(`ā³ ${elapsed}s elapsed...`); await new Promise(resolve => setTimeout(resolve, 10000)); } } console.log(isDone ? ' Complete!' : 'ā° Timeout'); return isDone; } ``` ### 2. Polling with Content Count Tracking Track synced content during polling: ```typescript async function pollWithContentTracking(feedId: string) { let isDone = false; let previousCount = 0; while (!isDone) { const status = await graphlit.isFeedDone(feedId); isDone = status.isFeedDone.result || false; // Check how many items synced so far const contents = await graphlit.queryContents({ feeds: [{ id: feedId }], limit: 1 // Just get count, not all items }); const currentCount = contents.contents.results.length; if (currentCount > previousCount) { console.log(`šŸ“„ Synced ${currentCount} items so far...`); previousCount = currentCount; } if (!isDone) { await new Promise(resolve => setTimeout(resolve, 10000)); } } console.log(` Sync complete! Total: ${previousCount} items`); } ``` ### 3. Parallel Feed Polling Poll multiple feeds simultaneously: ```typescript async function pollMultipleFeeds(feedIds: string[]) { const pollPromises = feedIds.map(async (feedId) => { let isDone = false; while (!isDone) { const status = await graphlit.isFeedDone(feedId); isDone = status.isFeedDone.result || false; if (!isDone) { await new Promise(resolve => setTimeout(resolve, 10000)); } } return feedId; }); // Wait for all feeds to complete const completedFeeds = await Promise.all(pollPromises); console.log(`All ${completedFeeds.length} feeds synced!`); return completedFeeds; } // Usage const feedIds = [feed1Id, feed2Id, feed3Id]; await pollMultipleFeeds(feedIds); ``` ### 4. Exponential Backoff Polling Reduce API calls with backoff: ```typescript async function pollWithBackoff(feedId: string) { let isDone = false; let interval = 5000; // Start at 5 seconds const maxInterval = 60000; // Max 60 seconds while (!isDone) { const status = await graphlit.isFeedDone(feedId); isDone = status.isFeedDone.result || false; if (!isDone) { console.log(`Waiting ${interval / 1000}s before next check...`); await new Promise(resolve => setTimeout(resolve, interval)); // Exponential backoff interval = Math.min(interval * 1.5, maxInterval); } } console.log('Feed sync complete!'); } ``` ### 5. Polling with Timeout Promise Use Promise.race for cleaner timeout: ```typescript async function pollWithTimeout( feedId: string, timeoutMs: number = 600000 // 10 minutes ): Promise { const pollPromise = (async () => { let isDone = false; while (!isDone) { const status = await graphlit.isFeedDone(feedId); isDone = status.isFeedDone.result || false; if (!isDone) { await new Promise(resolve => setTimeout(resolve, 10000)); } } return true; })(); const timeoutPromise = new Promise((resolve) => { setTimeout(() => resolve(false), timeoutMs); }); const completed = await Promise.race([pollPromise, timeoutPromise]); if (!completed) { console.log('ā° Feed sync timeout'); } return completed; } ``` ### 6. Query Content After Completion Complete workflow from feed creation to content query: ```typescript // Create feed const feedResponse = await graphlit.createFeed({ name: 'News RSS', type: FeedTypes.Rss, rss: { uri: 'https://news.example.com/rss' } }); const feedId = feedResponse.createFeed.id; // Wait for sync const completed = await waitForFeedCompletion(feedId); if (completed) { // Query synced content const contents = await graphlit.queryContents({ feeds: [{ id: feedId }], orderBy: OrderByTypes.CreationDate, direction: OrderDirectionTypes.Descending, limit: 10 }); console.log('Latest synced items:'); contents.contents.results.forEach((item, index) => { console.log(`${index + 1}. ${item.name}`); }); } ``` ## Common Issues **Issue**: `isFeedDone()` always returns false\ **Solution**: Feed may be stuck. Check feed state with `getFeed()`. Look for error state. **Issue**: Polling times out but feed has content\ **Solution**: Feed may be partially synced but not "done". Increase timeout or query content anyway. **Issue**: `Feed not found` error\ **Solution**: Verify feed ID is correct. Check that feed wasn't deleted. **Issue**: Feed completes immediately (no content synced)\ **Solution**: Check feed configuration. May have OAuth token issues or incorrect settings. **Issue**: Memory leak from long polling\ **Solution**: Ensure you have proper timeout/max attempts. Don't poll indefinitely. ## Production Example **Poll helper function**: ```typescript async function waitForFeed(feedId: string, maxMinutes: number = 10): Promise { const maxAttempts = maxMinutes * 6; // 6 checks per minute (10s intervals) let attempts = 0; while (attempts < maxAttempts) { const status = await graphlit.isFeedDone(feedId); if (status.isFeedDone.result) { return true; } attempts++; await new Promise(resolve => setTimeout(resolve, 10000)); } return false; } // Usage const feedId = response.createFeed.id; const success = await waitForFeed(feedId, 10); if (success) { const contents = await graphlit.queryContents({ feeds: [{ id: feedId }] }); console.log(`Synced ${contents.contents.results.length} items`); } ``` # Query and List Feeds ## Feed: Query and List Feeds ### User Intent "I want to list all my feeds or find a specific feed by name" ### Operation * **SDK Method**: `graphlit.queryFeeds()` or `graphlit.getFeed()` * **GraphQL**: `queryFeeds` or `getFeed` query * **Entity Type**: Feed * **Common Use Cases**: List active feeds, find feed by name, check feed status, manage feeds ### TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { EntityState, FeedTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Query all feeds const allFeeds = await graphlit.queryFeeds(); console.log(`Total feeds: ${allFeeds.feeds.results.length}`); allFeeds.feeds.results.forEach((feed) => { console.log(`- ${feed.name} (${feed.type}): ${feed.state}`); }); // Query specific feed types const slackFeeds = await graphlit.queryFeeds({ types: [FeedTypes.Slack], }); console.log(`\nSlack feeds: ${slackFeeds.feeds.results.length}`); // Search by name const docFeeds = await graphlit.queryFeeds({ search: 'Documentation', }); console.log(`\nDocumentation feeds: ${docFeeds.feeds.results.length}`); // Get specific feed by ID const feedId = 'feed-id-here'; const feed = await graphlit.getFeed(feedId); console.log(`\nFeed: ${feed.feed.name}`); console.log(`Type: ${feed.feed.type}`); console.log(`State: ${feed.feed.state}`); ``` ## Query all feeds (snake\_case) all\_feeds = await graphlit.queryFeeds() print(f"Total feeds: {len(all\_feeds.feeds.results)}") for feed in all\_feeds.feeds.results: print(f"- {feed.name} ({feed.type}): {feed.state}") ## Query by type slack\_feeds = await graphlit.queryFeeds( filter=FeedFilterInput( types=\[FeedTypes.Slack] ) ) ## Get specific feed feed = await graphlit.getFeed(feed\_id) print(f"Feed: {feed.feed.name}") ```` **C#**: ```csharp using Graphlit; var client = new Graphlit(); // Query all feeds (PascalCase) var allFeeds = await graphlit.QueryFeeds(); Console.WriteLine($"Total feeds: {allFeeds.Feeds.Results.Count}"); foreach (var feed in allFeeds.Feeds.Results) { Console.WriteLine($"- {feed.Name} ({feed.Type}): {feed.State}"); } // Query by type var slackFeeds = await graphlit.QueryFeeds(new FeedFilter { Types = new[] { FeedSlack } }); // Get specific feed var feed = await graphlit.GetFeed(feedId); Console.WriteLine($"Feed: {feed.Feed.Name}"); ```` ### Parameters #### queryFeeds (Optional Filter) * **`types`** (FeedTypes\[]): Filter by feed type * `SLACK`, `GOOGLE`, `RSS`, `WEB`, etc. * **`search`** (string): Search by feed name * **`states`** (EntityState\[]): Filter by state * `ENABLED`, `DISABLED` #### getFeed (Required) * **`id`** (string): Feed ID ### Response #### queryFeeds ```typescript { feeds: { results: Feed[]; // Array of feeds } } interface Feed { id: string; name: string; type: FeedTypes; state: EntityState; creationDate: Date; // Type-specific config (slack, google, rss, web) } ``` #### getFeed ```typescript { feed: { id: string; name: string; type: FeedTypes; state: EntityState; slack?: SlackFeedProperties; google?: GoogleFeedProperties; rss?: RSSFeedProperties; web?: WebFeedProperties; } } ``` ### Developer Hints #### Feed States ```typescript // Check if feed is active const feeds = await graphlit.queryFeeds(); feeds.feeds.results.forEach(feed => { if (feed.state === EntityState.Enabled) { console.log(` ${feed.name} is active`); } else { console.log(` ${feed.name} is disabled`); } }); ``` **Important**: Disabled feeds don't sync new content. Re-enable with `updateFeed()`. #### Find Feed by Name ```typescript // Search for feed const results = await graphlit.queryFeeds({ search: 'Engineering Slack' }); if (results.feeds.results.length > 0) { const feed = results.feeds.results[0]; console.log(`Found feed: ${feed.id}`); } else { console.log('Feed not found'); } ``` #### Filter by Type ```typescript // Get all Slack feeds const slackFeeds = await graphlit.queryFeeds({ types: [FeedTypes.Slack] }); // Get all web crawls const webFeeds = await graphlit.queryFeeds({ types: [FeedTypes.Web] }); // Multiple types const cloudFeeds = await graphlit.queryFeeds({ types: [FeedTypes.Site, FeedTypes.Email] }); ``` #### Check Feed Details ```typescript // Get full feed configuration const feed = await graphlit.getFeed(feedId); // Check type-specific config if (feed.feed.type === FeedTypes.Slack && feed.feed.slack) { console.log('Slack channel:', feed.feed.slack.channel); } if (feed.feed.type === FeedTypes.Web && feed.feed.web) { console.log('Starting URL:', feed.feed.web.uri); console.log('Read limit:', feed.feed.web.readLimit); } ``` ### Variations #### 1. List All Feeds Get all feeds: ```typescript const feeds = await graphlit.queryFeeds(); console.log(`You have ${feeds.feeds.results.length} feeds`); ``` #### 2. Filter by Type Only specific feed types: ```typescript const slackFeeds = await graphlit.queryFeeds({ types: [FeedTypes.Slack] }); console.log('Slack feeds:'); slackFeeds.feeds.results.forEach(feed => { console.log(`- ${feed.name}`); }); ``` #### 3. Search by Name Find feeds matching search: ```typescript const docFeeds = await graphlit.queryFeeds({ search: 'docs' }); console.log(`Found ${docFeeds.feeds.results.length} documentation feeds`); ``` #### 4. Get Feed Details Retrieve specific feed: ```typescript const feed = await graphlit.getFeed(feedId); console.log(`Feed: ${feed.feed.name}`); console.log(`Type: ${feed.feed.type}`); console.log(`Created: ${feed.feed.creationDate}`); console.log(`State: ${feed.feed.state}`); ``` #### 5. List Active Feeds Only Filter by state: ```typescript const activeFeeds = await graphlit.queryFeeds({ states: [EntityState.Enabled] }); console.log(`Active feeds: ${activeFeeds.feeds.results.length}`); ``` #### 6. Feed Inventory Report Generate feed summary: ```typescript const feeds = await graphlit.queryFeeds(); const byType = feeds.feeds.results.reduce((acc, feed) => { acc[feed.type] = (acc[feed.type] || 0) + 1; return acc; }, {} as Record); console.log('Feed Inventory:'); Object.entries(byType).forEach(([type, count]) => { console.log(` ${type}: ${count}`); }); const activeCount = feeds.feeds.results.filter( f => f.state === EntityState.Enabled ).length; console.log(`\nActive: ${activeCount} / ${feeds.feeds.results.length}`); ``` ### Common Issues **Issue**: `Feed not found` error\ **Solution**: Verify feed ID is correct. Feed may have been deleted. Use `queryFeeds()` to list all feeds. **Issue**: Search returns no results\ **Solution**: Search is case-sensitive. Try partial matches. Use `queryFeeds()` without filter to see all feeds. **Issue**: Wrong feed type returned\ **Solution**: Check `types` filter includes correct FeedTypes enum value. **Issue**: Feed appears but no content syncing\ **Solution**: Check feed `state`. If `DISABLED`, feed is not syncing. Re-enable with `updateFeed()`. ### Production Example **Feed management dashboard**: ```typescript // Get all feeds with details const feeds = await graphlit.queryFeeds(); console.log('=== FEED INVENTORY ===\n'); // Group by type const byType = new Map(); feeds.feeds.results.forEach(feed => { const type = feed.type; if (!byType.has(type)) { byType.set(type, []); } byType.get(type)!.push(feed); }); // Print by type for (const [type, feedList] of byType) { console.log(`\n${type} Feeds (${feedList.length}):`); for (const feed of feedList) { const status = feed.state === EntityState.Enabled ? '' : ''; console.log(` ${status} ${feed.name} (${feed.id})`); // Check if syncing is done const isDone = await graphlit.isFeedDone(feed.id); const syncStatus = isDone.isFeedDone.result ? 'Complete' : 'Syncing'; console.log(` Status: ${syncStatus}`); // Count synced content const content = await graphlit.queryContents({ feeds: [{ id: feed.id }], limit: 1 // Just get count }); console.log(` Content: ${content.contents.results.length} items`); } } ``` **Find and update feed**: ```typescript // Find feed by name const results = await graphlit.queryFeeds({ search: 'Engineering Slack' }); if (results.feeds.results.length === 0) { console.log('Feed not found'); } else { const feed = results.feeds.results[0]; console.log(`Found feed: ${feed.name} (${feed.id})`); // Get full details const fullFeed = await graphlit.getFeed(feed.id); // Check if it's Slack feed if (fullFeed.feed.type === FeedTypes.Slack && fullFeed.feed.slack) { console.log('Channels:', fullFeed.feed.slack.channels?.map(c => c.name).join(', ')); } // Check sync status const isDone = await graphlit.isFeedDone(feed.id); console.log(`Sync complete: ${isDone.isFeedDone.result}`); // Count content const content = await graphlit.queryContents({ feeds: [{ id: feed.id }] }); console.log(`Synced items: ${content.contents.results.length}`); } ``` **Health check script**: ```typescript // Check all feeds are healthy const feeds = await graphlit.queryFeeds(); console.log('=== FEED HEALTH CHECK ===\n'); for (const feed of feeds.feeds.results) { const status = { name: feed.name, type: feed.type, enabled: feed.state === EntityState.Enabled, syncComplete: false, contentCount: 0 }; // Check sync status const isDone = await graphlit.isFeedDone(feed.id); status.syncComplete = isDone.isFeedDone.result || false; // Count content const content = await graphlit.queryContents({ feeds: [{ id: feed.id }] }); status.contentCount = content.contents.results.length; // Report const healthIcon = status.enabled && status.contentCount > 0 ? '' : ''; console.log(`${healthIcon} ${status.name}`); console.log(` Enabled: ${status.enabled}`); console.log(` Synced: ${status.syncComplete ? 'Yes' : 'In Progress'}`); console.log(` Content: ${status.contentCount} items\n`); } ``` # Update Feed Configuration ## User Intent "How do I update feed settings? Show me feed updates." ## Operation **SDK Method**: `updateFeed()`\ **Use Case**: Modify feed configuration *** ## Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; const graphlit = new Graphlit(); await graphlit.updateFeed({ id: 'feed-id', name: 'Updated Feed Name', schedulePolicy: { repeatInterval: 'PT30M', }, }); console.log('Feed updated'); ``` *** ## Updatable Fields **name**: Feed display name\ **schedulePolicy**: Sync frequency\ **readLimit**: Items per sync\ **workflow**: Change processing workflow *** # Messaging Connect team collaboration, email, and customer support platforms. ## Team Collaboration * [Slack](/api-guides/use-cases/feeds/messaging/feed-create-slack) - Sync Slack channels * [Microsoft Teams](/api-guides/use-cases/feeds/messaging/feed-create-microsoft-teams) - Teams channels and threads * [Discord](/api-guides/use-cases/feeds/messaging/feed-create-discord) - Discord servers ## Email * [Gmail](/api-guides/use-cases/feeds/messaging/feed-create-gmail) - Google email with OAuth * [Outlook Email](/api-guides/use-cases/feeds/messaging/feed-create-outlook-email) - Microsoft 365 email ## Customer Support * [Intercom Tickets](/api-guides/use-cases/feeds/messaging/feed-create-intercom) - Customer support conversations * [Intercom Articles](/api-guides/use-cases/feeds/messaging/feed-create-intercom-articles) - Help Center articles [ā† Back to Feeds](/api-guides/use-cases/feeds) # Create Discord Feed ## User Intent "How do I sync Discord server messages? Show me Discord feed configuration." ## Operation **SDK Method**: `createFeed()` with FeedTypes.Discord\ **Auth**: Bot token required *** ## Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedListingTypes, DiscordChannelsInput, DiscordGuildsInput, } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); const discordAuth: DiscordGuildsInput = { token: process.env.DISCORD_BOT_TOKEN!, }; // Enumerate guilds (servers) the bot can access const guildsResponse = await graphlit.queryDiscordGuilds(discordAuth); const guilds = guildsResponse.discordGuilds?.results ?? []; if (guilds.length === 0) { throw new Error('No Discord guilds available for the provided bot token'); } const guild = guilds[0]!; console.log(`Using Discord guild: ${guild.guildName} (${guild.guildId})`); // Enumerate channels inside the selected guild const channelsResponse = await graphlit.queryDiscordChannels({ guildId: guild.guildId!, token: discordAuth.token, }); const channels = channelsResponse.discordChannels?.results ?? []; if (channels.length === 0) { throw new Error('No Discord channels available in the selected guild'); } const channel = channels[0]!; console.log(`Using Discord channel: ${channel.channelName}`); const feed = await graphlit.createFeed({ name: 'Discord Server', type: FeedTypes.Discord, discord: { type: FeedListingTypes.Past, token: discordAuth.token, channel: channel.channelName!, readLimit: 500, includeAttachments: true, }, // Optional: assign workflow for custom processing // workflow: { id: workflow.createWorkflow.id } }); console.log(`Created Discord feed: ${feed.createFeed.id}`); ``` *** ## Configuration **token**: Discord bot token (from Developer Portal)\ **channel**: Discord channel name (enumerate via `queryDiscordChannels`)\ **type**: `FeedListingTypes.Past` or `FeedListingTypes.New`\ **readLimit**: Messages per sync window\ **includeAttachments**: Enable to capture file uploads *** ## Discord Bot Setup 1. Discord Developer Portal ā†’ Create Application 2. Bot ā†’ Add Bot 3. Copy bot token 4. Invite bot to server with Read Messages, Read Message History, and Attach Files permissions 5. Use helper queries to discover guild and channel names *** ## What Gets Synced * Text messages * Mentions * Embeds * Attachments * Message reactions (metadata) *** # Create Gmail Feed ## User Intent "How do I sync Gmail messages? Show me Gmail feed configuration." ## Operation **SDK Method**: `createFeed()` with FeedTypes.Email\ **Auth**: Google OAuth tokens *** ## Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedServiceTypes, EmailListingTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); const feed = await graphlit.createFeed({ name: 'Gmail Inbox', type: FeedTypes.Email, email: { type: FeedServiceTypes.GoogleEmail, google: { type: EmailListingTypes.Past, clientId: process.env.GOOGLE_CLIENT_ID!, clientSecret: process.env.GOOGLE_CLIENT_SECRET!, refreshToken: process.env.GOOGLE_REFRESH_TOKEN!, readLimit: 100, inboxOnly: true, excludeSentItems: true }, includeAttachments: true, }, // Optional: add workflow for content processing // workflow: { id: workflow.createWorkflow.id } }); console.log(`Created Gmail feed: ${feed.createFeed.id}`); ``` *** ## Configuration **type**: `EmailListingTypes.Past` or `EmailListingTypes.New`\ **readLimit**: Number of emails to sync\ **inboxOnly**: Only sync inbox (excludes other labels)\ **excludeSentItems**: Exclude sent messages\ **includeSpam**: Include spam folder (default: false)\ **includeDeletedItems**: Include trash (default: false)\ **includeAttachments**: Download email attachments\ **filter**: Gmail search filter (e.g., `"from:user@example.com has:attachment"`) *** ## Gmail OAuth Setup 1. Create project in Google Cloud Console 2. Enable Gmail API 3. Create OAuth 2.0 credentials (Web application) 4. Add redirect URI 5. Get refresh token via OAuth flow *** ## What Gets Synced * Email subject, body, and headers * Sender and recipient information * Timestamps and labels * Attachments (if enabled) * Thread relationships *** ## Filtering Examples **By sender**: ```typescript filter: "from:boss@company.com" ``` **By date range**: ```typescript filter: "after:2024/01/01 before:2024/12/31" ``` **With attachments**: ```typescript filter: "has:attachment" ``` **By label**: ```typescript filter: "label:important" ``` *** ## Related * [Outlook Email Feed](/api-guides/use-cases/feeds/messaging/feed-create-outlook-email) - Microsoft email * [Slack Feed](/api-guides/use-cases/feeds/messaging/feed-create-slack) - Team messaging * [Feed Operations](/api-guides/use-cases/feeds/feed-query) - Query and manage feeds # Create Intercom Feed ## User Intent "How do I sync Intercom conversations? Show me Intercom feed configuration." ## Operation **SDK Method**: `createFeed()` with FeedTypes.Intercom\ **Auth**: API token *** ## Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedServiceTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); const feed = await graphlit.createFeed({ name: 'Customer Support', type: FeedTypes.Intercom, intercom: { type: FeedServiceTypes.IntercomTickets, token: process.env.INTERCOM_TOKEN!, readLimit: 100, }, // Optional: add workflow for content processing // workflow: { id: workflow.createWorkflow.id } }); console.log(`Intercom feed created: ${feed.createFeed.id}`); ``` *** ## Configuration **readLimit**: Max conversations\ **includeResolved**: Include closed tickets\ **assignee**: Filter by assigned user *** # Create Intercom Articles Feed ## User Intent "How do I sync Intercom Help Center articles? Show me Intercom Articles feed configuration." ## Operation **SDK Method**: `createFeed()` with FeedTypes.Intercom\ **Auth**: Intercom access token *** ## Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedServiceTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); const feed = await graphlit.createFeed({ name: 'Help Center', type: FeedTypes.Intercom, intercom: { type: FeedServiceTypes.IntercomArticles, accessToken: process.env.INTERCOM_TOKEN!, readLimit: 200, }, // Optional: add workflow for content processing // workflow: { id: workflow.createWorkflow.id } }); console.log(`Created Intercom Articles feed: ${feed.createFeed.id}`); ``` *** ## Configuration **type**: `FeedServiceTypes.IntercomArticles`\ **accessToken**: Intercom access token\ **readLimit**: Number of articles to sync *** ## Intercom Token Setup 1. Go to Intercom ā†’ Settings ā†’ Developers ā†’ Developer Hub 2. Create new app or select existing app 3. Configure OAuth scopes: `Read articles` 4. Generate access token 5. Set `INTERCOM_TOKEN` environment variable *** ## What Gets Synced * Help Center article titles and content * Article collections and sections * Article metadata (author, status, dates) * Article URLs and translations * Article statistics (views, reactions) *** ## IntercomArticles vs IntercomTickets **Use IntercomArticles for:** * Help Center content * Knowledge base articles * Self-service documentation * Product guides * FAQ content **Use IntercomTickets for:** * Customer support conversations * Support tickets * User inquiries * Chat transcripts *** ## Use Cases **Product Documentation**: ```typescript type: FeedServiceTypes.IntercomArticles ``` **Customer Self-Service**: ```typescript type: FeedServiceTypes.IntercomArticles, readLimit: 500 // Large knowledge base ``` **Multi-language Support**: ```typescript // Intercom automatically includes article translations type: FeedServiceTypes.IntercomArticles ``` *** ## Intercom API Scopes Required OAuth scopes: * `Read articles` - Access help center articles * `Read collections` - Access article collections Optional scopes: * `Read admins` - Get article author info * `Read teams` - Get team assignments *** ## Related * [Intercom Tickets Feed](/api-guides/use-cases/feeds/messaging/feed-create-intercom) - Support conversations * [Zendesk Articles](/api-guides/use-cases/feeds/project-management/feed-create-zendesk-articles) - Zendesk knowledge base * [Notion Feed](/api-guides/use-cases/feeds/project-management/feed-create-notion) - Notion docs # Create Microsoft Teams Feed ## User Intent "How do I sync messages from Microsoft Teams channels? Show me how to create a Teams feed with OAuth." ## Operation **SDK Method**: `createFeed()` with FeedTypes.MicrosoftTeams\ **OAuth**: Required via Developer Portal *** ## Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { FeedListingTypes, FeedTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); const feed = await graphlit.createFeed({ name: 'Engineering Team', type: FeedTypes.MicrosoftTeams, microsoftTeams: { type: FeedListingTypes.Past, clientId: process.env.MICROSOFT_CLIENT_ID!, clientSecret: process.env.MICROSOFT_CLIENT_SECRET!, refreshToken: process.env.MICROSOFT_REFRESH_TOKEN!, teamId: 'team-id', channelId: 'channel-id', readLimit: 1000, }, // Optional: add workflow for content processing // workflow: { id: workflow.createWorkflow.id } }); console.log(`Created Teams feed: ${feed.createFeed.id}`); ``` *** ## Configuration **Channels**: Specify channels or leave empty for all\ **readLimit**: Messages per channel (default: 100)\ **includeAttachments**: Sync file attachments *** ## OAuth Setup 1. Developer Portal ā†’ Connectors ā†’ Messaging 2. Authorize Microsoft Teams 3. Copy OAuth token 4. Use in feed creation *** ## What Gets Synced * Channel messages * Mentions * File attachments (if enabled) * Message metadata (author, timestamp) *** # Create Outlook Email Feed ## User Intent "How do I sync Outlook/Microsoft 365 email? Show me Outlook email feed configuration." ## Operation **SDK Method**: `createFeed()` with FeedTypes.Email\ **Auth**: Microsoft OAuth tokens *** ## Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedServiceTypes, EmailListingTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); const feed = await graphlit.createFeed({ name: 'Outlook Inbox', type: FeedTypes.Email, email: { type: FeedServiceTypes.MicrosoftEmail, microsoft: { type: EmailListingTypes.Past, clientId: process.env.MICROSOFT_CLIENT_ID!, clientSecret: process.env.MICROSOFT_CLIENT_SECRET!, refreshToken: process.env.MICROSOFT_REFRESH_TOKEN!, readLimit: 100, inboxOnly: true, excludeSentItems: true }, includeAttachments: true, }, // Optional: add workflow for content processing // workflow: { id: workflow.createWorkflow.id } }); console.log(`Created Outlook feed: ${feed.createFeed.id}`); ``` *** ## Configuration **type**: `EmailListingTypes.Past` or `EmailListingTypes.New`\ **readLimit**: Number of emails to sync\ **inboxOnly**: Only sync inbox folder\ **excludeSentItems**: Exclude sent messages\ **includeSpam**: Include junk folder (default: false)\ **includeDeletedItems**: Include deleted items (default: false)\ **includeAttachments**: Download email attachments\ **filter**: OData filter query *** ## Microsoft OAuth Setup 1. Register app in Azure Portal 2. Add Microsoft Graph API permissions (`Mail.Read`) 3. Create client secret 4. Configure redirect URI 5. Get refresh token via OAuth flow *** ## What Gets Synced * Email subject, body, and headers * Sender and recipient information * Timestamps and folder names * Attachments (if enabled) * Conversation threads *** ## Filtering Examples **By sender**: ```typescript filter: "from/emailAddress/address eq 'user@example.com'" ``` **By date**: ```typescript filter: "receivedDateTime ge 2024-01-01T00:00:00Z" ``` **With attachments**: ```typescript filter: "hasAttachments eq true" ``` **Unread only**: ```typescript filter: "isRead eq false" ``` *** ## Related * [Gmail Feed](/api-guides/use-cases/feeds/messaging/feed-create-gmail) - Google email * [Outlook Calendar Feed](/api-guides/use-cases/feeds/calendars/feed-create-outlook-calendar) - Microsoft calendar * [Microsoft Teams Feed](/api-guides/use-cases/feeds/messaging/feed-create-microsoft-teams) - Teams messaging # Create Slack Feed ## User Intent "I want to sync Slack channels into Graphlit for search and AI interactions" ## Operation * **SDK Method**: `graphlit.createFeed()` with Slack configuration * **GraphQL**: `createFeed` mutation * **Entity Type**: Feed * **Common Use Cases**: Slack channel sync, team communication search, chat history RAG ## TypeScript (Canonical) ```typescript import { Graphlit } from 'graphlit-client'; import { ContentTypes, FeedInput, FeedListingTypes, FeedTypes, SearchTypes, } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); // Optional: Query available Slack channels (requires token/OAuth) const channelsResponse = await graphlit.querySlackChannels({ token: process.env.SLACK_TOKEN!, }); console.log('Available channels:', channelsResponse.slackChannels?.results); // Step 1: Create a Slack feed for ONE channel (one feed per channel) const feedInput: FeedInput = { name: 'Engineering Slack', type: FeedTypes.Slack, slack: { type: FeedListingTypes.Past, channel: 'engineering', token: process.env.SLACK_TOKEN!, readLimit: 100, includeAttachments: true, }, }; const response = await graphlit.createFeed(feedInput); const feedId = response.createFeed.id; console.log(`Slack feed created: ${feedId}`); // Step 3: Poll for feed completion while (true) { const status = await graphlit.isFeedDone(feedId); if (status.isFeedDone.result) { break; } console.log('Still syncing Slack messages...'); await new Promise((resolve) => setTimeout(resolve, 10_000)); } console.log('Slack feed sync complete!'); ``` ## Parameters ### FeedInput (Required) * **`name`** (string): Display name for the feed * **`type`** (FeedTypes): Must be `SLACK` * **`slack`** (SlackFeedPropertiesInput): Slack-specific configuration ### SlackFeedPropertiesInput (Required) * **`type`** (FeedListingTypes): `Past` (backfill then continue) or `New` (new items only) * **`channel`** (string): Slack channel name (one channel per feed) * **`token`** (string): Slack token (Token auth) or use `connector` (Connector auth) * **`includeAttachments`** (boolean, optional): include file attachments * **`readLimit`** (int, optional): items per poll/run ### Optional * **`correlationId`** (string): For tracking in production * **`collections`** (EntityReferenceInput\[]): Auto-add synced messages to collections * **`workflow`** (EntityReferenceInput): Apply workflow to messages ## Response ```typescript { createFeed: { id: string; // Feed ID name: string; // Feed name state: EntityState; // ENABLED type: FeedTypes.Slack; // SLACK slack: { channel: string; includeAttachments?: boolean; type?: FeedListingTypes; } } } ``` ## Developer Hints ### OAuth Token Requirements **Slack OAuth Scopes Needed**: * `channels:read` - List public channels * `channels:history` - Read public channel messages * `groups:read` - List private channels (if needed) * `groups:history` - Read private channel messages (if needed) * `users:read` - Get user information **Getting a Slack Token**: 1. Create Slack App at 2. Add OAuth scopes under "OAuth & Permissions" 3. Install app to workspace 4. Copy Bot User OAuth Token (starts with `xoxb-`) ### Feed is Continuous Sync ```typescript // Feed continuously monitors for new messages const feed = await graphlit.createFeed(feedInput); // New messages appear automatically as content // No need to manually trigger sync ``` **Important**: Feeds run continuously. To stop syncing, disable or delete the feed. ### Polling for Initial Sync ```typescript // After creating feed, wait for initial sync const feedId = response.createFeed.id; let isDone = false; while (!isDone) { const status = await graphlit.isFeedDone(feedId); isDone = status.isFeedDone.result || false; if (!isDone) { await new Promise(resolve => setTimeout(resolve, 10000)); } } // Now query synced messages const messages = await graphlit.queryContents({ feeds: [{ id: feedId }], types: [ContentTypes.Message] }); console.log(`Synced ${messages.contents.results.length} messages`); ``` ### Channel Discovery ```typescript // List all channels user has access to const channels = await graphlit.querySlackChannels({ token: slackToken }); const channelNames = channels.slackChannels?.results ?? []; // One feed per channel (recommended if you want multiple channels) for (const channel of channelNames.filter((c) => c.startsWith('eng-') || c.startsWith('dev-'))) { await graphlit.createFeed({ name: `Slack #${channel}`, type: FeedTypes.Slack, slack: { type: FeedListingTypes.Past, channel, token: slackToken, }, }); } ``` ## Variations ### 1. Sync Specific Channels Only Create multiple feeds (one per channel): ```typescript await graphlit.createFeed({ name: 'Slack #customer-support', type: FeedTypes.Slack, slack: { type: FeedListingTypes.Past, channel: 'customer-support', token: slackToken, }, }); ``` ### 2. Sync with Auto-Collection Automatically add messages to a collection: ```typescript // Create collection first const collectionResponse = await graphlit.createCollection({ name: 'Slack Messages' }); // Create feed with collection const feedInput: FeedInput = { name: 'Team Slack', type: FeedTypes.Slack, slack: { type: FeedListingTypes.Past, token: slackToken }, collections: [{ id: collectionResponse.createCollection.id }] }; ``` ### 3. Sync with Entity Extraction Extract people and topics from messages: ```typescript // Create workflow for entity extraction const workflowResponse = await graphlit.createWorkflow({ name: 'Extract Slack Entities', extraction: { jobs: [{ connector: { type: EntityExtractionServiceTypes.ModelText, modelText: { extractedTypes: [ ObservableTypes.Person, ObservableTypes.Organization, ObservableTypes.Label ] } } }] } }); // Create feed with workflow const feedInput: FeedInput = { name: 'Slack with Extraction', type: FeedTypes.Slack, slack: { type: FeedListingTypes.Past, token: slackToken }, workflow: { id: workflowResponse.createWorkflow.id } }; ``` ### 4. Query Synced Messages Search through synced Slack messages: ```typescript // After feed sync completes const results = await graphlit.queryContents({ feeds: [{ id: feedId }], types: [ContentTypes.Message], search: 'deployment issues', searchType: SearchTypes.Hybrid }); results.contents.results.forEach(msg => { console.log(`${msg.name}: ${msg.summary}`); }); ``` ### 5. Multi-Channel Pattern with Filtering Sync multiple channels and filter by date: ```typescript const feedInput: FeedInput = { name: 'Recent Engineering Discussions', type: FeedTypes.Slack, slack: { type: FeedListingTypes.Past, channel: 'engineering', token: slackToken } }; const response = await graphlit.createFeed(feedInput); // Wait for sync await waitForFeedCompletion(response.createFeed.id); // Query only recent messages const lastWeek = new Date(); lastWeek.setDate(lastWeek.getDate() - 7); const recentMessages = await graphlit.queryContents({ feeds: [{ id: response.createFeed.id }], creationDateRange: { from: lastWeek, to: new Date() } }); ``` ## Common Issues **Issue**: `Invalid token` error\ **Solution**: Ensure Slack token has required OAuth scopes. Regenerate token with correct scopes. **Issue**: `Channel not found`\ **Solution**: Use `querySlackChannels()` to confirm the exact channel name. The app/user must have access to the channel. **Issue**: Feed created but no messages syncing\ **Solution**: Verify token has `channels:history` scope and the app/user has access to the channel. **Issue**: Feed sync taking too long\ **Solution**: This is normal for channels with many messages. Use `isFeedDone()` to poll. Initial sync can take minutes for large channels. ## Production Example # Create Zendesk Feed # Cloud Storage Sync documents from cloud storage providers and code repositories. ## Cloud Storage * [Google Drive](/api-guides/use-cases/feeds/cloud-storage/feed-create-google-drive) - Google Drive files * [OneDrive](/api-guides/use-cases/feeds/cloud-storage/feed-create-onedrive) - Microsoft OneDrive * [SharePoint](/api-guides/use-cases/feeds/cloud-storage/feed-create-sharepoint) - SharePoint document libraries * [Dropbox](/api-guides/use-cases/feeds/cloud-storage/feed-create-dropbox) - Dropbox files * [Box](/api-guides/use-cases/feeds/cloud-storage/feed-create-box) - Box enterprise storage ## Object Storage * [AWS S3](/api-guides/use-cases/feeds/cloud-storage/feed-create-aws-s3) - Amazon S3 buckets * [Azure Blob Storage](/api-guides/use-cases/feeds/cloud-storage/feed-create-azure-storage) - Azure storage * [Google Cloud Storage](/api-guides/use-cases/feeds/cloud-storage/feed-create-google-cloud-storage) - GCS buckets ## Code Repositories * [GitHub Repository](/api-guides/use-cases/feeds/cloud-storage/feed-create-github) - GitHub repo files and code [ā† Back to Feeds](/api-guides/use-cases/feeds) # Create AWS S3 Feed ## User Intent "How do I sync files from AWS S3 bucket? Show me S3 feed configuration." ## Operation **SDK Method**: `createFeed()` with FeedTypes.Site\ **Feed Service**: AWS S3\ **Auth**: AWS credentials *** ## Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedServiceTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); const feed = await graphlit.createFeed({ name: 'S3 Documents', type: FeedTypes.Site, site: { type: FeedServiceTypes.S3Blob, bucketName: 'my-documents-bucket', prefix: 'documents/', region: 'us-west-2', accessKey: process.env.AWS_ACCESS_KEY!, secretKey: process.env.AWS_SECRET_KEY!, }, // Optional: add workflow for content processing // workflow: { id: workflow.createWorkflow.id } }); console.log(`Created S3 feed: ${feed.createFeed.id}`); ``` *** ## Configuration **bucketName**: S3 bucket name\ **prefix**: Folder path in bucket\ **region**: AWS region\ **accessKey/secretKey**: AWS credentials *** # Create Azure Storage Feed ## User Intent "How do I sync files from Azure Blob Storage? Show me Azure Storage feed configuration." ## Operation **SDK Method**: `createFeed()` with FeedTypes.Site\ **Feed Service**: Azure Blob Storage\ **Auth**: Connection string or SAS token *** ## Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedServiceTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); const feed = await graphlit.createFeed({ name: 'Azure Documents', type: FeedTypes.Site, site: { type: FeedServiceTypes.AzureBlob, azureBlob: { accountName: 'mystorageaccount', containerName: 'documents', prefix: 'reports/', storageAccessKey: process.env.AZURE_STORAGE_KEY!, }, }, // Optional: add workflow for content processing // workflow: { id: workflow.createWorkflow.id } }); console.log(`Created Azure feed: ${feed.createFeed.id}`); ``` *** ## Configuration **containerName**: Blob container name\ **prefix**: Folder path\ **connectionString**: Azure Storage connection string *** # Create Box Feed ## User Intent "How do I sync files from Box? Show me Box feed configuration." ## Operation **SDK Method**: `createFeed()` with FeedTypes.Site\ **Feed Service**: Box\ **OAuth**: Required *** ## Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedServiceTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); const feed = await graphlit.createFeed({ name: 'Company Box', type: FeedTypes.Site, site: { type: FeedServiceTypes.Box, token: process.env.BOX_TOKEN!, folderPath: '/Engineering', fileTypes: ['pdf', 'docx'], }, // Optional: add workflow for content processing // workflow: { id: workflow.createWorkflow.id } }); console.log(`Created Box feed: ${feed.createFeed.id}`); ``` *** # Create Dropbox Feed ## User Intent "How do I sync files from Dropbox? Show me Dropbox feed configuration." ## Operation **SDK Method**: `createFeed()` with FeedTypes.Site\ **Feed Service**: Dropbox\ **OAuth**: Required *** ## Code Example (TypeScript) ```typescript import { Graphlit } from 'graphlit-client'; import { FeedTypes, FeedServiceTypes } from 'graphlit-client/dist/generated/graphql-types'; const graphlit = new Graphlit(); const feed = await graphlit.createFeed({ name: 'My Dropbox', type: FeedTypes.Site, site: { type: FeedServiceTypes.Dropbox, token: process.env.DROPBOX_TOKEN!, folderPath: '/Work/Documents', fileTypes: ['pdf', 'docx', 'txt'], }, // Optional: add workflow for content processing // workflow: { id: workflow.createWorkflow.id } }); console.log(`Created Dropbox feed: ${feed.createFeed.id}`); ``` *** --- [Next Page](/llms-full.txt/1)