> For the complete documentation index, see [llms.txt](https://docs.graphlit.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.graphlit.dev/api-guides/use-cases/feeds/web/feed-create-web-crawl.md).

# Create Web Crawl Feed

## User Intent

"I want to crawl a website and ingest all pages for search and AI interactions"

## Operation

* **SDK Method**: `graphlit.createFeed()` with web configuration
* **GraphQL**: `createFeed` mutation
* **Entity Type**: Feed
* **Common Use Cases**: Website crawling, documentation ingestion, site mirroring, web scraping

## TypeScript (Canonical)

```typescript
import { Graphlit } from 'graphlit-client';
import { ContentTypes, FeedTypes } from 'graphlit-client/dist/generated/graphql-types';

const graphlit = new Graphlit();

const response = await graphlit.createFeed({
  name: 'Company Documentation',
  type: FeedTypes.Web,
  web: {
    uri: 'https://docs.example.com',
    readLimit: 500,
    includeFiles: true,
    allowedDomains: ['docs.example.com'],
    excludedPaths: ['/api/', '/archive/'],
  },
});

const feedId = response.createFeed.id;
console.log(`Web crawl feed created: ${feedId}`);

// Poll for crawl completion
while (true) {
  const status = await graphlit.isFeedDone(feedId);
  if (status.isFeedDone.result) {
    break;
  }

  console.log('Still crawling website...');
  await new Promise((resolve) => setTimeout(resolve, 15_000));
}

console.log('Web crawl complete!');

const pages = await graphlit.queryContents({
  feeds: [{ id: feedId }],
  types: [ContentTypes.Page],
});

console.log(`Crawled ${pages.contents.results.length} pages`);
```

## Parameters

### FeedInput (Required)

* **`name`** (string): Feed name
* **`type`** (FeedTypes): Must be `WEB`
* **`web`** (WebFeedPropertiesInput): Web crawl configuration

### WebFeedPropertiesInput (Required)

* **`uri`** (string): Starting URL to crawl
  * Crawl starts here and follows links
* **`readLimit`** (int): Max pages to crawl
  * Safety limit to prevent runaway crawls
  * Typical: 100-1000 pages

### Optional (Highly Recommended)

* **`includeFiles`** (boolean): Download linked files (PDFs, docs)
  * Default: false
  * Recommended: true for comprehensive ingestion
* **`allowedDomains`** (string\[]): Domains to crawl
  * Prevents crawling external sites
  * Example: `['docs.example.com', 'help.example.com']`
* **`excludedPaths`** (string\[]): URL paths to skip
  * Example: `['/admin/', '/login/', '/api/']`
* **`allowedPaths`** (string\[]): Only crawl these paths
  * Example: `['/docs/', '/guides/']`

### Other Optional

* **`correlationId`** (string): For tracking
* **`collections`** (EntityReferenceInput\[]): Auto-add pages to collections
* **`workflow`** (EntityReferenceInput): Apply workflow to pages

## Response

```typescript
{
  createFeed: {
    id: string;              // Feed ID
    name: string;            // Feed name
    state: EntityState;      // ENABLED
    type: FeedTypes.Web;     // WEB
    web: {
      uri: string;           // Starting URL
      readLimit: number;     // Page limit
      includeFiles: boolean;
      allowedDomains: string[];
      excludedPaths: string[];
    }
  }
}
```

## Developer Hints

### Always Set Domain Restrictions

**Without restrictions** (dangerous):

```typescript
//  BAD - Could crawl entire internet
const feed = {
  web: {
    uri: 'https://example.com',
    readLimit: 10000
  }
};
```

**With restrictions** (safe):

```typescript
//  GOOD - Stays on specified domains
const feed = {
  web: {
    uri: 'https://docs.example.com',
    readLimit: 1000,
    allowedDomains: ['docs.example.com']
  }
};
```

**Important**: Always use `allowedDomains` to prevent crawling external sites.

### Read Limit Strategy

```typescript
// Small site (< 100 pages)
readLimit: 150

// Medium site (100-1000 pages)
readLimit: 1500

// Large site (1000+ pages)
readLimit: 5000

// Documentation site (typical)
readLimit: 500-1000
```

**Important**: Set `readLimit` higher than expected page count to ensure complete crawl.

### Path Filtering Patterns

```typescript
// Exclude admin/auth pages
excludedPaths: ['/admin/', '/login/', '/logout/', '/auth/']

// Exclude API docs
excludedPaths: ['/api/', '/swagger/']

// Only crawl documentation
allowedPaths: ['/docs/', '/guides/', '/tutorials/']

// Exclude specific sections
excludedPaths: ['/blog/', '/news/', '/archive/']
```

### File Ingestion

```typescript
// Include linked files (PDFs, Word docs, etc.)
const feed = {
  web: {
    uri: 'https://docs.example.com',
    readLimit: 500,
    includeFiles: true,  // Download and ingest PDFs, DOCX, etc.
    allowedDomains: ['docs.example.com']
  }
};

// Then query files separately
const files = await graphlit.queryContents({
  feeds: [{ id: feedId }],
  types: [ContentTypes.File]
});

const pages = await graphlit.queryContents({
  feeds: [{ id: feedId }],
  types: [ContentTypes.Page]
});

console.log(`Crawled ${pages.contents.results.length} pages and ${files.contents.results.length} files`);
```

## Variations

### 1. Basic Documentation Crawl

Simplest site crawl:

```typescript
const feed = await graphlit.createFeed({
  name: 'Docs Crawl',
  type: FeedTypes.Web,
  web: {
    uri: 'https://docs.example.com',
    readLimit: 500,
    allowedDomains: ['docs.example.com']
  }
});
```

### 2. Crawl with File Downloads

Include PDFs and documents:

```typescript
const feed = await graphlit.createFeed({
  name: 'Docs with Files',
  type: FeedTypes.Web,
  web: {
    uri: 'https://docs.example.com',
    readLimit: 1000,
    includeFiles: true,
    allowedDomains: ['docs.example.com']
  }
});
```

### 3. Targeted Path Crawl

Only specific sections:

```typescript
const feed = await graphlit.createFeed({
  name: 'API Documentation Only',
  type: FeedTypes.Web,
  web: {
    uri: 'https://docs.example.com/api',
    readLimit: 200,
    allowedDomains: ['docs.example.com'],
    allowedPaths: ['/api/']  // Only /api/* pages
  }
});
```

### 4. Multi-Domain Crawl

Crawl multiple related domains:

```typescript
const feed = await graphlit.createFeed({
  name: 'All Company Docs',
  type: FeedTypes.Web,
  web: {
    uri: 'https://docs.example.com',
    readLimit: 2000,
    allowedDomains: [
      'docs.example.com',
      'help.example.com',
      'support.example.com'
    ]
  }
});
```

### 5. Crawl with Workflow

Apply preparation workflow to pages:

```typescript
// Create workflow for better extraction
const workflow = await graphlit.createWorkflow({
  name: 'Web Page Prep',
  preparation: {
    jobs: [{
      connector: {
        type: FilePreparationServiceTypes.ModelDocument,
        modelDocument: {
          specification: { id: visionSpecId }
        }
      }
    }]
  }
});

// Create feed with workflow
const feed = await graphlit.createFeed({
  name: 'Docs with Prep',
  type: FeedTypes.Web,
  web: {
    uri: 'https://docs.example.com',
    readLimit: 500,
    includeFiles: true,
    allowedDomains: ['docs.example.com']
  },
  workflow: { id: workflow.createWorkflow.id }
});
```

### 6. Exclude Unwanted Sections

Skip specific paths:

```typescript
const feed = await graphlit.createFeed({
  name: 'Docs (No Archive)',
  type: FeedTypes.Web,
  web: {
    uri: 'https://docs.example.com',
    readLimit: 1000,
    allowedDomains: ['docs.example.com'],
    excludedPaths: [
      '/archive/',
      '/deprecated/',
      '/old-versions/',
      '/beta/',
      '/admin/'
    ]
  }
});
```

## Common Issues

**Issue**: Crawl takes very long\
**Solution**: This is normal for large sites. Use higher polling interval (30-60 seconds). Crawls can take 15-60 minutes for 500+ pages.

**Issue**: Not all pages crawled\
**Solution**: Increase `readLimit`. Check `allowedDomains` and `allowedPaths` aren't too restrictive. Some pages may not be linked.

**Issue**: Crawling external sites\
**Solution**: Set `allowedDomains` to prevent crawling links to external sites.

**Issue**: Crawling unwanted sections\
**Solution**: Use `excludedPaths` to skip admin, login, API routes, etc.

**Issue**: Files (PDFs) not ingested\
**Solution**: Ensure `includeFiles: true` is set. Check files are actually linked from crawled pages.

**Issue**: Duplicate pages\
**Solution**: Graphlit deduplicates by URL. If site has URL parameters, duplicates may occur. Use `excludedPaths` to skip query params.

## Production Example

**Complete documentation ingestion**:

```typescript
// 1. Create collection
const collection = await graphlit.createCollection({
  name: 'Product Documentation'
});

// 2. Create preparation workflow
const workflow = await graphlit.createWorkflow({
  name: 'Web Prep',
  preparation: {
    jobs: [{
      connector: {
        type: FilePreparationServiceTypes.ModelDocument,
        modelDocument: {
          specification: { id: visionSpecId }
        }
      }
    }]
  }
});

// 3. Create web crawl feed
const feed = await graphlit.createFeed({
  name: 'Documentation Crawl',
  type: FeedTypes.Web,
  web: {
    uri: 'https://docs.example.com',
    readLimit: 1000,
    includeFiles: true,
    allowedDomains: ['docs.example.com'],
    excludedPaths: ['/api/', '/admin/', '/login/']
  },
  collections: [{ id: collection.createCollection.id }],
  workflow: { id: workflow.createWorkflow.id }
});

// 4. Wait for crawl
console.log('Starting crawl...');
const feedId = feed.createFeed.id;
let isDone = false;
let checkCount = 0;

while (!isDone) {
  const status = await graphlit.isFeedDone(feedId);
  isDone = status.isFeedDone.result || false;
  
  if (!isDone) {
    checkCount++;
    
    // Show progress
    const crawled = await graphlit.queryContents({
      feeds: [{ id: feedId }]
    });
    
    console.log(`Check ${checkCount}: ${crawled.contents.results.length} pages crawled so far...`);
    await new Promise(resolve => setTimeout(resolve, 30000)); // 30 seconds
  }
}

console.log(' Crawl complete!');

// 5. Get final stats
const pages = await graphlit.queryContents({
  feeds: [{ id: feedId }],
  types: [ContentTypes.Page]
});

const files = await graphlit.queryContents({
  feeds: [{ id: feedId }],
  types: [ContentTypes.File]
});

console.log(`\nCrawl Results:`);
console.log(`- ${pages.contents.results.length} pages`);
console.log(`- ${files.contents.results.length} files`);

// 6. Enable RAG queries
const answer = await graphlit.promptConversation({
  prompt: 'How do I configure authentication?',
  collections: [{ id: collection.createCollection.id }]
});

console.log(`\nRAG Answer: ${answer.message.message}`);
```

**Multi-site documentation aggregation**:

```typescript
// Crawl multiple documentation sites
const docSites = [
  { name: 'Main Docs', url: 'https://docs.example.com', limit: 1000 },
  { name: 'API Docs', url: 'https://api.example.com/docs', limit: 500 },
  { name: 'Help Center', url: 'https://help.example.com', limit: 300 }
];

const collection = await graphlit.createCollection({
  name: 'All Documentation'
});

for (const site of docSites) {
  const feed = await graphlit.createFeed({
    name: site.name,
    type: FeedTypes.Web,
    web: {
      uri: site.url,
      readLimit: site.limit,
      includeFiles: true,
      allowedDomains: [new URL(site.url).hostname]
    },
    collections: [{ id: collection.createCollection.id }]
  });
  
  console.log(`Started crawl: ${site.name}`);
}

// All sites crawl in parallel
console.log('All crawls started. Pages will be available as they\'re crawled.');
```