> For the complete documentation index, see [llms.txt](https://docs.graphlit.dev/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.graphlit.dev/api-guides/use-cases/knowledge-graph/observable-entity-deduplication.md).

# Understanding Entity Deduplication

## User Intent

"How does Graphlit handle duplicate entities? Why do I sometimes see 'Kirk Marple' and 'Kirk' as separate entities?"

## Operation

**Concept**: Entity resolution and deduplication\
**SDK Methods**: `queryObservables()` for finding potential duplicates\
**Entity**: Observable deduplication behavior

## Prerequisites

* Knowledge graph with entities
* Understanding of Observable model
* Multiple content sources with entity mentions

***

## How Deduplication Works

### Automatic Deduplication

**At Creation Time**:

```typescript
// When entities are extracted, Graphlit attempts to deduplicate
// "Kirk Marple" mentioned in 10 documents → 1 Observable with 10 Observations
```

**Deduplication Strategies**:

1. **Exact Name Match**: "Kirk Marple" = "Kirk Marple"
2. **Email Matching** (for Person): <kirk@graphlit.com> always same person
3. **URL Matching** (for Organization): graphlit.com domain
4. **Normalization**: Case-insensitive, whitespace trimming

### Race Conditions

**Problem**: Parallel processing can create duplicates

```typescript
// Document 1 and Document 2 processed simultaneously
// Both mention "Kirk Marple" for first time
// May create 2 separate Observables before deduplication runs
```

**When This Happens**:

* Multiple feeds syncing in parallel
* Batch ingestion of many documents
* High-frequency entity creation

**Future Improvement**: More robust entity resolution is roadmap item

***

## Finding Duplicates

### Query Similar Entities

```typescript
import { Graphlit } from 'graphlit-client';

const graphlit = new Graphlit();

// Find all "Kirk" variants
const kirkEntities = await graphlit.queryObservables({
  search: "Kirk",
  filter: { types: [ObservableTypes.Person] }
});

console.log(`Found ${kirkEntities.observables.results.length} entities matching "Kirk"`);

kirkEntities.observables.results.forEach(entity => {
  console.log(`  - ${entity.observable.name} (ID: ${entity.observable.id})`);
  console.log(`    Email: ${entity.observable.properties?.email || 'N/A'}`);
});
```

### Identify Potential Duplicates

```typescript
function findPotentialDuplicates(entities: Observable[]): Map<string, Observable[]> {
  const groups = new Map<string, Observable[]>();
  
  entities.forEach(entity => {
    const normalized = entity.observable.name.toLowerCase().trim();
    
    // Group by normalized name
    if (!groups.has(normalized)) {
      groups.set(normalized, []);
    }
    groups.get(normalized)!.push(entity);
  });
  
  // Return only groups with duplicates
  return new Map(
    Array.from(groups.entries()).filter(([_, group]) => group.length > 1)
  );
}

const allPeople = await graphlit.queryObservables({
  filter: { types: [ObservableTypes.Person] }
});

const duplicates = findPotentialDuplicates(allPeople.observables.results);

console.log(`Found ${duplicates.size} potential duplicate groups`);
duplicates.forEach((group, name) => {
  console.log(`\n${name}:`);
  group.forEach(entity => {
    console.log(`  - ID: ${entity.observable.id}`);
  });
});
```

***

## Working with Duplicates

### Query All Variants

```typescript
// Find content mentioning ANY variant of entity
const kirkVariants = await graphlit.queryObservables({
  search: "Kirk",
  filter: { types: [ObservableTypes.Person] }
});

const allKirkContent = await Promise.all(
  kirkVariants.observables.results.map(variant =>
    graphlit.queryContents({
      
        observations: [{
          type: ObservableTypes.Person,
          observable: { id: variant.observable.id }
        }]
      })
  )
);

const totalMentions = allKirkContent.reduce(
  (sum, result) => sum + result.contents.results.length,
  0
);

console.log(`Total content mentioning Kirk: ${totalMentions}`);
```

### Disambiguate by Properties

```typescript
// Use email or other properties to identify correct entity
const kirkWithEmail = kirkVariants.observables.results.find(
  entity => entity.observable.properties?.email === 'kirk@graphlit.com'
);

if (kirkWithEmail) {
  console.log(`Canonical Kirk Marple: ${kirkWithEmail.observable.id}`);
}
```

***

## Best Practices

### 1. Use Unique Identifiers

When available, use email (Person) or URL (Organization):

```typescript
// Search by email for Person entities
const person = await graphlit.queryObservables({
  search: "kirk@graphlit.com",
  filter: { types: [ObservableTypes.Person] }
});
```

### 2. Aggregate Across Variants

Combine mentions from all duplicate entities:

```typescript
const allVariants = await graphlit.queryObservables({
  search: "Kirk Marple OR Kirk",
  filter: { types: [ObservableTypes.Person] }
});

// Aggregate data from all variants
```

### 3. Normalize in UI

Display normalized names in UI:

```typescript
function normalizeEntityName(name: string): string {
  // "kirk marple" and "Kirk Marple" → "Kirk Marple"
  return name.split(' ')
    .map(word => word.charAt(0).toUpperCase() + word.slice(1).toLowerCase())
    .join(' ');
}
```

***

## Future Improvements

**Roadmap Items** (not yet available):

* Manual entity merging API
* More sophisticated entity resolution
* Cross-source entity linking
* Entity resolution confidence scores

**Current Limitations**:

* No API to manually merge duplicates
* Some race conditions create duplicates
* Name variants not always linked

**Workarounds**:

* Query all variants and aggregate
* Use unique identifiers (email, URL)
* Filter by properties to disambiguate

***

## Developer Hints

* Duplicates are rare but possible
* Most common after parallel batch ingestion
* Email/URL properties help disambiguation
* Query by identifier, not just name
* Future releases will improve resolution
* Not a critical issue for most use cases

***