Skip to content

Search

Deeplake supports four search modes. All use the <#> operator. All go through SQL.

Setup

Set DEEPLAKE_API_KEY and DEEPLAKE_WORKSPACE as environment variables (see Quickstart).

Set credentials first

export DEEPLAKE_API_KEY="your-token-here"
export DEEPLAKE_WORKSPACE="your-workspace"  # optional, defaults to "default"

import os
from deeplake import Client

client = Client()
WORKSPACE = os.environ["DEEPLAKE_WORKSPACE"]
TABLE = "documents"

API_URL="https://api.deeplake.ai"
TABLE="documents"
export DEEPLAKE_ORG_ID="your-org-id"

Ingest with automatic indexing

Use the index parameter in client.ingest() to automatically create indexes on specified columns after ingestion:

client.ingest("documents", {
    "title": ["Getting started", "Vector search", "Hybrid search"],
    "content": [
        "Deeplake unifies tables, files, and search.",
        "Use the <#> operator for similarity queries.",
        "Combine BM25 and vector for best results.",
    ],
    "embedding": [
        [0.1, 0.2, 0.3, 0.4, 0.5],
        [0.2, 0.3, 0.4, 0.5, 0.6],
        [0.3, 0.1, 0.2, 0.6, 0.4],
    ],
}, index=["embedding", "content"])

This creates a deeplake_index on both embedding (vector) and content (BM25), enabling all search modes below. For manual index creation, see Indexes.

Matches meaning, not keywords. Use when the query and the data don't share exact words.

Good for: "same bug, different error message", "scenes like this clip", "documents that imply this requirement."

Requires: a vector index on a FLOAT4[] column.

embedding = [0.1, 0.2, 0.3, 0.4, 0.5]  # from any encoder
emb_pg = "{" + ",".join(str(x) for x in embedding) + "}"

rows = client.query(f"""
    SELECT title, content, embedding <#> $1::float4[] AS score
    FROM "{WORKSPACE}"."{TABLE}"
    ORDER BY score DESC
    LIMIT 10
""", (emb_pg,))

curl -s -X POST "$API_URL/workspaces/$DEEPLAKE_WORKSPACE/tables/query" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DEEPLAKE_API_KEY" \
  -H "X-Activeloop-Org-Id: $DEEPLAKE_ORG_ID" \
  -d '{
    "query": "SELECT title, content, embedding <#> ARRAY[0.1,0.2,0.3,0.4,0.5]::float4[] AS score FROM \"YOUR_WORKSPACE\".\"documents\" ORDER BY score DESC LIMIT 10"
  }'

Matches exact words. Use when precise terms matter.

Good for: stack traces, function names, ticket IDs, error codes.

Requires: a BM25 index on a TEXT column.

search_text = "authentication timeout error"

rows = client.query(f"""
    SELECT title, content, content <#> $1 AS score
    FROM "{WORKSPACE}"."{TABLE}"
    ORDER BY score DESC
    LIMIT 10
""", (search_text,))

curl -s -X POST "$API_URL/workspaces/$DEEPLAKE_WORKSPACE/tables/query" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DEEPLAKE_API_KEY" \
  -H "X-Activeloop-Org-Id: $DEEPLAKE_ORG_ID" \
  -d '{
    "query": "SELECT title, content, content <#> 'authentication timeout error' AS score FROM \"YOUR_WORKSPACE\".\"documents\" ORDER BY score DESC LIMIT 10"
  }'

Note: BM25 scores use DESC ordering (higher score = more relevant match).

Combines vector and BM25 in a single query. Best default for most use cases.

Reduces "semantic drift" (vector-only problem) and "keyword brittleness" (BM25-only problem).

Requires: both a vector index and a BM25 index on the same table.

search_text = "fix authentication timeout"
embedding = [0.1, 0.2, 0.3, 0.4, 0.5]  # embed the same text
emb_pg = "{" + ",".join(str(x) for x in embedding) + "}"

rows = client.query(f"""
    SELECT title, content,
           (embedding, content)::deeplake_hybrid_record <#> deeplake_hybrid_record($1::float4[], $2, 0.5, 0.5) AS score
    FROM "{WORKSPACE}"."{TABLE}"
    ORDER BY score DESC
    LIMIT 10
""", (emb_pg, search_text))

curl -s -X POST "$API_URL/workspaces/$DEEPLAKE_WORKSPACE/tables/query" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DEEPLAKE_API_KEY" \
  -H "X-Activeloop-Org-Id: $DEEPLAKE_ORG_ID" \
  -d '{
    "query": "SELECT title, content, (embedding, content) <#> deeplake_hybrid_record(ARRAY[0.1,0.2,0.3,0.4,0.5]::float4[], 'fix authentication timeout', 0.5, 0.5) AS score FROM \"YOUR_WORKSPACE\".\"documents\" ORDER BY score DESC LIMIT 10"
  }'

The last two arguments (0.5, 0.5) are weights for vector and BM25 respectively. Adjust them based on your use case:

Weights Best for
0.7, 0.3 Conceptual queries where meaning matters more
0.5, 0.5 Balanced default
0.3, 0.7 Precise queries where exact keywords matter

Multi-vector search (MaxSim)

Uses a bag of embeddings per item instead of a single vector. Catches fine details.

Good for: a specific action inside a video, a small object inside an image, a key sentence inside a long document.

Requires: a vector index on a FLOAT4[][] column.

# Multi-vector embedding: list of lists (tokens x dim)
multi_embedding = [[0.1, 0.2, 0.3, 0.4], [0.5, 0.6, 0.7, 0.8]]
multi_pg = "ARRAY[" + ",".join(
    "ARRAY[" + ",".join(str(x) for x in vec) + "]" for vec in multi_embedding
) + "]::float4[][]"

rows = client.query(f"""
    SELECT title, file_id,
           embedding <#> {multi_pg} AS score
    FROM "{WORKSPACE}"."video_chunks"
    ORDER BY score DESC
    LIMIT 10
""")

curl -s -X POST "$API_URL/workspaces/$DEEPLAKE_WORKSPACE/tables/query" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DEEPLAKE_API_KEY" \
  -H "X-Activeloop-Org-Id: $DEEPLAKE_ORG_ID" \
  -d '{
    "query": "SELECT title, file_id, embedding <#> ARRAY[ARRAY[0.1,0.2,0.3,0.4],ARRAY[0.5,0.6,0.7,0.8]]::float4[][] AS score FROM \"YOUR_WORKSPACE\".\"video_chunks\" ORDER BY score DESC LIMIT 10"
  }'

Multi-vector search uses MaxSim-style late interaction scoring: each query token is matched to the best-matching part of the item.

Filtering

You can combine search with SQL WHERE clauses. Filtering happens before ranking, reducing compute.

embedding = [0.1, 0.2, 0.3, 0.4, 0.5]
emb_pg = "{" + ",".join(str(x) for x in embedding) + "}"

rows = client.query(f"""
    SELECT title, content,
           embedding <#> $1::float4[] AS score
    FROM "{WORKSPACE}"."{TABLE}"
    WHERE created_at > '2025-01-01'
      AND title LIKE '%%report%%'
    ORDER BY score DESC
    LIMIT 10
""", (emb_pg,))

curl -s -X POST "$API_URL/workspaces/$DEEPLAKE_WORKSPACE/tables/query" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DEEPLAKE_API_KEY" \
  -H "X-Activeloop-Org-Id: $DEEPLAKE_ORG_ID" \
  -d '{
    "query": "SELECT title, content, embedding <#> ARRAY[0.1,0.2,0.3,0.4,0.5]::float4[] AS score FROM \"YOUR_WORKSPACE\".\"documents\" WHERE created_at > '2025-01-01' AND title LIKE '%report%' ORDER BY score DESC LIMIT 10"
  }'

This is a key advantage of SQL-first retrieval: structured filters and similarity search in one query.

Choosing the right mode

Mode Use when
Vector Query is conceptual, no shared keywords
BM25 Query contains exact identifiers or terms
Hybrid Default choice, covers both cases
Multi-vector Items are complex (long text, images, video) and details matter

Next steps