Skip to content

Image Search

Image search (or Reverse Image Search) allows you to find visually similar images or retrieve images using natural language descriptions. This is powered by vector embeddings and high-performance indexing.

Objective

Build an image catalog, store visual embeddings, and perform semantic search to find the most relevant images.

Prerequisites

  • Deeplake SDK: pip install deeplake and AI stack: pip install torch transformers pillow accelerate (Python SDK tab)
  • curl and a terminal (REST API tab)
  • A Deeplake API token.

Set credentials first

export DEEPLAKE_API_KEY="your-token-here"
export DEEPLAKE_WORKSPACE="your-workspace"  # optional, defaults to "default"

Complete Code

import os
import uuid
import torch
from PIL import Image
from deeplake import Client
from transformers import AutoModel, AutoProcessor

# 1. Setup ColQwen3 Visual Encoder
MODEL_ID = "TomoroAI/tomoro-colqwen3-embed-4b"
device = "cuda" if torch.cuda.is_available() else "cpu"

processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModel.from_pretrained(
    MODEL_ID, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map=device
).eval()

def get_image_embedding(image_path):
    img = Image.open(image_path).convert("RGB")
    inputs = processor.process_images(images=[img])
    inputs = {k: v.to(device) for k, v in inputs.items()}
    with torch.inference_mode():
        out = model(**inputs)
    return out.embeddings[0].cpu().float().numpy()

# 2. Setup
client = Client()

# 3. Create table and ingest images with multi-vector embeddings
client.query("""
    CREATE TABLE IF NOT EXISTS "image_catalog" (
        _id UUID PRIMARY KEY,
        image IMAGE,
        filename TEXT,
        embedding FLOAT4[][]
    ) USING deeplake
""")

ds = client.open_table("image_catalog")

image_paths = ["beach.png", "city.png"]
ds.append({
    "_id": [str(uuid.uuid4()) for _ in image_paths],
    "image": [open(p, "rb").read() for p in image_paths],
    "filename": [os.path.basename(p) for p in image_paths],
    "embedding": [get_image_embedding(p) for p in image_paths],
})
ds.commit()

client.create_index("image_catalog", "embedding")

# 4. Semantic Visual Retrieval
query_text = "sunset over the ocean"
query_inputs = processor.process_texts(texts=[query_text])
query_inputs = {k: v.to(device) for k, v in query_inputs.items()}
with torch.inference_mode():
    query_emb = model(**query_inputs).embeddings[0].cpu().float().numpy().tolist()

# Format multi-vector as PG array literal: {{v1},{v2},...}
emb_pg = "{" + ",".join(
    "{" + ",".join(str(v) for v in row) + "}" for row in query_emb
) + "}"

results = client.query(f"""
    SELECT filename, embedding <#> '{emb_pg}'::float4[][] AS score
    FROM image_catalog ORDER BY score DESC LIMIT 5
""")

for r in results:
    print(f"Match: {r['filename']} (Score: {r['score']:.4f})")

# Requires: export DEEPLAKE_API_KEY="..." (see quickstart)
# Requires: export DEEPLAKE_ORG_ID="your-org-id"
API_URL="https://api.deeplake.ai"
TABLE="image_catalog"

# 1. Create table
curl -s -X POST "$API_URL/workspaces/$DEEPLAKE_WORKSPACE/tables/query" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DEEPLAKE_API_KEY" \
  -H "X-Activeloop-Org-Id: $DEEPLAKE_ORG_ID" \
  -d '{
    "query": "CREATE TABLE IF NOT EXISTS \"'$DEEPLAKE_WORKSPACE'\".\"'$TABLE'\" (id SERIAL PRIMARY KEY, filename TEXT, embedding FLOAT4[][]) USING deeplake"
  }'

# 2. Insert metadata + embeddings (placeholder embeddings)
curl -s -X POST "$API_URL/workspaces/$DEEPLAKE_WORKSPACE/tables/query" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DEEPLAKE_API_KEY" \
  -H "X-Activeloop-Org-Id: $DEEPLAKE_ORG_ID" \
  -d '{
    "query": "INSERT INTO \"'$DEEPLAKE_WORKSPACE'\".\"'$TABLE'\" (filename, embedding) VALUES ($1, $2::float4[][])",
    "params": ["beach.png", "{0.1,0.2,0.3}"]
  }'

curl -s -X POST "$API_URL/workspaces/$DEEPLAKE_WORKSPACE/tables/query" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DEEPLAKE_API_KEY" \
  -H "X-Activeloop-Org-Id: $DEEPLAKE_ORG_ID" \
  -d '{
    "query": "INSERT INTO \"'$DEEPLAKE_WORKSPACE'\".\"'$TABLE'\" (filename, embedding) VALUES ($1, $2::float4[][])",
    "params": ["city.png", "{0.1,0.2,0.3}"]
  }'

# 3. Search
curl -s -X POST "$API_URL/workspaces/$DEEPLAKE_WORKSPACE/tables/query" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $DEEPLAKE_API_KEY" \
  -H "X-Activeloop-Org-Id: $DEEPLAKE_ORG_ID" \
  -d '{
    "query": "SELECT filename, embedding <#> $1::float4[][] AS score FROM \"'$DEEPLAKE_WORKSPACE'\".\"'$TABLE'\" ORDER BY score DESC LIMIT 5",
    "params": ["{0.1,0.2,0.3}"]
  }'

Step-by-Step Breakdown

1. Multi-Vector Embeddings

ColQwen3 produces one embedding vector per visual token, giving a float4[][] per image (ColBERT-style late interaction). This captures more visual detail than a single vector, improving retrieval quality for complex queries.

2. The Vector Index

client.create_index() builds a deeplake_index on the embedding column. The index uses optimized C++ kernels to compute multi-vector similarity directly where the data resides.

3. Retrieval with <#>

The <#> operator computes similarity between query and stored multi-vectors. Higher scores represent closer matches. Use ORDER BY score DESC to get the top results.

4. Multi-Vector PG Format

Multi-vector embeddings use nested braces in PostgreSQL: {{0.1,0.2},{0.3,0.4}}. The formatting helper converts the 2D array returned by the model into this literal.

What to try next