EmbImage

Overview

CapyDB now supports image embeddings as well. By wrapping your base64 encoded image data in EmbImage, you can leverage both text and vision models to extract meaningful features from your images. This allows for semantic searches that go beyond traditional keyword matching.

Key Points:

• Image Data Handling: Provide your image data in base64 format.
• Dual-Model Support: Optionally use an embedding model and a vision model to generate rich, multi-modal representations.
• Asynchronous Processing: Image embeddings and chunking occur in the background, ensuring a responsive write experience.
• Automatic Chunking: Images are internally processed (and chunked, if applicable) for efficient embedding and indexing.

Basic Usage

Below is the simplest way to use EmbImage:

from capydb import EmbImage

# Storing a single image field to embed.
{
  "field_name": EmbImage("https://example.com/image.jpg", mime_type="image/jpeg")
}

This snippet creates an EmbImage object containing your image URL. The mime_type parameter is required to specify the image format. By default, no specific models are set and all other parameters remain optional.

Customized Usage with Parameters

If you have specific requirements (e.g., using a particular embedding or vision model), customize EmbImage by specifying additional parameters:

from capydb import EmbImage, EmbModels, VisionModels

{
    "field_name": EmbImage(
        url="https://example.com/image.jpg",  # URL to the image
        mime_type="image/jpeg",                # Required: specify the image format
        emb_model=EmbModels.TEXT_EMBEDDING_3_LARGE,  # Optionally specify an embedding model
        vision_model=VisionModels.GPT_4O,             # Optionally specify a vision model
        max_chunk_size=200,                           # Configure chunk sizes
        chunk_overlap=20,                             # Overlap between chunks
        is_separator_regex=False,                     # Are separators plain strings or regex?
        separators=[
            "\n\n",
            "\n",
        ],
        keep_separator=False                          # Keep or remove the separator in chunks
    )
}

After Saving

CapyDB processes your image data asynchronously. Once processed, it automatically adds a chunks field to each EmbImage for easy access to the internal representations.

{
    "field_name": EmbImage(
        url="https://example.com/image.jpg",
        mime_type="image/jpeg",
        chunks=["chunk1", "chunk2", "chunk3"],
        emb_model=EmbModels.TEXT_EMBEDDING_3_LARGE,
        vision_model=VisionModels.GPT_4O,
        max_chunk_size=200,
        chunk_overlap=20,
        is_separator_regex=False,
        separators=["\n\n", "\n"],
        keep_separator=False
    )
}

Parameter Reference

How It Works: Asynchronous Processing

Whenever you insert a document containing EmbImage into CapyDB, the following steps occur asynchronously:

1. Image Retrieval

   The image is retrieved from the provided URL.

2. Chunking (if applicable)

   Depending on your configuration, the image may be internally segmented into chunks for finer-grained processing.

3. Embedding

   The image (or its chunks) is transformed into vector representations using the specified embedding and/or vision model. This step extracts the semantic and visual features.

4. Indexing

   The resulting embeddings are indexed for efficient, semantic searches. These steps happen in the background, so while write operations are fast, query availability may have a slight delay.

Querying

Once the Embedding and Indexing steps are complete, your EmbImage fields become searchable. To perform semantic queries on image data, use the standard query operations. Please refer to the Query page for details.

How can we improve this documentation?

Got a question? Email us