> ## Documentation Index
> Fetch the complete documentation index at: https://docs.evermind.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Multimodal Memory

> Extract memories from images, documents, audio, and video alongside text

EverOS goes beyond text. You can attach images, PDFs, audio recordings, and other file types to your messages, and EverOS will extract relevant information from them to enrich the memory.

## Supported Content Types

<CardGroup cols={3}>
  <Card title="Images" icon="image">
    JPG, PNG, GIF, WebP

    Max size: **10 MB**
  </Card>

  <Card title="Documents" icon="file-lines">
    PDF, DOC, HTML, TXT

    Max size: **100 MB**
  </Card>

  <Card title="Audio & Video" icon="film">
    MP3, WAV, MP4, WebM

    Max size: **500 MB**
  </Card>
</CardGroup>

## How It Works

When you include multimodal content in a message, EverOS **preprocesses** each file into text: OCR for images, transcription for audio, parsing for PDFs and Office documents. It then extracts and stores relevant information as part of the user's memory, just like text messages.

The key difference from text-only messages is that the `content` field becomes an **array of content items** instead of a plain string.

### Content Item Structure

Each item in the content array has a `type` field that determines how EverOS processes it:

| Type | Description | Required Fields |
| - | - | - |
| `text` | Plain text content | `text` |
| `image` | Image file | `uri` |
| `audio` | Audio recording | `uri` |
| `doc` | Document (TXT, DOC) | `uri` |
| `pdf` | PDF document | `uri` |
| `html` | HTML content | `uri` |
| `email` | Email message | `uri` |

<Note>
  The `uri` field contains an object key returned by the upload flow (see below), not a public URL.
</Note>

## Upload with the Python SDK

<Tip>
  The Python SDK handles the upload for you: `client.upload(path)` presigns, uploads
  the bytes directly to S3, and returns the `object_key` to reference in a message.
</Tip>

```python theme={null}
from everos_cloud import EverOS

client = EverOS(api_key="your_api_key")

# 1) Upload the file — returns an object_key (file_type inferred from the extension)
image_key = client.upload("whiteboard.jpg")

# 2) Reference the key in a message's content array
client.add(
    session_id="session_gs_003",
    messages=[
        {
            "sender_id": "user_demo_001",
            "role": "user",
            "content": [
                {"type": "text", "text": "Here is the whiteboard from today's meeting."},
                {"type": "image", "uri": image_key, "name": "whiteboard.jpg", "ext": "jpg"},
            ],
        },
    ],
)
```

## Upload Flow (raw API)

If you are not using the SDK, uploading multimodal data is a **3-step process**:

<Steps>
  <Step title="Get a pre-signed upload URL">
    Call `POST /api/v2/object/sign` to get a pre-signed S3 upload URL and an `objectKey` for your file.

    ```bash theme={null}
    curl -X POST https://api.evermind.ai/api/v2/object/sign \
      -H "Authorization: Bearer <api_key>" \
      -H "Content-Type: application/json" \
      -d '{
        "objectList": [
          {
            "fileId": "my_image_001",
            "fileName": "meeting-whiteboard.png",
            "fileType": "image"
          }
        ]
      }'
    ```

    Response (key fields):

    ```json theme={null}
    {
      "result": {
        "data": {
          "objectList": [
            {
              "fileId": "my_image_001",
              "objectKey": "982a8db8c67be277/b3303282-2c49-4564-bb1a-6e415b781d0c",
              "objectSignedInfo": {
                "url": "https://s3.amazonaws.com/bucket-name",
                "fields": {
                  "key": "982a8db8c67be277/b3303282-...",
                  "policy": "eyJjb25k...",
                  "x-amz-algorithm": "AWS4-HMAC-SHA256",
                  "x-amz-credential": "...",
                  "x-amz-signature": "..."
                }
              }
            }
          ]
        }
      }
    }
    ```

    Save the `objectKey`. You will need it in Step 3.
  </Step>

  <Step title="Upload the file to S3">
    Use the returned `objectSignedInfo.url` and `fields` to upload your file via an S3 POST request.

    ```bash theme={null}
    curl -X POST "https://s3.amazonaws.com/bucket-name" \
      -F "key=982a8db8c67be277/b3303282-2c49-4564-bb1a-6e415b781d0c" \
      -F "policy=eyJjb25k..." \
      -F "x-amz-algorithm=AWS4-HMAC-SHA256" \
      -F "x-amz-credential=..." \
      -F "x-amz-signature=..." \
      -F "file=@meeting-whiteboard.png"
    ```

    A successful upload returns HTTP **204 No Content**.
  </Step>

  <Step title="Add memory with the objectKey">
    Call `POST /api/v2/memory/add` with `content` as an array. Set the `uri` field to the `objectKey` from Step 1.

    ```bash theme={null}
    curl -X POST https://api.evermind.ai/api/v2/memory/add \
      -H "Authorization: Bearer <api_key>" \
      -H "Content-Type: application/json" \
      -d '{
        "session_id": "session-1",
        "messages": [
          {
            "sender_id": "user_alice",
            "role": "user",
            "timestamp": 1711900000000,
            "content": [
              {
                "type": "text",
                "text": "Here is the whiteboard from today meeting"
              },
              {
                "type": "image",
                "uri": "982a8db8c67be277/b3303282-2c49-4564-bb1a-6e415b781d0c",
                "name": "meeting-whiteboard.png",
                "ext": "png"
              }
            ]
          }
        ]
      }'
    ```

    EverOS will process the image, extract relevant information, and store it as part of the user's memory.
  </Step>
</Steps>

### Batch Upload

You can sign multiple files in a single request by passing multiple items in the `objectList` array (up to 50):

```bash theme={null}
curl -X POST https://api.evermind.ai/api/v2/object/sign \
  -H "Authorization: Bearer <api_key>" \
  -H "Content-Type: application/json" \
  -d '{
    "objectList": [
      {"fileId": "img_001", "fileName": "photo.jpg", "fileType": "image"},
      {"fileId": "doc_001", "fileName": "report.pdf", "fileType": "file"},
      {"fileId": "vid_001", "fileName": "recording.mp4", "fileType": "video"}
    ]
  }'
```

Then upload each file to its respective pre-signed URL and include all `objectKey` values in your message content array.

## Mixing Text and Files

A single message can contain both text and multiple file attachments. EverOS processes all content items together to build a complete memory (`timestamp` is a unix millisecond value):

```json theme={null}
{
  "sender_id": "user_alice",
  "role": "user",
  "timestamp": 1711900000000,
  "content": [
    {"type": "text", "text": "Summary of the Q3 planning session"},
    {"type": "image", "uri": "<objectKey_1>", "name": "whiteboard.png", "ext": "png"},
    {"type": "pdf", "uri": "<objectKey_2>", "name": "q3-plan.pdf", "ext": "pdf"},
    {"type": "audio", "uri": "<objectKey_3>", "name": "recording.mp3", "ext": "mp3"}
  ]
}
```

<Note>
  **Text items have no limit** per message. Non-text items (image, audio, doc, etc.) are limited to **10 per message**. The total request body must be under **300 KB** (file content is stored on S3, not in the request body).
</Note>

## Content Item Reference

| Field | Type | Description |
| - | - | - |
| `type` | string | **Required.** One of: `text`, `image`, `audio`, `doc`, `pdf`, `html`, `email` |
| `text` | string \| null | Content body. For `type: text`, this is the actual text |
| `uri` | string \| null | Object key from the upload flow (returned by `POST /api/v2/object/sign` or `client.upload()`) |
| `name` | string \| null | File name |
| `ext` | string \| null | File extension (e.g., `png`, `mp3`, `pdf`) |
| `source` | string \| null | Content source (e.g., `google_doc`, `notion`, `confluence`, `zoom`) |
| `source_info` | object \| null | Source-related traceability metadata |
| `extras` | object \| null | Type-specific extra fields |

## Next Steps

<CardGroup cols={2}>
  <Card title="Cloud Quickstart" icon="rocket" href="/cloud/quickstart">
    Add and retrieve memories with the EverOS SDK
  </Card>

  <Card title="API Reference" icon="code" href="/api-reference/introduction">
    Full request/response schemas for the Memory and Storage endpoints
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.