> ## Documentation Index
> Fetch the complete documentation index at: https://cortex-ad5578da.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Upload App Sources

> Specifically built to handle knowledge from workplace apps. Parsing, chunking, and indexing depends on the kind of app (eg - gmail, slack, documents, etc). 

export const TableOfContents = ({title = 'On this page', items, minHeadingLevel = 2, maxHeadingLevel = 3, className = '', activeClassName = 'text-primary dark:text-primary-light border-primary dark:border-primary-light hover:border-primary dark:hover:border-primary-light', inactiveClassName = 'hover:text-gray-900 dark:text-gray-400 dark:hover:text-gray-300'}) => {
  const [toc, setToc] = useState(items ?? []);
  const [activeId, setActiveId] = useState('');
  useEffect(() => {
    if (items && items.length) return;
    if (typeof document === 'undefined') return;
    const selectors = [];
    for (let lvl = minHeadingLevel; lvl <= maxHeadingLevel; lvl++) {
      selectors.push(`h${lvl}`);
    }
    const nodes = Array.from(document.querySelectorAll(selectors.join(','))).filter(el => el.id);
    const built = [];
    let currentTop = null;
    nodes.forEach(el => {
      const level = Number(el.tagName.slice(1));
      const node = {
        id: el.id,
        label: el.textContent.trim(),
        href: `#${el.id}`,
        children: []
      };
      if (level === minHeadingLevel) {
        built.push(node);
        currentTop = node;
      } else if (level > minHeadingLevel && currentTop) {
        currentTop.children.push(node);
      } else {
        built.push(node);
      }
    });
    setToc(built);
  }, [items, minHeadingLevel, maxHeadingLevel]);
  useEffect(() => {
    if (typeof document === 'undefined') return;
    const applyHash = () => {
      const h = window.location.hash.replace('#', '');
      if (h) setActiveId(h);
    };
    applyHash();
    const observer = new IntersectionObserver(entries => {
      entries.forEach(entry => {
        if (entry.isIntersecting) {
          setActiveId(entry.target.id);
        }
      });
    }, {
      rootMargin: '0px 0px -70% 0px',
      threshold: 0.1
    });
    const ids = toc.flatMap(i => [i, ...i.children ?? []]).map(i => i.id);
    ids.forEach(id => {
      const el = document.getElementById(id);
      if (el) observer.observe(el);
    });
    window.addEventListener('hashchange', applyHash);
    return () => {
      window.removeEventListener('hashchange', applyHash);
      observer.disconnect();
    };
  }, [toc]);
  const Item = ({node, depth = 0}) => {
    const isActive = activeId === node.id;
    return <li className="toc-item relative ml-6" data-depth={depth}>
        <a href={node.href} className={`py-1 block font-medium ${isActive ? activeClassName : inactiveClassName}`} style={depth > 0 ? {
      marginLeft: `${depth}rem`
    } : {}} onClick={() => setActiveId(node.id)}>
          {node.label}
        </a>
        {node.children && node.children.length > 0 && <>
            {node.children.map(child => <Item node={child} depth={depth + 1} key={child.href} />)}
          </>}
      </li>;
  };
  const data = toc && toc.length ? toc : items || [];
  return <div className={`text-gray-600 text-sm leading-6 w-[18rem] pb-4 -mt-10 pt-10 hidden xl:block ${className}`} id="table-of-contents-custom">
      <ul id="table-of-contents-custom-content" className="toc">
        <li className="toc-item relative">
          <div className="text-gray-700 dark:text-gray-300 font-medium flex items-center space-x-2 py-1">
            <svg width="16" height="16" viewBox="0 0 16 16" fill="none" stroke="currentColor" strokeWidth="2" xmlns="http://www.w3.org/2000/svg" className="h-3 w-3">
              <path d="M2.44434 12.6665H13.5554" strokeLinecap="round" strokeLinejoin="round"></path>
              <path d="M2.44434 3.3335H13.5554" strokeLinecap="round" strokeLinejoin="round"></path>
              <path d="M2.44434 8H7.33323" strokeLinecap="round" strokeLinejoin="round"></path>
            </svg>
            <span>{title}</span>
          </div>
        </li>
        {data.map(node => <Item node={node} key={node.href} />)}
      </ul>
    </div>;
};

<Panel>
  <TableOfContents />
</Panel>

<Tip> Hit the `Try it` button to try this API now in our playground. It's the best way to check the full request and response in one place, customize your parameters, and generate ready-to-use code snippets.</Tip>

### Sample

```bash expandable theme={null}
curl -X 'POST' \
  'https://api.usecortex.ai/ingestion/upload-app' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "app_sources": {
    "id": "string",
    "tenant_id": "string",
    "sub_tenant_id": "string",
    "title": "",
    "type": "",
    "description": "",
    "note": "",
    "url": "",
    "timestamp": "",
    "content": {
      "text": "",
      "html_base64": "",
      "csv_base64": "",
      "markdown": "",
      "files": [
        {
          "additionalProp1": {}
        }
      ],
      "layout": [
        {
          "additionalProp1": {}
        }
      ]
    },
    "tenant_metadata": {
      "additionalProp1": {}
    },
    "document_metadata": {
      "additionalProp1": {}
    },
    "meta": {
      "additionalProp1": {}
    },
    "attachments": [
      {
        "id": "",
        "url": "",
        "title": "",
        "content_type": "",
        "content_url": "",
        "misc": {
          "additionalProp1": {}
        },
        "content": {
          "text": "",
          "html_base64": "",
          "csv_base64": "",
          "markdown": "",
          "files": [
            {
              "additionalProp1": {}
            }
          ],
          "layout": [
            {
              "additionalProp1": {}
            }
          ]
        }
      }
    ]
  },
  "tenant_id": "string",
  "sub_tenant_id": "",
  "upsert": true
}'
```

### SDK Examples

<Tabs>
  <Tab title="TypeScript">
    ```ts expandable theme={null}
    const result = await client.upload.uploadAppSources({
      tenant_id: "tenant_1234",
      sub_tenant_id: "sub_tenant_4567",
      body: [
        {
          id: "user-guide-1",
          title: "Feature X Guide",
          content: { text: "How to use feature X" },
          document_metadata: { source: "database" }
        },
        {
          id: "user-guide-2", 
          title: "Feature Y Guide",
          content: { text: "How to use feature Y" },
          document_metadata: { source: "api" }
        }
      ]
    });
    ```
  </Tab>

  <Tab title="Python (Sync)">
    ```python expandable theme={null}
    # Async usage is similar, just use async_client and await
    result = client.upload.upload_app_sources(
        tenant_id="tenant_1234",
        request=[
            {
                "id": "user-guide-1",
                "title": "Feature X Guide", 
                "content": {"text": "How to use feature X"},
                "document_metadata": {"source": "database"}
            },
            {
                "id": "user-guide-2",
                "title": "Feature Y Guide",
                "content": {"text": "How to use feature Y"}, 
                "document_metadata": {"source": "api"}
            }
        ],
        sub_tenant_id="sub_tenant_4567"
    )
    ```
  </Tab>
</Tabs>

Works similar to the upload endpoint but is specifically designed to upload multiple app sources (e.g., Gmail, Slack, Notion) in a single request for processing and indexing.
Each app upload is handled using specialized pipelines inside Cortex and can include various content types with rich metadata.

## Supported Apps

The following apps are currently supported for app source uploads:

**File Storage & Cloud Services:**

* `drive` - Google Drive
* `dropbox` - Dropbox Business
* `dropboxpersonal` - Dropbox Personal
* `onedrive` - Microsoft OneDrive
* `sharepoint` - Microsoft SharePoint

**CRM & Sales:**

* `intercom` - Intercom
* `salesforce` - Salesforce
* `hubspot` - HubSpot

**Communication & Collaboration:**

* `msteams` - Microsoft Teams
* `gmail` - Gmail
* `slack` - Slack
* `outlook` - Microsoft Outlook

**Project Management:**

* `jira` - Atlassian Jira
* `confluence` - Atlassian Confluence
* `shortcut` - Shortcut
* `linear` - Linear
* `asana` - Asana

**Productivity & Organization:**

* `notion` - Notion
* `googlecalendar` - Google Calendar

## App Source Processing Pipeline

When you upload app sources, each source goes through specialized processing pipelines tailored to the specific app type:

### 1. **Immediate Upload & App Detection**

* All app sources are immediately accepted and stored securely
* App type is automatically detected (Gmail, Slack, Notion, etc.)
* Each source is routed to its specialized processing pipeline
* You receive a confirmation response with individual `file_id`s for tracking

### 2. **App-Specific Processing Phase**

Each app source is processed using specialized pipelines:

* **Gmail**: Email parsing, thread reconstruction, attachment handling
* **Slack**: Message threading, channel context, user mentions
* **Notion**: Page hierarchy, block structure, database relationships
* **Documents**: Format-specific parsing (PDF, DOCX, etc.)
* **Custom Apps**: Configurable parsing based on app metadata

### 3. **Content Extraction & Normalization**

* **Multi-format Support**: Text, HTML, CSV, Markdown, and file attachments
* **Context Preservation**: Maintaining app-specific context and relationships
* **Metadata Enrichment**: Extracting app-specific metadata and timestamps
* **Content Cleaning**: Normalizing content while preserving structure

### 4. **Intelligent Chunking**

* App-aware chunking strategies preserve context and relationships
* Thread-based chunking for Gmail and Slack conversations
* Hierarchical chunking for Notion pages and databases
* Metadata is preserved and associated with each chunk

### 5. **Embedding Generation**

* Each chunk is converted into high-dimensional vector embeddings
* Embeddings capture semantic meaning and app-specific context
* Vectors are optimized for similarity search and retrieval
* Cross-app relationship embeddings for related content

### 6. **Indexing & Database Updates**

* Embeddings are stored in our vector database for fast similarity search
* Full-text search indexes are created for keyword-based queries
* App-specific metadata is indexed for filtering and faceted search
* Cross-references are established between related app sources

### 7. **Quality Assurance**

* App-specific quality checks ensure processing accuracy
* Content validation verifies extracted text completeness
* Relationship validation ensures proper context preservation
* Embedding quality is assessed for optimal retrieval performance

<Note>
  **Processing Time**: App sources are processed in parallel using specialized pipelines. Most sources are fully processed and searchable within 2-5 minutes. Complex sources with multiple attachments may take up to 10 minutes. You can check processing status using the individual document IDs returned in the response.
</Note>

> **Recommended**: For optimal performance, limit each batch to a maximum of **20 app sources** per request. Send multiple batch requests with an interval of **1 second** between each request.

> **File ID Management**: When you provide a `file_id` as a key in the `document_metadata` object, that specific ID will be used to identify your content. If no `file_id` is provided in the `document_metadata`, the system will automatically generate a unique identifier for you. This allows you to maintain consistent references to your content across your application while ensuring every piece of content has a unique identifier.

### **Duplicate File ID Behavior**

When you upload app sources with `file_id`s that already exist in your tenant:

* **Overwrite Behavior**: Each existing app source with a matching `file_id` will be **completely replaced** with the new source
* **Processing**: Each new app source will go through its specialized processing pipeline independently
* **Search Results**: Previous search results and embeddings from old app sources will be replaced with the new sources' content
* **Idempotency**: Uploading the same app sources with the same `file_id`s multiple times is safe and will result in the same final state

<Warning>
  **Important**: When overwriting existing app sources, all previous chunks, embeddings, and search indexes associated with those `file_id`s will be permanently removed and replaced. This action cannot be undone.
</Warning>

**Example Success Response for Duplicate File IDs in App Upload:**

```json theme={null}
{
  "message": "App sources uploaded successfully. Sources with existing file_ids have been overwritten.",
  "document_ids": ["gmail_123456", "slack_789012", "notion_345678", "drive_901234"],
  "overwritten_file_ids": ["gmail_123456", "slack_789012"],
  "status": "success"
}
```

## Attachments Field Structure

The `attachments` field allows you to include additional files, documents, or content alongside your main app source. Each attachment supports multiple content formats and can contain nested structures for complex documents.

### Attachment Object Structure

```json theme={null}
{
  "attachments": [
    {
      "id": "unique_attachment_id",
      "url": "https://example.com/document.pdf",
      "title": "Document Title",
      "content_type": "application/pdf",
      "content_url": "https://api.example.com/content/123",
      "misc": {
        "custom_field": "value"
      },
      "content": {
        "text": "Plain text content",
        "html_base64": "base64_encoded_html",
        "csv_base64": "base64_encoded_csv",
        "markdown": "# Markdown content",
        "files": [{"name": "file.pdf", "data": "base64_data"}],
        "layout": [{"type": "section", "content": "..."}]
      }
    }
  ]
}
```

### When to Use Each Field

**Core Identification Fields:**

* `id` (optional): Unique identifier for the attachment. If not provided, system generates one automatically.
* `title` (optional): Human-readable name for the attachment.
* `url` (optional): External URL where the attachment can be accessed.
* `content_type` (optional): MIME type of the attachment (e.g., "application/pdf", "text/plain").
* `content_url` (optional): API endpoint URL for retrieving attachment content.

**Content Storage Fields:**
Use these fields to store different types of content directly in the attachment:

* `content.text`: Use for plain text content. Best for simple text documents, notes, or extracted text from other formats.
* `content.html_base64`: Use for HTML content encoded in base64. Ideal for web pages, rich text documents, or formatted content that needs to preserve HTML structure.
* `content.csv_base64`: Use for CSV data encoded in base64. Perfect for tabular data, spreadsheets, or structured data exports.
* `content.markdown`: Use for Markdown-formatted content. Great for documentation, README files, or any content that uses Markdown syntax.
* `content.files`: Use for binary file attachments as an array of file objects. Each file object should contain at least a `name` and `data` field (base64 encoded).
* `content.layout`: Use for structured document layouts as an array of layout objects. Useful for complex documents with sections, headers, or custom formatting.

**Metadata Field:**

* `misc` (optional): Dictionary for storing custom metadata, additional properties, or app-specific information about the attachment.

### Content Format Guidelines

**For Text Content:**

```json theme={null}
{
  "content": {
    "text": "This is plain text content that will be processed and indexed."
  }
}
```

**For HTML Content:**

```json theme={null}
{
  "content": {
    "html_base64": "PGgxPkhlbGxvIFdvcmxkPC9oMT4="
  }
}
```

**For CSV Data:**

```json theme={null}
{
  "content": {
    "csv_base64": "TmFtZSxBbW91bnQKSm9obiwxMDAKSmFuZSwyMDA="
  }
}
```

**For Markdown:**

```json theme={null}
{
  "content": {
    "markdown": "# Document Title\n\nThis is **markdown** content with formatting."
  }
}
```

**For File Attachments:**

```json theme={null}
{
  "content": {
    "files": [
      {
        "name": "document.pdf",
        "data": "JVBERi0xLjQKJcfsj6IKNSAwIG9iago8PAovVHlwZSAvUGFnZQovUGFyZW50IDMgMCBSCi9NZWRpYUJveCBbMCAwIDU5NSA4NDJdCi9SZXNvdXJjZXMgPDwKL0ZvbnQgPDwKL0YxIDIgMCBSCj4+Cj4+Ci9Db250ZW50cyA0IDAgUgo+PgplbmRvYmoK..."
      }
    ]
  }
}
```

### Best Practices

1. **Choose the Right Format**: Use the content field that best matches your data type for optimal processing.
2. **Base64 Encoding**: Always encode binary data (HTML, CSV, files) in base64 format.
3. **File Size Limits**: Keep individual attachments under 10MB for optimal processing performance.
4. **Metadata Usage**: Use the `misc` field to store app-specific metadata that might be useful for filtering or organization.
5. **Content Type Specification**: Always specify `content_type` when possible to help with proper content processing.

## Error Responses

All endpoints return consistent error responses following the standard format. For detailed error information, see our [Error Responses](/api-reference/error-responses) documentation.
