AI Agent: attachment-bearing document conversation memory fails to reload with multiple physical tenants

[Szik]

Describe the Bug

The AI Agent Task connector fails to reload document-backed conversation memory containing a document attachment when one Connectors runtime serves multiple physical tenants.

The initial agent invocation reads the attachment, requests a tool, persists the conversation and completes. The BPMN tool completes successfully. When the agent is invoked again to consume the tool result, it fails with a JOB_NO_RETRIES incident:

Failed to load conversation from documentReference | Error variables: {error={type=java.lang.RuntimeException, message=Failed to load conversation from documentReference}}

Reproduced in physical tenants default, tenanta, and tenantb. The saved conversation document is readable through the owning tenant's document API and contains the original nested attachment reference. The equivalent text-only conversation passes, including four user turns, real tool execution, agent history, memory reload and document retention/deletion.

This was discovered while validating #8996, but is separate from #8977's startup issue, whose multi-client startup and outbound-execution checks passed on the rebuilt image. The identified missing-context implementation already exists at PR merge base 28d211c645049fff3fb430c9e621c8129dbf5545; this report is not claiming a new regression introduced by #8996. Attribution is based on source comparison, not a pre-PR runtime A/B test.

Steps to Reproduce

1. Run a Self-Managed orchestration cluster with physical tenants enabled and configure one Connectors runtime with three named clients under camunda.clients: default, tenanta, and tenantb, each routed and authenticated to its own physical tenant. The tested configuration had no designated primary and job streaming enabled.
2. Upload a small valid text/plain document through the target physical tenant's POST /core/physical-tenants/{physicalTenantId}/v2/documents. Verify an owning-tenant GET returns the exact uploaded bytes. Retain the returned Camunda document reference.
3. Deploy an AI Agent Task v2 process with job type io.camunda.agenticai:aiagent:task:2, an ad-hoc tool container, and these input mappings (the values below describe the task's mapped input):

{
  "data": {
    "context": null,
    "userPrompt": {
      "prompt": "Read the attachment and calculate a=5,b=3 using the available tool.",
      "documents": ["<replace with the actual document reference object from step 2>"]
    },
    "tools": {
      "containerElementId": "Agent_Tools",
      "toolCallResults": []
    },
    "memory": {
      "storage": {"type": "camunda-document", "timeToLive": "PT48H"},
      "contextWindowSize": 50
    }
  }
}

4. Use resultVariable=agent, carry agent.context into data.context on re-entry, and pass the completed BPMN tool's results to data.tools.toolCallResults. A suitable existing scaffold is agentic-ai-connectors.bpmn. Replace its dummy AI task with the v2 connector and configure an OpenAI-compatible provider. The tested tool was a FEEL script calculating 3 * (a + b), returning 24.
5. Start the process in logical tenant <default> through the selected physical tenant's API. Have the model request the tool on the first invocation. The agent saves its conversation, the tool completes, and the process loops back to the AI task with the existing context.
6. Observe the incident on that second AI invocation, before a second model request. Repeat in the other physical tenants.
7. Control: repeat with data.userPrompt.documents=[]. The tool-return cycle and subsequent conversation turns complete successfully.

The reproduction used a deterministic OpenAI-compatible SSE model simulator. It returned a valid tool call on the first invocation and a final answer after receiving the tool result. The attachment was present in the captured initial model request; no simulator error occurred. Engine execution, authentication, BPMN tools, agent-history APIs and AWS-backed document storage were real.

Observed evidence

• Owning-tenant attachment upload/download succeeded; reads through the two other physical tenants returned 404.
• The first agent job completed and the actual BPMN tool reached COMPLETED.
• The saved conversation document remained readable and contained schemaVersion: 1, the user attachment, and the assistant tool call. Its relevant structure was:

{
  "messages": [{
    "role": "user",
    "content": [{
      "type": "document",
      "document": {
        "camunda.document.type": "camunda",
        "storeId": "aws",
        "documentId": "<uploaded-document-id>",
        "contentHash": "<returned-content-hash>",
        "metadata": {"contentType": "text/plain"}
      }
    }]
  }]
}

• The next agent job failed with the error quoted above. No cross-tenant data disclosure was observed.
• Fresh reproduction IDs preserved for investigation:

Physical tenant	Process instance	Failed job	Incident
default	2251799813689300	2251799813689356	2251799813689360
tenanta	4503599627374748	4503599627374805	4503599627374809
tenantb	6755399441059780	6755399441059837	6755399441059841

At capture time on 2026-09-21, the PR image and these incidents were left running in QA namespace qa-pr8996-mt-camunda-platform. Documents in this reproduction have 48-hour expiry; the test namespace is ephemeral. The reporter retained the BPMNs, request/response evidence, conversation JSON, model captures and standalone runtime logs.

Expected Behavior

The agent should reload its saved conversation, resolve nested document attachments through the job's physical tenant, incorporate the completed tool result and continue. Attaching a valid document should not break the otherwise successful document-memory flow.

Source investigation

• AgenticAiConnectorsAutoConfiguration supplies the shared @ConnectorsObjectMapper to the conversation store.
• CamundaDocumentConversationStore retains one shared serializer, although document storage is selected per physical tenant.
• CamundaDocumentConversationSerializer calls objectMapper.treeToValue(tree, DocumentContent.class) without setting DocumentFactory.PHYSICAL_TENANT_ID_ATTRIBUTE.
• DocumentDeserializer rejects nested document deserialization without that attribute when multiple tenant factories are configured.

The missing attribute and multi-factory deserializer also exist unchanged at the PR merge base. This source path explains the attachment-specific failure, but the live runtime logs expose only the outer Failed to load conversation from documentReference wrapper. The nested Java exception was not captured, so the source diagnosis should not be mistaken for an observed stack trace.

Environment

• OS: Linux containers on GKE, Self-Managed Helm deployment.
• Library/runtime version: Connectors 8.11.0-SNAPSHOT, exact tested head 4497f021620d0fd90133c20ca0da3237c94facfe.
• Image: registry.camunda.cloud/team-connectors/connectors-bundle:8.11.0-pr8996-run35599078523-a1.
• Published and running image digest: sha256:a9d0d6d96a289ff54e33cb421805f79d8d93962c7906fe1f188e3b1e774fafdc.
• Artifact build, image publication.
• Camunda version: camunda/camunda:8.10-SNAPSHOT, physical tenancy enabled (MT=true); three named clients and physical tenants, logical tenant <default>.
• Conversation storage: Camunda document store backed by AWS; AI Agent Task v2.
• No claim about affected released versions or the earliest affected release. No affects/8.10 or affects/8.11 label was available when filing; older release labels would be misleading.