Connection timeout when using ChatMessageImageContentItem and AzureSearchChatExtensionConfiguration

Question

Connection timeout when using ChatMessageImageContentItem and AzureSearchChatExtensionConfiguration

Rahul Navalakha 25

I have a chat client application utilizing Azure Open AI java SDK. Our use case is to query the OpenAI LLM by sending image as context along with user message and also use RAG data in the query.

The application invokes getChatCompletions endpoint using text user message and Azure AI Search options to the Chat (RAG). This works fine until I start using an image along with text user messages. When adding ChatMessageImageContentItem into the mix, the connection times out. We are including the image as a data URL with base64 encoding. As a test we tried to query the LLM using image and text user content without Azure AI Search options and it works fine. As soon as we mix image+text in the chat completiong with RAG search options, we run into the time out.

Looking for some guidance on what could be wrong with the strategy here.

Anshika Varshney 7,970 Reputation points Microsoft External Staff Moderator

2026-03-05T19:31:49.74+00:00

Hi Rahul Navalakha,

From what is described, this usually comes down to the request taking longer to complete when images are included, rather than a hard service failure. When ChatMessageImageContent is used, the request payload becomes much larger and the model needs extra time to process both the text and the image. If the client or HTTP library has a shorter timeout configured, the connection can get closed before the service finishes generating the response, which shows up as a connection timeout.

Another thing to watch is how the image is being sent. Large images or uncompressed base64 payloads can noticeably increase request size and processing time. In practice, resizing the image to a reasonable resolution and avoiding unnecessary metadata helps reduce latency and makes the call more reliable.

This behavior can also feel intermittent because it depends on current load and how complex the prompt plus image combination is. Text only requests may succeed quickly, while image-based requests hit the timeout more easily under the same settings.

What has helped others is increasing the client-side timeout to allow enough time for multimodal processing, keeping image sizes modest, and testing the same request with and without images to confirm that the delay is image related rather than a general connectivity issue. Once those are aligned, the timeouts usually stop occurring and the same setup works consistently.

Do let me know if you have any further queries.

Thankyou!
Rahul Navalakha 25 Reputation points

2026-03-05T19:42:56.05+00:00

Hi,

Thank You for the details. I have tried setting a longer timeout on the OpenAI HTTP client and also using a small image (2kb) for testing but did not see any change in behavior. The timeouts are consistent even with basic text and small image in the payload.
SRILAKSHMI C 14,815 Reputation points Microsoft External Staff Moderator

2026-03-06T06:23:02.0333333+00:00
Hi Rahul Navalakha,

Thank you for testing with a longer client timeout and a smaller image

Since the timeout still occurs even with a very small image and extended client timeout, the behavior is likely related to how the request is being processed when multimodal content (image + text) is combined with Azure AI Search chat extensions (RAG).

A couple of areas would be validating:

1. Current support for image inputs with Azure AI Search chat extensions

At the moment, chat extensions that integrate Azure AI Search are primarily designed for text-based prompts. When an image content item (ChatMessageImageContentItem) is included in the same request together with AzureSearchChatExtensionConfiguration, the service may not process the request as expected, which can manifest as long processing times or connection timeouts.

Since your tests show that:

Image + text without Azure AI Search works

Text + Azure AI Search works

Image + text + Azure AI Search times out

this pattern strongly suggests the interaction between multimodal input and the search extension is the triggering factor rather than the image size or client timeout.

2. Request processing flow with extensions

When Azure AI Search extensions are used, the service performs several additional steps before sending the final prompt to the model:

Query rewriting

Azure AI Search retrieval

Document grounding

Final prompt construction

When an image content item is included, the service needs to handle both multimodal processing and retrieval orchestration, which may cause the request to stall depending on the SDK implementation or request format.

Troubleshooting steps

To further isolate the issue, it would help to test the following:

• Try sending the same request through REST API or a simple HTTP client instead of the Java SDK to confirm whether the behavior is SDK-related.

• Verify the model being used supports image inputs (for example, GPT-4o or other multimodal-capable models).

• Enable HTTP logging in the Java SDK to confirm whether the request is reaching the service or timing out before a response is returned.

• As a workaround, consider a two-step approach:

Send the image + user prompt to the model to extract relevant text or description.

Use that text as the query input for the Azure AI Search RAG call.

This approach avoids mixing multimodal content with the chat extension in the same request.

Thank you!