Connection reuse

The DashScope SDK supports connection reuse in both Java and Python:

Java SDK: A built-in connection pool is enabled by default. Configure parameters like maximum connections and timeout durations.
Python SDK: Enable connection reuse by passing a custom HTTP client. It supports both synchronous and asynchronous calls.

Before you begin

Before running the code examples:

Java SDK

The Java SDK includes a built-in connection pool enabled by default. Adjust maximum connections and timeout settings to optimize for your workload.

Parameters

Parameter	Description	Default value	Unit	Notes
`connectTimeout`	Connection establishment timeout.	120	seconds	In low-latency scenarios, set a shorter timeout to reduce wait time.
`readTimeout`	Data read timeout.	300	seconds
`writeTimeout`	Data write timeout.	60	seconds
`connectionIdleTimeout`	Idle connection timeout in the pool.	300	seconds	In high-concurrency scenarios, extend the idle timeout to avoid frequent connection creation.
`connectionPoolSize`	Maximum connections in the pool.	32	connections	In high-concurrency scenarios, too few connections cause blocking, timeouts, and frequent reconnections. Too many connections overload the server. Balance the count based on your workload.
`maximumAsyncRequests`	Maximum concurrent requests across all hosts (global limit). Must be ≤ `connectionPoolSize`; otherwise, requests may block.	32	requests
`maximumAsyncRequestsPerHost`	Maximum concurrent requests per host. Must be ≤ `maximumAsyncRequests`.	32	requests

Code example

The following example configures connection pool parameters (timeouts, maximum connections) and calls a model service. Adjust the parameters to optimize concurrency and resource usage.

// Recommended DashScope SDK version >= 2.12.0
import java.time.Duration;
import java.util.Arrays;

import com.alibaba.dashscope.aigc.generation.Generation;
import com.alibaba.dashscope.aigc.generation.GenerationParam;
import com.alibaba.dashscope.aigc.generation.GenerationResult;
import com.alibaba.dashscope.common.Message;
import com.alibaba.dashscope.common.Role;
import com.alibaba.dashscope.exception.ApiException;
import com.alibaba.dashscope.exception.InputRequiredException;
import com.alibaba.dashscope.exception.NoApiKeyException;
import com.alibaba.dashscope.protocol.ConnectionConfigurations;
import com.alibaba.dashscope.protocol.Protocol;
import com.alibaba.dashscope.utils.Constants;

public class Main {
  public static GenerationResult callWithMessage() throws ApiException, NoApiKeyException, InputRequiredException {
    // Singapore (ap-southeast-1). For another region, replace ap-southeast-1 in the URL.
    // API keys are region-specific. See https://docs.qwencloud.com/developer-guides/getting-started/regions
    Generation gen = new Generation(Protocol.HTTP.getValue(), "https://ap-southeast-1.maas.qwencloudapi.com/api/v1");
    Message systemMsg = Message.builder()
        .role(Role.SYSTEM.getValue())
        .content("You are a helpful assistant.")
        .build();
    Message userMsg = Message.builder()
        .role(Role.USER.getValue())
        .content("Who are you?")
        .build();
    GenerationParam param = GenerationParam.builder()
        // If you have not configured the environment variable, replace with your API key: .apiKey("sk-xxx")
        .apiKey(System.getenv("DASHSCOPE_API_KEY"))
        // This example uses qwen-plus. Change the model name as needed.
        .model("qwen-plus")
        .messages(Arrays.asList(systemMsg, userMsg))
        .resultFormat(GenerationParam.ResultFormat.MESSAGE)
        .build();

    return gen.call(param);
  }
  public static void main(String[] args) {
    // Connection pool configuration
    Constants.connectionConfigurations = ConnectionConfigurations.builder()
        .connectTimeout(Duration.ofSeconds(10))  // Timeout for establishing a connection, default 120s
        .readTimeout(Duration.ofSeconds(300)) // Timeout for reading data, default 300s
        .writeTimeout(Duration.ofSeconds(60)) // Timeout for writing data, default 60s
        .connectionIdleTimeout(Duration.ofSeconds(300)) // Timeout for idle connections in the connection pool, default 300s
        .connectionPoolSize(256) // Maximum connections in the connection pool, default 32
        .maximumAsyncRequests(256)  // Maximum concurrent requests, default 32
        .maximumAsyncRequestsPerHost(256) // Maximum concurrent requests per host, default 32
        .build();

    try {
      GenerationResult result = callWithMessage();
      System.out.println(result.getOutput().getChoices().get(0).getMessage().getContent());
    } catch (ApiException | NoApiKeyException | InputRequiredException e) {
      // Use a logging framework to record exception information
      System.err.println("Failed to call the model service: " + e.getMessage());
    }
    System.exit(0);
  }
}

Python SDK

The Python SDK supports connection reuse by passing a custom HTTP client (httpx.Client / httpx.AsyncClient). It supports two calling methods: asynchronous HTTP (coroutine-based) and synchronous HTTP.

Asynchronous HTTP

For asynchronous scenarios, use httpx.AsyncClient with httpx.Limits to enable connection reuse. httpx.Limits supports configuring parameters like connection limits:

Parameter	Description	Default value	Notes
`max_connections`	Total connection limit.	None (unlimited)	In high-concurrency scenarios, increase this value to improve concurrency.
`max_keepalive_connections`	Maximum number of keep-alive connections.	None (unlimited)	Controls the number of active connections kept in the pool, preventing excessive load on a single server.
`keepalive_expiry`	Keep-alive timeout for connections.	5.0 (seconds)	Idle keep-alive connections are closed automatically after this duration.

Code example

import asyncio
import httpx
import os
import dashscope

async def main():
  # Configure connection parameters
  limits = httpx.Limits(
    max_connections=100,           # Total connection limit
    max_keepalive_connections=30,  # Maximum number of keep-alive connections
  )

  # Create a custom HTTP client and pass it to the Client constructor
  async with httpx.AsyncClient(limits=limits) as http_client:
    # Singapore (ap-southeast-1). For another region, replace ap-southeast-1 in the URL.
    # API keys are region-specific. See https://docs.qwencloud.com/developer-guides/getting-started/regions
    aclient = dashscope.AsyncClient(
      api_key=os.getenv("DASHSCOPE_API_KEY"),
      base_url="https://ap-southeast-1.maas.qwencloudapi.com/api/v1",
      http_client=http_client,  # Pass the custom HTTP client
    )
    response = await aclient.chat.completions.create(
      model="qwen-plus",
      messages=[{"role": "user", "content": "Hello, please introduce yourself"}],
    )
    print(response)

asyncio.run(main())

Synchronous HTTP

For synchronous scenarios, use httpx.Client to enable connection reuse. Multiple requests within the same HTTP client reuse the underlying TCP connection, avoiding the overhead of repeatedly establishing connections.

Code example: single call

import httpx
import os
import dashscope

# Use the with statement to ensure the HTTP client is closed correctly
with httpx.Client() as http_client:
  # Singapore (ap-southeast-1). For another region, replace ap-southeast-1 in the URL.
  # API keys are region-specific. See https://docs.qwencloud.com/developer-guides/getting-started/regions
  client = dashscope.Client(
    api_key=os.getenv("DASHSCOPE_API_KEY"),
    base_url="https://ap-southeast-1.maas.qwencloudapi.com/api/v1",
    http_client=http_client,  # Pass the custom HTTP client
  )
  response = client.chat.completions.create(
    model="qwen-plus",
    messages=[{"role": "user", "content": "Hello"}],
  )
  print(response)

Code example: multiple calls with a shared HTTP client

To reuse an HTTP client across multiple calls:

import httpx
import os
import dashscope

# Create an HTTP client and configure connection reuse parameters
http_client = httpx.Client(
  limits=httpx.Limits(
    max_connections=100,          # Total connection limit
    max_keepalive_connections=30, # Maximum number of keep-alive connections
  ),
)

# Singapore (ap-southeast-1). For another region, replace ap-southeast-1 in the URL.
# API keys are region-specific. See https://docs.qwencloud.com/developer-guides/getting-started/regions
client = dashscope.Client(
  api_key=os.getenv("DASHSCOPE_API_KEY"),
  base_url="https://ap-southeast-1.maas.qwencloudapi.com/api/v1",
  http_client=http_client,  # Pass the custom HTTP client
)

try:
  # Reuse the same HTTP client across multiple calls
  response1 = client.chat.completions.create(
    model="qwen-plus",
    messages=[{"role": "user", "content": "Hello"}],
  )
  print(response1)

  response2 = client.chat.completions.create(
    model="qwen-plus",
    messages=[{"role": "user", "content": "Introduce yourself"}],
  )
  print(response2)
finally:
  # Ensure the HTTP client is closed correctly
  http_client.close()

Best practices

Java SDK: Set connectionPoolSize and maximumAsyncRequests based on your workload. Balance the connection count: too few cause blocking, too many overload the server.
Python SDK: Use the with statement to automatically manage the HTTP client lifecycle and ensure correct resource cleanup.
Choose the right method: Use async calls for async applications (asyncio, FastAPI). Use sync calls for traditional sync applications.

Error codes

If a model call fails (the Python SDK throws a DashScopeError exception, and HTTP calls return an error message), see Error messages to resolve the issue.