Skip to main content
Tool calling

Web extractor

Fetch URLs for context

For Qwen4-Max, use Responses web_search + web_extractor, or Anthropic web_search + web_fetch, to use the web extractor.
For tasks involving mathematical calculations or data analytics, add code_interpreter alongside web extractor to improve accuracy.

Getting started

Call web extractor through the Responses API to summarize a web page. The examples below use web_search and web_extractor with qwen4-max in thinking mode.
  • Python
  • Node.js
  • curl
import os
from openai import OpenAI

client = OpenAI(
  # If the environment variable is not configured, replace with: api_key="sk-xxx"
  api_key=os.getenv("DASHSCOPE_API_KEY"),
  # Singapore (ap-southeast-1). For another region, replace ap-southeast-1 in the URL.
  # API keys are region-specific. See https://docs.qwencloud.com/developer-guides/getting-started/regions
  base_url="https://ap-southeast-1.maas.qwencloudapi.com/compatible-mode/v1"
)

response = client.responses.create(
  model="qwen4-max",
  input="Please visit the official QwenCloud documentation, find the code interpreter topic and summarize it",
  tools=[
    {"type": "web_search"},
    {"type": "web_extractor"}
  ],
  extra_body={
    "enable_thinking": True
  }
)

# Uncomment to view intermediate output
# print(response.output)
print("=" * 20 + "Response" + "=" * 20)
print(response.output_text)

# Print tool invocation count
usage = getattr(response, "tool_usage", {})
print("=" * 20 + "Tool Invocation Count" + "=" * 20)
if usage:
  print(f"Web Extractor invocations: {usage.get('web_extractor', {}).get('num_requests')}")

Response structure

The response contains the model's generated text plus metadata about tool usage.
FieldDescription
output_textThe model's final text response, grounded in the extracted web content
output[]Array of intermediate items including web_extractor_call objects (each with goal and output fields showing what URL was fetched and the extracted content)
usage.x_tools.web_extractor.countNumber of web extractor invocations in this request
usage.x_tools.web_search.countNumber of web search invocations in this request

How it works

  1. Include web_extractor (and typically web_search) in the tools array of your API request, along with a prompt that references a URL or topic.
  2. The model determines which pages to fetch, retrieves their content, and appends it to the context as additional input tokens.
  3. The model generates a response grounded in the retrieved content.
Extracted web content increases input tokens and affects billing. See Billing for details.

When to use web extractor

ScenarioTool configurationWhy
Answer questions about a specific URLweb_extractor (optionally with web_search)The model fetches and reads the full page content, not just a search snippet
Research a topic across the webweb_search + web_extractorweb_search finds relevant pages; web_extractor reads them in full
Quick factual lookup (no specific URL)web_search aloneSearch snippets are often sufficient; cheaper and faster
Use web_extractor when you need the model to read the actual content of a page -- not just a search-result summary.

Invocation methods

Web extractor supports two APIs. The Responses API provides the most control over tool behavior -- use it for new integrations.
APITool configurationStreaming requiredNotes
Responses API (recommended)Add web_search and web_extractor to toolsNoExposes intermediate tool execution status
Anthropic-compatibleAdd web_search (and web_fetch for web extractor) to toolsNoTargets Anthropic clients like Claude Code
When using qwen4-max or qwen3.8-max, set enable_thinking to true.
  • Responses API
  • Anthropic-compatible
response = client.responses.create(
  model="qwen4-max",
  input="Summarize the content at https://example.com/article",
  tools=[
    {"type": "web_search"},
    {"type": "web_extractor"}
  ],
  extra_body={
    "enable_thinking": True
  }
)

Stream web extractor events

For general streaming concepts (SSE protocol, how to enable streaming, and token usage), see Streaming output. This section covers only the event types specific to web extractor.
Web extraction can take time. Use streaming to receive reasoning steps, tool calls, and response text as they happen. The Responses API exposes intermediate execution status for each tool call, making it the best choice for streaming. When streaming with the Responses API, the following event types indicate progress through the extraction and generation pipeline:
Event typeDescription
response.reasoning_text.deltaIncremental reasoning text from the model's thinking process
response.output_item.doneA tool call has completed. Check item.type for web_extractor_call to get the extraction result
response.output_text.deltaIncremental response text
response.completedThe response is complete. The usage field contains tool invocation counts
To handle the web_extractor_call event in a stream, check for the event type and read its goal and output fields:
for chunk in stream:
  if chunk.type == 'response.output_item.done':
    if hasattr(chunk, 'item') and chunk.item.type == 'web_extractor_call':
      print(f"Fetched: {chunk.item.goal}")
      print(f"Content: {chunk.item.output}")
  elif chunk.type == 'response.output_text.delta':
    print(chunk.delta, end='', flush=True)
  elif chunk.type == 'response.completed':
    usage = getattr(chunk.response, "tool_usage", {})
    if usage:
      print(f"Web Extractor invocations: {usage.get('web_extractor', {}).get('num_requests')}")

Supported models

deepseek-v4.1-flash-1017 supports the built-in web_extractor tool in the Responses API and web_fetch in the Anthropic-compatible API in all regions: Singapore, China North 2 (Beijing), Germany (Frankfurt), China (Hong Kong), Japan (Tokyo), and US (Virginia). kimi-k3.1-pro (deployed on QwenCloud) supports the built-in web_extractor tool in the Responses API and web_fetch in the Anthropic-compatible API in all regions: Singapore, China North 2 (Beijing), Germany (Frankfurt), China (Hong Kong), Japan (Tokyo), and US (Virginia). Responses API
  • Qwen-Max: Qwen4-Max (qwen4-max), Qwen3.8-Max series, Qwen3.7-Max series
  • Qwen-Plus: Qwen3.7-Plus series, Qwen3.6-Plus series, Qwen3.5-Plus series
  • Qwen-Flash: qwen3.8-flash
  • DeepSeek: deepseek-v4.1-flash, deepseek-v4.1-flash-1017, deepseek-v4-pro-0813, deepseek-v4-flash-0731, deepseek-v4-pro
  • GLM: glm-5.2
  • Kimi: kimi-k3
  • Qwen3.8 open-source series

Other models

The following models also support this tool but may not perform as well as the recommended models. Only supported through the Responses API.
  • Qwen-Flash: qwen3.8-flash, Qwen3.7-Flash series, Qwen3.6-Flash series, Qwen3.5-Flash series
  • Qwen3.8 open-source series (qwen3.8-2.4t-a95b)
  • Qwen3.6 open-source series (excluding qwen3.6-27b)
  • Qwen3.5 open-source series
  • DeepSeek: deepseek-v4.1-flash, deepseek-v4.1-flash-1017, deepseek-v4-pro-0813, deepseek-v4-flash-0731, deepseek-v4-pro
  • GLM: glm-5.2

Limitations

  • Web extractor retrieves publicly accessible pages only. Pages behind authentication or paywalls return empty content.
  • Very large pages may be truncated before being added to the context window.
  • Dynamic content rendered exclusively by JavaScript may not be fully captured.
  • The extracted content counts as input tokens, which increases both latency and cost for large pages.

Error handling

When extraction fails, the model does not raise an error. Instead, the web_extractor_call item in the response output returns empty or partial content, and the model generates its response based on whatever context is available. Common failure scenarios:
ScenarioBehavior
URL is unreachable (404, 500, DNS failure)Extraction returns empty content; model responds using other available context
Page load times outPartial or empty content returned
Non-HTML content (PDF, images)Content may not be extracted; model falls back to other tools or general knowledge
To verify whether extraction succeeded, inspect the web_extractor_call items in response.output or check usage.x_tools.web_extractor.count for the number of successful invocations.

Billing

Web extractor costs have two components:
The prices listed below are list prices. For current promotions and discounted pricing, visit the Model Marketplace.
ComponentDetails
Model costExtracted web content is appended to the prompt, which increases input tokens. Billed at the model's standard token price. See Pricing for pricing.
Web search feePer 1,000 search invocations: $10.00 in Singapore, $0.573411 in China North 2 (Beijing), Germany (Frankfurt), China (Hong Kong), Japan (Tokyo), and US (Virginia). Web extractor typically triggers web search internally.
Web extractor feeFree for a limited time.
For qwen4-max Responses, read tool counts from the top-level tool_usage: web_search.num_requests is the billed search count, web_extractor.num_requests is the billed extraction count, and pdf_page_parser.num_requests is the parsed page count. For streaming, read these fields from response.completed.response.tool_usage. Output item counts do not determine these counts. Legacy counters apply only to models that still return them.