Reduce Agent Context Cost by Loading Tools on Demand

Tool definitions use context before the model handles the user's request. The cost is small for a short tool list and can become material when every request carries many long schemas.

Measure the overhead first

1. Serialize the exact tool definitions sent with a representative request.
2. Measure their tokens with the tokenizer for the selected model.
3. Separate fixed tool tokens from user, retrieval, and response tokens.
4. Repeat the measurement for the routes that use different tool sets.

Do not infer the saving from a tool count. Names, descriptions, nested schemas, and examples can make two tools very different in size.

Use a small discovery catalog

Keep a short catalog in the initial context. Each entry needs an identifier, a plain description, and enough routing information to decide whether the tool applies. Load the complete instructions only after selection.

{
  "calculate_lcoe": {
    "description": "Calculate LCOE from submitted project assumptions",
    "input": "project assumptions"
  }
}

Keep execution bounded

1. Select a tool from the catalog.
2. Load its full instructions and input schema.
3. Validate the submitted input.
4. Run the tool with its existing permission and time limits.
5. Validate and record the output.

Lazy loading changes context assembly. It must not weaken tool authorization, input validation, or execution limits.

Account for the trade-offs

• Routing errors: A small catalog can omit detail that helps the model select the correct tool.
• Added latency: Fetching instructions creates another step before execution.
• Cache drift: Catalog metadata and full instructions need the same version.
• Debugging: Run records must show which catalog entry and instruction version were used.

Load only the tools the task needs

If tool definitions are a meaningful part of request cost, compare a fixed tool list with catalog-based loading on the same task set. Keep the design only if selection quality remains acceptable and the measured saving exceeds the added routing cost.