Exclude External MCP tools from Apple On-Device; fix stale 4K context claims
Live-caught by Rune: a trivial one-tool request ("how many files in my
Downloads folder?") failed with "Provided 10,466 tokens, but the
maximum allowed is 8,192" — on the very first real test of tool
calling. Measured the real cause with Rune's actual live tool
configuration (via a throwaway, uncommitted diagnostic test, not
guessed): registering all 49 currently-enabled tools cost ~3,300
tokens of descriptions alone, before the system prompt, the user's
message, or any tool result even factored in. One External MCP server
(Obsidian, 16 tools) was 63% of that footprint by itself — third-party
tool descriptions are outside Confab's control and can be arbitrarily
verbose, unlike Confab's own deliberately-concise built-in tools.
Checked the fix approach with Rune before implementing (a real
product/scope decision, not something to guess): exclude External MCP
tools specifically for Apple On-Device (other providers' 100K+-token
windows absorb this fine) rather than building a full per-provider
tool-selection UI right now. Re-measured after the fix: 33 tools,
~1,212 tokens — comfortable headroom in a real conversation. Confab's
own built-in tools (files, bash, calendar, mail, contacts, Paperless,
Maps) are unaffected.
Also fixed while investigating: the real error dynamically reported
the actual 8,192-token limit (Apple's own error carries this), which
exposed that Confab's own hardcoded "4K context window" claims
(ModelInfo.contextLength, capabilities.maxContextLength, the model
description text, and a stale hardcoded "(4,096 tokens)" in the
macOS 26.x-only GenerationError fallback path) were wrong for the
current beta. Updated the verified ones to 8192; left the macOS
26.x-only error message without a specific number since whether that
runtime shares the same limit is genuinely unverified.
429 tests, clean.
This commit is contained in:
@@ -2002,7 +2002,18 @@ Don't narrate future actions ("Let me...") - just use the tools.
|
||||
// has no durable id to restore from later even if we tried (⌘S-only persistence, no
|
||||
// autosave). See AppleFoundationProvider.respondWithTools's persistTranscriptId doc comment.
|
||||
let persistTranscriptId = currentConversationId
|
||||
// External MCP servers' tool descriptions are outside Confab's control and can be
|
||||
// arbitrarily verbose — live-caught 2026-08-28: with a real-world set of enabled
|
||||
// integrations, registering every tool (49 of them) cost ~3,300 tokens of descriptions
|
||||
// alone before the system prompt, the user's message, or any tool result even factored in,
|
||||
// blowing straight through the 8K budget on a single trivial request. One External MCP
|
||||
// server (Obsidian, 16 tools) was 63% of that footprint by itself. Confab's own built-in
|
||||
// tools are deliberately kept concise; third-party servers aren't, and there's no way to
|
||||
// trim someone else's tool descriptions — so exclude External MCP tools specifically for
|
||||
// Apple On-Device rather than for every provider (their 100K+-token windows absorb this
|
||||
// fine). Built-in tools (files, bash, calendar, mail, contacts, paperless, maps) still work.
|
||||
let tools = MCPService.shared.getToolSchemas(onlineMode: onlineMode)
|
||||
.filter { !ExternalMCPManager.shared.isExternalTool($0.function.name) }
|
||||
let baseInstructions = effectiveSystemPrompt
|
||||
// Excludes the just-appended current-turn user message — only consulted by
|
||||
// AppleFoundationProvider if it actually has to rebuild the session, for best-effort
|
||||
|
||||
Reference in New Issue
Block a user