Exclude External MCP tools from Apple On-Device; fix stale 4K context claims

Live-caught by Rune: a trivial one-tool request ("how many files in my
Downloads folder?") failed with "Provided 10,466 tokens, but the
maximum allowed is 8,192" — on the very first real test of tool
calling. Measured the real cause with Rune's actual live tool
configuration (via a throwaway, uncommitted diagnostic test, not
guessed): registering all 49 currently-enabled tools cost ~3,300
tokens of descriptions alone, before the system prompt, the user's
message, or any tool result even factored in. One External MCP server
(Obsidian, 16 tools) was 63% of that footprint by itself — third-party
tool descriptions are outside Confab's control and can be arbitrarily
verbose, unlike Confab's own deliberately-concise built-in tools.

Checked the fix approach with Rune before implementing (a real
product/scope decision, not something to guess): exclude External MCP
tools specifically for Apple On-Device (other providers' 100K+-token
windows absorb this fine) rather than building a full per-provider
tool-selection UI right now. Re-measured after the fix: 33 tools,
~1,212 tokens — comfortable headroom in a real conversation. Confab's
own built-in tools (files, bash, calendar, mail, contacts, Paperless,
Maps) are unaffected.

Also fixed while investigating: the real error dynamically reported
the actual 8,192-token limit (Apple's own error carries this), which
exposed that Confab's own hardcoded "4K context window" claims
(ModelInfo.contextLength, capabilities.maxContextLength, the model
description text, and a stale hardcoded "(4,096 tokens)" in the
macOS 26.x-only GenerationError fallback path) were wrong for the
current beta. Updated the verified ones to 8192; left the macOS
26.x-only error message without a specific number since whether that
runtime shares the same limit is genuinely unverified.

429 tests, clean.
This commit is contained in:
2026-08-28 13:08:00 +02:00
parent b62dd10d7b
commit d357bb1a33
2 changed files with 23 additions and 4 deletions
+12 -4
View File
@@ -32,12 +32,17 @@ final class AppleFoundationProvider: AIProvider {
// ChatViewModel.generateAppleOnDeviceToolResponse() not the shared chatWithToolMessages(...)
// every other provider's tool loop uses. See the Apple Intelligence tool-calling plan
// (/Users/rune/.claude/plans/apple-intelligence-tool-calling-plan.md).
//
// Context window: live-observed as 8,192 tokens (not the 4,096 this used to say) via a real
// `contextSizeExceeded` error on macOS 27 beta 2026-08-28 `mapLanguageModelError` below reports
// the real number dynamically from that error, which is how the discrepancy was caught. Unverified
// whether macOS 26.x runtimes share this same limit; revisit if that turns out to differ.
let capabilities = ProviderCapabilities(
supportsStreaming: true,
supportsVision: false,
supportsTools: true,
supportsOnlineSearch: false,
maxContextLength: 4096
maxContextLength: 8192
)
// MARK: - Models
@@ -47,8 +52,8 @@ final class AppleFoundationProvider: AIProvider {
ModelInfo(
id: "apple-on-device",
name: "Apple On-Device (Beta)",
description: "On-device Apple Intelligence model. Private, free, and works offline. 4K context window. Apple's Foundation Models framework is still in active beta (currently macOS 27 beta) — expect occasional generation errors and rough edges.",
contextLength: 4096,
description: "On-device Apple Intelligence model. Private, free, and works offline. 8K context window. Apple's Foundation Models framework is still in active beta (currently macOS 27 beta) — expect occasional generation errors and rough edges. Tool calling doesn't include External MCP servers (their tool descriptions are outside Confab's control and easily exceed the on-device context budget) — Confab's own built-in tools (files, bash, calendar, mail, contacts, Paperless, Maps) still work.",
contextLength: 8192,
pricing: ModelInfo.Pricing(prompt: 0, completion: 0),
capabilities: ModelInfo.ModelCapabilities(
vision: false,
@@ -393,7 +398,10 @@ final class AppleFoundationProvider: AIProvider {
private func mapGenerationError(_ error: LanguageModelSession.GenerationError) -> Error {
switch error {
case .exceededContextWindowSize:
return ProviderError.unknown("Apple Intelligence context limit exceeded (4,096 tokens). Start a new chat or enable Progressive Summarization in Settings → Advanced.")
// Unlike LanguageModelError.contextSizeExceeded above, this macOS 26.x-only case carries
// no token-count payload and whether 26.x runtimes share the 8,192 limit observed on
// macOS 27 beta is unverified, so no specific number is claimed here.
return ProviderError.unknown("Apple Intelligence context limit exceeded. Start a new chat or enable Progressive Summarization in Settings → Advanced.")
case .rateLimited:
return ProviderError.rateLimitExceeded
case .guardrailViolation:
+11
View File
@@ -2002,7 +2002,18 @@ Don't narrate future actions ("Let me...") - just use the tools.
// has no durable id to restore from later even if we tried (S-only persistence, no
// autosave). See AppleFoundationProvider.respondWithTools's persistTranscriptId doc comment.
let persistTranscriptId = currentConversationId
// External MCP servers' tool descriptions are outside Confab's control and can be
// arbitrarily verbose live-caught 2026-08-28: with a real-world set of enabled
// integrations, registering every tool (49 of them) cost ~3,300 tokens of descriptions
// alone before the system prompt, the user's message, or any tool result even factored in,
// blowing straight through the 8K budget on a single trivial request. One External MCP
// server (Obsidian, 16 tools) was 63% of that footprint by itself. Confab's own built-in
// tools are deliberately kept concise; third-party servers aren't, and there's no way to
// trim someone else's tool descriptions so exclude External MCP tools specifically for
// Apple On-Device rather than for every provider (their 100K+-token windows absorb this
// fine). Built-in tools (files, bash, calendar, mail, contacts, paperless, maps) still work.
let tools = MCPService.shared.getToolSchemas(onlineMode: onlineMode)
.filter { !ExternalMCPManager.shared.isExternalTool($0.function.name) }
let baseInstructions = effectiveSystemPrompt
// Excludes the just-appended current-turn user message only consulted by
// AppleFoundationProvider if it actually has to rebuild the session, for best-effort