Exclude External MCP tools from Apple On-Device; fix stale 4K context claims
Live-caught by Rune: a trivial one-tool request ("how many files in my
Downloads folder?") failed with "Provided 10,466 tokens, but the
maximum allowed is 8,192" — on the very first real test of tool
calling. Measured the real cause with Rune's actual live tool
configuration (via a throwaway, uncommitted diagnostic test, not
guessed): registering all 49 currently-enabled tools cost ~3,300
tokens of descriptions alone, before the system prompt, the user's
message, or any tool result even factored in. One External MCP server
(Obsidian, 16 tools) was 63% of that footprint by itself — third-party
tool descriptions are outside Confab's control and can be arbitrarily
verbose, unlike Confab's own deliberately-concise built-in tools.
Checked the fix approach with Rune before implementing (a real
product/scope decision, not something to guess): exclude External MCP
tools specifically for Apple On-Device (other providers' 100K+-token
windows absorb this fine) rather than building a full per-provider
tool-selection UI right now. Re-measured after the fix: 33 tools,
~1,212 tokens — comfortable headroom in a real conversation. Confab's
own built-in tools (files, bash, calendar, mail, contacts, Paperless,
Maps) are unaffected.
Also fixed while investigating: the real error dynamically reported
the actual 8,192-token limit (Apple's own error carries this), which
exposed that Confab's own hardcoded "4K context window" claims
(ModelInfo.contextLength, capabilities.maxContextLength, the model
description text, and a stale hardcoded "(4,096 tokens)" in the
macOS 26.x-only GenerationError fallback path) were wrong for the
current beta. Updated the verified ones to 8192; left the macOS
26.x-only error message without a specific number since whether that
runtime shares the same limit is genuinely unverified.
429 tests, clean.
This commit is contained in:
@@ -32,12 +32,17 @@ final class AppleFoundationProvider: AIProvider {
|
||||
// ChatViewModel.generateAppleOnDeviceToolResponse() — not the shared chatWithToolMessages(...)
|
||||
// every other provider's tool loop uses. See the Apple Intelligence tool-calling plan
|
||||
// (/Users/rune/.claude/plans/apple-intelligence-tool-calling-plan.md).
|
||||
//
|
||||
// Context window: live-observed as 8,192 tokens (not the 4,096 this used to say) via a real
|
||||
// `contextSizeExceeded` error on macOS 27 beta 2026-08-28 — `mapLanguageModelError` below reports
|
||||
// the real number dynamically from that error, which is how the discrepancy was caught. Unverified
|
||||
// whether macOS 26.x runtimes share this same limit; revisit if that turns out to differ.
|
||||
let capabilities = ProviderCapabilities(
|
||||
supportsStreaming: true,
|
||||
supportsVision: false,
|
||||
supportsTools: true,
|
||||
supportsOnlineSearch: false,
|
||||
maxContextLength: 4096
|
||||
maxContextLength: 8192
|
||||
)
|
||||
|
||||
// MARK: - Models
|
||||
@@ -47,8 +52,8 @@ final class AppleFoundationProvider: AIProvider {
|
||||
ModelInfo(
|
||||
id: "apple-on-device",
|
||||
name: "Apple On-Device (Beta)",
|
||||
description: "On-device Apple Intelligence model. Private, free, and works offline. 4K context window. Apple's Foundation Models framework is still in active beta (currently macOS 27 beta) — expect occasional generation errors and rough edges.",
|
||||
contextLength: 4096,
|
||||
description: "On-device Apple Intelligence model. Private, free, and works offline. 8K context window. Apple's Foundation Models framework is still in active beta (currently macOS 27 beta) — expect occasional generation errors and rough edges. Tool calling doesn't include External MCP servers (their tool descriptions are outside Confab's control and easily exceed the on-device context budget) — Confab's own built-in tools (files, bash, calendar, mail, contacts, Paperless, Maps) still work.",
|
||||
contextLength: 8192,
|
||||
pricing: ModelInfo.Pricing(prompt: 0, completion: 0),
|
||||
capabilities: ModelInfo.ModelCapabilities(
|
||||
vision: false,
|
||||
@@ -393,7 +398,10 @@ final class AppleFoundationProvider: AIProvider {
|
||||
private func mapGenerationError(_ error: LanguageModelSession.GenerationError) -> Error {
|
||||
switch error {
|
||||
case .exceededContextWindowSize:
|
||||
return ProviderError.unknown("Apple Intelligence context limit exceeded (4,096 tokens). Start a new chat or enable Progressive Summarization in Settings → Advanced.")
|
||||
// Unlike LanguageModelError.contextSizeExceeded above, this macOS 26.x-only case carries
|
||||
// no token-count payload — and whether 26.x runtimes share the 8,192 limit observed on
|
||||
// macOS 27 beta is unverified, so no specific number is claimed here.
|
||||
return ProviderError.unknown("Apple Intelligence context limit exceeded. Start a new chat or enable Progressive Summarization in Settings → Advanced.")
|
||||
case .rateLimited:
|
||||
return ProviderError.rateLimitExceeded
|
||||
case .guardrailViolation:
|
||||
|
||||
@@ -2002,7 +2002,18 @@ Don't narrate future actions ("Let me...") - just use the tools.
|
||||
// has no durable id to restore from later even if we tried (⌘S-only persistence, no
|
||||
// autosave). See AppleFoundationProvider.respondWithTools's persistTranscriptId doc comment.
|
||||
let persistTranscriptId = currentConversationId
|
||||
// External MCP servers' tool descriptions are outside Confab's control and can be
|
||||
// arbitrarily verbose — live-caught 2026-08-28: with a real-world set of enabled
|
||||
// integrations, registering every tool (49 of them) cost ~3,300 tokens of descriptions
|
||||
// alone before the system prompt, the user's message, or any tool result even factored in,
|
||||
// blowing straight through the 8K budget on a single trivial request. One External MCP
|
||||
// server (Obsidian, 16 tools) was 63% of that footprint by itself. Confab's own built-in
|
||||
// tools are deliberately kept concise; third-party servers aren't, and there's no way to
|
||||
// trim someone else's tool descriptions — so exclude External MCP tools specifically for
|
||||
// Apple On-Device rather than for every provider (their 100K+-token windows absorb this
|
||||
// fine). Built-in tools (files, bash, calendar, mail, contacts, paperless, maps) still work.
|
||||
let tools = MCPService.shared.getToolSchemas(onlineMode: onlineMode)
|
||||
.filter { !ExternalMCPManager.shared.isExternalTool($0.function.name) }
|
||||
let baseInstructions = effectiveSystemPrompt
|
||||
// Excludes the just-appended current-turn user message — only consulted by
|
||||
// AppleFoundationProvider if it actually has to rebuild the session, for best-effort
|
||||
|
||||
Reference in New Issue
Block a user