Size the router's LLM pool to the active policy's local candidates

[QuManu]

Proposal size

Minor feature (a single PR)

Updates

No response

User story

As a user with a router policy that alternates between several local LLM candidates, I want the router's pool to keep all of them resident together, so that switching between candidates doesn't force a reload every time. Today this is painful because max_loaded_models is one shared capacity limit across every model type, so a policy alternating between local candidates evicts and reloads one on every switch, even when there is no real memory pressure.

High-level design

• This adds a new config key, llm_pool_autosize (boolean). When enabled, the router temporarily raises the LLM pool's effective capacity to fit the number of distinct local LLM candidates the active router policies require, instead of capping strictly at max_loaded_models. When disabled, today's behavior is unchanged. max_loaded_models remains a hard cap.

• The /health endpoint reports the currently applied floor (candidate_floor) plus a per-policy breakdown, so operators can see why the pool grew. The setting is also live-togglable through /internal/set; changing it (or max_loaded_models) takes effect immediately, evicting down to the new limit rather than waiting for the next model load.

Breaking changes

This is a self-contained C++ change fully covered by CI (ctest -L cpp-ci). Two dedicated unit test files (test_llm_candidate_floor.cpp, test_routing_helper_reconcile.cpp) exercise the floor/residency logic and the routing-helper reclaim logic respectively. It has no external dependency (model registry, backend versions, etc.) that would need ongoing human curation.

Maintenance plan

This touches router concurrency: floor changes race against in-flight model loads, config changes, and shutdown. These races were reviewed across multiple rounds by two maintainers and are covered by targeted concurrency unit tests (busy-LRU handling, shutdown ordering, generation-guarded reconciliation).

Separately, with auto_evict disabled, llm_pool_autosize can raise the number of resident LLMs without a hard backstop evicting one under VRAM pressure. The router logs a warning when this condition is active, but it's a real tradeoff worth stating explicitly.

Risks

No response

[QuManu]

@jeremyfowers

[QuManu]

@SlawomirNowaczyk @fl0rianr tagging you both here too, since you've been reviewing the underlying PR (#3399) and already discussed the scope question there. Feedback welcome.

[SlawomirNowaczyk]

Thanks, @QuManu. I've been thinking about this, and I have a feeling we are patching the symptoms, not the cause.

It seems that max_loaded_models conflates two different things: working-set sizing (how many models a policy needs warm) and a hard memory cap. Given that, llm_pool_autosize keeps the single scalar and overrides it dynamically, which leads to all the extra complexity and confusion.

Also, we already solved this more cleanly once: the RoutingHelper residency class in model_residency.h lets a policy keep its distinct helper models warm, decoupled from max_loaded_models. Whatever we do here needs to be aligned: we cannot have two separate mechanisms for what is essentially the same problem.

I believe the high-level design concerns should be:

1. • let the user set hard resource usage constraints, but do it explicitly; we cannot ship with "max_loaded_models": 1 as the default anymore. But this needs to be more fine-grained (a constraint on the number of models is mostly useless)
2. • provide suitable heuristics so that Lemonade is not wasting resources on stuff that's not being used -- but that needs to be smarter than per-model LRU, and this what we should target.

However, point 2 cannot be solved without fixing point 1.

@QuManu, could you look into this decoupling, and with fixed max_loaded_models, check if reusing the RoutingHelper residency solution would work for this user story as well?

[fl0rianr]

Maybe the clean abstraction is not to autosize a pool at all, but to let an active policy publish its required warm model set to a generic residency manager; resource limits and eviction policy can then stay orthogonal to that.