What are people using for multi-model backends? What about swapping configs?

I am trying to plan and deploy a machine that serves models for coding, Hermes, and whatever else. It's got multiple GPUs in it, and I want the flexibility to run different configurations (i.e. I might want to run two smaller models when I'm using Hermes and doing some less-intensive coding, swap to one big model across multiple GPUs when only Hermes is running and I'm not using anything for coding, or swap to one larger model that is better at coding and tool calls when I'm more focused on being productive). I have been down what feels like a massive rabbit hole exploring how to optimize for the best performance of local models (shout out to the club-3090 GitHub repo for both being an incredible and an amazing ego check!) to ensure I get the most performance, but the tear-down and build up of different model configurations seems to be the Achilles heel of all the solutions I have evaluated. I'm especially trying minimize the amount of manual intervention if I want to try a new model (Omni seems promising!) or I want to tune my setup.

llamaswap, LiteLLM, and llamactl all have their plusses and minuses. And other, lesser-known options crop up that seem promising--like GPUStack--but have their own issues (like being really geared towards enterprise).

I assume that I'm just going to wind up with something simple and just make peace with the idea that performance is the enemy of flexibility and every permutation I try will simply require a time investment to tune and deploy regardless of how worthwhile it turns out to be... But, I also figured that folks with capable rigs have already dealt with this and it's better to ask here than it is to waste time relearning what the community already has found.

What are you using or what have you found that is worth looking into? Thank you in advance, kind redditors, for your help!

Oh, in case it's helpful, this is a rig with up to four 3090's on an older Threadripper (3945WX)--and the permutations I have in mind are pretty much the ones above: big coding models, big "general" models, and some combination with a general model (e.g. Gemma 4 or Qwen3.6 MoE) usually up on at least one card for Hermes). I'm trying to keep the process of using new models as self-contained as possible so it can be orchestrated by Hermes and I'm isolating any bespoke tooling (like the 3090-club patched vLLM recipes) as much as possible. EDIT: Also adding that the rig will have ~128GB of DDR4-2400 RAM pieced together from older systems.