Hybrid Inference: Four-Location AI Architecture

# Hybrid Inference: One AI Experience, Four Places to Think

A technical blueprint for connecting cloud models, organizational infrastructure, personal AI computers, and the physical edge.

The next useful AI architecture will not put every request in the cloud. It will not force every request onto a laptop, either.

A public research task, a confidential engineering document, a coding session, and a robot interpreting its surroundings have different requirements. They should not inherit the same inference location simply because somebody selected a default model.

Hybrid inference means placing each workload where its data policy, quality requirements, latency budget, and economics can be satisfied. Cloud services, organizational data centers, individual computers, and edge devices become complementary execution locations.

The destination is one coherent user experience, backed by explicit choices about where information goes and where work happens. That is an architecture to build and operate, not a capability to assume every agent already provides.

## 1. Four locations, four different strengths

Cloud inference provides managed capacity, access to hosted frontier models, and an alternative to buying infrastructure for occasional demand. Microsoft Foundry, OpenAI, Anthropic, and xAI are important parts of this layer. Cloud is often appropriate for approved public information, bursty work, or tasks that justify capabilities unavailable on your hardware.

Organizational inference places serving infrastructure in a company-controlled data center or private environment. NVIDIA data-center GPUs and AMD Instinct accelerators can support shared model services. The organization gains operational control but also owns capacity planning, patching, tenant isolation, availability, and incident response. Ownership of a server is not itself a sovereignty guarantee.

Individual inference brings models to a person's machine. NVIDIA DGX Spark and RTX workstations belong here, but they are not the same product. DGX Spark uses an Arm-based GB10 platform with 128GB of unified memory. RTX workstations typically use discrete GPU memory; available capacity varies substantially by card. AMD Ryzen AI Max systems, commonly discussed as Strix Halo, offer another high-memory local path. [1][2]

The newer hardware deserves precise naming. HP's September 15 announcement identifies the ZBook Ultra G3a 16-inch, co-engineered with AMD, with up to Ryzen AI Max+ PRO 495 and 192GB unified memory. GMKtec's EVO-X5 Pro also specifies the Max+ PRO 495 and up to 192GB, with up to 160GB allocated to graphics. HP expects availability in October; GMKtec schedules launch for September 28. These are announced capabilities, not independently measured throughput. [3][4]

For the systems being discussed under the shorthand "Gorgon," use those manufacturer product names when buying or deploying. The cited announcements establish the processor and system specifications, not that codename. Do not confuse Ryzen AI Max PRO 400 with the separate Ryzen AI 400 family, or add graphics allocation to the shared-memory total.

Edge inference runs near sensors and actuators: cameras, gateways, industrial equipment, and robots. NVIDIA Jetson Orin and Jetson Thor address this territory. Their relevant constraints include power, thermals, sensor bandwidth, intermittent connectivity, and response deadlines, not just parameter count. [5]

## 2. Sovereignty is about the entire data path

Data residency asks where data is stored or processed. Sovereignty also concerns jurisdiction, administrative control, access, key custody, operational dependencies, and the ability to continue under constrained connectivity. Encryption is important, but it does not answer all those questions.

Trace the complete path: source documents, embeddings, retrieval, prompts, model execution, tool calls, memory, caches, logs, backups, and support diagnostics. A local vector database followed by a cloud model call still sends retrieved content to the cloud. A local model with a cloud browser or search tool can still create outbound disclosures.

Microsoft's deployment documentation makes another important distinction: Global and DataZone inference processing locations differ from storage-at-rest commitments. The location of an Azure resource alone is not enough to establish where every inference request is processed. Check the exact deployment type and model terms. [6]

Separate the data plane from the control plane. You may choose cloud management with local inference, but inventory what management metadata, diagnostics, credentials, and remote-access capabilities cross the boundary.

Microsoft Foundry Local runs inference on end-user devices. Foundry Local on Azure Local is a distinct enterprise deployment path using Kubernetes-native operations; its current documentation labels that offering preview and deployment access as request-based. Neither statement means every hosted proprietary model is available to download and operate privately. [7][8]

For workloads that must remain within an approved boundary, enforce that boundary outside the model: authenticated endpoints, network policy, restricted tools, controlled logs, and explicit deployment allowlists. Ask compliance and legal owners to evaluate the actual design, not a marketing label.

## 3. Build a policy router, not just a model picker

A model picker asks which model the user wants. A policy router asks which destinations are permitted, capable, healthy, and economical for this particular task.

First establish identity, data classification, jurisdiction, allowed recipients, and tool permissions. Enforce these as hard constraints. Do not send an unclassified confidential document to an external model just to ask whether it is confidential. Unknown classifications need a conservative route or review.

Then filter by capability: modality, context length, supported tool schemas, structured-output behavior, required languages, and evaluated task quality. Finally select among eligible endpoints using measured queue delay, time to first token, completion latency, throughput, cost, and current health.

The optimization is: minimize expected cost among destinations that meet the policy, quality, and latency requirements. It is not: assign privacy a small penalty and let a cheaper endpoint outweigh it.

Treat fallbacks as another policy decision. If an organizational model is unavailable, confidential work should queue, move to another approved endpoint, or fail visibly. It must not silently become a public API request. Summarization or redaction is not automatic authorization to cross the boundary.

Route task stages when useful: local extraction, private retrieval, organization-hosted synthesis, and cloud analysis of separately approved public material. Pass bounded, authorized artifacts between stages rather than forwarding the full conversation everywhere.

This is workload placement, not arbitrary model-layer splitting across the internet. Moving between unrelated models generally requires rebuilding context. Their KV caches are not portable conversation files. Distributed inference and prefill/decode disaggregation need compatible runtimes and carefully engineered interconnects.

## 4. Model choice is a deployment contract

OpenAI's hosted models and its downloadable gpt-oss releases are different choices. Anthropic's documented Claude offerings are hosted services, not a blanket self-hosting entitlement. xAI's hosted Grok services and released Grok 2 weights likewise have different capabilities and terms. Grok 2 uses a community license; downloadable does not mean unrestricted. [9][10][11]

DeepSeek, Z.ai's GLM, Alibaba's Qwen, and NVIDIA Nemotron broaden the open-weight options. But choose an exact checkpoint, not a family logo. DeepSeek-V3.2, GLM-4.5, Qwen3-235B-A22B, and NVIDIA Nemotron 3 Nano are concrete examples with their own architecture, license, precision, and serving requirements, not a ranking of the newest or best models. [12][13][14][15]

Nous Research adds another distinction: Hermes model releases and Hermes Agent are separate artifacts. A Hermes model has checkpoint-specific and base-model terms. Hermes Agent is software that can call different inference providers; running it locally does not establish that its inference is local. [16]

Maintain a deployment manifest containing model/version, weight provenance, license, tokenizer, chat template, quantization, runtime, tool parser, allowed data classes, and evaluation results. Pin these for reproducibility.

OpenAI-compatible APIs reduce integration work, but they do not guarantee identical behavior. Roles, reasoning fields, tool-call serialization, image support, streaming events, cancellation, and JSON-schema enforcement can differ. Test the complete client-model-runtime combination.

## 5. Memory capacity is not inference performance

A useful first approximation is: weight memory equals total parameter count multiplied by bits per weight, divided by eight. Then add quantization metadata, KV cache or recurrent state, runtime workspaces, activations, and operating-system headroom.

For illustration, 300 billion parameters at a uniform four bits require about 150GB of raw weight storage in decimal units, before those additions. A vendor's "300B locally" statement is therefore not a promise of large context, several simultaneous users, or fast output. Mixed precision, offloading, model architecture, and runtime implementation all change the result.

Mixture-of-experts models also have total and activated parameter counts. Activating fewer experts per token reduces some computation; it does not make the other experts' weights disappear. Offloading moves the storage and transfer burden elsewhere.

For conventional attention, KV-cache memory grows with retained tokens, concurrent sequences, attention layers, KV heads, head dimension, and cache precision. Grouped-query attention, sliding windows, cache quantization, and recurrent/hybrid architectures change that calculation.

Benchmark prefill and decode separately. Long-prompt prefill, single-user token generation, and heavily batched serving stress hardware differently. Include time to first token, inter-token latency, complete-task latency, and p95 queueing under realistic load.

DGX Spark, RTX, Ryzen AI Max, and Jetson have different CPU architectures, memory systems, power envelopes, and software paths. Validate CUDA, ROCm, ONNX execution providers, drivers, and exact model kernels where applicable. An NPU TOPS figure does not establish that a large language model will execute on that NPU.

## 6. Tokenomics: optimize cost per successful task

Here, tokenomics means the economics of inference, not a cryptocurrency incentive scheme. Dollars per million tokens is a useful billing unit, but a poor standalone business metric.

Cost per successful task = all attributable model, tool, infrastructure, operations, and review costs / tasks that meet the acceptance criteria.

Include failed attempts in the numerator. A cheaper model that repeatedly emits invalid tool calls, needs longer prompts, or requires expensive repair may cost more per completed outcome.

For hosted inference, distinguish uncached input, cache writes, cache reads, output, billed reasoning where applicable, and separately billed tools. Include search, code execution, network transfer, and retained storage when charged. Different tokenizers can produce different bills for the same text. Use actual provider usage, without double-counting reasoning already billed as output. [17]

For owned hardware, count amortization, power and cooling, maintenance, networking, storage, operations, and productive utilization. An already-purchased workstation can have low incremental cost without being economically free. A mostly idle private cluster can have expensive tokens.

A simple break-even model is Q = F / (Ccloud - Vlocal), where F is fixed local cost per period, Ccloud is comparable cloud cost per successful task, and Vlocal is local variable cost per successful task. It applies only when the denominator is positive and local capacity can meet the same quality, latency, and reliability requirements. It is an accounting model, not a hardware recommendation.

Caching and batching can improve economics, but neither is free of constraints. Isolate caches by authorized scope, invalidate stale results, and avoid cross-tenant disclosure. Batching may improve throughput while increasing waiting time. Measure the service people experience, not just GPU utilization.

## 7. Runtimes, agents, and clients are different layers

Ollama is a convenient entry point for local model serving, but it also supports cloud models. A localhost URL is not proof of local inference: the local service may forward a cloud-model request. Ollama documents controls to disable its cloud features; those controls do not disable other applications' outbound traffic. [18]

For shared serving, evaluate stacks such as vLLM, SGLang, NVIDIA TensorRT-LLM, or Triton according to their supported models and hardware. Microsoft explicitly distinguishes its client-oriented Foundry Local runtime from server frameworks designed for concurrent users. Automatic hardware acceleration is not automatic sovereignty-aware cloud routing. [7]

OpenClaw and Nous Research Hermes Agent sit above inference, coordinating tools and tasks. Their integrations illustrate why adapters matter: Hermes documents Ollama through an OpenAI-compatible endpoint, while OpenClaw's current Ollama guidance uses the native API and warns against substituting the compatibility endpoint for that integration. [19][20]

Autonomous, the personal-AI-computer company, is relevant through its hardware/software approach and Autonomous Grid. Grid distinguishes locally operated routing from a hosted-relay mode. Owning the inference machine does not make the relay path irrelevant to privacy or availability. [21][22]

Cursor is a development client and service, not simply a local inference runtime. Its BYOK documentation says requests still pass through Cursor's backend for final prompt building. Bringing a key is not the same as bringing an offline execution boundary. [23]

Grok Bot is also distinct from Grok model weights. xAI documents persistent computers running in Cursor's cloud, accessed through desktop and mobile clients. That is useful cloud execution, but installing its app on a personal computer does not relocate those computers onto the device. [24]

Evaluate every agent across three independent questions: Where does the interface run? Where does inference run? Where do tools execute? Add a fourth for persistent memory.

## 8. Edge AI is where placement becomes physical

A camera or robot may generate more raw data than it is sensible to transmit continuously. Local perception can turn that stream into bounded events, reducing bandwidth and keeping selected raw observations near their source. Events and derived features can still be sensitive; local preprocessing is not automatic anonymization.

NVIDIA Jetson provides a platform for that near-sensor compute. Wendy Labs addresses another essential layer: WendyOS and Wendy Agent, development tooling, simulation, device deployment, and fleet operations. Managing an edge fleet from the cloud and running inference on the devices are compatible choices, but they must be configured and evaluated separately. [5][25]

Robotics requires separation between high-level reasoning, perception/planning, and bounded control. A language model can help interpret an instruction or propose a plan. It should not be treated as the sole safety controller. Independent protective mechanisms, constrained commands, tested motion planning, and defined safe states remain necessary.

A local model can still miss a deadline, misread a scene, or produce an invalid action. A real-time kernel does not certify the complete system. Design for camera failure, thermal throttling, depleted power, stale maps, and loss of connectivity, not just an impressive connected demonstration.

Provision models and dependencies before disconnected operation. Use authenticated, versioned releases, staged rollout, rollback, and device health reporting. Higher-risk physical actions need their own authorization and validation, regardless of where inference runs.

## 9. Seamless UX should not mean invisible data movement

Users should not need to memorize model catalogs to finish a task. Give them one task view with progress, pause/cancel, resumable work, and clear outcomes. The interface can abstract endpoint complexity while still showing meaningful boundaries: "on this device," "organization-hosted," or "approved cloud service."

Maintain canonical task state outside provider-specific chat histories: approved inputs, source references, intermediate artifacts, pending actions, completed actions, and acceptance checks. Store that state under the same appropriate protection as its contents. Do not assume switching models preserves tool state, hidden reasoning, or context.

Use idempotency keys or equivalent transaction safeguards for side effects. If a network timeout occurs after a tool executed, reconcile its outcome before retrying on another model. A seamless fallback that sends an email twice is not seamless.

Record minimal routing receipts: policy decision, endpoint and model version, execution location, token/tool usage, latency, and result status. Keep prompt bodies and sensitive retrieval out of routine telemetry; do not collect hidden reasoning as an audit strategy. If cloud use is blocked, explain the limitation instead of quietly changing the policy.

## 10. Start with one workflow, not one universal model

Consider a hypothetical maintenance-support workflow. A Jetson-equipped device detects an equipment event. An organization-hosted model reviews permitted maintenance history. A technician's computer prepares a local draft. A cloud model analyzes separately approved public manuals when additional capability is useful. A person authorizes consequential work; machine safety remains in its dedicated controls.

That is not four copies of the same chatbot. It is a task divided according to data, timing, capabilities, and responsibility.

Start with a representative evaluation set and a load profile. Test every proposed route for task quality, malformed actions, privacy boundaries, p95 latency, disconnected behavior, and cost per successful outcome. Run fault-injection exercises for endpoint failure and ambiguous tool completion. Expand only when the evidence supports it.

Cloud intelligence, organizational control, personal compute, and edge autonomy are not mutually exclusive strategies. The opportunity is to combine them without making users manage the complexity or surrender control of their data.

Hybrid inference succeeds when the experience feels unified, the placement remains accountable, and the system knows when not to route.

## Sources

1. NVIDIA DGX Spark hardware (https://docs.nvidia.com/dgx/dgx-spark/hardware.html)

1. AMD: Strix Halo local robotics demonstration (https://rocm.blogs.amd.com/ecosystems-and-partners/rai-lemonade-agents/README.html)

1. HP: ZBook Ultra G3a announcement (https://www.hp.com/us-en/newsroom/press-releases/2026/hp-redefines-the-mobile-workstation-for-the-era-of-agentic-ai.html)

1. GMKtec: EVO-X5 Pro and September 28 launch (https://www.gmktec.com/blogs/news/gmktec-evo-x5-pro-the-next-generation-agentic-pc-with-192gb-memory-and-local-300b-ai)

1. NVIDIA: Jetson Thor and physical AI (https://developer.nvidia.com/blog/introducing-nvidia-jetson-thor-the-ultimate-platform-for-physical-ai/)

1. Microsoft: inference deployment types and processing locations (https://learn.microsoft.com/azure/ai-foundry/openai/how-to/deployment-types)

1. Microsoft: Foundry Local (https://learn.microsoft.com/azure/foundry-local/what-is-foundry-local)

1. Microsoft: Foundry Local on Azure Local preview (https://learn.microsoft.com/azure/azure-sovereign-clouds/private/foundry-local/overview)

1. OpenAI: gpt-oss (https://github.com/openai/gpt-oss)

1. Anthropic: Claude models overview (https://platform.claude.com/docs/en/models/overview)

1. xAI: Grok 2 model card and license (https://huggingface.co/xai-org/grok-2)

1. DeepSeek-V3.2 model card (https://huggingface.co/deepseek-ai/DeepSeek-V3.2)

1. Z.ai: GLM-4.5 model card (https://huggingface.co/zai-org/GLM-4.5)

1. Alibaba: Qwen3-235B-A22B model card (https://huggingface.co/Qwen/Qwen3-235B-A22B)

1. NVIDIA: Nemotron 3 Nano model card (https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16)

1. Nous Research: Hermes 3 model card (https://huggingface.co/NousResearch/Hermes-3-Llama-3.1-8B)

1. Anthropic: token, caching, and tool pricing mechanics (https://platform.claude.com/docs/en/about-claude/pricing)

1. Ollama: local operation and cloud controls (https://docs.ollama.com/faq)

1. Hermes Agent: inference providers (https://hermes-agent.nousresearch.com/docs/integrations/providers)

1. OpenClaw: Ollama integration (https://docs.openclaw.ai/providers/ollama)

1. Autonomous Personal AI Computer (https://github.com/autonomous-ai/autonomous-computer)

1. Autonomous Grid architecture (https://github.com/autonomous-ai/autonomous-grid/blob/main/docs/ARCHITECTURE.md)

1. Cursor: BYOK and request routing (https://cursor.com/help/models-and-usage/api-keys)

1. xAI: Grok Bot execution model (https://docs.x.ai/grok-bot/overview)

1. Wendy Labs: physical AI platform (https://wendy.dev/faqs)