MCDMA: Bridging Mac Studio and NVIDIA Sparks via RDMA

# MCDMA: Making a Mac Studio and two Sparks work together

I have a 256 GB M3 Ultra Mac Studio and two NVIDIA GB10 Spark-class systems on my desk, and I want them to do useful work on the same model.

That is why I started MCDMA. The Sparks give me CUDA and Blackwell Tensor Cores; the Studio gives me a large pool of unified memory and high local memory bandwidth. I want to find out whether combining them can make local inference faster, or let me run a model that none of them can hold alone.

The first job was getting the machines to move data through real hardware RDMA. I now have a native macOS ConnectX-5 driver and userspace provider passing byte-verified READ and WRITE between the Studio and one Spark, with either machine initiating. I have since verified transfers between buffers written and checked by Metal and CUDA kernels, in both directions.

That gives me a working connection for the experiments I actually care about: moving a running request between CUDA and Metal, splitting a larger model across all three machines, and giving a machine useful work when it would otherwise be waiting.

## Why I wanted to build this

Prompt processing and token generation stress a model differently. Prefill processes the input and builds its attention state, with enough parallel matrix work to make the Sparks' Tensor Cores an interesting fit. At low batch sizes, decode often spends much of its time reading weights and attention state from memory.

NVIDIA specifies 273 GB/s of memory bandwidth for each Spark, while Apple specifies over 800 GB/s for M3 Ultra. That is the hardware difference I want to investigate. It does not tell me which machine will win on a particular model, but it gives me a reason to test where each part of a request should run. NVIDIA hardware specifications, Apple's M3 Ultra Studio announcement. (https://docs.nvidia.com/dgx/dgx-spark/hardware.html) (https://www.apple.com/newsroom/2025/03/apple-unveils-new-mac-studio-the-most-powerful-mac-ever/)

Prefill on the Sparks and decode on the Studio is the obvious first experiment. Prefill/decode separation already exists in systems such as NVIDIA Dynamo; my question is when it pays off across the CUDA and Metal machines I already own. (https://docs.nvidia.com/dynamo/dev/knowledge-base/concepts/system-architecture/disaggregated-serving)

I also want to understand the transformer well enough to make better choices than assigning a permanent job to each box. Once prefill finishes, could the Sparks draft tokens, run resident experts or start the next request while the Studio decodes? How much state has to move before that becomes worthwhile?

## The hardware and the constraint

The Studio has 256 GB of memory and runs macOS 27. Each Spark has a nominal 128 GB, and the pair already has its own ConnectX-7 connection.

On the Mac side, an OWC Mercury Helios 5S Thunderbolt 5 PCIe enclosure holds a Mellanox ConnectX-5 Ex MCX516A-CDAT. The card has two QSFP28 ports, each rated for 100 GbE. NVIDIA card specifications. (https://networking-docs.nvidia.com/connectx5enhw/specifications)

The full layout will connect each Spark to one of those ports. Both Sparks also have USB-C links to the Studio. The verified RDMA tests so far use one Spark-to-Studio connection; using both card ports together still needs testing.

Both card ports share the enclosure's connection to the Mac. OWC advertises up to 6,000 MB/s, about 48 Gb/s, for the Helios 5S, and the recorded Ethernet bring-up negotiated at 40 Gb/s. Neither the card's rating nor the enclosure specification is a measured RDMA payload rate for this setup. I have new 100 GbE cables coming and will measure the resulting link and throughput. OWC enclosure specifications. (https://www.owc.com/solutions/mercury-helios-5s)

This is about £800 of networking hardware, which is a lot for a personal experiment. I nearly cancelled the order, then went ahead and sold a third Spark to help fund the research. I want to see how far I can take the hardware I have.

## What I had to build

Apple recognising the Mellanox card as an Ethernet device did not give me a working ConnectX RDMA path. MCDMA began with an experimental DriverKit implementation and has progressed to a native macOS driver and userspace provider.

The driver communicates with the card's firmware, maps and registers memory, creates the queues, posts operations and checks hardware completions. The ConnectX-5 now appears in macOS verbs discovery, and native READ and WRITE pass byte verification in both directions, including with direct userspace posting.

I checked more than a successful return code. The tests verified the transferred bytes, checked the physical NIC's RDMA counters, and attempted a remote WRITE without permission. The NIC rejected that request and left the target buffer unchanged.

The demonstrated path now includes data produced and checked by the GPUs themselves. A Metal kernel filled the source buffer, the NICs moved it through RDMA, and a CUDA kernel checked every destination word. The reverse direction passed too, as did READ operations initiated by either host.

Those tests used Metal shared buffers on the Studio and CUDA mapped host allocations on the Spark. The GPUs and NICs accessed the same registered storage, with no application-level payload staging copy. Across the kernel-posting and userspace BlueFlame tests, all 24 GPU-buffer operations passed, along with deliberately incorrect-data checks and guard-region checks.

I also built and tested a reusable adapter for existing CUDA device allocations. It keeps the original application pointer and uses explicit CUDA copies through a registered communication buffer. Applications can instead allocate their communication buffers in shared storage from the outset to use the tested path without payload copies.

Direct RDMA registration of cudaMalloc device memory failed on the tested Spark, and Metal private buffers are not supported by the current registration path. CPUs still coordinate allocation, GPU work and RDMA completion. The shared-buffer path works; the next job is integrating it into the inference engine.

## Where the latency is now

Mac-initiated 4 KiB RDMA transfers now measure 7.625 µs WRITE and 6.042 µs READ, with Spark-initiated transfers at 3.680 µs WRITE and 5.536 µs READ.

These are pooled medians from three matching runs on native version 0.1.17, with userspace BlueFlame posting and a small Metal keepalive workload running continuously on the Studio. Each figure includes all 3,000 measurements after warmup, using registered host buffers at queue depth one. The timings measure submission to application-observed completion under that GPU-active condition; individual completion times vary. The GPU-buffer tests verified correctness separately and did not measure GPU-to-GPU latency or model performance.

## What I want this to unlock

The first model experiment is a complete handoff. For a model that fits on the Studio and across the Spark pair, I would keep its weights resident in both places, prefill on the Sparks, then transfer the KV cache and other required state so the Studio can continue generation.

Then I want to overlap the work. My first idea was to hand over completed state in chunks of 25 prompt tokens while the Sparks process the next chunk. The exact answer still depends on the full prompt, but the transfer can start before prefill finishes.

The hard part is making the runtimes agree. Model version, token positions, precision and cache layout all have to match. Loading the same weights on CUDA and Metal does not make their cache bytes interchangeable. I will count packing, conversion and GPU synchronisation in the result, because that is the cost a real request pays.

## One model across all three machines

The experiment I am particularly interested in is a model with roughly 400 GB of resident weights across the Studio and both Sparks. That means 400 GB at the chosen weight representation, not 400 billion parameters.

There is about 512 GB of nominal memory across the three machines. It stays separate, and each machine still needs room for its OS, KV cache, activations and runtime workspace.

One layout to test puts about 200 GB of weights across the Spark pair and 200 GB on the Studio. The Sparks could use tensor parallelism to share calculations within their layers, while the Studio runs another group of layers as a pipeline stage.

The weights stay on the machine that uses them. Activations cross the stage boundary, and each stage keeps the cache for its own layers. All required stages participate in both prefill and decode.

I have not run that model across the cluster yet. Fitting it would be a useful capacity result in its own right; making it fast would require balancing the stages and finding enough concurrent work to keep them occupied.

## Give waiting machines useful work

For a single request, I want to try speculative decoding. A smaller model proposes a block of tokens, the target model verifies them together, and the runtime commits the accepted tokens. The result depends on how many proposals survive and how much verification costs. NVIDIA's speculative decoding documentation. (https://nvidia.github.io/TensorRT-LLM/1.1.0/features/speculative-decoding.html)

For several requests, continuous batching lets a worker advance independent sequences together. That can increase total throughput, but I also want to measure what happens to the person waiting for an answer. NVIDIA's batching and scheduling documentation. (https://nvidia.github.io/TensorRT-LLM/1.3.0rc15/features/paged-attention-ifb-scheduler.html)

I want to compare fixed roles with a scheduler that gives a free machine the next useful job. Moving state or loading weights can cost more than the work saves, and a Spark holding part of a sharded model cannot simply abandon that job.

The Neural Engine is another target. A small compatible drafter running through Core ML could propose tokens for a larger model on Metal to verify. I would need to confirm which operations actually ran on the Neural Engine and whether accepted tokens per second improved. Apple's Core ML documentation. (https://developer.apple.com/documentation/coreml/mlcomputeunits)

## Use the extra links

Both Sparks also have USB-C connections to the Studio, and I want to see whether those links can carry useful data alongside QSFP.

One test would send completed KV chunks over both paths according to their measured arrival times. Another would reserve the quicker path for state needed immediately while the other moves a prefix cache or adapter for the next request.

That needs explicit ordering and ownership so the consumer knows which bytes are ready. The existing USB software transport is not hardware RDMA, and shared controllers or memory traffic may limit the gain. The measurement is whether the model spends less time waiting.

## The 25 experiments

I developed the idea register with Claude and Codex, drawing on existing research as well as my own questions. The shared-buffer work has now passed its first correctness tests. This is the wider experiment programme, with inference integration and model benchmarks still ahead.

1. Move a compatible KV cache from a CUDA runtime to MLX and continue the request on the Mac.

2. Keep attention state on the Studio when a Spark would otherwise have to evict or recompute it.

3. Transfer compact latent attention caches in their native representation, together with the other state needed to resume.

4. Try tree speculative decoding, proposing several continuations and verifying the useful branches.

5. Use a small Neural Engine drafter with a larger Metal model as verifier.

6. Route requests using prompt length, expected answer length and measured queueing time.

7. Share compatible prefix caches to avoid processing the same prompt prefix again.

8. Keep mixture-of-experts weights on different machines and send activations to the experts needed.

9. Mirror generation checkpoints, then test controlled handover and recovery after a failure.

10. Stream completed prefill state while the remaining prompt is still being processed.

11. Build on the verified CUDA/Metal shared buffers, testing runtime integration and visibility under sustained workloads.

12. Compare a persistent GPU worker reacting to transfers with ordinary kernel launches.

13. Compute attention where each part of its KV cache lives, then combine the partial results correctly.

14. Split model layers into pipeline stages and measure stage balance and microbatching.

15. Put a vector index on another machine and compare retrieval methods at equal search quality.

16. Keep LoRA adapters ready on another machine and measure the cost of switching them in.

17. Test contrastive decoding across hosts and measure whether the quality gain justifies the extra work.

18. Escalate difficult requests through a model cascade, reusing state only when it is compatible.

19. Investigate remote optimiser state for LoRA training.

20. Move a running request when the time saved exceeds the cost of migration.

21. Separate control and payload traffic and choose a transport from measured completion costs.

22. Identify identical compatible state by content so it does not need storing or transferring twice.

23. Combine predictions from models on different hosts and measure quality against compute cost.

24. Compress transferred state and count compression time, including quality checks for lossy formats.

25. Change which machine drafts and which verifies as load and acceptance rates change.

## What would convince me

I want a reproducible model result. I will compare a single Spark, the Spark pair, the Studio and the combined cluster wherever the same workload fits, using time to first token, accepted tokens per second, total response time, quality and peak memory.

A larger model that only fits across the cluster is a capacity result. A faster answer needs an end-to-end comparison that includes communication and synchronisation. I will report both, including experiments where the extra machine makes things worse.

The driver has given me a way to ask these questions on real hardware. My next target is one model request that finishes sooner because the Sparks and Studio work together, followed by the larger-model pipeline experiment.

My next step is to prepare an oMLX PR that integrates the tested RDMA transport and GPU-buffer APIs into the inference engine. That means using compatible shared communication buffers where possible, making any copies from existing CUDA allocations explicit, and connecting transfer completion to the model's execution. Then I can benchmark real KV-cache handoffs and distributed inference, including all conversion and synchronisation costs.

If you work on CUDA or Metal runtimes, I would like to compare notes on handing a running request between them without losing state or spending the benefit on conversion. I will publish the experiments and measurements as I go.

The driver, setup guide and verification instructions are on GitHub. Feel free to try MCDMA in your own projects and share what you build. This is experimental software, and I will keep documenting what works and what still needs fixing. (https://github.com/ashhart/mcdma)