How to expose a local NVIDIA GPU to a Kubernetes cluster on Windows by using native Docker CE in WSL2 instead of Docker Desktop.
Adapted from @RDarrylRWith LLM token prices climbing all the time, more teams are looking at running smaller models locally, and a lot of local GPUs handle that size of workload fine. For people stuck using Windows, getting one of them visible to a Kubernetes cluster can be a challenge. The nvidia-smi tool on WSL2 is the way to check for GPUs but it many cases it will show the card but the workload will still fail. The node is usually the problem. Docker Desktop runs its control plane under sysbox-runc with no GPU passthrough, so containerd inside it never reaches /dev/nvidia*. The solution is to use native Docker CE in WSL2 and Minikube started with "--gpus=all". Check out this nice article from Vishnu Hari Dadhich. https://t.co/X2sz4kta9m #vLLM #Kubernetes #NVIDIA #Minikube #LocalAI
