CUDA Lock-In: What It Actually Costs and How to Keep Options Open

CUDA is the reason NVIDIA’s position is durable. The hardware advantage is real but contestable; the software ecosystem is what makes switching expensive. Understanding where that cost actually sits is useful whether you intend to switch or not.

What the lock-in consists of

It is not one thing. It is four layers, and they differ enormously in how portable they are.

Framework code — ordinary PyTorch or JAX written against high-level APIs. This is the most portable layer, and for a large share of research code, moving it is closer to a configuration change than a rewrite.

Optimised libraries — the highly tuned kernels for attention, convolution and matrix operations that make the difference between usable and unusable performance. Equivalents exist elsewhere but maturity varies, and the gap is workload-specific.

Custom kernels — hand-written CUDA. This is genuinely locked. Porting means rewriting, and the people who can do it well are scarce.

Tooling and habit — profilers, debuggers, deployment paths, and the accumulated knowledge of everyone on the team. Underrated, and frequently the largest cost in practice because it does not appear on any plan.

What the alternatives look like

AMD’s ROCm has improved substantially and mainstream framework support is real; the friction shows up in the long tail of libraries and in operational maturity. Apple silicon is genuinely good for local development on unified memory, and irrelevant for data-centre deployment. Specialised accelerators can be excellent for inference at scale and are not general-purpose research platforms. Compiler-based routes such as Triton and various intermediate representations aim to make kernels portable, and are promising rather than finished.

The honest summary is that for standard training and inference on mainstream architectures, alternatives are viable. For anything that depends on bleeding-edge kernels or an obscure library, they usually are not yet.

How to keep the door open cheaply

You do not need to switch to reduce the cost of switching.

Stay at the framework level wherever possible. Every drop into a custom kernel is a portability decision. Sometimes necessary; worth making consciously.

Isolate the parts that are not portable. If you must write custom kernels, keep them behind a clear interface with a slower generic fallback. The fallback is also useful for testing.

Containerise properly. Reproducible environments make trying a different platform a weekend experiment instead of a project.

Benchmark on something else once a year. Not to switch — to know what switching would cost. That number changes, and knowing it is worth something at renewal time.

When lock-in is the right choice

Portability is a cost, not a virtue. Maintaining an abstraction layer takes engineering time, and generic code is frequently slower.

If your workloads are stable, your performance requirements are demanding, and your team’s expertise is deep, committing fully is a defensible decision. The mistake is not committing. It is committing by default, without ever pricing the alternative, and then discovering the price during a procurement negotiation where you have no leverage.

Know what it would cost. Then choose.