Why models end up air-gapped
Some environments have no internet access because the data is sensitive, network policy prohibits outbound traffic, or both. Hosted model APIs, deployment-time pulls from Docker Hub, and setup-time installs from PyPI are unavailable.
Security policy, client data handling rules, classified networks, unreliable connectivity, and requirements to keep inference data inside the building can all drive the decision. Whatever the reason, every required artifact must be present before deployment into the isolated environment.
What has to be local
AI tooling can reach the internet in easy-to-miss places: a model server may fetch a tokenizer config on first boot, a Python package may check for updates during import, or an embedding model may pull a vocabulary file from a CDN. In an air-gapped environment, those requests can delay startup, log warnings, or stop the service entirely.
The full dependency list for offline deployment usually includes:
- Model weights. The model files in the runtime's expected format, such as GGUF, safetensors, or ONNX. Store them locally so the runtime does not fetch them during load.
- Tokenizer and vocabulary files. Libraries such as Hugging Face Transformers often download these separately from the model weights. Point environment variables and library settings to local paths, and prevent network requests at runtime.
- Runtime dependencies. Package Python dependencies, system libraries, CUDA user-space libraries, and cuDNN in the image, and install the compatible GPU driver on the host before deployment.
- Container images. Built and tagged on a connected machine, then transferred to the air-gapped environment via a local registry, a file export, or physical media.
- Configuration files. Prompt templates, system instructions, generation parameters, serving configs. Version these alongside the model.
A common failure: staging has internet access, so a library quietly downloads a 200 KB config file on startup. The same request times out or is blocked in production, leaving an opaque service error. Disable network access during the test; removing the API key alone does not exercise this failure mode.
Container and registry strategy
Container images are the cleanest way to package everything. On a connected machine, build a versioned image containing the model weights, tokenizer files, Python environment, required CUDA runtime libraries, serving framework, and configuration. Test that image with the network disabled before transferring it.
Transfer options depend on the environment:
- Local registry. Run a private container registry inside the air-gapped network. Import approved image archives through the transfer process, then push them to the registry from within the isolated network.
- Image export. Use
docker saveor equivalent to write the image to a tar file. Move the tar via an approved transfer mechanism. Load it withdocker loadon the target machine. - Physical media. For truly disconnected environments, approved physical media is a common transfer method.
Whichever method you use, record the image hash, the model version inside it, and the date of transfer. When something breaks six months later, you need to know exactly what's running.
Dependency management without a package manager
For containerized targets, bake every dependency into the image. Host deployments require every dependency to be preinstalled and version-locked.
Python needs special handling because many tools assume network access. Pip reaches for PyPI, Hugging Face libraries for the Hub, and NLTK and spaCy for separate data or model packages. Each can use local packages, caches, or data, but the setup differs and must be configured explicitly.
On a connected machine, download every required package, model, and data file into a local directory or cache, transfer it, and configure the application to use those local paths. Verify the result with all outbound connections hard-blocked; partial firewall restrictions can leave hidden network dependencies untested.
Evaluation without external services
Evaluation harnesses often depend on hosted LLM judges, embedding APIs, or external dataset servers. Those services are unavailable in an air-gapped environment.
Package the evaluation stack for offline operation as well:
- Eval datasets stored locally, versioned, and included in the deployment package.
- Scoring models (if using model-graded evaluation) running locally alongside the system under test.
- Deterministic scoring functions that require no external calls, including string matching, structural checks, and rubric-based rules.
- Results written to local storage with enough detail to reconstruct any individual test case.
Run the full eval suite on the air-gapped machine after deployment. This validates the transferred installation and can expose differences in library versions, GPU drivers, or file permissions that were absent on the build machine.
Updates and patching
Model updates in an air-gapped environment follow a deliberate cycle: build and test on a connected machine, package the update, transfer it through the approved channel, deploy it on the target, run the eval suite, and promote it to production only after the results match expectations.
The process is slower than pulling a new image tag and restarting a pod. Every change is intentional, tested, and recorded. Pinning the transferred image also prevents an upstream base-image update from silently changing the deployment.
Keep the current deployment and the previous known-good version on the target machine. If a new model introduces a regression, switch back through configuration immediately and avoid another multi-day transfer.
Version record. For each deployed version, track: model identity and quantization, container image hash, transfer date, eval results on the target machine, who approved the promotion, and the rollback path. This record is also useful during audits and procurement reviews where the reviewer wants to know exactly what's running and how it got there.
Operational overhead
Operating the system adds substantial work around inference:
- Transfer logistics. Every update, security patch, or config change has to go through the transfer process. If that process involves paperwork, review, and physical media, a five-minute code change on a connected machine becomes a multi-day deployment cycle.
- Driver and library compatibility. GPU drivers, CUDA versions, Python libraries, and the model runtime must remain compatible. In an air gap, each update is a coordinated operation.
- Monitoring without cloud services. Cloud monitoring services such as Datadog and CloudWatch are unavailable in a fully disconnected environment. Monitoring has to run locally, and someone has to maintain it and review its output.
- Debugging without Stack Overflow. When something breaks at 2 AM, the person on call can't Google the error message. Documentation, runbooks, and known-issue logs need to be thorough and local.
Budget for transfer, maintenance, monitoring, and support from the start. Inference hardware alone does not reflect the full operating cost of an air-gapped deployment.
Testing the air gap
A reliable predeployment test for offline operation is to run the complete deployment with outbound network access blocked. Use a test environment whose egress firewall denies all outbound traffic, then deploy the stack, run the application and eval suite, and inspect the logs for connection timeouts, DNS failures, or requests to external endpoints.
Do this before every deployment to a real air-gapped environment. A dependency that worked offline three months ago might have added a network call in a minor version bump. The test catches it before the operations team discovers it on the wrong side of the air gap.