The Pulse
Cohere Opens Confidential Inference in Model Vault Beta
Cohere has opened a public beta of confidential computing for its Model Vault inference service, running on NVIDIA H100 GPUs on Microsoft Azure. Customer-side software verifies the processing environment before sending protected requests, w
AI.info Team · Written with AI · fact-checked ·
“With Model Vault and Confidential Computing, customers can verify the environment that will process their data before they send it.”
Phil Blunsom, CTO at Cohere
Cohere opened a public beta of confidential computing for its Model Vault service on October 22, adding hardware-backed safeguards and customer-side verification to its managed AI inference offering. The beta runs on NVIDIA H100 GPUs using Microsoft Azure’s confidential computing infrastructure. Cohere says customers can check the environment that will handle a request before sending sensitive information to it.
H100 beta extends protection to the GPU
Model Vault provides dedicated, managed inference for Cohere models. The new option is designed to protect prompts, documents, responses and model execution while inference is underway—the state often called data in use. Cohere says hardware-enforced isolation protects workload memory outside the confidential boundary, while NVIDIA Confidential Computing protects GPU memory and encrypts traffic between the CPU and GPU.
The release combines components from three companies: Microsoft supplies the underlying confidential computing infrastructure, NVIDIA provides confidential-computing capabilities for its H100 GPUs, and Cohere operates Model Vault’s model-serving and verification layers. That division matters because a protected CPU alone would not cover the accelerator doing much of the model’s work. Cohere’s technical explanation says the boundary spans the confidential virtual machine and the GPU performing inference.
Customers check the destination before sending data
Remote attestation is the beta’s central verification mechanism. Before a protected request leaves a customer’s infrastructure, customer-side software checks cryptographically signed evidence about the destination, including its hardware, firmware and software configuration. The evidence is compared with an approved security policy; if the environment fails that check, the client refuses to send the request.
That check is paired with encryption of the application payload itself. Cohere’s technical account says Model Vault uses Oblivious HTTP, or OHTTP, to encrypt sensitive request contents to a public key associated with the attested environment. Intermediary systems can route the encrypted message, but only the OHTTP gateway inside that environment has the corresponding private key to decrypt it; the response is protected on its way back.
Customers continue using Model Vault APIs and familiar request formats, Cohere says. The company presents the combination of end-to-end payload encryption and remote attestation as a way to avoid relying solely on a provider’s assurance that requests stay private. The claim is narrower than a guarantee that no information is exposed: authentication and routing details, including the API credential and model identifier, remain outside the OHTTP-protected payload, according to Cohere’s technical account.
Confidential computing leaves risks in view
Cohere’s announcement names financial records, clinical information, privileged legal documents, public-sector material, research and proprietary source code as potential workloads for the beta. The company says the service does not replace application security, compliance obligations, contractual safeguards or data-residency requirements. Confidential computing protects data during processing; it does not determine where that processing takes place.
The technical account also identifies remaining risks and dependencies. Physical and side-channel attacks fall outside the stated protection boundary, and an operator can still interrupt service even if it cannot read protected workload memory. Software inside the confidential environment remains vulnerable to application flaws, including prompt injection, while timing, message size and traffic volume can remain observable.
Customers must also decide which hardware, firmware, measured software and attestation infrastructure they accept as trustworthy; attestation supplies evidence, but policy determines whether that evidence is sufficient. Cohere says it intends to open-source the confidential inference stack, which would give customers and researchers more material to inspect. For now, the concrete offer is a public beta on H100 GPUs in Azure, with requests withheld when customer-side verification does not approve the environment.