Dstack TEE is an open-source framework for confidential AI, based on Trusted Execution Environments (TEEs). It is currently maintained by the Linux Foundation's Confidential Computing Consortium. It is used by confidential AI platforms including NEAR AI, Phala, and RedPill AI. One of Dstack TEE's projects is private-ai-gateway, a reference implementation of a server running their Attested Confidential Inference (ACI) Specification. ACI is a framework for managing the TEE-based routing of inference requests to various upstream inference providers, including both TEE and non-TEE upstreams. The private-ai-gateway reference implementation is used directly by some of the confidential AI platforms.
ACI specified an X-Upstream-Verification header, which was designed to allow users to ensure only TEE-based upstream inference providers would be chosen for their requests. This header was also specified by ACI to have required as the default, meaning users would need to manually opt-out in order to use non-TEE inference providers. The specification made it simple for users to mandate that their prompts would only ever be in plaintext within TEE environments. However, the reference implementation of private-ai-gateway mismanaged this header, resulting in it being ignored whenever a non-TEE provider was available, defeating the purpose of the header.
I raised this issue with the Dstack-TEE maintainers, and they provided a fix, refactoring ACI and private-ai-gateway to instead support a aci_verified flag in the body of the request. When the aci_verified flag is set to true, only TEE-based upstream providers will be used. It is worth noting that this now defaults to false, which is different to the behaviour of the previous X-Upstream-Verification header.
The full details of the issue and fixes can be found here: https://github.com/Dstack-TEE/private-ai-gateway/issues/92