nCompass (YC W24) leverages Ori Serverless Kubernetes to make AI inference 2x cost-effective

Reimagining the inference API

After being a part of the YC Winter 2024 batch, nCompass secured $1.7 M in seed funding and has since focused on building its inference platform. nCompass’ new self-serve inference API, powered by their proprietary technology, optimizes GPU utilization for inference, enabling users to scale up their AI services and manage 100s of requests per second on a single GPU, while cutting hardware expenses by up to 2x. 

nCompass’ initial optimizations ensure that at high request rates, the time-to-first-token (TTFT) doesn’t explode the same way that it does with vLLM. Performance results shared by nCompass portray up to 18x better TTFT than vLLM at high workloads. This means that as their user base scales, the availability and quality-of-service of their API offering won’t degrade. Alongside an API, nCompass also provides dedicated model instance deployments as well as on-prem deployments that minimize users’ GPU bills.

Source: Benchmark testing provided by nCompass

Ori Serverless Kubernetes, a balance of simplicity and flexibility

Although Kubernetes is a powerful tool that is an industry standard in container orchestration, implementations can turn complex very quickly. nCompass experimented with a few cloud providers before choosing Ori Serverless Kubernetes for its balance of simplicity and flexibility.

Unlike other GPU Kubernetes providers whose fully managed platforms create vendor lock-in, nCompass found Ori’s Vanilla Kubernetes with customization options a refreshing change. Ori Serverless Kubernetes’ flexibility is also helping nCompass plan for future innovations within their platform such as on-premise deployment with minimal changes to their API architecture.

Achieving inference economics at scale with Ori

Chart your own AI reality with Ori Serverless Kubernetes

Ori Serverless Kubernetes is an AI infrastructure service that combines powerful scalability, simple management and affordability enabling you to scale into the future of AI.

• Top-tier GPUs with autoscaling: Choice of NVIDIA H100 PCIe, H100 SXM, L40S, and L4 GPUs, with seamless scaling of inference clusters.

• Pay as you go: Transparent and predictable pricing with per-minute billing.

• Simple management: Ori takes care of cluster management and load balancing, so you can focus on serving your models faster and wider.