Docs

Frequently Asked Questions (FAQ)

Common questions and answers about the Chutes API, SDK and platform.

General Questions

What is Chutes?

Chutes is a decentralized, serverless inference platform for open-source AI models. The quickest way to use it is the hosted, OpenAI-compatible API: point any OpenAI-style client at the base URL https://llm.chutes.ai/v1, send a Chutes API key (prefix cpk_) as a Bearer token in the Authorization header, and pick a model from https://llm.chutes.ai/v1/models.

curl https://llm.chutes.ai/v1/chat/completions \
  -H "Authorization: Bearer $CHUTES_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "deepseek-ai/DeepSeek-V4-Flash-0731-TEE",
       "messages": [{"role": "user", "content": "hello"}]}'

The models are served on Bittensor subnet 64 (SN64), where independent GPU operators compete to serve the workload. If you need your own code or model, the Python SDK and chutes CLI build and deploy a custom chute. Start with the Starter Guide for the API, or the Quick Start Guide for the SDK.

How is Chutes different from other platforms?

Feature	Chutes	Traditional Cloud	Other AI Platforms
Setup Time	Minutes	Hours/Days	Hours
Scaling	Automatic (0→∞)	Manual	Limited
Pricing	Pay-per-use	Always-on	Subscription
GPU Access	Latest hardware	Limited selection	Restricted
Code Style	Simple Python	Complex configs	Platform-specific

Who should use Chutes?

Perfect for:

AI/ML engineers building production applications
Startups needing scalable AI infrastructure
Researchers requiring powerful GPU compute
Companies wanting serverless AI deployment

Use cases:

LLM chat applications
Image/video generation services
Real-time AI APIs
Batch processing workflows
Model inference at scale

Is Chutes suitable for production?

Before you run a production workload, review the pages that set out what Chutes commits to:

Privacy Policy: for public LLM inference, Chutes does not log, store, or persist the content of your API requests or responses.
Trusted Execution Environments: confidential compute for sensitive workloads.
Data Processing Agreement: governs how Chutes processes personal data on behalf of customers.
Pricing: current per-model prices and plans.

For reliability, scaling, security and observability guidance on chutes you deploy, see the Production Readiness Guide.

Getting Started

How do I get started with Chutes?

Install the SDK
```
pip install chutes
```
Create account and authenticate
```
chutes auth login
```

Deploy your first chute

from chutes.chute import Chute

chute = Chute(username="myuser", name="hello-world")

@chute.cord(public_api_path="/hello")
async def hello():
    return {"message": "Hello, World!"}

chutes deploy

Do I need Docker experience?

No! Chutes handles containerization automatically. However, if you need custom dependencies, you can optionally use our class:

from chutes.image import Image

# Simple dependency installation
image = (
    Image(username="myuser", name="my-app", tag="1.0")
    .from_base("nvidia/cuda:12.4.1-runtime-ubuntu22.04")
    .run_command("pip install transformers torch")
)

chute = Chute(
    username="myuser",
    name="my-app",
    image=image
)

What programming languages are supported?

Currently, Chutes supports Python only. We're considering other languages based on user demand.

Python versions supported:

Python 3.8+
Recommended: Python 3.10 or 3.11

Can I use my existing Python code?

Yes! Chutes is designed to work with existing Python codebases. You typically just need to:

Wrap your functions with decorators
Add any dependencies to an if needed
Deploy with

Deployment & Usage

How long does deployment take?

First deployment: 5-15 minutes (includes image building)
Code-only updates: 1-3 minutes
No-code config updates: 30 seconds

Can I deploy multiple versions?

Yes! Each deployment creates a new version:

# Deploy new version
chutes deploy

# List versions
chutes chutes versions <chute-name>

# Rollback to previous version
chutes chutes rollback <chute-name> --version v1.2.3

How does scaling work?

Chutes automatically scales based on traffic:

Scale to zero: No requests = no costs
Auto-scaling: Handles traffic spikes automatically
Cold start optimization: Fast instance startup

# Configure scaling behavior
chute = Chute(
    username="myuser",
    name="my-app",
    min_replicas=0,    # Scale to zero
    max_replicas=100   # Scale up to 100 instances
)

Can I deploy the same model multiple times?

Yes! You can have multiple deployments:

# Production deployment
prod_chute = Chute(
    username="myuser",
    name="llm-prod",
    node_selector=NodeSelector()
)

# Development deployment
dev_chute = Chute(
    username="myuser",
    name="llm-dev",
    node_selector=NodeSelector()
)

How do I handle different environments?

Use environment variables and different chute names:

import os

environment = os.getenv("ENVIRONMENT", "dev")
chute_name = f"my-app-{environment}"

chute = Chute(username="myuser", name=chute_name)

Performance & Optimization

How can I optimize performance?

Model optimization:

# Use optimized engines
from chutes.chute.template.vllm import build_vllm_chute

chute = build_vllm_chute(
    username="myuser",
    name="fast-llm",
    model_name="microsoft/DialoGPT-medium",
    engine_args={
        "gpu_memory_utilization": 0.9,
        "enable_chunked_prefill": True,
        "use_v2_block_manager": True
    }
)

Hardware selection:

# Choose appropriate hardware
from chutes.chute import NodeSelector

node_selector = NodeSelector(
    gpu_count=1,
    min_vram_gb_per_gpu=24,
    include=["h100", "a100"]  # High-performance GPUs
)

Caching strategies:

from functools import lru_cache

@lru_cache(maxsize=1000)
def expensive_computation(input_hash):
    return compute_result(input_hash)

What's the latency for API calls?

Typical latencies:

Warm instances: 50-200ms
Cold start: 5-30 seconds (depending on model size)

How do I minimize cold starts?

# Keep minimum replicas warm
chute = Chute(
    username="myuser",
    name="low-latency-app",
    min_replicas=1  # Always keep 1 instance warm
)

# Optimize startup time
@chute.on_startup()
async def setup(self):
    # Load models efficiently
    self.model = load_model_optimized()

Can I use multiple GPUs?

Yes! Specify multiple GPUs in your node selector:

# Multi-GPU setup
node_selector = NodeSelector(
    gpu_count=4,  # Use 4 GPUs
    min_vram_gb_per_gpu=40
)

# Distribute model across GPUs
@chute.on_startup()
async def setup(self):
    self.model = load_model_distributed(device_map="auto")

Pricing & Billing

How does pricing work?

There are two billing models, both listed on chutes.ai/pricing:

Hosted models (the OpenAI-compatible API): pay per token, priced per 1M input tokens and per 1M output tokens for each model. No subscription and no minimum.
Private chutes you deploy: billed by the second at the GPU's hourly rate while the instance runs, plus a one-time deployment fee equal to 3× the hourly rate.

Per-model prices are also in the price field of https://llm.chutes.ai/v1/models.

How can I control costs?

Scale to zero:

chute = Chute(
    username="myuser",
    name="cost-optimized",
    min_replicas=0  # No idle costs
)

Choose appropriate hardware:

# Cost-effective GPUs for development
node_selector = NodeSelector(
    include=["l40", "a6000"],  # Less expensive than H100
    exclude=["h100", "h200"]
)

Monitor usage:

# Check current usage
chutes account usage

# Set billing alerts
chutes account alerts --threshold 100

Do you offer volume discounts?

The Enterprise plan includes volume discounts, custom rate limits and dedicated support; contact sales or email support@chutes.ai. See chutes.ai/pricing for current pricing.

Features & Capabilities

What AI frameworks are supported?

Officially supported:

PyTorch: Full support with CUDA optimization
Transformers: Hugging Face models and pipelines
VLLM: High-performance LLM inference
SGLang: Structured generation for LLMs
Diffusers: Image/video generation models

Community supported:

TensorFlow/Keras
JAX/Flax
ONNX Runtime
OpenCV
scikit-learn

Can I use custom models?

Absolutely! Upload your models several ways:

# From Hugging Face Hub
model_name = "your-username/custom-model"

# From local files
image = Image().copy("./my-model/", "/opt/model/")

# From cloud storage
image = Image().run([
    "wget https://storage.example.com/model.bin -O /opt/model.bin"
])

Do you support streaming responses?

Yes! Perfect for LLM chat applications:

from typing import AsyncGenerator

@chute.cord(public_api_path="/stream")
async def stream_generate(self, prompt: str) -> AsyncGenerator[str, None]:
    async for token in self.model.stream_generate(prompt):
        yield f"data: {token}\n\n"

Can I run background jobs?

Yes! Use the decorator:

@chute.job()
async def process_batch(self, batch_data: List[str]):
    results = []
    for item in batch_data:
        result = await self.process_item(item)
        results.append(result)
    return results

# Trigger job
@chute.cord(public_api_path="/submit_batch")
async def submit_batch(self, data: List[str]):
    job_id = await self.process_batch(data)
    return {"job_id": job_id}

Is there a Python client library?

Yes! Use the generated client or standard HTTP:

# Generated client (coming soon)
from chutes.client import ChuteClient

client = ChuteClient("https://your-chute.chutes.ai")
result = await client.predict(text="Hello world")

# Standard HTTP requests
import httpx

async with httpx.AsyncClient() as client:
    response = await client.post(
        "https://your-chute.chutes.ai/predict",
        json={"text": "Hello world"}
    )
    result = response.json()

Technical Details

What regions are available?

Compute on Chutes is supplied by independent GPU operators (miners) on Bittensor subnet 64; see Mining on Chutes. When you deploy your own chute, NodeSelector chooses hardware by GPU count, minimum VRAM per GPU, maximum hourly price per GPU, and GPU types to include or exclude. It has no region parameter. For how request data is handled, see the Privacy Policy.

What GPU types are available?

For hosted models you do not pick hardware: you call a model by name and pay per token. For chutes you deploy yourself, request hardware with NodeSelector. The NodeSelector reference lists the GPU type identifiers it accepts, and current GPU hourly rates for private chutes are on chutes.ai/pricing.

How does networking work?

Public endpoints: HTTPS with automatic SSL certificates
Load balancing: Automatic traffic distribution

What about data persistence?

Temporary storage (included):

Container filesystem
Cleared on restart/redeploy

Persistent storage (optional):

chute = Chute(
    username="myuser",
    name="persistent-app",
    storage_gb=100  # 100GB persistent disk
)

# Access at /opt/storage/
@chute.cord(public_api_path="/save")
async def save_data(self, data: str):
    with open("/opt/storage/data.txt", "w") as f:
        f.write(data)

Can I access the underlying infrastructure?

Chutes is serverless, so direct infrastructure access isn't available. However, you get:

System info: CPU, memory, GPU details via APIs
Metrics: Performance monitoring and alerts
Logs: Comprehensive application and system logs
Debug endpoints: Custom debugging interfaces

Troubleshooting

My deployment is failing. What should I check?

Validate configuration:
```
chutes chutes validate --file chute.py
```

Check build logs:

chutes chutes logs --build-logs <chute-name>

Verify resource availability:
```
chutes nodes list --available
```
Common fixes:
- Reduce GPU requirements
- Use more flexible node selector
- Check dependency versions

I'm getting out of memory errors. How do I fix this?

Immediate fixes:

# Request more VRAM
node_selector = NodeSelector(min_vram_gb_per_gpu=48)

# Or reduce batch size
engine_args = {"max_num_batched_tokens": 1024}

# Enable memory optimization
engine_args = {"gpu_memory_utilization": 0.85}

See the Troubleshooting Guide for more details.

How do I debug performance issues?

# Add performance monitoring
import time

@chute.cord(public_api_path="/predict")
async def predict(self, input_data):
    start_time = time.time()
    result = await self.model.predict(input_data)
    duration = time.time() - start_time
    self.logger.info(f"Prediction took {duration:.2f}s")
    return result

# Check resource usage
@chute.cord(public_api_path="/stats")
async def get_stats(self):
    return {
        "gpu_memory": torch.cuda.memory_allocated(),
        "cpu_percent": psutil.cpu_percent()
    }

Integrations

Can I integrate with my existing CI/CD?

Yes! Chutes works with any CI/CD system:

GitHub Actions:

name: Deploy to Chutes
on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Setup Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.10"
      - name: Install Chutes
        run: pip install chutes
      - name: Deploy
        run: chutes deploy --name my-app-prod
        env:
          CHUTES_API_KEY: ${{ secrets.CHUTES_API_KEY }}

Does it work with monitoring tools?

Yes! Export metrics to your preferred tools:

# Prometheus metrics
@chute.cord(public_api_path="/metrics")
async def metrics(self):
    return generate_prometheus_metrics()

# Custom webhooks
@chute.cord(public_api_path="/predict")
async def predict(self, input_data):
    result = await self.model.predict(input_data)

    # Send to monitoring
    await send_to_datadog(metric="prediction_count", value=1)

    return result

Can I use it with databases?

Absolutely! Connect to any database:

# PostgreSQL example
import asyncpg

@chute.on_startup()
async def setup(self):
    self.db = await asyncpg.connect(
        host=os.getenv("DB_HOST"),
        user=os.getenv("DB_USER"),
        password=os.getenv("DB_PASSWORD"),
        database=os.getenv("DB_NAME")
    )

@chute.cord(public_api_path="/query")
async def query_data(self, query: str):
    rows = await self.db.fetch("SELECT * FROM table WHERE condition = $1", query)
    return [dict(row) for row in rows]

Security & Privacy

How secure is my data?

For public LLM inference, Chutes does not log, store, or persist the content of your API requests or responses. In standard API mode, request data may be read transiently in memory to route and run the request, but it is never written to persistent storage. In the optional end-to-end encryption mode, Chutes' servers cannot read the content at all. The Privacy Policy lists the metadata that is still collected, such as timestamps, token counts and model identifiers.

Models that expose confidential_compute: true run inside trusted execution environments backed by Intel TDX and NVIDIA Protected PCIe; see Trusted Execution Environments and the security architecture. The Data Processing Agreement governs how Chutes processes personal data on behalf of customers.

Private and user-created chutes run third-party code, so these guarantees depend on that code. Review a chute author's policies before sending it sensitive data.

Can I use private models?

Yes! Several options for private models:

# Private Hugging Face models (requires token)
os.environ["HUGGINGFACE_HUB_TOKEN"] = "your_token"

# Upload during build
image = Image().copy("./private-model/", "/opt/model/")

# Download from private S3
image = Image().run([
    "aws s3 cp s3://private-bucket/model.bin /opt/model.bin"
]).env("AWS_ACCESS_KEY_ID", "your_key")

Still have questions?

Community: Join our Discord for community support
Documentation: Check our comprehensive docs
Support: Email for technical assistance
Sales: Contact

We're constantly updating this FAQ based on user feedback. If you have a question not covered here, please let us know!