Prismix
Modal GPU Cloud 9 min read

Modal Guide 2025: Serverless GPU Cloud for AI & Python

Modal lets you run Python functions on cloud GPUs with a single decorator. No servers to manage, no Kubernetes, no idle costs. This guide covers everything from first deploy to production Llama inference.

What Is Modal?

Modal is a serverless cloud platform purpose-built for Python and AI. Key properties:

  • Run any Python function on cloud GPUs — decorate with @app.function(gpu="A10G")
  • Build custom container images in Python code (no Dockerfile)
  • Persistent volumes, scheduled cron jobs, HTTP endpoints — all in the same file
  • Credit-based pricing — pay only for compute seconds used
  • Cold starts typically 1–5 seconds for pre-built images

Modal is popular for fine-tuning LLMs, running Whisper transcription at scale, serving Llama inference, and batch image generation jobs.

Installation & Setup

pip install modal

# Authenticate (opens browser for token)
modal token new

The token is stored in ~/.modal.toml. New accounts receive $30 free credits. No credit card required to start.

Hello GPU: Your First Modal Function

import modal

app = modal.App("hello-gpu")

@app.function(gpu="A10G")
def run_on_gpu():
    import torch
    return f"CUDA available: {torch.cuda.is_available()}, device: {torch.cuda.get_device_name(0)}"

@app.local_entrypoint()
def main():
    result = run_on_gpu.remote()
    print(result)
modal run hello_gpu.py
# Output: CUDA available: True, device: NVIDIA A10G

Custom Images & Dependencies

Define your container environment in Python — no Dockerfile needed: 

image = (
    modal.Image.debian_slim(python_version="3.11")
    .apt_install("ffmpeg")                     # system packages first
    .pip_install("transformers", "torch", "accelerate")
)

@app.function(image=image, gpu="A100")
def transcribe(audio_bytes: bytes) -> str:
    from transformers import pipeline
    pipe = pipeline("automatic-speech-recognition", model="openai/whisper-large-v3")
    return pipe(audio_bytes)["text"]

Modal caches images layer-by-layer. After the first build, reruns using the same image start in seconds.

Volumes, Secrets & Schedules

import os

# Persistent volume — survives container restarts
vol = modal.Volume.from_name("model-cache", create_if_missing=True)

# Secret from Modal dashboard
secret = modal.Secret.from_name("my-openai-secret")

@app.function(
    volumes={"/cache": vol},
    secrets=[secret],
    gpu="A10G",
    schedule=modal.Period(hours=24),   # run daily
    timeout=3600                       # 1 hour max
)
def daily_job():
    api_key = os.environ["OPENAI_API_KEY"]
    # download model to /cache on first run, reuse on subsequent runs
    vol.commit()   # flush writes

Web Endpoints & Keep-Warm

from fastapi import FastAPI

web_app = FastAPI()

@web_app.get("/predict")
async def predict(prompt: str):
    return {"result": run_model(prompt)}

@app.function(
    image=image,
    gpu="A10G",
    keep_warm=1    # keep 1 container hot to eliminate cold starts
)
@modal.asgi_app()
def fastapi_app():
    return web_app
modal deploy inference.py
# Deploys to: https://your-org--inference-fastapi-app.modal.run

GPU Types & Pricing

GPU VRAM Price/hr Best for
T4 16 GB ~$0.16 Inference for small models, batch jobs
A10G 24 GB ~$0.59 Llama 7B–13B inference, SD image gen
A100 (40 GB) 40 GB ~$1.60 Llama 70B inference, fine-tuning 7B
A100 (80 GB) 80 GB ~$2.50 Fine-tuning 13B–70B, large batch inference
H100 80 GB ~$4.50 Training runs, maximum throughput inference

Modal vs Replicate vs RunPod vs Vast.ai

Factor Modal Replicate RunPod Vast.ai
Custom code Full Python Cog framework SSH / Docker SSH / Docker
Pre-built models No Yes (1000+) Some No
Serverless Yes Yes Hybrid No (reserved)
A10G price/hr ~$0.59 ~$0.59 ~$0.44 ~$0.20–0.35
Best for Custom pipelines, fine-tuning Calling open-source models via API Persistent GPU pods, Docker Cheapest GPU, advanced users

Monitor Your AI Infrastructure

Modal apps often call OpenAI, Anthropic, or HuggingFace. Track uptime for every upstream dependency and get instant alerts when any API degrades — before your users notice.

Monitor AI API Status Free →