What is the OpenAI Vision API?

The OpenAI Vision API lets GPT-4o and GPT-4o-mini analyze images natively. You pass images as base64-encoded data URIs or public URLs in the messages array using the image_url content type. The model can describe, extract text (OCR), compare, and reason about images just like text.

What image formats does GPT-4o Vision support?

GPT-4o Vision supports PNG, JPEG, GIF, and WebP image formats. The maximum file size per image is 20 MB. Video is not supported — for video analysis you must extract individual frames and pass them as separate image content blocks.

What is the detail parameter in OpenAI Vision?

The detail parameter controls how GPT-4o processes an image. 'low' always uses 85 tokens and processes the image as a 512x512 thumbnail — fast and cheap. 'high' tiles large images into 512x512 chunks (max 1024 tokens per tile) for much higher accuracy on fine detail. 'auto' lets the model choose based on the input image size.

How does GPT-4o-mini vision compare to GPT-4o?

GPT-4o-mini supports the same image_url content type and detail parameter as GPT-4o but costs roughly 30x less per token. It works well for simple tasks like OCR on clean documents, image classification, and basic scene description. GPT-4o is recommended for complex chart reading, handwritten text, multi-step visual reasoning, and medical or scientific images.

OpenAI Vision API Multimodal 9 min read

OpenAI Vision API Guide 2025: Analyze Images with GPT-4o

GPT-4o and GPT-4o-mini can analyze images natively — no separate model, no separate endpoint. Pass an image URL or base64-encoded data alongside your text prompt in the standard chat completions messages array, and the model returns a detailed analysis. This guide covers every aspect of the Vision API: content types, the detail parameter, OCR, screenshot analysis, chart reading, multi-image comparison, streaming, and cost optimization.

What Is OpenAI Vision?

OpenAI Vision is the multimodal capability built into GPT-4o and GPT-4o-mini. Unlike earlier approaches that required a separate vision model, GPT-4o is natively multimodal — it processes images and text in the same forward pass, enabling richer cross-modal reasoning.

Common use cases include:

OCR and document analysis — extract text from scanned documents, receipts, invoices, PDFs rendered as images
UI and screenshot analysis — describe what is on screen, find bugs in a UI, extract form field values
Chart and data extraction — read bar charts, line graphs, pie charts, and tables from images
Object detection and scene description — count objects, identify products, describe scenes
Medical and scientific imaging — analyze X-rays, microscopy images, lab results (with appropriate disclaimers)

Images are passed using the image_url content type in the messages array — the same endpoint as text completions (POST /v1/chat/completions).

Basic Image URL Example

The simplest way to use Vision is to pass a publicly accessible image URL. The model fetches and processes the image server-side — no upload required:

vision_url.py

from openai import OpenAI

client = OpenAI()  # uses OPENAI_API_KEY env var

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What is in this image? Describe it in detail."
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/4/47/PNG_transparency_demonstration_1.png/280px-PNG_transparency_demonstration_1.png"
                    }
                }
            ]
        }
    ],
    max_tokens=500,
)

print(response.choices[0].message.content)

The content array mixes text and image_url blocks freely. You can have multiple text and image blocks in any order.

Base64 Image Encoding

For local files or private images, encode them as base64 and pass as a data URI. Supported formats: PNG, JPEG, GIF, WebP. Maximum size per image: 20 MB.

vision_base64.py

import base64
from openai import OpenAI

client = OpenAI()

def encode_image(path: str) -> str:
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

image_b64 = encode_image("screenshot.png")

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Describe the UI elements visible in this screenshot."
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/png;base64,{image_b64}"
                    }
                }
            ]
        }
    ],
    max_tokens=800,
)

print(response.choices[0].message.content)

Change the MIME type to match your file: image/jpeg, image/gif, or image/webp. The data URI scheme (data:image/png;base64,...) is the same format used in HTML and CSS.

Multiple Images in One Request

GPT-4o can process multiple images in a single request — useful for comparison, before/after analysis, or multi-page document processing:

multi_image.py

import base64
from openai import OpenAI

client = OpenAI()

def b64(path):
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "Compare these two UI screenshots. What changed between v1 and v2?"},
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{b64('ui_v1.png')}"}
                },
                {
                    "type": "image_url",
                    "image_url": {"url": f"data:image/png;base64,{b64('ui_v2.png')}"}
                }
            ]
        }
    ],
    max_tokens=1000,
)

print(response.choices[0].message.content)

Each additional image costs additional tokens based on its size and the detail setting. When passing many images, consider using detail: "low" to keep costs predictable.

The detail Parameter: low, high, auto

The detail field inside image_url controls how GPT-4o tiles and processes the image:

detail value	Token cost	Processing	Best for
`low`	Always 85 tokens	Resized to 512×512	Quick classification, scene summary, cost-sensitive tasks
`high`	85 + 170 per 512×512 tile	Adaptive tiling up to 2048px	OCR, charts, fine text, detailed diagrams
`auto` (default)	Varies	Model decides	General use when cost is not a primary concern

Setting detail explicitly

{
    "type": "image_url",
    "image_url": {
        "url": "https://example.com/chart.png",
        "detail": "high"   # or "low" or "auto"
    }
}

For a 1024×1024 image at high, the model creates four 512×512 tiles plus one 512×512 overview = 5 tiles × 170 tokens + 85 base = 935 tokens. At low, the same image costs only 85 tokens.

Practical Use Cases with Code

Three of the most common production use cases for GPT-4o Vision, each with a ready-to-run code snippet:

Screenshot / UI Analysis

ui_analysis.py

import base64
from openai import OpenAI

client = OpenAI()

with open("app_screenshot.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": (
                "You are a QA engineer. List every UI element visible in this screenshot "
                "and identify any visual bugs, misaligned elements, or UX issues."
            )},
            {"type": "image_url", "image_url": {
                "url": f"data:image/png;base64,{b64}",
                "detail": "high"
            }}
        ]
    }],
    max_tokens=1500,
)
print(response.choices[0].message.content)

OCR / Document Text Extraction

ocr_extract.py

import base64
from openai import OpenAI

client = OpenAI()

with open("invoice.jpg", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": (
                "Extract all text from this invoice image. "
                "Return a structured JSON with: vendor, date, line_items (list), total."
            )},
            {"type": "image_url", "image_url": {
                "url": f"data:image/jpeg;base64,{b64}",
                "detail": "high"
            }}
        ]
    }],
    max_tokens=1000,
    response_format={"type": "json_object"},
)
print(response.choices[0].message.content)

Chart Reading and Data Extraction

chart_extract.py

import base64
from openai import OpenAI

client = OpenAI()

with open("revenue_chart.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": (
                "Extract the data from this bar chart as JSON. "
                "Include: chart_title, x_axis_label, y_axis_label, "
                "and data_points as an array of {label, value} objects."
            )},
            {"type": "image_url", "image_url": {
                "url": f"data:image/png;base64,{b64}",
                "detail": "high"
            }}
        ]
    }],
    max_tokens=800,
    response_format={"type": "json_object"},
)
print(response.choices[0].message.content)

Streaming Vision Responses

Streaming works identically for vision requests — just add stream=True. This is especially useful for long descriptions or document analysis where you want the user to see partial results immediately:

vision_stream.py

import base64
from openai import OpenAI

client = OpenAI()

with open("diagram.png", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

stream = client.chat.completions.create(
    model="gpt-4o",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Explain this architecture diagram step by step."},
            {"type": "image_url", "image_url": {
                "url": f"data:image/png;base64,{b64}",
                "detail": "high"
            }}
        ]
    }],
    max_tokens=1500,
    stream=True,
)

for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)
print()  # newline after stream ends

Streaming vision is available in both the Python SDK and via the raw HTTP API with Server-Sent Events. The image is processed before the first token streams — expect a brief pause at the start proportional to image size.

GPT-4o vs GPT-4o-mini Vision

Both models support the Vision API with identical syntax. The tradeoff is quality vs cost:

Model	Input price	Vision quality	Best for
gpt-4o	$2.50 / 1M tokens	Excellent	Complex charts, handwriting, multi-step reasoning, medical images
gpt-4o-mini	$0.15 / 1M tokens	Good	Simple OCR, image classification, scene description, high-volume pipelines

GPT-4o-mini is approximately 16x cheaper on input tokens. For a pipeline processing 10,000 invoices/day at detail: "high" (avg ~600 image tokens each), switching from gpt-4o to gpt-4o-mini cuts image token costs from ~$15/day to ~$0.90/day. Always benchmark accuracy on your specific task before committing to the cheaper model.

Vision API Limits and Pricing

Key limits and pricing details to know before building vision pipelines:

Max image size: 20 MB per image (after base64 decoding)
Supported formats: PNG, JPEG, GIF, WebP only — no PDF, SVG, or video
Video: not supported — extract frames and pass as separate image blocks
Max images per request: no hard limit documented, but practical limit is ~10-20 before hitting context length
Image tokens at low detail: always 85 tokens regardless of image dimensions
Image tokens at high detail: 85 base + 170 per 512×512 tile; a 1024×1024 image = 935 tokens; a 2048×2048 image = ~1785 tokens
Rate limits: image requests count toward your standard TPM (tokens per minute) quota — large high-detail images can consume quota quickly
No storage: OpenAI does not store images after processing; images sent via URL are fetched in real time

Token estimation for a 1920×1080 screenshot at high detail

# 1920x1080 scaled to fit 2048x2048 -> stays 1920x1080
# Tiles: ceil(1920/512) x ceil(1080/512) = 4x3 = 12 tiles
# Plus 1 overview tile = 13 tiles total
# Token cost: 13 * 170 + 85 = 2295 image tokens
# At gpt-4o:      2295 * $2.50/1M  = ~$0.0057 per image
# At gpt-4o-mini: 2295 * $0.15/1M  = ~$0.00034 per image

Monitor OpenAI API Status in Real Time

GPT-4o Vision calls count against OpenAI's API quota. Prismix monitors OpenAI's status in real time — so when the Vision API has elevated latency or errors, you know before your users do.

Monitor OpenAI Status →