Prismix
MLflow MLOps 9 min read

MLflow Guide 2025: LLM Experiment Tracking & Model Management

MLflow started as an experiment tracker for classical ML. In 2024-2025 it became a serious LLM observability and evaluation platform — with auto-tracing for OpenAI, LangChain, and LlamaIndex built in.

What Is MLflow?

MLflow is an open-source MLOps platform maintained by Databricks. It covers four areas of the ML lifecycle:

  • Experiment Tracking — log parameters, metrics, and artifacts for every model training run or LLM call.
  • Model Registry — version, stage (staging/production), and serve models from a central registry.
  • Model Serving — deploy models as REST endpoints (local or cloud).
  • LLM Observability (new in 2.x) — tracing, prompt registry, and LLM evaluation.
pip install mlflow

Starting the Tracking Server & Logging Runs

Start the local MLflow UI with one command, then instrument your code:

Start tracking server

# Starts at http://localhost:5000
mlflow server --host 127.0.0.1 --port 5000

log_run.py — log params, metrics, artifacts

import mlflow

mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("my-llm-experiment")

with mlflow.start_run():
    # Log hyperparameters
    mlflow.log_param("model", "gpt-4o-mini")
    mlflow.log_param("temperature", 0.7)
    mlflow.log_param("max_tokens", 512)

    # Run your LLM call here...
    accuracy = 0.87
    latency_ms = 342

    # Log metrics
    mlflow.log_metric("accuracy", accuracy)
    mlflow.log_metric("latency_ms", latency_ms)

    # Log an artifact (e.g., the prompt template file)
    mlflow.log_artifact("prompt_template.txt")

MLflow Tracing for LLMs

MLflow 2.13+ includes automatic tracing for major LLM libraries. Enable it with one line before your first call:

autolog_tracing.py

import mlflow
import openai

mlflow.set_tracking_uri("http://localhost:5000")

# Auto-instrument OpenAI — logs every prompt, completion, token count
mlflow.openai.autolog()

# Now all OpenAI calls are automatically traced
client = openai.OpenAI()
response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "What is RAG?"}]
)
print(response.choices[0].message.content)
# Check http://localhost:5000 — you'll see the trace with timing + tokens

LangChain and LlamaIndex tracing

# LangChain auto-tracing
mlflow.langchain.autolog()

# LlamaIndex auto-tracing
mlflow.llama_index.autolog()

See also: LangChain guide · LlamaIndex guide · OpenAI API guide

LLM Evaluation with mlflow.evaluate()

mlflow.evaluate() runs LLM-as-judge metrics against a dataset of prompts and model outputs. Built-in metrics include answer correctness, faithfulness, toxicity, and relevance:

llm_eval.py

import mlflow
import pandas as pd

mlflow.set_tracking_uri("http://localhost:5000")

# Evaluation dataset
eval_data = pd.DataFrame({
    "inputs": ["What is MLflow?", "What is LangChain?"],
    "predictions": [
        "MLflow is an open-source MLOps platform.",
        "LangChain is a framework for building LLM apps.",
    ],
    "ground_truth": [
        "MLflow is an open-source platform for ML lifecycle management.",
        "LangChain is a Python and JavaScript framework for LLM pipelines.",
    ]
})

with mlflow.start_run():
    results = mlflow.evaluate(
        data=eval_data,
        targets="ground_truth",
        predictions="predictions",
        model_type="question-answering",
        evaluators="default",   # uses LLM-as-judge internally
    )
    print(results.metrics)

Model Registry

The MLflow Model Registry lets you version, annotate, and stage models for deployment:

model_registry.py

import mlflow.pyfunc

# Register a model from a run
model_uri = "runs:/<run-id>/model"
mlflow.register_model(model_uri, "my-rag-pipeline")

# Load a versioned model
model = mlflow.pyfunc.load_model("models:/my-rag-pipeline/1")
output = model.predict({"question": "What is RAG?"})

MLflow vs LangSmith vs W&B vs Langfuse

Tool Open source LLM tracing Classical ML Best for
MLflow Yes Yes (2.x) Yes Full MLOps, self-hosted
LangSmith No Yes No LangChain ecosystem
Weights & Biases Partial Yes (Weave) Yes Deep learning, training runs
Langfuse Yes Yes No LLM-only, self-hosted option

Also see: DSPy guide

Monitor the APIs Your MLflow Pipelines Call

MLflow traces are only half the picture — if OpenAI or Anthropic is down, your pipeline fails before MLflow even logs a trace. Prismix monitors LLM API uptime so you catch outages before your evals break.

Check LLM API Status →