• L
• T
• F
• R
• E

Large language models (LLMs) are revolutionizing the financial trading landscape by enabling sophisticated analysis of vast amounts of unstructured data to generate actionable trading insights. These advanced AI systems can process financial news, social media sentiment, earnings reports, and market data to predict stock price movements and automate investment strategies with unprecedented accuracy. 

The Strategic Technology Analysis Center (STAC) has been developing benchmarks for the workloads key to the financial industry for over 15 years. They have developed the STAC-AI benchmark to help companies assess the end-to-end retrieval-augmented generation (RAG) and LLM inference pipeline.

This post presents the results achieved on the STAC-AI LANG6 benchmark across multiple NVIDIA platforms. We will also share some recommendations on how any user can benchmark NVIDIA TensorRT LLM according to the specifications of their dataset.

STAC-AI LANG6 (Inference-Only) Benchmark

In the broader context of a RAG pipeline, STAC-AI LANG6 is the part of the benchmark focusing on LLM inference performance. The benchmark tests the hardware and software stack on the Llama 3.1 8B Instruct and Llama 3.1 70B Instruct models in combination with the following custom datasets: 

• EDGAR4: The prompts are summarizations of the relationship of a company to one of various physical and financial concepts (such as commodities, currencies, interest rates, and real estate sectors). It uses EDGAR 10‑K paragraphs from a single security filing for a single year. The input/output sequence length aims to model medium-length requests.
• EDGAR5: Questions covering several different aspects of a complete 10‑K filing. The document type is the complete text of a single EDGAR 10‑K filing. The input/output sequence length aims to model long-context requests.

These datasets, based on EDGAR filings, model medium and long-context summarization for financial trading and investment advice use cases. The prompts ask the model to perform analysis and summarization of annual reports (10-K filings) for thousands of public companies over the past five years.

The benchmark also tests two different inference scenarios, batch mode and interactive mode: 

• Batch (offline) mode: All requests are given at once, and all responses are collected at once. Only throughput is measured. 
• Interactive (online) mode: Requests arrive at pseudo-random times. The mean arrival rate λ (the average number of requests the system receives every second) can be set to model different usage scenarios. The benchmark collects metrics such as reaction time (RT), total words per second (WPS), and Output Rate (WPS/user), but does not set any constraint on them. RT is analogous to time to first token (TTFT) in other benchmarks, and Output Rate to words/second/user.

Note that interactive mode does not cover the combination of Llama 3.1 70B Instruct with EDGAR5.

The benchmark checks the quality of the output and word count with respect to a control set of LLM-generated responses. 

While other benchmarks allow all preprocessing, an important differentiator of STAC-AI is the need to apply chat templates and tokenize requests during inference. Real deployments may prefer to have this work done on the server side to protect their system prompts, thus imposing more load on the CPU.

Hardware and software stack

This post highlights STAC-AI audits run for an on-premises NVIDIA Hopper-based server submitted by HPE, an on-premises NVIDIA RTX PRO 6000 Blackwell Server Edition system submitted by Supermicro and Red Hat, and NVIDIA HGX B200 on Lambda.

• The HPE ProLiant Compute DL384 Gen12, powered by the NVIDIA GH200 Grace Hopper Superchip, provides an efficient single-server solution. To see detailed results, refer to the STAC report on HPE ProLiant DL384 Gen12 server with two NVIDIA GH200 NVL2 Superchips.
• A cloud-based instance provided by Lambda, based on NVIDIA HGX B200. The system uses eight NVIDIA Blackwell B200 GPUs in an HGX platform, connected with NVIDIA NVLink and NVIDIA NVSwitch for high-speed GPU-to-GPU communication. Each NVIDIA Blackwell B200 GPU includes 180 GB of HBM3e memory and 8 TB/s of memory bandwidth for large-model inference. For detailed benchmark results, see the Llama 3.1 8B and Llama 3.1 70B companion STAC report on Lambda 1-Click Cluster Cloud Instance with NVIDIA B200 SXM6 Blackwell Series GPUs.
• Another on‑premises option is the Supermicro AS -5126GS-TNRT in the two NVIDIA RTX PRO 6000 Blackwell Server Edition configuration, which pairs two Blackwell GPUs in a single server for AI development and deployment. Each RTX PRO 6000 Blackwell GPU includes 96 GB of memory, supplying the node with substantial aggregate GPU memory for larger models, larger batch sizes, or more concurrent jobs within the same system footprint. For details about the results, see the STAC report on Supermicro SuperServer SYS-222C-TN with two NVIDIA RTX PRO 6000 Blackwell Series GPUs. The full stack was deployed on Red Hat OpenShift, demonstrating the containerized Kubernetes platform introduces no measurable overhead for GPU-intensive LLM inference workloads.  

As the benchmark requires post-training quantization as part of the benchmarking procedure, the models were quantized using NVIDIA TensorRT Model Optimizer. To leverage the most performant kernels available for each deployment, quantization was performed to FP8 on NVIDIA Hopper and to NVFP4 on NVIDIA Blackwell. 

To achieve the best performance for both Hopper and Blackwell, TensorRT LLM inference framework was used for efficient model execution. These quantized models were run using TensorRT LLM PyTorch runtime for a familiar, native PyTorch development experience while maintaining peak performance. 

Benchmarking results on STAC-AI LANG6

Benchmarking results for both batch mode and interactive mode are detailed in this section.

Batch mode

For batch mode, NVIDIA Blackwell delivers significant speedups in all scenarios. Table 1 shows the WPS and requests per second (RPS) achieved. 

The full reports with more details across both interactive and batch modes can be found in the reports published by STAC.

Single-GPU performance was also derived to account for the different number of GPUs on each system. Although STAC-AI does not measure per-GPU performance, the results shown in Figure 1 illustrate the throughput difference between single GPUs from each of the systems. 

Interactive mode 

The balance between token economics (dependent on throughput) and user experience (dependent on interactivity metrics such as RT and WPS/user) is a crucial factor in modern LLM inference.

Interactive mode showcases the tradeoff across the interactivity-throughput Pareto front by selecting a range of arrival rates. Interactivity is measured by both RT and WPS/user. To facilitate visualization, the inverse of WPS/user, defined as interword latency (IWL), or \(\frac{1}{WPS/user}\), is used. In the graphs we use the 95th percentile of both metrics.

As seen in Figure 2, the NVIDIA HGX B200 system achieves a better tradeoff between throughput and both RT and IWL across the board. IWL (solid, lower is better) and RT (dashed, lower is better) are plotted versus interactive-mode throughput across model/dataset scenarios.

How to benchmark TensorRT LLM with your custom data

While the STAC benchmark uses proprietary data and metrics, you can benchmark TensorRT LLM against models tailored to your specific dataset characteristics. This tutorial walks you through quantizing a model, preparing your dataset, and running performance benchmarks—all customized for your use case.

Prerequisites:

• A Docker image that includes TensorRT LLM (TensorRT LLM Release, for example).
• An NVIDIA GPU that is large enough to serve your model at the desired quantization level. You can find a support matrix for quantization in TensorRT LLM documentation.
• A Hugging Face account and token, along with access to the gated models of Llama 3.1 8B Instruct or Llama 3.1 70B Instruct. You can set the HF_TOKEN environment variable to your token, and all subsequent commands will use this token.

Step 1: Launch the container

The containers maintained by NVIDIA contain all of the required dependencies pre-installed. Change into an empty directory with enough space for the models and their quantizations. You can start the container on a machine with NVIDIA GPUs with the following command. Make sure you specify your Hugging Face token.

docker run-it --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 \
            --gpus=all \
            -u $(id -u):$(id -g) \
            -e USER=$(id -un) \
            -e HOME=/tmp \
            -e TRITON_CACHE_DIR=/tmp/.triton \
            -e TORCHINDUCTOR_CACHE_DIR=/tmp/.inductor_cache \
            -e HF_HOME=/workspace/model_cache \
            -e HF_TOKEN=<your_huggingface_token> \
            --volume "$(pwd)":/workspace \
            --workdir /workspace \
            nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc2

Step 2: Clone the repositories

Model quantization reduces model size and improves inference speed. Use NVIDIA Model Optimizer to quantize Llama 3.1 8B Instruct to NVFP4 format. First, clone the Model Optimizer repository for the quantization example:

git clone https://github.com/NVIDIA/TensorRT-Model-Optimizer.git -b 0.37.0

Step 3: Quantize the model

Next, execute the Hugging Face example script with the chosen model and quantization format—in this case, Llama 3.1 8B Instruct using NVFP4 quantization.

bash TensorRT-Model-Optimizer/examples/llm_ptq/scripts/huggingface_example.sh \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --quant nvfp4

Step 4: Generate synthetic data

Use the benchmark utility to generate a synthetic dataset with the token distribution needed for a task. This example creates 30,000 requests with a fixed input sequence length of 2,048, and an output sequence length of 128. Nonzero standard deviations better approximate real traffic, if you have access to that information.

trtllm-bench \
  --model meta-llama/Llama-3.1-8B-Instruct \
  prepare-dataset \
  --output dataset_2048_128.json \
  token-norm-dist \
  --input-mean 2048 \
  --output-mean 128 \
  --input-stdev 0 \
  --output-stdev 0 \
  --num-requests 30000

Step 5: Run the benchmark

The trt-llm bench command can run the generated requests in an offline fashion, sending all requests at once to TensorRT LLM runtime (closely matching STAC-AI’s batch mode). 

While some options are available in the CLI API, the full LLM API can be accessed through a YAML file passed with the extra_llm_api_options parameter. For the purposes of this example, enable CUDA Graphs padding. To learn about more options, see the TensorRT LLM API Reference.

cat > llm_options.yml << 'EOF'
cuda_graph_config:
  enable_padding: True
EOF

Finally, run the benchmark, specifying the model, the dataset, and the options:

trtllm-bench \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --model_path /workspace/TensorRT-Model-Optimizer/examples/llm_ptq/saved_models_Llama-3_1-8B-Instruct_nvfp4 \
  throughput \
  --dataset dataset_2048_128.json \
  --backend pytorch \
  --extra_llm_api_options llm_options.yml

This will output various metrics such as the request throughput, the tokens/second/GPU, and more.

Get started with TensorRT LLM benchmarking

NVIDIA HGX B200 on Lambda significantly advanced performance on the STAC-AI LANG6 benchmark for LLM inference in financial services. NVIDIA Blackwell delivered up to 2.8x the performance of previous architectures, achieving both higher throughput and consistently maintaining superior interactivity.

The NVIDIA RTX PRO 6000 Blackwell results highlight the flexibility of the Blackwell platform. Running on Red Hat OpenShift, the two-GPU Supermicro system delivered competitive LLM inference performance. This means organizations can right-size their deployment, from a space- and cost-efficient server to a full-scale data center node, maintaining the performance benefits of Blackwell NVFP4 precision.

Alongside the new record, NVIDIA Hopper continues to deliver strong, valuable results for LLM inference workloads. Even more than three years after its initial release, Hopper proves highly effective in both batch and interactive inference scenarios, maintaining good performance metrics even at high throughput, and confirming its continued relevance for financial institutions.

To set up and run your own performance evaluations, explore the TensorRT LLM Benchmarking Guide.