/
LiteLLM User Guide Spring Updated v1

LiteLLM User Guide Spring Updated v1

 

API Access for the AI Sandbox

This document covers LiteLLM Release Version-v1.83.10 Stable

University of Oklahoma Libraries

Digital Scholarship & Data Services (DSDS)

Last Updated: May 2026

 

 

About This Guide

This guide is your reference for using LiteLLM, the API-based AI gateway provided by OU Libraries. LiteLLM lets you write code that talks to multiple AI models through a single, standardized interface. Whether you’re building a research pipeline, prototyping a chatbot, or teaching an AI workshop, this guide covers everything from your first API call to managing teams and budgets.

 

Who This Guide is For

This guide is designed for:

• Researchers automating AI workflows

• Students learning API-based AI tools

• Developers building applications

• Instructors creating AI-powered classroom activities

You should be comfortable with:

• Basic or intermediate Python or other programming languages.

• Running terminal commands

• Editing scripts

1. What Is LiteLLM?

LiteLLM is an open-source AI gateway that gives you a single, OpenAI-compatible API to call multiple large language models. Instead of learning a different SDK for every provider, you write your code once and swap models by changing a single string. The OU Libraries instance connects you to commercial models on AWS Bedrock and open-source models on the National Research Platform (NRP).

How Is It Different from LibreChat?

LibreChat is the no-code chat interface — you type and get responses. LiteLLM is the programmatic layer underneath: it is for users who want to write Python scripts, build applications, run batch jobs, or integrate AI into their research workflows through code.

 

LibreChat

LiteLLM

Interface

Chat window (no code)

API calls (Python, curl, etc.)

Best For

Exploring, brainstorming, drafting

Automation, pipelines, research apps

Skill Level

No coding required

Basic Python recommended

Key Feature

Switch models mid-chat

Virtual API keys, spend tracking, batch jobs

Key Features (v1.83.10)

  • OpenAI-compatible API — use the same code format regardless of which model you call

  • Virtual API keys — private keys used to authenticate your scripts for each project with individual spend tracking

  • Multi-threshold budget alerts — get notified at 75%, and ~95% of your spending limit

  • Concurrent budget windows — track daily and monthly spending simultaneously

  • Prompt compression — BM25-based trimming for long-context prompts before they reach the model

  • Multimodal support — send text, images, documents and code to models that support them

  • Team management — set rate limits, set quotas, and track usage across research groups

  • Per-team guardrail controls — teams can opt out of specific guardrails without admin config changes

  • Usage dashboard — monitor requests, tokens, and costs in real time

  • AWS Bedrock guardrails — basic content moderation and model access policies

2. Getting Started

LiteLLM requires basic familiarity with Python or command-line tools. If you’re new to APIs, the steps below will walk you through it.

Step 1: Request Access

  1. Email dsds@ou.edu to request a LiteLLM account or submit the AI Resources Request Form.

  2. You’ll be assigned to a Team specific to your project or use case.

  3. Log in to the LiteLLM dashboard: litellm.lib.ou.edu/ui

Step 2: Create a Virtual API Key

  1. In the left sidebar, click "Virtual Keys."

  2. Click "Create New Key."

  3. Select your Team, choose a model (or all available models), and name your key.

  4. Save the key immediately — you will not be able to see it again after this step.

Section of 'Virtual Keys' tab on LiteLLM Application

⚠️ Important: API Key Security

Treat your API key like a password. Do not share it, commit it to a public repository, or paste it into shared documents. Each user should have their own key for proper usage tracking.

Always load your key from an environment variable rather than hardcoding it. Use the following example to load the API keys.

 

# Set the variable in your terminal (bash/zsh): export LITELLM_API_KEY="sk-your-key-here" # Then in Python, load it securely: import os api_key = os.environ.get("LITELLM_API_KEY")

Step 3: Test Your Key

Use the "Test Key" option in the LiteLLM sidebar to verify your key works before writing code. Select either "Current UI Session" or "Virtual Key" as the source, then pick a model you assigned to the key.

Screenshot of Test Key section of LiteLLM Application.

Step 4: Set Up Your Environment

Install the OpenAI Python package (LiteLLM uses the same format):

pip install openai

You can work locally (terminal, VS Code, Jupyter Notebook) or use the OU Nautilus portal:

Note

You do not need to install the litellm Python package to use the OU-hosted service. The openai package is all you need. See Section 2 (Security Advisory) for important warnings about the litellm PyPI package.

 

Why We Use the OpenAI Library

The OU-hosted LiteLLM proxy exposes an OpenAI-compatible API, which means any client that can talk to OpenAI can talk to our proxy — you just point the ‘base_url’ to our endpoint instead of OpenAI's. The ‘openai’ Python package is lightweight, well-maintained, and handles this seamlessly.

 

Step 5: Make Your First API Call

Here’s a minimal Python script to send a prompt and print the response:

 

import os from openai import OpenAI # Create a client that points to OU's LiteLLM gateway client = OpenAI( api_key=os.environ.get("LITELLM_API_KEY"), # loaded from env var base_url="https://litellm.lib.ou.edu/v1" # OU endpoint, not ) # Send a chat-completion request response = client.chat.completions.create( model="anthropic.claude-3-haiku-20240307-v1:0", #model string from Section 5 messages=[ # Each message has a role ("system", "user", or "assistant") # and content (the actual text). {"role": "user", "content": "Explain the law of supply and demand in two sentences."} ] ) # The response object contains a list of choices; we grab the first one print(response.choices[0].message.content)

3. API Endpoints

All models — whether hosted on AWS Bedrock or the National Research Platform — are accessed through a single base URL: https://litellm.lib.ou.edu/v1. You never need to change this URL when switching between models.

All endpoints follow the OpenAI format: /v1/chat/completions, /v1/embeddings, etc. This means any tool or library built for the OpenAI API will work with LiteLLM by simply changing the base_url and api_key.

4. Available AI Models

LiteLLM connects to the same model set as LibreChat, accessible programmatically through API calls. AWS Bedrock models are commercial (funded by OU Libraries). NRP models are open-sourced and hosted by the National Research Platform.

Model

Provider

Type

Multimodal

Best For

Cost Tier

Notes

Claude Sonnet 4.5

AWS Bedrock

Commercial

Yes

Writing, analysis, coding, everyday use

$3.00 / $15.00 per 1M tokens (input / output)

Great all-rounder

Claude 3 Haiku

AWS Bedrock

Commercial

Yes

Quick answers, drafting, low-cost testing

$0.25 / $1.25 per 1M tokens (input / output)

Fast and efficient

Amazon Nova Lite

AWS Bedrock

Commercial

Yes

Lightweight tasks, summaries, testing

$0.06 / $0.24 per 1M tokens (input / output)

Amazon’s budget model

Qwen3 Coder 30B

AWS Bedrock

Open Source

Text

Code generation, debugging, programming

NRP*

Optimized for coding

Qwen3 32B

AWS Bedrock

Open Source

Text

Reasoning, analysis, research tasks

NRP*

Strong reasoning model

Qwen3

NRP

Open Source

Text

General chat, quick tasks, testing

NRP*

Lightweight Qwen variant

Kimi

NRP

Open Source

Text

Long documents, extended context tasks

NRP*

By Moonshot AI

Gemma 4

NRP

Open Source

Yes

General use, multimodal tasks

NRP*

By Google

Gemma 4 Small

NRP

Open Source

Text

Quick testing, lightweight tasks

NRP*

Compact Google model

GLM-4.7

NRP

Open Source

Text

Multilingual tasks, general reasoning

NRP*

By Zhipu AI

MiniMax M2

NRP

Open Source

Text

Creative writing, conversational tasks

NRP*

By MiniMax

 

About Claude 4.6 Opus

Opus 4.6 is the most capable (and most expensive) model available ($5/$25 per 1M tokens input / output). It is not enabled by default. To request access, contact dsds@ou.edu or fill in the request form, with your project description and use case.

 

AI Models Policy

AWS Bedrock prices shown are on-demand rates per million tokens (input / output) in standard US regions, sourced from the Amazon Bedrock pricing page. Pricing may change—please verify on the AWS pricing page before quoting figures elsewhere.

 

*The NRP is a community-owned research and education platform connecting researchers and educators to foster collaboration, accelerate innovation, and share resources. Supported by over 50 institutions, including leadership from UC San Diego, the University of Nebraska-Lincoln, and the Massachusetts Green High Performance Computing Center, the NRP provides access to cutting-edge technologies in AI, high-performance computing, data storage, and networking. Open to all nonprofit higher education institutions, from community colleges to top research universities, the NRP advances learning and scientific breakthroughs with support from the U.S. National Science Foundation, Department of Energy, and Department of Defense among others.

 

Use is governed by the NRP Acceptable Use Policy and the LLM fair-use limits: NRP Acceptable Use Policy • NRP LLM fair-use page.
NRP services are not approved for HIPAA, FERPA, PII, or other regulated data.

5. What Can You Build?

LiteLLM’s API access unlocks use cases that go far beyond what a chat interface can do. Here are real scenarios where OU users are putting it to work:

Scenario

Suggested Model

Audience

Batch-summarize 200 PDFs

Claude Sonnet 4.5

Researchers

Build a Q&A chatbot for a course

Claude 3 Haiku

Instructors

Compare model outputs for a study

Any / Multiple

Researchers

Generate synthetic training data

Qwen3 32B

Data Scientists

Automate code review feedback

Qwen3 Coder 30B

Students / Devs

Multilingual text classification

GLM-4.7 or Gemma 4

Researchers

Rapid prototyping in a workshop

Claude 3 Haiku

Instructors

Image analysis pipeline

Gemma 4 / Claude

Any

Long-context document analysis

Kimi

Researchers

Creative writing experiments

MiniMax M2

Students

Example: Batch-Summarize PDFs

This script reads every PDF in a folder, sends its text to an LLM, and writes a one-paragraph summary to a CSV file. It uses the PyPDF library for text extraction. Check out the example script for batch summarization of PDF files.

import os, csv from pathlib import Path from openai import OpenAI from pypdf import PdfReader client = OpenAI( api_key=os.environ.get("LITELLM_API_KEY"), base_url="https://litellm.lib.ou.edu/v1" ) pdf_folder = Path("./papers") results = [] for pdf_path in sorted(pdf_folder.glob("*.pdf")): # Extract text from the PDF reader = PdfReader(pdf_path) full_text = "\n".join(page.extract_text() or "" for page in reader.pages) # Truncate to ~10,000 characters to stay within token limits text_chunk = full_text[:10000] response = client.chat.completions.create( model="anthropic.claude-3.5-sonnet-v2", messages=[ {"role": "system", "content": "You are a research assistant. Summarize the following paper in one paragraph."}, {"role": "user", "content": text_chunk} ] ) summary = response.choices[0].message.content results.append({"file": pdf_path.name, "summary": summary}) print(f"Summarized: {pdf_path.name}") # Write results to CSV with open("summaries.csv", "w", newline="") as f: writer = csv.DictWriter(f, fieldnames=["file", "summary"]) writer.writeheader() writer.writerows(results) print(f"Done. {len(results)} summaries saved to summaries.csv")

Example: Simple Q&A Chatbot

This script creates a simple interactive chatbot that answers questions using context you provide (for example, course material). It maintains conversation history so the model remembers prior exchanges. Check out the example script for building a Econ 101 chatbot.

import os from openai import OpenAI client = OpenAI( api_key=os.environ.get("LITELLM_API_KEY"), base_url="https://litellm.lib.ou.edu/v1" ) # Provide context the chatbot should reference SYSTEM_PROMPT = """You are a helpful teaching assistant for ECON 101. Answer questions based on the following course material: - Supply: the quantity of a good that producers are willing to sell at a given price. - Demand: the quantity of a good that consumers are willing to buy at a given price. - Equilibrium: the price at which supply equals demand. If the student asks something outside this material, say so politely.""" messages = [{"role": "system", "content": SYSTEM_PROMPT}] print("ECON 101 Q&A Bot (type 'quit' to exit)") while True: user_input = input("You: ") if user_input.lower() in ("quit", "exit"): break messages.append({"role": "user", "content": user_input}) response = client.chat.completions.create( model="anthropic.claude-3-haiku-20240307-v1:0", messages=messages ) reply = response.choices[0].message.content messages.append({"role": "assistant", "content": reply}) print(f"Bot: {reply}\n")

6. Streaming Responses

By default, the API waits for the entire response to be generated before returning it. For interactive applications (chatbots, live demos, workshop prototyping), you can enable ‘streaming’ to receive the response token by token as it is generated. This dramatically improves perceived latency. Check out the example script for streaming responses.

import os from openai import OpenAI client = OpenAI( api_key=os.environ.get("LITELLM_API_KEY"), base_url="https://litellm.lib.ou.edu/v1" ) # Pass stream=True to get a generator of partial responses stream = client.chat.completions.create( model="anthropic.claude-3-haiku-20240307-v1:0", messages=[ {"role": "user", "content": "Write a haiku about data science."} ], stream=True # enables token-by-token streaming ) # Print each token as it arrives (no newline between tokens) for chunk in stream: token = chunk.choices[0].delta.content or "" print(token, end="", flush=True) print() # final newline

7. Error handling

When running batch jobs or production scripts, you will encounter rate limits, timeouts, and transient errors. Wrapping your API calls in a retry function prevents your entire job from failing a single hiccup. Check out the example script for error handling.

import os, time from openai import OpenAI, RateLimitError, APITimeoutError, APIError client = OpenAI( api_key=os.environ.get("LITELLM_API_KEY"), base_url="https://litellm.lib.ou.edu/v1" ) def call_with_retry(messages, model, max_retries=5): """Call the API with exponential backoff on transient errors.""" for attempt in range(max_retries): try: response = client.chat.completions.create( model=model, messages=messages ) return response.choices[0].message.content except RateLimitError: wait = 2 ** attempt # 1s, 2s, 4s, 8s, 16s print(f"Rate limited. Retrying in {wait}s...") time.sleep(wait) except APITimeoutError: print(f"Timeout on attempt {attempt + 1}. Retrying...") time.sleep(2) except APIError as e: print(f"API error: {e}. Retrying...") time.sleep(2) raise RuntimeError(f"Failed after {max_retries} retries") # Usage: result = call_with_retry( messages=[{"role": "user", "content": "Summarize this text..."}], model="anthropic.claude-3-haiku-20240307-v1:0" ) print(result)

8. Asynchronous and Concurrent Requests

If you need to process many items (for example, 200 PDFs), sending requests one at a time is slow. Python's asyncio library and the AsyncOpenAI client let you run multiple requests concurrently while respecting rate limits. Check out the example script for asynchronous and concurrent requests.

 

import os, asyncio from openai import AsyncOpenAI client = AsyncOpenAI( api_key=os.environ.get("LITELLM_API_KEY"), base_url="https://litellm.lib.ou.edu/v1" ) # Semaphore limits how many requests run at the same time SEM = asyncio.Semaphore(5) # adjust based on your rate limits async def summarize(text, filename): async with SEM: response = await client.chat.completions.create( model="anthropic.claude-3-haiku-20240307-v1:0", messages=[ {"role": "system", "content": "Summarize in one paragraph."}, {"role": "user", "content": text[:10000]} ] ) return {"file": filename, "summary": response.choices[0].message.content} async def main(): # Build a list of tasks (one per document) tasks = [ summarize("Full text of document 1...", "doc1.pdf"), summarize("Full text of document 2...", "doc2.pdf"), # ... add more tasks as needed ] results = await asyncio.gather(*tasks) for r in results: print(f"{r["file"]}: {r["summary"][:80]}...") asyncio.run(main())

Tip

Start with a semaphore of 5 and increase gradually. If you start seeing rate-limit errors, reduce the concurrency. For very large jobs (500+ items), contact DSDS in advance to arrange a temporary rate-limit increase.

9. Teams, Budgets & Rate Limits

Teams

When your account is created, you’re assigned to a Team. Teams let groups of users share API key quotas, track collective usage, and control which models members can access.

Role

Permissions

Admin

Add members, control permissions, set model access, view team spend logs

User

Use shared resources and keys, view personal usage — cannot modify settings

Budgets

As part of the UL pilot program, each user is seeded with a $25 spending limit. Admins will email your team when you reach approximately 75% of your budget and again when you are close to 100%. Once the limit is reached, API calls will fail with a "Budget exceeded" error (see Section 13: Troubleshooting).

Visit the AI Sandbox Project page for full details.

Rate Limits

Rate limits exist to ensure fair access and control costs. They apply at both the user and team level, and expensive commercial models have stricter limits than open-source alternatives.

  • Check your usage dashboard regularly (left sidebar → Usage)

  • Use lighter models (Haiku, Nova Lite, or NRP models) for testing and iteration

  • Request limit increases early if you’re planning a large batch job (Contact admins)

10. Best Practices

For Efficient Usage

  • Start with simple test prompts before scaling to batch jobs

  • Use the smallest model that meets your needs — Nova lite, Haiku or Qwen3 for testing, GPT-OSS or Sonnet for production

  • Monitor your usage dashboard to stay within budget

For Research Projects

  • Estimate your token needs before starting large analyses or contact admins for larger usage.

  • Document which models, parameters, and prompts you use for reproducibility

  • Version-control your scripts and configurations

  • Validate findings by cross-checking with different models

For Classroom Use

  • Contact DSDS before onboarding a class (instructor) — we can set up dedicated teams and quotas

  • Share working code templates with students to reduce setup friction

  • Start out with low-cost models and pivot to specific models (AWS, NRP models) to support coursework demands

  • Monitor team usage during active projects

11. Privacy & Data Guidelines

LiteLLM is hosted within the OU Libraries’ AWS environment. Prompts and responses may be logged in the system for debugging, but follow these guidelines:

 

  • Do not send FERPA-protected student data or HIPAA-protected health information

  • Do not include passwords, SSNs, or personal identification in prompts

  • Do not share your API key — each user needs their own for proper tracking

  • Use this service for educational and research purposes only (not commercial)

13. Troubleshooting

Error

Cause

Fix

Rate limit exceeded

Too many requests in a short window

Wait for the quota reset, check Usage dashboard, or request higher limits. See Section 8 for retry logic.

Model not found

Incorrect model string

Check the model table in Section 5; copy the exact model's name including version strings.

Invalid API key

Key is wrong, expired, or revoked

Verify your key in the dashboard. Regenerate if needed. Do not share keys.

Budget exceeded

Spending limit reached

Email dsds@ou.edu with project details to request more credits.

Slow responses

High server load or large prompt

Try a lighter model, reduce prompt size, or use prompt compression.

Connection refused

Wrong base URL or network issue

Verify you are using https://litellm.lib.ou.edu/v1 as the base URL.

12. Frequently Asked Questions

Do I need coding skills?

Yes, basic Python is preferred. If you’re not comfortable with code, start with LibreChat (the chat interface) instead.

 


Can I use this for commercial projects?

No. LiteLLM is strictly for OU educational and research use.

 

How long do API keys last?

By default, keys remain active unless revoked. You can set expiration dates during key creation for added security. For best practices, delete the unused key(s)

 

Can I share my API key?

No. Each user should have their own key for proper usage of tracking and accountability.

 

What happens to my data?

Prompts and responses may be stored temporarily for logging and debugging. No personal, FERPA, or HIPAA data is allowed. This sandbox is for general, non-sensitive research and learning use only.

 

Can I use this for thesis research?

Yes, but follow your department’s research guidelines and cite AI assistance appropriately. Please reach out to the DSDS team ahead of using this for your research work, as this is a pilot project and is not guaranteed beyond the available pilot funding.

 

How do I get Claude 4.6 Opus access?

Opus is available on request due to its higher cost. Email dsds@ou.edu with your project description, estimated token usage, and timeline.

 

Can I request more tokens or higher limits?

Yes, subject to pilot program funding and approval. Email dsds@ou.edu with your current project description, storage requirements, and timeline.

 

Can I run LiteLLM on my own servers?

This guide covers the OU-hosted instance.
Yes, you can run the LiteLLM instance. LiteLLM is open-source and can be self-hosted — see docs.litellm.ai for details.

 

13. Security Advisory

⚠️ Critical: LiteLLM PyPI Package Vulnerabilities

Recent supply-chain attacks have targeted the litellm package on PyPI. Before installing any version of the litellm Python package on your own machine, verify you are using a known-safe version.

For most users of the OU-hosted LiteLLM instance, you do not need the litellm package at all. You only need the openai package (see Section 3). The litellm package is only required if you are self-hosting your own LiteLLM proxy.
Versions:
litellm==1.82.7 (payload in proxy_server.py, triggers on import) and litellm==1.82.8 (malicious .pth file, triggers on any Python startup — the more dangerous one)

Related security disclosures:

Security Update — March 2026

CVE-2026-42208 — SQL Injection in LiteLLM Proxy

14. Get Help & Share Feedback

Your feedback shapes the AI Sandbox. Whether you have questions, hit bugs, or want new features, reach out.

 

When Reporting an Issue, Include:

  • The exact error message (copy and paste)

  • The model's name you were using

  • Steps you took before the error

  • Your username/email and the time the error occurred

Contact

Primary Support: dsds@ou.edu

Technical Contact: varunsaya@ou.edu

15. Useful Links