Comprehensive LiteLLM Configuration Guide (config.yaml with all options included)

LiteLLM is most feature-rich proxy but when it comes to write great docs, they are failing a lot! I've used my research agents to bring all relevant information about config.yaml of LiteLLM to make it comprehensive guide that can be useful for most of people in here. Also useful for copy-paste and discuss with AIs like o1-pro while building config files. If you need quick copy paste to tools like Claude / Thinkbuddy to discuss that, visit my Gist and click 'Raw' button to make it easy to copy paste: https://gist.github.com/yigitkonur/4053c1e31f9351da1d91a0ca410442fd

Table of Contents

• Understanding config.yaml Structure
• environment_variables Section: Defining Variables within YAML
• model_list Section: Defining Models and Deployments
  • Model List Parameter Summary
  • model_name
  • litellm_params
    • model
    • api_key
    • api_base
    • Provider-Specific Settings (within litellm_params)
    • temperature
    • max_tokens
    • rpm
    • tpm
  • model_info
    • id
    • mode
    • input_cost_per_token
    • output_cost_per_token
    • max_tokens
    • base_model
    • metadata
    • Additional Fields (within model_info)
• litellm_settings Section: Configuring the LiteLLM Library
  • litellm_settings Parameter Summary
  • Logging & Callback Settings
    • success_callback
    • failure_callback
    • callbacks
    • service_callbacks
    • turn_off_message_logging
    • redact_user_api_key_info
    • langfuse_default_tags
    • modify_params
    • enable_preview_features
    • set_verbose
    • json_logs
  • Networking & Timeout Settings
    • request_timeout
    • force_ipv4
    • forward_traceparent_to_llm_provider
  • Fallback & Reliability Settings
    • default_fallbacks
    • content_policy_fallbacks
    • context_window_fallbacks
    • disable_end_user_cost_tracking
    • disable_end_user_cost_tracking_prometheus_only
    • key_generation_settings (Enterprise Feature)
    • disable_add_transform_inline_image_block
    • disable_hf_tokenizer_download
  • Caching Settings
    • cache
    • cache_params
      • type
      • Redis-specific host
      • Redis-specific port
      • Redis-specific password
      • Redis-specific namespace
      • Redis-specific redis_startup_nodes
      • Redis-specific service_name
      • Redis-specific sentinel_nodes
      • S3-specific s3_bucket_name
      • S3-specific s3_region_name
      • S3-specific s3_aws_access_key_id
      • S3-specific s3_aws_secret_access_key
      • S3-specific s3_endpoint_url
      • Qdrant Specific (Semantic Cache) qdrant_semantic_cache_embedding_model
      • Qdrant Specific (Semantic Cache) qdrant_collection_name
      • Qdrant Specific (Semantic Cache) qdrant_quantization_config
      • Qdrant Specific (Semantic Cache) similarity_threshold
      • Cache Params applies to all - supported_call_types
      • Cache Params applies to all - mode
      • Cache Params applies to all - ttl
  • Callback Settings (callback_settings)
    • callback_settings.otel.message_logging
• general_settings Section: Proxy General Settings
  • general_settings Parameter Summary
  • Default Model Selection
    • completion_model
    • embedding_model
    • image_generation_model
    • moderation_model
    • infer_model_from_keys
  • Master Key & Authentication
    • master_key
    • enable_jwt_auth (Enterprise Feature)
    • litellm_jwtauth (Enterprise Feature)
    • allowed_routes
    • admin_only_routes (Enterprise Feature)
    • allowed_ips
    • enforce_user_param
    • enable_oauth2_proxy_auth (Enterprise Feature)
    • use_x_forwarded_for
    • custom_auth (Enterprise Feature or Advanced Usage)
    • allow_user_auth (Deprecated)
    • custom_sso (Enterprise Feature or Advanced Usage)
  • Database & Persistence
    • database_url
    • database_connection_pool_limit
    • database_connection_timeout
    • allow_requests_on_db_unavailable
    • disable_spend_logs
    • disable_adding_master_key_hash_to_db
    • store_model_in_db (Enterprise Feature)
    • store_prompts_in_spend_logs
    • disable_prisma_schema_update (Advanced Usage)
    • proxy_batch_write_at
    • proxy_budget_rescheduler_min_time
    • proxy_budget_rescheduler_max_time
  • Key Management & Encryption
    • key_management_system (Enterprise Feature)
    • key_management_settings (Enterprise Feature)
    • use_azure_key_vault (Enterprise Feature)
    • use_google_kms (Enterprise Feature)
    • default_team_disabled (Enterprise Feature)
    • custom_key_generate (Advanced Usage)
    • Encryption Salt (Environment Variable)
  • Rate Limiting & Quotas
    • max_parallel_requests
    • global_max_parallel_requests
    • max_request_size_mb
    • max_response_size_mb
    • proxy_budget_rescheduler_min_time
    • proxy_budget_rescheduler_max_time
  • Monitoring, Alerting & Health Checks
    • background_health_checks
    • health_check_interval
    • health_check_details
    • alerting
    • alerting_threshold
    • alerting_args
    • alert_types
    • alert_to_webhook_url
    • spend_report_frequency (Enterprise Feature)
    • forward_openai_org_id
    • forward_client_headers_to_llm_api
    • use_client_credentials_pass_through_routes (Enterprise Feature)
    • allow_client_side_credentials (Use with Caution)
  • Other Miscellaneous Settings
    • service_account_settings (Enterprise Feature)
    • provider_budget_config (Advanced Feature)
    • model_group_alias
    • retry_after (Advanced Feature)
    • num_retries
• router_settings Section: Routing & Load-Balancing Settings
  • router_settings Parameter Summary
  • Routing Strategy & Model Selection
    • routing_strategy
    • model_group_alias
    • default_litellm_params
    • default_max_parallel_requests
    • default_priority
    • polling_interval
    • caching_groups
    • assistants_config (Advanced/Enterprise Feature)
  • Multi-Instance Coordination & Scaling
    • redis_host
    • redis_port
    • redis_password
    • redis_url
    • client_ttl
    • cache_responses
    • routing_strategy_args
  • Pre-Call Checks & Validation
    • enable_pre_call_checks
    • optional_pre_call_checks
  • Failover & Retry Policies
    • allowed_fails
    • cooldown_time
    • disable_cooldowns
    • retry_policy
    • allowed_fails_policy
    • fallbacks
    • content_policy_fallbacks
    • default_fallbacks
    • max_fallbacks
    • num_retries
    • model_group_retry_policy (SDK-only/advanced)
  • Timeouts & Debugging
    • timeout
    • stream_timeout
    • debug_level
    • set_verbose (Deprecated)

Understanding config.yaml Structure

The config.yaml file serves as the central configuration hub for the LiteLLM Proxy Server. It's a YAML-formatted file that consolidates all the settings needed to define how your proxy will operate, including: which LLM models you'll use, how to connect to them, how to route requests, how to manage authentication, and how to customize various aspects of the proxy's behavior. A well-structured and understood config.yaml is essential for a functional and secure LiteLLM Proxy deployment.

The config.yaml file is organized into several top-level sections, each responsible for a specific area of configuration. These sections are:

• environment_variables (Optional): This section allows you to define environment variables directly within the config.yaml file. This can be convenient for managing credentials and endpoints within the configuration itself, especially in deployment environments like Kubernetes, where managing external environment variables might be more complex.

• model_list (Required): This is a crucial section that defines all the LLMs and model deployments that your LiteLLM Proxy will manage and serve. It's where you specify which models you want to use, their backend configurations (provider, API keys, endpoints, etc.), and any associated metadata like cost information or context window limits.

• litellm_settings: This section contains general settings that configure the behavior of the core LiteLLM Python library within the proxy. These are settings that apply broadly across the proxy, influencing things like logging, timeouts, fallback mechanisms, and caching behavior.

• callback_settings (Optional): This section lets you fine-tune the behavior of specific logging and monitoring callbacks that you've enabled in litellm_settings. For example, you can use it to control message logging for the OpenTelemetry (otel) callback.

• general_settings: This section encompasses proxy server-level settings. These are configurations that apply to the proxy server as a whole, including settings for the master key, database connections, authentication methods (like JWT or OAuth2), global rate limits, alerting, and various feature flags.

• router_settings: This section focuses on the proxy's routing and request handling configurations. It includes settings for choosing the load balancing strategy (e.g., round-robin, least-busy, latency-based), defining retry policies, setting concurrency limits, and configuring how fallbacks to alternative models should work.

Key Principles to Keep in Mind:

• YAML Format: The config.yaml file must be a valid YAML file. YAML is a human-readable data serialization format that uses indentation (spaces, not tabs) to define structure.
• Environment Variable Overrides: Most configuration values in config.yaml can also be set via environment variables. If a setting is defined both in config.yaml and as an environment variable, the YAML setting typically takes precedence (except where noted in this guide).
• Precedence: Settings defined in router_settings generally override similar settings in litellm_settings for router-specific operations. This allows for fine-grained control over the router's behavior.
• Restart Required: Changes to config.yaml generally require restarting the LiteLLM Proxy Server for the changes to take effect.

environment_variables Section: Defining Variables within YAML

Type: Mapping (String to String)

Optional: Yes

This section provides a way to define environment variables directly within your config.yaml file. While you can always set environment variables externally in your operating system or deployment environment, the environment_variables section offers some convenience, especially in certain contexts:

• Centralized Configuration: It allows you to keep all your configuration settings, including environment variables, in a single, centralized config.yaml file.
• Kubernetes Deployments: This is particularly helpful in Kubernetes deployments, where you might prefer to define environment variables within your configuration files (e.g., ConfigMaps or Secrets) rather than setting them separately in the Kubernetes environment.
• Simplified Management: For some deployments, managing environment variables within the config.yaml can simplify setup and configuration.

Each entry under environment_variables is a key-value pair:

• Key: The name of the environment variable (e.g., REDIS_HOST, OPENAI_API_KEY).
• Value: The string value to be assigned to that environment variable.

Example:

environment_variables:
  REDIS_HOST: "cache.example.com"
  REDIS_PORT: "6379"
  OPENAI_API_KEY: "sk-..."

In this example, the LiteLLM Proxy will operate as if the environment variables REDIS_HOST, REDIS_PORT, and OPENAI_API_KEY were set externally in the system environment.

Referencing Defined Environment Variables:

Within other parts of your config.yaml file, you can reference these defined environment variables using the syntax: "os.environ/VAR_NAME". This tells LiteLLM to fetch the actual value from the environment (either defined in the environment_variables section or externally).

Example:

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: openai/gpt-3.5-turbo
      api_key: "os.environ/OPENAI_API_KEY" # Referencing the OpenAI API key defined in environment_variables

Here, "os.environ/OPENAI_API_KEY" will be replaced with the value of the OPENAI_API_KEY environment variable (either sk-... from the example above, or a value set externally if one exists).

Note: If an environment variable is defined both in the environment_variables section and externally in your system environment, the YAML setting (within config.yaml) typically takes precedence.

model_list Section: Defining Models and Deployments

Type: List of Mappings

Required: Yes

The model_list section is mandatory and forms the core of your LiteLLM Proxy configuration. It's where you define all the LLMs and model deployments that your proxy will manage and make accessible to your applications. Each entry in the model_list represents a single model or model alias that the proxy will serve.

Structure of a model_list Entry:

Each entry in the model_list is a mapping (dictionary) that contains the following key components:

• model_name (String, Required): This is the user-friendly alias or name that client applications will use to refer to this specific model deployment when making API requests to the proxy. This is the name your application code will use.
• litellm_params (Object, Required): This is a nested mapping (object) that contains the core parameters that the LiteLLM library will use to actually interact with the backend LLM API. This is where you specify the actual model identifier (provider and model name), API keys, base URLs, and other provider-specific settings.
• model_info (Object, Optional): This is an optional nested mapping that allows you to provide metadata and descriptive attributes about the model, such as cost per token, context window size, and a description. While optional, it's highly recommended to provide this information for monitoring, cost tracking, and management purposes.

Example model_list Section:

model_list:
  - model_name: gpt-3.5-turbo-azure # User-facing alias: "gpt-3.5-turbo-azure"
    litellm_params:
      model: azure/my-deployment-name # Actual Azure OpenAI deployment name
      api_base: "https://my-azure-endpoint.openai.azure.com/" # Azure endpoint
      api_key: "os.environ/AZURE_API_KEY" # API Key (loaded from environment variable)
      api_version: "2023-07-01-preview" # Azure API version
      temperature: 0.7 # Default temperature
    model_info:
      max_tokens: 4096 # Context window size
      input_cost_per_token: 0.0000015  # Input cost per 1k tokens
      output_cost_per_token: 0.0000020 # Output cost per 1k tokens
      mode: "completion"  # Model type
      metadata: "Azure OpenAI GPT-3.5 Turbo deployment in EU region" # Description

  - model_name: llama2-chat-local # User-facing alias: "llama2-chat-local"
    litellm_params:
      model: openai/llama-2-13b-chat-hf # Using OpenAI-compatible interface
      api_base: "http://localhost:8000/v1" # Local server URL
      api_key: "none" # No API key needed for local model
    model_info:
      id: "llama2-chat-13b-local" # Unique ID (optional but recommended)
      max_tokens: 4096
      mode: "completion"
      input_cost_per_token: 0 # No cost for local model (example)
      output_cost_per_token: 0

This example defines two model deployments:

1. gpt-3.5-turbo-azure: An alias for an Azure OpenAI deployment of GPT-3.5 Turbo. The litellm_params specify the necessary details to connect to the Azure OpenAI service, including the deployment name, API base URL, API key (loaded from an environment variable), and API version. The model_info section provides metadata like the context window size (max_tokens) and estimated costs per token.
2. llama2-chat-local: An alias for a locally served LLaMA 2 model (presumably using an OpenAI-compatible interface like vLLM). The litellm_params use the openai/ prefix to indicate an OpenAI-compatible API, and the api_base points to a local server URL. No API key is needed (api_key: "none").

Key Concepts within model_list:

• Model Aliases: The model_name acts as an alias. Your client applications use this alias when making API requests, without needing to know the specific backend provider or deployment details. This abstraction allows you to change the backend model without modifying your application code, as long as the model_name remains consistent.
• Provider Routing: The litellm_params.model value (e.g., "azure/...", "openai/...", "anthropic/...") tells LiteLLM which provider's API to use for this model. The proxy uses this to route requests to the correct backend.
• Environment Variables: It's a best practice to use environment variables (referenced via "os.environ/VAR_NAME") to store sensitive information like API keys, rather than hardcoding them directly in the config.yaml file.
• model_info for Transparency: The model_info section is highly recommended for transparency and manageability. It provides valuable metadata that can be used for monitoring, cost tracking, and displaying model information in the proxy's Admin UI (if enabled).

Model List Parameter Summary

model_name

YAML Key: model_name

Type: String

Environment Variable: N/A

Default Value: N/A (Required Parameter)

Description: The model_name parameter defines the user-facing alias or name of a model deployment within your LiteLLM Proxy configuration. This is the name that client applications will use when making API requests to the proxy. It acts as a logical identifier, allowing you to abstract away the complexities of backend model names and provider-specific details. Each model_name within the model_list must be unique.

Example YAML:

model_list:
  - model_name: gpt-3.5-turbo # User-facing alias
    litellm_params:
      model: azure/gpt-turbo-small-eu # Backend model details
      # ... other litellm_params ...

Example Environment Variable: N/A (YAML-only parameter)

litellm_params

YAML Key: litellm_params

Type: Object (Mapping)

Environment Variable: N/A

Default Value: N/A (Required Parameter)

Description: The litellm_params section is a required mapping within each model_list entry. It contains the core configuration parameters that the LiteLLM library uses to interact with the actual backend LLM API. These parameters dictate how LiteLLM will call the underlying model provider. At a minimum, it must include the model parameter, but can include a wide range of settings to customize API calls.

Example YAML:

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params: # Defining litellm_params for this model
      model: azure/gpt-turbo-small-eu
      api_base: "https://my-azure-endpoint.openai.azure.com/"
      api_key: "os.environ/AZURE_API_KEY_EU"
      # ... other litellm_params ...

Example Environment Variable: N/A (YAML-only section header)

model (litellm_params)

YAML Key: model

Type: String

Environment Variable: N/A

Default Value: N/A (Required Parameter within litellm_params)

Description: The model parameter, nested within litellm_params, is required and specifies the actual backend model identifier. This string tells LiteLLM which LLM provider to use and which specific model from that provider to target. The format of this string is provider-specific, and typically follows the pattern "provider_name/model_identifier". For example:

• "openai/gpt-3.5-turbo": Specifies OpenAI's GPT-3.5 Turbo model.
• "azure/gpt-turbo-small-eu": Specifies an Azure OpenAI deployment named "gpt-turbo-small-eu".
• "anthropic/claude-instant-1": Specifies Anthropic's Claude Instant-1 model.

Example YAML:

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: azure/gpt-turbo-small-eu # Specifying the backend model (Azure OpenAI)
      # ... other litellm_params ...

Example Environment Variable: N/A (YAML-only parameter)

api_key (litellm_params)

YAML Key: api_key

Type: String

Environment Variable: Corresponding environment variable depends on the provider (e.g., OPENAI_API_KEY, ANTHROPIC_API_KEY, AZURE_API_KEY, etc.)

Default Value: N/A (Required for most providers)

Description: The api_key parameter, within litellm_params, is used to provide the API key or authentication token required to access the backend LLM provider's API. Whether this parameter is strictly required depends on the specific provider and model. Many commercial providers (like OpenAI, Anthropic, Azure) require an API key for authentication. It's highly recommended to not hardcode API keys directly in your config.yaml file. Instead, use environment variable references like "os.environ/API_KEY_NAME" to fetch the key from environment variables for better security and easier management.

Example YAML:

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: openai/gpt-3.5-turbo
      api_key: "os.environ/OPENAI_API_KEY" # Referencing OpenAI API key from environment variable
      # ... other litellm_params ...

Example Environment Variable:

export OPENAI_API_KEY="sk-..." # Setting OpenAI API key in environment

api_base (litellm_params)

YAML Key: api_base

Type: String

Environment Variable: Corresponding environment variable depends on the provider (e.g., AZURE_API_BASE, custom endpoint environment variables).

Default Value: Provider-specific default base URL.

Description: The api_base parameter, under litellm_params, allows you to specify a custom base URL for the LLM provider's API endpoint. This is often necessary when you are not using the default public endpoint of a provider. Common use cases include:

• Azure OpenAI: Azure OpenAI deployments require a custom api_base URL that points to your specific Azure endpoint.
• Open-Source Model Gateways: When using open-source models served through gateways like vLLM or custom inference servers, you'll need to set api_base to the URL of your gateway.
• Non-Default Regions or Endpoints: Some providers might offer APIs in different regions or with custom endpoints, requiring you to override the default api_base.

If api_base is not provided, LiteLLM will use a provider-specific default base URL where applicable.

Example YAML:

model_list:
  - model_name: azure-gpt-3.5-turbo
    litellm_params:
      model: azure/gpt-turbo-small-eu
      api_base: "https://my-azure-endpoint.openai.azure.com/" # Custom api_base for Azure OpenAI
      api_key: "os.environ/AZURE_API_KEY_EU"
      # ... other litellm_params ...

Example Environment Variable:

export AZURE_API_BASE="https://my-azure-endpoint.openai.azure.com/" # Setting Azure API base URL in environment

Provider-Specific Settings (within litellm_params)

YAML Key: Provider-specific parameter names (e.g., api_version, aws_region_name, project_id, etc.)

Type: Depends on the parameter (String, Integer, Boolean, etc.)

Environment Variable: Corresponding environment variable depends on the provider and parameter.

Default Value: Provider-specific defaults.

Description: Beyond the common parameters like api_key and api_base, the litellm_params section can include provider-specific settings. These parameters are unique to each LLM provider and are used to configure provider-specific behaviors or features. Examples include:

• api_version (Azure OpenAI): Specifies the API version to use for Azure OpenAI deployments.
• aws_region_name (AWS Bedrock): Sets the AWS region for Bedrock models.
• project_id (Google Vertex AI): Specifies the Google Cloud Project ID for Vertex AI models.
• deployment_id (Azure OpenAI): Used to target a specific deployment within Azure OpenAI.

Refer to the "Providers" section of this documentation and the official documentation of each LLM provider to identify available provider-specific parameters and their usage.

Example YAML:

model_list:
  - model_name: azure-gpt-4
    litellm_params:
      model: azure/gpt-4-deployment
      api_base: "https://my-azure-endpoint.openai.azure.com/"
      api_key: "os.environ/AZURE_API_KEY_EU"
      api_version: "2023-07-01-preview" # Provider-specific parameter: Azure API Version
      # ... other litellm_params ...

Example Environment Variable:

export AZURE_API_VERSION="2023-07-01-preview" # Setting Azure API version in environment

temperature (litellm_params)

YAML Key: temperature

Type: Number (Float or Integer)

Environment Variable: N/A

Default Value: Provider-specific default (often 1.0 for OpenAI, but can vary).

Description: The temperature parameter, within litellm_params, controls the randomness or creativity of the LLM's output. It's a common parameter across many LLM providers, though the exact range and interpretation might slightly vary.

• Lower values (e.g., 0.1, 0.0): Result in more deterministic, focused, and predictable output. The model will tend to choose the most likely next tokens, making it suitable for tasks requiring factual accuracy or code generation.
• Higher values (e.g., 0.7, 1.0): Introduce more randomness and creativity into the output. The model will consider a wider range of possible next tokens, leading to more surprising, diverse, and potentially less predictable text. Useful for creative writing, brainstorming, or tasks where originality is valued over strict accuracy.

A temperature of 0 makes the output almost entirely deterministic (always choosing the most likely token), while values approaching 1 (or sometimes higher, depending on provider) maximize randomness.

Example YAML:

model_list:
  - model_name: creative-writer
    litellm_params:
      model: openai/gpt-3.5-turbo
      api_key: "os.environ/OPENAI_API_KEY"
      temperature: 0.8 # Setting higher temperature for creative output
      # ... other litellm_params ...
  - model_name: code-generator
    litellm_params:
      model: openai/gpt-3.5-turbo
      api_key: "os.environ/OPENAI_API_KEY"
      temperature: 0.0 # Setting lower temperature for deterministic code generation
      # ... other litellm_params ...

Example Environment Variable: N/A (YAML-only parameter)

max_tokens (litellm_params)

YAML Key: max_tokens

Type: Integer

Environment Variable: N/A

Default Value: Provider-specific default (often varies significantly by model).

Description: The max_tokens parameter, in litellm_params, sets a limit on the maximum number of tokens the LLM should generate in its response. This is a crucial parameter for controlling output length, latency, and cost.

• Token Limit: max_tokens specifies the upper bound on the generated text length. The LLM may stop generating before reaching this limit if it naturally completes its response (e.g., reaches a stop sequence or end-of-text token).
• Cost Control: Limiting max_tokens is a direct way to control the cost of LLM API calls, as you are billed per token.
• Latency Control: Shorter max_tokens values generally result in lower latency, as the model has less text to generate.
• Context Window Consideration: Be mindful of the model's maximum context window when setting max_tokens. The sum of input tokens (prompt) and max_tokens should not exceed the model's context limit.

Example YAML:

model_list:
  - model_name: concise-summary
    litellm_params:
      model: openai/gpt-3.5-turbo
      api_key: "os.environ/OPENAI_API_KEY"
      max_tokens: 256 # Limiting response length for concise summaries
      # ... other litellm_params ...
  - model_name: detailed-answer
    litellm_params:
      model: openai/gpt-4
      api_key: "os.environ/OPENAI_API_KEY"
      max_tokens: 1024 # Allowing longer responses for detailed answers
      # ... other litellm_params ...

Example Environment Variable: N/A (YAML-only parameter)

rpm (litellm_params)

YAML Key: rpm

Type: Integer

Environment Variable: N/A

Default Value: No default (no model-level RPM limit unless explicitly set).

Description: The rpm parameter, within litellm_params, sets a requests-per-minute (RPM) rate limit specifically for this particular model deployment. This is a deployment-level rate limit, distinct from global or per-key rate limits configured elsewhere.

• Model-Specific Rate Limiting: The rpm limit applies only to the model deployment in which it is defined. It does not affect other models or deployments in your model_list.
• Concurrency Control: rpm helps manage concurrency at the model level, preventing overload of specific backend deployments, especially slower or resource-intensive models.
• Resource Management: Use rpm to control the rate of requests to different models based on their capacity, cost, or priority.

Example YAML:

model_list:
  - model_name: gpt-3.5-turbo-low-rpm
    litellm_params:
      model: azure/gpt-turbo-small-eu
      api_key: "os.environ/AZURE_API_KEY_EU"
      rpm: 6 # Setting RPM limit to 6 requests per minute for this deployment
      # ... other litellm_params ...
  - model_name: gpt-3.5-turbo-high-rpm
    litellm_params:
      model: openai/gpt-3.5-turbo
      api_key: "os.environ/OPENAI_API_KEY"
      rpm: 100 # Higher RPM limit for a more robust deployment
      # ... other litellm_params ...

Example Environment Variable: N/A (YAML-only parameter)

tpm (litellm_params)

YAML Key: tpm

Type: Integer

Environment Variable: N/A

Default Value: No default (no model-level TPM limit unless explicitly set).

Description: The tpm parameter, under litellm_params, defines a tokens-per-minute (TPM) rate limit specifically for this model deployment. Similar to rpm, this is a deployment-level limit, separate from global or per-key TPM limits.

• Token-Based Rate Limiting: tpm limits the total number of tokens that can be processed by this model deployment within a minute. This includes both input (prompt) tokens and output (completion) tokens.
• Granular Control: tpm provides finer-grained control over usage compared to rpm, as it accounts for the length and complexity of requests, not just the number of requests.
• Cost Optimization: Use tpm to manage costs more precisely, particularly for models with variable token pricing or when you want to limit total token consumption for a deployment.

Example YAML:

model_list:
  - model_name: gpt-4-economy
    litellm_params:
      model: openai/gpt-4
      api_key: "os.environ/OPENAI_API_KEY"
      tpm: 50000 # Setting TPM limit to 50,000 tokens per minute for cost-conscious deployment
      # ... other litellm_params ...
  - model_name: gpt-4-performance
    litellm_params:
      model: openai/gpt-4
      api_key: "os.environ/OPENAI_API_KEY"
      tpm: 200000 # Higher TPM limit for a performance-focused deployment
      # ... other litellm_params ...

Example Environment Variable: N/A (YAML-only parameter)

model_info

YAML Key: model_info

Type: Object (Mapping)

Environment Variable: N/A

Default Value: No default (optional section).

Description: The model_info section is an optional mapping within each model_list entry. It allows you to provide metadata and descriptive attributes about the model deployment. While not strictly required for the proxy to function, model_info is highly recommended for:

• Monitoring and Observability: Provides data points for dashboards and monitoring systems.
• Cost Tracking: Enables accurate cost calculations and spend analysis.
• Model Management UI (Admin UI): Populates information in the proxy's Admin UI (if enabled) for model browsing and documentation.
• API Endpoint /model/info: Data in model_info is surfaced via the proxy's /model/info endpoint, allowing clients to programmatically retrieve model metadata.

model_info can include fields like cost per token, context window size, model mode (completion, embedding, etc.), and custom metadata.

Example YAML:

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: azure/gpt-turbo-small-eu
      api_key: "os.environ/AZURE_API_KEY_EU"
    model_info: # Defining model_info for metadata
      max_tokens: 4096
      mode: "completion"
      metadata: "Azure OpenAI GPT-3.5 Turbo (EU deployment)"
      # ... other model_info fields ...

Example Environment Variable: N/A (YAML-only section header)

id (model_info)

YAML Key: id

Type: String

Environment Variable: N/A

Default Value: If not provided, LiteLLM may auto-generate a unique ID, but explicit definition is recommended for consistency.

Description: The id parameter, within model_info, allows you to specify a unique identifier for the model deployment. While technically optional, providing a custom id is strongly recommended for:

• Consistent Referencing: Ensures a stable, human-readable identifier for the model, especially when managing multiple deployments or versions.
• Monitoring and Logging: Makes it easier to track and analyze logs and metrics associated with a specific model instance.
• Admin UI Clarity: Provides a clear identifier in the proxy's Admin UI and /model/info endpoint.

If you don't provide an id, LiteLLM may automatically generate one, but relying on auto-generated IDs can make management and tracking less predictable.

Example YAML:

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: azure/gpt-turbo-small-eu
      api_key: "os.environ/AZURE_API_KEY_EU"
    model_info:
      id: "azure-gpt-3.5-turbo-eu-01" # Custom ID for this deployment
      max_tokens: 4096
      # ... other model_info fields ...

Example Environment Variable: N/A (YAML-only parameter)

mode (model_info)

YAML Key: mode

Type: String

Environment Variable: N/A

Default Value: If not provided, LiteLLM may attempt to infer the mode based on the model name, but explicit definition is recommended.

Description: The mode parameter, within model_info, specifies the general function or type of the model deployment. This informs the proxy about how to handle the model, particularly for tasks like health checks and API routing. Common values include:

• "completion": For chat completion and text completion models (most common).
• "embedding": For embedding models.
• "image_generation": For image generation models.
• "audio_transcription": For audio transcription models.

Setting mode correctly is important for features like health checks, which may perform mode-specific validations (e.g., health checks for image models expect mode: "image_generation"). While the proxy might infer the mode in some cases, it's best practice to explicitly define the mode for clarity and robustness.

Example YAML:

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: azure/gpt-turbo-small-eu
      api_key: "os.environ/AZURE_API_KEY_EU"
    model_info:
      mode: "completion" # Explicitly defining mode as "completion"
      max_tokens: 4096
      # ... other model_info fields ...

Example Environment Variable: N/A (YAML-only parameter)

input_cost_per_token (model_info)

YAML Key: input_cost_per_token

Type: Number (Float or Integer)

Environment Variable: N/A

Default Value: If not provided, LiteLLM may use a default cost from its internal model cost map, if available for the model.

Description: The input_cost_per_token parameter, within model_info, specifies the cost in USD (or your chosen currency) per 1,000 input tokens for this model deployment. This is used for internal cost tracking and reporting within LiteLLM Proxy.

• Cost Tracking Accuracy: Providing accurate cost data is crucial for precise spend tracking and budget enforcement.
• Custom Pricing: Use this parameter to define custom pricing if the default cost map is inaccurate or if you have negotiated specific pricing with your provider.
• Cost Units: Costs are always interpreted in USD (or your chosen base currency) and are typically represented as a very small decimal number (e.g., 0.0000015 for $0.0015 per 1,000 tokens).

If input_cost_per_token is not provided, LiteLLM may attempt to use a default cost from its internal model cost map. However, for accurate cost tracking, it's recommended to explicitly define both input_cost_per_token and output_cost_per_token.

Example YAML:

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: azure/gpt-turbo-small-eu
      api_key: "os.environ/AZURE_API_KEY_EU"
    model_info:
      mode: "completion"
      input_cost_per_token: 0.0000015 # Specifying input cost per 1K tokens
      output_cost_per_token: 0.0000020
      # ... other model_info fields ...

Example Environment Variable: N/A (YAML-only parameter)

output_cost_per_token (model_info)

YAML Key: output_cost_per_token

Type: Number (Float or Integer)

Environment Variable: N/A

Default Value: If not provided, LiteLLM may use a default cost from its internal model cost map, if available for the model.

Description: The output_cost_per_token parameter, within model_info, defines the cost in USD (or your chosen currency) per 1,000 output tokens generated by this model deployment. This is used in conjunction with input_cost_per_token for comprehensive cost tracking.

• Complete Cost Picture: You should always provide both input_cost_per_token and output_cost_per_token to get a full picture of the cost implications of using a model.
• Pricing Variations: Different models and providers may have varying costs for input and output tokens. Ensure you configure these values accurately based on your provider's pricing.

If output_cost_per_token is omitted, LiteLLM might attempt to use a default cost from its internal model cost map. However, for reliable cost tracking, it's recommended to explicitly set both input and output cost parameters.

Example YAML:

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: azure/gpt-turbo-small-eu
      api_key: "os.environ/AZURE_API_KEY_EU"
    model_info:
      mode: "completion"
      input_cost_per_token: 0.0000015
      output_cost_per_token: 0.0000020 # Specifying output cost per 1K tokens
      # ... other model_info fields ...

Example Environment Variable: N/A (YAML-only parameter)

max_tokens (model_info)

YAML Key: max_tokens (within model_info)

Type: Integer

Environment Variable: N/A

Default Value: If not provided, LiteLLM may use a default max tokens value from its internal model database, if available for the model.

Description: The max_tokens parameter, when placed inside the model_info section (note: this is different from litellm_params.max_tokens), specifies the maximum context length or token limit of the model itself. This value represents the total number of tokens (input + output) that the model can handle in a single request.

• Context Window Enforcement: LiteLLM Proxy uses this max_tokens value to enforce request size limits. It can prevent requests that exceed the model's context window from being sent to the backend, avoiding errors.
• Informational and Reporting: This value is also used for informational purposes, such as reporting the model's context window via the /model/info endpoint and in logs.

While LiteLLM might use a default max_tokens from its internal database if you omit it, it's best practice to explicitly define model_info.max_tokens for each model, ensuring accurate context window awareness and limit enforcement.

Example YAML:

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: azure/gpt-turbo-small-eu
      api_key: "os.environ/AZURE_API_KEY_EU"
    model_info:
      mode: "completion"
      max_tokens: 4096 # Specifying max context tokens for GPT-3.5 Turbo
      input_cost_per_token: 0.0000015
      output_cost_per_token: 0.0000020
      # ... other model_info fields ...

Example Environment Variable: N/A (YAML-only parameter)

base_model (model_info)

YAML Key: base_model

Type: String

Environment Variable: N/A

Default Value: No default value.

Description: The base_model parameter, within model_info, is an optional field that allows you to specify the underlying base model name or version that a given model_name alias corresponds to.

• Versioning and Lineage Tracking: Useful when you have custom models or model aliases that are based on specific versions of foundation models (e.g., a fine-tuned model based on a particular GPT-4 snapshot).
• Clarity and Documentation: Helps clarify the origin and basis of a model alias, especially when managing multiple versions or custom deployments.
• Cost Mapping (Internal): In some cases, LiteLLM might use base_model to help determine the correct cost mapping for models that don't explicitly report their costs.

In many cases, you can omit base_model, especially for standard models where the model_name is already self-explanatory. However, it's good practice to use it when you want to clearly document the underlying model lineage or need it for accurate cost tracking of custom models.

Example YAML:

model_list:
  - model_name: my-custom-gpt-4-model
    litellm_params:
      model: azure/my-custom-gpt-4-deployment
      api_key: "os.environ/AZURE_API_KEY_EU"
    model_info:
      mode: "completion"
      base_model: "gpt-4-1106-preview" # Indicating this custom model is based on GPT-4 preview version
      max_tokens: 8192
      # ... other model_info fields ...

Example Environment Variable: N/A (YAML-only parameter)

metadata (model_info)

YAML Key: metadata

Type: String, or any other data type serializable to YAML (e.g., Number, Boolean, List, Mapping).

Environment Variable: N/A

Default Value: No default value.

Description: The metadata parameter, within model_info, is a highly flexible and optional field that allows you to attach arbitrary descriptive information to a model deployment. This metadata is free-form and can be used for various purposes:

• Documentation and Descriptions: Provide human-readable descriptions of the model, its purpose, version, or any relevant notes for documentation or UI display.
• UI Labels and Categorization: Use metadata to categorize models, add tags, or provide labels for easier filtering and organization in the Admin UI or model selection interfaces.
• Custom Application Logic: Your application code can retrieve and use this metadata via the /model/info endpoint to implement custom logic based on model attributes.

metadata can be a simple string, or it can be a more complex data structure (like a dictionary or list) if you need to store structured information.

Example YAML:

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: azure/gpt-turbo-small-eu
      api_key: "os.environ/AZURE_API_KEY_EU"
    model_info:
      mode: "completion"
      metadata: "OpenAI GPT-3.5 Turbo model (2023-03-01 snapshot)" # Simple string metadata
      max_tokens: 4096
      # ... other model_info fields ...
  - model_name: llama2-chat-v2
    litellm_params:
      model: openai/llama-2-13b-chat-hf
      api_base: "http://localhost:8000/v1"
      api_key: "none"
    model_info:
      mode: "completion"
      metadata:
        version: 2 # Using structured metadata (dictionary)
        organization: "HuggingFace"
        description: "Local LLaMA-2 13B Chat model, version 2"
      max_tokens: 4096
      # ... other model_info fields ...

Example Environment Variable: N/A (YAML-only parameter)

Additional Fields (within model_info)

YAML Key: Any custom string (e.g., version, organization, notes, etc.)

Type: Any data type serializable to YAML.

Environment Variable: N/A

Default Value: No default value.

Description: The model_info section is extensible. You can add any custom fields you need beyond the predefined parameters (like id, mode, max_tokens, etc.). These custom fields will be stored and returned by the /model/info API, but generally do not affect the proxy's core operation.

• Application-Specific Metadata: Use custom fields to store information that is relevant to your specific application, organization, or workflow.
• Record-Keeping and Tracking: Add fields for version numbers, organization names, notes, tags, or any other data you want to associate with each model deployment for record-keeping purposes.
• UI Customization (Advanced): In highly customized setups, you could potentially use these fields to drive dynamic behavior in custom UIs or dashboards that interact with the proxy's /model/info endpoint.

Example YAML:

model_list:
  - model_name: gpt-3.5-turbo
    litellm_params:
      model: azure/gpt-turbo-small-eu
      api_key: "os.environ/AZURE_API_KEY_EU"
    model_info:
      mode: "completion"
      input_cost_per_token: 0.0000015
      output_cost_per_token: 0.0000020
      version: "2023-03-01" # Custom field: Model version
      organization: "OpenAI" # Custom field: Provider organization
      notes: "Original snapshot of GPT-3.5 Turbo" # Custom field: Notes/description
      max_tokens: 4096
      # ... other model_info fields ...

Example Environment Variable: N/A (YAML-only parameters)

litellm_settings Section: Configuring the LiteLLM Library

This section allows you to configure the behavior of the underlying LiteLLM Python library within the proxy. These settings globally influence logging, networking, caching, and fallback behaviors. Many of these settings correspond to functions within the litellm module or internal flags. While most values can also be set via environment variables (prefixed with LITELLM_), defining them in config.yaml under litellm_settings is often more convenient. If a setting is defined in both litellm_settings and router_settings, the router_settings value will take precedence specifically for router operations.

litellm_settings Parameter Summary

Logging & Callback Settings

This subsection within litellm_settings allows you to configure how the proxy handles logging and integrates with various monitoring and observability platforms. You can define callbacks for successful and failed requests, enable detailed debugging, and control what information is logged.

success_callback

YAML Key: success_callback

Type: Array of Strings

Environment Variable: LITELLM_CALLBACKS (comma-separated string)

Default Value: [] (Empty list, no success callbacks enabled by default).

Description: The success_callback parameter, within litellm_settings, is a list of strings that specifies which callback integrations should be executed after a request to the LLM API is successfully completed. Each string in the list should be the name of a logging or monitoring callback that you want to invoke on successful requests. Common examples include: "langfuse", "helicone", "wandb", "sentry", "mlflow", etc. These callbacks are used for logging, monitoring, and observability purposes.

Example YAML:

litellm_settings:
  success_callback: ["langfuse", "helicone"] # Enabling Langfuse and Helicone callbacks for successful requests
  # ... other litellm_settings ...

Example Environment Variable:

export LITELLM_CALLBACKS="langfuse,helicone" # Setting success callbacks via environment variable

failure_callback

YAML Key: failure_callback

Type: Array of Strings

Environment Variable: LITELLM_CALLBACKS (comma-separated string, same as success_callback)

Default Value: [] (Empty list, no failure callbacks enabled by default).

Description: The failure_callback parameter, in litellm_settings, is an array of strings that defines which callback integrations should be executed when a request to the LLM API fails (results in an error). Similar to success_callback, each string should be the name of a logging/monitoring callback. Failure callbacks are essential for capturing and reporting errors, enabling you to monitor the health and reliability of your LLM application. You might use callbacks like "sentry", "slack", or custom error logging functions in failure_callback.

Example YAML:

litellm_settings:
  failure_callback: ["sentry"] # Enabling Sentry callback for failed requests
  # ... other litellm_settings ...

Example Environment Variable:

export LITELLM_CALLBACKS="sentry" # Setting failure callback via environment variable (using LITELLM_CALLBACKS, same as success_callback)

callbacks (litellm_settings)

YAML Key: callbacks

Type: Array of Strings

Environment Variable: LITELLM_CALLBACKS (comma-separated string, same as success_callback and failure_callback)

Default Value: [] (Empty list, no general callbacks enabled by default).

Description: The callbacks parameter, within litellm_settings, is an array of strings specifying callback integrations that should be executed regardless of whether the LLM API call succeeds or fails. These are general-purpose callbacks that should run for every request outcome. Common use cases for callbacks include:

• Comprehensive Logging: Callbacks like "otel" (OpenTelemetry) for tracing or "lunary" for general logging that you want to capture for all requests, both successful and failed.
• Auditing: Callbacks for audit logging, where you need a record of every request attempt, regardless of success.
• Metrics Collection: Callbacks that increment counters or update metrics dashboards for every request, irrespective of the outcome.

If a callback is listed in both callbacks and either success_callback or failure_callback, it will only be executed once per request, avoiding redundant callback invocations.

Example YAML:

litellm_settings:
  callbacks: ["otel"] # Enabling OpenTelemetry callback for all requests (success and failure)
  # ... other litellm_settings ...

Example Environment Variable:

export LITELLM_CALLBACKS="otel" # Setting general callback via environment variable (using LITELLM_CALLBACKS, same as success_callback and failure_callback)

service_callbacks

YAML Key: service_callbacks

Type: Array of Strings

Environment Variable: N/A

Default Value: [] (Empty list, no service callbacks enabled by default).

Description: The service_callbacks parameter, within litellm_settings, is a list of strings that defines service health monitoring callbacks. These callbacks are not triggered by individual LLM API requests. Instead, they are designed to capture internal service-level events and failures within the LiteLLM Proxy itself, such as:

• Database Connection Issues: Errors connecting to or interacting with the proxy's database (Postgres).
• Cache Backend Failures: Problems with Redis, S3, or other caching backends.
• Internal Service Errors: Exceptions or failures within the proxy's own components or services.

service_callbacks are typically used for operational monitoring of the proxy's health and infrastructure, rather than for logging individual LLM request details. Examples include "datadog" and "prometheus" for reporting internal metrics and errors to monitoring platforms.

Example YAML:

litellm_settings:
  service_callbacks: ["datadog", "prometheus"] # Enabling Datadog and Prometheus service health callbacks
  # ... other litellm_settings ...

Example Environment Variable: N/A (YAML-only parameter)

turn_off_message_logging

YAML Key: turn_off_message_logging

Type: Boolean

Environment Variable: N/A

Default Value: false (Message logging is enabled by default).

Description: The turn_off_message_logging parameter, in litellm_settings, is a boolean flag that controls whether the content of prompts (messages) and responses are logged to your configured callbacks.

• Privacy and Security: Setting turn_off_message_logging: true prevents the actual text of conversations from being logged to external logging/monitoring systems. This is crucial for privacy-sensitive applications or when dealing with confidential data in prompts and responses.
• Reduced Log Volume: Disabling message logging can significantly reduce the volume of logs generated by the proxy, which can be beneficial for cost optimization and easier log management, especially in high-throughput environments.
• Metadata Logging Still Active: Even when message logging is turned off, the proxy will still log metadata about requests, such as token counts, costs, timestamps, model names, and error details. Only the actual text content of prompts and responses is suppressed.

Example YAML:

litellm_settings:
  turn_off_message_logging: true # Disabling message content logging for privacy
  success_callback: ["langfuse"] # Still logging metadata to Langfuse, but no message text
  # ... other litellm_settings ...

Example Environment Variable: N/A (YAML-only parameter)

Note: In code, you can achieve the same effect by setting litellm.turn_off_message_logging=True globally in your application or proxy initialization code.

redact_user_api_key_info

YAML Key: redact_user_api_key_info

Type: Boolean

Environment Variable: N/A

Default Value: false (User API key information is logged by default).

Description: The redact_user_api_key_info parameter, within litellm_settings, is a boolean flag that controls whether information related to the end-user's API key (virtual key) is redacted (removed or anonymized) from logs sent to your configured callbacks.

• Privacy of User Credentials: When enabled (redact_user_api_key_info: true), the proxy will attempt to remove or anonymize user-specific credential information from logs before they are sent to external logging systems (like Langfuse, OpenTelemetry, etc.). This includes:
  • The hashed virtual API key itself.
  • User IDs, team IDs, or any other identifiers associated with the virtual key.
• Log Anonymization: This feature helps prevent accidental exposure of user-specific credential information in external logs, enhancing privacy and security, especially in multi-tenant environments where user API keys are used.
• Supported Loggers: Redaction is supported for specific loggers, including Langfuse, OpenTelemetry, Logfire, and ArizeAI (as of the documentation). Check the official documentation for the most up-to-date list of supported loggers.

Example YAML:

litellm_settings:
  redact_user_api_key_info: true # Redacting user API key info from logs for privacy
  success_callback: ["langfuse"] # Langfuse logging enabled, but user key info will be redacted
  # ... other litellm_settings ...

Example Environment Variable: N/A (YAML-only parameter)

Note: There is no direct environment variable equivalent for this setting; it must be configured in config.yaml.

langfuse_default_tags

YAML Key: langfuse_default_tags

Type: Array of Strings

Environment Variable: N/A

Default Value: [] (Empty list, no default Langfuse tags are added by default).

Description: The langfuse_default_tags parameter, within litellm_settings, is an array of strings that allows you to specify a list of default tags to be automatically added to every log event sent to Langfuse, when using the Langfuse logging integration.

• Enhanced Langfuse Filtering and Analysis: Tags are key-value pairs that can be attached to log events in Langfuse. By defining default tags, you can automatically enrich your Langfuse logs with useful metadata, making it easier to filter, group, and analyze requests within the Langfuse platform.
• Predefined Metadata: Common tags you might want to include by default could be:
  • "proxy_base_url": The base URL of your LiteLLM Proxy instance.
  • "user_api_key_alias": The alias or identifier of the user API key used for the request.
  • "semantic-similarity": Any semantic similarity scores or metrics calculated during request processing.
  • "cache_hit": Indicates if the response was served from the cache.
  • "user_api_key_user_id": The User ID associated with the API key.

Example YAML:

litellm_settings:
  success_callback: ["langfuse"] # Langfuse logging enabled
  langfuse_default_tags: ["proxy_base_url", "user_api_key_alias", "cache_hit"] # Adding default tags to Langfuse logs
  # ... other litellm_settings ...

Example Environment Variable: N/A (YAML-only parameter)

Note: These tags will then be automatically applied to every log event sent to Langfuse by the proxy, aiding in filtering and analysis within the Langfuse UI.

modify_params

YAML Key: modify_params

Type: Boolean

Environment Variable: N/A

Default Value: false (Parameter modification is disabled by default).

Description: The modify_params parameter, within litellm_settings, is an advanced boolean flag that enables a powerful, but potentially risky, feature: allowing programmatic modification of request parameters just before the proxy sends the request to the backend LLM provider.

• Custom Request Transformations: When modify_params: true, the proxy allows custom code (typically via a plugin or callback) to intercept and alter the litellm_params dictionary or the request body itself on the fly. This opens up possibilities for highly customized request handling.
• Advanced Use Cases: modify_params is primarily intended for advanced and enterprise scenarios, such as:
  • Custom Plugin Logic: Implementing complex plugins that need to dynamically adjust API requests based on real-time conditions, user context, or external data.
  • Provider-Specific Parameter Injection: Programmatically injecting provider-specific parameters that are not directly exposed through standard LiteLLM parameters, but needed for certain backend models.
  • Request Augmentation: Adding extra information or context to the request payload based on custom logic.

Important Security and Stability Considerations:

• Security Risks: Enabling modify_params: true introduces significant security risks if not used carefully. Malicious or poorly written custom code could potentially:
  • Expose sensitive data by logging or transmitting modified requests.
  • Bypass security checks or rate limits.
  • Cause unexpected behavior or errors in the proxy or backend LLM.
• Stability Risks: Incorrectly modifying parameters can lead to malformed API requests, causing errors or unpredictable LLM behavior.
• Advanced Feature – Use with Caution: This parameter is marked as an advanced feature and should be used only if you fully understand the risks and have a clear, well-justified use case. Thoroughly test and audit any custom code that modifies request parameters.

Example YAML:

litellm_settings:
  modify_params: true # Enabling request parameter modification (advanced feature - use with caution!)
  # ... other litellm_settings ...

Example Environment Variable: N/A (YAML-only parameter)

Note: The actual logic for how parameters are modified is not configured via config.yaml. modify_params: true simply enables the capability. The custom modification logic itself must be implemented in separate Python code, usually within a plugin or callback, that is designed to work with this feature.

enable_preview_features

YAML Key: enable_preview_features

Type: Boolean

Environment Variable: N/A

Default Value: false (Preview features are disabled by default).

Description: The enable_preview_features parameter, within litellm_settings, is a boolean flag that controls whether experimental or preview features within the LiteLLM library are enabled.

• Access to New or Experimental Functionality: LiteLLM may introduce new features or integrations that are initially marked as "preview" or "experimental". These features are typically under active development and may have limited stability, documentation, or support.
• Early Access and Testing: Enabling enable_preview_features: true allows you to access and test these preview features, giving you early access to new capabilities and the opportunity to provide feedback.
• Potential Instability or Changes: Preview features are not considered stable and are subject to change, removal, or movement to the Enterprise edition in future releases. Do not rely on preview features in production environments unless you are prepared for potential instability and changes.
• Documentation Check: Always consult the official LiteLLM documentation to understand which features are currently behind the enable_preview_features flag and what their limitations or caveats are.

Example YAML:

litellm_settings:
  enable_preview_features: true # Enabling preview features (use with caution!)
  # ... other litellm_settings ...

Example Environment Variable: N/A (YAML-only parameter)

Example Use Case: At the time of writing, enabling Azure's "O1" series models with streaming requires setting enable_preview_features: true.

set_verbose

YAML Key: set_verbose

Type: Boolean

Environment Variable: LITELLM_VERBOSE=1 (or any non-zero value)

Default Value: false (Verbose logging is disabled by default).

Description: The set_verbose parameter, within litellm_settings, is a boolean flag that enables verbose debug logging from the core LiteLLM Python library itself. When enabled, LiteLLM will output detailed debug messages to the proxy's console (stdout), providing very granular information about its internal operations, API calls, and processing steps.

• Detailed Debugging Information: Verbose logging is invaluable for troubleshooting complex issues, understanding the internal workings of LiteLLM, or debugging custom integrations. It can reveal:
  • Detailed request and response payloads sent to and received from LLM providers.
  • Internal function calls and execution flow within the LiteLLM library.
  • Variable values and internal state during request processing.
• High Log Volume: Verbose logging generates a very large volume of log output. Do not enable set_verbose: true in production environments unless absolutely necessary for debugging a specific issue, as it can impact performance, consume disk space, and potentially expose sensitive information in logs.
• Development and Testing Tool: set_verbose: true is primarily intended as a development and debugging tool for local testing or controlled staging environments.

Example YAML:

litellm_settings:
  set_verbose: true # Enabling verbose debug logging (for development/debugging only!)
  # ... other litellm_settings ...

Example Environment Variable:

export LITELLM_VERBOSE=1 # Enabling verbose logging via environment variable

Note: Setting the environment variable LITELLM_VERBOSE=1 (or any non-zero value) has the same effect as setting set_verbose: true in config.yaml. You can also set litellm.set_verbose=True programmatically in your code.

json_logs

YAML Key: json_logs

Type: Boolean

Environment Variable: LITELLM_JSON_LOGS=true (or any value interpretable as boolean true, e.g., "1", "yes")

Default Value: false (Logs are output in plain text format by default).

Description: The json_logs parameter, within litellm_settings, is a boolean flag that controls the format of the proxy's logs.

• JSON Structured Logs: When json_logs: true, the proxy will output all logs (including request logs, error logs, and other proxy events) in JSON (JavaScript Object Notation) format instead of the default plain text format.
• Structured Logging Benefits: JSON logs are highly beneficial for:
  • Log Aggregation and Analysis: JSON format is easily parsed and ingested by log management systems (like Elasticsearch, Splunk, Datadog Logs, etc.), enabling efficient centralized logging, searching, and analysis.
  • Programmatic Processing: JSON logs are structured data, making it much easier to programmatically parse, filter, and process log information for monitoring dashboards, automated alerts, or custom analytics tools.
  • Integration with Observability Pipelines: JSON logs are well-suited for integration into modern observability pipelines and systems that expect structured log data.

Example YAML:

litellm_settings:
  json_logs: true # Enabling JSON formatted logs for structured logging
  # ... other litellm_settings ...

Example Environment Variable:

export LITELLM_JSON_LOGS=true # Enabling JSON logs via environment variable

Note: Enabling json_logs: true will cause the proxy to output all logs in JSON format to stdout (standard output). If you are shipping logs to a JSON log aggregator, this is highly recommended. The documentation notes that enabling JSON logs also causes the raw POST request payload to be logged as JSON as well, which can be useful for debugging but consider any potential security implications of logging request bodies in detail.

Networking & Timeout Settings

This subsection of litellm_settings controls network-related configurations, primarily focusing on timeouts and connection behaviors for requests made to the backend LLM providers. These settings are crucial for ensuring responsiveness and handling network issues.

request_timeout

YAML Key: request_timeout

Type: Integer (representing seconds)

Environment Variable: N/A

Default Value: While the documentation notes 6000 seconds (100 minutes), this seems exceptionally high and might be a typo. A more typical default, based on OpenAI SDK defaults and general API timeout practices, would be closer to 600 seconds (10 minutes) or even less. You should test and confirm the actual default behavior in your LiteLLM Proxy version.

Description: The request_timeout parameter, within litellm_settings, sets the timeout duration in seconds for all LLM API calls made by the LiteLLM library within the proxy. This timeout applies to the entire duration of a single API call to the backend LLM provider, including network latency, server-side processing, and response streaming.

• Preventing Hanging Requests: request_timeout is crucial for preventing indefinite hanging or stalled requests. If a call to the upstream model takes longer than the specified timeout, the proxy will abort the request and raise a Timeout error.
• Resource Management: Timeouts help free up resources on the proxy server and prevent it from getting stuck waiting for unresponsive or slow LLM providers.
• User Experience: Setting a reasonable timeout ensures that clients receive timely responses or error indications, rather than waiting indefinitely for a response that may never come.
• Recommended Value: It's generally recommended to set a request_timeout to a reasonable value, balancing responsiveness with allowing enough time for legitimate LLM processing. A value like 30-60 seconds might be a good starting point for many applications, but you should adjust it based on your expected LLM response times and application requirements.

Example YAML:

litellm_settings:
  request_timeout: 30 # Setting request timeout to 30 seconds
  # ... other litellm_settings ...

Example Environment Variable: N/A (YAML-only parameter)

Note: There is also a router_settings.timeout parameter, which sets a timeout for the overall request handling within the proxy router, including retries and fallbacks. The litellm_settings.request_timeout is specifically for the underlying LLM API call itself. If both are set, router_settings.timeout typically takes precedence for overall request handling, while litellm_settings.request_timeout governs the individual LLM call timeout.

force_ipv4

YAML Key: force_ipv4

Type: Boolean

Environment Variable: N/A

Default Value: false (IPv6 is preferred if available, otherwise IPv4 is used).

Description: The force_ipv4 parameter, in litellm_settings, is a boolean flag that, when set to true, forces all outgoing LLM API requests from the proxy to use IPv4 (Internet Protocol version 4) exclusively.

• IPv6 Compatibility Issues Workaround: In certain network environments or with specific LLM providers, IPv6 connectivity might cause issues, such as HTTPX connection errors (particularly observed with Anthropic API). Setting force_ipv4: true can act as a workaround by forcing the proxy to use IPv4, potentially resolving these connectivity problems.
• IPv4-Only Environments: In environments where IPv6 is not fully supported or reliably configured, forcing IPv4 ensures consistent and predictable network communication for LLM API calls.
• Default IPv6 Preference: By default (force_ipv4: false), the proxy will attempt to use IPv6 if it is available and properly configured in the system environment. If IPv6 is not available or fails, it will fall back to IPv4 automatically. Setting force_ipv4: true disables this IPv6 preference and forces IPv4-only communication.

Example YAML:

litellm_settings:
  force_ipv4: true # Forcing IPv4 for all LLM API requests (workaround for IPv6 issues)
  # ... other litellm_settings ...

Example Environment Variable: N/A (YAML-only parameter)

Note: Only enable force_ipv4: true if you are experiencing specific IPv6 connectivity issues with your LLM provider APIs. In most modern network environments, leaving it at the default false is recommended, allowing the proxy to leverage IPv6 if available.

forward_traceparent_to_llm_provider

YAML Key: forward_traceparent_to_llm_provider

Type: Boolean

Environment Variable: N/A

Default Value: false (Traceparent header is not forwarded by default).

Description: The forward_traceparent_to_llm_provider parameter, within litellm_settings, is a boolean flag that controls whether the proxy will forward any incoming traceparent header it receives from the client application to the upstream LLM API call.

• Distributed Tracing Propagation: The traceparent header is part of the W3C Trace Context standard for distributed tracing. When a client application initiates a trace (e.g., using OpenTelemetry or a similar tracing system) and includes a traceparent header in its request to the proxy, enabling forward_traceparent_to_llm_provider: true will propagate this trace context to the backend LLM provider.
• End-to-End Traceability: This allows for end-to-end traceability of requests across your entire system, from the client application, through the LiteLLM Proxy, and into the LLM provider's infrastructure (if the provider also supports and honors trace context propagation). This is invaluable for debugging and performance monitoring of distributed LLM applications.
• Self-Hosted or Internal LLMs: forward_traceparent_to_llm_provider: true is generally safe and useful when you are proxying to self-hosted LLMs or internal model services that you control or trust and that are designed to handle trace context propagation.
• Caution with External Services: Exercise caution when enabling this for external, third-party LLM services (like OpenAI, Anthropic, Google Vertex AI, AWS Bedrock). External services may not recognize or correctly handle the traceparent header. In some cases, forwarding unknown headers to external services might even cause errors or unexpected behavior. The documentation explicitly warns against using this with external services like AWS Bedrock or Google Vertex AI, as they may reject requests with unknown headers.
• Default Off for Safety: The default value is false for safety, to avoid unintended issues with external services.

Example YAML:

litellm_settings:
  forward_traceparent_to_llm_provider: true # Enabling traceparent header forwarding for internal LLM backend
  # ... other litellm_settings ...

Example Environment Variable: N/A (YAML-only parameter)

Important: Only enable forward_traceparent_to_llm_provider: true if you are certain that your LLM backend (typically a self-hosted or internal service) is designed to accept and handle traceparent headers. For external, third-party LLM providers, leave this setting at its default false unless you have explicit guidance from the provider to enable trace context propagation.

Fallback & Reliability Settings

This subsection within litellm_settings provides options for configuring fallback models and other reliability features. These settings help your proxy handle errors and ensure service continuity when primary LLMs are unavailable or exceed context limits.

default_fallbacks

YAML Key: default_fallbacks

Type: Array of Strings (List of model names)

Environment Variable: N/A

Default Value: [] (Empty list, no default fallbacks configured).

Description: The default_fallbacks parameter, within litellm_settings, is an array of model names that defines a global fallback list. This list specifies which alternative models the proxy should attempt to use as a last resort if a request to the originally intended model fails for any reason (except for ContentPolicyViolationError and ContextWindowExceededError, which have their own dedicated fallback settings).

• Global Safety Net: default_fallbacks acts as a catch-all fallback mechanism. If a request to a model fails and no model-specific fallback is defined (via router_settings.fallbacks or request-level fallbacks parameter) and the error is not a content policy or context window error, the proxy will try the models listed in default_fallbacks in the order they are specified.
• Reliability and Redundancy: Use default_fallbacks to enhance the reliability of your LLM application by ensuring that there are always backup options if primary models become unavailable or encounter issues.
• Ordered Fallback Attempts: The proxy will attempt to use the fallback models in the list sequentially. If the first fallback model also fails, it will try the next one, and so on, until either a fallback succeeds or the list is exhausted.
• Model Names from model_list:** The model names listed in default_fallbacks must be valid model_name aliases that are defined in your model_list section of config.yaml.

Example YAML:

litellm_settings:
  default_fallbacks: ["claude-instant-1", "gpt-3.5-turbo"] # Global fallback list: try Claude Instant-1, then GPT-3.5 Turbo
  # ... other litellm_settings ...

In this example, if a request fails for any model and no specific fallback is set for that model, the proxy will first attempt to use the model aliased as "claude-instant-1". If that also fails, it will then try "gpt-3.5-turbo". If both fallbacks fail, the request will ultimately result in an error.

Example Environment Variable: N/A (YAML-only parameter)

Note: default_fallbacks is a global fallback mechanism. For more granular control, you can define model-specific fallbacks using router_settings.fallbacks (see Router Settings section).

content_policy_fallbacks

YAML Key: content_policy_fallbacks

Type: Array of Objects (List of mappings)

Environment Variable: N/A

Default Value: [] (Empty list, no content policy fallbacks configured).

Description: The content_policy_fallbacks parameter, within litellm_settings, is an array of objects used to specify fallback models specifically for ContentPolicyViolationError errors. This error typically indicates that the requested content (prompt or completion) violated the content policy of the LLM provider (e.g., OpenAI, Anthropic, etc.).

• Content Moderation Handling: content_policy_fallbacks allows you to define a specific fallback strategy to handle content policy rejections gracefully. Instead of simply returning an error to the client, the proxy can automatically retry the request with a different, potentially less restrictive, model.
• Model-Specific Fallbacks for Content Policy: You can define fallbacks on a per-model basis. Each object in the content_policy_fallbacks list is a mapping that associates a source model (or model group) with a list of fallback models to try if a ContentPolicyViolationError occurs for the source model.
• Example Mapping: content_policy_fallbacks: [{"gpt-3.5-turbo-small": ["claude-opus"]}] means: "If model gpt-3.5-turbo-small returns a ContentPolicyViolationError, retry the request with claude-opus."
• List of Fallbacks: For each source model, you can specify a list of fallback models to try in order. The proxy will attempt the fallbacks sequentially until one succeeds or the list is exhausted.
• Model Names from model_list:** The model names used as source models and fallback models must be valid model_name aliases defined in your model_list section.

Example YAML:

litellm_settings:
  content_policy_fallbacks: # Defining fallbacks specifically for ContentPolicyViolationError
    - gpt-3.5-turbo-small: ["claude-instant-1", "gpt-3.5-turbo-large"] # If gpt-3.5-turbo-small fails content policy, try Claude Instant-1, then GPT-3.5-turbo-large
    - gpt-4: ["claude-2"] # If gpt-4 fails content policy, try Claude-2
  # ... other litellm_settings ...

In this example, if a request to gpt-3.5-turbo-small results in a ContentPolicyViolationError, the proxy will first retry the request with claude-instant-1. If that also fails (or also results in a content policy error), it will then try gpt-3.5-turbo-large. Separate fallback mappings are defined for gpt-4 to use claude-2 as a content policy fallback.

Example Environment Variable: N/A (YAML-only parameter)

Note: content_policy_fallbacks is specifically for handling ContentPolicyViolationError errors. For other types of errors, use the general default_fallbacks or router_settings.fallbacks mechanisms.

context_window_fallbacks

YAML Key: context_window_fallbacks

Type: Array of Objects (List of mappings)

Environment Variable: N/A

Default Value: [] (Empty list, no context window fallbacks configured).

Description: The context_window_fallbacks parameter, within litellm_settings, is an array of objects used to specify fallback models specifically for ContextWindowExceededError errors. This error occurs when the input prompt (or combined prompt and response length) is too long and exceeds the maximum context window of the model.

• Context Length Handling: context_window_fallbacks provides a mechanism to automatically handle requests that are too long for a given model's context window. Instead of failing, the proxy can intelligently retry the request with a model that has a larger context window.
• Model-Specific Fallbacks for Context Window Errors: Similar to content_policy_fallbacks, you define fallbacks on a per-model basis. Each object in the context_window_fallbacks list is a mapping that associates a source model (or model group) with a list of fallback models with larger context windows.
• Example Mapping: context_window_fallbacks: [{"gpt-3.5-turbo-small": ["gpt-3.5-turbo-large", "claude-opus"]}] means: "If model gpt-3.5-turbo-small returns a ContextWindowExceededError, first retry with gpt-3.5-turbo-large, and if that also fails (or also gets a context window error), then try claude-opus."
• Ordered Fallback Attempts: The proxy will try the fallback models in the list sequentially until a model with a sufficiently large context window is found or the list is exhausted.
• Model Names from model_list:** The model names used as source models and fallback models must be valid model_name aliases defined in your model_list.

Example YAML:

litellm_settings:
  context_window_fallbacks: # Defining fallbacks specifically for ContextWindowExceededError
    - gpt-3.5-turbo-small: ["gpt-3.5-turbo-large", "claude-opus"] # If gpt-3.5-turbo-small gets context window error, try gpt-3.5-turbo-large, then Claude Opus
    - gpt-3.5-turbo: ["gpt-4"] # If gpt-3.5-turbo (4K context) gets context window error, try gpt-4 (8K+ context)
  # ... other litellm_settings ...

In this example, if gpt-3.5-turbo-small (perhaps a model with a 4K context window) gets a request that's too long, the proxy will first try gpt-3.5-turbo-large (e.g., a 16K context model). If that still fails due to context length, it will then attempt to use claude-opus (potentially a model with a very large context). A separate fallback is defined for gpt-3.5-turbo (assuming it has a smaller context than gpt-4).

Example Environment Variable: N/A (YAML-only parameter)

Note: context_window_fallbacks is specifically for ContextWindowExceededError errors. For other error types, use the general default_fallbacks or router_settings.fallbacks mechanisms.

disable_end_user_cost_tracking

YAML Key: disable_end_user_cost_tracking

Type: Boolean

Environment Variable: N/A

Default Value: false (End-user cost tracking is enabled by default).

Description: The disable_end_user_cost_tracking parameter, within litellm_settings, is a boolean flag that controls whether the proxy should track costs at the end-user level. By default, LiteLLM Proxy tracks and records the cost of each request, attributing it to the specific end-user (if a user identifier is provided in the request or associated with the API key). This per-user cost tracking is used for:

• Usage Dashboards: Generating dashboards that show LLM spend broken down by user, team, or project.
• Cost Control and Budgeting: Enforcing budgets and quotas at the user or team level.
• Detailed Usage Analysis: Providing granular usage data for cost optimization and resource allocation.

Setting disable_end_user_cost_tracking: true will turn off this per-user cost tracking. The proxy will still track aggregate usage and costs, but it will not record or report costs at the individual end-user level.

• Privacy Considerations: You might disable per-user cost tracking for privacy reasons, if you do not want to associate costs with specific users.
• Simplified Cost Reporting: If you only need aggregate cost metrics and do not require per-user spend breakdowns, disabling this feature can simplify your data and reduce the amount of data stored in the proxy's database.

Example YAML:

litellm_settings:
  disable_end_user_cost_tracking: true # Disabling per-user cost tracking for privacy
  # ... other litellm_settings ...

Example Environment Variable: N/A (YAML-only parameter)

Note: When end-user cost tracking is disabled, the proxy will still track and report aggregate costs (total spend across all requests), but the per-user breakdown will not be available.

disable_end_user_cost_tracking_prometheus_only

YAML Key: disable_end_user_cost_tracking_prometheus_only

Type: Boolean

Environment Variable: N/A

Default Value: false (End-user cost tracking is enabled in Prometheus metrics by default).

Description: The disable_end_user_cost_tracking_prometheus_only parameter, within litellm_settings, is a boolean flag that provides more granular control over end-user cost tracking. When set to true, it disables per-user cost tracking *specifically in Prometheus metrics, but **continues to record per-user cost data in the proxy's database spend logs*.

• Selective Metric Exposure: This setting allows you to control where per-user cost information is exposed. You might use this if:
  • You want to retain detailed per-user spend logs in your internal database for audit or detailed analysis.
  • But you do not want to expose per-user cost metrics via Prometheus, perhaps for privacy or security reasons when Prometheus metrics are exposed externally or to a wider audience.
• Database Logs Still Retain Detail: Even when Prometheus tracking is disabled via this flag, the proxy will still write per-user spend information to the spend_logs database table, ensuring that detailed spend data is still captured internally.

Example YAML:

litellm_settings:
  disable_end_user_cost_tracking_prometheus_only: true # Disabling per-user cost tracking in Prometheus metrics only
  # ... other litellm_settings ...

Example Environment Variable: N/A (YAML-only parameter)

In summary: disable_end_user_cost_tracking turns off per-user cost tracking everywhere (database and Prometheus). disable_end_user_cost_tracking_prometheus_only turns it off only in Prometheus metrics, while keeping detailed per-user logs in the database.

key_generation_settings

YAML Key: key_generation_settings

Type: Object (Mapping)

Environment Variable: N/A

Default Value: If not provided, default key generation settings apply (no special restrictions beyond normal authentication).

Description: The key_generation_settings parameter, within litellm_settings, is an advanced mapping (object) that controls who is allowed to generate API keys (virtual keys) via the proxy's admin endpoints (e.g., the /key/generate API or the Admin UI). This is primarily an enterprise-level configuration, relevant when you expose a UI or API for users or teams to create their own API keys for accessing the proxy.

• Granular Key Generation Control: key_generation_settings allows you to define fine-grained rules and restrictions on API key generation, enabling you to implement policies around who can create keys and what requirements must be met during key creation.
• Sub-Keys for Key Types: The key_generation_settings object can contain two sub-keys:
  • team_key_generation: Settings specifically for generating team API keys (keys associated with a team or organization).
  • personal_key_generation: Settings for generating personal API keys (keys not tied to a team).
• Enterprise Feature: This is primarily an enterprise feature used in multi-tenant scenarios or when you need to enforce strict governance over API key issuance.

Configuration Options within key_generation_settings Sub-Keys:

For both team_key_generation and personal_key_generation, you can define the following restrictions:

• allowed_team_member_roles / allowed_user_roles: (Array of Strings): Lists the user roles that are permitted to generate keys of this type. Roles are typically defined within your user authentication system (e.g., "admin", "proxy_admin", "developer", etc.). Only users with roles listed here will be authorized to create keys. If not specified, there's no role-based restriction beyond general authentication.
• required_params: (Array of Strings): Lists parameters that are required to be provided when generating a key of this type. For example, you might require that team admins always provide "tags" when creating team API keys for cost tracking purposes. If a required parameter is missing during key generation, the request will be rejected.

Example YAML:

litellm_settings:
  key_generation_settings: # Enterprise Feature: Controlling key generation
    team_key_generation: # Settings for team API keys
      allowed_team_member_roles: ["admin"] # Only team members with "admin" role can create team keys
      required_params: ["tags"] # Team keys must be created with "tags" parameter
    personal_key_generation: # Settings for personal API keys
      allowed_user_roles: ["proxy_admin", "key_manager"] # Only users with "proxy_admin" or "key_manager" roles can create personal keys
      # No required_params specified for personal keys (optional)
  # ... other litellm_settings ...

In this example:

• Only team members with the "admin" role can create team API keys, and they must provide a "tags" parameter when creating a team key.
• Only users with the "proxy_admin" or "key_manager" roles can create personal API keys. There are no additional parameter requirements for personal keys.

Example Environment Variable: N/A (YAML-only parameter)

Note: key_generation_settings is an enterprise feature primarily relevant for multi-tenant deployments or organizations with strict API key governance policies. If unspecified, there are no special restrictions on key generation beyond the normal authentication requirements of the proxy.

disable_add_transform_inline_image_block

YAML Key: disable_add_transform_inline_image_block

Type: Boolean

Environment Variable: N/A

Default Value: false (Automatic #transform=inline addition is enabled by default).

Description: The disable_add_transform_inline_image_block parameter, within litellm_settings, is a boolean flag specifically related to integration with Fireworks AI models. When set to true, it disables the automatic addition of #transform=inline to the image_url parameter for non-vision Fireworks AI models.

• Fireworks AI Specific Behavior: Fireworks AI's API might require image URLs to include #transform=inline as a query parameter for certain models, especially for image generation calls when the model itself is not a vision model.
• Automatic Parameter Appending (Default): By default (disable_add_transform_inline_image_block: false), LiteLLM Proxy will automatically append #transform=inline to the image_url when making image generation requests to Fireworks AI models that are not vision models. This is to ensure compatibility with Fireworks AI's API requirements.
• Disabling Auto-Addition (Optional): Setting disable_add_transform_inline_image_block: true will turn off this automatic parameter appending. You might want to disable it if:
  • You are using a Fireworks AI model where this parameter is not needed or causes issues.
  • You want to have explicit control over the image_url and do not want the proxy to modify it automatically.
• Fireworks AI Context: This setting is only relevant when you are using Fireworks AI models. If you are not using Fireworks AI, you can safely ignore this parameter.

Example YAML:

litellm_settings:
  disable_add_transform_inline_image_block: true # Disabling auto-addition of #transform=inline for Fireworks AI
  # ... other litellm_settings ...

Example Environment Variable: N/A (YAML-only parameter)

Note: If you are not using Fireworks AI models, you can typically leave this parameter at its default false value. Only adjust it to true if you specifically need to disable the automatic #transform=inline appending for Fireworks AI models, and understand the implications for your Fireworks AI integration.

disable_hf_tokenizer_download

YAML Key: disable_hf_tokenizer_download

Type: Boolean

Environment Variable: N/A

Default Value: false (Hugging Face tokenizer download is enabled by default).

Description: The disable_hf_tokenizer_download parameter, within litellm_settings, is a boolean flag that controls whether LiteLLM should attempt to download Hugging Face tokenizer files.

• Hugging Face Tokenizer Usage (Default): By default (disable_hf_tokenizer_download: false), LiteLLM will try to use the appropriate tokenizer for Hugging Face models. This often involves downloading tokenizer files from the Hugging Face Hub when a Hugging Face model is first used via the proxy. Using the correct tokenizer is essential for accurate token counting and optimal performance with Hugging Face models.
• Forcing OpenAI Tokenizer (Alternative): Setting disable_hf_tokenizer_download: true instructs LiteLLM to avoid downloading Hugging Face tokenizers. Instead, it will default to using the OpenAI tokenizer for all models, including Hugging Face models.
• Use Cases for Disabling HF Tokenizer Download: You might want to disable Hugging Face tokenizer download in scenarios such as:
  • Offline Environments: In environments with limited or no internet access, preventing tokenizer downloads can be necessary.
  • Restricted Environments: In highly secure or restricted environments where downloading external files is prohibited or undesirable.
  • Startup Speed Optimization: Downloading tokenizer files can sometimes add to the startup time of the proxy, especially on slower networks. Disabling downloads can speed up proxy startup, at the potential cost of less accurate tokenization for Hugging Face models.
  • Forced OpenAI Tokenizer: If you specifically want to ensure that the OpenAI tokenizer is used for all models, even Hugging Face models, for consistency or compatibility reasons.

Example YAML:

litellm_settings:
  disable_hf_tokenizer_download: true # Disabling Hugging Face tokenizer downloads, forcing OpenAI tokenizer usage
  # ... other litellm_settings ...

Example Environment Variable: N/A (YAML-only parameter)

Important Considerations:

• Tokenizer Accuracy: Using the OpenAI tokenizer for Hugging Face models may lead to less accurate token counting and potentially suboptimal performance for some models, as Hugging Face models are often trained and optimized with their specific tokenizers.
• Tokenization Mismatches: Forcing the OpenAI tokenizer might lead to tokenization mismatches and unexpected behavior with certain Hugging Face models that heavily rely on their specific tokenization schemes.
• Default Behavior Recommendation: Unless you have a specific reason to disable Hugging Face tokenizer downloads (like offline environments or forced OpenAI tokenizer usage), it's generally recommended to leave disable_hf_tokenizer_download: false (the default) to ensure proper tokenizer usage for Hugging Face models.

Caching Settings

The cache and cache_params settings, located under litellm_settings, are used to configure the proxy's caching behavior. You'll define the caching backend, set time-to-live (TTL) values, and customize which types of requests are cached.

cache (litellm_settings)

YAML Key: cache

Type: Boolean

Environment Variable: N/A

Default Value: Often false by default, but can depend on the specific distribution or setup of LiteLLM Proxy. Check your default config.yaml or distribution-specific documentation.

Description: The cache parameter, within litellm_settings, is the master switch for enabling or disabling caching of LLM responses within the LiteLLM Proxy Server.

• Global Caching Control: Setting cache: true turns on caching globally for the proxy. This means that, by default, the proxy will attempt to cache eligible requests and retrieve cached responses whenever possible, using the cache backend and settings configured in cache_params.
• Disabling Caching: Setting cache: false completely disables caching throughout the proxy. No responses will be cached, and the proxy will always forward requests to the backend LLM provider, regardless of any cache_params configuration.
• cache_params Requirement: If you set cache: true, you must also configure the cache_params section in litellm_settings. cache_params is where you specify the type of cache backend (Redis, memory, S3, etc.) and its connection details. If cache: true but cache_params is missing or incorrectly configured, the proxy may fail to start or caching might not function correctly.
• Default Off (Often): Caching is often disabled by default in many distributions of LiteLLM Proxy, meaning cache is typically set to false initially. You must explicitly enable caching by setting cache: true and configuring cache_params if you want to use caching.

Example YAML:

litellm_settings:
  cache: true # Master switch: Enabling caching globally
  cache_params: # Required: Cache backend configuration
    type: "redis"
    host: "cache.example.com"
    port: 6379
    password: "your_redis_password"
    # ... other cache_params ...
  # ... other litellm_settings ...

Example Environment Variable: N/A (YAML-only parameter)

Note: cache: true is just the master switch. The actual behavior of caching (what gets cached, how long it's cached, etc.) is further controlled by the settings within the cache_params section.

cache_params (litellm_settings)

YAML Key: cache_params

Type: Object (Mapping)

Environment Variable: N/A

Default Value: None (Required if cache: true)

Description: The cache_params parameter, within litellm_settings, is a required mapping (object) when you have enabled caching by setting cache: true. cache_params is where you configure the details of your cache backend, specifying:

• type: The type of cache backend to use (Redis, memory, S3, Qdrant, etc.).
• Backend-Specific Settings: Connection details and configurations that are specific to the chosen type of cache backend (e.g., Redis host, port, password; S3 bucket name, region, credentials).
• Common Cache Settings: General caching behavior settings like supported_call_types, mode, and ttl.

If cache: true is set, but cache_params is missing, incomplete, or incorrectly configured, the proxy may fail to start, or caching functionality may not work as expected.

Example YAML (Redis Cache Configuration):

litellm_settings:
  cache: true # Caching is enabled
  cache_params: # Cache backend configuration
    type: "redis" # Using Redis as cache backend
    host: "cache.example.com" # Redis server hostname
    port: 6379 # Redis server port
    password: "your_redis_password" # Redis password (if required)
    namespace: "litellm.proxy.cache" # Namespace for Redis keys
    supported_call_types: ["acompletion", "aembedding"] # Caching only chat completions and embeddings
    mode: "default_off" # Caching is opt-in per request
    ttl: 600 # Cache TTL of 600 seconds (10 minutes)
  # ... other litellm_settings ...

Example Environment Variable: N/A (YAML-only section header)

Note: The specific parameters within cache_params depend heavily on the type of cache backend you choose. Refer to the documentation for each cache type (Redis, S3, Qdrant, etc.) for details on the required and optional settings.

type (cache_params)

YAML Key: type

Type: String

Environment Variable: N/A

Default Value: N/A (Required Parameter within cache_params when cache: true)

Description: The type parameter, nested within cache_params, is required and specifies the type of cache backend that the LiteLLM Proxy should use for caching responses. This parameter determines where and how the proxy will store and retrieve cached data. Valid values for type include:

• "redis": Configures the proxy to use a Redis database as the cache backend. Redis is a popular in-memory data store, well-suited for caching due to its speed and persistence.
• "memory": Uses in-memory caching. Cached responses are stored in the proxy server's RAM. This is the simplest option to configure, and very fast, but the cache is not persistent (data is lost if the proxy restarts) and has limited capacity.
• "s3": Uses AWS S3 (Simple Storage Service) as the cache backend. S3 is a scalable object storage service, suitable for large caches and persistent storage, especially in AWS environments.
• "qdrant": Enables Qdrant semantic caching. This is an advanced caching strategy that uses a Qdrant vector database to store and retrieve cached responses based on the semantic similarity of prompts, rather than exact string matching.
• "disk": Uses a local disk-based cache. Cached responses are stored as files on the proxy server's local file system. Provides persistence between restarts but may be slower than in-memory or Redis.
• "hosted": Uses a hosted cache managed by the LiteLLM team (at api.litellm.ai). This is a convenient option for fast caching without needing to manage your own infrastructure.

Example YAML (Redis Cache Type):

litellm_settings:
  cache: true
  cache_params:
    type: "redis" # Setting cache type to Redis
    host: "cache.example.com"
    port: 6379
    # ... other Redis-specific settings ...

Example Environment Variable: N/A (YAML-only parameter)

Note: The choice of type depends on your application's requirements, scale, persistence needs, and infrastructure. "redis" is often a good balance of performance and persistence for many production setups. "memory" is suitable for testing or simple, non-persistent caching. "s3" and "qdrant" are for more advanced, large-scale, or semantic caching scenarios. "disk" is a persistent alternative when Redis or S3 are not feasible. "hosted" provides convenience for quick setup.

Redis-Specific Settings (within cache_params)

(Introductory text explaining Redis-specific settings)

host (cache_params -> Redis)

YAML Key: host

Type: String

Environment Variable: N/A

Default Value: N/A (Required when cache_params.type: "redis")

Description: The host parameter, within cache_params when using type: "redis", specifies the hostname or IP address of your Redis server. This is the network address where the LiteLLM Proxy will connect to access the Redis database for caching.

• Redis Server Address: Provide the hostname (e.g., "redis.example.com") or IP address (e.g., "192.168.1.100") of your Redis instance.
• Local or Remote Redis: The Redis server can be running locally on the same machine as the proxy, or it can be a remote Redis instance accessible over the network.
• Required for Redis Cache: This parameter is required when you configure cache_params.type: "redis". If host is missing, the proxy will not be able to connect to Redis and caching will not function correctly.

Example YAML (Redis host):

litellm_settings:
  cache: true
  cache_params:
    type: "redis"
    host: "cache.example.com" # Specifying Redis server hostname
    port: 6379
    # ... other Redis settings ...

Example Environment Variable: N/A (YAML-only parameter)

port (cache_params -> Redis)

YAML Key: port

Type: Integer

Environment Variable: N/A

Default Value: 6379 (Standard default Redis port)

Description: The port parameter, within cache_params and when type: "redis", specifies the port number on which your Redis server is listening for connections.

• Redis Port Number: Provide the port number of your Redis instance. The standard default Redis port is 6379, and if your Redis server is using the default port, you can often omit this parameter as it will default to 6379.
• Required for Non-Default Ports: If your Redis server is running on a non-default port, you must specify the correct port value.
• Integer Value: port must be an integer representing a valid port number.

Example YAML (Redis port):

litellm_settings:
  cache: true
  cache_params:
    type: "redis"
    host: "cache.example.com"
    port: 6380 # Specifying a non-default Redis port (e.g., 6380)
    # ... other Redis settings ...

Example Environment Variable: N/A (YAML-only parameter)

password (cache_params -> Redis)

YAML Key: password

Type: String

Environment Variable: N/A

Default Value: No default value (no password assumed by default).

Description: The password parameter, under cache_params when using type: "redis", is used to provide the password for your Redis server, if your Redis instance is configured with password-based authentication.

• Redis Authentication: If your Redis server requires a password for client connections, you must provide the correct password using this parameter.
• Security Best Practice: It is highly recommended to secure your Redis instance with a password in production environments to prevent unauthorized access to your cache data.
• Omit for Passwordless Redis: If your Redis server does not require a password (e.g., in a development or trusted network environment), you can omit this parameter. Do not provide a password if your Redis server is not configured to require one, as this could cause connection errors.

Example YAML (Redis password):

litellm_settings:
  cache: true
  cache_params:
    type: "redis"
    host: "cache.example.com"
    port: 6379
    password: "your_redis_password" # Providing Redis password for authentication
    # ... other Redis settings ...

Example Environment Variable: N/A (YAML-only parameter)

Security Note: For enhanced security, consider storing the Redis password in an environment variable or a secure secret management system and referencing it in your config.yaml using "os.environ/REDIS_PASSWORD" instead of hardcoding the password directly in the YAML file.

namespace (cache_params -> Redis)

YAML Key: namespace

Type: String

Environment Variable: N/A

Default Value: "litellm.caching.caching" (Default namespace used by LiteLLM Caching).

Description: The namespace parameter, within cache_params and when type: "redis", specifies a prefix or namespace that will be added to all keys used by LiteLLM Proxy when storing data in Redis.

• Key Collision Prevention: Using a namespace is crucial when your Redis instance is shared with other applications or services. It prevents key collisions by ensuring that LiteLLM Proxy's cache keys are prefixed with a unique namespace, isolating them from keys used by other applications in the same Redis database.
• Organization and Management: Namespaces can also improve the organization and manageability of your Redis cache, making it easier to identify and manage keys related to the LiteLLM Proxy.
• Default Namespace: If you don't provide a namespace, LiteLLM will use a default namespace, typically "litellm.caching.caching". While the default namespace might be sufficient for many setups, it's still best practice to explicitly define a custom namespace, especially in shared Redis environments.
• Custom Namespace String: You can set namespace to any string value you choose. Choose a string that is descriptive and unique to your LiteLLM Proxy deployment.

Example YAML (Redis namespace):

litellm_settings:
  cache: true
  cache_params:
    type: "redis"
    host: "cache.example.com"
    port: 6379
    password: "your_redis_password"
    namespace: "my-litellm-proxy-cache" # Setting a custom namespace for Redis keys
    # ... other Redis settings ...

Example Environment Variable: N/A (YAML-only parameter)

Recommendation: Always set a custom namespace when using Redis caching in production or shared Redis environments. Choose a namespace that is specific to your LiteLLM Proxy deployment to avoid potential key conflicts.

redis_startup_nodes (cache_params -> Redis)

YAML Key: redis_startup_nodes

Type: List of Objects (List of mappings)

Environment Variable: N/A

Default Value: N/A (Optional, only used for Redis Cluster)

Description: The redis_startup_nodes parameter, within cache_params when type: "redis", is used specifically when connecting to a Redis Cluster. Redis Cluster is a distributed Redis setup that provides high availability and scalability. redis_startup_nodes is a list of objects, where each object defines a Redis Cluster node that the proxy can use to bootstrap the connection to the cluster.

• Redis Cluster Bootstrapping: When connecting to a Redis Cluster, the proxy needs to initially connect to one or more startup nodes in the cluster to discover the cluster topology and routing information. redis_startup_nodes provides the addresses of these nodes.
• List of Nodes: The value of redis_startup_nodes should be a list, where each item in the list is a mapping (object) containing:
  • host: The hostname or IP address of a Redis Cluster node.
  • port: The port number of the Redis Cluster node.
• Multiple Startup Nodes: You can provide multiple startup nodes in the list for redundancy and fault tolerance during cluster discovery. The proxy will attempt to connect to the first available node in the list.
• Cluster-Specific Setting: redis_startup_nodes is only used for Redis Cluster setups. Do not use it when connecting to a standalone Redis server or Redis Sentinel.

Example YAML (Redis Cluster redis_startup_nodes):

litellm_settings:
  cache: true
  cache_params:
    type: "redis"
    redis_startup_nodes: # Defining Redis Cluster startup nodes
      - host: "redis-node-1.example.com" # Address of first startup node
        port: 7001
      - host: "redis-node-2.example.com" # Address of second startup node (redundancy)
        port: 7002
    # ... other Redis settings ...

In this example, the proxy is configured to connect to a Redis Cluster. It is provided with two startup nodes: redis-node-1.example.com:7001 and redis-node-2.example.com:7002. The proxy will use these to discover the cluster and establish connections to all nodes in the cluster.

Example Environment Variable: N/A (YAML-only parameter)

Note: When using Redis Cluster, you typically do not need to specify host and port directly under cache_params. Instead, you should use redis_startup_nodes to define the cluster nodes. Also, you generally do not need to use service_name or sentinel_nodes when connecting to a Redis Cluster.

service_name (cache_params -> Redis)

YAML Key: service_name

Type: String

Environment Variable: N/A

Default Value: N/A (Required when using Redis Sentinel)

Description: The service_name parameter, within cache_params and when type: "redis", is used specifically when connecting to a Redis Sentinel setup. Redis Sentinel provides high availability for Redis by monitoring master and replica instances and performing automatic failover in case of master failures. service_name specifies the name of the Redis Sentinel service that the proxy should connect to.

• Redis Sentinel Service Name: Provide the service name of your Redis Sentinel setup. This is the name configured for your master Redis instance within the Sentinel configuration.
• Sentinel-Specific Setting: service_name is only used for Redis Sentinel setups. Do not use it when connecting to a standalone Redis server or Redis Cluster.
• High Availability: Redis Sentinel provides high availability by automatically promoting a replica to master if the current master fails. The proxy, when configured with Sentinel, will automatically connect to the current master instance as managed by Sentinel.

Example YAML (Redis Sentinel service_name):

litellm_settings:
  cache: true
  cache_params:
    type: "redis"
    service_name: "mymaster" # Specifying Redis Sentinel service name
    sentinel_nodes: # Defining Sentinel node addresses
      - ["redis-sentinel1.example.com", 26379]
      - ["redis-sentinel2.example.com", 26379]
    # ... other Redis settings ...

In this example, the proxy is configured to connect to a Redis Sentinel setup named "mymaster".

Example Environment Variable: N/A (YAML-only parameter)

Note: When using Redis Sentinel, you typically do not need to specify host and port directly under cache_params. Instead, you should use service_name and sentinel_nodes to define the Sentinel setup. Also, you generally do not need to use redis_startup_nodes when connecting via Redis Sentinel.

sentinel_nodes (cache_params -> Redis)

YAML Key: sentinel_nodes

Type: List of Lists (List of [host, port] pairs)

Environment Variable: N/A

Default Value: N/A (Required when using Redis Sentinel)

Description: The sentinel_nodes parameter, within cache_params and when type: "redis", is also used specifically for Redis Sentinel setups. It is a list of lists, where each inner list is a pair of [host, port] representing the address of a Redis Sentinel node.

• Sentinel Node Addresses: Provide a list of addresses for your Redis Sentinel instances. You should list multiple Sentinel nodes for redundancy. The proxy will use these addresses to connect to the Sentinel cluster and discover the current master Redis instance.
• Host-Port Pairs: Each entry in the sentinel_nodes list should be a list containing two elements:
  • The hostname or IP address of a Sentinel node (string).
  • The port number of the Sentinel node (integer).
• Sentinel-Specific Setting: sentinel_nodes is only used for Redis Sentinel setups. Do not use it when connecting to a standalone Redis server or Redis Cluster.

Example YAML (Redis Sentinel sentinel_nodes):

litellm_settings:
  cache: true
  cache_params:
    type: "redis"
    service_name: "mymaster"
    sentinel_nodes: # Defining Redis Sentinel node addresses
      - ["redis-sentinel1.example.com", 26379] # Address of first Sentinel node
      - ["redis-sentinel2.example.com", 26379] # Address of second Sentinel node (redundancy)
    # ... other Redis settings ...

In this example, the proxy is configured with two Sentinel node addresses: redis-sentinel1.example.com:26379 and redis-sentinel2.example.com:26379. The proxy will use these to connect to the Sentinel cluster and discover the master Redis instance.

Example Environment Variable: N/A (YAML-only parameter)

Note: When using Redis Sentinel, you typically do not need to specify host and port directly under cache_params. Instead, use service_name and sentinel_nodes. Also, you generally do not need to use redis_startup_nodes when connecting via Redis Sentinel.

S3-Specific Settings (within cache_params)

(Introductory text explaining S3-specific settings)

s3_bucket_name (cache_params -> S3)

YAML Key: s3_bucket_name

Type: String

Environment Variable: N/A

Default Value: N/A (Required when cache_params.type: "s3")

Description: The s3_bucket_name parameter, within cache_params when using type: "s3", specifies the name of the AWS S3 bucket that the LiteLLM Proxy should use for caching responses.

• S3 Bucket Identifier: Provide the name of your S3 bucket as a string. Ensure that the bucket exists and that the AWS credentials you provide (via s3_aws_access_key_id and s3_aws_secret_access_key, or IAM roles) have read and write permissions to this bucket.
• Required for S3 Cache: This parameter is required when you configure cache_params.type: "s3". If s3_bucket_name is missing, the proxy will not be able to use S3 for caching.

Example YAML (S3 s3_bucket_name):

litellm_settings:
  cache: true
  cache_params:
    type: "s3"
    s3_bucket_name: "my-litellm-proxy-cache-bucket" # Specifying S3 bucket name
    s3_region_name: "us-west-2"
    s3_aws_access_key_id: "os.environ/AWS_ACCESS_KEY_ID"
    s3_aws_secret_access_key: "os.environ/AWS_SECRET_ACCESS_KEY"
    # ... other S3 settings ...

Example Environment Variable: N/A (YAML-only parameter)

Security Note: Ensure that your S3 bucket is properly secured with appropriate access policies to protect your cached data.

s3_region_name (cache_params -> S3)

YAML Key: s3_region_name

Type: String

Environment Variable: N/A

Default Value: N/A (Required when cache_params.type: "s3")

Description: The s3_region_name parameter, within cache_params when type: "s3", specifies the AWS region in which your S3 bucket is located. AWS regions are geographic locations where AWS services are hosted (e.g., "us-east-1", "eu-west-2", "ap-southeast-2").

• Bucket Region: Provide the AWS region name as a string (e.g., "us-west-2"). The region name must match the region where your s3_bucket_name bucket is actually created.
• Required for S3 Cache: This parameter is required when you configure cache_params.type: "s3". If s3_region_name is missing, the proxy will not be able to access your S3 bucket.

Example YAML (S3 s3_region_name):

litellm_settings:
  cache: true
  cache_params:
    type: "s3"
    s3_bucket_name: "my-litellm-proxy-cache-bucket"
    s3_region_name: "us-west-2" # Specifying AWS region of the S3 bucket
    s3_aws_access_key_id: "os.environ/AWS_ACCESS_KEY_ID"
    s3_aws_secret_access_key: "os.environ/AWS_SECRET_ACCESS_KEY"
    # ... other S3 settings ...

Example Environment Variable: N/A (YAML-only parameter)

Note: Specifying the correct s3_region_name is essential for the proxy to connect to your S3 bucket in the correct AWS geographic location. Incorrect region names will lead to connection failures.

s3_aws_access_key_id (cache_params -> S3)

YAML Key: s3_aws_access_key_id

Type: String or os.environ/... reference

Environment Variable: AWS_ACCESS_KEY_ID (if not using os.environ reference in YAML).

Default Value: N/A (Required when cache_params.type: "s3")

Description: The s3_aws_access_key_id parameter, within cache_params when type: "s3", is used to provide your AWS Access Key ID for authenticating with AWS S3. This is one part of your AWS credentials required to access your S3 bucket for caching.

• AWS Credential: Provide your AWS Access Key ID as a string. This is a sensitive credential, so it is highly recommended to use an environment variable reference (like "os.environ/AWS_ACCESS_KEY_ID") rather than hardcoding the access key directly in your config.yaml file.
• Required for S3 Cache (Unless Using IAM Roles): Unless your proxy is running in an AWS environment with an IAM role that grants it access to the S3 bucket, you must provide AWS credentials (Access Key ID and Secret Access Key) to authenticate with S3.
• Security Best Practice: Storing AWS credentials in environment variables or using IAM roles is significantly more secure than embedding them directly in configuration files.

Example YAML (S3 s3_aws_access_key_id with environment variable reference):

litellm_settings:
  cache: true
  cache_params:
    type: "s3"
    s3_bucket_name: "my-litellm-proxy-cache-bucket"
    s3_region_name: "us-west-2"
    s3_aws_access_key_id: "os.environ/AWS_ACCESS_KEY_ID" # Referencing AWS Access Key ID from environment variable
    s3_aws_secret_access_key: "os.environ/AWS_SECRET_ACCESS_KEY"
    # ... other S3 settings ...

Example Environment Variable:

export AWS_ACCESS_KEY_ID="AKIA..." # Setting AWS Access Key ID in environment

Security Note: Never hardcode your AWS Access Key ID directly in your config.yaml file. Always use environment variables or IAM roles for managing AWS credentials securely.

s3_aws_secret_access_key (cache_params -> S3)

YAML Key: s3_aws_secret_access_key

Type: String or os.environ/... reference

Environment Variable: AWS_SECRET_ACCESS_KEY (if not using os.environ reference in YAML).

Default Value: N/A (Required when cache_params.type: "s3")

Description: The s3_aws_secret_access_key parameter, within cache_params when type: "s3", is used to provide your AWS Secret Access Key for authenticating with AWS S3. This is the second part of your AWS credentials required to access your S3 bucket.

• AWS Credential: Provide your AWS Secret Access Key as a string. This is a highly sensitive credential. Always use an environment variable reference (like "os.environ/AWS_SECRET_ACCESS_KEY") to fetch the secret key from environment variables. Never hardcode the secret key directly in your config.yaml file.
• Required for S3 Cache (Unless Using IAM Roles): Similar to s3_aws_access_key_id, you must provide the Secret Access Key (or use IAM roles) for the proxy to authenticate with S3, unless your proxy is running in an AWS environment with an IAM role that provides S3 access.
• Security Imperative: Protecting your AWS Secret Access Key is of paramount importance. Treat it as a highly sensitive secret and follow AWS best practices for credential management.

Example YAML (S3 s3_aws_secret_access_key with environment variable reference):

litellm_settings:
  cache: true
  cache_params:
    type: "s3"
    s3_bucket_name: "my-litellm-proxy-cache-bucket"
    s3_region_name: "us-west-2"
    s3_aws_access_key_id: "os.environ/AWS_ACCESS_KEY_ID"
    s3_aws_secret_access_key: "os.environ/AWS_SECRET_ACCESS_KEY" # Referencing AWS Secret Access Key from environment variable
    # ... other S3 settings ...

Example Environment Variable:

export AWS_SECRET_ACCESS_KEY="abcd1234..." # Setting AWS Secret Access Key in environment

Security Warning: Absolutely never hardcode your AWS Secret Access Key directly in your config.yaml file or any version control system. Always use environment variables or IAM roles for secure credential management. Compromising your AWS Secret Access Key can have serious security implications for your AWS account.

s3_endpoint_url (cache_params -> S3)

YAML Key: s3_endpoint_url

Type: String

Environment Variable: N/A

Default Value: If not provided, defaults to the standard AWS S3 endpoint URL for the specified s3_region_name.

Description: The s3_endpoint_url parameter, within cache_params when type: "s3", is an optional parameter that allows you to specify a custom S3-compatible endpoint URL.

• Non-AWS S3 Services: s3_endpoint_url is primarily used when you are not using Amazon's own AWS S3 service, but rather an S3-compatible object storage service from another provider. Examples include:
  • Backblaze B2 Storage: Backblaze B2 is an S3-compatible cloud storage service.
  • Cloudflare R2 Storage: Cloudflare R2 is another S3-compatible object storage service.
  • MinIO: MinIO is an open-source object storage server that is S3-compatible and can be self-hosted.
• Custom Endpoints: By providing a custom s3_endpoint_url, you can instruct the LiteLLM Proxy to connect to these S3-compatible services instead of the default AWS S3 endpoint.
• AWS S3 Default Endpoint: If you are using AWS S3 and your bucket is in a standard AWS region, you can omit s3_endpoint_url. In this case, the proxy will automatically use the standard AWS S3 endpoint URL for the specified s3_region_name.

Example YAML (S3 s3_endpoint_url for Backblaze B2):

litellm_settings:
  cache: true
  cache_params:
    type: "s3"
    s3_bucket_name: "my-b2-bucket-name"
    s3_region_name: "us-west-004" # Backblaze B2 region code
    s3_aws_access_key_id: "os.environ/B2_APPLICATION_KEY_ID" # B2 application key ID
    s3_aws_secret_access_key: "os.environ/B2_APPLICATION_KEY" # B2 application key
    s3_endpoint_url: "https://s3.us-west-004.backblazeb2.com" # Custom endpoint URL for Backblaze B2
    # ... other S3 settings ...

In this example, the proxy is configured to use Backblaze B2 storage as the cache backend, specifying the custom s3_endpoint_url for Backblaze B2.

Example Environment Variable: N/A (YAML-only parameter)

Note: When using AWS S3 itself, you typically do not need to provide s3_endpoint_url, and can rely on the default AWS S3 endpoint for your specified s3_region_name. Only use s3_endpoint_url when connecting to non-AWS S3-compatible services.

Qdrant (Semantic Caching) Settings (within cache_params)

(Introductory text explaining Qdrant-specific settings for semantic caching)

qdrant_semantic_cache_embedding_model (cache_params -> Qdrant)

YAML Key: qdrant_semantic_cache_embedding_model

Type: String (Must be a model_name defined in your model_list)

Environment Variable: N/A

Default Value: N/A (Required when using Qdrant semantic cache)

Description: The qdrant_semantic_cache_embedding_model parameter, within cache_params when using type: "qdrant" or enabling semantic cache, specifies the model_name of an embedding model that the proxy should use to generate embeddings for cache keys in Qdrant.

• Embedding Model for Semantic Search: Semantic caching relies on vector embeddings to measure the semantic similarity between prompts. You need to designate an embedding model that the proxy will use to generate these embeddings.
• model_name Reference: The value of qdrant_semantic_cache_embedding_model must be a valid model_name that is already defined in your model_list section of config.yaml. This model_name should correspond to an embedding model (e.g., OpenAI's text-embedding-ada-002, or a local embedding model you have configured).
• Embedding Model Configuration: You must ensure that the model_name you specify here is indeed configured as an embedding model in your model_list, with appropriate litellm_params for calling the embedding API (e.g., API key, API base, etc.).
• Required for Semantic Cache: qdrant_semantic_cache_embedding_model is required when you are using Qdrant semantic caching (either type: "qdrant" or by enabling semantic cache features). If you are using Qdrant semantic cache, but this parameter is missing, the proxy will not be able to generate embeddings for cache keys and semantic caching will not function.

Example YAML (Qdrant qdrant_semantic_cache_embedding_model):

litellm_settings:
  cache: true
  cache_params:
    type: "qdrant" # Using Qdrant semantic cache
    qdrant_semantic_cache_embedding_model: "openai-embedding" # Using "openai-embedding" model for generating embeddings
    qdrant_collection_name: "litellm_cache_collection"
    # ... other Qdrant settings ...
model_list:
  - model_name: openai-embedding # Defining the embedding model used for semantic cache
    litellm_params:
      model: openai/text-embedding-ada-002
      api_key: "os.environ/OPENAI_API_KEY"
    model_info:
      mode: "embedding" # Important: Marking this model as "embedding" mode
      # ... other model_info ...
  - model_name: gpt-3.5-turbo # Example completion model
    litellm_params:
      model: azure/gpt-turbo-small-eu
      api_key: "os.environ/AZURE_API_KEY_EU"
      # ... other litellm_params ...
    model_info:
      mode: "completion" # Marking this model as "completion" mode
      # ... other model_info ...

In this example, qdrant_semantic_cache_embedding_model: "openai-embedding" specifies that the model named "openai-embedding" (which is also defined in model_list as an embedding model using OpenAI's text-embedding-ada-002) should be used to generate embeddings for the Qdrant semantic cache.

Example Environment Variable: N/A (YAML-only parameter)

Note: It's crucial that the model_name you specify for qdrant_semantic_cache_embedding_model actually corresponds to a properly configured embedding model in your model_list and that its model_info.mode is set to "embedding".

qdrant_collection_name (cache_params -> Qdrant)

YAML Key: qdrant_collection_name

Type: String

Environment Variable: N/A

Default Value: N/A (Required when using Qdrant semantic cache)

Description: The qdrant_collection_name parameter, within cache_params when type: "qdrant" or enabling semantic cache, specifies the name of the Qdrant collection that the LiteLLM Proxy should use to store vector embeddings for semantic caching.

• Qdrant Collection Identification: Provide the name of your Qdrant collection as a string. Ensure that the collection exists in your Qdrant database. If the collection does not exist, LiteLLM may attempt to create it automatically (depending on Qdrant configuration and permissions), but it's generally recommended to create the collection beforehand and configure its settings appropriately (e.g., vector dimensions, distance metric).
• Required for Qdrant Semantic Cache: qdrant_collection_name is required when you are using Qdrant semantic caching. If this parameter is missing, the proxy will not know which Qdrant collection to use for storing and searching embeddings.

Example YAML (Qdrant qdrant_collection_name):

litellm_settings:
  cache: true
  cache_params:
    type: "qdrant" # Using Qdrant semantic cache
    qdrant_semantic_cache_embedding_model: "openai-embedding"
    qdrant_collection_name: "litellm_cache_collection" # Specifying Qdrant collection name
    qdrant_api_base: "http://localhost:6334"
    # ... other Qdrant settings ...

In this example, "litellm_cache_collection" is specified as the name of the Qdrant collection to be used for semantic caching.

Example Environment Variable: N/A (YAML-only parameter)

Best Practice: It's recommended to use a dedicated Qdrant collection specifically for the LiteLLM Proxy's semantic cache. Choose a descriptive and unique collection name to avoid conflicts with other data in your Qdrant database.

qdrant_quantization_config (cache_params -> Qdrant)

YAML Key: qdrant_quantization_config

Type: String

Environment Variable: N/A

Default Value: No quantization configured by default (vectors are stored as full-precision floats).

Description: The qdrant_quantization_config parameter, within cache_params when type: "qdrant" or enabling semantic cache, allows you to specify a quantization method for vector embeddings stored in Qdrant. Quantization is a technique to reduce the memory footprint and potentially improve search speed of vector embeddings in a vector database like Qdrant.

• Vector Quantization Options: Currently, the documentation mentions support for "binary" quantization. Other quantization methods might be supported or added in future versions of LiteLLM.
  • "binary": Binary quantization reduces the size of vectors by representing them using binary codes (0s and 1s) instead of full-precision floating-point numbers. This can significantly reduce storage space and improve search performance, especially for very large datasets.
• Performance Trade-offs: Quantization typically involves a trade-off between storage/performance gains and potential loss of precision in the vector embeddings. Binary quantization is a more aggressive form of quantization and may result in some loss of accuracy in semantic similarity search compared to using full-precision vectors.
• Optional Optimization: qdrant_quantization_config is optional. If you omit this parameter, vectors will be stored in Qdrant using their original full-precision floating-point representation (no quantization). Quantization is typically used for optimizing storage and performance when dealing with very large caches or when memory/performance is a critical concern.

Example YAML (Qdrant qdrant_quantization_config):

litellm_settings:
  cache: true
  cache_params:
    type: "qdrant" # Using Qdrant semantic cache
    qdrant_semantic_cache_embedding_model: "openai-embedding"
    qdrant_collection_name: "litellm_cache_collection"
    qdrant_quantization_config: "binary" # Enabling binary quantization for Qdrant vectors
    qdrant_api_base: "http://localhost:6334"
    # ... other Qdrant settings ...

In this example, qdrant_quantization_config: "binary" enables binary quantization for vectors stored in the Qdrant cache.

Example Environment Variable: N/A (YAML-only parameter)

Recommendation: Consider using quantization (e.g., "binary") if you are planning to build a very large semantic cache with Qdrant, where storage space and search performance are critical. If accuracy is paramount and you have sufficient resources, you can omit qdrant_quantization_config to use full-precision vectors. Test and evaluate the impact of quantization on your specific use case to determine the optimal setting.

similarity_threshold (cache_params -> Qdrant)

YAML Key: similarity_threshold

Type: Number (Float)

Environment Variable: N/A

Default Value: N/A (Required when using Qdrant semantic cache)

Description: The similarity_threshold parameter, within cache_params when type: "qdrant" or enabling semantic cache, is a floating-point number that defines the cosine similarity threshold for determining a cache hit in Qdrant semantic caching.

• Semantic Similarity Matching: Semantic caching, unlike exact-match caching, retrieves cached responses based on the semantic similarity of the incoming prompt to prompts that are already in the cache. Cosine similarity is a common metric used to measure semantic similarity between vector embeddings.
• Threshold Value: similarity_threshold sets the minimum cosine similarity score required for a cached response to be considered a match for an incoming request. The cosine similarity score ranges from 0 to 1, where:
  • 1.0: Perfect cosine similarity (vectors are identical in direction).
  • 0.0: No cosine similarity (vectors are orthogonal).
  • Values closer to 1.0: Indicate higher semantic similarity.
  • Values closer to 0.0: Indicate lower semantic similarity.
• Tuning for Cache Hit Rate vs. Relevance: The similarity_threshold value directly impacts the cache hit rate and the relevance of cached responses:
  • Higher threshold (closer to 1.0): Makes semantic matching more strict. Only prompts that are very semantically similar to cached prompts will result in a cache hit. This leads to a lower cache hit rate but potentially higher relevance of cached responses (as they are very close in meaning to the current request).
  • Lower threshold (closer to 0.0): Makes semantic matching more lenient. Prompts that are somewhat semantically similar can trigger a cache hit. This results in a higher cache hit rate but potentially lower relevance of cached responses (as they might be less semantically aligned with the current request).
• Experimentation and Tuning: The optimal similarity_threshold value depends on your specific application and use case. You will likely need to experiment and tune this value to find a balance between cache hit rate and the desired level of semantic relevance for cached responses. A typical starting point might be around 0.8 or 0.85.

Example YAML (Qdrant similarity_threshold):

litellm_settings:
  cache: true
  cache_params:
    type: "qdrant" # Using Qdrant semantic cache
    qdrant_semantic_cache_embedding_model: "openai-embedding"
    qdrant_collection_name: "litellm_cache_collection"
    similarity_threshold: 0.8 # Setting cosine similarity threshold to 0.8
    qdrant_api_base: "http://localhost:6334"
    # ... other Qdrant settings ...

In this example, similarity_threshold: 0.8 means that a cached response will only be considered a match if the cosine similarity between the incoming prompt's embedding and a cached prompt's embedding is 0.8 or higher.

Example Environment Variable: N/A (YAML-only parameter)

Recommendation: Start with a similarity_threshold around 0.8 and then adjust it based on your testing and evaluation of cache hit rates and the semantic relevance of retrieved cached responses for your application. Lower the threshold to increase cache hits (potentially at the cost of relevance), and raise it to decrease cache hits but increase relevance.

Common Cache Settings (within cache_params)

(Introductory text explaining common cache settings that apply regardless of the backend type)

supported_call_types (cache_params -> Common)

YAML Key: supported_call_types

Type: Array of Strings

Environment Variable: N/A

Default Value: If not set, a default list might be used (check documentation for default behavior in your LiteLLM Proxy version). Commonly defaults to ["acompletion", "atext_completion", "aembedding", "atranscription"].

Description: The supported_call_types parameter, within cache_params, is an array of strings that allows you to specify which types of API calls should be cached by the LiteLLM Proxy. This provides granular control over what gets cached and what does not.

• Selective Caching: By default, LiteLLM Proxy is capable of caching several types of API calls:
  • "acompletion": Asynchronous chat completion calls (corresponding to /chat/completions endpoint).
  • "atext_completion": Asynchronous text completion calls (/completions endpoint).
  • "aembedding": Asynchronous embedding calls (/embeddings endpoint).
  • "atranscription": Asynchronous audio transcription calls (/audio/transcriptions endpoint).
• Customize Caching Scope: Using supported_call_types, you can customize which of these call types are actually cached. For example:
  • supported_call_types: ["acompletion", "aembedding"]: Only cache chat completions and embeddings, but not text completions or transcriptions.
  • supported_call_types: ["acompletion"]: Only cache chat completions, and disable caching for all other call types.
  • supported_call_types: []: Disable caching for all call types, even if cache: true is set. This is effectively a way to disable caching selectively without turning off the master cache switch.
• Performance Optimization: Use supported_call_types to optimize caching for your specific application's needs. You might choose to cache only the most frequently used or expensive call types, while excluding less critical or less cacheable calls.

Example YAML:

litellm_settings:
  cache: true
  cache_params:
    type: "redis"
    # ... Redis settings ...
    supported_call_types: ["acompletion", "aembedding"] # Only caching chat completions and embeddings
    # ... other cache settings ...

In this example, only asynchronous chat completion (acompletion) and embedding (aembedding) calls will be cached. Text completions (atext_completion) and transcriptions (atranscription) will not be cached, even though caching is generally enabled (cache: true).

Example Environment Variable: N/A (YAML-only parameter)

Note: If supported_call_types is not explicitly set in your config.yaml, LiteLLM Proxy may use a default list of call types that are enabled for caching (check the documentation for your specific version for the default list). However, it's best practice to explicitly configure supported_call_types to clearly define the scope of your caching strategy.

mode (cache_params -> Common)

YAML Key: mode

Type: String

Environment Variable: N/A

Default Value: "default_off" (Caching is opt-in per request by default).

Description: The mode parameter, within cache_params, controls the default caching behavior of the LiteLLM Proxy. It determines whether caching is enabled by default for all eligible requests, or if it needs to be explicitly enabled on a per-request basis.

• "default_off": Caching is opt-in per request. This is the default mode. In "default_off" mode, caching is disabled by default for all requests. To enable caching for a specific request, the client application must explicitly request caching, typically by including a caching: true parameter in the API call (e.g., in the request headers or body, depending on the client library).
• "default_on": Caching is opt-out per request (Note: This mode might be less common in stable versions and might be considered more of a conceptual option; verify if "default_on" is actually supported in your version's documentation). In a hypothetical "default_on" mode, caching would be enabled by default for all eligible requests. To disable caching for a specific request, the client application would need to explicitly opt-out, likely using a parameter like caching: false in the API call.

Example YAML (Caching Mode: Default Off):

litellm_settings:
  cache: true
  cache_params:
    type: "redis"
    # ... Redis settings ...
    mode: "default_off" # Caching is opt-in per request (default mode)
    # ... other cache settings ...

Example Environment Variable: N/A (YAML-only parameter)

Recommendation: "default_off" mode is generally recommended for production environments and provides the most control. It ensures that caching is only applied when explicitly requested by the client application, giving you granular control over caching behavior and preventing unintended caching of sensitive or dynamic data. The "default_on" mode, if supported, might be useful for testing or development where you want caching to be enabled broadly, but is generally less suitable for production due to the lack of explicit control. Always check the documentation for your specific LiteLLM Proxy version to confirm the actually supported caching modes and their default behavior.

ttl (cache_params -> Common)

YAML Key: ttl

Type: Integer (representing seconds)

Environment Variable: N/A

Default Value: No default TTL is specified in the provided documentation snippets. You may need to check the full documentation for the default TTL value in your specific LiteLLM Proxy version. If no default is documented, it's likely that cache entries might persist indefinitely unless explicitly evicted or the cache backend has its own eviction policies. It's highly recommended to explicitly set a ttl value.

Description: The ttl parameter, within cache_params, sets the time-to-live (TTL) in seconds for cache entries. TTL determines how long a cached response remains valid and will be served from the cache before it is considered expired.

• Cache Expiration: ttl is a fundamental parameter for cache management. It defines the maximum age of a cached response. After the specified TTL duration has elapsed since the response was cached, the cached entry is considered stale or expired.
• Cache Invalidation: When a cached response is expired due to TTL, the proxy will not use it for subsequent requests, even if the request matches the cache key. Instead, the proxy will forward the request to the backend LLM provider, get a fresh response, and potentially cache the new response (replacing the expired entry).
• Balancing Freshness and Performance: The ttl value represents a trade-off between:
  • Cache Hit Rate and Performance: Longer TTL values (e.g., hours or days) increase the likelihood of cache hits and improve performance, as cached responses remain valid for longer. However, they might serve stale or outdated information if the underlying LLM or data changes frequently.
  • Data Freshness and Accuracy: Shorter TTL values (e.g., minutes or seconds) ensure that responses are more up-to-date and accurate, reflecting recent changes in the LLM or data, but they reduce the cache hit rate and may increase API costs and latency.
• Appropriate TTL Value: The optimal ttl value depends heavily on your application's requirements for data freshness, the frequency of changes in the underlying LLM or data, and your performance/cost optimization goals. For relatively static data or tasks where slightly stale data is acceptable, you can use longer TTL values. For applications requiring highly dynamic or real-time data, use shorter TTL values or disable caching altogether.

Example YAML (Cache TTL of 10 minutes):

litellm_settings:
  cache: true
  cache_params:
    type: "redis"
    # ... Redis settings ...
    ttl: 600 # Setting cache TTL to 600 seconds (10 minutes)
    # ... other cache settings ...

In this example, cache entries will expire and be considered invalid after 600 seconds (10 minutes) from when they were initially cached.

Example Environment Variable: N/A (YAML-only parameter)

Recommendation: Always explicitly set a ttl value that is appropriate for your application's needs. If data freshness is critical, use shorter TTLs. If performance and cost reduction are prioritized and some staleness is acceptable, use longer TTLs. Monitor your cache hit rates and application behavior to fine-tune the ttl value for optimal performance and data accuracy. If no caching is desired for certain data, disable caching for those call types using supported_call_types or disable caching altogether with cache: false.

Callback Settings (callback_settings)

This section is used for fine-tuning of the individual callbacks configured in litellm_setting. You will be able to customize settings of selected callback.

callback_settings.otel.message_logging

YAML Key: callback_settings.otel.message_logging

Type: Boolean

Environment Variable: N/A

Default Value: true (Message logging in OpenTelemetry is enabled by default).

Description: The callback_settings.otel.message_logging parameter, within the top-level callback_settings section and specifically under the otel key, is a boolean flag that controls whether the OpenTelemetry (otel) callback should log message content and responses. This setting provides fine-grained control over what data is included in OpenTelemetry traces specifically for the OTEL callback.

• OTEL-Specific Message Logging Control: This parameter applies only to the OpenTelemetry logging integration. It does not affect other logging callbacks or the general turn_off_message_logging setting.
• Message Content in Traces (Default): By default (message_logging: true or if this parameter is omitted), the OpenTelemetry callback will log the full content of prompts (messages) and responses within the OpenTelemetry traces. This provides detailed context within traces, which is helpful for debugging and understanding the flow of conversations or requests.
• Disabling Message Content in OTEL Traces: Setting message_logging: false will instruct the OTEL callback not to log message content or responses in the traces it generates. Only high-level metadata (like request parameters, timestamps, etc.) will be included in the OTEL spans.
• Privacy or Trace Volume Reduction: You might disable message logging in OTEL traces for:
  • Privacy Reasons: If you are concerned about sensitive data appearing in your OpenTelemetry traces, especially if traces are exported to external systems or accessed by a wider audience.
  • Reduced Trace Volume: Omitting message content can significantly reduce the size and volume of OpenTelemetry traces, which can be beneficial for performance in high-throughput systems or when dealing with large trace datasets.

Example YAML:

litellm_settings:
  callbacks: ["otel"] # Enabling OpenTelemetry callback
callback_settings: # Fine-tuning callback behavior
  otel: # Settings specific to the "otel" callback
    message_logging: false # Disabling message content logging in OTEL traces

In this example, OpenTelemetry tracing is enabled for all requests (via callbacks: ["otel"]), but the message_logging: false setting under callback_settings.otel ensures that the actual text content of prompts and responses will not be included in the generated OTEL traces. Only metadata will be traced.

Example Environment Variable: N/A (YAML-only parameter)

Note: callback_settings.otel.message_logging: false provides a way to control message logging specifically for OpenTelemetry, without affecting other logging callbacks or the global turn_off_message_logging setting, which controls message logging across all callbacks.

general_settings Section: Proxy General Settings

The general_settings section contains a wide range of proxy-wide configurations, affecting overall proxy behavior, security, authentication, and database interactions. These are general settings that are not specific to a particular model or routing strategy.

general_settings Parameter Summary

Default Model Selection

completion_model

YAML Key: completion_model

Type: String (Must be a model_name defined in your model_list)

Environment Variable: N/A

Default Value: None (No default completion model is set). If not configured, requests without a model parameter will likely result in an error.

Description: The completion_model parameter, within general_settings, defines the default model_name (as defined in your model_list) that the proxy should use when a client application sends a completion or chat request (to endpoints like /v1/completions or /v1/chat/completions) without explicitly specifying a model parameter in the request body.

• Default Model for Unqualified Requests: This setting acts as a fallback. In practice, well-behaved API clients should always specify a model in their requests. However, completion_model ensures that if a request does arrive without a model, the proxy has a designated default model to use.
• Model Alias from model_list:** The value of completion_model must be a valid model_name that you have already defined in your model_list section of config.yaml. This ensures that the default model is one that the proxy is configured to serve.
• Ensuring a Default Behavior: Setting completion_model provides a safety net, guaranteeing that even if a client omits the model parameter, the proxy can still process the request using a predefined default model. This can be useful in scenarios where you want to enforce a specific model as the standard choice, or for simplifying client-side code in certain use cases.

Example YAML:

general_settings:
  completion_model: "gpt-3.5-turbo" # Setting "gpt-3.5-turbo" as the default completion model
  # ... other general_settings ...
model_list:
  - model_name: gpt-3.5-turbo # Defining "gpt-3.5-turbo" in model_list
    litellm_params:
      model: azure/gpt-turbo-small-eu
      api_key: "os.environ/AZURE_API_KEY_EU"
      # ... other litellm_params ...

In this configuration, if a client sends a request to /v1/chat/completions without including "model": "..." in the request body, the proxy will automatically route that request to the model deployment aliased as "gpt-3.5-turbo" (which, in this example, is configured to use an Azure OpenAI deployment).

Example Environment Variable: N/A (YAML-only parameter)

Note: While completion_model provides a default, it's generally best practice for client applications to always explicitly specify the model in their API requests to avoid ambiguity and ensure they are targeting the intended model.

embedding_model

YAML Key: embedding_model

Type: String (Must be a model_name defined in your model_list)

Environment Variable: N/A

Default Value: None (No default embedding model is set). If not configured, requests to /v1/embeddings without a model parameter will likely result in an error.

Description: The embedding_model parameter, within general_settings, defines the default model_name (from your model_list) that the proxy should use for embedding requests (to the /v1/embeddings endpoint) when the client request does not specify a model parameter.

• Default Embedder: Similar to completion_model, embedding_model sets a default for embedding-related API calls. If a client sends a request to /v1/embeddings omitting the model parameter, the proxy will assume this default model for generating embeddings.
• Model Alias from model_list:** The value of embedding_model must be a valid model_name that is already defined in your model_list section of config.yaml. This ensures that the default embedder is one that the proxy is configured to manage and serve.
• Directing Embedding Calls: Setting embedding_model is useful if you want to direct all embedding-related API calls to a specific embedding model without requiring clients to always specify the model name in their requests. This can simplify client code and enforce a consistent embedding strategy.

Example YAML:

general_settings:
  embedding_model: "text-embedding-ada-002" # Setting "text-embedding-ada-002" as the default embedding model
  # ... other general_settings ...
model_list:
  - model_name: text-embedding-ada-002 # Defining "text-embedding-ada-002" in model_list
    litellm_params:
      model: openai/text-embedding-ada-002
      api_key: "os.environ/OPENAI_API_KEY"
      # ... other litellm_params ...

In this configuration, if a client sends a request to /v1/embeddings without specifying "model": "..." in the request body, the proxy will automatically use the model deployment aliased as "text-embedding-ada-002" as the default embedder.

Example Environment Variable: N/A (YAML-only parameter)

Note: While embedding_model provides a default embedder, it's generally good practice for API clients to explicitly specify the model in their /v1/embeddings requests to avoid reliance on defaults and ensure they are using the intended embedding model.

image_generation_model

YAML Key: image_generation_model

Type: String (Must be a model_name defined in your model_list)

Environment Variable: N/A

Default Value: None (No default image generation model is set). If not configured, requests to /images/generations may not be handled or might result in an error.

Description: The image_generation_model parameter, within general_settings, defines the default model name (from your model_list) that the proxy should use for image generation requests (to the /images/generations endpoint). Crucially, this setting overrides any model parameter provided in the client's image generation request payload.

• Forcing a Specific Image Model: When image_generation_model is set, the proxy will always route image generation requests to this specified model, regardless of whether the client includes a model parameter in their request. This effectively enforces the use of a particular image generation model for all image-related API calls.
• Model Alias from model_list:** The value of image_generation_model must be a valid model_name that you have already defined in your model_list section of config.yaml.
• Enforcing a Single Image Model Policy: This setting is useful when you want to strictly control which image generation model is used by your application and ensure that all image requests are routed to a specific, designated model. For example, you might want to force all image generation to use DALL-E 2 and prevent users from requesting other image models via the proxy.

Example YAML:

general_settings:
  image_generation_model: "dalle-2" # Setting "dalle-2" as the default and *only* image generation model
  # ... other general_settings ...
model_list:
  - model_name: dalle-2 # Defining "dalle-2" in model_list
    litellm_params:
      model: openai/dall-e-2
      api_key: "os.environ/OPENAI_API_KEY"
      # ... other litellm_params ...

In this configuration, all requests to the /images/generations endpoint, regardless of any model parameter they might include, will be routed to the model deployment aliased as "dalle-2".

Example Environment Variable: N/A (YAML-only parameter)

Note: Setting image_generation_model effectively ignores any model parameter provided by the client in image generation requests. This enforces a strict policy of using the configured default model for all image-related API calls. If you want to allow clients to choose image models, do not set image_generation_model.

moderation_model

YAML Key: moderation_model

Type: String (Must be a model_name defined in your model_list)

Environment Variable: N/A

Default Value: None (No default moderation model is set). If not configured, moderation requests might not be handled correctly or may result in an error.

Description: The moderation_model parameter, within general_settings, defines the default model_name (from your model_list) that the proxy should use for moderation requests (to the /moderations endpoint).

• Default Moderation Model: When a client application sends a request to the /moderations endpoint, which is typically used for content moderation checks, the proxy will use the model specified by moderation_model to perform the moderation, unless a different moderation model is explicitly configured in the request itself (Note: OpenAI's standard /moderations endpoint does not typically accept a model parameter in the request body; this setting is relevant if LiteLLM's proxy supports multiple moderation models or custom moderation endpoints in the future).
• Model Alias from model_list:** The value of moderation_model must be a valid model_name that is already defined in your model_list section of config.yaml. This should be a model that is suitable for content moderation tasks (e.g., OpenAI's moderation models or similar).
• Ensuring a Moderation Model is Used: Setting moderation_model ensures that the proxy has a designated model to use for moderation checks, even if the client's request does not explicitly specify one.

Example YAML:

general_settings:
  moderation_model: "openai-moderation" # Setting "openai-moderation" as the default moderation model
  # ... other general_settings ...
model_list:
  - model_name: openai-moderation # Defining "openai-moderation" in model_list
    litellm_params:
      model: openai/text-moderation-stable
      api_key: "os.environ/OPENAI_API_KEY"
      # ... other litellm_params ...

In this configuration, when a client sends a request to /moderations without a model parameter, the proxy will use the model deployment aliased as "openai-moderation" for performing the content moderation check.

Example Environment Variable: N/A (YAML-only parameter)

Note: In practice, OpenAI's standard /moderations endpoint does not typically allow specifying a model parameter in the request body. The moderation_model setting in general_settings is more relevant if LiteLLM Proxy were to support multiple moderation models or custom moderation endpoints in the future. Currently, it mainly serves to define a default moderation model within your proxy configuration.

infer_model_from_keys

YAML Key: infer_model_from_keys

Type: Boolean

Environment Variable: N/A

Default Value: false (Model is not inferred from API keys by default).

Description: The infer_model_from_keys parameter, within general_settings, is a boolean flag that, when set to true, instructs the proxy to attempt to infer the intended model for a request based on the API key (virtual key) used in the request.

• API Key-Based Model Routing: This feature is relevant in scenarios where you issue different API keys that are scoped to specific models or model groups. For example, you might issue one set of API keys that are only authorized to access GPT-4 models, and another set for GPT-3.5 models.
• Key-Based Inference Logic: When infer_model_from_keys: true, the proxy will examine the API key used in an incoming request. If the API key is configured to be associated with a specific model (or set of models), the proxy will attempt to automatically route the request to that model, even if the client specifies a different model in the request body.
• Prioritizing Key-Based Routing: If model inference from the API key is successful, it will take precedence over any model parameter provided by the client. In essence, the API key becomes the primary factor in determining the model to be used. If model inference from the key is not possible (e.g., the key is not associated with a specific model), the proxy will then fall back to using the model parameter from the client request (or the completion_model/embedding_model defaults if no model is specified in the request either).
• Multi-Tenant Scenarios: infer_model_from_keys is particularly useful in multi-tenant environments where you want to enforce model access control at the API key level, ensuring that each user or team is restricted to using only the models they are authorized to access.

Example YAML:

general_settings:
  infer_model_from_keys: true # Enabling model inference from API keys
  # ... other general_settings ...
# Example model_list (showing model_name aliases, not directly related to infer_model_from_keys):
model_list:
  - model_name: gpt-4-key-scoped
    litellm_params:
      model: azure/gpt-4-deployment
      api_key: "os.environ/AZURE_GPT4_API_KEY" # This key would be associated with gpt-4 in key management
  - model_name: gpt-3.5-turbo-key-scoped
    litellm_params:
      model: openai/gpt-3.5-turbo
      api_key: "os.environ/OPENAI_GPT35_API_KEY" # This key would be associated with gpt-3.5-turbo in key management

In this configuration, if a request comes in with an API key that is internally marked as being associated with "gpt-4-key-scoped", the proxy will route it to the Azure GPT-4 deployment (as defined by the gpt-4-key-scoped model alias), regardless of what model parameter the client might have included in the request body. If a request comes with a key associated with "gpt-3.5-turbo-key-scoped", it will be routed to the OpenAI GPT-3.5 Turbo deployment, and so on.

Example Environment Variable: N/A (YAML-only parameter)

Note: infer_model_from_keys: true is typically used in conjunction with the proxy's virtual key management system. You would need to configure your API keys (via the Admin UI or API) to associate them with specific models or model groups for this feature to be effective. If you are not using virtual keys or have not configured key-to-model associations, enabling infer_model_from_keys: true will likely have no effect.

Master Key & Authentication

This subsection within general_settings focuses on configuring the master API key, which provides administrative access to the proxy, and enabling/configuring various authentication methods (JWT, OAuth2) for enhanced security.

master_key (general_settings)

YAML Key: master_key

Type: String (Secret, high-entropy string)

Environment Variable: PROXY_MASTER_KEY

Default Value: None (Proxy will not start in key-managed mode if master_key is not set).

Description: The master_key parameter, within general_settings, defines the master API key for your LiteLLM Proxy Server. This is a critical security credential that acts as an administrator key for the proxy.

• Admin Authentication: The master_key is used to authenticate access to privileged admin endpoints of the proxy, such as:
  • Generating new API keys (virtual keys).
  • Listing, updating, or deleting API keys.
  • Viewing usage statistics and spend reports.
  • Accessing the Admin UI (if enabled).
• Security Importance: The master_key is a highly sensitive secret. Protect it carefully. Anyone who possesses the master_key gains administrative control over your LiteLLM Proxy instance.
• High Entropy String: The master_key value should be a strong, randomly generated string with high entropy (i.e., difficult to guess). It is often recommended to generate a key that resembles an OpenAI API key (starting with sk- followed by a long random string), but any sufficiently random string will work.
• Required for Key-Managed Mode: Setting master_key is essential if you want to run the LiteLLM Proxy in "key-managed" mode, where the proxy manages API keys (virtual keys), enforces rate limits, tracks usage, and provides admin functionalities. If master_key is not set, the proxy may run in a limited or "single-key" mode, without key management features.