Large Language Models (LLMs)

Karini AI supports integrations with the following Large Language Model providers and custom models. Using these models, users can create model endpoints in the Karini AI model hub.

  1. Amazon Bedrock

  2. OpenAI

  3. Azure OpenAI

  4. Databricks

  5. Anyscale

  6. Amazon SageMaker

Add New Model Endpoint

To add a new model endpoint to the model hub, do the following:

  1. On the Model Endpoints menu, select Large language model endpoints(LLM) tab and click Add New.

  2. Select a model provider and associated model id in the list.

  3. User has option to override default configurations such as temperature, max tokens and pricing.

  4. By default, the organization level credentials are used to access the model. User can User can optionally overwrite credentials with a new set of model credentials.

  5. User can test the model endpoint request and response by using the Test endpoint button.

Review Model Endpoints

User can review the created model endpoints under Large language model endpoints(LLM) tab. It includes following information:

  1. Model provider and model id

  2. Max tokens, Min tokens and Temperature: The default values are displayed based on model specifications from the model provider. User has the ability to override them.

  3. Model Price: The default price displays public pricing of the model inference per 1000 input and output tokens. This price is used in Karini AI Dashboards to calculate cost. User has the ability to override this price if needed - such as in case of special pricing agreement with the model provider.

  4. Link to view the the recipes and prompts in which the model endpoint is used.

  5. Link to view the model information including the cost and usage dashboard for the model endpoint.

Available LLM Configurations

The following table describes LLMs that are available for integration with Karini AI model hub. It also includes links to model provider reference documentation offering detailed information on model specifications, usage instructions, and API endpoints for effective integration and utilization.

Provider
Models
Config Parameters
Reference

Amazon Bedrock

  1. Meta Llama3 8B Instruct

  2. Meta Llama3 70B Instruct

  3. Anthropic Claude 3 Haiku

  4. Anthropic Claude 3 Sonnet

  5. Anthropic Claude v2.1

  6. Anthropic Claude v2

  7. Mistral Mistral 7B Instruct

  8. Mistral Mixtral 8x7B Instruct

  9. Mistral Mistral Large

  10. Cohere Command R Plus

  11. Cohere Command R

  12. Cohere Command Text

  13. Cohere Command Light Text

  14. Amazon Titan Text Express

  15. Amazon Titan Text Lite

  16. Jurrasic-2 Ultra

  17. Jurasic-2 Mid

  1. Temperature

  2. Max Tokens

Azure OpenAI

  1. GPT 3.5 Turbo

  2. GPT 4O

  3. GPT-4

  1. Temperature

  2. Max Tokens

  3. Azure OpenAI API Base

  4. Azure OpenAI Deployment Name

OpenAI

  1. GPT-4O

  2. GPT-4O-2024-05-13

  3. GPT-4-Turbo

  4. GPT-4-Turbo-2024-04-09

  5. GPT-3.5-Turbo

  6. GPT-3.5-Turbo-0125

  7. GPT-4-0125-Preview(Legacy)

  8. GPT-4-Turbo-Preview(Legacy)

  9. GPT-4-1106-Preview(Legacy)

  10. GPT-4(Legacy)

  1. Temperature

  2. Max Tokens

Anyscale

  1. Google Gemma 7B

  2. Meta Llama 3 8B

  3. Meta Lama 3 70B

  4. Mistral 7B Instruct

  5. Mixtral 8x7B Instruct

  6. Mixtral 8x22B Instruct

  1. Temperature

  2. Max Tokens

Databricks

  1. Foundation Models

    1. Databricks DBRX Instruct

    2. Meta Lama 3 70B Instruct

    3. Mistral 8x7B Instruct

    4. Llama 2 70B Chat (Legacy)

  1. Databricks External Models

  1. Databricks Custom Models

  1. Temperature

  2. Max Tokens

  3. Endpoint URL: Databricks model Endpoint URL

Amazon SageMaker

  1. Temperature

  2. Max Tokens

  3. Model Endpoint Name: SageMaker model endpoint name

PreviousEmbeddings ModelsNextPrompt Management

Last updated