Features
Endpoints
Endpoints
Orquesta splits Prompts and Endpoint, although both have comparable features.
Prompts are used when you use Orquesta as a Prompt Manager and want to have full control over the actual LLM call and logging. Read more about Prompts.
Within Endpoints, Orquesta handles all complexity for you, including the actual LLM call and full logging of responses. On top of that, Endpoints have more superpowers, where you can configure retries and fallback models.
When logging metrics you will only have to log human-in-the-loop feedback and your custom metadata. All other metrics are logged for you by Orquesta, and you can view all these in the Observability tools.
Activate the models supported in the gateway.
Create your first Endpoint.
Add a variant in the Endpoint.
Configure your Endpoint (retries and fallback model).
Activate your models in the model garden
To use the Endpoints, you have to activate the models, that are supported by Orquesta's LLM Gateway. Head over to the Model Garden and activate the models you want to interact with.
Check out the supported models
Creating your first Endpoint
In the Endpoints section, click on the Add Endpoint button to create your first Endpoint in Orquesta.
Provide your Endpoint Key and Domain, and Publish.
Variants based on custom context
You can create variants based on the custom context of your user or application in Endpoint, and the process is very simple. Click on the Add Variant button, set up your prompt and model to use, and build business rules for the custom context.
Fallbacks
This refers to a secondary or alternative language model that is used as a backup when the primary model is unavailable or encounters issues. The purpose of a fallback model is to ensure the continuity and reliability of language model operations in case of unexpected failures or downtime of the primary model.
To use a fallback model in Orquesta, simply click on the Fallback Model button. Select the fallback model of your choice and configure it if necessary.
Retries
Retries refer to the number of attempts made to execute an LLM call or a specific task. Retries are used to handle situations where an operation may fail for various reasons, such as network issues, system errors, or resource constraints.
When an LLM call fails, Orquesta automatically initiates a retry mechanism to reattempt the task. The number of retries determines how often the platform will try to perform the operation before considering it a permanent failure or using the fallback model. This helps to improve the chances of successful task completion.
The difference between Prompts and Endpoints
Orquesta splits Prompts and Endpoint, although both have comparable features.
Prompts are used when you use Orquesta as a Prompt Manager and want to have full control over the actual LLM call and logging.
While in Endpoints, Orquesta handles all complexity for you, including the actual LLM call and full logging of responses. On top of that, Endpoints have more superpowers, where you can configure retries and fallback models. When logging metrics you will only have to log human-in-the-loop feedback and your custom metadata. All other metrics are logged for you by Orquesta, and you can view all these in the Observability tools.
Wrap up
You can use Endpoints using no-code, where Orquesta handles all the infrastructure of executing the LLM call and logging all interactions for observability. Using Endpoints to generate an LLM response based on your use case with Orquesta provides a low-latency, secure connection to the Endpoints API service, getting out-of-the-box metrics and logging for your LLMs.
The Endpoints API is a scalable and efficient way to interact with Multiple LLMs using smart business rules. One key and endless possibilities, allowing you and your team to streamline your operations and focus on what matters most.
References
Endpoint usage using Python SDK and Node.js SDK.
Endpoints
Orquesta splits Prompts and Endpoint, although both have comparable features.
Prompts are used when you use Orquesta as a Prompt Manager and want to have full control over the actual LLM call and logging. Read more about Prompts.
Within Endpoints, Orquesta handles all complexity for you, including the actual LLM call and full logging of responses. On top of that, Endpoints have more superpowers, where you can configure retries and fallback models.
When logging metrics you will only have to log human-in-the-loop feedback and your custom metadata. All other metrics are logged for you by Orquesta, and you can view all these in the Observability tools.
Activate the models supported in the gateway.
Create your first Endpoint.
Add a variant in the Endpoint.
Configure your Endpoint (retries and fallback model).
Activate your models in the model garden
To use the Endpoints, you have to activate the models, that are supported by Orquesta's LLM Gateway. Head over to the Model Garden and activate the models you want to interact with.
Check out the supported models
Creating your first Endpoint
In the Endpoints section, click on the Add Endpoint button to create your first Endpoint in Orquesta.
Provide your Endpoint Key and Domain, and Publish.
Variants based on custom context
You can create variants based on the custom context of your user or application in Endpoint, and the process is very simple. Click on the Add Variant button, set up your prompt and model to use, and build business rules for the custom context.
Fallbacks
This refers to a secondary or alternative language model that is used as a backup when the primary model is unavailable or encounters issues. The purpose of a fallback model is to ensure the continuity and reliability of language model operations in case of unexpected failures or downtime of the primary model.
To use a fallback model in Orquesta, simply click on the Fallback Model button. Select the fallback model of your choice and configure it if necessary.
Retries
Retries refer to the number of attempts made to execute an LLM call or a specific task. Retries are used to handle situations where an operation may fail for various reasons, such as network issues, system errors, or resource constraints.
When an LLM call fails, Orquesta automatically initiates a retry mechanism to reattempt the task. The number of retries determines how often the platform will try to perform the operation before considering it a permanent failure or using the fallback model. This helps to improve the chances of successful task completion.
The difference between Prompts and Endpoints
Orquesta splits Prompts and Endpoint, although both have comparable features.
Prompts are used when you use Orquesta as a Prompt Manager and want to have full control over the actual LLM call and logging.
While in Endpoints, Orquesta handles all complexity for you, including the actual LLM call and full logging of responses. On top of that, Endpoints have more superpowers, where you can configure retries and fallback models. When logging metrics you will only have to log human-in-the-loop feedback and your custom metadata. All other metrics are logged for you by Orquesta, and you can view all these in the Observability tools.
Wrap up
You can use Endpoints using no-code, where Orquesta handles all the infrastructure of executing the LLM call and logging all interactions for observability. Using Endpoints to generate an LLM response based on your use case with Orquesta provides a low-latency, secure connection to the Endpoints API service, getting out-of-the-box metrics and logging for your LLMs.
The Endpoints API is a scalable and efficient way to interact with Multiple LLMs using smart business rules. One key and endless possibilities, allowing you and your team to streamline your operations and focus on what matters most.
References
Endpoint usage using Python SDK and Node.js SDK.
Endpoints
Orquesta splits Prompts and Endpoint, although both have comparable features.
Prompts are used when you use Orquesta as a Prompt Manager and want to have full control over the actual LLM call and logging. Read more about Prompts.
Within Endpoints, Orquesta handles all complexity for you, including the actual LLM call and full logging of responses. On top of that, Endpoints have more superpowers, where you can configure retries and fallback models.
When logging metrics you will only have to log human-in-the-loop feedback and your custom metadata. All other metrics are logged for you by Orquesta, and you can view all these in the Observability tools.
Activate the models supported in the gateway.
Create your first Endpoint.
Add a variant in the Endpoint.
Configure your Endpoint (retries and fallback model).
Activate your models in the model garden
To use the Endpoints, you have to activate the models, that are supported by Orquesta's LLM Gateway. Head over to the Model Garden and activate the models you want to interact with.
Check out the supported models
Creating your first Endpoint
In the Endpoints section, click on the Add Endpoint button to create your first Endpoint in Orquesta.
Provide your Endpoint Key and Domain, and Publish.
Variants based on custom context
You can create variants based on the custom context of your user or application in Endpoint, and the process is very simple. Click on the Add Variant button, set up your prompt and model to use, and build business rules for the custom context.
Fallbacks
This refers to a secondary or alternative language model that is used as a backup when the primary model is unavailable or encounters issues. The purpose of a fallback model is to ensure the continuity and reliability of language model operations in case of unexpected failures or downtime of the primary model.
To use a fallback model in Orquesta, simply click on the Fallback Model button. Select the fallback model of your choice and configure it if necessary.
Retries
Retries refer to the number of attempts made to execute an LLM call or a specific task. Retries are used to handle situations where an operation may fail for various reasons, such as network issues, system errors, or resource constraints.
When an LLM call fails, Orquesta automatically initiates a retry mechanism to reattempt the task. The number of retries determines how often the platform will try to perform the operation before considering it a permanent failure or using the fallback model. This helps to improve the chances of successful task completion.
The difference between Prompts and Endpoints
Orquesta splits Prompts and Endpoint, although both have comparable features.
Prompts are used when you use Orquesta as a Prompt Manager and want to have full control over the actual LLM call and logging.
While in Endpoints, Orquesta handles all complexity for you, including the actual LLM call and full logging of responses. On top of that, Endpoints have more superpowers, where you can configure retries and fallback models. When logging metrics you will only have to log human-in-the-loop feedback and your custom metadata. All other metrics are logged for you by Orquesta, and you can view all these in the Observability tools.
Wrap up
You can use Endpoints using no-code, where Orquesta handles all the infrastructure of executing the LLM call and logging all interactions for observability. Using Endpoints to generate an LLM response based on your use case with Orquesta provides a low-latency, secure connection to the Endpoints API service, getting out-of-the-box metrics and logging for your LLMs.
The Endpoints API is a scalable and efficient way to interact with Multiple LLMs using smart business rules. One key and endless possibilities, allowing you and your team to streamline your operations and focus on what matters most.
References
Endpoint usage using Python SDK and Node.js SDK.
