Core Concepts

Understand the core concepts and glossary of topics we use throughout Orquesta

Fundamentals

Context

The Context is the environment (dev-test, production, etc.) and setting where the subject that requests the Prompt or Remote Config are being evaluated operates. Your own team defines the Context, which consists of a set of Fields and mirrors your Data Model. Our evaluation engine attempts to match the conditions set in our Configuration Matrix as strictly as possible to the provided Context to provide the correct Return Value.

Data model

In the Data model, field is the major concept to look out for. A Field mirrors a field that occurs within your data model. Each Field has a unique key and type which could either be a Boolean, Date, List, Number, and String. A set of Fields can be combined as a specific Context sent as a payload for an Evaluation.

Domain

Domains enable you to group your Prompt or Remote Config logically. Additionally, you can set specific read, write, and admin permissions per domain for collaboration. For example, a Backend domain where Backend Engineers have write permissions, but the Frontend Engineers and Product Owners only have view permissions.

Evaluation

An Evaluation happens each time a Prompt or Remote Config with Context is sent to Orquesta. With our distributed architecture, this Evaluation happens on the Edge and is returned to your systems in milliseconds globally.

Rollback

The process of reverting to a previous version of a prompt or remote configuration. This can be done for a variety of reasons, such as to fix a bug, to revert to a previous configuration.

Versions

Each unique state of a Prompt or Remote Config is saved as a different version. This allows you to track changes over time and rollback to previous configurations if necessary. Versions are usually numbered sequentially and may also include metadata like who made changes and when.

Workspace

The workspace is where you collaborate with your team in setting up configurations, and AI prompts. Each workspace is an isolated environment with unique Prompts, Remote Config, Domains, and Teams.

LLM Ops

Code Snippet

This is a small block of reusable code that is often provided to demonstrate how to integrate the selected LLM Ops or Prompt in various programming environments like Node, Python, or CURL. These snippets serve as a quick reference for developers to understand the required syntax and parameters for making API calls.

Gateway

A Gateway is a component that provides access to LLMs. It is responsible for authenticating and authorizing users, and for routing requests to the appropriate LLM.

Gateways can play a number of important roles including: Security, Load balancing, Routing.

Model Garden

Orquesta is LLM-agnostic. We want to get out of your way in working with public, private and custom LLM Providers and Models. Within the Model Garden, you can enable providers and specific models your product works with.

Playground

Orquesta provides you with a playground where you can quickly assess each model can generate a response, evaluate the relevance, coherence, and accuracy of each model's answers. You can also compare how well each model understands the nuance or context behind a given prompt and by fine-tuning its provided parameters, it can help improve the response quality on the selected models.

You can create multiple playgrounds to identify the limitations or flaws in each model's understanding or output generation by creating a side-by-side comparison, making it easier to spot differences in quality and efficiency.

The playground provides an immediate way to judge performance metrics like speed and accuracy, all this are constantly auto saved so you don't have to save all the time.

Prompt

In Orquesta, you maintain and manage the entire lifecycle of your prompts for your AI-infused products. Each Prompt is a combination of the actual prompt, provider, model and set of hyperparameters. This allows you to design, experiment and optimize Prompts for different use cases and as new models come to market continuously.

Prompt Overview

The prompt overview provides a table of information about the prompt's key, domain, version, requests, costs, latency, score, last updated, and the user that created it. It also help you understand how your prompts are performing.

Prompt Tokens

These are specific tokens that make up the input prompt you give to the language model. Token count is often crucial because many language models have a maximum token limit for each interaction.

Prompt Variant

A prompt variant is a variation of a prompt that is used to generate text from a LLM. Prompt variants are used to improve the quality and diversity of the generated text. They can be created by: changing the wording of the prompt, adding or removing information from the prompt, using different formats for the prompt.

Remote Configurations

Active Remote Configs

The number of different configurations within the remote configuration service that are actively being requested and used by the application.

Average Response Time

The average time it takes for the server to return a configuration after it has been requested.

Configuration Matrix

A Rule is configured with our highly intuitive Configuration Matrix. Anybody that knows Excel can contribute to managing and operating your systems. The matrix contains Fields that are used to evaluate against a Context. Each constellation of Fields can have a unique return value. The Evaluation of each request happens top-to-bottom, and the first matched Context is returned.

Default Value

A default value is a value that is assigned to a parameter if no value is explicitly provided. Default values are useful for ensuring that parameters are always assigned a value, even if the user forgets to specify one.

Hit Rate (row matched per request)

This is the ratio of requests that have a context that matches a specific value. The default value is not returned.

LLM Model

A large language model (LLM) is a type of artificial intelligence that can generate and understand human-like text. LLMs are trained on massive datasets of text and code, which allows them to learn the statistical relationships between words and phrases. This knowledge allows them to perform a wide range of tasks, such as:

• Translation: LLMs can translate text from one language to another.
  
  

• Summarization: LLMs can generate summaries of long or complex texts.
  
  

• Creative writing: LLMs can generate creative text formats, such as poems, code, scripts, musical pieces, email, letters, etc.
  
  

• Question answering: LLMs can answer questions in a comprehensive and informative way.
  
  

LLM Provider

An LLM Provider is a company or organization that provides access to large language models (LLMs). Some of the LLM provider available include: OpenAI, Cohere, LangChain, HuggingFace, Anthropic, Google and Replicate.

Remote Configuration

Configurations of systems are often fragmented across different sources and can be complex. Orquesta enables your teams to feed configurations remotely from a single source of truth to all your systems. Your teams can adjust parameters, deploy updates, or troubleshoot issues without the need for code changes and expensive deployments.

All layers in your stack can consume Orquesta Rules, including your frontend, backend, CI/CD pipelines, and infrastructure. Additionally, your systems can behave differently based on Context for personalization and localization use cases.

Return Value

A type-safe value that a prompt or remote configuration returns to your systems after an Evaluation. Each prompt or remote configuration also has a default value that is returned in case no Context is matched.

Requests

The number of times a particular remote configuration is requested by the application during a specific time period.

Simulator

This a tool that enables you to test any of your prompt's matrix rows. It provides the capability to verify the default value when the simulated fields do not match or to see the respective return value when they do match.

The results are based on whatever values you input or select for the supported properties of the prompt matrix. This functionality aids in debugging and validating the behavior of your prompts before or after deploying them into a live environment.

Total Requests

The cumulative number of times a particular remote configuration has been requested since over a specified time period.

Analytics

Costs

Costs helps teams and organizations understand and manage their LLMOps costs more effectively.

Latency

Latency is the time it takes for a large language model (LLM) to process a request and return a response. This is an important metric to consider when using LLMs, it can be affected by a number of factors, including:

• The size and complexity of the LLM

• The type of request (Chat or Completion) being made

• The network connection between the user and the LLM

Monitoring & Logging

Monitoring and logging is the process of collecting and analyzing data about the performance and usage of LLMs. This data can be used to identify and troubleshoot problems, improve the performance of LLMs, and understand how LLMs are being used.

p50

P50 is the percentile of latency at which 50% of requests are served. In other words, it is the amount of time it takes for half of all requests to be completed.

P50 is an important metric to track because it can help teams identify and address performance bottlenecks in their applications. For example, if the P50 latency is high, it may indicate that the platform is not able to handle the volume of requests that it is receiving.

p99

The p99 refers to the 99th percentile of a given metric. In other words, it is the value that is higher than 99% of all other values in the metric.

p99 is often used to measure the performance of LLMs in production, it can also be used to measure the quality of LLM outputs.

Score

Score is a measure of the quality of an LLM's output or a metric that is used to evaluate the performance of a language model on a prompt or remote configuration. The score can be calculated using a variety of methods, depending on the specific task. For example, for a task such as question answering, the score could be calculated based on the accuracy of the LLM's answers.

Continue reading

Quick start: Get started in a couple of simple steps and be up in 10 minutes. Read more 🡢

Developers: Use our SDKs, integrations and REST API to connect your entire stack. Read more 🡢

Troubleshooting

If you encounter any issues while using Orquesta, send us a message in the chat at the bottom right or see our Contact page