LLM Experimentation Playground

Orquesta’s LLM Playground is an experimentation playground where you can interact with multiple Large Language Models (LLMs), perform comparisons based on the LLM response, and keep track of metrics such as costs and latency of experiments and provide human feedback.

Use Cases

The LLM Playground can be used by your teams for a big set of use cases

• Easily test existing use cases and prompts with new models

• Compare new use cases across different models and collect insights on quality, performance and costs

• Experiment with changing hyperparameters of models

• Experiment with new and updated prompts to get a quick sense of the performance of a model

• Provide feedback on responses to create datasets for future fine-tuning

Setting up a LLM Playground

Create a new Playground. The playground is auto-saved, so you can easily come back to your experiments or hand it over to a team member to continue your experiments.

Playground Tools

The playground comprises experiment blocks, and you can add up to 6 blocks by clicking on the add comparison button.

You can delete a block within the playground by clicking on the delete button, clear a conversation within a block and copy the entire response.

Configuring Playground Block

For your model, you can select a model by clicking on the Model selection dropdown (the prompt type will be displayed beside it - Chat or Completion), and you will be able to access the activated models available in the playground. To get a better response from the model, the playground offers a unique feature that helps configure the model by clicking on the gear icon. Some of these parameters include Number of words, Frequency Penalty, Temperature, Top K and Top P.

To input your prompts, you use the prompt field within the block; type in your prompt and press the send button. You can sync chats in the playground to easily test the same prompt across different model configurations.

For each response in the playground, you can access its costs and latency.

You can create variables using opening and closing curly braces {{ }} and then place the variable key inside it within the prompt. To do this, you will have to create the variable, for example, {{context}} and automatically, it gets added to the sidebar; edit the variable to your satisfaction in the input field. This is useful for RAG use cases and working with large pieces of context in your prompt engineering.

Internal feedback

Feedback is very important, as it helps to know how the model is performing based on the output it renders. The playground uses "thumbs up" and "thumbs down" to provide feedback for each response, making it easy for internal team members or domain experts to add feedback. The collected feedback can then be used for model fine-tuning as a data set.

Logs

All the runs are recorded in the Logs tab in the Playground. Some of the logged metrics include cost (input cost, output cost, total cost), provider/model, economics (input_tokens, output_tokens, tokens_per_second, first_token_latency), latency, prompt, status, and the LLM response.