Human in the Loop

Human in the Loop is a concept often used in artificial intelligence and large language models (LLMs) which involves humans in the decision-making or validation process when using language models, especially in situations where the model's predictions or actions may be uncertain, risky, or require human judgment and expertise.

This concept is also know as Reinforcement Learning from Human Feedback (RLHF).

RHLF is an approach that leverages human-provided feedback to train or fine-tune reinforcement learning models. This concept is particularly relevant in Large Language Models (LLMs) because it helps improve their performance, safety, and alignment with human values.

This can be applied in several ways including:

1. Data annotation and supervision: Since LLMs require a substantial amount of data for training, therefore humans are often involved in curating and labeling datasets to ensure the model learns from high-quality, relevant information.
   
   

2. Fine-tuning: After pre-training on a large amount of text, LLMs can be fine-tuned on specific tasks or domains. Human experts can be involved in this fine-tuning process to adjust the model's behavior, making it more applicable to particular tasks or industries.
   
   

3. Evaluation and validation: LLMs can generate text, answer questions, or make recommendations, but the quality of these outputs can vary. Human reviewers can assess and validate the model's responses, ensuring they meet certain quality standards, are factually accurate, and align with ethical guidelines.
   
   

4. Content filtering and moderation: To prevent the generation of harmful or inappropriate content, human reviewers are employed to filter and moderate the model's output. This is crucial in applications like content generation, chatbots, and AI-driven customer support.

In Orquesta, there are several ways a human expert can help improve the performance of an LLM and guide it's output or response.

Thumbs up and Thumbs down

This are a simple yet effective way for humans to provide feedback on the responses generated by a Large Language Model (LLM). These feedback mechanisms are similar to the familiar concept of "liking" or "disliking" content on various online platforms, and they serve as a means of guiding the LLM's learning and improving its responses. Here's how these concepts work:

1. Thumbs Up (Positive Feedback):

   When a user interacts with an LLM and receives a response they find helpful, accurate, or satisfying, they can provide a "Thumbs Up" or a positive rating to that response. This is a way of indicating that the LLM's output was valuable and aligned with the user's intent.
   
   

2. Thumbs Down (Negative Feedback):

   Conversely, if a user receives a response from the LLM that is unhelpful, incorrect, offensive, or otherwise unsatisfactory, they can provide a "Thumbs Down" or a negative rating to that response. This feedback signals that the LLM's output needs improvement.

Using the add metrics method

Orquesta provides users with the ability to use the addMetrics() method to add metadata, metrics and information about the interaction with the LLM to the request log. This will further help the human or domain expert add more custom metrics to the log.

For example: Add metrics to your request log

prompt.addMetrics({
  score: 70,
  latency: 3000,
  llm_response: 'Using Orquesta is super amazing!',
  economics: {
    prompt_tokens: 1200,
    completion_tokens: 750,
    total_tokens: 1950,
  }
});

Metadata

Metadata is a set of key-value pairs that you can use to add custom information to the log. It typically includes additional information or context related to the request or response, which can be helpful for various purposes. Here's how you can pass metadata to the addMetrics method:

prompt.addMetrics({
  metadata: {
    custom: 'custom_metadata',
    chain_id: 'ad1231xsdaABw',
    total_interactions: 200,
    timestamp: Date.now(),
    user_clicks: 20,
    selected_option: 'option1',
  }
});

Review of response

A human expert can assess and evaluate the generated text produced by the model in response to a given prompt or query. This review process is crucial for assessing the quality, relevance, and safety of the LLM's output, as well as ensuring it aligns with the intended purpose.

Advantages of Human in the Loop in LLMs

1. Improved Quality and Trust: Human reviewers can help improve the quality and accuracy of LLM outputs, making them more reliable and trustworthy. This is particularly important in applications where errors or biases can have significant consequences.

2. Adaptability and Customization: HITL allows LLMs to be customized for specific tasks or industries. Human input ensures that the model aligns with domain-specific requirements and can handle nuances and complexities.

3. Ethical Control: HITL can be used to prevent the generation of harmful, biased, or inappropriate content by providing human oversight and moderation. This is essential for maintaining ethical standards.

4. Addressing Uncertainty: LLMs can sometimes produce uncertain or ambiguous responses. Human reviewers can resolve such uncertainty, making the model more useful in situations where clarity is essential.

5. Continuous Learning: Human reviewers can provide feedback to LLMs, helping them learn from their mistakes and improve over time. This iterative feedback loop contributes to ongoing model refinement.

6. Compliance and Regulation: In regulated industries or areas with strict guidelines, HITL can ensure that LLM outputs conform to legal and industry standards.

Wrap up

In essence, the Human in the loop approach emphasizes the importance of human feedback and metadata to improve the performance of AI systems. Several key elements contribute to the HITL process, including "Thumbs up" and "Thumbs down" feedback, the use of the "add metrics" method, metadata integration, and the review of AI-generated responses.

Get started today.

Check out Orquesta documentation.

Follow us on Twitter, LinkedIn and GitHub.