Reviewing and Managing Annotations

This guide explains how to collect user feedback through annotations and use that feedback to improve your AI prompts in PromptOwl .

Overview

Annotations are user feedback messages attached to AI responses. They help you:

• Understand how well your prompts are performing
• Identify areas for improvement
• Build evaluation datasets for testing
• Track user satisfaction over time

Understanding Annotations

What Are Annotations?

Annotations combine two types of feedback:

1. Sentiment - Quick thumbs up/down rating
2. Detailed Feedback - Written comments explaining the rating

Types of Annotations

Collecting Annotations (User View)

Adding Feedback to a Response

Users can annotate any AI response:

1. Hover over an AI response in the chat
2. Click the Annotation icon (speech bubble)
3. The annotation modal opens

The Annotation Modal

The modal shows:

• Preview of the AI response (first 200 characters)
• Sentiment buttons (thumbs up/down)
• Text area for detailed feedback

Providing Feedback

1. Select Sentiment (optional):

   • Click Thumbs Up for positive feedback
   • Click Thumbs Down for negative feedback
   • Click again to deselect

2. Write Details (required):

   • Describe what was good or bad
   • Suggest improvements
   • Note any inaccuracies
   • Maximum 2,000 characters

3. Click Submit to save

Annotating Entire Conversations

For overall conversation feedback:

1. Open the conversation history panel
2. Click Annotate Conversation in the header
3. Provide sentiment and feedback
4. Submit the annotation

Reviewing Annotations (Admin View)

Administrators can review all feedback through the Monitor interface.

Accessing the Monitor

1. Open a prompt from the Dashboard
2. Click Monitor in the top navigation
3. Navigate to the All Annotations tab

The All Annotations View

This view displays all feedback collected for your prompt:

Table Columns

Filtering and Sorting

• Search: Find specific feedback by keyword
• Sort: Order by most recent first
• Filter by Sentiment: View only positive or negative feedback

Viewing Annotation Context

To see the full conversation:

1. Click on any annotation row
2. The conversation panel opens on the right
3. View the complete message history
4. See where the annotation was placed

Sentiment Analysis

Understanding Sentiment Badges

Sentiment Patterns to Watch

• Consistent negative sentiment → Prompt needs major revision
• Mixed sentiment on same topics → Inconsistent AI responses
• Positive sentiment declining → Recent changes may have issues
• Negative on specific questions → Knowledge gaps to address

Exporting Annotations

Export to CSV

Download all annotations for external analysis:

1. Go to the All Annotations tab
2. Click Export CSV
3. Save the downloaded file

The CSV includes:

• User information
• Question and response text
• Annotation content
• Sentiment values
• Timestamps

Use Cases for Exports

• Share with stakeholders
• Analyze trends in spreadsheets
• Create reports on AI performance
• Archive feedback for compliance

Creating Evaluation Sets

Turn high-quality annotations into test cases for prompt evaluation.

What Are Eval Sets?

Eval Sets are collections of question-response pairs with expected outcomes. Use them to:

• Test prompt changes before publishing
• Compare different prompt versions
• Ensure quality doesn’t regress

Creating an Eval Set from Annotations

1. Go to the All Annotations tab
2. Check the boxes next to relevant annotations
3. Click Save to Eval Set
4. Name your eval set
5. Click Create

Best Annotations for Eval Sets

Include annotations that:

• Represent common user questions
• Show clear expected behavior
• Cover edge cases
• Include both positive and negative examples

Running Evaluations

1. Go to the prompt’s Eval tab
2. Select your eval set
3. Run the evaluation
4. Compare results against expected outcomes

Using Annotations to Improve Prompts

Feedback Analysis Workflow

1. Review Weekly: Check annotations at least weekly
2. Identify Patterns: Look for recurring issues
3. Prioritize Fixes: Address frequent negative feedback first
4. Update Prompts: Make targeted improvements
5. Test Changes: Use eval sets to verify fixes
6. Monitor Results: Track if sentiment improves

Common Issues and Solutions

Best Practices

For Collecting Quality Feedback

• Enable annotations for all production prompts
• Train users on how to give useful feedback
• Make it easy - ensure annotation button is visible
• Follow up on critical feedback quickly

For Reviewing Feedback

• Schedule regular reviews (daily or weekly)
• Look for patterns not just individual complaints
• Track sentiment trends over time
• Celebrate positive feedback with your team

For Acting on Feedback

• Prioritize by impact - fix issues affecting most users
• Test before publishing - use eval sets
• Document changes - note why changes were made
• Close the loop - let users know issues are fixed

Enterprise Settings

Enabling Annotations

Annotations can be enabled/disabled at the enterprise level:

1. Go to Admin → Settings
2. Find Response Annotation setting
3. Toggle to enable/disable

Related Settings

Troubleshooting

Annotation button not visible

1. Check enterprise settings have annotations enabled
2. Verify user is logged in
3. Ensure user owns the conversation
4. Check permission settings for the prompt

Cannot submit annotation

1. Verify feedback text is not empty
2. Check text is under 2,000 characters
3. Ensure network connection is stable
4. Try refreshing the page

Annotations not appearing in Monitor

1. Verify you have access to the prompt
2. Check you’re viewing the correct prompt
3. Refresh the All Annotations view
4. Check date filters aren’t excluding recent feedback

Export not working

1. Check browser allows downloads
2. Try a different browser
3. Verify there are annotations to export
4. Check for popup blockers

Privacy and Data Handling

What’s Stored

• Annotation text
• Sentiment value
• Timestamp
• User who submitted
• Associated conversation

Data Retention

• Annotations are retained with conversations
• Deleting a conversation removes its annotations
• Export data for archival before deletion

Access Control

• Only prompt owners/admins can view all annotations
• Users can only annotate their own conversations
• Team members see annotations for shared prompts

Related Guides

• Adding and Managing Users
• Editing and Maintaining Prompts
• Adding and Organizing Documents