Skip to main content

Test Run Management

Test run management involves overseeing and analyzing the results of test suites executed against the AI assistant. Admins can use this functionality to gain insights into the AI's performance, identify areas for improvement, and ensure the quality and reliability of Ayushma's responses.

Accessing Test Run Results​

  1. Navigate to Test Suites: From the admin dashboard, access the "Test Suites" section.
  2. Select Test Suite: Choose the specific test suite for which you want to view run results.
  3. View Test Runs: Ayushma typically displays a list of past test runs associated with the selected test suite, including information such as:
    • Project: The project against which the test run was executed.
    • Start and End Times: The timestamps indicating when the test run started and finished.
    • Status: The current status of the test run, such as "Completed," "Running," "Canceled," or "Failed."
    • Evaluation Metrics: Summary metrics such as average BLEU score and cosine similarity, providing a high-level overview of the AI's performance during the run.

Analyzing Test Run Results​

  • Individual Test Case Review: Admins can delve into the results of each test case within a run, examining:
    • Question: The specific query or prompt presented to the AI assistant.
    • Human Answer: The expected or reference answer for the question.
    • AI Answer: The response generated by the AI assistant during the test run.
    • Cosine Similarity: A metric indicating the semantic similarity between the AI's answer and the human answer.
    • BLEU Score: A metric measuring the overlap and fluency of the AI's answer compared to the human answer.
    • (Optional) References: If reference documents were used during the test run, the specific documents that informed the AI's response might be listed.
    • Feedback: Admins or reviewers can provide feedback on individual test cases, offering qualitative insights and suggestions for improvement.
  • Aggregate Metrics: Ayushma might present aggregated metrics for the entire test run, such as:
    • Average Cosine Similarity: The mean cosine similarity score across all test cases in the run.
    • Average BLEU Score: The mean BLEU score across all test cases in the run.
    • Pass/Fail Rate: The percentage of test cases where the AI's response met predefined criteria for success.
  • Visualization: Ayushma may offer visualizations such as charts or graphs to represent the distribution of scores or highlight patterns in the AI's performance.

Additional Insights​

  • Feedback System: The code suggests the presence of a feedback system, allowing admins or reviewers to provide qualitative feedback on individual test cases. This feedback can be valuable for understanding the nuances of the AI's performance and guiding further improvements.
  • Test Run Logs: Ayushma might maintain detailed logs of test runs, capturing information about the AI's processing steps, intermediate outputs, and any errors encountered during the evaluation. These logs can be helpful for debugging and troubleshooting purposes.

Benefits of Test Run Management​

  • Performance Insights: Test run management provides detailed insights into the strengths and weaknesses of the AI assistant, enabling targeted improvements and optimizations.
  • Error Analysis: By analyzing test cases where the AI underperformed, admins can identify patterns in errors and take corrective actions, such as refining the training data, adjusting model parameters, or adding more relevant reference documents.
  • Model Selection: Comparing the results of test runs across different AI models can help admins choose the most effective model for a particular project or use case.
  • Quality Assurance: Thorough analysis of test run results is essential for quality assurance, ensuring that the AI assistant meets the required standards of accuracy and reliability before deployment in real-world medical scenarios.