Evaluation failing on every testset

(2) Share

Report

Posted on by BV-29041200-0

Dear Support team & Copilot Studio users,

I've been using the Copilot Studio Evaluation functionality succesfully and happily for the last months, almost every day.

Yesterday, the Copilot Studio Evaluation functionality ran into problems.

When I launch the Evaluation of a testset (large or short with 2 question-expectedResponse pairs) after 10-20 seconds the page hangs and shows the message "Preparing testcases". On the right pannel I see "Score N.v.t. and No score available". "N.v.t. stands for "Not applicable" in Dutch, the locale I'm using).

I can reproduce the same issue in different environments and with different users (personal and service account user).

Last, but also relevant, if I test the questions in the testset in the Copilot Studio Test pannel, there I do receive relevant answers and the agent works as expected.

Does someone know what has changed in Copilot Studio Evaluation functionality and how to workaround this problem?

Thanks in advance!

Best, Begoña

Categories:

General topics

I have the same question (0)

All responses (4)

Answers (0)

Sort by

Sayali-MSFT Microsoft Employee on at

Like (1)

Report
Copy link

Link copied!

Hello BV-29041200-0 ,

Based on the reported behavior, this appears more likely to be a platform-side issue affecting the Evaluation workflow rather than an agent configuration problem. The issue reproduces across multiple environments and user accounts, including very small test sets, while the agent continues to return valid responses in the Test pane. Since the failure occurs during the "Preparing testcases" stage and no evaluation score is generated, further investigation of the Evaluation service/backend logs is recommended.

Was this reply helpful? Yes No
BV-29041200-0 12 on at

Like (0)

Report
Copy link

Link copied!

Hello Sayali,

Thanks very much for your prompt response!

I have access to the App Insights Telemetry events when the evaluation run was launched. Unfortunately there's not enough information available to understand where the failure occurs.

Could you please advise me on where to find the Evaluation service logs ?

I much appreciate further directions.

Thanks a lot!

Begoña

Was this reply helpful? Yes No
11manish 3,052 on at

Like (1)

Report
Copy link

Link copied!
Given that:

It worked for months,

Failed suddenly,

Reproduces across environments and users,

Agents themselves still answer correctly,

this strongly points to a backend Copilot Studio Evaluation service issue or regression, rather than a configuration problem in your agent.

I would recommend checking whether other users in the community are reporting the same behavior and, if no active advisory exists, opening a Microsoft Support ticket with:

Environment ID(s)

Timestamp of failed evaluations

Browser HAR file

Screenshot showing the "Preparing testcases" state

That information should allow Microsoft to trace the evaluation job and determine whether the evaluation orchestration service is failing behind the scenes.

Was this reply helpful? Yes No
Suggested answer

Haque 3,486 on at

Like (0)

Report
Copy link

Link copied!

Hi @BV-29041200-0,

Can you please verify that the MS Copilot Studio connector is enabled and authorised for your tenant and users? Admins should check Power Platform Admin Center > Data policies and ensure the connector is not blocked.

Apparently seems like a platfrom issue, please monitor service health and community forums for updates, if this is a known issue to MS, they may release fixes or guidance.

Bare minimum: Sometimes stale tokens or cached data cause UI hangs. Please try different browsers or clear broswer caches.

Evaluation Service: The Evaluation service logs for Copilot Studio are not exposed directly to users through the portal or App Insights telemetry by default, which can make troubleshooting evaluation hangs challenging.

Alternatively, please access the Power Platform Admin Center and check the Environment’s Audit Logs and Power Platform Analytics for any errors or warnings related to evaluation runs.

Also, please use the Microsoft 365 Security & Compliance Center or Azure Monitor if your organization has integrated logging and monitoring for Power Platform and Copilot Studio.

I am sure some clues I tried to give. If these clues help to resolve the issue brought you by here, please don't forget to check the box Does this answer your question? At the same time, I am pretty sure you have liked the response!

Was this reply helpful? Yes No