Power Platform Community Forum Thread Details

Product

Power Automate Desktop

Issue Description

I have built a Power Automate Desktop flow that retrieves a patient's appointment number and then opens the corresponding appointment from a table displayed on the screen.

The process works as follows:

The flow receives the appointment number as input.

An OCR action scans the appointment table and extracts all appointment numbers along with their coordinates.

The flow compares the extracted appointment numbers with the input appointment number.

Once a match is found, the flow uses the coordinates returned by OCR to click on the appointment number and open the appointment details.

Problem

In most cases, the flow works as expected. However, there are instances where OCR correctly identifies the appointment number, but the coordinates returned for that appointment number are incorrect.

As a result, the mouse click occurs in the wrong location, causing the workflow to fail or open an incorrect record.

Expected Behavior

When OCR identifies the correct appointment number, it should return the corresponding coordinates accurately so that the flow can click the correct appointment entry.

Actual Behavior

OCR successfully identifies the appointment number, but occasionally returns incorrect coordinates, leading to an incorrect click position.

Troubleshooting Performed

Verified that the appointment number is correctly identified by OCR.

Reviewed OCR output data and confirmed that the matched appointment number exists in the OCR results.

Tested the flow multiple times to rule out intermittent application performance issues.

Observed that the issue occurs only in certain cases and is not consistently reproducible

Questions

Has anyone experienced similar issues with OCR coordinate extraction in Power Automate Desktop?

Are there any known limitations or accuracy concerns with OCR coordinate mapping?

Would you recommend an alternative approach (for example, UI elements, image recognition, anchor-based targeting, or additional validation steps) to improve reliability?

Additional Information

Any suggestions, best practices, or workarounds for improving OCR coordinate accuracy would be greatly appreciated.

Thanks in advance for your help!

Categories:

Power Automate Desktop

When your flow clicks the wrong place is it consistently clicking same wrong place? Or when the OCR actions clicks the wrong place is it random place it clicks?
Is it always the same appointment numbers that the flow fails on or is it random?
Is the appointment numbers short? If they are short like "3145" then if there is some other text that might contain that like phone number: "020 3145 0126" then the OCR might be clicking that text instead of the appointment number.

Have you checked that there is only one occurrence of that appointment number when it fails? Usually if OCR clicks "wrong place" it actually just found another instance of that same text and clicked the first it found. So if anywhere in the screen there is that same appointment number elsewhere than just where the flow should click on then that could be the reason that it clicked wrong place. For example maybe if you run the flow from the editor and the appointment number is saved on a variable it might try to click that Power Automate Desktop editors variable value instead of the system you are automating. Or if in the system there is that same number visible somewhere else then it might click there since that came up first for the OCR.

Would you recommend an alternative approach (for example, UI elements, image recognition, anchor-based targeting, or additional validation steps) to improve reliability?

Yes. If possible always use UI elements rather than OCR or image recognition. UI elements are much more reliable. So create UI selector that selects only the appointment number UI element that you want the flow to click and just change the appointment number to be dynamic in the selector.