Finding elements in HarmonyOS scripts: combining image match, color, and OCR

HarmonyOS automation means a program takes over repeated manual work on a HarmonyOS device. You connect through developer mode plus USB debugging, without root and without changing the system. Image match, color match, and OCR all sit on that connection.

Initialize first, then how image match, color, and OCR fit together

In a HarmonyOS automation script, every image-related recognition capability hangs on the image and ocr objects. One action has to happen first: initialize OpenCV. Skip it and recognition stays empty, with no error, and you can sit there for a long time.

After initialization, switch image storage to the mat format while you are there. The vendor’s measurements are direct: memory down by half to eighty percent, CPU down by twenty to thirty percent, and speed up by one to two times. On batch jobs in HarmonyOS cluster control, the device count magnifies that difference.

// Initialize OpenCV first. Image match and color match both depend on it.
image.initOpenCV();
// Captured images, loaded images, image match, and color match all switch to the mat format, which is faster and uses less memory.
image.useOpencvMat(1);

Screenshots for recognition usually use a full-screen capture. If the flow has to grab many frames in a row, use streaming screenshots so each frame does not open a new connection. Recycle image objects when you are done. A long-running script that does not recycle them eats memory bit by bit.

An image match takes a template image and finds it inside the current screenshot. The return value includes the matched coordinates, and those coordinates are what you tap.

let screen = image.captureFullScreen();
let tpl = image.readBitmap("/sdcard/ec/tpl_confirm_btn.png");
let pos = image.findImage(screen, tpl, { threshold: 0.9 });
if (pos) {
    logd("找到确认按钮: " + JSON.stringify(pos));
} else {
    logw("没找到确认按钮");
}
image.recycle(screen);

Three things from practice. Crop the template on the target device. A screenshot from model A running on model B can differ in size and rendering, and the match rate drops. Do not make the template too small. A small one looks like too many places on screen, and a slightly looser threshold false-matches. Keep changing text out of the template. Leave text to OCR, and keep only stable graphic features in the template. Start the threshold from an experience value, then look at the actual similarity in the log, and settle it where it just stops false-matching. Devices at different resolutions may need the threshold tuned separately.

Color matching compares color values, and it is much lighter than image matching. Single-point comparison fits a fixed position’s state, such as a button whose color differs when it can be tapped and when it cannot. Comparing the color at that position is enough, and it is much faster than an image match.

// Single-point color comparison: whether the color at a coordinate falls inside the allowed range
let c = image.getPixelBitmap(screen, x, y);
let hit = image.cmpColor(screen, "#FF5252", 0.9, 0, x, y);

Multi-point color matching is about uniqueness. One color can appear in many places on screen. Bind the colors of several points with relative offsets and the signature becomes unique. A reverse use is finding a region by color first: use color to mark a rough area, then image-match inside that area. That is faster than a full-screen match, and more accurate. Color matching is weak against change. A theme color change, a dark-mode switch, or a new background image can all break it. It fits a fast filter and a fallback. It is a poor sole basis for a decision. What color and graphics cannot settle is usually text, and that is when you switch to OCR.

OCR has a few more choices than image and color matching. Pick the model type first. The built-in PP-OCRv6_small is the faster route, and it is enough for ordinary upright interfaces. The optional types also include ocrLite, tesseract, and Onnx V4 and V5. Different models trade precision against speed. The initialization below uses the PPOCR V6 route.

ocr.releaseAll();
let engine = ocr.newOcr();
let ok = engine.initOcr({
    type: "paddleOcrOnnxV6",
    numThread: 2,
    matMode: 1,
    maxSideLen: 640,
    doAngleFlag: 0,
    mostAngleFlag: 0
});
if (!ok) {
    loge("OCR 初始化失败: " + engine.getErrorMsg());
    return;
}

Try parameters in this order. Padding is the one most worth moving first. It expands a ring of white around the text box. When the text was not fully boxed, raising it improves the result immediately, at the cost of more time. The detection-box confidence threshold: raise it and recall drops while accuracy goes up; lower it and you find more, with more false hits. maxSideLen caps the longest image edge at 640, which is noticeably faster, and very small text may not be recognized. Text-direction detection and angle voting only need to be on when the image is upside down. On an ordinary upright interface, leaving them off saves time. Recognition returns an array. Each item has the text, a confidence, and a coordinate range, and you can tap from the coordinates. While debugging, print both label and confidence to the log. When recognition is off, you can see whether it missed the text or read it wrong.

YOLO is for cases the earlier routes do not cover: the target changes shape, the position is not fixed, and you know which category it belongs to. It adds a model file. Initialization and tuning are both more complicated than what came before, and the training method matches the Android side. If image match, color, and OCR together keep the flow stable, you do not need a model. After you have tried those three and they genuinely cannot solve it, then use YOLO.

When recognition fails, check in this order

When recognition fails, this order usually locates the problem. First confirm you got a screenshot. If the screenshot comes back empty, every later recognition fails. Then confirm the coordinate systems match. On HarmonyOS Next, taps and recognition are both based on screenshot coordinates. If you set a screen size along the way, confirm both sides use the same set. Then confirm OpenCV is initialized and the mat mode switch succeeded. Last, print the recognized coordinates together with that screenshot to the log. A lot of “element not found” cases found it and the coordinates were off. Put them next to the screenshot and it is obvious. When a script crashes, save that screenshot before you exit. Reading the log afterward is less useful than seeing the screen as it was.

---

About EasyClick: A mobile automation script development platform covering three phone ecosystems: Android without root, iOS without jailbreak, and HarmonyOS. It also offers Apple script writing, Apple cluster control, Android cluster control, remote cloud control, and AI phone cluster control. → Explore all products