You snap a bug on screen, drop the screenshot into the planner, and get a ready-made task card. What happens under the hood? This article walks through the recognition pipeline—from the image to a structured task with a title and quadrant.
Why parse screenshots
Copying text is not always practical. Sometimes the task is an error screenshot, a photo of a whiteboard, or a capture of a messenger chat where selecting text is impossible or takes too long. A screenshot carries context more faithfully than a retelling.
How the pipeline works
Step 1. Image upload
The user picks a file (JPEG, PNG, or WebP) or drags it into the form. The image is sent to the server as base64 or in a multipart request.
Step 2. A vision model describes the image
The image is passed to a vision model (for example, GLM-4.5V via Hugging Face or Groq Vision). The model returns a text description of what it sees on the screenshot—perhaps “a Telegram chat screenshot discussing a report deadline” or “a 500 error on the login page.”
Step 3. A text model builds the task
The description from step 2 goes to a text LLM with a prompt: “From this screenshot description, formulate a task: title, description, Eisenhower quadrant.” The result is returned to the user as a draft card.
Why a two-step process
You could feed the image straight into one model and ask for a task in one shot. In practice, a two-step flow is more reliable:
- Vision models are strongest at describing what they see, without forcing structure prematurely
- Text models are better at wording titles and setting priorities
- If the vision model mis-describes something, you see it in the intermediate output—easier to debug than a black box
Limitations
Fine print
If text on the screenshot is small or blurry, the vision model may miss it or misread it. Higher resolution and contrast generally improve results.
Busy layouts
A screen full of tabs, pop-ups, and notifications can confuse the model. It does not know which part of the screen is “main.” Crop to the area that matters.
Handwriting
Photos of whiteboards or handwritten notes are harder to read than on-screen text. Vision models improve each generation, but there is still no perfect handwriting OCR.
One task at a time
If the screenshot shows several distinct tasks (for example, a long list in a messenger), the model will tend to merge them into one card. Prefer separate screenshots or text-based parsing.
Which models are used
In AI Planner, vision providers are wired through an OpenAI-compatible API. Priority: Groq Vision, then xAI Vision, then Hugging Face (GLM-4.5V). Whichever key comes first in configuration is the one that runs.
The text step follows the same idea: DeepSeek, Groq, Hugging Face. Models are chosen to stay accessible or free-tier friendly so the service can run without a subscription.
Summary
Screenshot parsing is a two-step process: a vision model describes the image, a text model builds the task. It works well on sharp captures with clear text; less so on handwritten notes and cluttered layouts. Try it—photograph a bug or a slice of chat and drop it into AI Planner.