What is data annotation and how it shapes your computer vision model

What is data annotation in plain terms

Data annotation is the process of adding structured labels to data. A photo of a warehouse shelf is just pixels until someone marks where each product sits and which product it is. Once those marks exist in a format a training script can read, the image becomes a training example.

The label is the answer you want the model to give. When you annotate an image with a box around a forklift and the class "forklift", you are telling the model: given these pixels, this is the answer. Repeat that across thousands of images with varied lighting, angles and backgrounds, and a detector learns to find forklifts on its own.

Annotation is not the same thing as data collection or data cleaning, although the three overlap. Collection gets the images. Cleaning removes duplicates, corrupt files and images that do not belong. Annotation adds the ground truth. Most labeling problems that show up later can be traced back to skipping one of the first two steps.

Why labels matter more than model choice

A model can only be as precise as the labels it learns from. If half your annotators include the side mirror in a car box and half do not, the model learns a fuzzy boundary and your evaluation numbers become hard to trust. Swapping architectures rarely fixes that. Fixing the labels usually does.

Labels also define what your model is allowed to know. If your schema has no class for "partially occluded pallet", the model never learns that case exists. Annotation decisions are product decisions, and they deserve the same care as the training code.

The main types of image annotation

Each annotation type answers a different question. Pick the simplest type that answers yours, because richer labels cost more time per image and more review effort.

For object detection, start with bounding box annotation. When you need exact shapes, area or pixel level output, move to an image segmentation tool with polygons and brush masks. Our deeper comparison of bounding box vs segmentation covers when the extra effort pays off.

Classes, attributes and hierarchy

A class is the main name of an object, such as "car" or "scratch". An attribute is a property you want to record without creating a new class, such as "occluded", "color: red" or "severity: minor". Keeping these separate prevents class explosion. Instead of "red car occluded" as its own class, you keep one class and three attributes.

Hierarchical classes help when you want fine labels today but coarse training later. You can label "sedan", "SUV" and "van" under a parent "vehicle", then decide at export time which level to train on.

Who does data annotation

There are three common setups, and many teams mix them.

• In house labelers who know the domain. Best for specialized images such as industrial defects or crop disease, where context matters.
• ML engineers labeling their own data. Fast for small pilots, expensive at scale because engineer time is the scarcest resource on the team.
• Managed annotators working to your guidelines. Useful when volume grows faster than your team. On ImageLabeling, data annotation services provide dedicated annotators from 899 USD per month per annotator on Business and Enterprise plans, inside the same review workflow your own team uses.

Whoever labels, the same rule holds: a reviewer who did not draw the label checks it before it reaches training data.

Where AI pre-labeling fits

Modern annotation rarely starts from a blank canvas. An AI model can propose boxes, polygons, masks and classes for your schema, each with a confidence score, and a person accepts, edits or rejects each proposal. This changes the job from drawing to verifying, which is faster for most images and keeps a human responsible for every label. Auto labeling on ImageLabeling works this way: the model suggests, your labelers decide, and nothing enters a dataset version without passing review.

How to write annotation guidelines that hold up

Guidelines are the single most important document in a labeling project. They turn personal judgment into a shared rule. Good guidelines are short, visual and specific about edge cases.

• Define each class in one sentence and show two correct examples and one common mistake.
• State the tightness rule for boxes, for example "edges touch the outermost visible pixel, do not include shadows".
• Say what to do with occlusion: label the visible part only, or estimate the full extent. Pick one.
• Set a minimum size, such as "skip objects smaller than 10 pixels on the short side", so labelers do not argue about specks.
• Explain truncation at the image edge: label it and mark the attribute "truncated".
• Define reflections, screens, posters and toys. A picture of a car on a billboard is either a car or ignored, and the guideline must say which.
• List what to do when unsure: flag the image to the issue queue instead of guessing.

Update the guideline every time a reviewer sends the same mistake back twice. A guideline that never changes usually means nobody reads it.

Edge cases worth deciding on day one

Crowds and dense scenes are the classic trap. If forty bottles sit on a shelf, decide whether each bottle gets a box or whether a group box is allowed. Groups save time but confuse detectors trained to count.

Ambiguous classes are the second trap. A "truck" and a "van" overlap at the edges. Write the deciding feature (cargo area separate from cab, or not) instead of trusting intuition. If two experienced people disagree on a sample image, the guideline is incomplete.

A typical data annotation workflow

Most computer vision teams follow a similar loop. The details vary, the order rarely does.

• Upload images by drag and drop, a zip file or a cloud bucket URL.
• Define the label schema: classes, colors, attributes and any hierarchy.
• Run AI pre-labeling on a batch so labelers start from proposals.
• Labelers accept, edit or reject each object, using keyboard shortcuts to keep speed up.
• Reviewers approve images or send them back with a comment.
• Freeze a dataset version and split it into train, validation and test.
• Export in the format your training code reads, such as COCO JSON or YOLO txt.
• Train, evaluate, find failure cases, and label more images that look like them.

The data annotation platform page shows how ImageLabeling handles each step, from upload to versioned export.

Formats are part of the job

Annotation is only useful if your training pipeline can read it. COCO JSON stores all images and labels in one file, YOLO uses one text file per image with normalized coordinates, and Pascal VOC uses one XML file per image. Picking a format early avoids conversion bugs later. Our guide to COCO vs YOLO format compares them field by field.

How to measure annotation quality

You cannot fix what you do not measure. Three checks catch most problems.

• Review rate: every image, or a fixed share of images, passes through a second person before training.
• Consensus: a sample of images is labeled by two or more people independently, then compared. Low agreement points to a weak guideline, not a weak labeler.
• Per labeler QA: track how often each person's work is sent back and for what reason. This turns review comments into coaching instead of blame.

When you compare dataset versions, look at what changed: labels added, edited or removed. A sudden jump in removed labels after a guideline update is expected. A jump with no guideline change is a warning.

Common data annotation mistakes

• Labeling before the schema is stable, then relabeling everything when a class splits in two.
• Mixing guideline versions inside one dataset without recording which images used which rule.
• Leaving duplicates and near duplicates in both training and test splits, which inflates evaluation results.
• Ignoring background images. A detector needs images where the object is absent, labeled as empty, to learn when not to fire.
• Treating AI proposals as truth. Pre-labels speed things up, but every accepted label is still a human decision.

What data annotation costs in practice

Cost depends on three variables: images, label complexity and review depth. A classification label takes seconds. A detailed segmentation mask on a busy scene can take minutes. Review roughly adds a second pass on top.

The useful number is cost per accepted label, not cost per hour. Track it per class. If one class costs five times more than the others, it is usually a guideline problem (labelers hesitate) or an image problem (the object is tiny or blurred), and both are fixable.

ImageLabeling plans are monthly or yearly subscriptions, starting with Starter at 49 USD per month or 24 USD per month billed yearly, with 1,000 AI pre-labeled images per month and three seats. See pricing for Team, Business and Enterprise limits.

Data annotation across computer vision domains

The basic idea of data annotation is the same everywhere, but each domain brings its own traps. Knowing them in advance saves a round of relabeling.

• Manufacturing defect detection. Defects are small, rare and often look like dust, glare or normal texture. Guidelines need close up reference images and a clear rule for borderline cases. Background images of good parts matter as much as defect images.
• Retail shelves. Scenes are dense, products repeat, and packaging changes often. Hierarchical classes help: label the product family now and the exact item later if needed.
• Agriculture and aerial imagery. Objects are tiny relative to the image, and the same field looks different by season and time of day. Tiling large images before labeling keeps objects visible at a useful zoom.
• Robotics and warehouses. The robot cares about what it can grasp or avoid, so occlusion and truncation rules must match how the robot uses the prediction.
• Medical research images. Labels often require trained specialists, and the guideline must reflect the study protocol. Annotation for research and model training is different from diagnosis, and the labeling tool is not a diagnostic device.
• Safety and PPE. Small items like helmets, vests and gloves are partly hidden by posture. Decide whether a helmet held in a hand counts as worn.

Annotation for images versus video

Video is a sequence of images, and many teams label it as such: extract frames at a fixed interval, remove near duplicates, then label the frames like any other image set. Labeling every frame of a video is rarely worth it, because consecutive frames carry almost the same information. Sampling one frame every second or two, plus extra frames around rare events, gives better coverage for the same effort. On ImageLabeling, frames are uploaded as images and labeled in the standard editor.

Getting started with your first labeled dataset

Start small and concrete. Pick one model question, write a one page guideline, label two hundred images, review all of them, then train a quick baseline. The baseline tells you which classes and conditions need more examples, and that list becomes your next labeling batch.