Labeled dataset building guide for computer vision teams

Start your labeled dataset from the production use case

Every good labeled dataset begins with a sentence that describes what the model will do in production. "Detect missing products on retail shelves from fixed ceiling cameras" is a useful sentence. "Detect products" is not. The specific version tells you which cameras, which distances, which lighting and which errors matter.

From that sentence, write down three things before you collect a single image.

• The decision the model supports, and what a miss or a false alarm costs.
• The conditions in production: cameras, resolution, angles, light, motion.
• The output you need: boxes, polygons, segmentation masks, keypoints or image classes.

The output type is a cost decision as much as a technical one. Boxes are fast to draw. Masks take longer but describe shape precisely. If you are unsure, our comparison of bounding box vs segmentation explains when each one pays off.

Collect images that match production conditions

The most common dataset mistake is collecting images that are easy to get instead of images that look like production. Stock photos, product renders and clean studio shots teach a model a world it will never see. Collect from the real cameras, at real distances, across real shifts and seasons whenever you can.

Aim for coverage, not just volume. Plan your collection across the conditions that change appearance.

Remove near duplicates before labeling. Video frames sampled every frame are almost identical and waste labeling time while inflating your image count. Sample frames at an interval that produces visible change, and drop frames that add nothing new. In ImageLabeling, frames are uploaded as regular images by drag and drop, as a zip, or from a cloud bucket URL.

If you are wondering how big the first collection should be, read how many images to train object detection for practical starting points and how to measure them with learning curves.

Design the label schema before anyone draws

The label schema is the contract between your labelers and your model. It lists the classes, their colors, the attributes each object can carry, and the annotation type for each class. Changing it halfway through means relabeling, so spend an hour on it now.

• Keep classes mutually exclusive. If an object could fit two classes, the schema is unclear.
• Merge classes the business does not need to separate. Each extra class costs examples.
• Use attributes for properties that vary within a class, such as occluded, truncated, color or condition.
• Use hierarchical classes when you may need both a coarse and a fine label later, for example vehicle and then car, truck, van.
• Add an explicit class or attribute for uncertain cases, so labelers do not guess silently.

In ImageLabeling the schema lives with the project: classes with colors, attributes per object, hierarchical classes, and reusable templates you can copy into the next project. The AI pre-labeling reads the same schema, so the drafts it produces already use your class names.

Write labeling guidelines with pictures, not paragraphs

Guidelines turn your schema into consistent labels across people and weeks. The best guidelines are short, visual, and focused on edge cases, because labelers rarely disagree on the easy objects.

A practical guideline document covers these points.

• A definition and two or three example images for each class.
• Box or polygon tightness: where the edge goes, whether shadows count.
• Occlusion rules: label the visible part, or estimate the full extent.
• Truncation at the image border and the minimum visible size to label.
• Groups and crowds: when to label each object and when to skip.
• What to do when unsure: flag the object or open an issue instead of guessing.

Treat the guideline as a living document. Every time a reviewer sends work back for the same reason twice, add a picture example to the guideline. Version the document alongside the dataset, so you know which rules a given set of labels followed.

Label faster with AI pre-labeling and a keyboard first editor

Manual labeling from an empty canvas is where most dataset budgets go. AI pre-labeling draws a first draft for every image in your schema, and labelers accept, edit or reject each object. On images with many objects, correcting a draft is far less work than drawing from zero.

With auto labeling in ImageLabeling, the AI pre-labels boxes, polygons, segmentation masks, keypoints and image classes, and each object shows a confidence score. Labelers accept confident objects with one keystroke and focus their attention where the model is unsure. The editor is keyboard first: B draws a box, P a polygon, K a keypoint, A accepts, and D moves to the next image. For pixel level work, the image segmentation tool adds a brush and polygon tools for masks.

Two habits keep pre-labeling honest.

• Never accept a draft without looking at it. Pre-labels are a starting point, not ground truth.
• Watch for systematic errors in the drafts, such as a class the model always misses, and correct the guideline or the schema if needed.

If your team does not have the capacity to label in house, data annotation services give you dedicated, vetted annotators who work inside the same project and follow your guidelines.

Review every batch before it enters the labeled dataset

Labels should not enter the training data until someone other than the labeler has checked them. A simple two stage workflow works for most teams: the labeler submits, a reviewer approves or sends the image back with a comment.

• Review a fixed share of every labeler's work, and all work from new labelers.
• Use comments that point to the guideline rule, so feedback teaches.
• Run consensus on a sample: several labelers label the same images and you compare the results.
• Track recurring problems in an issue queue instead of chat threads.

For a deeper look at agreement metrics, gold sets and per labeler scoring, see our guide on annotation quality control. In ImageLabeling, the full review workflow with consensus is part of Team, and Business adds a QA score per labeler.

Freeze versions and split train, validation and test once

A labeled dataset changes constantly. New images arrive, reviewers fix labels, classes get merged. If you train on a moving target, you cannot tell whether a better metric came from the model or from different labels. Dataset versions solve this.

• Freeze a version before every training run and record its name with the experiment.
• Compare versions to see which labels were added, changed or removed.
• Create the train, validation and test split once and keep it stable across versions.
• Keep near duplicate images and frames from the same sequence in the same split, so the test set does not leak.
• Add new images to the training split by default, and to validation and test only on purpose.

A common split is a large training share with smaller validation and test shares, but the right proportion depends on how much data you have. What matters most is that the validation and test sets represent production conditions, including the hard cases. ImageLabeling stores dataset versions (3 on Starter, unlimited on other plans), compares them, and creates the train, validation and test split for export.

Export in the format your training code expects

The last step is getting labels into your training pipeline without conversion scripts that silently drop data. Decide the format early, because it affects what annotation types you can carry.

The coco annotation tool and yolo labeling tool pages explain each export in detail, and our article on coco vs yolo vs pascal voc compares the formats side by side. You can also import existing labels in the same formats, so earlier work is not lost. On Team plans and above, the API lets you trigger exports and fetch labels from your own pipeline.

Common labeled dataset mistakes and how to avoid them

Most dataset problems show up weeks later as a model that will not improve. They are much cheaper to prevent at the start than to diagnose after training.

• Changing the schema mid project without relabeling earlier images. Old and new labels then follow different rules, and the model learns both.
• Letting the test set drift. If test images are swapped or relabeled between runs, metrics stop being comparable.
• Labeling only easy images. A dataset full of clear, centered objects scores well in validation and fails in production.
• Ignoring negatives. Without images that contain no target objects, false positives climb in real scenes.
• Skipping review for speed. Unreviewed labels look like progress until the model plateaus.
• Losing track of which labels trained which model. Without frozen versions, you cannot reproduce or explain a result.

Keep the labeled dataset alive after the first model

A dataset is never finished. Once the first model runs, it becomes your best tool for improving the data. Run it on new production images, collect the cases where it is wrong or unsure, label those, and add them to the next version. This loop focuses labeling effort on the images that teach the model something new, instead of on more of what it already knows.

Set a simple rhythm for it: a regular batch of fresh production images, pre-labeled by the current model, corrected and reviewed, frozen as a new version, then used for retraining. Over time the dataset grows exactly where the model is weakest. On Business plans and above, webhooks notify your pipeline when a batch is reviewed, so retraining can start without anyone checking the queue by hand.