Crop Counter

  • Beta
  • Agriculture
  • Built by InsightML

Crop Counter finds each plant in a top-down field photo and marks it as a point, so every count comes with the locations behind it. The code, training notebooks and trained wheat decoder are open source.

In Beta. Open source under AGPL-3.0. To use it commercially without the AGPL conditions, ask us about a commercial licence. The hosted wheat-count endpoint on our portal still runs the previous model, which returns a count only. Its API reference (endpoints, rate limits and errors) is in the repository.

Three rows of field images, each beside the model's heatmap. Top: a drone image of maize with red rings on 160 predicted tassels and green dots on 159 labelled ones. Middle: a close view of wheat with red rings on 43 predicted heads. Bottom: emerging wheat seedlings on bare soil with red rings on 70 predicted plants and green dots on 66 labelled ones. Each heatmap shows one bright spot per object on a dark background.
Three tasks, three trained decoders, the same code. Red rings are predicted points, green dots hand labels, and the right column is the model's heatmap. Top: an MTC-UAV validation image, 160 predicted against 159 labelled. Middle: a GWHD 2021 test image, 43 predicted. Bottom: a held-out quadrat from the repository's demo set, 70 predicted against 66 labelled.

What it's for

  • Plant establishment

    Count emerged wheat plants in a quadrat photo and convert the count to plants per square metre. This is the task the shipped decoder was trained for.

  • Heads and tassels

    The same code counts wheat heads and maize tassels. Both are worked examples in the repository, with notebooks, reports and trained decoders.

  • Your own crop

    Label one point per plant in CVAT or COCO format and train a new decoder. The maize example trained in about 2 hours on one RTX 4070 Ti Super, and Apple Silicon is supported.

Architecture

The count comes from points the model has located: 43 means 43 peaks, each one at a place in the photo. A density model sums a map instead, and can reach the right total for the wrong reasons. Only the decoder is trained. The backbone’s large-scale pretraining handles texture and shape, so a few thousand point-labelled images are enough.

Crop Counter model architectureA top-down RGB image goes through a frozen DINOv3 ConvNeXt-B backbone, which gives four feature maps at strides 4, 8, 16 and 32 with 128, 256, 512 and 1024 channels. A 1×1 convolution projects each map to 192 channels. The trained pyramid decoder starts from the stride-32 map and, three times, upsamples by 2, concatenates the projected map at the new stride and applies two 3×3 convolution, GroupNorm and GELU layers, ending at stride 4. A 1×1 head gives one logit channel per class. A sigmoid turns the logits into a heatmap with one peak per plant. Decoding keeps local maxima in a 3×3 window that score at least tau, and point non-maximum suppression drops duplicates. The output is a point and a class for each plant, and the count is the number of points. In training only, each labelled point becomes a Gaussian peak with sigma 2, overlapping peaks merged by maximum, and a focal loss on the logits trains the decoder; no gradients reach the backbone.Frozen · pretrained, no gradientsTrained · 3.36 M parametersDecode-time settings · no weightsBackbone · DINOv3 ConvNeXt-BPyramid decoderTraining only2× upsample2× upsample2× upsamplesigmoidgradients: decoder onlynone to the backboneRGB imageany H × W divisible by 32Stage 1stride 4 · 128 chStage 2stride 8 · 256 chStage 3stride 16 · 512 chStage 4stride 32 · 1024 ch1×1 → 1921×1 → 1921×1 → 1921×1 → 192Fuse · stride 4concat → 2 × (3×3 conv, GN, GELU)Fuse · stride 8concat → 2 × (3×3 conv, GN, GELU)Fuse · stride 16concat → 2 × (3×3 conv, GN, GELU)Seed · stride 32the projected stage 4 map1×1 head · stride 4one logit map per classHeatmapone peak per plantDecodelocal max in a 3 × 3 windowkeep peaks scoring ≥ τpoint NMS drops duplicatesPoints (x, y) + classcount = number of pointsPoint labelsGaussian per point · σ 2 · merged by maxFocal loss · α 2 · β 4Crop Counter model architectureA top-down RGB image goes through a frozen DINOv3 ConvNeXt-B backbone, which gives four feature maps at strides 4, 8, 16 and 32 with 128, 256, 512 and 1024 channels. A 1×1 convolution projects each map to 192 channels. The trained pyramid decoder starts from the stride-32 map and, three times, upsamples by 2, concatenates the projected map at the new stride and applies two 3×3 convolution, GroupNorm and GELU layers, ending at stride 4. A 1×1 head gives one logit channel per class. A sigmoid turns the logits into a heatmap with one peak per plant. Decoding keeps local maxima in a 3×3 window that score at least tau, and point non-maximum suppression drops duplicates. The output is a point and a class for each plant, and the count is the number of points. In training only, each labelled point becomes a Gaussian peak with sigma 2, overlapping peaks merged by maximum, and a focal loss on the logits trains the decoder; no gradients reach the backbone.Frozen · pretrained, no gradientsTrained · 3.36 M parametersDecode-time settings · no weightsBackbone · frozenDINOv3 ConvNeXt-BPyramid decodertrainedTraining only1×11×11×11×1sigmoidRGB imageany H × W divisible by 32Stage 1stride 4 · 128 chStage 2stride 8 · 256 chStage 3stride 16 · 512 chStage 4stride 32 · 1024 chFuse · stride 4concat → 2 conv blocksFuse · stride 8concat → 2 conv blocksFuse · stride 16concat → 2 conv blocksSeed · stride 32projected stage 41×1 head · stride 4one logit map per classHeatmapone peak per plantDecodelocal max in a 3 × 3 windowkeep peaks scoring ≥ τpoint NMS drops duplicatesPoints (x, y) + classcount = number of pointslaterals 1×1 → 192 ch · ladder 2× upsamplePoint labels → a Gaussian per point (σ 2),merged by max, never summed → focal loss(α 2, β 4). Gradients reach the decoder only.

How an image becomes points. Shown with the default ConvNeXt-B backbone; the small and large backbones change the stage widths, not the layout. Every part is in dinov3_pyramid.py and heatmap.py.

Design choices

  • ConvNeXt, not a ViT

    A hierarchical backbone gives real stride-4, 8, 16 and 32 maps for the decoder to fuse. With no positional embeddings it trains on 768 px tiles and runs on any image size divisible by 32.

  • A hybrid decoder

    Uniform 1×1 laterals and a single fine-scale output come from FPN. Merging each step by concatenation and a double conv block, rather than adding, comes from U-Net.

  • Targets merged by max

    Each labelled point becomes a Gaussian peak (σ 2 at stride 4). Overlapping peaks are merged by taking the maximum, never the sum, so two close plants stay two maxima with a dip between them.

  • Focal loss

    A penalty-reduced focal loss in the CenterNet style (α 2, β 4) on the raw logits, normalised by the number of peaks. The head starts at p ≈ 0.02 everywhere so empty ground does not swamp early training.

  • τ set after training

    The score threshold is swept on validation images once training ends, never fixed in advance. Each class can have its own threshold, read off one sweep.

  • Count and location scored apart

    Count error (MAE, RMSE, bias) and location (precision, recall and F1 with one-to-one matching) are reported separately. A good count with a poor F1 means over- and under-counts cancelling out.

Model card

Task
Find and count individual plants, or parts of plants such as wheat heads and maize tassels, in top-down RGB images. One output channel per class, so one model can count several kinds of object.
Inputs
A top-down RGB image of any size whose sides divide by 32, such as a quadrat photo, a field-scout crop or a drone image. The model is fully convolutional, so it trains on fixed-size tiles and runs on whole images without tiling.
Outputs
A list of points, each with an (x, y) position, a class and a score, plus the count per class. Every counted plant is a point you can check on the photo. The heatmap behind the points comes out at a quarter of the input resolution.
Architecture
A frozen DINOv3 ConvNeXt backbone (base by default; small and large also supported) gives four feature maps at strides 4, 8, 16 and 32. A trained pyramid decoder projects each map to 192 channels with a 1×1 convolution, then works up from stride 32 to stride 4. At each step it upsamples by 2, concatenates the map at that stride and applies two 3×3 convolution, GroupNorm and GELU layers. A 1×1 head gives one logit channel per class. Points come from a local-maximum test, a score threshold τ and point non-maximum suppression.
Size
3.36 million trained parameters, all in the decoder (a 13 MB checkpoint). The frozen backbone is Meta's DINOv3 ConvNeXt-B, a gated download from Meta that is not shipped with the code.
Training data
The shipped wheat emergence decoder: 1,153 point-annotated photos of emerging wheat in UK farm fields from autumn 2024, wheat and volunteer plants merged into one class, split by grower into 1,031 training and 122 validation photos. The worked examples train separate decoders on two public datasets: GWHD 2021 wheat heads (3,657 training images) and MTC-UAV maize tassels (200 training images).
Headline metric
MAE 8.59 · F1 0.729 Wheat emergence, shipped decoder: an average count error of 8.59 plants per photo, and F1 0.729 for locating each plant.
How measured
On 122 validation photos from growers whose photos were held out of training. The 1,153 labelled photos, training and validation together, average about 57 plants each. The checkpoint (lowest validation loss, epoch 24 of 50) and τ (swept to 0.35) were both chosen on these photos. F1 matches predicted to labelled points one-to-one within 24 px.
Metric
WDA 0.740 Wheat heads, GWHD 2021 test set: weighted domain accuracy 0.740 across 18 domains, with every prediction kept.
How measured
On 1,382 test images not used in training or tuning. A predicted point is correct if it falls inside an unmatched labelled box, taken in descending score order. Raising the score cut-off to 0.23, a value read off the test set itself, gives 0.742. The 2021 challenge's best entry scored 0.700 with box matching at IoU ≥ 0.5, which is a different rule.
Metric
MAE 14.4 · F1 0.902 Maize tassels, MTC-UAV drone images: an average count error of 14.4 tassels per image, and F1 0.902 for locating each tassel.
How measured
On the 106 validation images, about 207 tassels each. The dataset has no test split, and τ (0.3) and the best checkpoint (epoch 39 of 40) were chosen on these images.

Results

One trained decoder per task, on the same frozen backbone. Count error is the average absolute difference between predicted and labelled counts per image.

TaskDataScored onCount errorF1Notes
Wheat emergence1,153 farm photos, private122 val8.590.729Shipped decoder. τ 0.35, swept on the same photos
Wheat headsGWHD 2021, public1,476 val2.630.879τ 0.2, NMS radius 5. Test set: WDA 0.740
Maize tasselsMTC-UAV, public106 val14.40.902τ 0.3, NMS radius 5, 1,536 px tiles

A 13-epoch retrain of the wheat emergence recipe with the packaged code, about 3 hours on an Apple M-series laptop, reached 8.64 and 0.726 on the same 122 photos. The full reports are in the repository’s examples.

Limits

  • The scores are optimistic. For wheat emergence and maize tassels, τ and the best checkpoint were chosen on the same images that were scored, and there is no separate test set. Only the wheat-head example has one.
  • τ is a decode-time setting, not something the model learns, and it has to be calibrated for each dataset and class. The same wheat weights give an average count error of 13.83 at τ 0.3 and 8.59 at τ 0.35.
  • The wheat-head score uses our own point-in-box matching, adapted from the 2021 challenge's box matching at IoU ≥ 0.5, so it is not a like-for-like comparison with that leaderboard.
  • On maize tassels it counts less accurately than TasselNetV3 (average error 14.4 against 8.8 tassels per image). TasselNetV3 does not output a location for each tassel, so the localisation scores can't be compared.
  • Smeared or blurred patches can be decoded as two points. The wheat-head and maize examples raised the NMS radius from 1.5 to 5 to stop this.
  • Needs individual plants visible from above in colour photos. Multispectral, greenhouse and tray images are untested.
  • The hosted wheat-count endpoint on the InsightML portal still runs the previous model, a density-map counter that returns a count only. To get points from this model today, run the open-source code.

Run it

Python 3.12 or later, on CUDA, Apple Silicon or CPU. You fetch the DINOv3 backbone from Meta yourself; python -m cropcounter.weights says where to put it. Then count the demo photos with the shipped decoder:

import torch
from cropcounter import (
    CropTileDataset, load_checkpoint, records_from_folder, predict_prob, decode_classes,
)

device = torch.device("mps")  # or "cuda" / "cpu"
model, cfg = load_checkpoint("weights/decoder_best.pt", device, weights_dir="weights")
model.eval()

images = "examples/demo/data/val/images"
records = records_from_folder(images)
dataset = CropTileDataset(records, images, train=False, output_stride=cfg.output_stride)

for record, item in zip(records, dataset):
    prob = predict_prob(model, item["image"], device)  # (1, C, h, w) heatmap
    points, scores, class_ids = decode_classes(
        prob, cfg.class_names, tau=0.35, k=cfg.k, nms_radius=cfg.nms_radius,
        output_stride=cfg.output_stride, width=record.width, height=record.height,
    )
    print(record.name, len(points), "plants")

Training a new decoder is one command, python -m cropcounter.train –config …. The code, the shipped decoder and the trained decoders for wheat heads and maize tassels are released under AGPL-3.0, so a service built on them has to publish its source too. For other terms, ask us about a commercial licence.

Credits

Built by InsightML: Freddie Allen-Stevens and Liam Cracknell. Wheat emergence photos from farmers in BOFIN’s SLIMERS programme, point-annotated by an annotation vendor; the 20-photo demo set is anonymised and published with the programme’s permission. The worked examples use the Global Wheat Head Dataset 2021 (David et al.) and MTC-UAV (Lu et al.). The backbone is Meta’s DINOv3, used under Meta’s own licence.