DiveInsight

  • Research
  • Marine
  • Built by InsightML

Follows marine life through dive video, outlines each animal frame by frame, and suggests what it is.

Research: in development. We're looking for marine biologists and dive professionals on Koh Tao to review the footage with us. Until an expert has checked a sample, there is no accuracy figure.

Pipeline output on our own GoPro footage from Koh Tao, Thailand. No expert has checked the outlines or names yet; the shark, barracuda and scad names were blind-checked against the footage.

What it's for

The aim is to automate reef biodiversity surveys from the footage divers already shoot, so that a recreational diver with a GoPro can take part as a citizen scientist.

  • Know what you saw

    A list of the animals on a dive, each with its own outlined clip, so a diver doesn't have to identify them from memory.

  • Turn dives into survey data

    Every animal is followed through the clip and named where the model is confident. That is the starting point for counting species on a reef from ordinary dive video.

  • Label faster

    Outlines and tracks load into an annotation tool as a first draft, so people correct labels instead of drawing them.

Model card

Task
Find, outline and track each animal in dive video, then suggest a species or family name for it.
Inputs
Dive video. Built and tested on 4K GoPro footage from recreational dives.
Outputs
An outline of each animal on every frame, a track that follows it through the clip, a suggested name at species or family level (or none), and a list of what each clip contains.
Architecture
A pipeline of models. A detector chooses where to look, including our own on a frozen DINOv3 backbone. SAM 2 outlines and tracks each animal. BioCLIP 2 compares 4K crops of each animal with a Koh Tao species list, and rules fall back to genus or family when it isn't sure.
Detector benchmark
On the open Community Fish Detection benchmark (all 17 sources), our DINOv3 box detector scores AP 0.520 (AP50 0.786) with the backbone fine-tuned, and 0.397 with it frozen. RF-DETR-Medium, re-scored on the same frames, scores 0.602 (0.840), so ours is not the most accurate detector on this benchmark.
Headline metric
None yet
Why
No expert has reviewed the pipeline’s output yet. We’ll quote a number once a marine biologist or dive professional has checked a sample with us. If that could be you, get in touch.
Scale so far
~1,000 clips filmed by our own divers (Nov 2026). A first pass over our footage produced about 3 million outlines, all unreviewed.

Limits

  • No accuracy figure yet, because no expert has reviewed the output.
  • Names are suggestions. The naming stage failed its own held-out test in November: 1 of 5 clips passed.
  • A "tracked" count is a count of tracks, not fish. In a big school one fish can become several tracks.
  • All the footage is diver GoPro video from Koh Tao reefs. Other regions and fixed cameras are untested.
  • An outline can jump to the background when an animal swims behind coral.
  • Coral and open water are sometimes outlined as fish.

How it works

  1. Dive video

    4K GoPro footage is cut into overlapping windows and scaled to work frames 1024 px on the long side.

  2. Detect

    A detector finds candidate animals and chooses where the segmenter should look. A text prompt alone wasn't enough: in our tests it found nothing or two fish on the same clip, depending on the frame it was attached to.

  3. Outline and track

    SAM 2 turns each detection into an outline and follows it through the window. Tracks are stitched across windows.

  4. Name

    BioCLIP 2 scores 4K crops of each animal against a Koh Tao species list. Below a confidence bar, rules fall back to genus or family, or leave it unnamed.

  5. Outputs

    Outlines on every frame, a named track list, a list for each dive, and annotated clips like the ones above.

Build story

DiveInsight: teaching a pipeline to watch a dive

Where it stands, what we measured, and what still goes wrong.

  1. Choosing a detector

    We trained detectors on a DINOv3 backbone and scored them on the open Community Fish Detection benchmark. Our box detector reached AP 0.520 with the backbone fine-tuned and 0.397 with it frozen. RF-DETR-Medium, re-scored on the same frames, reached 0.602.

    On our own dive footage the picture changed. In a side-by-side comparison the frozen-backbone detectors raised far fewer false alarms, where RF-DETR marked fins and a dive watch as fish. For a product divers use, that matters more than a benchmark score.

  2. A detector has to choose where to look

    We tested whether SAM 3 could find fish from a text prompt alone. On the same clip it found nothing when prompted at the first frame, and two fish when prompted at the 98th. So a detector picks the frame and the place, and SAM 2 does the outlining.

  3. A first pass over our footage

    We ran the pipeline over short clips from our own Koh Tao dives in the cloud. It gave about 3 million outlines, which went into a hosted annotation tool for review.

    The problems were plain to see: coral outlined as fish, two fish merged into one, and dense scenes hitting the cap on how many objects we track at once.

  4. Naming what we see

    We built a species list for Koh Tao from public iNaturalist, GBIF and WoRMS records, and used BioCLIP 2 to suggest a name for each tracked animal. The rules give a species name only above a confidence bar. Otherwise they give a genus or family name, or nothing.

    We wrote the test down before running it, on five clips the rules had never seen. One passed. On one school, two independent blind checks of eight tracks labelled “Fusilier” named all eight as scads. Names stay suggestions until that’s fixed.

  5. What's next

    Collect more clips from divers. Make the pipeline cheaper to run on each dive. Put DiveInsight on the InsightML Portal, where it’s in development.

Credits

Built by InsightML.

Footage: our own dives on Koh Tao, Thailand. Detector benchmark: the Community Fish Detection Dataset. Species list: iNaturalist, GBIF and WoRMS records. Models: DINOv3, SAM 2 and BioCLIP 2.