DiveInsight
Follows marine life through dive video, outlines each animal frame by frame, and suggests what it is.
Research: in development. We're looking for marine biologists and dive professionals on Koh Tao to review the footage with us. Until an expert has checked a sample, there is no accuracy figure.

Blacktip reef shark · Tanote Bay 
Green sea turtle · Light House 
Pink skunk anemonefish · Chumphon Pinnacle 
Barracuda, named to family · Chumphon Pinnacle 
Oxeye scad school · Southwest Pinnacle
Pipeline output on our own GoPro footage from Koh Tao, Thailand. No expert has checked the outlines or names yet; the shark, barracuda and scad names were blind-checked against the footage.
What it's for
The aim is to automate reef biodiversity surveys from the footage divers already shoot, so that a recreational diver with a GoPro can take part as a citizen scientist.
Know what you saw
A list of the animals on a dive, each with its own outlined clip, so a diver doesn't have to identify them from memory.
Turn dives into survey data
Every animal is followed through the clip and named where the model is confident. That is the starting point for counting species on a reef from ordinary dive video.
Label faster
Outlines and tracks load into an annotation tool as a first draft, so people correct labels instead of drawing them.
Model card
- Task
- Find, outline and track each animal in dive video, then suggest a species or family name for it.
- Inputs
- Dive video. Built and tested on 4K GoPro footage from recreational dives.
- Outputs
- An outline of each animal on every frame, a track that follows it through the clip, a suggested name at species or family level (or none), and a list of what each clip contains.
- Architecture
- A pipeline of models. A detector chooses where to look, including our own on a frozen DINOv3 backbone. SAM 2 outlines and tracks each animal. BioCLIP 2 compares 4K crops of each animal with a Koh Tao species list, and rules fall back to genus or family when it isn't sure.
- Detector benchmark
- On the open Community Fish Detection benchmark (all 17 sources), our DINOv3 box detector scores AP 0.520 (AP50 0.786) with the backbone fine-tuned, and 0.397 with it frozen. RF-DETR-Medium, re-scored on the same frames, scores 0.602 (0.840), so ours is not the most accurate detector on this benchmark.
- Headline metric
- None yet
- Why
- No expert has reviewed the pipeline’s output yet. We’ll quote a number once a marine biologist or dive professional has checked a sample with us. If that could be you, get in touch.
- Scale so far
- ~1,000 clips filmed by our own divers (Nov 2026). A first pass over our footage produced about 3 million outlines, all unreviewed.
Limits
- No accuracy figure yet, because no expert has reviewed the output.
- Names are suggestions. The naming stage failed its own held-out test in November: 1 of 5 clips passed.
- A "tracked" count is a count of tracks, not fish. In a big school one fish can become several tracks.
- All the footage is diver GoPro video from Koh Tao reefs. Other regions and fixed cameras are untested.
- An outline can jump to the background when an animal swims behind coral.
- Coral and open water are sometimes outlined as fish.
How it works
Dive video
4K GoPro footage is cut into overlapping windows and scaled to work frames 1024 px on the long side.
Detect
A detector finds candidate animals and chooses where the segmenter should look. A text prompt alone wasn't enough: in our tests it found nothing or two fish on the same clip, depending on the frame it was attached to.
Outline and track
SAM 2 turns each detection into an outline and follows it through the window. Tracks are stitched across windows.
Name
BioCLIP 2 scores 4K crops of each animal against a Koh Tao species list. Below a confidence bar, rules fall back to genus or family, or leave it unnamed.
Outputs
Outlines on every frame, a named track list, a list for each dive, and annotated clips like the ones above.
Select a stage to see the detail.
Build story
DiveInsight: teaching a pipeline to watch a dive
Where it stands, what we measured, and what still goes wrong.
Choosing a detector
We trained detectors on a DINOv3 backbone and scored them on the open Community Fish Detection benchmark. Our box detector reached AP 0.520 with the backbone fine-tuned and 0.397 with it frozen. RF-DETR-Medium, re-scored on the same frames, reached 0.602.
On our own dive footage the picture changed. In a side-by-side comparison the frozen-backbone detectors raised far fewer false alarms, where RF-DETR marked fins and a dive watch as fish. For a product divers use, that matters more than a benchmark score.
A detector has to choose where to look
We tested whether SAM 3 could find fish from a text prompt alone. On the same clip it found nothing when prompted at the first frame, and two fish when prompted at the 98th. So a detector picks the frame and the place, and SAM 2 does the outlining.
A first pass over our footage
We ran the pipeline over short clips from our own Koh Tao dives in the cloud. It gave about 3 million outlines, which went into a hosted annotation tool for review.
The problems were plain to see: coral outlined as fish, two fish merged into one, and dense scenes hitting the cap on how many objects we track at once.
Naming what we see
We built a species list for Koh Tao from public iNaturalist, GBIF and WoRMS records, and used BioCLIP 2 to suggest a name for each tracked animal. The rules give a species name only above a confidence bar. Otherwise they give a genus or family name, or nothing.
We wrote the test down before running it, on five clips the rules had never seen. One passed. On one school, two independent blind checks of eight tracks labelled “Fusilier” named all eight as scads. Names stay suggestions until that’s fixed.
What's next
Collect more clips from divers. Make the pipeline cheaper to run on each dive. Put DiveInsight on the InsightML Portal, where it’s in development.
Credits
Built by InsightML.
Footage: our own dives on Koh Tao, Thailand. Detector benchmark: the Community Fish Detection Dataset. Species list: iNaturalist, GBIF and WoRMS records. Models: DINOv3, SAM 2 and BioCLIP 2.