Recording session leakage inflates accuracy in marine mammal call classification

Erzsebet Frey 

Splitting clips at random overstated the accuracy of a marine mammal call classifier by up to 23 percentage points, because clips from the same recording tape landed in both training and test data. I classified 4,761 recordings of nine Arctic and subarctic marine mammal species from the Watkins Marine Mammal Sound Database, recorded on 148 tapes between 1949 and 1997, at two levels: species and taxonomic group (baleen whales, toothed whales, pinnipeds). A random forest on 90 acoustic features, a small transformer trained from scratch on log mel spectrograms and linear classifiers on embeddings from the pretrained BirdNET and Perch 2.0 models were evaluated with 5 fold cross validation grouped by tape, and the random forest and both embedding classifiers were also scored with random splits.

With random splits the random forest reached 97% accuracy for both tasks. With whole tapes held out, species accuracy fell to 74% (balanced accuracy 49%, against 89% with random splits), while group accuracy fell to 91% (balanced accuracy 71%). The loss was concentrated in species recorded on few tapes: F1 for narwhal fell from 0.91 to 0.02 and for ringed seal from 0.73 to 0.00, with most of their clips predicted as killer whale. The transformer matched the random forest on balanced accuracy for groups (71%) but trailed it for species (42%). Embeddings from the pretrained BirdNET and Perch 2.0 models did best on unseen tapes (species balanced accuracy 66% and 69%) but were still inflated by 28 and 26 points under random splits. Unsupervised clustering recovered neither species nor groups.

Random splits of clip level bioacoustic data therefore measure partly how well a model recognises recording conditions, not animals, even when the model is pretrained. I recommend grouping cross validation folds by recording session, reporting balanced accuracy and per class scores, and reporting the random split score alongside as a measure of leakage.