<p>The new R package ohun allows to do exactly that (diagnose the performance of detection routines).</p>
<p><a href="https://marce10.github.io/ohun/index.html" rel="nofollow noreferrer">https://marce10.github.io/ohun/index.html</a></p>
<p>It takes a reference table and the detection output and returns a bunch of performance metrics, including number of true positives, false positives, false negatives, precision and recall. It also provides some time related metrics like the amount of overlap to true positives.</p>
<p>I am still working on it so would be happy to hear suggestions for new features.</p>
<p>Edit: to echo @lostanlen, the <a href="http://tut-arg.github.io/sed_eval/" rel="nofollow noreferrer">python package sed_eval</a> for evaluating (diagnosing) sound event detection. It has lots of cool features to evaluate segment-based and event-based detection</p>

CC BY-SA 4.0
2022-08-10T11:43:57.920Z
Awesome you are working on it! would be nice to have a summary of species / events etc. per file or in total as a higher level of comparison. Would be nice to have long-term visualization of annotations for overview and quick visula comparison.
Pavel Linhart
CC BY-SA 4.0
2022-08-10T17:10:01.070Z
Could you elaborate a bit on your suggestions about visualizing annotations? You mean labeled spectrograms?
Marcelo Araya-Salas
CC BY-SA 4.0
2022-08-11T09:36:31.437Z
Visualizing ground truth events next to detections so that you have quick graphical overview where ground truth and detection match or disagree along with the numeric performance metrics. I posted a question related to this problem: https://bioacoustics.stackexchange.com/questions/1043/tools-to-visualize-and-compare-species-detections-in-long-term-recordings
Pavel Linhart
CC BY-SA 4.0
2022-08-16T22:38:15.793Z
Hello @MarceloAraya-Salas, i'm afraid there is a problem in your algorithm for counting true positives. https://github.com/maRce10/ohun/blob/master/R/diagnose_detection.R#L170-L183

As you noticed, you may count true positives multiple times and obtain recall values above 1. You have a heuristic to prune them but it leads to a suboptimal TP. The optimal way to address this is via bipartite graph matching (Hopcroft-Karp)

See mir_eval package: https://craffel.github.io/mir_eval/
implementation by Brian McFee https://github.com/craffel/mir_eval/pull/258
lostanlen
CC BY-SA 4.0
2022-08-17T19:46:43.313Z
thanks I will look at it
Marcelo Araya-Salas
CC BY-SA 4.0
2022-07-11T17:12:23.230Z
2022-08-26T18:29:00.647Z
2022-08-26T18:29:00.647Z
Marcelo Araya-Salas
<p>I am also looking for tools doing that and seems like people usually do not share them.</p>
<p>Some time ago I tested sed_eval Python tool from Tuomas Virtanen and his Audio Research Group. You can find it here: <a href="https://webpages.tuni.fi/arg/index.php/code" rel="nofollow noreferrer">https://webpages.tuni.fi/arg/index.php/code</a> .
I used it on my Raven selection tables but needed to tweak the selection tables before.</p>

CC BY-SA 4.0
2022-08-09T16:03:59.143Z
Hi Pavel and welcome, thanks for your response. If your last paragraph refers to the other response, please remove it and add it as a comment of this response when you have enough points to do so (50).
Noil
CC BY-SA 4.0
2022-08-10T11:46:15.090Z
@Noil Thanks, I did so.
Pavel Linhart
CC BY-SA 4.0
2022-08-09T11:25:15.273Z
2022-08-10T11:45:38.317Z
2022-08-10T11:45:38.317Z
Pavel Linhart
<p>Is there any software option for diagnosing the performance of an automatic acoustic detection routine? I am thinking of something that can take the output of an automatic detection routine (e.g., a table with the time position of the detected signals) and compare it to a reference table (e.g., a table with the time position of the target signals) and return basic signal detection indices like true positive rate, recall, precision and so on.</p>

CC BY-SA 4.0
2022-07-08T20:01:36.520Z
@CathleenBalantic that would be a great additional question. I've asked myself that!
selene
CC BY-SA 4.0
2022-07-08T21:43:22.350Z
Thanks @selene, I added this question here: https://bioacoustics.stackexchange.com/questions/510/what-criteria-should-be-used-to-decide-whether-an-automated-detection-overlaps-w
Cathleen B
CC BY-SA 4.0
2022-07-05T20:00:12.507Z
I don't know of any stand-alone software, but I think there are some MATLAB-based tools that do this for Raven output tables. I am hoping someone more familiar with Raven can chime in,
selene
CC BY-SA 4.0
2022-07-05T20:13:15.273Z
Something I struggle with is, "what constitutes an overlap between an automated detection and a reference annotation"? I've written my own code for this, and depending on the motivating question, I've gone back and forth between coding it as "a true positive means the centroid of a detection needs to be contained within an annotation in the reference table", vs. "ANY overlap/portion of the detection occurring within an annotation in the reference table counts as a true positive". If there are software options for diagnosing performance, it will be interesting to dig into how they handle this.
Cathleen B
<automated-detection><diagnose-performance><signal-detection>
Software options for diagnosing the performance of a detection routine
# Software options for diagnosing the performance of a detection routine

(Asked by: username_0 on 2022-07-05)

Is there any software option for diagnosing the performance of an automatic acoustic detection routine? I am thinking of something that can take the output of an automatic detection routine (e.g., a table with the time position of the detected signals) and compare it to a reference table (e.g., a table with the time position of the target signals) and return basic signal detection indices like true positive rate, recall, precision and so on.

**Tags:** automated-detection, diagnose-performance, signal-detection

> **Comments:**
> **username_1**: I don't know of any stand-alone software, but I think there are some MATLAB-based tools that do this for Raven output tables. I am hoping someone more familiar with Raven can chime in,  
> **username_2**: Something I struggle with is, "what constitutes an overlap between an automated detection and a reference annotation"? I've written my own code for this, and depending on the motivating question, I've gone back and forth between coding it as "a true positive means the centroid of a detection needs to be contained within an annotation in the reference table", vs. "ANY overlap/portion of the detection occurring within an annotation in the reference table counts as a true positive". If there are software options for diagnosing performance, it will be interesting to dig into how they handle this.  
> **username_1**: @CathleenBalantic that would be a great additional question. I've asked myself that!  
> **username_2**: Thanks @username_1, I added this question here: https://bioacoustics.stackexchange.com/questions/510/what-criteria-should-be-used-to-decide-whether-an-automated-detection-overlaps-w  

---

## 2 answers

### Answer by username_0 on 2022-07-11. Score: 8

The new R package ohun allows to do exactly that (diagnose the performance of detection routines).

<https://marce10.github.io/ohun/index.html>

It takes a reference table and the detection output and returns a bunch of performance metrics, including number of true positives, false positives, false negatives, precision and recall. It also provides some time related metrics like the amount of overlap to true positives.

I am still working on it so would be happy to hear suggestions for new features.

Edit: to echo @username_4, the [python package sed\_eval](http://tut-arg.github.io/sed_eval/) for evaluating (diagnosing) sound event detection. It has lots of cool features to evaluate segment-based and event-based detection

> **Comments:**
> **username_3**: Awesome you are working on it! would be nice to have a summary of species / events etc. per file or in total as a higher level of comparison. Would be nice to have long-term visualization of annotations for overview and quick visula comparison.  
> **username_0**: Could you elaborate a bit on your suggestions about visualizing annotations? You mean labeled spectrograms?  
> **username_3**: Visualizing ground truth events next to detections so that you have quick graphical overview where ground truth and detection match or disagree along with the numeric performance metrics. I posted a question related to this problem: https://bioacoustics.stackexchange.com/questions/1043/tools-to-visualize-and-compare-species-detections-in-long-term-recordings  
> **username_4**: Hello @username_0, i'm afraid there is a problem in your algorithm for counting true positives. https://github.com/maRce10/ohun/blob/master/R/diagnose_detection.R#L170-L183  As you noticed, you may count true positives multiple times and obtain recall values above 1. You have a heuristic to prune them but it leads to a suboptimal TP. The optimal way to address this is via bipartite graph matching (Hopcroft-Karp)  See mir_eval package: https://craffel.github.io/mir_eval/ implementation by Brian McFee https://github.com/craffel/mir_eval/pull/258  
> **username_0**: thanks I will look at it  

---

### Answer by username_3 on 2022-08-09. Score: 2

I am also looking for tools doing that and seems like people usually do not share them.

Some time ago I tested sed\_eval Python tool from Tuomas Virtanen and his Audio Research Group. You can find it here: <https://webpages.tuni.fi/arg/index.php/code> .
I used it on my Raven selection tables but needed to tweak the selection tables before.

> **Comments:**
> **username_5**: Hi username_3 and welcome, thanks for your response. If your last paragraph refers to the other response, please remove it and add it as a comment of this response when you have enough points to do so (50).  
> **username_3**: @username_5 Thanks, I did so.  

---

