<p>I would cross-correlate detected signal with annotated signal (reference) then you can transform you metrics into a question, how good are the two signals correlated, for which the answer could be easier discussed as it follows standard statistics.</p>

CC BY-SA 4.0
2022-07-10T00:49:27.523Z
That would only apply for stereotyped signals. Not all detection methods are trying to find sounds that closely match a template though.
Marcelo Araya-Salas
CC BY-SA 4.0
2022-07-10T04:22:46.780Z
yes, but OP asked for similarity (overlap) with reference signal.
WMXZ
CC BY-SA 4.0
2022-07-10T23:30:41.763Z
Cathleen mentions that "a true positive means the centroid of a detection needs to be contained within an annotation in the reference table". So it is overlap in time. not similarity.
Marcelo Araya-Salas
CC BY-SA 4.0
2022-07-11T14:28:32.727Z
Marcelo is correct, I meant an overlap in time. An automated detection might overlap one partial note out of a three-note song by a target species. If I'm trying to diagnose performance of the detector, do I count that detection or not? The answer is usually "it depends on the question" so I'm curious about the heuristics others use. (Apologies for any imprecise terminology; I'm coming at this from a biology background.)
Cathleen B
CC BY-SA 4.0
2022-07-11T15:01:23.530Z
I'm confused but try to understand.  Assume you have a three-note song and your detector detected only one note? is that the question?  Was the detector trained for this three-note song (is the three-note song in the reference table)?
WMXZ
CC BY-SA 4.0
2022-07-11T16:18:40.040Z
Some detector tells you not only if something is detected but also the start and end times of the detection. That start and end might include only a portion of the signal of interest (compared to the annotated start and end times), so a partial overlap, or might be a perfect match. Overlap can be seen as a continuous parameter, and it seems that Cathleen is asking about cutoff values for deciding whether a detection is good enough to be counted as a true positive.
Marcelo Araya-Salas
CC BY-SA 4.0
2022-07-11T22:16:07.370Z
@MarceloAraya-Salas that is correct. I'm reading through ohun's documentation and it seem like you have thought deeply about this and had to make some relevant decisions in your performance evaluation functions. I'm interested to know how your software handles it. Let me know if there's an edit I can make to my question to make it more clear (or feel free to make that edit).
Cathleen B
CC BY-SA 4.0
2022-07-09T04:52:15.703Z
2022-07-09T04:52:15.703Z
WMXZ
<p>In the R package <a href="https://marce10.github.io/ohun/" rel="noreferrer">ohun</a> any overlapping detection is taken as a true positive, as it would be arbitrary to pick up an overlap threshold to be applied by default. However, the performance output (function <a href="https://marce10.github.io/ohun/reference/diagnose_detection.html" rel="noreferrer">diagnose_detection</a>) also includes a summary about the overall overlap of the true positives (&quot;overlap.to.true.positives&quot;, a number between 0 and 1). If users want to exclude some detected signals with low overlaps they can get the amount of overlap for each detection (in a detection table) with the function <a href="https://marce10.github.io/ohun/reference/label_detection.html" rel="noreferrer">label_detection</a>. That would add an extra column that you can use to remove those with little overlap.</p>
<p>Here is an example on how to do that:</p>
<pre><code>#remotes::install_github(&quot;maRce10/ohun&quot;)
library(ohun)

# load example data
data(&quot;lbh_reference&quot;)

# create a simulated detection ouput ##
# copy reference
sim_detection &lt;- lbh_reference

# add some time to the first 3 rows so they barely overlap with the original data
sim_detection$start[1:3] &lt;- sim_detection$start[1:3] + 0.1
sim_detection$end[1:3] &lt;- sim_detection$end[1:3] + 0.1

# check peformance (look at the overlap.to.true positives) ##
# for a perfect detection
diagnose_detection(reference = lbh_reference, detection = lbh_reference)

# for the simulated detection
diagnose_detection(reference = lbh_reference, detection = sim_detection)

# remove low overlap ones #
# get amount of overlap
lab_detection &lt;- label_detection(reference = lbh_reference, detection = sim_detection)

print(lab_detection$overlap)

# exclude those below 0.5
filtered_detection &lt;- lab_detection[lab_detection$overlap &gt; 0.5, ]

# check performance again (of course recall will go down!)
diagnose_detection(reference = lbh_reference, detection = filtered_detection)
</code></pre>
<p>links:
ohun: <a href="https://marce10.github.io/ohun/" rel="noreferrer">https://marce10.github.io/ohun/</a></p>
<p>label_detection: <a href="https://marce10.github.io/ohun/reference/label_detection.html" rel="noreferrer">https://marce10.github.io/ohun/reference/label_detection.html</a></p>
<p>diagnose_detection: <a href="https://marce10.github.io/ohun/reference/diagnose_detection.html" rel="noreferrer">https://marce10.github.io/ohun/reference/diagnose_detection.html</a></p>

CC BY-SA 4.0
2022-07-12T21:37:59.583Z
2022-07-13T12:49:08.057Z
2022-07-13T12:49:08.057Z
Marcelo Araya-Salas
<p>For those who use automated detection and routinely need to evaluate detector performance, what constitutes an overlap between an automated detection and a reference annotation?</p>
<p>Most of my bioacoustics work is with terrestrial species (birds). I've written plenty of code for comparing whether automated detections are lining up with annotations in a &quot;truth&quot; data set. Depending on the motivating question, I've gone back and forth between coding it as, &quot;a true positive means the centroid of a detection needs to be contained within an annotation in the reference table&quot;, vs. &quot;ANY overlap/portion of the detection occurring within an annotation in the reference table counts as a true positive&quot;. Depending on motivating question and vocalization type of interest, it seems this decision point is another place where error might be introduced. There are likely creative, robust ways to handle this that I'm overlooking. For those routinely tasked with comparing an automated detector's performance to a truth/reference annotation data set, how do you typically handle this?</p>
<p>Inspired by this relevant question: <a href="https://bioacoustics.stackexchange.com/questions/295/software-options-for-diagnosing-the-performance-of-a-detection-routine">Software options for diagnosing the performance of a detection routine</a></p>

<automated-detection><classifier><spectrogram-annotation><machine-learning><diagnose-performance>
What criteria should be used to decide whether an automated detection overlaps with a reference annotation?
# What criteria should be used to decide whether an automated detection overlaps with a reference annotation?

(Asked by: username_0 on 2022-07-08)

For those who use automated detection and routinely need to evaluate detector performance, what constitutes an overlap between an automated detection and a reference annotation?

Most of my bioacoustics work is with terrestrial species (birds). I've written plenty of code for comparing whether automated detections are lining up with annotations in a "truth" data set. Depending on the motivating question, I've gone back and forth between coding it as, "a true positive means the centroid of a detection needs to be contained within an annotation in the reference table", vs. "ANY overlap/portion of the detection occurring within an annotation in the reference table counts as a true positive". Depending on motivating question and vocalization type of interest, it seems this decision point is another place where error might be introduced. There are likely creative, robust ways to handle this that I'm overlooking. For those routinely tasked with comparing an automated detector's performance to a truth/reference annotation data set, how do you typically handle this?

Inspired by this relevant question: [Software options for diagnosing the performance of a detection routine](https://bioacoustics.stackexchange.com/questions/295/software-options-for-diagnosing-the-performance-of-a-detection-routine)

**Tags:** automated-detection, classifier, spectrogram-annotation, machine-learning, diagnose-performance


---

## 2 answers

### Answer by username_2 on 2022-07-12. Score: 8 | Accepted answer

In the R package [ohun](https://marce10.github.io/ohun/) any overlapping detection is taken as a true positive, as it would be arbitrary to pick up an overlap threshold to be applied by default. However, the performance output (function [diagnose\_detection](https://marce10.github.io/ohun/reference/diagnose_detection.html)) also includes a summary about the overall overlap of the true positives ("overlap.to.true.positives", a number between 0 and 1). If users want to exclude some detected signals with low overlaps they can get the amount of overlap for each detection (in a detection table) with the function [label\_detection](https://marce10.github.io/ohun/reference/label_detection.html). That would add an extra column that you can use to remove those with little overlap.

Here is an example on how to do that:

```
#remotes::install_github("maRce10/ohun")
library(ohun)

# load example data
data("lbh_reference")

# create a simulated detection ouput ##
# copy reference
sim_detection <- lbh_reference

# add some time to the first 3 rows so they barely overlap with the original data
sim_detection$start[1:3] <- sim_detection$start[1:3] + 0.1
sim_detection$end[1:3] <- sim_detection$end[1:3] + 0.1

# check peformance (look at the overlap.to.true positives) ##
# for a perfect detection
diagnose_detection(reference = lbh_reference, detection = lbh_reference)

# for the simulated detection
diagnose_detection(reference = lbh_reference, detection = sim_detection)

# remove low overlap ones #
# get amount of overlap
lab_detection <- label_detection(reference = lbh_reference, detection = sim_detection)

print(lab_detection$overlap)

# exclude those below 0.5
filtered_detection <- lab_detection[lab_detection$overlap > 0.5, ]

# check performance again (of course recall will go down!)
diagnose_detection(reference = lbh_reference, detection = filtered_detection)

```

links:
ohun: <https://marce10.github.io/ohun/>

label\_detection: <https://marce10.github.io/ohun/reference/label_detection.html>

diagnose\_detection: <https://marce10.github.io/ohun/reference/diagnose_detection.html>


---

### Answer by username_1 on 2022-07-09. Score: 1

I would cross-correlate detected signal with annotated signal (reference) then you can transform you metrics into a question, how good are the two signals correlated, for which the answer could be easier discussed as it follows standard statistics.

> **Comments:**
> **username_2**: That would only apply for stereotyped signals. Not all detection methods are trying to find sounds that closely match a template though.  
> **username_1**: yes, but OP asked for similarity (overlap) with reference signal.  
> **username_2**: username_0 mentions that "a true positive means the centroid of a detection needs to be contained within an annotation in the reference table". So it is overlap in time. not similarity.  
> **username_0**: username_2 is correct, I meant an overlap in time. An automated detection might overlap one partial note out of a three-note song by a target species. If I'm trying to diagnose performance of the detector, do I count that detection or not? The answer is usually "it depends on the question" so I'm curious about the heuristics others use. (Apologies for any imprecise terminology; I'm coming at this from a biology background.)  
> **username_1**: I'm confused but try to understand.  Assume you have a three-note song and your detector detected only one note? is that the question?  Was the detector trained for this three-note song (is the three-note song in the reference table)?  
> **username_2**: Some detector tells you not only if something is detected but also the start and end times of the detection. That start and end might include only a portion of the signal of interest (compared to the annotated start and end times), so a partial overlap, or might be a perfect match. Overlap can be seen as a continuous parameter, and it seems that username_0 is asking about cutoff values for deciding whether a detection is good enough to be counted as a true positive.  
> **username_0**: @username_2 that is correct. I'm reading through ohun's documentation and it seem like you have thought deeply about this and had to make some relevant decisions in your performance evaluation functions. I'm interested to know how your software handles it. Let me know if there's an edit I can make to my question to make it more clear (or feel free to make that edit).  

---

