<p>FFmpeg can also generate spectrograms from audio as video:
<a href="https://ffmpeg.org/ffmpeg-filters.html#showspectrum" rel="nofollow noreferrer">https://ffmpeg.org/ffmpeg-filters.html#showspectrum</a>
<a href="https://ffmpeg.org/ffmpeg-filters.html#showcqt" rel="nofollow noreferrer">https://ffmpeg.org/ffmpeg-filters.html#showcqt</a>
This should be precise to the frame level, as long as your video and audio tracks are aligned. But it is a similar solution as you showed with dynaSpec in R.</p>

CC BY-SA 4.0
2022-07-29T20:26:41.440Z
2022-07-29T20:26:41.440Z
Jörg Rychen
<p>Many species communicate using multimodal signals, and I study a bird species that emits a specific vocalization type associated with a typical body posture. Is there any software or package in R that allows to annotate for example the start and end of an acoustic event and the visual display or posture of a animal from a video recording?</p>
<p>I have been using BORIS software (Behavioral Observation Research Interactive Software <a href="http://www.boris.unito.it/" rel="noreferrer">http://www.boris.unito.it/</a>) which is very good for creating ethogram, and annotate visual elements from a video recording. It is also possible to show the spectrogram of the audio from the video. But the resolution is not that good, and there is some latency between the sound of the video and what is displayed on spectrogram. I had to annotate acoustic events separately from the video using PRAAT (I could use Raven as well) and then join data from the video anotation and from the audio annotation.</p>
<p>Another approach that could fix the issue with the poor quality spectrogram in BORIS software, could be to first create a dynamic spectrogram using <code>dynaSpec package</code> in <code>R</code>, then combine both the original video with the dynamic spectrogram to get the best quality of both the video and the spectrogram perfectly synchronized with the audio (here a example of a video I made of some birds singing using <code>dynaSpec package</code>: <a href="https://www.youtube.com/watch?v=Q-tuPoQR4TU" rel="noreferrer">https://www.youtube.com/watch?v=Q-tuPoQR4TU</a>).</p>
<p>Is there other tools that allow to interactively annotate both acoustic and visual parts of a multimodal display?</p>

CC BY-SA 4.0
2022-07-27T10:44:38.073Z
This is a really good question, and I'd love to hear people's answers. One comment: you might find a good tool that is not based on R - it seems to me likely that good custom software may be needed for a/v sync - so my hunch is that looking outside of R may be advisable.
Dan Stowell
<spectrogram-annotation>
Annotate acoustic and visual events of a multimodal signal
# Annotate acoustic and visual events of a multimodal signal

(Asked by: username_0 on 2022-07-25)

Many species communicate using multimodal signals, and I study a bird species that emits a specific vocalization type associated with a typical body posture. Is there any software or package in R that allows to annotate for example the start and end of an acoustic event and the visual display or posture of a animal from a video recording?

I have been using BORIS software (Behavioral Observation Research Interactive Software <http://www.boris.unito.it/>) which is very good for creating ethogram, and annotate visual elements from a video recording. It is also possible to show the spectrogram of the audio from the video. But the resolution is not that good, and there is some latency between the sound of the video and what is displayed on spectrogram. I had to annotate acoustic events separately from the video using PRAAT (I could use Raven as well) and then join data from the video anotation and from the audio annotation.

Another approach that could fix the issue with the poor quality spectrogram in BORIS software, could be to first create a dynamic spectrogram using `dynaSpec package` in `R`, then combine both the original video with the dynamic spectrogram to get the best quality of both the video and the spectrogram perfectly synchronized with the audio (here a example of a video I made of some birds singing using `dynaSpec package`: <https://www.youtube.com/watch?v=Q-tuPoQR4TU>).

Is there other tools that allow to interactively annotate both acoustic and visual parts of a multimodal display?

**Tags:** spectrogram-annotation

> **Comments:**
> **username_1**: This is a really good question, and I'd love to hear people's answers. One comment: you might find a good tool that is not based on R - it seems to me likely that good custom software may be needed for a/v sync - so my hunch is that looking outside of R may be advisable.  

---

## 1 answers

### Answer by username_2 on 2022-07-29. Score: 2

FFmpeg can also generate spectrograms from audio as video:
<https://ffmpeg.org/ffmpeg-filters.html#showspectrum>
<https://ffmpeg.org/ffmpeg-filters.html#showcqt>
This should be precise to the frame level, as long as your video and audio tracks are aligned. But it is a similar solution as you showed with dynaSpec in R.


---

