Investigation of the Efficiency of the Phase-Based Method Based on Gabor Filter Banks for Audio Recovery from Video Recordings

Abstract

This paper addresses the problem of passive acoustic signal recovery through the analysis of video sequences of vibrating objects. Sound waves propagating through a medium exert varying pressure on physical objects, inducing microscopic vibrations of their surfaces. The amplitude of such vibrations typically amounts to fractions of a pixel (10−3 -- 10−2 pixels), posing a challenge for detection via classical computer vision methods based on optical flow.
This study extends the "visual microphone" approach proposed in [1]. Based on computer vision principles [3], the Fourier transform shift property serves as the theoretical foundation, mathematically linking spatial object displacement to a linear phase shift. With the general mathematical model being preserved, the frequency- spatial decomposition stage is modified. Specifically, this research evaluates the efficiency of a custom implementation of a complex Gabor filter bank (in contrast to the Steerable Pyramid used in the original work).
An optimized C++ software implementation of the algorithm is presented. The proposed solution employs convolution via the Fast Fourier Transform (FFT) and batch parallel frame processing (OpenMP), yielding signicantly higher computational efficiency compared to naive spatial convolution.
A comparative analysis regarding robustness to sensor noise [18] is conducted, demonstrating the superiority of the phase-based approach over intensity-based methods using experimental data. Results obtained with a high-speed camera (2200 fps) confirm the feasibility of recovering acoustic information within the bandwidth limited by the Nyquist frequency [15, 16] and capturing tonal signals from lightweight objects at a distance of several meters. Furthermore, the impact of incoherent sensor noise is effectively mitigated through spatial weighting and filtering.