Investigation of the Efficiency of the Phase-Based Method Based on Gabor Filter Banks for Audio Recovery from Video Recordings
DOI:
https://doi.org/10.32603/2071-2340-2025-3-2Keywords:
visual microphone, Gabor filters, phase-based analysis, audio recovery from video, sub-pixel motion, multi-scale analysisAbstract
This paper addresses the problem of passive acoustic signal recovery through the analysis of video sequences of vibrating objects. Sound waves propagating through a medium exert varying pressure on physical objects, inducing microscopic vibrations of their surfaces. The amplitude of such vibrations typically amounts to fractions of a pixel (10−3 -- 10−2 pixels), posing a challenge for detection via classical computer vision methods based on optical flow.
This study extends the "visual microphone" approach proposed in [1]. Based on computer vision principles [3], the Fourier transform shift property serves as the theoretical foundation, mathematically linking spatial object displacement to a linear phase shift. With the general mathematical model being preserved, the frequency- spatial decomposition stage is modified. Specifically, this research evaluates the efficiency of a custom implementation of a complex Gabor filter bank (in contrast to the Steerable Pyramid used in the original work).
An optimized C++ software implementation of the algorithm is presented. The proposed solution employs convolution via the Fast Fourier Transform (FFT) and batch parallel frame processing (OpenMP), yielding signicantly higher computational efficiency compared to naive spatial convolution.
A comparative analysis regarding robustness to sensor noise [18] is conducted, demonstrating the superiority of the phase-based approach over intensity-based methods using experimental data. Results obtained with a high-speed camera (2200 fps) confirm the feasibility of recovering acoustic information within the bandwidth limited by the Nyquist frequency [15, 16] and capturing tonal signals from lightweight objects at a distance of several meters. Furthermore, the impact of incoherent sensor noise is effectively mitigated through spatial weighting and filtering.
References
1. A. Davis, M. Rubinstein, N. Wadhwa, G. J. Mysore, F. Durand, and W. T. Freeman, “The visual microphone: passive recovery of sound from video,” ACM Trans. Graph. (Proc. SIGGRAPH), vol. 33, no. 4, pp. 1–10, art. no. 79, 2014. DOI: https://doi.org/10.1145/2601097.2601119
2. B. D. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” in Proc. DARPA Image Understanding Workshop, 1981, pp. 121–130.
3. J. R. Janesick, Scientific Charge-Coupled Devices, Bellingham, WA, USA: SPIE Press, 2001, doi: 10.1117/3.374903. DOI: https://doi.org/10.1117/3.374903
4. N. Wadhwa, M. Rubinstein, F. Durand, and W. T. Freeman, “Phase-based video motion processing,” ACM Transactions on Graphics (TOG). (Proc. SIGGRAPH), vol. 32, no. 4, pp. 80:1–80:10, 2013, (art. no. 80), doi: 10.1145/2461912.2461966. DOI: https://doi.org/10.1145/2461912.2461966
5. D. J. Fleet and A. D. Jepson, “Computation of component image velocity from local phase information,” International Journal of Computer Vision, vol. 5, no. 1, pp. 77–104, 1990. DOI: https://doi.org/10.1007/BF00056772
6. A. Torralba, P. Isola, and W. T. Freeman, Foundations of Computer Vision, Cambridge, MA: MIT Press, 2024.
7. R. C. Gonzalez and R. E. Woods, Digital Image Processing, 3rd Eds., Moscow: Tekhnosfera, 2019 (in Russian).
8. V. A. Soifer, Metody komp’yuternoj obrabotki izobrazhenij[Methods of Computer Image Processing], Moscow: Fizmatlit, 2003 (in Russian).
9. V. V. Starovoitov and A. A. Golub, Tsifrovaya obrabotka izobrazhenij i komp’yuternaya grafika [Digital Image Processing and Computer Graphics], Minsk: OIPI NAN Belarusi, 2020 (in Russian).
10. J. Portilla and E. P. Simoncelli, “A parametric texture model based on joint statistics of complex wavelet coefficients,” International Journal of Computer Vision, 2000. DOI: https://doi.org/10.1023/A:1026553619983
11. Software implementation based on the article in C++, 2025. [Online]. Available: https://gitverse.ru/rremca/Gabor-Visual-Microphone
12. W. T. Freeman and E. H. Adelson, “The design and use of steerable filters,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 13, no. 9, pp. 891–906, 1991. DOI: https://doi.org/10.1109/34.93808
13. J. G. Daugman, “Uncertainty relation for resolution in space, spatial frequency, and orientation optimized by two-dimensional visual cortical filters,” J. Opt. Soc. Am. A, vol. 2, no. 7, pp. 1160–1169, 1985,doi: 10.1364/JOSAA.2.001160. DOI: https://doi.org/10.1364/JOSAA.2.001160
14. P. J. Burt and E. H. Adelson, “The laplacian pyramid as a compact image code,” IEEE Trans. Commun., vol. COM-31, no. 4, pp. 532–540, 1983. DOI: https://doi.org/10.1109/TCOM.1983.1095851
15. E. P. Simoncelli, W. T. Freeman, E. H. Adelson, and D. J. Heeger, “Shiftable multiscale transforms,” IEEE Trans. Inf. Theory, vol. 38, no. 2, pp. 587–607, 1992. DOI: https://doi.org/10.1109/18.119725
16. M. Meingast, C. Geyer, and S. S. Sastry, “Geometric models of rolling-shutter cameras,” arXiv preprint, 2005, (cs/0503076).
17. C. H. Knapp and G. C. Carter, “The generalized correlation method for estimation of time delay,” IEEE Trans. Acoust., Speech, Signal Process., vol. 24, no. 4, pp. 320–327, 1976, doi: 10.1109/TASSP.1976.1162830. DOI: https://doi.org/10.1109/TASSP.1976.1162830
18. A. B. Sergienko, Tsifrovaya obrabotka signalov [Digital Signal Processing: A Textbook for Universities], 3rd Eds., St. Petersburg, Russia: BHV-Peterburg, 2011 (in Russian).
Published
Issue
Section
License
Copyright (c) 2025 Alexander Spiridonov, Maxim Beskonchin
This work is licensed under a Creative Commons Attribution 4.0 International License.
