We perform the spectrogram re-synthesis step with the use of a phase vocoder or a short-timescale Fourier transform framework (where the columns of the image/spectrogram are treated as individual spectra). The processed spectrogram can then be transferred back to the time-domain for sound reproduction (e.g., in loudspeakers or headphones). In the time-frequency processing step, we use optical pattern detection and recognition techniques to analyze and process the spectrogram. In addition, we encode the signal power of each time-frequency point with a grayscale or hue value. This spectrogram can be interpreted as an image in which every pixel corresponds to a quantized time slot and frequency band (time is depicted along the x-axis and frequency is shown along the y-axis). In our audio processing chain, we include a sound-analysis step in which Fourier transforms of consecutive overlapping frames of input samples are performed to produce a spectrogram. With the newer generations of digital signal processors, however, we have been able to develop new algorithms 9 and to implement them in real time on mobile devices. Until recently, sound processing has generally been performed with the use of time-domain algorithms. In addition, we introduce our novel sound-synthesis technique that is based in the time-frequency domain. We also outline various basic optical sound-processing systems and their performance parameters. In our work, 7,8 we provide an overview of optical techniques that can be used for sound processing, and of current trends in this research field.
The sound waveform is then recovered via frame-by-frame processing of the recorded vibrations. In the visual microphone system, a high-speed camera is used to record the sound-induced vibrations of an object. In the optical microphone, the vibration of the reflective membrane is used as a measure of the variation in light intensity at the photodetector. Schematic diagrams of (a) an optical microphone and (b) a visual microphone. This type of analysis of temporal variations in videos 6 can thus serve as the basis for new research directions.įigure 1. For this kind of application, the video frame rate should be similar to that of typical audio sampling rates (i.e., from 8–44.1kHz). By subsequently conducting frame-by-frame processing, the vibration of the object can be analyzed to recover the sound waveform. In this setup, sound events in a room (e.g., those coming from speakers) cause an object to vibrate slightly and a high-speed camera is used to record the object. Another recent development is the visual microphone 5-see Figure 1(b)-which is a camera-based sound-capturing system. For example, optical audio reconstruction 2 and laser microphones 3, 4 have been demonstrated. Optical microphones have thus gained a great deal of interest and have found a number of applications (ranging from pattern recognition to the generation of sound effects). In contrast, with optical microphones 1-see Figure 1(a)-the vibration of the reflective membrane is directly measured to assess the light intensity variation at the photodetector. In the past, the recording of sound with microphones has mainly been achieved with the use of a condenser and moving-coil electronic components that transform a mechanical sound wave into an electric signal. In addition, the capture of sound with optical and visual microphone techniques offers a range of new possibilities for specific sound-sensing scenarios. Recently, however, storage and distribution of sound and video have been completely replaced by specific audio and video coding standards. To date, such developments have led to sound being optically recorded on film, compact disc systems, and digital video discs. With the invention of photoelectrical cells and laser-based optics, however, more sophisticated optical techniques for sound recording, storing, and processing have also been, and continue to be, developed. This can be considered the first kind of optical recording and storing technique for sound. In other words, optical characters are used to capture sound and to deliver a symbolic recording of a melody. For example, the notation of sound events (i.e., through the musical notation system) offers a graphical representation of sound in terms of quantized frequency along a quantized time axis.
From an engineering perspective, there are several aspects to the relationship between optics and sound.
0 kommentar(er)
