Creating Sound Effects Via Algorithm

I have great respect for sound designers in that they not only have to create effects that not only sound totally real but, in many cases, better than real. That last part is the key that will continue to keep them working despite a new algorithm from some MIT researchers that can independently add realistic sound effects to silent videos.

The researchers from MITCSAIL used artificial intelligence to enable a computer to learn the movements and surfaces occurring in a video and insert the appropriate sound effects. As you can see from the video below, the results are impressive.

The computer associates what it sees in the video with the appropriate sound from a database, then inserts it as needed. That should send a chill down a few sound designer’s spines.

In order to prove that the method was effective, the researchers did an A/B test on a on a number of test subjects. They showed one video that had the sound effects inserted using normal foley techniques and the other using the algorithm. In most cases, the test subjects failed to notice a difference between the two, and in some cases even preferred the one generated by the algorithm.

If you’re a sound designer reading this and fearing for your job, you needn’t be worried – yet. Although the algorithm shows promise, it has a long way to go. It’s only useful for very short clips as it tends to misfire on longer ones and play sounds at the wrong times, and it’s dependent on the sound library that it has available.

These problems will no doubt be worked out as development continues, but remember, only a sound designer good judgement can make something sound “better than live,” and judgement isn’t exactly the strong suit of computers, at least not today.