{"id":84,"date":"2006-03-23T13:02:00","date_gmt":"2006-03-23T21:02:00","guid":{"rendered":"http:\/\/www.mccambridge.org\/blog\/2006\/03\/real-time-music-video-synthesis-by-reconstruction\/"},"modified":"2022-09-11T00:40:47","modified_gmt":"2022-09-11T00:40:47","slug":"real-time-music-video-synthesis-by-reconstruction","status":"publish","type":"post","link":"http:\/\/www.mccambridge.org\/blog\/2006\/03\/real-time-music-video-synthesis-by-reconstruction\/","title":{"rendered":"Real Time Music Video Synthesis by Reconstruction"},"content":{"rendered":"

The world truly is full of wonder.<\/p>\n

You have<\/em> to check this out. A guy named Sven König, who I assume is from Germany, wrote some amazing software for creating music videos in real time<\/a>. The idea is, you feed the system existing material (anything from existing music videos to someone giving a filmed interview). The system analyzes the input media, breaking it apart into sound\/video snippets that are, say, as long as a quarter note in the input song, or an eighth note, a sixteenth, etc. It then creates a sound signature of each sample (I assume performing some sort of time\/frequency analysis), and stores all the samples along with their signatures in a database. Then, when you run the software, you sing\/talk\/beatbox\/make random noise into a microphone. The software now takes apart your voice coming in, and generates signatures in the same manner. It mathes those signatures to the nearest existing sound samples in its database, and plays back the sound\/video clip corresponding to the closest match, instead of just playing back your own voice.<\/p>\n

What ends up happening is that if you were to say “Hello” into the mic, the computer would break that apart into maybe “Heh” and “Low”, and search its databse for some moment in time in one of the source media you fed it at the beginning, where it sounds like “heh” or “low”, and play back those two clips in order, thus reconstructing output sound that sounds like “Hello,” but isn’t being said by you.<\/p>\n

I’m not sure if that made any sense, but check out this video of Sven explaining his “sCrAmBlEd?HaCkZ!” software<\/a>, and pay attention to the examples, and I think you’ll understand better. It’s a really awesome idea. I don’t know if just nobody’s thought about it before, or if nobody has done it completely, or if I just wasn’t aware of it, but as a sound guy, I think it’s one of the coolest things I’ve ever<\/em> seen a computer do.<\/p>\n","protected":false},"excerpt":{"rendered":"

The world truly is full of wonder. You have to check this out. A guy named Sven König, who I assume is from Germany, wrote some amazing software for creating music videos in real time. The idea is, you feed the system existing material (anything from existing music videos to someone giving a filmed interview). […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"_links":{"self":[{"href":"http:\/\/www.mccambridge.org\/blog\/wp-json\/wp\/v2\/posts\/84"}],"collection":[{"href":"http:\/\/www.mccambridge.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.mccambridge.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.mccambridge.org\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.mccambridge.org\/blog\/wp-json\/wp\/v2\/comments?post=84"}],"version-history":[{"count":1,"href":"http:\/\/www.mccambridge.org\/blog\/wp-json\/wp\/v2\/posts\/84\/revisions"}],"predecessor-version":[{"id":1811,"href":"http:\/\/www.mccambridge.org\/blog\/wp-json\/wp\/v2\/posts\/84\/revisions\/1811"}],"wp:attachment":[{"href":"http:\/\/www.mccambridge.org\/blog\/wp-json\/wp\/v2\/media?parent=84"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.mccambridge.org\/blog\/wp-json\/wp\/v2\/categories?post=84"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.mccambridge.org\/blog\/wp-json\/wp\/v2\/tags?post=84"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}