Interesting Video Technology

Not really sure what to say about this video but it’s pretty cool. The video discusses specialized methods & technology for enhancing altering video.

Check it out

August 17th, 2008 @ 3:29 pm
Right On...Really cool
August 17th, 2008 @ 3:32 pm
very interesting stuff.
August 17th, 2008 @ 3:32 pm
great utility
August 17th, 2008 @ 3:35 pm
Wow... That's interesting. Not sure what to say about it either, but I look forward to utilizing those algorithm based depth maps and such to enhance my work.
August 17th, 2008 @ 3:35 pm
August 17th, 2008 @ 3:41 pm
thats so amazing. My favorite would have to be the library shot and the camera stabalization.
Some Guy
August 17th, 2008 @ 3:43 pm
Wow! This is the future. I love all these technical stuff like this or automatic 3D scene creation from one photograph etc.
Especially I loved the motion stabilization. It was even better than a dolly shot. Almost artificial, like a 3D model. The downside of this technology is that I have to make photos of everything I shoot, which almost doubles the filming time...
Kim A. Strandli
August 17th, 2008 @ 3:46 pm
It is really impressive what people can do :) I hope this technology comes to the marked in a relatively low price soon in the future! Maybe Adobe should give them a big bag of money, and release it with the Creative Suite package? ;P
August 17th, 2008 @ 3:49 pm
looks fun :)
August 17th, 2008 @ 3:54 pm
This stuff is interesting. Awesome
August 17th, 2008 @ 3:59 pm
this is amazing, i think this technology will be used a lot in the future
August 17th, 2008 @ 4:04 pm
Yeah, I saw this a while back on Motionographer. I watched it 3 times I was so amazed. My favorite way to utilize this technology would be some cool stuff with the painterly effects. Like a live action version of Van Gogh's "Starry Starry Night."

Too bad you can't have anything moving in the shot like a car or person. But still really amazing stuff.
August 17th, 2008 @ 4:11 pm
mmmm.. a new science?!
August 17th, 2008 @ 4:21 pm
So how do you incorporate this into AE?
August 17th, 2008 @ 4:47 pm
One of the person (Aseem Agarwala) behind this technology is now working in Adobe so I hope we might see this thing very soon in After Effects or Photoshop.
August 17th, 2008 @ 4:49 pm
sorry, ignore that above me ;)
August 17th, 2008 @ 5:25 pm
So this must have some kind of inherent 3D tracking in order to guess depth.
August 17th, 2008 @ 5:31 pm
Very cool. Interesting how so many solutions for video are converging on 3D technology.
August 17th, 2008 @ 5:47 pm
I don't think it is anything commercially available but rather a study. Although I'm sure if it is viable something may come of it.
Allen Ellis
August 17th, 2008 @ 5:47 pm
"The downside of this technology is that I have to make photos of everything I shoot, which almost doubles the filming time…"

Well I can certainly imagine someone creating a still camera that would attach to the video camera automatically taking pictures. If they could get that to work, we could see this technology in consumer video cameras before long.
August 17th, 2008 @ 5:49 pm
What system/software is used to achieve this?
August 17th, 2008 @ 5:56 pm
Thats incredibly useful, complicated but i see the obvious benefits. eagerly awaiting your 3D TRACKING tutorial andrew. Good Job, Your tutorials are increasingly helpful.
Jim Jimenson
August 17th, 2008 @ 6:08 pm
August 17th, 2008 @ 6:34 pm
lol, so fascinating that i forgot why i opened my browser.. ah i remember now... lol.
August 17th, 2008 @ 6:38 pm
That looks amazing. It produces very good results!

The object removal was flawless ...Thanks for sharing this.
August 17th, 2008 @ 6:40 pm
fantastic.they finally merged the two
August 17th, 2008 @ 6:42 pm
amazing!!!! how they did the video with the parking label?
which program they use??
Jim Jimenson
August 17th, 2008 @ 7:16 pm
Credibly the most incredibly incredible feat of incredible incredibility.
August 17th, 2008 @ 7:18 pm
amazing, thank you for sharing. I hope somthing useful comes from this technology.
August 17th, 2008 @ 7:26 pm
Fascinating stuff. Technology pushing the envelope even further. This will definitely be available in the near future. All of our current applications started out this way. I myself cant wait for something like this to available commercially on some level. This will only offer artists more tools to get the look they after. So long to throwing out poorly shot video. Now it may have life after all.....
August 17th, 2008 @ 7:43 pm
I saw this side about a year ago.

At that time some people on the Web where arguing that the Fx shown are fake...

To me the contents on the side havent change since that...

I mean, Im just wondering why big companys haven´t already bougt or at least are interested in the technology
August 17th, 2008 @ 8:10 pm
One thing of note, is the reference images require stereoscopic photography, not just a standard still camera, but a specialized, not quite consumer friendly yet...but think about the still image sensors on a standard vid cam, always quite high res...imagine a consumer cam that shoots vid, but takes a high res still every 20 frames or so, and stores it separately, this could theoretically be processed and ready in-cam, as you shoot, so the information is ready to process by the time you load it up in your program of choice...anyway, certainly fascinating stuff....on a related note, Microsoft is working on similar themed technology (pretty cool also). Have a look here:
August 17th, 2008 @ 8:44 pm
This is a very interesting technology, hope to see it soon on the market, it will save lot of time (and money) in a project.
Once a again thanks to Andrew for find this kind of info.
Waiting for the new tuts.
August 17th, 2008 @ 8:51 pm
I actually "Stumbled" upon this the other day, it is very awesome, like andy said, i dont think it'll be commercially available for a while, but once the technology starts picking up and others start to emulate, or they start making a marketable software and/or hardware product, yea...... that'll be fun! :p
August 17th, 2008 @ 9:54 pm
I guess you could turn all footage into HD :D
August 17th, 2008 @ 10:28 pm
You can do this with Photoshop CS3 on the cheap. If anything, why are you shooting over exposed footage to begin with, go out and buy some flags light the thing.
August 17th, 2008 @ 10:46 pm
Ahh, Wow! Interesting development. I can see the possibilities with applying effects to the photos first.
August 17th, 2008 @ 10:59 pm
they must develop a software.
August 17th, 2008 @ 11:32 pm
wow, that is awesome, thanks for sharing

don't have any other words, i am ''speechless''
August 18th, 2008 @ 12:28 am
I think it's kind of credit work from FX & animation university. It's cool. But, with VC tutorials, we can do simmilar things :)
August 18th, 2008 @ 1:02 am
by Allen Ellis
"Well I can certainly imagine someone creating a still camera that would attach to the video camera automatically taking pictures. If they could get that to work, we could see this technology in consumer video cameras before long."

I already know a professional photojournalist who does just that. He has a small HD video camera attached to the tripod collar on his lense what ever he is taking photos of he is also taking Hi-Def video. A cool little tip I picked up from other pros.
August 18th, 2008 @ 1:06 am
That looks so awesome. I am also currently studying that. It looks pretty advanced. I think you guys are interested in this too.

I really love this one..its from the same makers that andrew kramer linked us.

August 18th, 2008 @ 1:38 am
somebody here reads engadget, huh andrew :P
Daryl Booth
August 18th, 2008 @ 2:13 am
Holy CARP!

I will never trust a goverment vid clip agaian seeing what they can do with this stuff...
August 18th, 2008 @ 2:15 am

Microsoft have 3 guys on this
Adobe have 1 guy on this.

I don''t know, but i reckon they'll be looking to make this os easy to do that they could make it comercially available. All the guys on the project have an experienced background in bringing complex ideas into a userfriendly format.

I predict this will be ot in the next 5years in a simple but effective form. Could be a collaberation between adobe and microsoft.
Ori Lahav
August 18th, 2008 @ 2:55 am
Its a very creative use of 3D motion tracking (or at least something similar to it).
I wish them good luck with it, pretty interesting!

August 18th, 2008 @ 3:14 am
Would be cool Plug In for my AE =]
August 18th, 2008 @ 4:25 am
maybe a fake? :) the flower shoot looks to unbelievable imho.
August 18th, 2008 @ 4:40 am
*correction :P shot not shoot^^
G Man
August 18th, 2008 @ 5:14 am
wow...i love the depth finder function...that is amazing...
i would hate to see the math that goes into that...
August 18th, 2008 @ 5:28 am
Awesome!!! Thank you for the video.
August 18th, 2008 @ 5:57 am
Well, check out section 7.1.1 of the paper: "The current processing speed of our system is quite slow with five
minutes being spent on each video frame (resolution: 853 x 480);". Now think a 200 frame shot in 4K. That's about 20 minutes per frame if your're very lucky (and processing time scales linearly), which means 4000 minutes, or 66.66 hours, or 2.77 days. We're going to need very lose deadlines, even if they manage to optimize the algorithm for better performance.

I downloaded their sample code and looked into it, and while I have seen much worse, the code is not necessarily very easy (nor very well suited) to be integrated into an AfterEffects or OpenFX plugin as it is right now. There are absolutely no comments in the sources whatsoever, but hundreds of different code files with zillions of utility classes and functions and third party libraries, so it's not very easy to figure out what's really going on and which part does what and is or can be used for what etc. So as much as I regret it, it's just too much work for me to make an AE plugin out of the reference implementation. And that's without counting the licensing nightmare that comes with all the third party libs. We'll have to wait for a company like Adobe or Autodesk to help us out.

But while we're on the subject of research papers: I found the following Siggraph 2008 paper on an adaptation of last year's seam carving algorithm for video processing quite interesting. One of the authors works for Adobe now, and the algorithm itself is much less complex than the one Andrew linked to (I implemented the still image version of the seam carving algorithm last year as a spare time project, see my website link if you want to try it for yourself), so I think there is a fair chance that this is going to be in AfterEffects/Photoshop CS4.

August 18th, 2008 @ 7:08 am
Wow! I really liked the stabalizer and the removing of the sigh!
August 18th, 2008 @ 8:16 am
Very interesting, I bet maaltanon will dig this!
August 18th, 2008 @ 8:21 am
*maltaannon... whoopsy!
August 18th, 2008 @ 8:59 am
Woah! These concepts would be excellent for commerical and indy movie projects.
I love it. AS the person before states the code may not be as efficient as others with development and refining, plus a licensing agreement who knows how much better the software can get. For now the feat is amazing and welcome in all facets of the post universe. Thanks Andrew for pointing the way.
August 18th, 2008 @ 9:41 am
COOL check out each person in that project. all are far far far far futurestic BrainY guys. i too wanna move out!.

i found more than this invention. a croma without a green, blue or red screen. its just ausum!!!
August 18th, 2008 @ 10:34 am
First i thought, that is the direction that leeds to FX they use or better show in movies like Running Man (where Arnold´s face has been replaced digitally by clicking on ONE button only!)
But at the other hand i think a lot of the shown examples are possible with AE, too (remember: they are using still photographs for enhancement) or some special tool actually availible. I love such special tools a lot and they are making my live easier. Time will tell if it´s easier to enhance with that tool or still AE. But normaly you will get bad original footage in reality from private movie makers who will not (be able to) pay for enhancement of their footage. It´s the reason why they make their movies on very low budget cameras.
August 18th, 2008 @ 10:56 am
uhh... look at this:

that is or could be a great tutorial - isn´t it Andrew?

It´s so amazing but will be easy if you are teached by VC ;-)
August 18th, 2008 @ 10:59 am
just watched the video demo. blew my mind! this could save a lot of work. I also felt a sudden drop my IQ after watching.
August 18th, 2008 @ 11:30 am
@ bijandisco

If I remember correctly, Andrew posted this on the blog some way back.

Also, on Andrew's / Creatice Cow's Serious FX Training DVD it has a whole tutorial on making groups of people from one person (Sam dancing in this case). Can easily be done, if you have the motivation and patience to do so
August 18th, 2008 @ 11:52 am
@ thomas_017 i meant that is easy to do for all of us vc students, but didn´t know andrew posted it some time ago.
My point is: that looks so amazing but is so easy to realize
August 18th, 2008 @ 12:45 pm
you are awesome,
im getting into expressions,
and you are so right,
the power of controling my stuff ibecame much better now;
you should get payed by adobe too ^^ they cant show me their app like you did.

please continue with this,
this is much better then what ive learned at school:-)

best regards from germany
August 18th, 2008 @ 5:32 pm
I'll surely keep an eye or two opened for that
Rob Moffett
August 18th, 2008 @ 5:55 pm
Hey Andrew, if you like that, you will LOVE this...
August 18th, 2008 @ 6:47 pm
I can see the Department of Defense and Homeland Security putting this to good use to cover up and doctor footage in favor and against our nation as well as those abroad
Peter Somers
August 18th, 2008 @ 8:11 pm
Wow, the VideoTrace link is amazing ... Really takes the hard work out of modelling 3D objects from just XYZ plane photographs.
Anyone know if there is already a commercially available version of this somewhere?
August 18th, 2008 @ 9:19 pm
Those damn algorithms. I never liked math. Still don't.
August 19th, 2008 @ 2:08 am
wow, this is really amazing. if you can do this now, what can you do in the future? they just have to build al photocamera in a videocamara that makes 1 photo a sec. ore something. you will have endless oppertunities in the editing whit just one handeling.
August 19th, 2008 @ 2:10 am
but when can you get this sofware, what is the name of it?
August 19th, 2008 @ 3:59 am
@Rob Moffet: That's actually pretty simple to do manually, it's just a bit more work. Just load your footage into a 3D tracking application (Boujou, Matchmover, flame*, whatever), then load the camera data that is generated into your favorite 3d app (Maya, XSI, 3ds Max, …) and load your footage as a background/reference. Then start modelling. If your tracking app can export locators for arbitrary points (or the point cloud it used for tracking), make sure to take advantage of that so that you already have the z-depth of the key points of the model so you only have to estimate the position of the vertices inbetween, essentially reducing the modelling problem to a drawing-by-numbers exercise. Move around in time and adjust and extend your model where necessary. Then take screenshots of your footage at key points in time (i.e. approximate front view, side view etc.), duplicate your camera and use it as a projector to create a texture for each view via simple camera projection. Stitch them together in Photoshop and apply it to your model.

On the other hand, if you have a tool that can generate an approximate depth map for your footage (via optical flow or whatever method) and you have the camera data from the 3d tracking app, writing a simple program that can do the modelling is quite simple since it could easily "unproject" the 2d screen coordinates you click with your mouse into 3d vertices for your model. You could then export a simple 3d .obj model or something like that for use in a 3d app or a compositor that can handle 3d geometry (like Nuke or flame*).

August 19th, 2008 @ 8:29 am
August 19th, 2008 @ 8:59 pm
Video Trace!!! It's very cool!!!
August 20th, 2008 @ 3:58 pm
wow... why didn't YOU think of that Andrew?

August 20th, 2008 @ 8:37 pm
reminds me of the photosynth technology

video here
August 21st, 2008 @ 3:56 am
This has some great applications for video HDR and he sure does talk pretty...
August 21st, 2008 @ 6:07 am
Since everyone seems to be putting links up.. This is really cool

Research Studies
August 25th, 2008 @ 3:03 am
Kinda seems like they took matchmoving software one step further.
Cool as hell though!
December 3rd, 2008 @ 9:39 am
hey andrew can you do this to.. or this is for professional...
August 13th, 2010 @ 8:44 pm
Okay i'm gonna give a frank opinion now.............i like tech stuff like this but....................i couldn't really stand the video after the first minute...............i know one thing for sure.............if Andrew had made that video instead of that person............i would have loved it.
Anyway the video was still informative and cool.
December 14th, 2010 @ 8:29 am
Wow, that's some pretty cool stuff.
Before Posting
We're trying to encourage more thoughtful commenting.  Please take a moment to write a meaningful response to this post.
* Name
* Email (will not be published)
* field is required