I like this thought from Martin Geddes of why video conferences have never taken off:
“Without having something to gesticulate at â other participants, a diagram, the window â youâre left limp and lifeless. So perhaps thereâs a Superman-style blue backdrop screen type of technology that can re-insert those elements.”
Perhaps this is a job for EyeToy hackers, or those with an iSight, Quartz Composer and a spare afternoon?
—-
Bonus link: some lovely Quartz Composer work at zugakousaku.com
Petafloptimism 
The lack of normal interaction IS a key element. However, I’d suggest the un-reality of the image is more a factor than lack of a shared whiteboard or similar.
Have a read of this then click thru the Slashdot link to see something that DOES have a HUGE impact on Video Conferencing’s usability:
Holographic 3D on your desktop