HMDs are only part of VR though. Head tracking is just as important, and i believe the ability to process that with low enough latency to avoid the nausea inducing cognitive disconnect is what for the most part has taken this long.
I may be stating the obvious here, if so I apologize :P