Well it depends,just like Ayurveda,Vaastu Shastra is a branch of science or knowledge that was practiced in the ancient times.Vaastu will not bring any magical change in fortune or any such thing,that is mostly superstition which is unfortunately often attached to all ancient knowledge system.
While the Shastra itself is quite ancient,most of it was developed during the age of massive temple buildings.The two key ingredients of ancient temple building was physical engineering (building the structure itself) and its intended spiritual purpose.
The physical aspect is most obvious,because they used massive stone masonry,architects had to get the design and alignments absolutely perfect,because there wont be any recasting.
The spiritual aspect was metaphysical,like you mentioned It was believed(and not without reason)that the fundamental elements of nature, cosmos and even planetary bodies affected human lives,which impacted his physical and mental health.So when they raised an artificial structure,the space within the enclosure was thought to have disturbed the natural interplay of the elements.Thus the structure had to be designed in such way so that any artificial rearrangement of the environment was minimal and the natural flow primeval energies were not interfered with.
We don't know how much of this has scientific basis,however there is no doubt that even non traditional engineering gives much value to environmental aspect of structures,like proper access to natural lighting,flow of air,presence of water bodies,and its impact of people living with it.
There is no reason to believe that buildings not following Vaastu are unfit for habitation or vice-versa,but its still better than building a house with due consideration to such natural and time observed factors.