Many Buddhist philosophers accepted certain assertions that cannot be empirically proven, such as those regarding transmigration of souls, reincarnation, consciousness, etc. By modern standards they may appear as "idiots" for accepting such assertions as true. However, I think it is important to keep in mind that the ancient Buddhist philosophers and Siddhartha Gautama himself did not have access to the modern scientific knowledge that we do today. Given the absence of this knowledge, they lacked some key insights into the nature of the physical world and the physical human body that we have today. The state of scientific knowledge in the 6th century B.C.E. was not advanced enough to decisively support materialism; the Charvaks and other such materialists could not cite scientific journals and information to support their arguments, but instead had to resort to philosophical debates. Even in the 21st century, when scientific knowledge overwhelmingly supports materialism and increasingly shows how ludicrous and backwards religions are, there are many humans who still cling to their personal religions and reject scientific theories out of reasons of "faith". One can only imagine how the situation must have been 2500 years ago. The Charvaks and other rationalists must have had a very difficult time back then, and it is not too surprising that they went extinct and were never extremely popular.