The issue of brain is grossly over-exaggerated. Several of my classmates from school, left India and went to the UK for further studies or as they claim to do so. Most of them left, not because it's difficult to build a career in India and live, but rather because of an obsession to move to the UK. I don't know anyone who has moved to the US but all classmates I know, moved to the UK. And I can guarantee you, that most of them are not studious. Some are pursuing some good study field but that is about it.
This trend among Goans to move to the UK is very common. Punjabi Sikhs have an obsession to move to Canada as a preferred International destination, in the end they mostly opt for small time jobs such as Taxi Driver, Workers, cleaning jobs at airports, factory labour, or work at some food outlet such as McDonald's, KFC, Burger king etc. The same way, majority of Goans go to the UK and opt for low level jobs.
During my science practical, one of my classmates was speaking about someone's he knew who gone to the UK and become rich, he said this:
"He was a class 9th failure, he went to the UK, and earned tons of money by simply cleaning and washing toilets".
I honestly don't know how true his claim is but I would take everything with a pinch of salt.
Majority of the youth that we lose to the west, like you said, are the ones who are unable to get a basic education in India, in the long run, their skills would prove to be useless for India. Though there are outliers wherein we have indeed lost many bright minds to the west, which if would have stayed in India would have done a lot of R&D and scientific contribution to Bharat.
Some of these Goans, come back every now and then back from the UK, and oh boy, the way they have changed, I can truly say that moving to the UK cucked them.