As I said why would I be worried if I don't have anything to hide. As simple as that. But if I am planning a blast and purchasing plastic explosives for it, or learning making parcel bomb over dark web thats when I want my right to privacy protected. You know just in case. Trust me.
You're looking at it from a collective security perspective only.
There are other aspects to it.
1. Financial; a person with access to such large and well endowed database like Aadhaar has better chance of stealing my data.
2. Blackmail. Your private/sensitive data can be used by online gangs to blackmail and extort money.
3. Insurance scams; health data etc.