Our minds are always constantly reminding us about the kind of person we think we are, no matter where we are or what we're doing. It's always there to call up everything we think is wrong with ourselves. This can be triggered by a myriad of things, but it always comes down to feelings. They play a huge role when we picture in our minds who we are. Skinny people can feel fat, young people can feel old, smart people can feel dumb, talented people can feel like losers, etc.
When people judge themselves critically, they have an underlying psychological tendency to see others as judging them in the exact same way. When people live in a world where everyone judges them as harshly asthey judge theirselves, it becomes impossible for them to feel good and be confident.
If you can control these feelings, then you can control how you feel about yourself and control your self-esteem and self-confidence to a great degree. Naturally, there will be external happenstances that will challenge how we view ourselves, but that's relatively minor compared to the internal aspect.