Confidence is built by accomplishing goals as you go along. Some people & horses catch on quickly, while others it takes a bit more time. Each person & horse is unique & there isn't a set program that works for both the same so you have to adjust your learning/training accordingly.
Confidence comes when you &/or your horse learns the lesson & moves on to the next. Starting with the foundation for riding gives you & your horse a place to start. I emphasize ground work for both new horse & new rider. When you buy a horse, you don't really know what you've got until you get it home & settled in, though you may have an idea when you went to look at it. Unless you have gone every day for about a week or 2, you may not see it's daily normal behavior. Home trials are often available to see if you & this new horse is a good match or not.
For many beginner riders, taking lessons is essential in building confidence. Your instructor can see which areas you need to work on & where you might best excel. Even seasoned riders may have a troublesome area & go to an instructor & get constructive criticism in order to fix it & become better riders.
I think for most beginner riders their first challenge in riding is finding their balance. Balance on horseback is different because you have to find your center of balance while sitting on a moving horse. During my first lessons my instructor would tell me to sit up straight, sit back on my pockets & turn my toes out to a 45 degree angle. This helps you to sit the horse. Once you find your balance, allowing your hips to move with the motion, keeping your upper body still also helps to find your balance & it is easier on the horses back.
Being limber is a great asset. When I was in lessons, my instructor would have me lean down & touch my toe without gouging with my other leg/heel or to allow it to swing way out. It takes practice, but it helped me with balance & so I practiced often for about 10 minutes. Squeezing a ball between your knees, helps to build muscles in your legs. Those muscles help you to stay on & help prevent you from falling. It also allows you to grip with the inside of your knee & your lower leg to move freely so you can cue your horse.
All of these things, though they may seem small, will help build your confidence in riding. There are many other things to learn also that will build confidence as you go along. Bounce steps or double stair stepping, help build muscles in your legs that will aid you in getting on your horse so that you don't pull it over, or put excess strain on the back & withers. If you have trouble getting on from the ground, use a mounting block until you build those muscles up. One of the things that you should learn is sense of feel. When you pick up the reins you should be able to lightly pull until you feel the horse. Yanking or jerking on the reins is only going to result in a bad situation. Practice with a rope at home. Tie it to a chair & pull the rope only until you feel resistance, then release.
Confidence takes time to build, don't be hard on yourself if you don't get it the first time, or the first 10 times. You will get it! Once you do, you'll be proud that you accomplished something & are able to move onto the next lesson, AND don't compare yourself or your training with how far someone else has got. Everyone learns differently & they learn different things at different times. Only concentrate on your lessons & always pat yourself on the back for a job well done.