Started AFF - not sure if skydiving is for me?

Hi all, hope you're well! A little bit about me; i've always fancied the idea of flying and been an adrenaline junkie (done quite a lot of bungee jumps - although i know skydiving is quite different from it). I'm half way through my university undergrad degree - thought i could spend most weekends learning how to fly and invest the time and money into skydiving from now. I went for my AFF in the UK. Done 2 jumps only (AFF level 1 & 2). Although, I'm quite petite and weigh just about 42kgs. Not as fit as i'd like to be. I found the rig a little bit too heavy and the jumpsuit i was given really lose. They added additional weight by giving me a weight belt. Found this a bit heavy to handle as well. When I went for my jumps - after exiting the aircraft i was really worried about the weight + if me and the instructors were falling at the same rate. During my first jump i had a stable position and arch - felt weightless. It was great excluding the wind on my face (that i know i can get used to) and the noise and sensory overload the first few seconds. My second jump wasn't good - i didnt feel weightless, felt like i was falling really fast and felt like i was fighting the air... couldn't hold a steady position either... if it weren't for my instructors holding me i'm not sure how badly i'd have flipped around. After both the jumps i'd feel really exhausted and would find it super hard carrying the canopy back (Literally just a 8 min walk). I have decided to take some time off and work on my fitness - joined the gym, eating better, getting some yoga done. I can't stop thinking about skydiving and wanting to go back and learn how to fly and getting through the challenges i'm facing due to my weight. It's like a challenge... something i just have to do. despite not liking my jumps to an extent where i'd say I LOVED them... just confused if skydiving if for me or i'm being stubborn trying to do something I can't... 

A few thoughts in no particular order:

If you want it, go for it. Better fitness and flexibility are always a good thing, but there are some really out of shape people who jump and do just fine. Also some 'old and crunchy' types (including me) who do ok. 

Student gear sucks. It's big & heavy. It's "one size sort of fits most, but not really". Once you get through the progression and get your own stuff, it gets a lot better.

The instructors should know what they are doing. The 'big & baggy' jumpsuit slows you down. The weights speed you back up. Part of the job of the instructor is to make sure that they can fly with you. From their viewpoint, it's easier to make you go faster and work to stay down with you than have you go slower and fight to stay up with you (it's easier to speed up than slow down). It would have been better to find a tighter suit and skip the weights, but that may not have been an option. Perhaps you can find a smaller jumper who is willing to let you use one of their suits. Most jumpers have at least a couple. 

I'm not surprised you were exhausted. Much of it was mental, rather than physical, but that doesn't make it less real. Some places have golf carts or similar to bring jumpers back from a distance. It's a nice thing, especially for early students who are often overwhelmed and exhausted, even if everything goes well.

You don't say if you passed the levels.  What did your instructors say about how you did?  Many jumpers experienced setbacks, less than perfect jumps during their AFF progression.  Skydiving doesn't just come naturally to many people, it's something you have to work at to learn how to control yourself in the air.  Just a guess, but your experience on your second jump may indicate you tensed up and were fighting the air, which isn't at all unusual.  I think I was told I had to relax more after each of my first 3 or 4 jumps.  Did you meet the goals of the jump?  Were you aware of your altitude, and did you pull on time?  If so you did fine.  If not, it's common to have to re-do a level, or even multiple levels.  You will progress faster, and have more fun, if you are in good shape so that is always a worthwhile goal.  However you should be aware that if you take a long time off to get in shape you will have to go back a bit in your training.  The point of having several levels to AFF is to build survival skills, and too big of a gap will interfere with that.  If you are having trouble relaxing, maybe a tandem jump where you don't have the stress of performing can help you to relax and experience more of the fun, and help you get in the mindset to carry on with AFF.  Also as has been suggested a better fitting jumpsuit could help you to fall faster and have better control without fighting a lot of flapping fabric.  If you are very light, though, some added weight may still be needed so your instructors can fly with you.

If you can't stop thinking about skydiving, and your instructors didn't give you the "take up bowling instead" speech, you can work past these very common early self-doubts.