If your doc is saying it's stress is it possible that maybe you have anxiety issues? Try reading about
Anxiety and Panic Attack Symptoms to see if this sounds like your problem.
It sounds to me like when you are in new or stressful situations your body has a hard time coping very well but the one thing that may be an indication that it is something other than stress induced anxiety or panic attacks is if your stamina is declining considerably as time passes.
If that is the case then maybe you can ask to get tested for something called
Pulmonary Fibrosis. My mom has this and so do many many people and it's often mis-diagnosed or ignored because there isn't very much awareness about this. Some of the symptoms include always feeling short of breathe, heart arythmias, chronic fatigue, and trouble sleeping. You can read more about it at the link.
Hope this helps you getting the help you need to get healthy. There are solutions to any issue you may be having. You just have to find out what you're dealing with first : )