Then you need read this article. If that doesn’t convince you, then head on over to this site to see just how great you can get on a vegan diet.
If anything, being vegan may help you, depending on your current diet and other factors that may be limiting your ability to improve.