In addition to being physically fit, an athlete's performance will be benefited by developing many of the following skills and components listed below. Depending on the sport or activity, one will want to focus on many of these specific skills. Developing these skills can make a huge difference to an athlete's performance.
Agility, Power, Strength, Speed, Quickness, Balance, Endurance, Flexibility, Reaction Time/Timing, Coordination, Stamina, Accuracy, Concentration, Experience, Competitiveness, and Mental Preparation.
I can't tell you how many times I've made the mistake of judging a book by it's cover. I once played basketball against an older guy who was very over weight, and only about 5'10". Despite his appearance, he went off for 30 points in the game and nobody could stop him. I saw him do the same thing all season long. At other times I've been in awe of amazing physical specimens, but the intimidation stopped when it became apparent that they could hardly dribble a ball and breathe at the same time.
Being physically fit may be more beneficial in the long run than developing specific athletic qualities, but as a society we certainly seem to value those who run the fastest, jump the highest, or lift the most weight, as opposed to just being healthy.
Another aspect of fitness to consider is the ability to stay healthy for the long haul. I'm sure you've seen professional athletes who retire and a few years later, they've let their bodies go to pot. Sometimes participating in athletic events can take a toll on your body. It's not uncommon for athletes to experience concussions, dislocations, and joint injuries that can slow them down later in life.
Longevity is another area to consider when contrasting fitness and athleticism. If I had the choice of setting world records in my 20's or being in good enough condition to compete in the senior games in my 70's or 80's, I'd take the latter because of the positive health implications later in life.
There are many physically fit people who are also great athletes, so it's not like you have to pick one or the other, but if you did, which would you value more, fitness or athleticism?