What word modifies bad in the sentence If you do not exercise physically you may feel bad?
If physically modifies bad, it should be moved to the end of the
sentence in order for the sentence to flow smoothly. However, it
isn't at the end of the sentence. That tells us that bad is not
modified in this sentence. Rather, "physically" is telling us about
exercise; the exercise is physical exercise not mental.
Compare with this sentence where bad is modified:
If you do not exercise you may feel bad physically.