Now comes the real question:

I love physics. I love the class I'm in - I just can't do the math. Calculus is and will always be my downfall. So do I struggle through it and get a trerrible grade because I can't do the math or do I drop it now while I still can?

I don't need it to graduate. I'll still be a fulltime student if I drop it. But... It feels like giving in. Even if I really would like to, for all I enjoy the subject. I just can't integrate. 
I quit physics in our version of high school. I love science, but the math was beginning to go way over my head, or at least my speed of learning, (I would understand things if they were explained properly, with me asking a dozen questions...) plus the teacher was super boring -_- (compared to the ones I'd had earlier, who had made things interesting)

I can't say what would be right for you, but I think you're right in pondering if this is something you necessarily need to be doing, because you have so much going on already. Sometimes it's not about giving up, but about priorities.
I ended up dropping it last night. My GPA can't take the hit if I'm to go on to grad school, however much I love the subject. I feel a little guilty, but ultimately its the best decision. I hope.