Now comes the real question:

I love physics. I love the class I'm in - I just can't do the math. Calculus is and will always be my downfall. So do I struggle through it and get a trerrible grade because I can't do the math or do I drop it now while I still can?

I don't need it to graduate. I'll still be a fulltime student if I drop it. But... It feels like giving in. Even if I really would like to, for all I enjoy the subject. I just can't integrate. 
I ended up dropping it last night. My GPA can't take the hit if I'm to go on to grad school, however much I love the subject. I feel a little guilty, but ultimately its the best decision. I hope.