Whats the most difficult part of physics?

To me, the hardest part of physics was really just wrapping my head around the context (when to apply what principles + equations) and the explanations. Depending on how good your teacher is at explaining, and how clear his diagrams are, you will also be affected by the preciseness and guiding ability of your textbooks. When I took up physics during my high school years, I recall going through no less than 7 different physics textbooks before I found one which finally enlightened me.

Also, good luck to you if your teacher/books you're using is complicated and long-winded like hell! If you're stuck with such material, then the best thing to do is source out problem questions (online can be a great source) or workbooks with nothing but physics questions. Even better if you can find a physics book with nothing but diagrams, questions and worked solutions. Then practice, practice, practice. The more questions you do, the more accustomed you'll grow to them. Once you've done a certain amount, you would be able to automatically think 'Oh, this looks really familiar to that question about the helicopter moving at X speed, I can apply similar steps and equations here'.

Other problems you may encounter would be remembering equations. But really, equations and definitions in physics comes almost hand in hand. If you remember the equation, you'll remember the worded definition. If you remember the worded definition, it's most likely you know what the equation is.

To be very honest, math in physics is not very complicated. If you like Maths, have a brain for numbers, and have a calculator, math in physics is literally nothing. Anyway, don't be nervous about it. Approach the subject with an open, clear mind, concentrate during class and work hard with your books and assignments, and surely physics will reward you back for your hardwork with decent grades that you deserve. Best of luck!