You can hold a conversation in a café and then freeze completely when the same conversation happens on the phone. Nothing about your level changed between those two situations. Almost everything else did.
You lost half the information
In person you read lips, expression, gesture and the situation itself. On the phone all of that is gone and the audio is compressed, so you are decoding degraded sound with no context to repair it. This is why phone calls feel like a sudden drop in level. It is not a drop in level; it is a drop in available information.
There is no time to recover
Face to face, a confused expression makes the other person slow down automatically. On a call, silence just sounds like a bad line, so nobody adjusts and the pressure keeps building. You have to ask for what a face would have requested for you: please speak slowly, could you repeat that, I did not catch the last part.
Prepare the call on paper
Write your opening sentence, the reason you are calling and the three questions you need answered. Read them if you must — nobody can see you. Write the likely replies too. Half of phone panic comes from not recognising the answer to your own question.
Start with calls where failure is free
Ring a shop to ask their opening hours. Ring a restaurant to ask whether they are open on Sunday. These take thirty seconds and nothing goes wrong if they collapse. The aim is to make the format ordinary before you use it for something that matters.
Phone calls are a separate skill on top of speaking, and they are learnable the same way: a script, a few low-stakes repetitions, and the phrases that buy you a second attempt.