You can discuss your job and follow a film, and then someone says a price and you are lost. Numbers are learned in week one and remain a weak point for years, and the reason is not that you forgot them.
Recognising is not the same as decoding at speed
You know the words. The problem is that a price arrives as a single fast burst, often with a currency and a decimal, and you have to convert it while the conversation continues. Numbers also rarely come with context to repair a gap. Miss a word in a sentence and the rest of the sentence tells you what it was; miss a digit and nothing does.
Some languages make it structurally harder
Several European languages say the units before the tens, so you cannot start building the number until the whole word has arrived. Others build on twenty, or on ten thousand rather than a million. This is not a sign you are bad at the language. It is an extra operation your first language never made you perform.
Drill them out of order and at speed
Counting from one to a hundred teaches you a sequence, not recognition. Use random numbers: prices, phone numbers, years, times, quantities, spoken fast. Have someone read a list at natural pace while you write digits. Ten minutes of this is worth more than any amount of counting.
Learn the repair phrases too
You will mishear numbers for a long time, so learn to check without breaking the conversation: sorry, how much? did you say fifteen or fifty? can you write it down? Asking is normal, and confirming a number is a thing native speakers do to each other constantly.
Treat numbers as a separate skill and give them their own ten minutes. It is the smallest piece of practice with the most obvious payoff in real situations.