Earbuds connect while within charging case

I have a FP3+ and am using the Fairphone earbuds. The earbuds seem to sometimes connect to the phone while in their charging case. This is happening every few days now. This is very annoying, especially when it happens when I’m using the phone. Suddenly the sound is gone. I can ‘solve’ this by disconnecting bluetooth, but that is not desired.

Does anyone have the same problem and/or know about a solution to stop the earbuds connecting while in the charge case?