Sounds interesting and I tried it. But it did not change anything. Anyway, thank you for this hint!
Seems like I can not blame the Jitsi app for the problem, as it works fine on other phones. And as I see in this thread, similar or the same audio problems exist also in Zoom and WhatsApp, when used on the FP3(+).
So I guess this needs to be fixed by Fairphone in their firmware.