Same problem for me with FP4 and e/OS/ 1.17 (Android 12). I am actually glad that the problem is not related to e/OS/ but to Fairphone 4 in general so I had hope that this issue gets fixed quickly. However, seems to be not the case 
Update:
Oh wait a second, I just checked and it is fixed in the latest version e/OS/ 1.18. They literalley have it in their changelog. They fixed the volume issue and I can confirm that it seems to work now. The lowest volume is much lower now, nice 