Phone call audio really loud in the beginning

Everytime I make or receive a phone call the volume is really loud in the beginning. I have to press volume up and volume down to make the volume lower (pressing only volume down doesn’t do anything because the volume thinks it is already in the lowest). I think this is related to the absolute volume setting which I had to disable from developer settings because the sound volume resolution is so coarse (only fifteen steps). Is there anyway to fix this issue with having also the absolute volume disabled? Thanks!