You are right. If the sensor is covered, the auto-brightness is not decreasing anymore without reasons. So it seems, that just very little amount of light, reflected in a disadvantegeous way, can cause this behaviour.
I would say this is some kind of “design-related peculiarity” which can not be easily changed. Except putting tape on it or turn off the auto-brightness, of course.