You took the perfect example! In 2014, two of my close relatives had those phones. My girlfriend's Note 3 would wait 2 days before needing to be charged, whereas my flatmate's S4 mini would need to be charged every night (they had pretty much the same use of the phone). And that perfectly makes sense:
On the Note 3, the typical screen consumption was 20 mAh per hour (depending on the phone use, of course), and the typical other activities consumption was 60 mAh per hour. The phone consumed 20 + 60 = 80 mAh per hour. So the 3200 mAh battery had juice for 3200 / 80 = 40 hours (from 8 am till midnight the next day).
In the S4 Mini, the screen consumption was 20 * 50 / 90 = 11 mAh per hour and the typical other activities consumption was still 60 mAh per hour. The phone consumed 11 + 60 = 71 mAh per hour. So the 1900 mAh battery had juice for 1900 / 71 = 26 hours.
The calculus mistake you make is implicitly assuming that the screen is the only energy consumer. If it was the case, increasing the screen size and the battery size by the same rate would indeed be pointless.