I read that some people are convinced about the phone's size, given the bigger screen say, 4.5 or 4.7".
I don't think it will have any negative affect on the phone. Its how the device is built is what matters. For a reference, check the LG G2, and how perfectly it scales the display size across the device, without really affecting the overall phone. Thinner bezels, not only on the sides, but also at top & bottom ... THATs the formulat of achieving perfect ergonomics.
I believe the smaller screen iPhones will share the same fate, with display size getting a smaller and reasonable bump ... with overall dimensions being the same.