While I appreciate the fact that statistically an IQ of 160 is as far from "normal" as an IQ of 40 is, I don't really think it is an appropriate comparison. I have worked with children at both ends of the normal distribution, and the kids with an IQ of 40 really do struggle more and stand out more. Perhaps because they stand out more, people are forced to accommodate them better. Maybe it is because i have always lived in college towns with a high concentration of people in the two standard deviations above the mean range, but it is possible to "pass" as bright without your peers picking up on just where you are functioning.
And yeah, you're totally right- people with very low IQs do stand out a lot more than "equally" gifted kids. Low IQ isn't just impacting their ability to learn, but physiological stuff, too. As a species, we have a preference for intelligence as well, so while a super-genius may be off-putting, we still have admiration for someone who is profoundly gifted, whereas many people have almost a repulsion for those with serious disabilities. I guess I just mean that the gap is just as wide, and imagining what it's like in the middle is going to be harder and harder the farther you are from average.
Height might be a better analogy. Someone who is 7' tall will stick out just as someone who is 4', but the shorter person will have a harder go of life, from struggling to get around to being patronized or mocked. We value height, so we may stare at the tall person, we don't have the same reaction when we see a very short person. ("We" is used generally, of course.)