A side effect of a variety of illnesses (uncontrolled diabetes, for example), medication (estrogens or testosterone-reducing drugs), and some types of prostate surgery is something called “penile atrophy."
This doesn’t mean the penis literally shrink, it just means that the penis loses its ability to enlarge. A penis that is always in its normally compressed, small state may appear to be permanently small, but that’s just because it’s not longer getting large on a regular basis.