Games became big after people in this age group had reached adulthood. Women, especially, have real lives and little time for gaming.
Atari and Nintendo were only just coming out when I was in 4th or 5th grade. That was some 30 years ago, and I can't imagine many people over 20 at that time really getting into video games ;-)
Yeah, seems the logic of the generation before us was that eventually you have to grow up and put away your toys. Then along came us who said, why should we? ;-) lol