Population shift is one major factor, to be sure.
Desegregation is another. Here's a list of when the original SEC programs became desegregated:
Tennessee: 1967
Auburn: 1969
Florida: 1969
Vanderbilt: 1969
Mississippi State: 1969
South Carolina: 1969
Alabama: 1970
Georgia: 1970
LSU: 1971
Ole Miss: 1971
Now here's a list of players (perhaps incomplete) who were in high school in the deep south during the era of segregation, who went to historically black colleges, and who are now in the NFL Hall of Fame:
John Stallworth (AL) Alabama A+M
Larry Little (FL) Bethune-Cookman
"Bullet Bob" Hayes (FL) Florida A+M
Willie Brown (MS) Grambling
Buck Buchanan (AL) Grambling
Willie Davis (AR) Grambling
Charlie Joiner (LA) Grambling
Lem Barney (MS) Jackson State
Walter Payton (MS) Jackson State
Jackie Slater (MS) Jackson State
Art Shell (SC) Maryland-Eastern Shore
Ken Houston (TX) Prairie View
Harry Carson (SC) South Carolina State
Deacon Jones (SC) South Carolina State
Mel Blount (GA) Southern
That's just the Hall of Famers. Now add in star players like Ken Riley, Harold Carmichael, Ed "Too Tall" Jones, Jethro Pugh, Raymond Chester, Claude Humphrey, Al Beauchamp, Barney Chavous, Ken Ellis, Isiah Robertson, Sam Adams, Henry Lawrence, Nemiah Wilson, John Gilliam, Otis Taylor, Rayfield Wright, Verlon Biggs, Essex Johnson, Gary "Big Hands" Johnson, Carl "Big Daddy" Hairston, Emerson Boozer, Dick Westmoreland, Glen Edwards, and Robert Brazile.
Now add in literally dozens more players who had careers in the NFL.
That's an awful lot of talent to leave standing on the other side of the schoolhouse door.
It's no wonder that the SEC schools in particular, and southern schools in general, developed more powerful football programs after desegregation.