"Market size" is flawed. In bigger markets, there's a shitload of other competing things and less of a connection between the market and any one thing.
Market size matters, to tv execs, in ad sales because the sales are done before the fact so the potential audience size have a greater effect on ad rates than the actual numbers, especially in college sports which tends to be volatile.
It's exactly why the Big 10 needed Rutgers and Maryland. It was a low stakes wager that those two programs would become relevant in two huge markets.
Of course you're correct that this hardly ever ever comes to fruition because college sports teams in large metro areas generally don't rate well. Even in a place like Houston, which you'd think would do very well, are generally lukewarm for live sports ratings. If you look at sports ratings it's always mid sized markets that lead ratings of live sports, places like Kansas City, Tampa, Minneapolis, Raleigh, Salt Lake, and Birmingham.