It's my opinion that until the program wins the World Series, attendance is not going to be what it needs to be. The program was on the cusp of this during that stretch in the mid 2000's, but regardless of how good the team was, they flamed out in the World Series.
Fans started to feel like the program was just good enough to be highly ranked, but didn't have what it takes to win the World Series. I think the 2020 team was poised to prove that wrong until Covid derailed the season.
Baseball in February and March is another problem for attendance. It's just too damn cold to sit outside on bleachers for 3 + hour baseball games. If the program was coming off winning the World Series, more fans would likely show up when it's 40 degrees.