The real equalizer in all of this is that there are only 32 NFL teams. So there are only 32 GM spots and maybe 100 top other executive GM positions.
You can't be real picky when are opportunity to advance comes along. You either take it or you might have to wait another 3-4 years before you are looked at for a similar position.