I think at least part of it was issues with the production companies. From comments from Amber, Sarah and Emma, it does not sound like Fox or ME (whoever makes these decisions) were very open to letting actors do other things. It's why Amber didn't want to sign a contract like the others. She wanted more freedom to do other projects.

So if you're having to turn down work because of a show that isn't paying much relatively speaking, the grass will surely look greener on the other side.