What would happen if you had a symphony or musical going on and you took the maestro away, would the musicians suddenly suck or not know what they were playing? Do they really pay attention to the guy with the stick moving his hand, or is that just for show/theatrical purposes?
If they were really good, someone in the front (depends on the ensemble), would probably just stand up and cue the sections that need cuing. If it was something simple, say like a John Phillip Sousa march, and the musicians were even mildly good, they'd be fine. If it was something more complex with multiple tempo or time signature changes, it could get rough. But if the group is well rehearsed, they'd probably be ok.
This is pretty accurate. In fact, you see it with some big bands, where the leader will direct from the piano or some other instrument that he/she is also playing, so he or she mainly directs during tempo changes or other key passages. The larger the group, the more necessary the conductor.
Interesting sidebar: most folks don't associate improvisation with classical music. But some concertos were written allowing the cadenza (solo portion for the featured pianist/violinist, etc.) to be improvised if the performer so desired. There's a certain ending notated in the music for the soloist to use at the end of the improvisation, so that the conductor knows it's time to prepare the orchestra to come back in.
