Depends on whether you pay at the table or at a cash register on your way out.
If they bring you the ticket, take your card, and return the ticket with a place for tip, you should tip. how much is your call (not more than 20% unless they brought you free crap, brunch is almost always overpriced and never very good so your overtipping no matter what).
If you have to wait in line at a cash register to pay, throw a $1 and whatever change is in your pocket on the table for the bus boy. If you're one of these weirdos that doesn't ever carry cash, you have a built-in excuse not to tip, use it.