From The Guardian:

Going backstage to see mates afterwards throws up interesting moral conundrums, particularly if you’re an actor yourself and know how things work. If you’ve enjoyed the show, all is sweetness and light: gazes can be returned, questions can be asked and backs can be slapped. But what if it’s the biggest turkey before Christmas?

The author suggests you’ve got three options in this situation: tell the honest truth, lie, or say something without saying something.  I’ll pose a fourth option (and this is a big secret of mine that I’m giving up here, so appreciate it): ask them how they feel.  Always ask them how they feel.  Even when the show was great, ask them how they feel.  Theater people want praise not so much because of the praise (although the praise is great), but because they want people to care about them.  If they’re your friends, you do care about them, even if you don’t like the show.  So ask how they feel about the performance that night, and work from there.
Unless you’re coming to see my show.  Then just tell me you loved it.  I’ll know you’re telling the truth.