It can't be ignored because it eats at you. Remember that it's just a voice and only has power when you've allowed it to have power. Stand Up Comedy is a series of highs and lows. What I do when I'm feeling doubt when I'm doing well is walk from venue to venue. I need the city's sights and sounds to engulf me. It keeps me focused on the task at hand.
While I'm walking, I'm observing the city and remembering my set. That also helps me out with what I'm going to say next at the other Mic. I hope this helps you.