Bayindir the hero as 10-man Man Utd knock Arsenal out FA Cup
United's stand-in goalkeeper saved a penalty from Martin Odegaard in normal time and then denied Kai Havertz in the shootout as the wasteful Gunners had another day to forget in front of goal.
Bruno Fernandes had fired the visitors into the lead early in the second half before Diogo Dalot was sent-off on the hour mark for a second bookable offence.
Gabriel Magalhaes quickly levelled for Arsenal, but they wasted a host of chances to win the tie in 90 minutes and were made to pay.
United were perfect from the spot as the much-criticised Josua Zirkzee stroked home the winning penalty in front of the jubilant travelling support.
"We deserved to pass through this round because we suffered all together and showed character," said United boss Ruben Amorim.
United's reward is a meeting with the club's legendary former striker and caretaker boss Ruud van Nistelrooy, who will return to Old Trafford as Leicester manager in round four.
Defeat is another hammer blow to Mikel Arteta's attempts to end a five-year trophy drought for Arsenal, who also lost the first leg of their League Cup semi-final 2-0 to Newcastle on Tuesday.
"It was unbelievable," said Arteta. "(Based on) The performance we deserved to win the game by a mile, but the reality is that we are out and the only thing we will be judged on is that."
After four consecutive defeats in all competitions, United were much improved in a 2-2 draw away at Premier League leaders Liverpool last weekend.
Amorim's gameplan was a similar one as the visitors were happy to sit back and contain Arsenal before looking to spring on the counter-attack.
Jesus adds to injury woes
However, neither side posed much attacking threat in a flat first half.
And Arsenal's injury problems mounted before the break when Gabriel Jesus was carried off on a stretcher with a suspected serious knee injury.
If the first 45 minutes lacked for action, the second period more than compensated.
United hit the Gunners with a sucker punch seven minutes after the restart.
Alejandro Garnacho took advantage of a slip by Gabriel to gallop into Arsenal territory and teed up Fernandes to blast brilliantly into the top corner.
Shorn of the injured Bukayo Saka, Arsenal's attack has lacked creativity and goal threat in recent weeks.
But just as United were beginning to believe they could deliver another result on the road for Amorim, they were let down by Dalot's indiscipline.
The Portuguese full-back needlessly dived in on Mikel Merino when already on a booking and was swiftly shown a second yellow card by referee Andy Madley.
Within seconds Arsenal had made the man advantage count.
Bayindir flapped at Martinelli's cross and punched the ball into the path of the Brazilian centre-back, whose shot deflected in off Matthijs de Ligt.
The home side were gifted the chance to turn the tie around when Harry Maguire was harshly penalised for a challenge on Havertz which sparked a melee among both sets of players.
Maguire angrily shouted "cheat" towards Havertz, who then went head-to-head with Manuel Ugarte.
But if Bayindir was partly culpable for the Arsenal equaliser, the Turkish international made amends with a brilliant save to turn Odegaard's penalty behind.
United needed more Bayindir heroics to make it to extra-time as he tipped over Declan Rice's header.
Havertz somehow fired over from under the shadow of the United crossbar and Rice drilled a shot inches wide as Arsenal failed to kill the tie off.
United, by contrast, were ruthless from the spot as Fernandes, Amad Diallo, Leny Yoro, Lisandro Martinez and Zirkzee converted to progress.