Photo Credit: Getty Image
He and two other hostages were then able to get out "without a shot being fired", following a 10-hour siege.
The hostage-taker, named as British citizen Malik Faisal Akram, was shot dead after a stand-off with police.
US President Joe Biden has described the hostage-taking incident as "an act of terror".
It began at around 11:00 local time (17:00 GMT) on Saturday.
Rabbi Charlie Cytron-Walker told BBC partner CBS News the group had been praying when he heard a click that turned out to be the hostage-taker's gun, and he and three others were held captive.
"We were threatened the entire time but fortunately none of us were physically injured," he said.
One hostage was released after six hours, while the other three escaped several hours later.
"When I saw an opportunity, when he wasn't in a good position, I made sure the gentlemen were still with me, they were ready to go," Rabbi Cytron-Walker recalled. "The exit wasn't too far away, I told them to go."
He then threw a chair at the gunman and headed for the door.
"It was terrifying. It was overwhelming and we're still processing. It's been a lot," Rabbi Cytron-Walker said.
He added getting back to the synagogue following the incident "won't be easy but it's an important thing".
The service was live-streamed and captured the audio of the man demanding the release of Aafia Siddiqui, a Pakistani neuroscientist who is currently serving an 86-year prison term in the US, law enforcement officials told local media.
Siddiqui was convicted of trying to kill US military officers while in custody in Afghanistan. Thousands took to the streets in Pakistan to protest against her conviction in 2010.
A brother of Malik Faisal Akram issued a statement apologising to the victims and saying he had been suffering from "mental health issues".
Friends of Akram in his home town of Blackburn told the BBC his mental health had been getting worse and said they were surprised that he had been able to travel to the US.
CBS News reported that Akram arrived in the US two weeks before the siege, landing in New York.
Mr Biden said on Sunday that the assumption was that "he got the weapons on the street" and "apparently spent the first night in a homeless shelter".
Two teenagers have been arrested in England as part of the investigation into the incident. Their ages and genders have not been revealed.
British Foreign Secretary Liz Truss described the incident as an "act of terrorism and anti-Semitism".
"We stand with US in defending the rights and freedoms of our citizens against those who spread hate," she said.
When the FBI said this investigation would have "a global reach" they weren't exaggerating.
The FBI, who are leading it, have deployed their Joint Terrorism Taskforce with teams following up leads in the US, Britain and possibly also in Pakistan.
The immediate priority is to establish whether or not the hostage-taker had help from anyone else, either here in Britain or during the two weeks he was in the US prior to entering the synagogue.
Britain's Counter Terrorism Police and MI5 will be crucial in uncovering Malik Faisal Akram's so-called "digital trail" - the clues left behind on his phone, computer or credit card purchases.
