"He got out of his car and walked back to us and he said, 'what do you think happened?'" Rich Sanchez, 26, of La Verne, said about Bryson. "I said, 'well, you just hit me.'"
Sanchez said he thought Bryson may have received preferential treatment.
That's because prosecutors announced Tuesday that they have decided not to file criminal charges against Bryson.
Authorities said his blood tested negative for alcohol and controlled substances.
Sanchez said that after the accident he and Bryson agreed they would pull over after the train passed and exchange insurance information.
Instead of pulling over, Sanchez said Bryson put his car in reverse, hit the Buick again, and sped away.
The brothers then called 911 and followed Bryson down San Gabriel Boulevard.
"We're thinking we're not just going to let him get away," Sanchez said.
Bryson then hit a center divider and rear-ended another vehicle, Sanchez said.
"At that point we were able to pull in right behind him and kind of pin him between the two cars--mine and the other one he was just hit," Sanchez said.
Bryson was found unconscious and slumped over his wheel, his car still running. He was transported to a local hospital and charged with hit and run.
Following the accidents, a Commerce Department spokesperson said that Bryson, 68, had suffered a seizure at the time.
The White House subsequently announced that he would be taking a leave of absence to undergo tests and evaluations.
Bryson then announced his resignation less than two weeks after the crashes.
According to the police, Bryson was found unconscious at the wheel of his car after two accidents on Saturday, June 9.
The first crash happened around 5 p.m. in the 400 block of San Gabriel Boulevard, according to San Gabriel police.
When Bryson continued to drive his Lexus into Rosemead, he allegedly crashed into a Honda Accord around 5:10 p.m. near the intersection of San Gabriel Boulevard and Hellman Avenue.
Bryson voluntarily took a breathalyzer test that detected no alcohol use, the Los Angeles Sheriff's Department said.
According to a Commerce official, Bryson was on personal time with no security detail when the accidents occurred and was driving his own vehicle.
The official, who spoke on condition of not being identified, told CNN that Bryson was given medication to treat the seizure.
Bryson, who never had a seizure before, was in touch with doctors who are monitoring his progress, the source said.
Bryson had limited recall and the cause of the seizure, as well as the exact sequence of events, was not clear, the official said.