Being out at night without permission was dangerous for slaves. That they didn't know what was ahead made their trip even more frightening.
And they knew that he was going to take them south for more war work, forcing them to leave their families in Hampton behind.
"Whatever they had, they knew they would get a better shake from the Northern troops," says Gerri Hollins of Hampton, keeper of the flame for the Contraband Historical Society, which has the lot of Baker, Mallory and Townsend as its genesis.
The contraband story is one of courage. It's one of hope.
And it's one of blind luck.
"They had no idea what they could expect at Fort Monroe," says Robert F. Engs, a history professor at the University of Pennsylvania. He's the author of "Freedom's First Generation," about blacks in Hampton from the Civil War through Reconstruction.
Had Baker, Mallory and Townsend escaped two days earlier, they probably would have been sent back by Fort Monroe's post commander, Col. Justin Dimick. The Fugitive Slave Act required it.
So did President Abraham Lincoln, who censured other officers for trying to free slaves. Lincoln was still trying to hold the union together.
Had Maj. Gen. Benjamin Butler not been sent to Fort Monroe on May 22 -- "to keep him out of trouble," Engs says -- the slaves probably would have been returned.
It had happened to others who sought refuge at the fort.
Had Hampton not voted on May 23 to endorse the ordinance of Virginia's secession, the slaves probably would have been returned.
But on that night, Virginia was no longer a part of the United States.
That confluence of events gave Butler, a lawyer by trade, the evidence he needed to tell Confederate Maj. John Cary -- an emissary sent by Mallory to reclaim his property -- that Baker, Mallory and Townsend would be staying at Fort Monroe.
It was a reunion of sorts.
Butler and Cary had met at the 1860 Democratic National Convention in Charleston, S.C., where both voted for Jefferson Davis to be the party's nominee to run against the Republicans' Lincoln. Stephen Douglas ultimately got the Democratic nod.
(In 1864, Butler turned down an invitation to run as Lincoln's vice president, saying he would do so only if the president promised "that within three months after his inauguration, he would die.")
In his memoirs, Butler writes that he told Cary on May 24, 1861: " 'I mean to take Virginia at her word, as declared in the ordinance of secession passed yesterday. I am under no constitutional obligations to a foreign country, which Virginia now claims to be.'