"There were still a lot of formative years there, (when) having three older brothers who were really good role models was really beneficial. They were achievers. … I felt like I had a lot to live up to."
Swofford cleared the bar his siblings set. He served as president of the student body at Wilkes Central High, the fourth Swofford brother elected, and starred in football, basketball and baseball.
"Johnny, he was the (family's) best all-around athlete," Jim says.
Swofford grew up a Duke fan, cheering for Jim at Wallace Wade Stadium and for the Blue Devils' basketball team when it played at nearby Wake Forest. A spinoff of the Southern Conference, the ACC was in its infancy, a cozy, eight-member league stretching from Maryland to South Carolina.
"College athletics, to me, was the Atlantic Coast Conference," Swofford says.
He soon joined the ACC family, attending North Carolina on a Morehead and playing quarterback and defensive back for Bill Dooley's Tar Heels.
Mononucleosis shelved Swofford for a season, and injuries interrupted others. He started at quarterback for portions of his sophomore and junior years, but in a 1970 loss to South Carolina threw three interceptions.
"The next Monday," he says, pausing for effect, "I was a defensive back."
Swofford was relegated to special teams as a senior in 1971, but the Tar Heels won the ACC and played in the Gator Bowl.
The yin and yang of his football career "really did give me an appreciation for the whole experience that college athletes go through," Swofford says, "both losing and winning, and playing and not playing, being hurt and unable to play. For what I've ended up doing, career-wise, it was a great experience to have because it was all over the map."
Swofford's college journey included a 1969 appearance on The Ed Sullivan Show, a week after he threw three touchdown passes in a victory over Vanderbilt. An entertainment kingmaker, Sullivan was showcasing Oliver, a singer whose singles "Good Morning Starshine" and "Jean" had gone gold.
Oliver was John's brother Bill, using his middle name for the stage.
"Johnny, stand up and take a bow, would you please?" Sullivan said during the show.
Dressed in a gray, double-breasted leisure suit, Swofford obliged.
During Swofford's redshirt freshman year at North Carolina, the school hired Homer Rice as athletic director, and after the 1968 season, Rice spoke at the football team's banquet.
"He talked about the importance of goal setting, attitude and positive thinking," Swofford says. "I'd never really heard anybody talk about those things in the way he did."
Almost immediately, Swofford began plotting a career in sports administration. He courted Rice as a mentor and majored in industrial relations.
"Basically, that was a business degree without accounting," Swofford says, "and I was very happy to avoid accounting."
"You could tell from that twinkle in his eye, whatever you want to call it, that he had that charisma thing," Rice says. "You just knew, this guy's going to make it. He's going to do whatever he wants to do.