“When you describe it like that,” says Marshall Brickman, “it sounds like I’ve never been able to stick with anything I like!” I had given Brickman a quick run-through of his career highs, from scoring hits with folk band the Tarriers in the 60s, moving into comedy to become head writer on The Tonight Show with Johnny Carson, then winning an Oscar as co-writer of Annie Hall in the 70s, followed by a Tony in 2006 for co-writing the musical Jersey Boys. “My life,” he says, “is no example of how to plan a creative life whatsoever. My only philosophy is that I pick projects where I don’t mind having lunch with the people.”

When he was first approached to write Jersey Boys, based on the life of Frankie Valli and the Four Seasons, Brickman turned it down. He only changed his mind when he met founding member Bob Gaudio, who told him about their colourful lives, which featured petty crime and encounters with the m afia. “They regaled me with all these stories, many of which were included in the libretto – and then I listened to their music.”

Jersey Boys opened on Broadway in 2005 and won four Tonys, including best musical. It has been seen by more than 65 million people worldwide – and this week makes a return to the London stage. After its success, it was even adapted into a film directed Clint Eastwood. “Can I take the fifth on that?” he says when asked what he made of the result.

Brickman was born in Rio de Janeiro but grew up in New York. His parents were leftist activists – Paul Robeson sang at a fundraiser at their home – and the young Brickman became entranced by folk music. “It was like an ancillary connection to the progressive movement,” he says, speaking via Zoom from his Manhattan apartment. He says when he first “heard the banjo, it made me levitate. There was something compelling to it.”

Brickman’s banjo-playing, and his parents’ political associations, led him to be invited to Moscow as a 16-year-old in 1958. “I played on the stage of the Bolshoi,” he recalls. “I won a gold medal in the international talent competition. I think it’s in my closet.” The Russians wanted him to tour China but his parents said he had to return to school. Brickman ended up studying in Wisconsin, where he quickly became part of the folk scene. “Our apartment was the centre of folk music,” he says, “and people who passed through would stay with us.”

In January 1961 a young folk singer, on his way to visit New York for the first time, crashed at Brickman’s apartment. “He showed up in a suit and tie and said he was going to go to New York and become bigger than Elvis.” The young singer, who called himself Bob Dylan, made his New York debut days later and signed with Columbia soon after.

Brickman, meanwhile, joined the Tarriers, a band that also boasted Alan Arkin in its lineup. They were managed by the legendary Jack Rollins, who also managed Dick Cavett, a writer on The Tonight Show with Johnny Carson. Cavett wanted to quit, to try his hand at standup, and Brickman was offered a writing job. “On my third day,” he recalls, “the head writer quit and said, ‘You’re now the head writer.’ He handed me his joke file and a box of cigars, which you had to have as a writer.”

Brickman was head writer for three years, starting in 1967. “It was a highly tense, highly sexualised environment,” he says. “There was a lot of stuff going on. Somebody should some day do a movie – probably for the Hustler channel.”

Four times a year, The Tonight Show would record from California. When Brickman was in Los Angles, he would hang out with John Phillips. Brickman had been in a folk band called the New Journeymen with Phillips before the latter formed the Mamas and the Papas. “It was such a drugged-up scene,” he says. “John was the pope of drugs. You’d go to his house and it was like something out of Coleridge. There was John and his wife Michelle sitting on enormous cushions with these strange little waifs floating around. There was a big pile of mescaline on the table and, before you knew it, you didn’t know where you were.”

One night in the summer of 1969, Brickman got a call from Phillips to say there was a party at a friend’s home and did he want to come. Brickman passed up the invitation. “I’m a Jew from New York and I didn’t like to feel out of control,” he says. “And I had to get up early and work.” It was only the following day he learnt that five people, including the actress Sharon Tate, had been murdered at the party under the direction of the cult leader Charles Manson.

Rollins also managed a young New York standup who would open for the Tarriers when they were in the Big Apple. “I would stand at the back of The Bitter End and listen to this guy who was getting no laughs,” he says. “He was using material that was totally fresh and weird. It was like discovering a great author you never knew existed.” Rollins suggested that Brickman and the standup, called Woody Allen, work on material. “I would go over to his house and we would write jokes,” he says. “His housekeeper would bring out a tuna fish sandwich for each of us.”

Allen and Brickman moved on to films, collaborating on Sleeper in 1973, then later on Annie Hall and Manhattan. “With Annie Hall, we were trying to show off, trying to prove how clever we were. It’s a slice of what life was like at a particular time and place, but it can’t give you any clues as to how to write a movie. There were endless re-shoots and the first cut was two hours 40 minutes.” Still, Annie Hall went on to win four Oscars including best original screenplay, which Brickman accepted as Allen always refuses to attend. “After we won the Oscar, I became bankable. The spill light off the Oscar lasts a long time.”

Given the allegations against Allen, which he has always denied, what would Brickman say to anyone who felt uncomfortable watching one of his movies now? “They’re fools because Woody didn’t do anything,” he says. “I think the hatchet job they did on HBO is nothing to be proud of. I’m just waiting for the backlash. Sometimes, there will be a piece in one of the papers and they’ll put Harvey Weinstein and Bill Cosby and Woody Allen in the same sentence. I think that’s terrible. I think that this cancel culture has ruined too many lives already.”

I tell Brickman I have always loved Manhattan but since reading that Allen, when he was 41, had dated a 16-year-old aspiring model called Babi Christina Engelhardt, it had made watching the film – which stars Allen as a middle-aged man in a relationship with a teenage girl – problematic for me. “Yes, I think he was seeing someone,” he says, adding that it was “consensual”. He goes on: “Does an artist get a pass on things that people who aren’t artists aren’t allowed? I can’t answer that. But I’m the wrong person to ask. I love Woody.”

Brickman and Allen still take walks through the city. “Every once in a while, we’ll hook up and talk,” he says. “Many years ago, we would walk through the park and see two old guys sitting on a bench eating a sandwich. Now we walk through and say, ‘That’s us.’”

Brickman says he would like to write with Allen again. “It would be fun, just for the process,” he says, but in the meantime he remains, even at 82, busy. His fothcoming projects include a musical based on the life of Roy Rogers, king of the cowboys. “My early training was in television and you had to deliver something every night,” he says. “That was the ethic I was educated on. And what else would I be doing? When you enjoy it, why would you not do it?”

• Jersey Boys begins performances at the Trafalgar theatre, London on 28 July.