James Fox is telling me how, at the age of 74, he has finally got his confidence back. Not before time. It's half a lifetime since he walked away from acting, at the height of his fame, to become a door-to-door salesman for an obscure Christian movement.

In his new film, A Long Way from Home, he is desperately moving as Joseph, a man who has retired to France with his wife, only to fall for a young woman travelling with her boyfriend. It's his most dominant role since his incendiary Chas in 1970's Performance, but it couldn't be more different: Chas was a terrifying cockney gangster in a movie about decadence; Joseph is a restrained, depressed presence in a film about quiet disappointment. Fox's eyes are hollowed out as he observes the world passing by from a distance. He might be dapper and superior in manner, but he aches with longing.

When I arrive at the bar of the boutique London hotel, he is waiting for me, every bit as dapper as Joseph – beautiful navy blue suit, wonderful head of silver hair, military straight-back – and every bit as diffident. He sips on a glass of water, speaks quietly in short, fractured sentences, and chews on his upper lip and fingers.

Fox has an amazing story to tell. There's the acting dynasty: his grandmother and mother were actors, his father was a theatrical agent, his brother Edward is an actor, his brother Robert a producer, his children Lawrence, Jack and Lydia all actors. There's life as a child star, making his debut at 11 in 1950's The Miniver Story. There's the tremendous success from the early 1960s onwards that resulted in two great films about fractured identity: The Servant with Dirk Bogarde, and Performance with Mick Jagger. And, of course, there is walking away from it all at the age of 30.

"I haven't played this interesting a part for years," he says. He's right. In the 34 years since his comeback, there have been memorable roles – humanist headmaster Cyril Fielding in A Passage to India, Nazi-appeasing Lord Darlington in The Remains of the Day, haughty former spy Anthony Blunt in A Question of Attribution – but not enough. The trouble is, says Fox, when he returned to acting, he didn't believe he was up to the best roles. "I came back as a different man and obviously as a worse actor. I hadn't been practising it for nine or 10 years. And I probably lost touch with who I was in terms of my craft, my identity – and other people's perception of me."

One popular theory about why he quit is that he took so much mescaline making Performance it left his head mashed. He says this isn't true. "I was the old pro around that set. I was completely straight. I've always been that in my work. I want to be on the script, I want to know where were are, that we've rehearsed this."

But after making Performance, everybody says ... Fox gently finishes the sentence for me: "I had a crackup." He smiles. So did he? "I don't know. I would have said I was going through some kind of crisis." There was so much going on, he says, and you sense even now he doesn't know where to start. Yes, he had certainly taken drugs when off duty – but, he protests, "I didn't take that much acid." His father was ill with cancer, he was breaking up with a girlfriend. And yes, he did find it hard to leave Performance behind.

"It was traumatic to make that film. I'm a Dirk Bogarde in that way. He always said he found it very hard to put the clothes back in the trunk." You can understand why. Everything about Performance was shocking: the violence, the sex (as well as an explicit scene with Anita Pallenberg, Fox snogs Jagger), the cross-dressing and its overall trippiness. Just watching it can get you stoned.

On the surface, Fox couldn't have been a more enviable symbol of swinging London: he drove flash cars, was papped at the hippest clubs, had girlfriends who were almost as famous as him. He went out with Sarah Miles, now as famous for drinking her own urine (for health reasons) as for her acting. I ask Fox if he ever had a glass of her pee. He starts to giggle and suddenly his tired face is transformed: he looks so boyish. "No, I didn't. I mean, come on! She only started that caper afterwards. Could you imagine!"

I don't know if it's the way the conversation's turned, but I tell him I fancy a glass of white wine. "Oh yes, I'll have one, too. Thank you." He's talking more easily about the past – how he might have looked the part, but he never felt it."I didn't have enough chutzpah. People like Michael Caine and Terry Stamp were good-looking, but they used to pull with their greater confidence and chat." Despite everything, he felt lonely and lost. "I saw a lot of things I didn't like about myself. I'd been having a great time, living a somewhat louche life. But now I felt I'd had it too easy. Guilt, I suppose. Relationships weren't very meaningful."

So he walked away. His crisis expressed itself in a number of ways. "After Performance, I went and did this incredibly naff thing at Christmas in Blackpool, playing Doctor in the House on stage. That looks like a bit of a breakdown, doesn't it?"

Then he left a profoundly secular life in London for Leeds, to work for the Navigators, an evangelical Christian ministry. He gave up his Lotus for an Austin Maxi and survived off a stipend from the Navigators and savings from his previous career. People must have been terrified when Chas rolled up at the door selling God. Well, Fox says, strictly speaking he wasn't selling God – he was raising funds. Doing what? "I was a telephone-sterilising service salesman." Sexy, I say. "Really sexy. God that was sexy!" He gets the giggles again and does an impression of Chas selling door-to-door: "Fackin buy this phone-sterilising stuff or I'll do ya." By the time the wine arrives, Fox has morphed from melancholic elderly man to giddy schoolboy.

Did people ask if he used to be James Fox when he knocked? "They did, yes. They said, 'What are you doing here?' But it was in Yorkshire and there weren't many film buffs there." After a few months, he started doing outreach work with students on behalf of the Navigators. His mission? "To encourage people to have a personal relationship with God." I bet he was a diffident evangelist. "Probably. That's the worst thing to be – a diffident evangelist! He he he he!"

You sense it's taken him decades to see the funny side. "It wasn't good casting – what I was doing for those nine years." Fox often uses the word casting to describe decisions he's made: as if somebody took them for him. Did nobody take you aside and ask what you were doing with your life? "No, I just didn't have the support group, that community. You need friends. A friend." He never had that? "I had a friend at school, a good schoolfriend. I had girlfriends." What about your family? "No, my family – they were so tolerant. They didn't speak." Is that tolerance or uptightness as in: best not to mention what's happening to James? "You said it. Yes, that English thing. Yes."

It might well have been a decade-long flip, yet much good came out of it. He met his wife, Mary, had five children, rediscovered his values. At the age of 40, he decided to return to acting. But for many years, he found it hard to reconcile his Christianity with his work. While Christianity is about subsuming ego, acting is about giving it free rein. As a result, he says he became a boring, inhibited actor. Did any director say: "James, we want a bit of the old Chas back?"

"No, I didn't work with people who took that much interest. Probably, they thought I was a bit of a weirdo anyway. They were wary of me." Does he think he was a weirdo? "Looking back, yes, I do see the weirdo they saw." It took him a while to realise he could still be a good man and play sinners: he was offered the Jeremy Irons role in The French Lieutenant's Woman, but turned it down for moral reasons.

After he returned to acting, he suffered depression, again struggling with his identity. When I watched him in A Long Way from Home, I worried that he was so convincing because he was playing himself. Yes, he says, there were times when he's felt like that, but not now. He no longer goes to church, or is involved with the formal side of religion, but he still considers himself a Christian – one who's more at ease with himself than ever before.

He talks about the love of his family, the support of his wife, how his children are his best friends. "They are so brilliant, what they've done for my life. They take the piss out of you, they're natural with you, just great. My family is the most grounding influence."

Is he ready to return to the more risque roles of old? Yes, definitely, he says – he's done enough repressed, dislocated poshos. It's time to play a hard man, a character with a touch of evil. He throws back his shoulders, puffs out his chest and, in perfectly menacing cockney, tells me: "I've still got a bit of Chas in me."

• A Long Way from Home is out on Friday.