Simon Amstell was in a car on the way to the premiere of his first feature film, and he was feeling anxious. This wasn’t surprising. Anybody might be expected to be nervous screening the first film they’d written and directed, a semi-autobiographical love story about intimacy. But those who have followed Amstell’s career, whether making musicians sweat on Popworld or the stand-up shows where he unpicks his mental health for laughs, might expect another level of anxiety, a kind of high-pitched hum of anxiety from the back seat.

This is a man who lives in a vacuum of discomfort (at an orgy in LA, having cautiously touched a bottom, “so hard it was like a cupboard,” he found himself frustrated by the casual sex and nudity, and really wishing someone was wearing a hat). Now 39, Amstell has always insisted on revealing the truth, however excruciating that may be. His film, titled Benjamin, tells the story of a man, not unlike Amstell, who is premiering his own feature; the first scene they shot was Mark Kermode giving Benjamin’s film a crushing review. So Amstell was in the car, nervous. And he describes what happened next in the wondrous tone of a boy relating a magic trick.

“Having fictionalised this exact event in the film, I thought about how the character Benjamin coped and I was able to get a distance between me and him, the person I used to be.” This was the pleasing result of a life spent making art mined from his own, sometimes unhappy, life. “It was suddenly just so obvious. If I act how he acts in the film I wrote, this won’t be a good night.”

We are in a café near his home in London, drinking mint tea, and he has the startled look of a squirrel one’s trying to lure closer with nuts. “It occurred to me, ‘Maybe, maybe the hard work is done, maybe I get to show the thing I made, and maybe these people are here because they’ve chosen to be here, and I don’t have to be anything other than who I am. I don’t have to be extra interesting; I am enough.’” In the cinema he sat in the middle of the audience, slightly to the left, “Sitting there I realised I could just… enjoy it. So I did.”

This realisation, though, didn’t come easily. As a boy growing up in Essex, Amstell was so shy he yearned for a sister, because then, he thought, he’d at least have someone who’d marry him. He made his stand-up debut when he was 13 and, in Help, his recent memoir, he recalls one specific joke. “You’re born, you go to school, you get a job, you get married, you have children and then you die. What’s the point of that?” Which mirrors uncannily the existential depths of his contemporary stand-up. But yes, it’s too perfect. “I found the VHS tape of the show and the line isn’t there. I think I may have made the memory up.”

Honesty is everything for Amstell, even when it ruins a good story. “When I first got a job in TV at 18,” he tells me, chuckling darkly, “a journalist from the local paper came to my mum’s house and interviewed me. And… he made me more likable. He pretended I’d said: ‘I can’t believe I’ve got this job, it’s my dream come true, I’m pinching myself. And I read it thinking, ‘This is outrageous!’ I’d been planning it since I was 13, it was no shock. It would’ve been shocking if it hadn’t happened!”

He wanted to be on TV in pursuit of acceptance, once removed. In “real” life he couldn’t be honest about who he was: his interest, for example, in the way Leonardo DiCaprio’s hair fell just so. In 2001 he became almost famous presenting Popworld, and then a bit more famous presenting Never Mind the Buzzcocks. He wrote and starred in a sitcom called Grandma’s House, based on his own Jewish family, then wrote Carnage, a mockumentary set in a vegan future. And he found success in America – in one stand-up show he describes walking offstage after appearing on the Late Show with David Letterman, when somebody congratulated him with: “If you’re not happy now, you’ll never be happy.” “I thought to myself,” he said: “‘I will never be happy.’”

His work has shifted and deepened, but all the while he’s been asking questions both of his guests and himself. What was he… for? After coming out to his family (his mother thought he was joking, his father recommended therapy, his uncle gave him a safe-sex manual and his aunt cried for two hours straight), he decided: “The point of me, if I’m here for anything, is to help young gay people know that they’re OK.” And yet, he wasn’t OK, not really. In his 30s he was diagnosed with “classic depression”, which irritated him. “I would have preferred not to have the standard version,” he wrote. “I didn’t think I was depressed, I thought I was profound.”

He started therapy and, after two years, he says, “I went to Peru and drank ayahuasca [a tea made of a plant that contains a natural hallucinogen, used today as a tool for psychological healing] with a shaman, then I came back and told my therapist I was probably done. And she agreed. Yes, she could see that something had happened.” In his stand-up shows, and then in Help, he talked about the first time he went to a rainforest to drink ayahuasca in the same way he talks about the awkwardness of cocktail parties – that is, without pretension, and with wit, despite these being feelings many find difficult to articulate. “I felt for a while that it was so profound an experience that I didn’t want to interfere with it, but then I decided,” he says, sipping his tea, “it was too good a story not to tell.”

When Amstell took ayahuasca, he had a series of experiences that changed him. First, he says, he returned to the womb and forgave himself, and emerged as “the person I’d hoped I might become, beyond my ‘temporary personality before I get to the good one’,” then the rainforest told him how to poo, and also, for the first time, he learned to feel.

Much of which had a clear effect on the work he made next. Benjamin (the title character played by Colin Morgan) is, Amstell says, “someone who’s so defended he’s unable to love and be loved. I didn’t know the film would be about this when I started writing it, but I discovered it after having enough therapy.” Morgan’s character is a mournful filmmaker who speaks with Amstell’s voice. Benjamin would be happy, he says, just making one great film, but, “ideally I’d have made that film and died”. His producer eases his anxiety by reminding him that it doesn’t matter what people think, because everybody who sees his film is going to die at some point anyway. And then Benjamin falls in love (Amstell himself has been in a relationship for six years) and has to learn how to stop talking long enough to experience intimacy.

“Making this film has been very… healing,” says Amstell. Though his language might come across in print as earnest, even Goop-y, the silky ease with which he discusses such topics as mortality and love means later I will walk home slightly lighter, as if I’ve accidentally overheard the meaning of life. He talks like this: 1) nervous laughter, leaning back as if to see objectively, in order to preempt the effect of saying, 2) something extremely soulful and profound, about the trouble with being human, before, 3) his voice rises to a friendly shriek at its absurdity, its impossible truth.

“The problem,” he continues, “is when all this stuff is in my head and I feel alone with these thoughts and ashamed of who I am; the solution seems to be to get it out on to paper, and now, into actors who can express it. First of all it organises all the thoughts so I can see them. Then I see how funny the thoughts are, that they’re the thoughts of a ridiculous person, not the thoughts of a wise man who understands the world.” After months of writing, rehearsing, shooting and editing, by the time Benjamin was finished, he says: “It doesn’t feel like my problem any more.It feels like I’m telling the story of someone I used to be, someone I’ve expunged from myself. Though, there are still moments when I feel…” He searches, “Compassionate? ‘This poor guy!,’ I go in the edit, ‘What he had to deal with.’ And what the people around him had to deal with!” He shrieks.

Did the ayahuasca cure him of his… “Personality? Maybe! If I feel any of that anxiety or depression approaching, I’m aware that it isn’t me. I can see it, and it’s helpful, having written it out of me in various different forms. Each time,” he says, thoughtfully, “that’s what happens – I write or perform a version of myself that’s causing me anxiety, and then I’m able to literally see on a big screen who I used to be and decide whether I want to carry on being him or if I’m OK with letting him go.”

The thing is, he very much is a wise man who understands the world. It’s most evident today in how he discusses his father, the man we met earlier telling him to have gay conversion therapy. He politely turned down invitations to his son’s stand-up shows and, after a time, Amstell found it easier (he wrote) to, “Stop thinking of him as my ‘father’ and start thinking of him as ‘the man who ejaculated.’” But the night before we met, Amstell received a rare text from him – he’d read the book. “There’s been a lot of awkward communication, and it’s been quite helpful for me writing a chapter called ‘Daddy’. I felt I’d dealt with it all and on my own.” Which meant he’d separated his father from what he calls his father’s “defence mechanisms… the things he has unconsciously attached himself to in order to cope with being alive.”

He pauses for a second. “I was listening to a podcast with Dr Gabor Maté about addiction, and when people come to him for treatment, the first thing he says is: ‘Well done for finding heroin, well done for finding something that would ease your pain for a while.’” To wish that his dad had been a better father, to wish the past was different, Amstell says, “is just another form of self-hate”.

Except, then his dad texted, “And he’s saying everything I thought I needed.” He sighs, theatrically. “It’ll be easier now, probably – we won’t be trying to fix each other. Enough time has gone by that we’ll be able to laugh about who we were – we’re not there anymore, in that emotional wrestling match, in the mud. Getting the text didn’t initially touch me deeply. But this morning as I was replying, the opening up of communication gave me a feeling of lightness that I thought I’d already achieved.”

Why did his dad get in touch now? The book came out 18 months ago. His eyes widen. “Oh God, he was waiting for the paperback, wasn’t he?” He hoots with laughter.

Every morning, he puts loud music on his wireless headphones, and leaps around his flat. His boyfriend will be meditating in the bedroom, and Amstell will be jumping, springing from the furniture. “I don’t make too much of a racket,” he insists. “They’re very beautiful, balletic movements. I started after Peru – something had happened where I felt more in my body. In one of the ceremonies I ended up becoming a cat, looking around the room and growling, and I wanted to keep that going.” When he’s finished leaping, he has some porridge, and then he writes three pages of stream-of-consciousness nonsense and, after he’s showered, sits down to work on his new script. What will this next film be about, I ask, and he pauses for a moment while he waits for the word to come. “I think… freedom?”

Benjamin is in cinemas now

Simon Amstell’s styling by Tanja Martin, assisted by Lauren Mitchell; grooming by Kristopher Smith at Terri Manduca using Bumble and Bumble and Bobbi Brown