The man tucked away in a smart office in the gigantic New York film studio complex does not look like someone who has hit the big time. He wears a blue beanie hat and sips cautiously on a cup of coffee through a close-cut golden beard. His face bears an expression similar to a rabbit transfixed by the lights of an oncoming car: a mix of fear, surprise and the certainty of onrushing doom. "I don't really recognise success," he says quietly and slowly. "I don't see myself as on an upwardly mobile trajectory. I see myself as on the edge of a cliff about to fall off."

Mainstream recognition has been a long time coming for 46-year-old New Yorker Jonathan Ames. For two decades he has been a fixture of the downtown semi-underground literary scene, carving a name for himself with wildly confessional essays. One of his most famous has the title, "I Shit My Pants in the South of France", the subject matter of which is self-explanatory, and he once had a one-man off-off Broadway show called Oedipussy. He has been the archetype of a certain kind of outrageous New York artist who puts their extremist life on full display to the world.

So the question is why does Ames now have an office in a television studio, two assistants and a fridge full of free drinks? The answer is the biggest twist yet: he is now the creator of a successful HBO television show in America called Bored to Death, which is about to cross the Atlantic. The show is an authentic Ames creation: a hard-boiled detective story surreally set amid the literary and hipster crowd of Brooklyn and Manhattan. Its basic premise follows a down-on-his luck writer – called, naturally, Jonathan Ames – who, in a fit of despair after a break-up, advertises his services as a private eye on Craigslist. Jason Schwartzman is Ames, with comic star Zach Galifianakis as his best friend Ray and a hilarious turn from Ted Danson as an ageing editor called George, who struggles manfully against boredom, herpes outbreaks and a desire for marijuana.

It also has a script that crackles with wit and somehow makes the whole thing believable and immensely enjoyable. Ames, the once-lonely scribe working by himself in his ratty apartment, now talks fondly of the communal creative process of TV and how he has bonded with the huge crew and cast over the endless meetings and long days. "You work together every day for 14 hours and I think it speaks to probably some old anthropological feeling inside of when you used to live in a small town and you saw all the same people all the time. It is very lovely," he says of his new TV family. But still the doubts remain. None of the trappings of success seem to relieve Ames of his angst. "Privately I can still feel wounded, feel terrible and hate myself and think: 'God, the emperor has no clothes!'" he says. Then he swiftly adds: "Not that I am an emperor. The pauper has no clothes!"

In some people such displays of insecurity could come across as an annoying affectation. But not with Ames. His writing career and performance art has managed a rare trick of putting out information that most people would blanche at and yet somehow also making the reader empathise with him. They cover things like an encounter with a transvestite (specifically, smoking crack with one on Christmas Day), his first successful experience with masturbation (he was so pleased he immediately told his mother), his worries over going bald, his fears of sexual inadequacy, his battles with booze and his bizarre life with a coterie of decadent downtown artist friends.

In essay after essay Ames made a literary splash by putting his crippling anxieties and sordid adventures out into the world. He is often compared to an X-rated version of the acclaimed essayist David Sedaris. But his persona is more like a Woody Allen figure, if Allen was less concerned about his therapist and more worried about sexual diseases, the risks of alcoholism and prostitutes. Ames certainly took to heart Socrates's famous exhortation that the "unexamined life is not worth living" – though the Greek sage probably did not have in mind Ames's detailed descriptions of getting an enema. Or trying, and failing, to attend a West Village orgy.

Even in Bored To Death Ames takes thinly veiled inspiration from his real life and familiar themes of sex and self-doubt crop up. In the first episode Ames (the character) is surprised when a pair of burly Israeli movers help his girlfriend shift all her belongings from his apartment. Disgusted at Ames's shock at their masculinity, one of them asks him: "What are you, another self-hating New York Jew?" Ames's character replies earnestly: "Yes, I am."

In later episodes Ames and Ray chat on the subway and head into typical Amesian territory as Ames wonders who – if they were gay lovers – would be on the top or bottom. Ames's own conflicts with booze also arise. In the first episode his girlfriend, played by Olivia Thirlby, states she is leaving him because he failed to keep his promise to give up drugs and booze. Ames's character protests: "It's dangerous to go cold turkey! I'm down to white wine!"

It is tempting to look for demons and traumatic experiences in Ames's past, but very little in his early life explains how he came to be who he is. He was born in Oakland, New Jersey, in 1964 to middle-class parents. His childhood was full of creativity (his mother wrote poetry) and he developed a deep love of books and journalism. Success even came early as he published his first novel, I Pass Like Night, in 1989 at the age of 25. But it was hardly a meteoric rise. For most of the next two decades he lived the hand-to-mouth existence of a struggling New York writer and performer, including a couple of years driving a cab.

Even as his name became a fixture of the New York arts scene his career did not translate into money. Not that Ames cares. He is the first to brush off accusations that his poverty meant anything profound. "Even when I was living below the poverty line as a novelist I was still living better than 99.5% of the human population of the world," he says. "But in my little, soft realm of trying to amuse a few dozen middle-class people with my books and articles, I did struggle to survive in my own way."

But amuse them he did. Whether it was in his infamous City Slicker column for the alternative New York Press newspaper, the famous Moth live poetry readings or his notorious boxing matches – where he went by the name of the Herring Wonder and had his nose broken (it's still crooked) – Ames became a household name for bohemian New Yorkers.

Long before the internet allowed everyone to overshare on Facebook, Ames carved out an outrageous niche by detailing his life. But it came at a price. He admits his past writings now cause him current problems either by offending those close to him or by making people wonder what sort of person he really is. "You get tired of exposing yourself and having people wonder about you… sometimes it doesn't feel honest in retrospect, because we change so much," he admits. "That's part of my problem. Someone will read something I wrote 10 years ago and to them its very resonant. They think: 'This is how he is right now.' But I've changed so much. Honesty ends up not having a shelf life."

It's a profound reflection, which, too, is typical Ames. On Bored to Death, Ted Danson's character suddenly reflects on the impossibility of ever truly loving someone or seeing something from someone else's point of view. "I am in your movie. You are in mine," he tells Ames (the character). "Two different films, really. We don't really know each other. We just make a guess at knowing each other, right? I think the same is true about love."

That speech echoes things Ames (the real person) often thinks about. Pressed on why he does not think his early work is honest any more, he reflects sadly on the elusive subject of reality and relationships. "Because of the nature of language there was no honesty, really, in my non-fiction. I mean there was a simulacrum of honesty. It is the limitation of language and not being able to express everything and all the complexities of any single encounter with another person," he says.

For all the shocking stuff Ames has put out into the world, the odd truth is that he is a gentle soul who now appears desperate for a bit of privacy. He has, according to many reports, recently been dating the singer-songwriter Fiona Apple. But Ames, who has gleefully described encounters with prostitutes, clams up at the mention of their current status. He looks away and quietly says: "I don't want to be precious but I don't want to talk about those kind of things. I'd feel funny about that." He says the words with such pain it is impossible to tell if he just wants to keep things out of the public gaze or whether he is dealing with the aftermath of a break-up. It is an uncomfortable moment.

Ames has left the life of a starving artist behind, but he comes across as stunned rather than happy. He has a degree of wealth and has won mainstream acclaim that has seen him break out of downtown Manhattan. "More people watch a single episode of my show than have read all my books," he says. But it also leaves him hankering, just slightly, after the old days. "I am somehow the boss of this TV show. But the show is the boss of me, really. I've got to be somewhere every day now and I became a novelist because I didn't like to be anywhere. I didn't like having a boss. I was my own boss and I didn't even like him. I was always napping and procrastinating."

But he has no choice now. Instead, he has plans. He is mulling over the idea of a screenplay for a movie adaptation of one of his novels and wondering if he could direct it. He is thinking of taking his love of noir detective fiction a step beyond Bored to Death and writing some genre novels. He is inundated with offers, but still he remains immovably sceptical. "I have a lot of low self-esteem in the bank that I can continue to work off for a while. I imagine I will continue to screw up in new ways."

Maybe that is his secret. Ames's insecurity and fears of failure are what fuels him. Just as they did when he was poor. What would Ames do if ever he felt confident about his achievements? "I don't know what it would be like to be that kind of person," he muses. "I wonder what it is like. But then I wonder what would push you to keep making things..." Perhaps, then, we should selfishly hope Ames never gains the confidence of his convictions.

Bored to Death starts on Sky Atlantic on Monday 28 March at 10.15pm

Paul Harris is US correspondent for the Observer and the Guardian