“Making this movie is like trying to complete a crossword puzzle while plunging down an elevator shaft.” So runs the epigraph to the second novel from the actor David Thewlis, who explains that these words were “spoken to the author by Marlon Brando, Australia, 1995”, when the two were filming John Frankenheimer’s famously tortuous adaptation of The Island of Dr Moreau. It seems to set the stage for a novel that will do for the movie industry what Thewlis’s satirical debut, The Late Hector Kipling, did for the Young British Artists, but the unsettling narrative that ensues is harder to pin down, folding showbiz send-up and creepy psychodrama into a giggly comedy of errors that rarely feels all that funny.

Jack, an English film director, is consoling himself after the death of his actor wife, Martha, with a silicone cast of her head (left over from when she played Anne Boleyn) and a library of audio clips. He reckons he can go one better by hiring Betty, a stage actor with a gift for mimicry, to make video calls to him while following a strict script under virtual house arrest in the south of France. “It’s the most bonkers thing I’ve ever heard,” her sister says, but Betty, struggling with alcoholism and a violent boyfriend, signs up in the hope that it might be a way to regain custody of her six-year-old son, Freddie.

The action resembles an unlikely splice of an experimental Tom McCarthy novel with a Howard Jacobson farce. While Jack shoots an autobiographical film in England – it involves a blow-by-blow reconstruction of finding his father in flagrante with his teacher – Betty, in France, stumbles upon an old phone with clues to the whereabouts of his mysteriously unmentioned son. “My God, Betty, you are like a detective,” says one of the characters she pumps for information, which you might feel rather betrays Thewlis’s anxiety about how to manage vital exposition. In general, the dialogue – if you’re being generous – would probably sound better performed:

“I quit.”
“Good, cos you’re fired.”
“Good, cos I quit.”
“Good, cos you were shit.”
“Goodbye.”
“Goodbye.”

When Freddie’s father hears coughing, he thinks it’s Betty’s lover; as she points out, it’s her nextdoor neighbour, who has emphysema. “You’re fucking Mrs Mason from next door?” he asks. There’s a strange undercurrent of nastiness to the gags. We see an elderly woman in hospital who “might have died about an hour ago by the looks of things”; a pivotal event doesn’t make front-page news because “some enterprising genius had thrown acid at a TV astrologer”.

Ultimately, you sense Thewlis hasn’t decided what kind of novel he wants to write. While Jack’s assistant says she’s “happy to go along with this caper”, Betty sees his enterprise, perhaps more shrewdly, as an “astonishingly elaborate masturbatory subterfuge, conceived in the mind of an ageing ghoul”. Yet while the book’s core is pitch-dark, involving a second struggling mother, driven to deadly negligence and using status to buy silence, so much of the narrative is played for cheap laughs that the force is lost. That may be the point, but it’s not the least of the novel’s problems that you can never quite tell.

• Shooting Martha is published by W&N (£14.99). To support the Guardian and Observer order your copy at guardianbookshop.com. Delivery charges may apply