The two adaptations may not seem closely related. This week the film version of Paula Hawkins’s bestseller The Girl on the Train, starring British actor Emily Blunt but transposed from south-east England to the New York commuter belt, was released in the UK. Meanwhile, later this month, a new dramatisation of Wilkie Collins’s The Moonstone, initially published as a weekly serial in 1868, will be screened by the BBC. The Girl on the Train is being trumpeted with all the noise that a major Hollywood studio can make; The Moonstone will be shown on five consecutive days. Yet the books from which they derive are closely connected. Hawkins’s novel is one recent, hugely successful incarnation of a type of fiction that Collins invented. We call it the “psychological thriller”.

Collins is simply one of the most influential of all English novelists. In the 1860s he published his four great “sensation novels” (as they were dubbed by contemporary reviewers): The Woman in White, No Name, Armadale and The Moonstone. Wildly popular for a time, his books were largely scorned as melodramatic crowd-pleasers after his death in 1889, but began to be reassessed in the late 20th century (now almost all of them are in print as “classic” editions). Look closely at his most powerful books and it is extraordinary how many of the features of the modern psychological thriller he foresaw.

The Moonstone is the story of a crime within a family. In the first instance, a famous diamond is stolen during a weekend house party at a country home. In trying to discover what has happened to the jewel, the novel’s characters, including the falsely accused hero Franklin Blake, begin stumbling on the dark secrets at the heart of their respectable social circle. What began as the story of a theft eventually takes in obsessive sexual passion, suicide, embezzlement and murder. Sergeant Cuff, the great detective who is dispatched from London, fails to find the thief but is the catalyst for the uncovering of these secrets. Collins elaborates his plot entirely through the accounts of witnesses and participants. The reader must piece it all together.

Calling a thriller “psychological” credits it with a kind of literary complexity. The very first recorded use of the term “psychological thriller” was in an admiring review of Somerset Maugham’s The Painted Veil in 1925. Most dictionaries of literary terms lack an entry for this genre, as if it were a figment of reviewers’ or publishers’ imaginations. A cynical interpretation would be that it is a thriller that an intelligent person is happy to be seen reading. Hawkins’s novel gained a place on Barack Obama’s summer reading list, thereby endorsed as the thinking person’s page-turner. A couple of years ago it was Gillian Flynn’s Gone Girl, also made into a glossy and violent film, that successfully filled this niche.

It is where genre fiction and literary fiction overlap. One of the books on this year’s Man Booker shortlist, Ottessa Moshfegh’s Eileen, is declared to be a “taut psychological thriller” in a quotation from an admiring reviewer blazoned on its cover. Certainly it displays some of the key features of the psychological thriller. It has a first-person narrator on whom we rely, yet whose view of the world sometimes seems warped. Eileen works as an administrator in a prison for young offenders, whom she likes to study. “The best was when I could see the hard face of a cold-hearted killer breaking through the chubby cheeks and callow softness of youth. That thrilled me.” Like those of a character in a novel by Ruth Rendell (the British doyenne of the psychological thriller), her thought processes are at once logical and twisted. Eileen has a narrative that, we know with delicious apprehension, is heading towards a nasty, probably violent, conclusion. There is already talk of a film adaptation of Moshfegh’s novel. This is hardly surprising: “psychological thriller” has long been a description applied equally to novels, on the one hand, and film or TV dramas, on the other.

In one way these stories are the opposite of the serial killer fantasies with which we have been diverting ourselves for several decades. In the psychological thriller the killer is not a dedicated monster, it is someone close to us, someone familiar. The genre thrives in domestic settings. Tell Amazon that you like “psychological thrillers” and you will be offered novels with titles such as The Woman Next Door by Cass Green, Follow You Home by Mark Edwards or Behind Closed Doors by BA Paris. It is no accident that the settings of both Gone Girl and The Girl on the Train are banal. Gone Girl is mostly located in a numbingly ordinary Missouri town where Nick and Amy move after their careers in New York falter. The Girl on the Train takes place in a fictional commuter town on the rail line from Euston.

Each novel concerns itself with the mysteries within marriages. Indeed, both rely on the unsettling, fascinating idea that a man or a woman might not really know the person to whom he or she is married. They are tales of shocking marital discoveries, seen from within. The trick of Flynn’s novel is to set the husband’s and the wife’s narratives alongside each other. For a long while it looks as though Nick has the secrets and knows better than Amy – but slowly you begin to understand that the reverse is the case. And indeed, that Nick has little idea of the full force of his wife’s insight and willpower.

This discovery of what might thrill us in domestic circumstances should also be credited to Collins. It was Henry James who famously saw his originality in these terms. “To Mr Collins belongs the credit of having introduced into fiction those most mysterious of mysteries, the mysteries which are at our own doors.” The bourgeois household is his stage: no wonder his mysteries turn so often on the testimonies of servants. Terrible things happen in familiar places. In The Woman in White the most frightening location is a comfortable family home in Hampshire, within a few minutes’ walk of a railway station.

In his great novels of the 1860s, Collins invented a series of narrative tricks and peculiar plot elements that thriller writers still draw on. The Girl on the Train depends on the fact that one of its narrators, Rachel, suffers from alcohol-induced memory loss. She struggles through the book to recover the memories that might explain a woman’s mysterious disappearance. She even wonders whether she might somehow be responsible. It was Collins who introduced to the English novel this strangest of possibilities: that a person might not know what they know – might not even know what they have done. In The Moonstone opium removes some characters from conscious control over their actions. In Armadale Lydia Gwilt, the beguiling villainess, swears by it. “Who was the man who invented laudanum? I thank him from the bottom of my heart.” (Laudanum was opium steeped in alcohol.) Collins’s close friend Charles Dickens was so intrigued by the fictional possibilities of opium addiction that he put it at the heart of his own, unfinished psychological thriller, The Mystery of Edwin Drood, in which the frighteningly obsessive choirmaster John Jasper is wedded to the drug.

One returns to Collins’s novels with the thrill of recognition. Even some of his unaccountable quirks have become conventions of thriller writing, though modern practitioners will have been unaware that they were following a pattern set by this unconventional Victorian. So, for instance, Collins liked to use physical disabilities in his mystery stories. The heroine of Hide and Seek (1854) is deaf. It was followed by The Dead Secret (1857), in which the heroine, Rosamond Treverton, marries a blind man. His later novel Poor Miss Finch (1872) features a blind heroine. This exploration of radically restricted perceptions seems to be another version of his interest in altered states of consciousness.

Some of this interest was rooted in experience. Collins himself had become an opium addict as a result of taking large doses of laudanum to relieve his rheumatism. He knew all about the narcotically induced trances of which he wrote. Indeed, it is hard to imagine that his stories could carry such conviction if he had not himself led such an unconventional life. The son of a painter, he came from a bohemian background and remained a sworn enemy to bourgeois conformity. He liked to dress idiosyncratically and he flaunted his Francophilia. He loved French cooking and approved of liberal French attitudes to sexual mores. He never married but had long-term relationships with two women. The first was Caroline Graves, a widow with whom he lived for almost three decades and whose daughter he adopted. The second was Martha Rudd, a servant he met on a holiday in Norfolk, by whom he had three children and whom he maintained in comfort in a separate London household. All were acknowledged and provided for in his will.

Though Collins was the pioneer, there were, of course, other sensation novelists. Writers like Mary Elizabeth Braddon, Charles Reade and Mrs Henry Wood thrilled the Victorian reading public with their tales of bigamy, fraud and sexual passion. Yet Collins was singular not just because he was the first, or because his plots remain the most intriguing, but because he was formally adventurous. Above all, he pioneered the use of multiple narrators. His preface to The Woman in White declared, “An experiment is attempted in this novel, which has not (so far as I know) been hitherto tried in fiction. The story of the book is told throughout by the characters of the book.” Collins made it possible that a narrator might even be a culprit. He showed how to extract thrills from narrative unease. In The Moonstone the characters are like witnesses in a courtroom. They include the garrulous steward, Gabriel Betteredge, the heroine Rachel Verinder’s absurdly moralistic cousin, Miss Clack, and the opium addict Ezra Jennings. Often they have misunderstood what they have seen. No Godlike narrator presides.

Multiple narration is common in the modern psychological thriller. The Girl on the Train has three alternating narrators. Gone Girl shifts back and forth between the narratives of Nick and of Amy. No one can quite be trusted. “I’m a big fan of the lie of omission,” Nick tells us in Gone Girl. A candid admission of deceitfulness, it is the novel’s way of alerting the reader to the need for suspicion. We are a third of the way through Flynn’s novel before Nick happens to mention that he has been conducting a torrid affair with one of his students for the previous year (though the intermittent call alerts of his disposable mobile phone have already made us aware that his narrative was not complete). “Now is the part where I have to tell you I have a mistress and you stop liking me … I have a pretty, young, very young mistress, and her name is Andie. I know. It’s bad.” (Worse in the film, where her age is reduced from 23 to 20.) “My vote for unreliable narrator of the year,” declares one reviewer of The Girl on the Train in a quote reproduced on the cover of the British edition of the novel. A phrase – “unreliable narrator” – coined in the early 1960s by a little known American prof, Wayne C Booth, in a densely academic book, The Rhetoric of Fiction, has become a standard term in book reviews and publishers’ puffs.

Collins liked to experiment with different narrators. His longest and most complicated novel, Armadale, begins as a conventional third-person narrative, but then expands to include letters and first-person testimonies, and finally, grippingly, a diary written by the villain Lydia Gwilt, a formidably intelligent, red-haired governess, a murderer and a blackmailer, who uses her sexual allure to trick men. (One contemporary reviewer called her “Lydia Guilt” by mistake). She is fond of quoting Byron’s poetry and for relaxation plays Beethoven sonatas on the piano. Though ruthless, she finds herself falling in love with one of her intended victims. Eventually, her love for him prevents her gaining the fortune she seeks. She schemes to cheat the novel’s hero out of his inheritance, finally deceiving him into spending the night in a private asylum in Hampstead, where, with the assistance of the corrupt alienist Dr Downward, she attempts to murder him with poison gas as he sleeps. All this would be so much melodrama if we did not have her own account, and in particular the journal in which she records her mental torments as well as her infernal plotting.

The documents that make up Collins’s narratives are not inert records: they become part of the story. In The Woman in White, in a coup de theatre that would be called postmodern if it happened in a novel of the early 21st century, the diary written by the novel’s wonderfully resourceful heroine, Marian Halcombe, which we have been reading for more than a hundred pages, is suddenly interrupted by a “Postscript by a Sincere Friend”. With a chill we realise that the new narrator is the ingenious villain Count Fosco himself, relishing the “unexpected intellectual pleasure” he has taken from reading the journal of the woman who has been trying to frustrate his nefarious schemes against her sister, Laura. Marian has fallen ill and he has purloined her narrative, joining the disconcerted reader in praising her talents and her courage. “Yes! These pages are amazing”, and increase his admiration of “this sublime creature”. “The presentation of my own character is masterly in the extreme.” How he regrets that he and she are foes. “I condole with her on the inevitable failure of every plan that she has formed for her sister’s benefit.”

Flynn, something of a devotee of Victorian fiction, has learned Collins’s trick. In Gone Girl – in a narrative twist that the film version of the story found hard to replicate – the apparently ingenuous diary being written by Amy turns out to be her device. “The diary, yes! We’ll get to my brilliant diary.” We are more than halfway through the book before we discover – one of those overturnings of expectations that Collins relished – that this diary, one of the narrative threads that we have been following, is a fiction Amy is using to incriminate her husband. It has become a familiar trick of literary fiction (Ian McEwan uses it in both Atonement and Sweet Tooth), but is peculiarly suited to the psychological thriller. The firm narrative ground disappears from under our feet. Collins’s interest in unsettling our confidence in a narrative is what makes him seem so modern. One of the first reviewers of Armadale noted shrewdly, “If it were the object of art to make one’s audience uncomfortable, without letting them know why, Mr Wilkie Collins would be beyond all doubt a consummate artist.”

Collins turned the apprehensions of his characters into the stuff of his fiction. “Nervous” is one of his favourite adjectives: characters suffer “nervous restlessness” or “nervous dread” or “violent nervous agitation”. Often their “nerves” are “shaken”; sometimes they are “shattered”. They are sensitised to the terrors of the plots in which they find themselves entangled. Their febrile states still often surface in psychological thrillers, as characters question their own fears and perceptions. Rachel, the main narrator in The Girl on the Train, descends from Collins. A partially recovering alcoholic, she struggles to tell the difference between her memory and her imagination. She is not sure what she saw on the night that a woman disappeared. Much of Hawkins’s novel is narrated in the present tense, enacting the feverishness of its main characters.

All of these are women and it has frequently been noted that contemporary psychological thrillers often feature “difficult” female protagonists. (In an article in the London Review of Books, critic Jacqueline Rose arraigned both Flynn and Hawkins for producing post-feminist fiction that in fact encouraged “hatred of women”.) Collins invariably put women – usually unconventional, difficult women – at the centre of his novels. In No Name, the resourceful Magdalen Vanstone uses disguise and elaborate deceit to try to win back her rightful inheritance. In Armadale the notional protagonist, Alan Armadale, is an impetuous, good‑hearted bore, but the villainess, Gwilt, is constantly surprising and intriguing. Collins felt and wrote passionately about the injustices that women suffered as a result of illiberal marriage and divorce laws, and intolerant attitudes to “fallen women”. After The Moonstone he started using his novels rather too bluntly to campaign on such issues. But he had shown later novelists how to combine narrative suspense with psychological complexity.

His novels have their Victorian preoccupations, naturally. They always depend on arcane legal issues, usually to do with the conditions of wills and rights of inheritance, about which Collins took detailed advice from lawyer friends. He had had a legal training, though he never practised as a lawyer, and was an avid reader of accounts of trials. Each of his novels also satisfied his first readers by achieving something like poetic justice at its conclusion. He knew that the Victorian public demanded happy endings. Yet his readers were also invited to glimpse the possibility that crimes could go unpunished. In a memorable exchange in The Woman in White, Count Fosco undermines the confidence expressed by Marian and Laura that “crimes cause their own detection”. Only if the criminal is a fool, he retorts. The Count strokes his pet mice and laughs “inwardly and silently” at the pious thought that criminal deeds bring retribution. “Ask Coroners who sit at inquests in large towns if that is true, Lady Glyde. Ask secretaries of life-assurance companies if that is true, Miss Halcombe. Read your own public journals.” All around them crimes take place that are not even identified as such – “bodies that are not found” and “wise criminals who escape”. Those crimes are still there for novelists to uncover.

• The Moonstone begins on 30 October on BBC1.