‘Do you want to see something really scary …” The horror anthology movie is a peculiar – some might think – imperfect beast. It’s been with us for well over 70 years and is close to the heart of many fans, not least my own. In fact, it was one of the key influences on Ghost Stories, which I co-wrote and co-directed with Andy Nyman.

But what makes these pictures so endlessly fascinating and appealing? Though there are examples from all over the world, this body rests on two sturdy British legs. One is the Ealing horror film Dead of Night, released in 1945; the other is the output of a small UK-based studio called Amicus, who were active throughout the 60s and 70s. The Ealing film was more obviously artistic and noble in its intention, while the Amicus titles were – on the surface at least – more shamelessly commercial. And the tussle between those two impulses gives us a big clue as to the enduring appeal of these curious films.

Anthology movies, sometimes known as portmanteau films, predate Dead of Night of course. Hollywood had produced a couple of notable successes in the early 40s. In 1942, 20th Century Fox released Tales of Manhattan, an all-star, multi-story piece about a tailcoat that is passed from one owner to another, while the following year saw Universal’s Flesh and Fantasy, which had a vaguely supernatural air without ever touching on anything horrific. Both films were inspired by the French movie Un Carnet de Bal (1937), directed by Julien Duvivier, who was then hired to work on Flesh and Fantasy.

Ealing Studios picked up the baton in 1944 with The Halfway House, another quasi-supernatural drama in which a miscellaneous group of people find themselves at a mysterious Welsh hotel – unaware it was destroyed by a bomb the previous year. Its success persuaded Ealing boss Michael Balcon to make another film in the same vein the following year. Reflecting on it in his autobiography, Balcon said: “We decided it would be a good idea to make a series of ghost stories joined by a suitable central thread, which would display the all-round talents of the creative team we had built up.” And it’s fair to say they had quite a team. Dead of Night has four directors: Basil Dearden, Charles Crichton, Alberto Cavalcanti and Robert Hamer. Cavalcanti was a feted documentary-maker who had also made the wartime classic Went the Day Well?; Crichton would go on to direct The Lavender Hill Mob and A Fish Called Wanda; Hamer would helm the brilliant Kind Hearts and Coronets, and Dearden gave us The Blue Lamp and The League of Gentlemen (the 1960 black-comedy thriller from which a certain comedy group thieved their name).

The stories in Dead of Night were a combination of adaptations (two from EF Benson, one from HG Wells) and original stories by Ealing staff writers. One of these, the through-line story that binds the others together, is Dead of Night’s secret weapon: it’s the best of the lot. An anxious architect (brilliantly played by an increasingly sweaty Mervyn Johns) is troubled by recurring dreams. His wife recommends he take up an invitation to a country house – only for him to get there and be hit by a terrible sense of deja vu. In an attempt to calm him down, the other guests tell him stories of their own supernatural experiences. But each time we come back to him, things get weirder and weirder. Dead of Night still packs a wallop. There is a third element to the film’s success: it was made by lovers of horror. Cavalcanti in particular was a huge admirer of Val Lewton’s groundbreaking B pictures being produced for RKO at the time – literate supernatural fantasies that relied on suggestion and atmosphere for their scares.

Dead of Night was a moderate success on its release, but it didn’t lead to any revolution in movie-making in the UK. No particular strand of intelligent, well-made supernatural horror films appeared in its wake. In fact, Dead of Night’s real impact was not to be felt for another two decades. Amicus was famously a studio without walls, cobbling together productions on a film-by-film basis. It was founded by two New Yorkers of contrasting temperaments, dynamic grifter Max J Rosenberg and bookish scriptwriter Milton Subotsky, who were drawn to the UK by the Eady Levy, a government sponsored film production scheme. (Matt Hancock, take note.) Their first movie, Dr Terror’s House of Horrors (1965) just happened to be a portmanteau horror film. As well as Dead of Night, it was inspired by the recent success of the Edgar Allan Poe adaptation Tales of Terror and Black Sabbath (adaptations of Chekhov and Tolstoy).

Dr Terror had to make do with Subotsky as its writer, but this was no bad thing. Subotsky was a massive fan of the genre and had a deep love of SF and horror, yet it was Rosenberg’s business smarts that lit the fuse. He realised that an anthology had a cost advantage: a conventional narrative required actors to be booked for several weeks of production and paid accordingly, but each individual story could be shot in a few days. This meant Amicus could afford to punch well above its weight, attracting big marquee turns with a reasonable daily rate, then have their names on the poster to market the movie for a fraction of what it would normally cost to employ them. Christopher Lee and Peter Cushing were snared in this way, and later productions would boast the likes of Jack Palance, Herbert Lom, David Warner and Donald Pleasence.

This ethos extended behind the camera, and Subotsky and Rosenberg thought big when it came to creative talent. Dr Terror director Freddie Francis was an A-list cinematographer who had shot Room at the Top, Sons and Lovers and Innocents. Roy Ward Baker was an industry veteran who’d made the original Titanic drama A Night to Remember. New talent would also be promoted – such as Kevin Connor, a gifted visual stylist who was still in his 30s when he directed From Beyond the Grave. Crucially, the films were made with love and care. If the budgets were low, the aspirations were high, and this attitude paid off on screen. Amicus were also enterprising in their source material, turning to Psycho author Robert Bloch for Torture Garden, British ghost story writer Ronald Chetwynd-Hayes for From Beyond the Grave, and in a particular stroke of genius the infamous EC horror comics of the 50s for Tales from the Crypt and The Vault of Horror.

By 1975, the cycle ground to a halt – along with the rest of the British film industry – after a productive decade. Subotsky attempted a revival with The Monster Club (1981), but the old magic was gone. However, the work done lingered in the imagination of a generation of genre fans, and the form became a fixture that is still with us today. The 80s saw Creepshow, Twilight Zone: The Movie, Cat’s Eye and VHS-only gem Screamtime. In the 90s, we got Tales from the Darkside: The Movie, Body Bag and Tales from the Hood. And it’s still going strong with the V/H/S/ franchise, The ABCs of Death and the soon to be released The Field Guide to Evil.

And the secret of their enduring appeal? For me it’s two-fold. It’s partly to do with that fusion of the highbrow and lowbrow in the best examples of the anthology horror, which is deeply entertaining. But there’s something older, too, something primal lurking in there, a connection to our most ancient form of storytelling. There’s an atavistic yearning deep inside us to be back round that flickering fire, keeping the terrors of the night at bay, as the yarn-spinner holds us bewitched: “You liked that one? Well, I’ve got another for you – but be warned, this one is really scary …”

• Ghost Stories is released on 6 April.