Vampire in Brooklyn, Thinner, The Dentist, Leprechaun 3 … the horror genre in the mid-90s was terrifying for all the wrong reasons. It was barely even a thing, at least outside of the very bottom shelf of Blockbuster, a place where kids would awkwardly hover before begging parents to let them watch some film about an evil laundry-folding machine.

But then Wes Craven, whose career had wound up in an equally dread-filled place, made Scream, a slasher film that had the balls to kill its biggest star in the opening scene while ridiculing the very genre it was trying to resurrect. It was a phenomenon, both at the box office and with critics, yet 21 years later, masked killers have been replaced with evil clowns, evil dolls and evil racists. Scream brought back the horror genre, yet ultimately killed the slasher film once again.

Last weekend, though, represented something of a last-minute gotcha moment, the bogeyman knifing his way back to life with the shock success of the high-concept slasher Happy Death Day. Box office pundits jumped out of their seats when the star-free, $4.8m budget film made $26.5m in its debut, only $5m less than Blade Runner 2049 the weekend before, a big star sequel that carried a $150m price tag.

Happy Death Day boasted a juicy premise, crudely described as Groundhog Day meets Scream (a college student is stuck in a time-loop that sees her murdered again and again), and arrived in a year that’s seen horror films rise to the top of the box office (It, Get Out, Split and Annabelle: Creation are all bona fide smashes). But it still posed something of a risk. A slasher film hasn’t made any real money since Texas Chainsaw 3D in 2013, and even then, it was hardly a seat-filler, with a final gross just $7m more than Happy Death Day’s opening weekend; plus, it benefited from brand awareness (something that’s died down enough to mean the latest chapter, Leatherface, has premiered on DirecTV). It also arrived after a number of major flops, including 2009’s Sorority Row, 2010’s My Soul to Take and 2011’s Scream 4.

The well-trodden formula of pretty young things being ripped apart by a seemingly unstoppable madman gradually became irrelevant even as the horror genre continued to make a profit. But why?

In the late 70s, as critics cooed over a golden age of prestige cinema, John Carpenter was bringing a violent, low-budget revolution to the outskirts, far away from the more socially acceptable films of the time. The minimalist horror film Halloween brought a new subgenre to date night audiences, one that was crafted through films such as Peeping Tom, Psycho and Black Christmas, and the $300k budget hit remains one of the most profitable independent films of all time. Its success led to inevitable sequels and the advent of other slasher franchises in the 80s.

But audiences grew understandably tired of watching Freddy and Jason slice moronic horny teens into pieces and Hollywood appeared equally bored. The formula was brutal and repetitive and existed within a world that was so impossible to identify with that none of the scares were cutting deep anymore. In 1994, after nine Friday the 13ths and six Nightmares on Elm Street, something shocking rose from the graveyard of gristle: an original idea.

Before the word meta had become commonplace, Craven hatched a nifty way to breathe new life into the Elm Street series. In his New Nightmare, the maniacal pedophile Freddy Krueger crossed over from the world of fiction to reality and haunted the actors from the original film as well as Craven himself. It was a daring idea and one that audiences were unprepared for (it was the poorest-performing Elm Street film at the box office) but the director was convinced he was on to something.

It was an under-appreciated forefather of 1996’s Scream, a dazzling, witty game-changer that added a much-needed layer of believability to what had become an easily parodied subgenre. For once, thanks to Kevin Williamson’s smart, biting script, the soon-to-be-butchered teens were as cine-literate as they were attractive, calling out slasher cliches and reawakening studios to the idea of horror as profitable. Like Halloween in 1978, it opened the floodgates to more, from Urban Legend to Valentine to I Know What You Did Last Summer to three more Screams to the inevitable remakes of Prom Night, Friday the 13th and, yep, Halloween.

But fatigue set in as creativity became scarce and the cycle drew to a close, with Japanese horror remakes (The Ring, The Grudge, One Missed Call), torture porn (Saw, Hostel, The Human Centipede), zombie films (Dawn of the Dead, 28 Days Later, World War Z) and found footage frights (Paranormal Activity, The Blair Witch Project, Cloverfield) all taking over. Even the return of the surviving cast of Scream, in the failed franchise-reviver Scream 4, couldn’t lure teens to the multiplex, the self-referential whodunnit no longer seeming quite as fresh.

Yet as all of those trends have similarly died, a gap has now emerged in the market. Later this month, the grisly reboot Jigsaw will dust off the rusty blades of the Saw franchise while the surprise success of Happy Death Day suggests that less torturous pleasures might again have their day. It’s landed in cinemas soon after news that Jamie Lee Curtis is returning to the Halloween franchise, a surprise for many reasons, not least the fact that her character was killed off in 2002’s painfully misjudged Halloween: Resurrection. The reboot, which will cannily ignore the majority of existing Halloween sequels, comes from the Pineapple Express director, David Gordon Green, and the Eastbound & Down star Danny McBride, two figures not associated with horror. But it will be overseen by Jason Blum, one of the most powerful producers working in Hollywood right now. His company Blumhouse has been behind the Purge and Insidious franchises as well as Get Out, Split and Happy Death Day.

The film lands next Halloween, months after the Scream franchise attempts another desperate stab for attention. After the fourth film failed to relaunch the series, MTV nabbed a small screen reboot that quite successfully stretched the formula over two seasons. The third will relaunch the story yet again, this time with Queen Latifah as producer and a predominantly black cast, including rapper Tyga, itself a deviation from a subgenre that has historically killed off black characters first. We can also expect a Happy Death Day sequel, with the director Christopher Landon already developing a follow-up.

This year has seen It become the biggest horror film of all time, making $604m worldwide, while Get Out could become one of the rare examples of the genre to gain Oscar nominations. So we can expect horror to thrive more than ever in the next few years.

Whether the slasher film can truly be part of this is still up for debate. There’s something so unsophisticated about a person in a mask stabbing teens, especially for a younger audience dazzled by effects-heavy haunted house films and less willing to head to the cinema in the first place.

Happy Death Day succeeded with its Groundhog Day gimmick (its audience was 55% under 25), a supernatural spin elevating it out of well-mined territory. Otherwise, would it have landed quite as well? Teen movies in general are no longer a box office draw. They’re rarely released in cinemas and when they are, even great reviews can’t propel them to success; last year’s Edge of Seventeen had a 95% Rotten Tomatoes rating yet couldn’t even crack $15m at the box office.

As with Scream in 1996, a true genre resurgence will only stick with ingenuity and skill. There’s a large audience for horror – but there’s a reason why audiences were drawn to the well-reviewed and well-marketed It and then failed to show up for the poorly reviewed and poorly marketed Flatliners just three weeks later. Not even Michael Myers can kill bad buzz.

• Happy Death Day is out in US cinemas now and in the UK on 20 October