Halloween spirits: literature's haunted houses

From Poe, King and Lovecraft to Woolf, Wharton and even Ali Smith, there's a long list of authors who've lurked in the literary shadow of the haunted house. So what's the explanation for such settings' continued power and popularity?

Horace Walpole's The Castle of Otranto, written in 1764, is often cited as the first gothic novel; an innovative combation of the old (magic, myths and the fantastic) and the new (realistic characters and conversations, described as though they actually took place). Set in a crumbling castle with all the now-classic gothic trappings (secret passageways, bleeding statues, unexplained noises and talking portraits), it introduced the haunted house as a symbol of cultural decay or change.

Homes are supposed to be places of safety and sanctuary; that's what makes it so terrifying when they turn on their tenants. They can also be an effective analogy for the collective fears of a community. Charlotte Riddell's The Uninhabited House (1875), for instance, tells the tale of River Hall, haunted by the phantom of its former owner, who committed suicide in mysterious circumstances. As with all good ghost stories, there are multiple threats and fears in play; it features the Victorian fascination with money and inheritances, and the impact on estates of spooks, ghosts and ghouls. River Hall's rightful owner refuses to live there, but remains haunted by the bleakness of her financial future if no paying tenant can be found.

Charlotte Perkins Gilman's disturbing short story "The Yellow Wallpaper" (1892) explores the idea of a house haunted by its current occupants. A place with "something queer about it", "let so cheaply and left so long untenanted", the isolated setting plays an important role in the narrator's increasing paranoia, hallucinations, and ultimate mental unraveling. In a powerful account of the way women were treated for "hysteria" in the 19th century, the narrator describes signs of violent damage in her room, prompting questions from the reader about who's really responsible.

So what's the source of these unquiet spirits? There are several options, according to Dr Montague in Shirley Jackson's The Haunting of Hill House (1959): "Some houses are born bad … What it was like before, whether its personality was molded by the people who lived here, or the things they did, or whether it was evil from the start, are all questions I cannot answer."

In Susan Hill's The Woman in Black (1983), betrayal and the desire for revenge fuel the terrifying goings-on at Eel Marsh House, along with Victorian-era tensions between past, present and future. This tension is echoed in Sarah Waters' The Little Stranger (2009). Although set in the 1940s, it explores similar themes; postwar anxiety and upheaval, old estates and families meeting financial ruin, progress conflicting with panic. Whatever the source, malevolence abounds at Hundreds Hall, prompting the Ayres family down a path of suspicion, self-doubt and destruction, uncertain whom they can trust.

What's the future of the fictional haunted house? Can contemporary authors continue to innovate?

Mark Z Danielewski's bestselling debut House of Leaves (2000) won international acclaim for its experimental approach, featuring multiple narrators, bizarre footnotes and unconventional page layouts. A manuscript by a blind author, Zampanò, recounts the experiences of the Navidson family. It describes video footage of the Navidsons exploring their continually changing home, with mysteriously appearing additions such as the "Five and a Half Minute Hallway," a never-ending spiral staircase and maze. But the conundrum is compounded further when another character – Johnny Truant – investigates and can find no evidence or history of the Navidsons or their home even existing.

Helen Oyeyemi also experiments with multiple narrators, along with the recurring haunted house themes of personal and cultural displacement, in her 2009 novel White is for Witching, a rich, nightmarish account of familial grief and guilt and its impact on twins Miri and Eliot. As well as being possessed by the ghosts of former family members, The Silver House has a voice of its own: "One evening [Miri] pattered around inside me … she dragged all my windows open, putting her glass down to struggle with the stiffer latches. I cried and cried for an hour."

In South African author Lauren Beukes's genre-bending The Shining Girls (2013), an abandoned building opens onto different times, compelling serial killer Harper Curtis to stalk the "shining girls'" of the title across decades, and acting as both motive and means. "'The House has been waiting for him,' he thinks. 'It called him here for a purpose. It wants to claim the fire in their eyes and snuff it out.'" Carefully paced and cunningly constructed, the non-linear narrative gives a fragmented, kaleidoscopic view of Chicago, Curtis and his victims through the years, portraying violence against women as an omnipresent cultural force.

So if societal fears are what fuel the continued reimaginings of haunted houses, what does the future hold? What current cultural anxieties could soon be incarnated as literature's next brilliantly bloodcurdling haunted house?

And in the meantime: which other haunted houses from literature have left you sleeping with the lights on and the doors double-locked? And is your favourite fear-inducing fictional abode a classic or contemporary take? I'd love your recommendations for my Halloween reading list …