The early history of Hallsands is unknown, but a chapel has existed there since at least 1506. The village was at a cave known as Poke Hole, and probably was not inhabited before 1600. The village grew in size during the 18th and 19th centuries, and by 1891 it had 37 houses, a spring, a public house called the London Inn, and a population of 159. Most residents of Hallsands at that time depended on fishing for a living, particularly crab fishing on the nearby Skerries Bank.
In the 1890s, following a scheme proposed by Sir John Jackson, it was decided to expand the naval dockyard at Keyham, near Plymouth, and dredging began offshore from Hallsands to provide sand and gravel for its construction. Soon, up to 1,600 tons of material was being removed each day, and the level of the beach began to drop, much to the alarm of local residents. The Board of Trade agreed to establish a local inquiry in response to protests from villagers, who feared that the dredging might destabilise the beach and thereby threaten the village. The inquiry found that the activity was not likely to pose a significant threat to the village, so dredging continued. By 1900, however, the level of the beach had started to fall. In 1900's autumn storms, part of the sea wall was washed away. In November 1900, villagers petitioned their Member of Parliament complaining of damage to their houses, and in March 1901 Kingsbridge Rural District Council wrote to the Board of Trade complaining of damage to the road. In September 1901 a new Board of Trade inspector concluded that further severe storms could cause serious damage and recommended that dredging be stopped. On 8 January 1902 the dredging licence was revoked. During 1902 the level of the beach recovered, but 1902 winter brought more storms and damage.
On 26 January 1917, a combination of easterly gales and exceptionally high tides breached Hallsands' defences, and by the end of that year only one house remained habitable. The villagers' fight for compensation took seven years.
According to Pathe News newsreel footage from 1960, the last inhabitant of the village was Mrs. Elizabeth Prettejohn.
The site of the old village at South Hallsands, is closed, although South Hams District Council has built a viewing platform, which is accessed from Trout's Apartments (formerly Trout's Hotel) in South Hallsands. Two houses remain intact, but in summer the owners spend much time repairing the damage that the easterly winds have caused over the winter.
In May 2012, an access road, viewing platform and two houses were affected by a 200 tonne landslide, leading to the houses being evacuated and the affected area cornered off by police. There was further damage to remaining houses at the site.
The beach at North Hallsands, known locally at the time as "Greenstraight", was for 99 years the only one remaining at Hallsands, as the beach beside the village no longer exists, having been removed by the 1917 storm. In 2016 this beach was reported to have been washed away by storms, leaving only a peat underlayer; there are no plans to restore the sand or protect houses in the village.
In 2006, the opera company 'Streetwise Opera' commissioned a new opera, Whirlwind, based on the story of Hallsands. It was written by Will Todd and Ben Dunwell and premièred at The Sage Gateshead, on 24 October 2006.
On 15 November 2010, Damon Albarn revealed on BBC Radio 4's Today Programme that Hallsands is the beach which inspired him and Jamie Hewlett to produce the Gorillaz Plastic Beach LP.
In the Julien Temple-directed documentary on Keith Richards, Keith Richards - The Origin of the Species, Richards told how he used to holiday as a child with his family at Hallsands.
The 1964 film The System directed by Michael Winner, starring Oliver Reed and Jane Merrow filmed scenes with the two actors at the ruins.