Residents of four tower blocks in Peckham who have been told they will need to move out temporarily claim their safety concerns were "ignored for decades".