Residents in an area of Taunton, used by travellers, are furious after the council have delayed a decision on the future of the site and the Green Party conference in Bristol.