A family has been threatened with eviction after complaints that their baby was crying.