New figures suggest more than a fifth of adults are not actively looking for work.