Asylum seekers have been 'harassed' while attending church, a cabinet meeting heard.