A joint letter from Guernsey, Jersey and The Isle of Man has been sent to David Cameron, outlining concerns over Brexit.