States: Iowa: Howard
Sorry, but I could not locate anything on Joseph Ray or his family at Ancestry.Com
I checked the 1880 Federal Census (Chester township, Howard county, Iowa) and did not find the Ray family living there. They may have been living in an adjoining township though or even in Minnesota. There appears to be several Joseph Ray's living in Iowa during the 1860/1870 Iowa Federal Census - none in Howard county.
Sorry, I couldn't have been of more help. Have you tried the Howard county Genealogical Society?