I think you meant to say that Mary and children were in the 1880 Grayson Co. census, not 1870, since the family, including George, was in White Co., IN in 1870.
The 1880 census lists children Sally, age 19, b. OH, Attie, age 9, b. IN, Mary, age 7, b. IN, and Elmer, age 2, b. TX. So whatever happened to George must have been between about 1878 and 1880 in Texas. I looked on findagrave.com but didn't find him.