Mila, I have been researching census records of this area for the last few days working on our SWARTOUT ancestry and I believe there is good evidence that this family group is the family of James SWARTOUT, b. 10 Oct 1792 in Fishkill, Dutchess Co., New York. He and his brothers can all be found together on the 1840 U. S. Census and this family group matches up very well. In addition, James' wife was believed to be Hannah EVERY, so that fits as well. Presumably, James passed away before the 1850 U. S. Census was taken.
As to the specific name you tried to read as Charles, it is very, very hard to decipher. I listed it as "Shuler," but it is very hard to say.
If anyone reads this post and can tell me where this family went after the 1850 U. S. Census, I would be very interested to hear. They appear to have left the area.