Harold, this is what I show for James Cady's parents.
305. STEPHEN5 CHAFFEE (JAMES4, JOSEPH3, JOSEPH2, THOMAS1) was born in WOODSTOCK, CON 9/25/1773. STEPHEN died 2/6/1841
SPRINGVILLE, NY, at 67 years of age.
He married RUTH UNDERWOOD in WOODSTOCK, CON, 1795. RUTH was born 11/17/1774. RUTH was the daughter of KINGSLEY
UNDERWOOD. RUTH died 8/8/1840 EVANS, NY, at 65 years of age. Her body was interred EVANS, NY.
He was baptized in WOODSTOCK, CON, 11/5/1773. Religion: religion unknown.
STEPHEN CHAFFEE and RUTH UNDERWOOD had the following children:
529 i. LUCY6 CHAFFEE was born 2/17/1797. LUCY died 1/11/1859 at 61 years of age. She married JOHN BUCK.
530 ii. AMASA CHAFFEE was born 11/28/1797. He married LYDIA W. NICHOLS.
531 iii. DAVID KINGSLEY CHAFFEE was born 1/18/1800.
532 iv. JAMES CHAFFEE was born CAZENOVIA, NY 12/26/1801. JAMES died 7/4/1803 at 1 year of age.
533 v. JAMES CADY CHAFFEE was born CAZENOVIA, NY 2/22/1804. He married LUCY A. CHAFFEE.
534 vi. OLIVER BUGBEE CHAFFEE was born CAZENOVIA, NY 4/20/1806. OLIVER died ATTICA, NY.
535 vii. RUBY UNDERWOOD CHAFFEE was born 4/23/1808. She married SYLVESTER BAKER.
536 viii. ALZINA CHAFFEE was born ATTICA, NY 12/18/1810. ALZINA died 2/21/1820 at 9 years of age.
537 ix. AMBROSE WORTHINGTON CHAFFEE was born ATTICA, NY 2/23/1813. He married MRS FLUERY.
It is reasonable from this lineage that James Cady had sons named as you mention. Not having my Chaffee Genealogy book with me, this is all I have at the moment. The book may show the second family and lineage. I'll look.
My mother is a Chaffee. Regards, Chloe