According to Jim S05 E19 Daddy Dearest

Jim believes that Bill, the father of one of Rubys and Gracies classmates is actually his real father that abandoned Jim when he was a child.