Did you ever consider asking the father to take him more or maybe going to court to force him to take custody (the latter is rare but is certainly does happen). Maybe you could ask (or sue) for child support that you could use to hire a nanny and get back some of your life back.