I've been working from home for at least four years, and the hardest part for me is the lack of coworkers. I just plain get lonely.
The time management issues and distractions -- not so much. Before I worked for myself, I was a freelancer, so I would usually organize my own time on my assignments. I would be given an assignment with a deadline and it was left to me to get from point A to point B. That's what I like most about working for myself -- if I want to work 6am to 2pm and then take the rest of the day off, I can. If I desperately need to sleep in until 8am, I can. As long as my work gets done, no one supervises me.
But oh my God, I have no wardrobe! I live in my pyjamas. I have lost the ability to dress myself everyday! LOL