Perhaps Sis. Bezas will respond with a more insightful answer but I can make a suggestion: Have a talk with your visiting teacher as well as the bishop. They would like to know how you feel because your home visits are important. And it's important to have a spiritual moment during these visits. They aren't merely social calls. Sometimes, these visits are the only spiritual moments people have during their entire months because either they are inactive or homebound.
As a new member, you are still vulnerable to so many things that could sideline your spiritual connection to the church. This is one of them. It's too easy to get both disillusioned, discouraged when you see longtime members make mistakes that shake your testimony.
If you talk to the bishop and your visiting teacher, they'll be understanding. But you must be honest! Others to confide in include the missionaries who still should be visiting you on occasion.
The only time a home/visiting teacher should bring children is when the offspring are old enough to learn from and/or enhance the purpose of the visit. At times, I had a visiting teacher who would bring her young one but he was not a distraction at all. It was a cheery time for me because I had just lost a little boy.
Please, please talk to your bishop and visiting teacher so they can make the necessary changes in your assignments. But you must be straightforward. Explain that you do not find these visits a spiritual treat and you actually dread them. Use those words so there is no confusing how strongly you feel.
The right home teacher is vital. I know. We at times have had absentee teachers during which time my husband and son ended up leaving the church. And now we have an excellent teacher who is their only lifeline to the church. He is a good representative of the Christ and he brings the Spirit with him each month.