For casual wear, a black boot with brown pants is acceptable, particularly with a neutral, light colored blouse or sweater. If you accent with a black purse and other black accessories - black hair fobs with rhinestones, black watch, black bead necklace or earthy black cord choker - you should look great.
For formal attire, always keep with the long line look recommended by Rebekah.
Sure, you can wear whatever is comfortable, but I like to save that look for sloughing around the house on a Saturday morning. I don't believe "anything goes," unless you are competing for "Worst Dressed Celebrity of the Year." Your outside is a reflection of your inside.