You have a lot going! That's great. What zone are you in?
I am in 7B I guess (the realtor said it's now zone 8 but on the back of seed packs it shows us as 7. So, I go with 7 when I buy things.)
I can't say for sure about the Goji. I have read that it should do ok in my zone but I have found VERY single bug and critter eats them. I have one big plant that I grew from seeds a while back and that one is inside now.
I had a few in the ground but the bugs and other things kept at them.
I grew about 12 other plants and gave them to friends and family (some in Ohio, one in Mississippi, one in Washington, and a few others here and there.) It'll be neat to see how they hold up for them too (everyone that I gave the plants to earlier in the year said the same thing that BUGS just love them.)
I'd love to grow rosemary. It's one of the plants that I haven't done yet. Maybe, next year. Is it an easy one to grow?
Dora Renee' Wilkerson