In ancient Israelite culture, the term "brothers" was used to refer to cousins and other relatives, not just siblings. In Genesis 13:8 and 14:16, Lot is called the "brother" of Abram, yet 11:27 indicates that Lot was actually Abram's nephew. In Genesis 29:15, Laban calls Jacob his "brother", though Jacob was his nephew (see Gen 28:2). In Leviticus 10:4, Moses calls Nadab and Abihu the "brethren" of Mishael and Elzaphan, while that very same verse describes them as first cousins once-removed. In I Chronicles 23:21-22, the word "brethren" clearly refers to cousins.
Scripture has more such examples, but these four sufficiently illustrate the broad usage of the term "brother" or "brethren" in the Bible.
Now, though the Bible speaks of Jesus' "brothers" and "sisters", nowhere does it actually call them children of Mary and Joseph. Jesus alone is called "the Son of Mary" (Mark 6:3), a phrase which in that culture denoted the only child of a widow. And Mary is always called simply "the mother of Jesus", never "the mother of Jesus, James, Joses, Simon, Judas and their sisters" or anything like that.
In fact, Saint Mark tells us that two of these "brother", James and Joses, had another mother, Mary the wife of Cleophas (Mark 15:40). That also indicates that they were not Jesus' half siblings or sons of the Blessed Virgin.