TL;DR: no, it does not seem to be the case that the mathematicians of Pythagoras had the narrowly defined meaning of practitioners of mathematics, but were rather more general scholars.
$\bullet$ Greek: The Greek origin of the word "mathematics" is "that which is learned", and early uses were consistent with this broad sense of the word. A History of Greek Mathematics describes how for Plato μαθηματα
simply means any subject of study; Archytas (428–347 BC) writes that "those concerned with mathemata were particularly interested in the speed of the stars, their risings and settings, and in geometry, arithmetic, astronomy, and also music – for these mathemata seem to be sisters."
The Greek philosopher Sextus EmpiricosEmpiricus (160–210 CE) explicitly used μαθηματικούς (mathematicians) to indicate a broad class of scholars, including rhetoricians, geometers, musicians, astrologers.
$\bullet$ Latin: As recorded in the Latin Dictionary of Lewis and Short, early uses of the noun mathematician in Latin had the modern meaning as well as the meaning of astronomer or astrologer, for example, in this famous quote from Augustine of Hippo (fourth century):
Quapropter bono christiano, sive mathematici, sive quilibet impie divinantium, maxime dicentes vera, cavendi sunt, ne consortio daemoniorum animam deceptam, pacto quodam societatis irretiant.
The good Christian should beware of mathematicians. The danger already exists that mathematicians have made a covenant with the devil to darken the spirit and confine man in the bonds of Hell.
$\bullet$ English: The Oxford English Dictionary records earliest known uses of the noun mathematician, which in the 16th century could mean both a person who is skilled in mathematics, as well as an astrologer. Ranulf Higden's Polychronicon is the earliest 15th century source.