The ancient astrologers were not concerned with the material stars. Plotinus in his Enneads was asked “are the planets causes?”, to which he argued that they are not causes, but instead the entire cosmos is as one body, and as a physician looks at one part of the body and is able to gather information about another, so a diviner can look at one part of the cosmos and determine information about another, and as he says this is how diviners are able to divine things through the flight of birds (as was common in the Roman empire in that time). In this way, horary astrology is also possible, and it provides immensely precise information. You could say it is the height of divination.
Al-Kindi proposed another theory, that the stars (and, everything else) exert spiritual rays that cause events on Earth. But these rays are spiritual, not material.
With that said, there are many factors in traditional astrology such as the North and South nodes of the Moon which are not material bodies at all. They are not physical bodies, yet were always considered and caused great effects. Similarly, the precession of the equinox has been known since nearly some 2000 years ago, but the position of the Zodiac never changed in traditional western astrology, despite the stars moving being well known. This is because the Zodiac was never based on the material stars.
The 28 Mansions of the Moon in traditional western astrology is similarly not placed by the position of a fixed star.
Traditional astrology was not concerned with the material bodies, but rather the spiritual heavens that it points towards. In fact, in their Neoplatonic view, the material world was more imperfect, unlike the higher spiritual world. All this is shown in the extreme precision and accuracy of traditional horary, electional, natal, mundane, and astrological magick.
It is only now modern astrology that takes on a materialist view, and ends up being innacurate for predictive purposes in many cases.