Actually, its a combination of the two.
The amount of zoox in a coral will determine the base colour (a lower count will give it a lighter colour, too much and its brown, too little and it's actually bleached), the polyps themselves may have a different colour and the growth tips (which exhibit a lot more of the anti-UV pigmentation) are usually brighter & more beautifully coloured.
IME, if your corals are growing fast, they are usually better coloured.