hi eChan, think no hard and fuss rule. It depends on your initial setup. If you start everything fresh and needs the bacteria to bloom, then in my opinion, you should remove them.....if you have live sand, rock, bacteria dose, etc, then good to have them to maintain good water quality. Just remember to reach through the cycling cycle - spike in amonia, etc and then back to zero.....if you put in too many media to improve your water, the initial reading may not represent well on your tank cycling....and you might have to take few reading to ascertain the cycling process....
just my humble comment and subject to correction from the experts....