First u need to identify what type of worms. The 3 most common seen, would be bristle worm, fire worm & bobbit worm.
It would be good to remove the bobbit, as it may attack smaller/slower fishes, although some reefers says they never had problem with it.
Fire worm too, remove it to reduce possibility that you may get stunk when u move the LR.
Bristle worm usually can leave inside as they are scavengers. But some reefers still don't trust them, so it is also removed.