The refuge was build to keep water clean , remove po4 and no3 . Dirts , rubbish , dead matter, detritus are the main cause of high no3 and po4.
If your tank water is good and the refuge is doing what it's supposed to do.Then I would leave those stuff alone , but if those dirt are causing problems , I would remove them. Don't worry about the little critters , there are lots more from where they first came from.