Bro.. my advise... if you want to get things right. Go buy a second or new tank with sump. Your method will not work, it might as of what you see and think it work.. but it like ticking bomb...one day it will flood your floor and kill your coral and fish.
Don't waste time trying... go get a new tank .. if really not possible. Then use canister. Your existing filter is good enough. I think your mistake is like most newbie, loading the tank with too many live stocks too fast or overload whereas the filter cannot manage.
Actually keep a reef tank is not really that difficult. First thing I need to manage the water. Go goggle on how to keep water. Only with good water then your live stock will let survive.
Also ... another key thing in success reefing is patience. Don't see something wrong only start to react on it.. sometime this will be over react and instead do more harm then good.
Remember patience is the key to success. .. this was once told to me by a old reefer...
Sent from my SM-A800F using Tapatalk