It can be done, but it defeats the primary advantage of using clams. Clams give off a scent that allows the fish to find the bait. It is like having chum on your hook. Throwing the shells and scraps over add to the chum slick. If you are drifting, the scent doesn't get out much further than the bait, and it is harder for the fish to track it down. If you happen to drift it close to a fish, there is still a chance it will take it, but probably no more than if you were using any other type of bait.