Revert part of "Alloc OpRemove to accept a class to remove. Tested by test_gradien.py:test_grad_clip"

This reverts part of commit 0e2d7a79.

We don't need that and it is safer to not introduce it.