safe_to_cpu was one of the reason that I was importing cuda. It was a
misunderstanding on my part, and the way is done now is somewhat more sane.