When we do this on an inputs/shared var and that this is the only use
of it in the graph, we should only transfer to the GPU the small
computation.