Make InputsToGpuOptimizer skip processing if all clients are transfers or outputs, no matter their number.

Also, fix the error condition.