This makes things easier for shape inference and gradient operations.
Also fix broadcasting pattern of the gradient.