I do not force the grad to be the same format. This would require
conversion.  I do not think we always enforce this already, so I do
not want to enforce that now.