This enable elemwise opt with flatten in the middle.
This can also enable more elemwise fusion that got split by flatten.