Is this a valid argument against clean meat?
This doesn't really answer your question in a qualitative way but could be useful data(?) anyway.
For me, it goes the opposite way–it's easier to not eat meat right now when I tell myself I'm not giving it up forever, just until there is an ethically acceptable source.
Also, I expect that if an alternative to factory farming becomes commonplace, the next generation's view of factory farming will shift to something like "how could this ever have been accepted", and I kind of don't want to be on the wrong side of that.
These are two pretty idiosyncratic potential cultured meat-based reasons to not eat meat now. But on the other hand I'm someone for whom your interlocutor's argument doesn't have an impact, which might help you decide how prevalent people for whom it does have an impact are.