I think that's a large part of it - whether your particular "appropriation" seems to imply you are an authority on a culture you know next to nothing about. Eating in the restaurant implies nothing. Even cooking a dish at home doesn't imply much. Leading a religious/cultural service does. Getting a culturally important image or phrase as a tattoo does. It "claims" the culture much more distinctly.

The latter, authority-laden activities tend to imply cultural knowledge and continuity, so it's disingenuous to perform those actions without the appropriate contextual knowledge, and could lead to a lot of misunderstandings about the source culture (transcription errors, like I mentioned above). That's disrespectful to the source.

Maybe the "claim on culture" is the real dividing line - though obviously it's a fuzzy one that different people will interpret differently.