This is a much-discussed aphorism (Philosophical Investigations II 190), and even now Wittgenstein scholars differ over how it should be interpreted. But everyone can agree that its meaning depends crucially on what Wittgenstein means by understanding (verstehen).
Here’s what he says about that in Philosophical Investigations §§531–532:
We speak of understanding a sentence in the sense in which it can be replaced by another which says the same; but also in the sense in which it cannot be replaced by any other. (Any more than one musical theme can be replaced by another.)
In the one case the thought in the sentence is something common to different sentences; in the other, something that is expressed only by these words in these positions.
Then has “understanding” two different meanings here? – I would rather say that these kinds of use of “understanding” make up its meaning, make up my concept of understanding.
For I want to apply the word “understanding” to all this.
At one level, understanding means grasping the general meaning of an expression. At this level, “I’m going to walk the dog” and “I’m going to take the dog for a walk” mean the same thing. If you heard someone (call her Alice) use either expression under the right circumstances, you’d be justified in forming the expectation that she would soon walk the dog. You’ll have understood her well enough to predict her behavior.