One side edge of the "window" is taller than the other - because of this, it never looks like the tall edge is farther away from you than the short edge. It makes it look like it's going back and forth because your brain thinks the front edge is the taller edge regardless of how close or far it really is. As an aid to this, the "window" is flat/thin enough that there is no side surface or 3rd dimension to give away the illusion when it points straight at you. You would normally notice the angles of the front surface as they relate to the side surface, but since it's so flat, there are no angles to be seen.

There is added confusion due to the pipe - it's obvious to your brain that the pipe is going all the way around because it's neither parallel nor perpendicular to the ground and it doesn't try to fool you.

I hope I didn't add to your confusion.