People who loiter around the temple are probably just trying to wait. They don't mean to look like they're up to no good, but it's hard to wait without looking suspicious when you're waiting outside a temple.
As you wait outside the temple, you can't help but adjust your tie (たい). When you wait you get nervous, and when you get nervous you fidget. So now you're messing with your tie a whole bunch.