46 months ago
Found it and fixed it finally. Was pretty simple in retrospect.
We run multiple frontends behind a load balancer. The date on the frontends varies by a bit between them. The largest delta is about two minutes. It's not particularly critical that they be perfectly in sync with each other.
What was happening was that if you posted a comment and it posted on a frontend that was ahead two minutes, there was a chance that later rendering that comment you'd render it on a system that was two minutes behind the system that posted it. So the comment would effectively be in the future. The way our "format time delta" function worked, if you passed in a negative time delta it'd print out "23 hours ago". Since the largest time delta was about 2 minutes, you'd have a reasonable chance posting a comment to see that bug during the first two minutes after that comment / topic / whatever was posted.
The simple short-term fix was to just treat negative time differences as no time difference (and say it was "0 seconds ago"). Longer term we'll probably do a better job of syncing up the clocks between the systems.