If you receive our newsletter, you may have noticed that you recently got a HUGE list of posts we've written recently. Well, except that they weren't all really that recent — some of those we two months old, and every week in between. Our regular newsletter is sent out automatically based on our RSS feed, and it turns out that our RSS feed was broken. Once we tracked it all down and got it fixed, all of the posts that had never gotten queued up for the newsletter shot out in one big go. Sorry about that. Aside from the crazy long newsletter though, I thought I'd share how I got this sorted out, because this is the kind of problem that can happen to anyone, and it is really annoying to track down.
When we finally realized there was a problem, we didn't know what was causing the problem, or therefor how to fix it. The error that Drupal.org was showing when trying to update the feed was "The feed from Drupalize.Me seems to be broken, because of error "Reserved XML Name" on line 1." That wasn't really helpful, but a little bit of Googling pointed us in the direction of having a space in the feed. Lo and behold when we went back and looked at the feed itself, you could see that there was a blank space right at the beginning of the first line. Hm, where did that come from?
We hadn't made any changes to our view for the feed, and we didn't have any custom theming going on for it, but we checked those things out anyway. Nothing proved helpful. Trying to review all of the code on our site for an extra space was massively daunting. Then it struck me that I could narrow my search very easily because we use version control. We knew the feeds worked fine originally, so we just need to check the changes between it working correctly and not. I tracked down the rough date of when the feeds started to fail, and then I found the point in time in our code where it worked correctly. We use tags for our production code, so on my local I checked out a tag that worked right (3.3.0). I rolled forward on the tags until I saw when the feed broke (3.3.2). Now since this was a hot fix release, the changes in it were pretty small and in only a few files. A quick look at the diff on the files and I saw the mistake right away: a space was accidentally added to the top of our template.php file, right before the opening PHP tag. That added a space to the beginning of ALL pages on the site.
Now, if this hadn't been so quick to find with my basic method here, the next thing to do would have been to use a really handy Git command called bisect. If you've never used bisect before, it lets you take two endpoints in your code, and then goes halfway between those two points. You can then test if the bug is still there. You can keep bisecting the commits until you zero in on the commit that introduced the bug.
So, luckily, once I remembered that version control is my friend in so many ways, I quickly found the bug, fixed it and rolled out the new hot fix. Our feeds are now working properly and we can share with the world again.