Going Public: Tech Fu Exhausted
July 30th, 2005This summer on 3hive, we’ve been besieged with bugs. Either on the MT side or the host side. I tend to think it’s on the MT side of things, as I host this site with the same people, iPower, and I haven’t experienced any of the issues that we are having.
Specifically, something seems to happen EVERY weekend. On a Thursday or Friday, we won’t be able to log in and all the comment forms and popups are broken until the next week, maybe Monday or Tuesday and things are fine again for a few days. There may have been one weekend where we could get in this month but it’s been very spotty all summer.
Initially, I thought it was a Perl plug-in problem, but that same plugin is on the server that hosts this site, and so I tend to think it’s something more severe. If 3hive were my site alone, I’d just dump the mysql and start over. However, it’s not just my site. And there’s a lot of valuable information there that would be tragic to lose. I have backups, but the very thought of a catastrophe has rendered me helpless.
I’ve got help tickets in, but I’m asking here if anybody has had similar issues with MT and if so, how did you resolve them? I suppose I could change hosts, but that’s an expense in both time and money…
As an aside, the host offers a one-click install of WordPress and to test to see if iPower was having mysql issues, I dumped a lot of our content into WordPress. It didn’t appear to have any issues, which is why I’m leaning towards MT.
I’m now going to pray and nap.
July 30th, 2005 at 3:35 pm
Without access to system logs or remote access to MySQL, all I can suggest is (a) disable plugins one by one and see if anything changes, or (b) see if you can pepper the Perl code with error logging stuff and send the output to a file in your home directory (I assume you have a login account). You may have to go an entire weekend without access to the logs “you” wrote because you can’t log in, but at least there’ll be something waiting for you when access is restored.
Just for yuks, again assuming you can actually login at iPower, and assuming iPower uses UNIX, run “crontab -l” to see what is being run every Thursday and Friday. Something may be borking just the 3Hive stuff. You never know.
What a pain in the ass.
July 30th, 2005 at 6:28 pm
I have been very happy with WordPress. I chose it over MT only because my provider, DreamHost, had a one-click install. Its use of MySQL was incidental but convenient, as I have a lot of experience with both MySQL and POSTGRESQL.
Are you saying you can’t log in to the host shell, to your database, or to the blog itself?
July 30th, 2005 at 8:29 pm
We can’t log into the blog. I can get to the database, and perform operations on it, but I’m wondering if the database itself is corrupt. In that case, next time I log in, I’ll have to export everything, including all the templates and start again from scratch.
This is the first time that I’ve been bitten this hard by Movable Type. I don’t think it’s an indication of crappy software or anything, and if we moved to another platform, other considerations would have to be given.
I still believe that MT is the most robust, easiest to use CMS software. PERIOD. Jeff Veen wrote about this in late 2004 after looking at a ton of CMS software.
WordPress is sweet, but to bend it, the code fu increases exponentially, whereas the tag system inside MT seems more intuitive. I don’t want this to get into a “Movable Type sucks” thing or a “WordPress kicks ass” thing. Both systems are awesome and let people publish online very quickly and easily.
Right now, we just want to keep sharing the sharing. I may have some late nights next week.
July 30th, 2005 at 10:01 pm
I agree with the “I jumped on the WordPress bandwagon and liked it so far” camp.
(thanks for helping me with my mac issues, btw. not relevant but wanted to mention :))
I also host with Liquidweb and have had no problems so far. The company I work for has a client whose blog traffic is comparable to Heathers - the comments import from Blogger to WP was evil and required much-handcoded workaround, but that was mostly Haloscans fault. Their export function sux rox, as we say here in the Motor City.
So yeah, you WILL have the late nights, I am living proof of that right now. The brick wall you hit when migrating isn’t always apparent at first - backup early and often, brother…but then I am preaching to the choir here.
The WP forums and support community at large are great - I owe a LOT of people a lot of props, which will end up being musical in nature.
Have you considered either Radio Blog http://radioblogclub.com/ (I use it on my site) or Lazlo’s SoundBlox? http://soundblox.blogspot.com/
for your streams at 3hive instead of linking to mp3’s?
July 30th, 2005 at 10:13 pm
The whole point of 3hive is to point out the good, legal mp3s so people can download them and stick them on a player and listen to them again and again and then they buy the stuff they like. We want to encourage musicians and their labels to share a little so that they can continue making music. Streams don’t really do that, and there are a million streaming sites.
Yes, I know there are a million mp3 blogs, but none with the taste and sweet design that 3hive has. Insert smiley here.
July 31st, 2005 at 1:08 am
I agree that MT is great stuff. Just try not to love it too much. In the end, it’s just software. It’s capable of having flaws just like any other software.
The fact that it uses Perl gives me hives, but mostly because Perl has taken on a life of its own with all the different versions of the same packages. CPAN can be great, but all it takes is for DBI to be swapped for an identically named but inferior DBI and things can start to go screwy. For that reason alone I never rely on Perl doing anything other than twiddling text (and for that reason alone I don’t use MT, even though the system itself rocks). Perl just gives me the creeps.
July 31st, 2005 at 3:16 pm
I don’t mean to suggest you move to a different blogging platform, especially since the problem has not yet been identified. In fact, different software might just make the problem harder to find if it’s not in MT.
Can you duplicate the 3hive installation on a system where you have complete control over logging levels and such? It sure would help when you try to track down the problem.
3hive does look good and I have appreciated the suggestions.
July 31st, 2005 at 7:39 pm
I am way too technically inept to give a half-assed opinion even.. So good luck, I hope you get the bugs worked out. My husband is a techy genius… but he is here in San Fran, so that doesn’t help you much now does it?
July 31st, 2005 at 8:50 pm
This is probably something that you had seen before, therefor won’t be much help, but I was having problems with my MT isntallation (I actually decided to use MT again thanks to Dooce), where it wouldn’t let me log in, and when it would let me log in I wasn’t able to post, update settings, or change templates. It turned out to be a problem with CPANEL. There’s a message about it in the MT news when you log in.
July 31st, 2005 at 9:10 pm
Woops, the message is called
“Intermittent “500 Server Errors” in MT (RESOLVED)”
August 1st, 2005 at 6:56 am
Oh crap. There was no Geek Alert on this one. On to checking for a new whore….
August 1st, 2005 at 11:00 am
*Heading with Mrs. Paul to whore check*
Either that or wait here til Heather finds all this geek talk….
August 1st, 2005 at 12:42 pm
Not to split hairs too finely, but if the dynamic parts of the site are failing on a regular, repeating schedule, it seems most likely it’s an issue with your Perl environment.
We’d be glad to help troubleshoot, but maybe we could try a test migration to MT3.2 and see if it’s interacting differently with your Perl setup?
August 1st, 2005 at 1:54 pm
An MT installation that doesn’t work on weekends? Man, I’d love that job…
Anyway, I agree with Anil. MT doesn’t have anything that is regular and time-sensitive, unless you’re using cron for period rebuilds or something or you have a plugin that does so. This sounds like some problem on the host.
If there are errors in your activity log or in your server’s error logs, you’ll want to put those in your help ticket as well.
In either case, I’m sure that our tech support wizards will answer and help you solve this mystery.
August 1st, 2005 at 1:56 pm
Oh also:
“I tend to think it’s on the MT side of things, as I host this site with the same people, iPower, and I haven’t experienced any of the issues that we are having.”
It could very well be the combination of your host and some data or code that is in that other MT installation. Quite often problems aren’t found until you have edge case intersections, which is what you may have stumbled across…
But I can assure you that if MT wanted to take off every weekend, we would consider that a bug and have had it fixed if we knew about it.
August 1st, 2005 at 2:15 pm
Anil & Jay, THANKS FOR RESPONDING TO THIS. I’ll contact you offsite.
I’m thinking that it’s a combination of things. There may well be a cron job on my host that is jacking something, if so, it’s not me or my doing. I think our mysql database might be hosed. The support staff of six apart have been awesome and have responded very quickly, including over the 4th of July. Because I dig on the MT tag structure, I’m holding out that we can keep the hive on an MT install.
At this point, I think my solution is to wait until I can log back into MT and then export everything out, including templates. Then trash the MT install, trash the database and start over, reinstalling everything, re-making the templates and then re-importing the entries. Re-install the plugins and test. Then, if it works, push the mother live.
It’s gonna be a fun week.
August 1st, 2005 at 3:03 pm
I’m sure you hate comments posted on your blog about an entry made by your wife, but she doesn’t have comments enabled, as you know. I’m posting about this entry:
http://www.dooce.com/archives/daily/07_13_2004.html
I don’t know if this will help or not, but it worked for Caden. One of us would sit in the back with him, hold his hand, and whisper to him. Since baby’s can sense the tension, you can’t just be doing it to shut them up. I’d talk to him. Have conversations with him. Hell, it’s better than listening to crying. At first he’d just keep screaming but pretty soon he’d quiet down so he could listen to what I had to say.
I’d do that for hours if necessary. A lot of work? Yep. But you should see how well that turned out! Now he’s 5 and a half and I have never once spanked him in his life. The reason? Simply getting a different pitch to my voice than he’s used to brings him to tears. He so used to talking to daddy now that my words are all I need.
Maybe it will work. Maybe not. But I thought I’d tell ya my experiences.
August 1st, 2005 at 4:50 pm
Mrs. Paul….we have something right here. I don’t think *that* guy REALLY reads Dooce.
August 1st, 2005 at 6:42 pm
Jon, whatever happened with this??
August 1st, 2005 at 10:57 pm
Curious to hear what happened to this. From my telephoto lens, I’m going with something fubar’d the lock on the necessary tables that are required for the login process. That best describes the behavior you’re seeing. Try this: next time you see the lock up, try to do the exact same queries that the login process does in the exact same order. See if you can do it. That will quickly tell you if in fact you have a SQL problem or a app level issue.
Best of luck.
August 2nd, 2005 at 4:17 pm
Um…yeah. What Steve S. said.
August 2nd, 2005 at 8:56 pm
* poke! *
You still nappin’, Jon?
August 2nd, 2005 at 9:37 pm
Have you considered looking at your error logs (there should be a error log option if your host uses cpanel), when you attempt to log in and googling the errors? Often perl has errors that have been discussed on newsgroups, etc.
Also, I had modified my headers (I use wordpress btw) recently in an attempt to redirect users to a subdomain, and then changed my mind and forgot about the headers…so instead of going to a.com/login.php, my browser kept redirecting to a.com/something/login.php I figured out the problem via lynx (I use liquidweb, and lynx is installed, so if your host is *nix based, lynx may work as every redirect requires you to press your enter key, so you can tell where you are going). Hope this helps.
August 2nd, 2005 at 9:42 pm
I looked at the error logs. Nothing came up that was conclusive. The recent issue with a Perl plugin doesn’t seem to have applied to our situation. So now I’m leaning to MySQL corruption. We’ll see. I’ve posted about my steps in an entry called “Elbow Grease”.