[Goober5000]

For the past week or so, Sesquipedalian has been experiencing long stretches of time when he cannot access HLP.  The forums either show a blank page or a database error.  Attached is a sample screenshot.

Any ideas?  The forum error log hasn't yielded any useful information (mostly because it's jam-packed with template errors caused by the unfinished theme...)

[attachment deleted by ninja]

[rev_posix]

Found the error in the apache error log:

[Fri Feb 25 20:37:21 2011] [error] [client $IP] PHP Warning:  mysql_real_escape_string() expects parameter 2 to be resource, boolean given in /var/www/clients/client2/web2/web/forums/Sources/Subs-Db-mysql.php on line 143, referer: http://www.hard-light.net/forums/index.php?board=80.0

EDIT: Did some digging and found this thread on the SMF forums:
http://www.simplemachines.org/community/index.php?topic=423482.0

Quick scan makes it sound like this is caused by the sql backend running out of open connections or dropping a connection right after it's established.  Looking at the config file for the site, it looks like it's set for 100 connections max for all the hosted sites.

After looking at the number of hosted siites on the machine, I'm thinking that a connection starvation may very well be the root cause.  Turning up logging on the database would help confirm that, but would really impact overall performance.

[Dilmah G]

This is what I was complaining to you about on IRC the other week.

[Fury]

A week huh? I have a hunch what may cause the problem. Please report if it occurs again.

[Goober5000]

According to Sesquipedalian, it's still happening.  As I type this, even.

EDIT: Hmm.  He cleared his cookies and his cache, and now he can view the boards, but not log in.

[Sesquipedalian]

Actually, after I cleared my cache and cookies, I still could not view anything on the boards. When I tried to load a private forum, I would be presented with the login page as expected, but with I tried to log in all I got was a blank page. I also got a blank page when I tried to go to the main forum index. Trying to go to specific forums or topics would show me as logged in in the banner area, but show only the Database Error message in the content area. This continued for about 15 minutes after I cleared the cookies and attempted the login. Then suddenly everything started loading properly again. I just hope it keeps working now...

[Fury]

If that kind of problem was server-side, I'm pretty sure there would have been a lot of people complaining by now. Since that is not the case and based on above post, it sounds like isolated problem.

If this isn't one of those HLP theme bugs, I'm not sure what it could be beyond SMF 2.0 RC5 bug. If this occurs again, please see if the problem also affects FSWiki and Game Warden. If those two aren't affected, then it's a forum software problem. If only forums are affected, you should try the default SMF theme, you can switch to it in your user profile settings. If those two are also affected, then it's a server issue, though it sounds improbable based on only two people have reported this. Should the latter be true, I don't know what possibly could cause it, unless rev_posix or Goober have altered something lately.

Anyway, can't be anything serious if it's only affecting two people.

[rev_posix]

No changes by me.  If I was going to, I'd run it by everyone else first. 

[Goober5000]

Nor me.

[Goober5000]

This is still a problem; Sesquipedalian still cannot access the forum from multiple browsers on multiple computers.

The last time he was able to access the forum he tried changing his theme back to the default, but that didn't help.

Did we ban all of Canada from HLP, or something?  Could something have gotten screwed up with his forum account?

[rev_posix]

This is still a problem; Sesquipedalian still cannot access the forum from multiple browsers on multiple computers.

When this happens, has he tried to access the wiki?  I'm guessing, since Fury mentioned checking it when this happens, that the wiki uses the same database as the forum, so if it is related to a DB connection starvation, Sesq' should be seeing it happening on both when one starts up.  That's the theory anyway.  If we knew the time this was happening and what URL was being access, digging in the apache logs could reveal something more.

Also, looking at the DB again, the Max_used_connections is only at 74.  Of course, this is the maximum since it was started, so what I'm seeing now might not be an accurate gauge of how busy it is when Sesq' is having problems.  All I know for sure is that the errors I found when this first came up are common with many pieces of software that use PHP for DB access, and it's almost always a lack of free connections to the database.

EDIT:  OR, just perhaps, something is all but locking the main mysql thread and PHP is timing out before it can handle the connection request...  So many different ways this one could go...

Did we ban all of Canada from HLP, or something?  Could something have gotten screwed up with his forum account?

Yep, .ca has been banned.  Damn their free healthcare, and cold meds with codeine in them, and....  wait, remind me again why so many places make fun of them? 

[Goober5000]

I spoke to Sesquipedalian and got some more information...

[11:11] Sesquipedalian: Well, there's never been any problem getting to the wiki, or the main page. I must say, I would be quite surprised if the wiki and the forums used the same database.
...
[11:21] Sesquipedalian: Okay, so I've cleared my cache and deleted the cookies from hard-light.net, and I will try to create a new user...
[11:24] Sesquipedalian: ...
[11:24] Sesquipedalian: ...
[11:25] Sesquipedalian: Blast! A blank page.
[11:29] Sesquipedalian: Well, maybe this will be relevant: whenever I run into either a blank page or the "Database Error" message, the server takes a long time to respond, whereas if I run into the message that I need to log in, that response comes very quickly.
[11:31] Sesquipedalian: Also, some of the cookies I get are for "hard-light.net" and others are for "www.hard-light.net", in case that makes a difference.
[11:34] Sesquipedalian: If Fury or rev_posix want to schedule a time to have me try certain actions so that he can watch the log in real time, I'd be willing to do that.
...
[11:35] Goober5000: when exactly did the blank page come up?
[11:38] Sesquipedalian: When I clicked the link to go to the registration page. However, I often get that even just trying to go to the main forum index (although that is unpredictable)
[11:39] Sesquipedalian: Oh, for heaven's sake. Now the registration page responded. I'm going to try making a new user while I have the chance.
[11:39] Goober5000: hahaha
[11:41] Sesquipedalian: Okay, so I made a new user, and got my email, and am now trying to log in...
[11:45] Sesquipedalian: Hm. Logging in seemed to return no response, but when I gave up and then tried to reload the main forum index, that loaded and shows my new user as logged in.
[11:45] Sesquipedalian: Of course, we've seen that before with my real user: occasionally working, and then later not working again.

[11:57] Sesquipedalian: By the way, the cookies from hard-light.net and www.hard-light.net look like they are redundant and might be conflicting with each other.
[11:58] Sesquipedalian: Cookies from hard-light.net:
exp_last_activity
exp_last_visit
exp_tracker
PHPSESSID
[11:59] Sesquipedalian: Cookies from www.hard-light.net:
exp_last_activity
exp_last_visit
PHPSESSID
[11:59] Sesquipedalian: They appear to have different values.
[12:00] Sesquipedalian: when I was logged in, there was another cookie from www.hard-light.net called HLPBB or something similar.
[12:00] Sesquipedalian: When I logged out, that was deleted

[Fury]

Of course forums, wiki, etc don't use same database. It's not relevant, the databases are still located on same server and use same mysqld process to access them. If there were connection issues, they would be seen on the wiki as well as Game Warden. Since there haven't been any reports to that end, it clearly isn't a server issue at the very least.

There hasn't been any problems with the forums accessing the database either. Not a single error email have been sent to notify about problems connecting to the database. There aren't any such errors logged into the forum error log either. Which means exactly one thing, there aren't any database problems as far as the forum software is concerned.

Which makes sense since otherwise we'd have a lot of people complaining, yet we have none except for one. I don't even know what's the deal in Dilmah's case since he hasn't posted anything since last Monday. For all we know that was one time issue Dilmah faced along with everyone else and is separate issue from what Sesq is having.

Unfortunately I don't have a solution or suggestions for the problem. It is quite curious he's getting database error messages, but everything seems to indicate it's a matter of connections simply timing out. It is actually consistent with above description of the problem, slow loading of pages.

If I had to guess, I'd say that everything points to a networking problem between Sesq and the server. That or some really obscure bug in the forum software, assuming that he really can browse the wiki and GW while forums aren't loading at all. However, it sounds like he didn't try GW at all and only sometimes tried the wiki. The only relevant question here is whether the wiki and GW works while the forums aren't loading. Browsing them while the forums are working is equally irrelevant.

If the answer to those is a no, then it's clearly networking issue and nothing we can do about it. If the answer to that is yes, then is is most likely forum software bug, again nothing we can do about unless there is a code fix somewhere in the official SMF forums. In the latter case someone would have to report it to the SMF devs, someone other than me.

[Dilmah G]

Well the symptoms were the same as Sesq's, more or less, but it only occurred on two occasions for me (once for about 3-4 days, the other for a few hours, I believe). Sorry I can't give more details, can't remember much of it at the moment.

[Goober5000]

Which makes sense since otherwise we'd have a lot of people complaining, yet we have none except for one.

Well, if they aren't able to access the forum, how are they supposed to complain?  We only know about Sesquipedalian's problem because he IMmed me.

[Fury]

IRC, IM, email? There are many ways to get the information through.

[Goober5000]

Just spoke to Sesquipedalian again.  He continues to have no problems accessing the wiki, and he doesn't have any trouble accessing Game Warden either.

On a hunch, I asked him to try using one of those free web proxies to view the forum.  This actually worked; he could sign in with his dummy account and everything was "fast and smooth".  He is using freeproxyserver.ca.

[Dilmah G]

Funny you say that, I came home from school about 5/6 hours ago, had Sesq's issue, and tried a proxy and it worked when I disabled 'remove objects' and 'remove scripts' under the options. Though now I can access the forums normally. Weird, huh?

[Fury]

I can't think of anything but Google's ads that would cause such erratic issue, especially if you can prevent it by disabling javascript. Sesq should try ad-blocker that blocks ads from loading. Note that ad-blocker may load ads but not display them, in which case problem may still persist. As far as I know, no other external javascript is used.

Internal javascript would cause these issues for everyone. This theory is consistent with what Dilmah has posted. So, block ads entirely from loading and see what happens.

[Sesquipedalian]

I don't really understand how or why, but I blame my ISP. I'd rebooted my router and modem recently, but that hadn't helped. However, since the proxy server worked, while blocking ads and disabling JavaScript had no effect, and whenever I got errors I always noticed that the server was responding very slowly, I came to suspect that the issue was indeed a networking issue. So I turned off the router and modem for an hour to make sure they did not reassign me the same IP address, and when I turned it back on, everything was fine again.

I still don't understand how a connection problem like that could have such site specific effects, but I guess it's fixed itself now. Thanks again for all your patience, Fury and rev_posix, especially since it really doesn't seem to have anything to do with you.