# On Long Threads



## Negrek (Dec 16, 2011)

So, this issue has apparently started brewing again, although people haven't been willing to say anything to my face about it. Long threads! They clog the server and cause errors! They should be locked and restarted! Negrek, why are you such a jerk, hanging around with all your long ASB threads that you blithely refuse to do anything about? How could you willingly detract from other people's forum experience like that?

That would be because I doubt "long" (greater than the thrillingly arbitrary 1000-post cutoff) threads actually have any significant impact on forum performance. And, in the interest of not having to repeat myself a ton, I'm just going to make a thread about it that people can froth over instead of addressing each person who brings it up individually.

How did the whole "long threads cause slowdown" thing get started, anyway? There was some nebulous muttering about it here, back when people were on a crusade against Forum Games. Somehow, "Well maybe long threads cause server overhead problems" went from being speculation to actual fact, and from actual fact to actually informing forum policy, without there being anything like proof brought up in the interim.

In fact, a simple Google search on "do long threads cause errors vBulletin" brings up no resounding "Yes! They're bad!" Responses tend to be mixed or outright negative, and the information available is contradictory and almost always hearsay. Ultimately, there is no consensus that long threads are more harmful than the same number of posts split up between multiple threads that indicates it actually has an effect that can be discerned from other factors that tend to come along with really long threads, i.e. lots of people online at once, lots of people viewing/editing a thread at once, and so on.

Unfortunately, Butterfree doesn't have error logging set up so she can see the number of 500 errors that occur, when, and what action was being performed at the time (I assume); otherwise, it would be fairly easy to benchmark this. We can therefore proceed only with indirect evidence. Most damning of all is the fact that periods of server fail come and go, while over time these threads have only been growing longer. The board was choking even before the Challenge Board hit 1000 posts. Then things died down for a while. Then, oh no! More server fail, and now the Challenge Board has more than 1000 posts! But then I didn't do anything about it, and the errors died down again. And now there are more errors, and the thread has even more posts! But, once again, no action is taken; the errors die down, and yet _*the thread has more posts in it than ever before*_. There is simply no correlation between the length of these large ASB threads and server performance.

Not only that, but even way back in the beginning, when forum games were the scapegoat, deleting them didn't actually stop the board problems. They died down a little while afterwards, yes, but the effect was not immediate, and _post hoc, ergo propter hoc_. Even moving to a new server, which should have led to more resources and therefore less of a problem for long threads, did not appreciably reduce spates of server fail.

So, there remains no evidence that I can see that long threads cause server traffic problems or errors. If you can provide some, I will listen to you and become a supporter of the "kill long threads" policy. If you can't, I will continue to ignore you, especially if you'd rather stew quietly than confront me about the issue.

Note that "but I always get more errors when I view the ASB Challenge Board!" is not a valid response for three reasons: a) confirmation bias b) anecdata c) lack of proper controls (could you be getting more errors there simply because you view it more than other threads?). If you can show that this is actually the case, i.e. with numbers in a controlled experiment, then it would be acceptable.

I dislike server fail as much as anybody else. If I thought locking long threads would actually do something to alleviate it, I would suck it up and deal with the inconvenience and just do so. However, I refuse to go out of my way to comply with a policy that has no sound basis in fact.

tl;dr if you want me to actually go to the trouble of getting rid of long threads, provide some actual evidence that they're bad for the board.


----------



## Zhorken (Dec 17, 2011)

Yeah, ten minutes of somewhat-scientific experimentation suggest that there's no correlation.  Loading the first and last pages of a few threads thirty times each at whatever the default page length is, these are the average times it took for each page to start transferring content (which would more or less be the time it takes for the server to generate  the page, which is where the slowdown would happen if thread length mattered):

*Homestuck fan club, page 1:* 1.263 seconds
*Homestuck fan club, page 36:* 1.146 seconds
*Challenge board, page 1:* 1.219 seconds
*Challenge board, page 58:* 1.380 seconds
*This thread, page 1 (and only):* 1.613 seconds (mostly hovering around 1.2 s; there was an oddball nine-second load time)
*My art thread, page 1 (and only):* 1.538 seconds (mostly hovering around 1.2 s, this time with several middling oddballs)

EDIT: It have to do with posting to long threads, or jumping to a certain post in long threads, or something, I guess.  The latter makes more sense than the former...  Might test that later I guess but I have Rice Krispie Squares to make.


----------



## ultraviolet (Dec 17, 2011)

I think I remember Butterfree saying something about the threads actually being a big deal wait I'm half asleep hang on I'll find it

edit: oh here



Butterfree said:


> Yes, and the lag, downtime and fatal errors  were because these forums are a memory hog. Kat e-mailed the host about  it and they explicitly went "uh, yeah, that forum you're running over  there has ridiculously long threads that take like five minutes to  query, you might want to prune some of that stuff".
> 
> Deleting helps a little more in that it altogether prevents people from  viewing the thread and removes them from search results for regular  users, but otherwise, yeah, those are probably fine being  closed.


to be honest, I've hardly had any errors at all any more - but then my timezone probably means I'm browsing when there's little traffic anyway, so.

edit edit: oh yeah and the 1000 posts thing is sort of arbitrary, yeah - but it seemed like the only thing we could really do about the problem at the time. :B


----------



## 1. Luftballon (Dec 17, 2011)

informally, by running wget(1) repeatedly with time(1) and subtracting the time actually transferring content: jumping to a post near the end of the challenge board takes 209 s to generate the page 75 times; jumping to this post, 93 loads in 74 s.

both are not really all that big next by the actual loading times, which averaged some 2.2 s (total ~165 s) and 1.4 s (total ~130 s), respectively (but mind, the challenge board page probably had more stuff on it, too), so. take that for what you will.


----------



## Negrek (Dec 17, 2011)

ultraviolet said:


> I think I remember Butterfree saying something about the threads actually being a big deal wait I'm half asleep hang on I'll find it


Hmm, that doesn't look right to me. Not that the host would have said that, necessarily, but that ~2000 post threads would be considered "ridiculously long."


----------



## 1. Luftballon (Dec 17, 2011)

well, we did once have at least one thread in which at least one user made at least 3000 posts, and just in that thread, wargle referenced a thread which was some 5k posts long.


----------



## Phantom (Dec 17, 2011)

Wait, one user made 3000 posts _in one thread_? Insane! 

Bet they were pissed when that thread was deleted.


----------



## Zhorken (Dec 17, 2011)

Eevee once said something about OFFSETs in the thousands making most RDBMSes cry, and it would make sense for an OFFSET to be involved here, for what that's worth.

More science, though.  I tested jumping to individual posts, and other than oddballs, the times were consistently 1-2 seconds regardless of how deep in the thread the post was.  The oddballs didn't happen on first loads or anything, either, so I assume the server was just plain having trouble at those points.

Here's the csv with all my results, if anyone wants to play with them; I can't be bothered actually firing up another Python prompt and messing with them this time.

EDIT: And here's my first round. (The plain old "time" in that one is time_starttransfer, which is a curl thing.)


----------



## Negrek (Dec 17, 2011)

Zhorken said:


> Eevee once said something about OFFSETs in the thousands making most RDBMSes cry, and it would make sense for an OFFSET to be involved here, for what that's worth.


Nope, no OFFSETs used in the vB code at all.


----------



## Butterfree (Dec 17, 2011)

All I know is the host's word on this, but I do agree that as far as I've noticed there hasn't been any marked overall decrease in 500 errors since long threads started getting deleted, and it sounds kind of generally fishy given one would _think_ that would at least make the threads in question load very slowly, which they don't. :/


----------



## Negrek (Dec 17, 2011)

Sooo, what ought I to do here? Close those threads, or leave them open for now?


----------



## ultraviolet (Dec 17, 2011)

is it perhaps more of a problem of user traffic, then? it explains why I get barely any errors at all and some get them lots. maybe it's only really a problem when lots of people are trying to view the same long thread, like the challenge board or some forum games threads or the fwee/grr threads.


----------



## bulbasaur (Dec 17, 2011)

Actually, I don't notice a difference in when or where the 500 errors occur - they happen at random locations (not just threads, but also user pages) and random times. Also, it seems that while I have a period of 500 errors, others have no difficulty coming on.


----------



## surskitty (Dec 17, 2011)

sorry about being obnoxious and passive aggressive, negrek


----------



## Negrek (Dec 17, 2011)

No worries. I wasn't trying to call you out specifically.


----------



## Luxcario (Dec 17, 2011)

bulbasaur said:


> Actually, I don't notice a difference in when or where the 500 errors occur - they happen at random locations (not just threads, but also user pages) and random times.


Exactly the same with me.


----------



## hopeandjoy (Dec 17, 2011)

Would have to agree there. I've gotten them just loading up the index a few times.


----------



## Phantom (Dec 17, 2011)

I seem to have the exact opposite, I will try for one page and get four 500's then quit because it's hopeless.

EDIT: I'm not sure if it's high traffic, I've been on when there is only a few people and I've gotten them.


----------



## Zero Moment (Dec 19, 2011)

When you guys say "500 errors," do you mean when it says that the server is busy?


----------

