“Threadless Numbers” is a series of posts I wrote on my now defunct t-shirt blog, Fantastic Blognanza. It is an analysis of the data I collected while doing research for my final paper in a sociology class, Technology and Society, beyond what I included in that paper. The paper itself may appear on this site sometime, but it will need revising before that can happen. Note that this is somewhat old research (2007), so might not be as applicable to the current community at Threadless.com, an online crowd-sourced t-shirt design competition/community/company. The analysis is also not what it would be if I were to do it now. Regardless, hope you enjoy, and feel free to comment/critique/question!
Table of Contents
– What Is Threadless?
– Threadless Numbers #1: Is it possible to predict how well a design submitted to Threadless will score?
– Threadless Numbers #2: What’s the difference between professional and amateur designers on Threadless?
– Threadless Numbers #3: What kind of t-shirts don’t get printed at Threadless?
– Threadless Numbers #4: Does being a Threadless “alumnus/a” give your submissions an advantage?
What Is Threadless?
[Excerpted from my paper, a description of Threadless circa 2007; a really good, extensive, current discussion of Threadless as a company] Anybody is welcome to submit a t-shirt design to Threadless. When they do, their design is visible in the voting section of the Threadless website for a week, over the course of which the Threadless community judges it. (Note that if the design scores too poorly, it will be removed from the running after 24 hours.) Designs are scored on a scale from one through five; additionally, a shirt can be rated “I’d buy it!” Based on the designs’ scores (which are made public after the scoring has finished) and the “I’d buy it!” votes (which remain hidden), Threadless prints the most popular ones in limited quantities, which are usually put up for sale about 90 days after their scoring has finished. The designer of the winning design receives $2000 in cash and prizes. Regardless of how well a particular design has scored, all are available for viewing in an extensive archive.
All designs that are currently in stock can be viewed and purchased exclusively through the Threadless website. They are sold for $15 (for guys) and $17 (for girls). In 2005, Threadless made $6.2 million and was expected to make $18 million in 2006. Although nearly all designs eventually sell out, some designs sell out rapidly, others will sell out during a $10 sale, and some will last months (but usually in only a couple sizes). Threadless prints new winning designs and reprints a couple sold out (or nearly sold out) ones each week.
Threadless Numbers #1
Is it possible to predict how well a design submitted to Threadless will score?
Because I’m a t-shirt nerd, I wrote my final for Technology and Society (it’s Sociology) last semester on Threadless, which involved gathering some data from the Threadless website. Even though my research has some problems, I found some intriguing stuff! I’m planning on making a small series of “Threadless Numbers” posts, beginning with this one. Some of the other topics I would like to cover are 1) the differences between how professional and amateur designers use Threadless, 2) who actually gets their designs printed, and 3) what losing submissions are like.
So, is it possible to predict how well a design submitted to Threadless will score?
In short: maybe, but not with the data I collected. My data, however, provides an indication that it should be possible to make a pretty good guess based on the number of comments a design has received.
What This Means for You
Because I’m guessing most of our readers don’t have much of a background in statistics, I’m going to leave some of that stuff out. Suffice it to say, there is a real correlation between the number of comments a design receives during scoring and its final score. In a perfect world, all you would have to do is replace x in the equation displayed on the graph above with the number of comments the design has received, and you’d get its final score.
So you can only predict the final score of a design in hindsight with my numbers, which isn’t much of a prediction at all. Because of this correlation, though, it would make sense that the comments on a submission before its scoring period has ended would also correlate with the final score, although this correlation is likely to be weaker unless you can control for things like the number of users scoring designs on a given day (or day of the week).
If the number of users varies consistently by the day of the week, surveying a large enough sample of designs and counting the number of comments left on each day of the week should yield enough information to control for this variability. Fortunately, Threadless tells you two useful things about submissions and comments: both what day a design was submitted and on which day each comment was left. I’d be impressed if somebody goes on to do this, but it would be a logical next step to what I’ve already done. Be sure to let me know what (if anything) you find!
Discussion of Data
If reading about possible problems with this data doesn’t float your boat, stop right here. If you’re still on the edge of your seat, read on.
One of the things that would most affect the regression is a data point that I consider an outlier. That point is for Disbelief, which has 344 comments and a score of 3.10. Including the outlier makes the correlation look more logarithmic than linear (the R2 is higher for a logarithmic trend line than a linear one, but lower than when excluding it). Without data on how individuals vote and comment, I can’t think of many conclusions to make from this fact. It would make sense, however, for the correlation to be logarithmic, because the maximum average score a design can have is 5, and even the most popular designs don’t score anywhere near that.
Another potential problem with my data is that I was only working with the 60 most recently printed designs as of May 6, 2007; because an overwhelming majority of submissions are pulled after 24 hours for having such low scores, it’s difficult to find data on non-winning designs that have finished their entire time in the running. I’m not sure how this would affect the data, but it seems as if it might.
A final problem is that I used the total number of comments (both negative and positive) left for a submission, because it is an easy metric to record from the Threadless website. Using the total number of positive comments would probably result in a stronger correlation.
So can I predict the final score of a design submitted to Threadless? Not really. But I hope one of you figures it out!
Threadless Numbers #2
What’s the difference between professional and amateur designers on Threadless?
Welcome to my second installment of “Threadless Numbers;” if you missed the first (on submission scores and comments), you can still read it above.
Even though the Threadless t-shirt design competition is open to all, those who actually get their designs printed could be primarily professional designers who have little actual involvement in the Threadless community, which would be antithetical to Threadless’s image as “design by the people, for the people.” Is this the case?
Because there are two parts to the problem (1) professional designers winning and 2) not being involved in the community), the answer is also two-fold.
Do Professional Designers Rule Threadless?
Well… yes and no, but mostly not really. Professionals (heuristically classified as “those members of Threadless who link to a designer’s portfolio or something similar on their profile”) might account for a greater total number of winning designs, but professionals and amateurs on average win about the same number of times.
Professional designers account for a greater total number of printed designs than do amateur designers. On the other hand, the number of designs printed on average by a member of either group is pretty much the same.
Even though the mean number of designs printed by professional designers (out of the 102 most recent winning designers as of May 6, 2007) is slightly higher, that difference is not statistically significant. Which means that the above graph is pretty much pointless… but whatever. It’s nice because it illustrates one point: it’s really hard to tell who on Threadless is a professional!
I judged whether each winner is a professional designer by whether or not they had a link to their professional design website in their profile. That data point at 14 printed designs is Glenn Jones, a.k.a. Glennz; even though he didn’t link to a website as of my research (and it still isn’t clear from his site that he is a professional), I know that Glenn is the creative director of Dashwood Design, a graphic design company in Auckland, New Zealand.
Are Professional Designers Really a Part of Threadless’s Community?
So professional designers don’t really win Threadless’s design competition any more than amateurs (even though they account for most of the winning designs); but do they really play as large a role in the community?
Why, yes they are! In fact, professional designers play an even larger role in Threadless’s community than do their amateur counterparts! Why might this be? I have several theories concerning this fact: that both a) professional designers are a welcome part of the community (I will return to this point shortly) and that b) the community is composed largely of professional designers!
My support for a comes largely from this Threadless blog post asking about opinions on professionals submitting to Threadless: only 15 responses (which is pretty meager), all of which are positive. And within those responses, eskimokiss a.k.a. Pascal Hoayek hit the nail on the head (I believe) as to why b might be the case: Threadless began with “a group of ‘professional’ designers releasing some creative energy by designing t-shirts.” So Threadless may not be exactly “by the people, for the people,” but more “by designers, for designers.” With its growing popularity, however, (and the pool of non-professional designers who might be interested in Threadless is larger than the number of professionals) it seems to be partially shifting toward “by designers, for the people.” Threadless Select and their recent decision to let “winning designers select a certain number of shirts to be printed every month, regardless of the voting results” (Walker 2007) may be responses to this trend.
Oh, and in case you’re wondering who scored 55095 submissions… it’s Ronin60 a.k.a. Cameron McEwan, member of Threadless number 7007 since June 20, 2002. He’s scored over 3000 more subs since May 6, 2007.
Implications and the Future
In sum, what all of this means is that professionals and amateurs have an equal shot at winning Threadless’s t-shirt design contest, and that professional designers participate heavily in Threadless’s community. Submitting designs is not the full extent of their contribution.
It will be interesting to see how things change; whether the printed designs will start to lean more toward either professionals or amateurs, or whether it will stay relatively balanced. I’m even more interested in whether more professional designers will show up who just submit entries and don’t do much else. On the other hand, participation in the community may well improve a designer’s chances of getting printed, so non-participatory professionals might not have as high odds of winning as do participators.
Interesting Things about the Data (and Problems)
May as well put this behind a cut [originally]… even though it actually is interesting! (To me.) And I only mention a p-value once, so it’s hardly more technical than the above discussion. It wouldn’t have surprised me to find a significant difference between the mean number of designs printed for professional designers and those for amateurs, whether because professionals are just “better” at designing (and so each of their designs is more likely to win) or because they design more prolifically (and so they have more chances at winning). Neither of these factors seems to have any affect, whether by the intervention of Threadless or by the will of the community. My guess is that it’s a little of each.
There is a big problem with my measurement of “participation in the Threadless community.” I chose to use “number of submissions scored” simply because it is the easiest metric to record from the Threadless site that seems reasonably to be related to participation. And it is reasonable, because voting is participation, and a form that is displayed publicly, at that. Whether voting is participation in the community (as opposed to just in Threadless’s business model), however, is open to debate.
That professional designers account for a greater total number of printed designs than amateurs cannot be explained by the fact that professional designers have on average been members of Threadless for a longer time than amateurs (586 days for pros compared to 364 for amateurs, p< .001), even though this time advantage might give professionals more experience with the community's preferences. Nor can professional designers submitting more designs (as I mentioned above, though I don't have the data to test this hypothesis) explain this discrepancy. Rather, instead of there being a difference in the nature of professional and amateur designers, there must be an imbalance in who actually submits winning designs: simply, more professional designers submit designs that get printed than do amateur designers.
Threadless Numbers #3
What kind of t-shirts don’t get printed at Threadless?
Because this feels like a lightweight week to me, I think it deserves a lightweight Threadless Numbers, and what’s more lightweight than losing Threadless t-shirts? But really I kid; there are a lot of good subs that never get printed, which leaves room for wonderful companies like Go Ape and Cotton Werks
What You Wanna Know
Who wins the t-shirt design contest on Threadless? What are their winning designs like? Well, I do have some information on that sort of thing, but…
What I”m Gonna Tell You
As of 5/6/2007, out of the most recent 120 submissions that had completed scoring, exactly 100% had been dropped from the scoring process after 24 hours because of the updated 1.5 rule. Basically, designs that aren’t scoring well get dropped after 24 hours. What this means is that most of the submissions to Threadless aren’t liked by the community. Which is why the focus of this “Threadless Numbers” is losing submissions.
Reasons for Disliking Designs
I couldn’t possibly claim to know why every person rates a particular t-shirt design poorly, but there are two large contributing factors that stood out to me in the comments of these designs: 1) the design is too similar to something previously printed by Threadless and 2) the design is too distant from Threadless’s aesthetic.
A case in point for a design too similar to a previous winner is Beware of the Innermost Gnome, which is reminiscent of Nesting Dolls (in that they both use nesting dolls for their gimmick). I wouldn’t be surprised to find that winning designs inspire others to think up designs that are somewhat similar; in this case, I think people probably didn’t like the colorway and naive style of Beware of the Innermost Gnome.
As for something that’s too different from Threadless’s aesthetic… I can’t find a single striking example, but if you’ve spent any time scoring t-shirt designs, I’m sure you’ll know what I’m talking about.
The Final Scores
The mean score of these submissions that had been dropped from the scoring process is 1.403, ranging from .99 for Skully to 1.72 for King. For comparison, the printed designs I discussed in Threadless Numbers #1 (above) averaged 2.772 (or 1.369 greater than the mean score for dropped designs), ranging from 2.17 for Le Voyage Dans la Lune to 3.77 for Can”t See the Forest but for the Socks.
Comments on Losing Submissions
Remember how I talked about the positive correlation between submissions’ final average scores and the number of comments they receive during scoring in “Threadless Numbers #1: Is it possible to predict how well a design submitted to Threadless will score?” Well, the average losing design receives 11 comments, compared to 91 for the winners. That these losing designs were dropped from the scoring process after 24 hours (or 1/7
What would this post be without a graph? Not much, that’s what. But don’t worry: no statistical analysis!
The number of comments (adjusted for being in the running for only one day) on dropped submissions is pretty random, and has a range almost as large as that for the printed designs. There are a couple things that would have made sense: 1) there would be more comments on subs that are either really good or really bad (or controversial), because somebody is more likely to have something to say about a design they feel strongly about than one for which they have a pretty neutral opinion. But without filling in the 1.7 to 2.2 range of scored submissions (which are generally either score too well to get dropped from the running or too poorly to get printed), it’s hard to say whether that’s the case.
2) Another possibility is the one which I originally proposed, that as the submission’s score increases, the number of comments the sub receives also increases. The correlation almost looks exponential, but having to adjust the number of comments for the dropped designs certainly introduces additional error, and missing a large portion of data is probably skewing the shape of the correlation; so the fit of an exponential curve is pretty bad.
I have no further generalizations to make at this time, save that more research needs to be done to make up for what mine lacks. If anybody wants to talk to me about doing their own research, I’d be more than happy to!
Threadless Numbers #4
Does being a Threadless “alumnus/a” give your submissions an advantage?
The question for this edition of Threadless Numbers, which will likely be the last for the foreseeable future, comes from MJ at Compete-tee-tion, and I thought it was a great one and quite a reasonable extension of the first “Threadless Numbers.” Let’s get on with it, then!
What”s at Stake
As I showed in the first “Threadless Numbers,” more comments are correlated with a higher score; could either getting more comments or a higher score (or both) be because the designer has had their submissions printed before (i.e. is an “alumnus/a”)?
Where Are the Numbers?
Sadly, nowhere. Okay, not nowhere… you can find them on the Threadless website. But I don’t have them. That isn’t going to stop me from talking a bit about this problem, though! I’m going to talk about the question, problems with it, and then lay out a specific project that would serve as a good beginning for anybody who might want to follow up on my research.
Spuriousness and Direction of Causality
There are some problems with trying to show that a design got either a lot of comments or a high score because they were a Threadless alum. It could be the case that a submission gets a higher score purely because the designer has a reputation of being “good” (as defined by the judgment of the Threadless community) because they’ve been printed before. (I’m dropping number of comments from this thought experiment because it’s simpler.) This story seems plausible. It might seem reasonable, then, to compare the scores of submissions from alum designers to those of unprinted designers. But there’s a glaringly huge problem with that: alum designers might have a higher average score not because they’ve had submissions printed in the past, but because their designs are “better” than those from unprinted designers. In other words, the correlation (if there is one) would be spurious, or caused by a third variable.
Let”s Try to Test It Anyway!
I wish we could! But here’s the thing: to control for the “goodness” of a submission we would have to quantify it. Besides the design’s average score (which is one of the variables we’re already trying to use in our correlation), how can we? We can’t use the comments on the sub because they are almost certainly caused by the same thing that causes it to have a higher average score. Unless the same design could be submitted under two users, one an alum and the other not, without the community somehow noticing (which is basically impossible), I can’t think of any other way. There is, however, something else we could do.
What Else Can We Do?
Although we can’t control for the “goodness” of a submission without operationalizing it somehow, we can still try to look at how Threadless alumnus/a status affects the number of comments on a submission, regardless of how “good” that submission is. Here’s how it would be set up: whether or not the designer of a submission is a Threadless alumnus/a is our categorical independent variable, while the number of comments on their post-alum submissions is our continuous dependent variable. Analysis would take the form of a comparison of means, controlling for the average score a design received.
Where Can I Get the Necessary Data?
I’ll tell you where! You’d have to look at designs that have completed scoring. Data for all three of the necessary variables can be obtained from this single page:
– the design’s average score (in the right-hand column)
– the number of comments the design received (in the right-hand column)
– whether or not the submitter is a Threadless alumnus/a (look for the little shield either next to a “This is my design” comment or on their profile page)
So what are you waiting for?