Do we need to rethink our site promotions and contests?

Question

Allow me to preface this with a bit of a disclaimer. I do not want to sound ungrateful or upset with the efforts of the SE team to support and promote this site. I think it's fantastic that SE is willing to create little contests and promotions and such for its sites, and I'm genuinely appreciative of the support they provide to all of the sites in the network. I very much enjoy Android Enthusiasts, and I don't see that changing any time soon.

Now, having said that, I feel the need to voice a bit of concern over the specifics of some of our site promotions. This was motivated initially by what I felt was an almost immediate flood of low-quality questions in audio for the current contest, so I decided to go back and do a little bit of datamining on the previous ones. What concerns me is that although we may be getting attention from the contests, we are:

1. Generally not retaining the users we attract
2. Causing a larger-than-usual influx of poor questions
3. (In my view) Run the risk of possibly disillusioning some of the more regular and loyal members

Now that we've had a few goes at some contests, I think we can start to analyze the results a bit. I'm going to go through the past contests in what should essentially be chronological order with a bit of data I collected by simply scanning the relevant tags and user profiles:

Kindle Fire Contest

This contest in some ways worked and in some ways felt like it kind of didn't. On the one hand, it brought attention to a lot of older unanswered questions, which is generally great. On the other hand, one user was so annoyed by what he perceived to be a flood of non-answers grabbing for loot, that he actually removed the tag from his own question so that it would not be part of the contest. There were certainly lots of cases of non-answers or half-answers attempting to get in on the contest, which provided a lot of clutter that people ended up needing to downvote since the rules dictated that anything with a score of zero was an eligible entry. In my opinion, this doesn't really encourage quality participation, but that's probably neither here nor there.

The winners of the contest also show some concerning stats. Of the 5 winners:

• 2 have not visited the site since the end of the contest
• 1 has visited but has made no other contributions to the site
  • Additionally, this user has only one post total (mind you, this is the post which won the contest for them) and it currently sits at a net score of -1 (!!)

Ice Cream Sandwich week

ICS week was the first tag-themed week. It was presented to the community on Meta before starting and received generally positive responses. Overall the week played out relatively well, but still seemed to encourage one-off posts to win sweet loot:

• 18 questions asked
• 1 of the 18 was closed
• 5 of the users (I count 14 total unique users, so ~35%) have no other activity on their profiles whatsoever
• The winning question ultimately had the 4.0-ice-cream-sandwich removed (!!) - It was quite arguably never about ICS to begin with.

Tablet week

Tablet week started immediately after ICS week, and was not presented to the userbase as a whole beforehand. It was also not one of the tags suggested in the ICS week's answers. I found this to be a bit unusual as tablet is a pretty broad tag, but it didn't seem like an unreasonable choice. This perhaps lent to the fact that there were very few entries (it's a bit of a low-traffic tag).

• 6 questions were asked (none closed)
• 1 user has no other posts on the site

Apps week

I have to be blunt here: this was a terrible tag choice, in my opinion. applications is absolutely notorious for being a dumping ground for questions where the author can't come up with any other relevant tags. It's vague, it's broad, and it attracts junk and (I think) should be actively discouraged whenever possible. I firmly believe that if this tag had been pitched to the community via Meta it would have been met with substantial resistance. The stats for this week were:

• 18 questions asked
• 3 closed
• 7 users (out of 17) have no other activity outside of the contest or the contest timeframe

Audio week

Similar to tablet week, this seems like an unusual tag choice to me. It is very broad, and it is also fairly low traffic (111 posts as of this writing, which includes the sudden influx of contest questions). Additionally, this week is based purely on views, which seems to further fly in the face of promoting high-quality contributions. Realistically, a winning entry needs only to be acceptable with effective link distribution. Audio week has also not fared so well, in my opinion, as it currently has:

• 13 questions asked
• 5 of the 13 (this is ~38%) have been closed, one has 3 close votes
• 7 of the users have (so far) no other activity on the site

I don't claim to have any solutions, but I do think that perhaps we should at least discuss the contests that have been run so far. As a community, I'm curious to know:

1. How do we feel about these contests and this data? Are we concerned? Do we consider these contests successful?
2. What other contest styles could we look at? Are there other options?
3. Should we be concerned about the fact that established, loyal users have no benefit over one-off throwaway accounts in these contests (which, in many cases appear to never return)?
4. Should the eligible posts be moderated more stringently or held to a higher standard than simply "Not a negative score and not closed"? Are view-based contests a good idea?

Answer 1

I've been lurking on Android-SE for months, on-and-off, mostly to flag a few things every day, to check for something I can answer (mostly non-existent, with the 1+ year old device I currently have). I've seen the promotions previously, and while a good idea generally (who wouldn't mind free stuff?!), I always thought something like "this probably won't work well enough". The thing is, such contests award only spurious contributions to the site in the space of a single week or so -- something that the analysis of eldarerathis proves. Moreover, it tends to reward not the loyal users, but rather some random people which were drawn to the site because of the promotion, not because they genuinely like the site.

You can't quite build a stable community this way, me thinking. What would work better is to reward long-lasting, positive contributions to the site, not only in the way of answering and asking questions, but general site governance. It's no big secret that meta participation is low here, chat is deserted, there are too many low-value questions and answers, the tags organization could use some love. Why not reward such activity instead?

For example, announce a "Device/manufacturer cleanup week", where all questions and answers about a particular handset or a group of related handsets are examined -- the obscure unanswerable ones are closed, some low-quality ones are deleted, answers are cleaned up, device-specific tags are removed where possible, etc. Some general suggestions following this train of thought:

• Announce this sufficiently in advance on meta and allow for opinions/suggestions/improvements before actually starting it.
• Devise a methodology of evaluating contributions (e.g. some sooper-seekrit SQL magic that collects number of retags/edits/deletions/helpful flags and spews a score of some kind). A scoreboard of sorts would be a good addition to promote competition.
• Restrict the contest to users that have joined the site X months/weeks ago and have scored a minimum of Y rep ("So, you want to win a new phone on Anroid-SE? Sorry kid, hang around for a while, help out with the site, and you'll have a running chance the next time around"). This will ward off drive-bys which only join the site because of the reward.
• Keep the rewards secret until the contest is over. If you don't know what you are going to win, whether it's random SE schwag signed by Jeff and Joel, or a new handset, you have the right motivation to participate -- you are not doing it for the reward itself, but rather to score nerd cred and help out a community you care about. After the contest is over, the SE team can decide what rewards are appropriate, judging from the overall level and quality of participation. Exceptional participation can be rewarded with a substantial award, and meh contributions can get a simple mug with Android logo, for example.

Finally, we don't need no stinking phones. No, really. I'd do all this kind of stuff for an Android-SE T-shirt. A shiny new phone doesn't seem a proper reward for spending basically a few minutes of one's time to answer a question. The more disproportionate the reward, the more people are inclined to cheat their way through to claim it. People won't join Android-SE only to get a T-shirt, but they will if the prize is an electronic gadget. That's why there is no significant positive effect from the competitions -- the prizes don't match the commitment. With cheaper prizes, the contest can be run much more frequently, every month even, and the SE financial investment would be pretty much the same.

And really finally, allow for converting a prize into a cash donation to a selected Android-related project. No use for a new phone if you already got one recently, it would probably end up on an eBay equivalent somewhere. I'd rather donate the full prize equivalent to a project than go through the hassle of selling unneeded stuff online to get only half of that.

Answer 2

There is so much text here, so I will keep it short:

I had the same thought. Tag based promotion just doesn't work well on AN.SX. IMHO the best promotion is purely reputation based, with a slight addition. Something like:

• Questions Week - earn XXX rep by asking questions within a week
• Answers Week - earn XXX rep by answering questions within a week
• New Users Week - every user between 0-200 rep, that earns XXX rep within a week

or just

• Reputation Week - earn XXX rep in a week

Where winners will be randomly chosen between the eligible users. Maybe this will also attract more users to edit the tag wiki to earn the rep.

Answer 3

I definitely agree that these are valid concerns. One of our biggest challenges is finding promotions that have an effect. It seems like we've found ways to generate a temporary spike in questions, but that doesn't necessarily translate into users that stick around. Our theory was that if you generate enough spikes, there will be the occasional user that sticks around. I don't think there has been enough time or enough contests to determine whether or not that is true, but I do think that promoting the site through a contest is a good thing, because it makes the site more visible. In other words, it's definitely good to look at how active people are after they enter a contest as one metric, but contests can also attract new users after the fact because of the new content.

That being said, it is important to make sure our questions and answers remain high quality, and contests that promote asking bad questions are always bad. That's why we decided to go with views for Audio Week instead of a random drawing. We figured that votes were more easily inflated than views, and that better questions would be clicked on more. However, it might be worthwhile to place more emphasis on votes in the future. Perhaps holding questions to a standard of "at least a score of 3" (or 5, or something) would be a better.

Answer 4

As a new user i know i don't have something to say. But personally i had never heard of this site before Android Guys wrote an article about your competition, and it was perfect because i had a question which i have looked for in the past month and haven't found any answers too. And suddenly i had a chance to both get a answer on my question and a chance to win a tablet.

It was a win win for me, and since i made my user here, i have tried to help a couple of users, and i am pretty sure i will stick here for a good while, because i like the way the site is build and how it is working.

So even though i'm a new user, and don't have anything to say, then you should know that the audio competition worked on me :)

[off topic]BTW: how do you run this site, there aren't any commercials on the site. Where do you get the money to run the server and buy the prices in the competitions? [/off topic]

Answer 5

Thanks for posting this; I agree fully.

Tags are tricky across all SE sites, but in my experience we do have somewhat unique problems with them here. Gaming has had issues with meta tags that we haven't, for example, but on the other hand most of their tags are really easy to evaluate. For example, their swtor tag is applied only to questions about The Old Republic and it's immediately obvious when a question should or shouldn't have the tag. Few would try to get away with tagging a StarCraft question with it in order to cheat at a contest, since they would have no chance of getting away with it.

That's not as true here. The closest we have to such specific tags are device tags, but they aren't supposed to be applied to every question involving the device. We get an immense number of questions where no one has any idea whether the issue is device-specific or not; it could be part of Android, specific to a chipset, specific to a custom UI or launcher, etc. Tags like audio are even worse. Does it cover call audio? Notifications? Music? With many tags we've attempted to address the distinctions in tag wikis, but no one reads them. It doesn't help that the contests provide no real advice on how the tag should be used (nor is that the right place for it, IMO).

I don't think promotions based on tags can work here. Device tags are the best candidates, yet bad overall: in addition to it being hard to determine when they should be used, I can't see a situation in which it makes sense to select a device tag. If the device is just being released, we won't get that many questions about it and anyone who already owns one probably wouldn't be too excited to win another. If the device hasn't been released, we'll get speculation and bad questions about easily-Googled announcements. If the device is old, no one will be interested. This applies to the OS version tags as well, really.

The drive-by user problem is pervasive here, in more than just contests. I think in order to combat this we need to have contests reward people for longer-term participation, so that they'll want to stay and be involved. I don't know the best way to go about that, but I think draw entries might work. Top user for the month? Get entered into a draw for the end of the year, winner gets the device of their choice. New user gaining 500+ rep in the first month? Win a case or a headset or other peripheral. Something like that. Good questions and good answers will also draw more traffic and thus more users; I think it's better to entice users organically with content instead of overtly with short-term promotions.

As for moderation, I did moderate more strictly during the Kindle Fire Contest. We had a few users being absolutely bombarded with crap and I was a bit liberal with deletions of bad and duplicate answers. For these other contests I've moderated just as I always do, but I agree that the relevant questions are disproportionately bad. I haven't looked at exact stats but you're right that many are flagged, closed, and improperly tagged. It's quite unpleasant to, effectively, tell someone that there were hidden terms to the contest (you have to use the tag appropriately, where appropriately is non-obviously defined) and their question doesn't qualify. It's even worse to see someone post crap and win instead of someone who posted a great question.

I also strongly agree that the team needs to take advantage of the community here. Grace Note, former Gaming mod and current SE employee, has helped with their promotions, but (as far as I can tell) there's no one even remotely as familiar with our site helping out with our promotions. Not taking advantage of us just doesn't make sense; even if the promotions were going well, making them better wouldn't hurt.

_{As an aside, I hate tablet with a passion and have been trying to figure out how to get rid of it. It's no better than if we had phone.}

---

Rather than make a novel in the comments I'm going to here respond to Lauren's comment:

I agree with most of your points here. Unfortunately, we don't have a Grace Note equivalent for the Android community. That's why we've made an effort to stay in contact with the moderators by posting on meta and via email. I know we've involved you personally in an email about upcoming promotions, and I'm always happy to talk to anyone who has ideas or feedback. Hopefully this thread will help us all talk and come up with a good solution.

Much of it has felt like a formality. The ICS contest you contacted us about was probably the least problematic, though certainly scaled back from the initial idea and (I think) correspondingly less successful. I think we'd all have benefitted from a bit more back-and-forth between the "hey there might be a thing" and "contest starts NOW!" communications.

By no means do I expect to be included or privy to what you're doing at SE; my job is just to moderate. What's really uncomfortable for me is being included just enough that I do have to acknowledge it, and at the same time knowing that if I had ever been asked about audio I would have been vehement in advising you against it. I'm left explaining it after the fact in a wall of text on Meta, which no doubt feels like an attack on you to some degree, for which I apologize. I don't doubt your intentions or abilities in any way; everyone I've ever been in contact with at SE has been awesome.

Right now I see two methods of dealing with the disjoint here:

• Promotions are designed on Meta. The team makes a post with an idea, the community gives feedback, the final form including complete rules and so on is determined before the contest ever starts. At least until we figure out a formula that works.
• Promotions are experiments. The team comes up with ideas and doesn't involve us with them, just acknowledging that things are experimental and there may be issues.

I don't really like the latter because we have so few involved users; when they have so many bad posts to clean up it gets tiring and it sort of feels like being dumped on. (This can be mitigated to some degree by contests that reward longer-term involvement.) Finding what works here is obviously difficult and it seems that trial and error will inevitably result in more to clean up.

The former has its problems too — it's slow and probably awkward for you, it may remove the element of surprise, Meta isn't built for back-and-forth, etc. Again I'm not sure what the best solution is here, but the current track doesn't seem to be working for anyone, and I think it's simply because, again, this stuff is hard. I'd feel quite bad if anyone came away from this feeling that they were to blame, because I don't think that's the case.

Answer 6

What did we end up with? Which solution should be used? I like the idea about the highest number of reputation points in a week..

What did the staff decide?