Recruiting subjects via CentralNotice: feedback needed - RCom-l

Daniel Mietchen

26 Jun 26 Jun

10:20 p.m.

Dear all, I had a look at the documents and find their approach reasonable. What I am not convinced about is the necessity of running the notice for two weeks - I would expect that getting 2000 completed questionnaires (even for 25min) out of the whole population of all Wikimedians should take less time. Anyway, I volunteer to take the survey and to report back to RCom as to how fit I see the survey design for the intended research purpose. As for the implications of using CentralNotice this time, we could think of limiting the number and length of slots available for such purposes for a certain period of time. Cheers, Daniel On Thu, Jun 23, 2011 at 7:44 PM, Dario Taraborelli <dtaraborelli(a)wikimedia.org> wrote:

...

All, I received an urgent request by a team of researchers who have been waiting for several months for an approval to run a survey of Wikipedia editors [1]. The team includes researchers based at Harvard and Sciences Po. One RCom member (Mayo) is also involved in this project. The project aims to recruit participants by running banners for logged-in users on the English Wikipedia and the team previously sought community consensus for this recruitment method via a discussion on the Admin Noticeboard [2]. The planning of this project predates the creation of the Research Committee and the community discussion was not explicitly mentioned in the project page, as a result most people on RCom (myself included) were entirely unaware of it. Over the last weeks we have successfully been able to channel new recruitment requests to Meta, where RCom and community members can discuss proposed recruitment methods for various studies. I told Jérôme and Mayo from the Sciences Po/Harvard team that their project should be no exception and I would like to solicit RCom members to comment on this proposal over the next days. The request is sensitive not only for its tight timeline (there was apparently a commitment to get the banner campaign started today and the team has already allocated engineers to this project next week), but also because it would be the first time ever that we use the CentralNotice for research projects, and this is a decision that may have implications for future studies. I'd like to have your thoughts on this proposal via its discussion page by Monday night (Pacific time) at the latest. I am particularly interested in hearing on this from community members (such as Ziko, WereSpielChequers, Milos, Steven). Because of her direct involvement in the project, Mayo won't be participating in the discussion. I attach below a letter that Jérôme addressed to the Research Committee to document the history of this project. I look forward to hearing from you. Best Dario [1] http://meta.wikimedia.org/wiki/Research:Dynamics_of_Online_Interactions_and… [2] http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Archiv… Begin forwarded message: From: Jérôme HERGUEUX <jerome.hergueux(a)sciences-po.org> Date: June 22, 2011 6:28:19 PM PDT To: dtaraborelli(a)wikimedia.org Cc: Mayo.Fuster(a)eui.eu Subject: Follow up Mayo chat Reply-To: Jérôme HERGUEUX <jerome.hergueux(a)sciences-po.org> x-mailer: ContactOffice Mail Dear Dario, Thank you for helping us out with this! Please find attached: 1. A document explaining rapidly the history of our research project and where we stand now. 2. The original code of the banner we prepared to advertize the study (normally the folks coding the banners at the Foundation should be aware of it). I've linked to our Early research protocol discussion on AN in the link section of our Talk Page on Meta: http://meta.wikimedia.org/wiki/Research_talk:Dynamics_of_Online_Interaction… I'll try to add some relevant info to our research project template very soon. Best, Jérôme (User:SalimJah). ________________________________ -------------------------------------------------------------------------- Tous les courriers électroniques émis depuis la messagerie de Sciences Po doivent respecter des conditions d'usages. Pour les consulter rendez-vous sur http://www.ressources-numeriques.sciences-po.fr/confidentialite_courriel.htm _______________________________________________ RCom-l mailing list RCom-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l

Reply

Dario Taraborelli

27 Jun 27 Jun

5:27 p.m.

I understand they are only targeting enwiki (Mayo, can you confirm?) and CentralNotice can be configured to serve language and project-specific banners. Dario On Jun 26, 2011, at 3:21 PM, Daniel Mietchen wrote:

...

Forgot to mention: Another question I am not clear about is whether the survey is available in all the languages covered by CentralNotice.

Reply

Daniel Mietchen

8:33 p.m.

Hi all, I just took the survey. Here are my observations: Paypal account: why ask before the survey is completed? And why make entering an email address compulsory if there is an option to donate the money to the WMF, under which there would be no need for an email address (other than to control for duplicate entries, perhaps). There is ambiguity in questions of the "did you have an idea" kind about other participants' behaviour: Although it sounded more like "did you have some imagination as to the probability distribution of play behaviour amongst other players", I took it to mean "did you *know* how the others played?". In my view, either question only makes sense as a way to estimate how carefully one has read the instructions (in which the correct answer was given, although only roughly for the first). I was surprised (and disappointed) that the game theoretical questions ended so quickly after all the effort that went into explaining (and understanding) the rules. Also, such decision-making scenarios tend to be different between single-play and repeated-play conditions, and in the latter case, depending upon whether the players (or their behaviour) change or not, and whether this is known to the others. The demographic part is fairly standard, though some questions (e.g. on country of citizenship, birth, birth of mother, birth of father) seem a bit far-fetched for an analysis of this kind (and it's annoying to enter if all four are the same). So here we have indeed the case of "repeated surveys with similar questions", as WereSpielChequers had put it. The questions right after the demographic part are probably the most relevant ones from an RCom perspective, and they are posed mostly such that their analysis promises interesting results. There was one that I found a bit odd, but can't remember any details now. The question on monthly gross income could do with a currency converter (not everyone, even on enwiki, thinks in USD). Typo: * Where you in a calm environment when you answered the questions? The technical implementation is excellent, and it took me exactly the 25min that had been advertised. Daniel On Mon, Jun 27, 2011 at 7:54 PM, Fuster, Mayo <Mayo.Fuster(a)eui.eu> wrote:

...

Hello!

I understand they are only targeting enwiki (Mayo, can you confirm?)

Yes, that's right. Have a nice day, Mayo _______________________________________________ RCom-l mailing list RCom-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l

Reply

WereSpielChequers

26 Jun 26 Jun

10:46 p.m.

I've made some specific comments on the talkpage, but hadn't appreciated that they were planning to run this for two weeks. Perhaps naively I'd assumed the plan was to switch this off once they'd got 2,000 responses. The more I think about this and the numbers involved, the more I move to the idea that we should be expanding our existing survey into an omnibus survey that carries questions for legitimate researchers like these. That way the researchers would benefit from a larger more robust dataset and the editors benefit from questions not being duplicated, research not becoming spam and from greater confidence re privacy. The researchers would get more data - perhaps 5,000 individual but anonymised records with common questions such as age band, education level, motive for editing etc, and of course their own question or questions - dress size, operating system, number of times editor has been abducted by UFOs or whatever. My experience is that if people are persuaded to do a survey they won't be bothered at the length so much as they are by questions they can't answer without looking something up or talking to someone else. But repeated surveys with similar questions are a turnoff. WSC On 26 June 2011 23:20, Daniel Mietchen <daniel.mietchen(a)googlemail.com> wrote:

...

Dear all, I had a look at the documents and find their approach reasonable. What I am not convinced about is the necessity of running the notice for two weeks - I would expect that getting 2000 completed questionnaires (even for 25min) out of the whole population of all Wikimedians should take less time. Anyway, I volunteer to take the survey and to report back to RCom as to how fit I see the survey design for the intended research purpose. As for the implications of using CentralNotice this time, we could think of limiting the number and length of slots available for such purposes for a certain period of time. Cheers, Daniel On Thu, Jun 23, 2011 at 7:44 PM, Dario Taraborelli <dtaraborelli(a)wikimedia.org> wrote:

All, I received an urgent request by a team of researchers who have been waiting for several months for an approval to run a survey of Wikipedia editors [1]. The team includes researchers based at Harvard and Sciences Po. One RCom member (Mayo) is also involved in this project. The project aims to recruit participants by running banners for logged-in users on the English Wikipedia and the team previously sought community consensus for this recruitment method via a discussion on the Admin Noticeboard [2]. The planning of this project predates the creation of the Research Committee and the community discussion was not explicitly mentioned in the project page, as a result most people on RCom (myself included) were entirely unaware of it. Over the last weeks we have successfully been able to channel new recruitment requests to Meta, where RCom and community members can discuss proposed recruitment methods for various studies. I told Jérôme and Mayo from the Sciences Po/Harvard team that their project should be no exception and I would like to solicit RCom members to comment on this proposal over the next days. The request is sensitive not only for its tight timeline (there was apparently a commitment to get the banner campaign started today and the team has already allocated engineers to this project next week), but also because it would be the first time ever that we use the CentralNotice for research projects, and this is a decision that may have implications for future studies. I'd like to have your thoughts on this proposal via its discussion page by Monday night (Pacific time) at the latest. I am particularly interested in hearing on this from community members (such as Ziko, WereSpielChequers, Milos, Steven). Because of her direct involvement in the project, Mayo won't be participating in the discussion. I attach below a letter that Jérôme addressed to the Research Committee to document the history of this project. I look forward to hearing from you. Best Dario [1] http://meta.wikimedia.org/wiki/Research:Dynamics_of_Online_Interactions_and… [2] http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Archiv… Begin forwarded message: From: Jérôme HERGUEUX <jerome.hergueux(a)sciences-po.org> Date: June 22, 2011 6:28:19 PM PDT To: dtaraborelli(a)wikimedia.org Cc: Mayo.Fuster(a)eui.eu Subject: Follow up Mayo chat Reply-To: Jérôme HERGUEUX <jerome.hergueux(a)sciences-po.org> x-mailer: ContactOffice Mail Dear Dario, Thank you for helping us out with this! Please find attached: 1. A document explaining rapidly the history of our research project and where we stand now. 2. The original code of the banner we prepared to advertize the study (normally the folks coding the banners at the Foundation should be aware of it). I've linked to our Early research protocol discussion on AN in the link section of our Talk Page on Meta: http://meta.wikimedia.org/wiki/Research_talk:Dynamics_of_Online_Interaction… I'll try to add some relevant info to our research project template very soon. Best, Jérôme (User:SalimJah). ________________________________ -------------------------------------------------------------------------- Tous les courriers électroniques émis depuis la messagerie de Sciences Po doivent respecter des conditions d'usages. Pour les consulter rendez-vous sur http://www.ressources-numeriques.sciences-po.fr/confidentialite_courriel.htm _______________________________________________ RCom-l mailing list RCom-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l

_______________________________________________ RCom-l mailing list RCom-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l

Reply

WereSpielChequers

28 Jun 28 Jun

12:41 p.m.

With Omnibus surveys there are tradeoffs - some people may have incompatible questions, though things like age and education bands should be able to resolve that. The targetting can also be different - but a filter should resolve that - if one researcher doesn't want a subset of participants then providing their selection criteria is a survey variable they should be able to exclude the ones they don't want. And I would predict that a big advantage to the researcher would be a greater response to a WMF officail survey - especially if it was an annual event. WSC On 27 June 2011 18:25, Dario Taraborelli <dtaraborelli(a)wikimedia.org> wrote:

...

Daniel, WSC – thanks for your feedback: I agree there is no reason for the banner to run for 2 weeks if they get their complete 2K sample before then. The omnibus survey also sounds like an interesting plan, but I suspect many researchers may feel uncomfortable not to have control on the exact design of the survey. On the other hand we definitely need to find ways to avoid redundant recruitment requests. I'll add a specific item on this issue to the agenda of the next RCom meeting. Dario On Jun 26, 2011, at 3:46 PM, WereSpielChequers wrote:

I've made some specific comments on the talkpage, but hadn't appreciated that they were planning to run this for two weeks. Perhaps naively I'd assumed the plan was to switch this off once they'd got 2,000 responses. The more I think about this and the numbers involved, the more I move to the idea that we should be expanding our existing survey into an omnibus survey that carries questions for legitimate researchers like these. That way the researchers would benefit from a larger more robust dataset and the editors benefit from questions not being duplicated, research not becoming spam and from greater confidence re privacy. The researchers would get more data - perhaps 5,000 individual but anonymised records with common questions such as age band, education level, motive for editing etc, and of course their own question or questions - dress size, operating system, number of times editor has been abducted by UFOs or whatever. My experience is that if people are persuaded to do a survey they won't be bothered at the length so much as they are by questions they can't answer without looking something up or talking to someone else. But repeated surveys with similar questions are a turnoff. WSC On 26 June 2011 23:20, Daniel Mietchen <daniel.mietchen(a)googlemail.com> wrote:

Dear all, I had a look at the documents and find their approach reasonable. What I am not convinced about is the necessity of running the notice for two weeks - I would expect that getting 2000 completed questionnaires (even for 25min) out of the whole population of all Wikimedians should take less time. Anyway, I volunteer to take the survey and to report back to RCom as to how fit I see the survey design for the intended research purpose. As for the implications of using CentralNotice this time, we could think of limiting the number and length of slots available for such purposes for a certain period of time. Cheers, Daniel On Thu, Jun 23, 2011 at 7:44 PM, Dario Taraborelli <dtaraborelli(a)wikimedia.org> wrote:

All, I received an urgent request by a team of researchers who have been waiting for several months for an approval to run a survey of Wikipedia editors [1]. The team includes researchers based at Harvard and Sciences Po. One RCom member (Mayo) is also involved in this project. The project aims to recruit participants by running banners for logged-in users on the English Wikipedia and the team previously sought community consensus for this recruitment method via a discussion on the Admin Noticeboard [2]. The planning of this project predates the creation of the Research Committee and the community discussion was not explicitly mentioned in the project page, as a result most people on RCom (myself included) were entirely unaware of it. Over the last weeks we have successfully been able to channel new recruitment requests to Meta, where RCom and community members can discuss proposed recruitment methods for various studies. I told Jérôme and Mayo from the Sciences Po/Harvard team that their project should be no exception and I would like to solicit RCom members to comment on this proposal over the next days. The request is sensitive not only for its tight timeline (there was apparently a commitment to get the banner campaign started today and the team has already allocated engineers to this project next week), but also because it would be the first time ever that we use the CentralNotice for research projects, and this is a decision that may have implications for future studies. I'd like to have your thoughts on this proposal via its discussion page by Monday night (Pacific time) at the latest. I am particularly interested in hearing on this from community members (such as Ziko, WereSpielChequers, Milos, Steven). Because of her direct involvement in the project, Mayo won't be participating in the discussion. I attach below a letter that Jérôme addressed to the Research Committee to document the history of this project. I look forward to hearing from you. Best Dario [1] http://meta.wikimedia.org/wiki/Research:Dynamics_of_Online_Interactions_and… [2] http://en.wikipedia.org/wiki/Wikipedia:Administrators%27_noticeboard/Archiv… Begin forwarded message: From: Jérôme HERGUEUX <jerome.hergueux(a)sciences-po.org> Date: June 22, 2011 6:28:19 PM PDT To: dtaraborelli(a)wikimedia.org Cc: Mayo.Fuster(a)eui.eu Subject: Follow up Mayo chat Reply-To: Jérôme HERGUEUX <jerome.hergueux(a)sciences-po.org> x-mailer: ContactOffice Mail Dear Dario, Thank you for helping us out with this! Please find attached: 1. A document explaining rapidly the history of our research project and where we stand now. 2. The original code of the banner we prepared to advertize the study (normally the folks coding the banners at the Foundation should be aware of it). I've linked to our Early research protocol discussion on AN in the link section of our Talk Page on Meta: http://meta.wikimedia.org/wiki/Research_talk:Dynamics_of_Online_Interaction… I'll try to add some relevant info to our research project template very soon. Best, Jérôme (User:SalimJah). ________________________________ -------------------------------------------------------------------------- Tous les courriers électroniques émis depuis la messagerie de Sciences Po doivent respecter des conditions d'usages. Pour les consulter rendez-vous sur http://www.ressources-numeriques.sciences-po.fr/confidentialite_courriel.htm _______________________________________________ RCom-l mailing list RCom-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l

_______________________________________________ RCom-l mailing list RCom-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/rcom-l

Reply