Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in...

43
Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia DR CNRS @ LIG [email protected] joint work with: Soudip Roy Chowdhury (INRIA), Carlos Castillo, Muhammad Imran (QCRI), Muhammad Rizwan Asghar (Saarland U.), Senjuti Basu Roy (UW Tacoma), Gautam Das, Habibur Rahman, Saravanan Thirumuruganathan (UT Arlington) SAGEO conference Grenoble Nov 26 th , 2014

Transcript of Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in...

Page 1: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

Task Assignment Optimization in Crowdsourcing

(and its applications to crisis management)

Sihem Amer-Yahia DR CNRS @ LIG

[email protected] joint work with:

Soudip Roy Chowdhury (INRIA), Carlos Castillo, Muhammad Imran (QCRI), Muhammad Rizwan Asghar (Saarland U.), Senjuti Basu Roy (UW Tacoma),

Gautam Das, Habibur Rahman, Saravanan Thirumuruganathan (UT Arlington)

SAGEO conference Grenoble Nov 26th, 2014

Page 2: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Traditional Media

Today’s Reporting

Crisis awareness and reporting

Onsite reporters Tweets, text messages, news reports

Report timeliness Mixed (reliance on experts; limited resources)

Report quality ✔ Mixed (noisy; rumor and misinformation)

Report cost High ✔

Page 3: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Traditional Media

Today’s Reporting

Future Reporting?

Crisis awareness and reporting

Onsite reporters Tweets, text messages, news reports

Online tweet analysis discovers and follows course of incident

Report timeliness Mixed (reliance on experts; limited resources)

✔ ✔ (if detection of crisis is done on time)

Report quality ✔ Mixed (noisy; rumor and misinformation)

✔ (via careful assignment)

Report cost High ✔ ✔ (within budget)

Page 4: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

CrowdMap from Ushahidi

•  Ushahidi is a non-profit software company that develops free software for information collection, visualization, and interactive mapping via citizen journalism.

•  Started with Kenya's disputed 2007 presidential election that collected eyewitness reports of violence reported by email and text message and placed them on a map.

•  CrowdMap: a place where volunteers go to “check-in” and add data about incident.

•  Future option for CrowdMap: start with a world map that shows crisis hotspots (determined by geo-tagged data).

Page 5: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

CrowdMap from Ushahidi

Page 6: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Talk outline

1.  Quick overview of existing crowdsourcing 2.  Task assignment 3.  Online tweet monitoring

Page 7: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Crowdsourcing

•  Crowdsourcing: a variety of tasks –  Micro-tasks: data gathering (e.g. picture/video tagging, opinion

solicitation (e.g. restaurant ratings,) –  Collaborative tasks: document editing (e.g., Wikipedia), creative

design, fansubbing, solution outsourcing (e.g., Netflix contest)

•  Existing systems –  Platforms: AMT, Turkit, Innocentive, CloudFlower, etc. –  Crowd: volatile, asynchronous arrival/departure, various levels of

attention/accuracy/expertise

•  3 primary processes –  Worker skill estimation –  Worker-to-task assignment –  Task accuracy evaluation

Page 8: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Challenges

•  Who Evaluates What and How? •  How to Estimate Worker Skills? •  How to Assign Tasks to Workers? •  How to do all of the above efficiently?

•  Magnified by: –  Human factors –  Scale

Page 9: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Limitations

•  Developing a crisis-specific platform for each disaster situation is costly

•  Recent research undertakes some challenges in silo, for specific cases: e.g. real-time crowdsourcing, highly volatile crowds, single worker skill –  Active learning strategies for task accuracy improvement [Boim

et. Al. 2012, Krager et. al. 2011, Ramesh et. al. 2012] –  Worker-to-task-assignment [Ho et. al. 2012]

•  Human involvement introduces uncertainty –  Worker availability –  Worker wage: deviations even among persons of the same

profile, due to workload, time –  Worker skill: may decline with workload, change with motivation

Page 10: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Talk outline

1.  Quick overview of existing crowdsourcing 2.  Task assignment 3.  Online tweet monitoring

Page 11: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

Optimization in Knowledge-Intensive

Crowdsourcing @VLDBJ 2015 (to appear)

Page 12: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Integrated Architecture

Skill discovery

and evaluation

Task-to-worker

assignment

Accuracy/ efficiency increases

Workers enter

Complete tasks

All kinds of tasks

Page 13: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Expressing Task Assignment

•  Input: tasks to complete, human workers •  Output: completed tasks

•  Each task has skill/quality/cost requirements •  Each worker has human factors: skill, expected

wage, acceptance ratio

•  Desirable properties: –  Task-centric: high quality tasks (relevant workers), low cost –  Worker-centric: balanced workload, good incentive (high pay,

relevant tasks) –  System-centric: low latency

Page 14: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Example: Maximize task quality under task-centric and worker-centric constraints

objective: maximize aggregated vt

aggregated worker skills and wages task quality constraint

task budget

Page 15: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Task Assignment Solution Overview

Task Assignment Problem is NP-hard (reduction using Multiple-Knapsack Problem) Our approach: •  Offline – Index Building for a workload of tasks •  Online – Index Maintenance when tasks occurs

–  How to replace a worker who is not available or does not accept a task?

Page 16: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Optimal Solution Offline Index Building IP-based

Page 17: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Approximation Solution for Offline Index Building •  Objective function submodular and becomes

monotonic when W2 = 0 •  Contribution to index building

–  A greedy deterministic algorithm with a 1-1/e approximation factor when submodular and monotonic

–  A greedy randomized algorithm with a 2/5 approximation factor when submodular

•  Contribution to index maintenance –  Solve a marginal IP –  Cluster workers to reduce size

Page 18: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Experimental Evaluation •  Quality Experiments using multiple Applications

–  Collaborative Document Editing –  20 workers asked to produce reports on 5 different topics: 1) Political unrest in Egypt, 2) NSA document leakage, 3) Playstation games, 4) All electric cars and 5) Global warming

•  Scalability experiments –  A collaborative crowd simulator

Page 19: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Quality Experiments •  A total of 230 workers hired on AMT

•  A set of 8 multiple choice questions per task, to assess skills

•  Study conducted in multiple phases –  Phase1- Skill and Cost of workers learned using benchmark

dataset –  Phase2- Task assignment –  Phase3- Completed tasks evaluated by crowd workers

Page 20: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

AMT worker distributions (Egypt task)

Page 21: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

AMT worker distributions (Egypt task)

Page 22: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Quality Assessment

•  Scale of 1-5 by 150 AMT workers •  Compared to Benchmark and Online-greedy

Page 23: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Summary of Quality Experiments (from translation task) •  Higher affinity impacts positively quality •  A large group (beyond size 10) is less effective •  Region-based affinity is more effective than age-

gender based

Page 24: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Opportunities

Page 25: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Traditional media

Basic Crowdsourcing

ECCO

Crisis awareness and reporting

Onsite reporters Tweets, text messages, news reports

Online tweet analysis discovers and follows course of incident

Task timeliness Mixed (reliance on experts but limited resources)

✔ ✔ (if detection of crisis is done on time)

Task quality ✔ Mixed (noisy; rumor and misinformation)

✔ (via careful assignment)

Task cost High ✔ ✔ (within budget)

Page 26: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Talk outline

1.  Quick overview of existing crowdsourcing 2.  Task assignment

–  Flexible optimization framework with task-centric, worker-centric goals and constraints

–  Applicable to collaborative tasks 3.  Online tweet monitoring

Page 27: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

Tweet4act Using Incident-Specific Profiles for

Classifying Crisis-Related Messages @ISCRAM 2013

Page 28: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Classifying incident-related tweets

-  Caution, warnings -  Alerts etc.

-  Damage -  Causalities etc.

-  Request for help -  Donations etc.

1.  Identify messages related to an incident. 2.  Classify incident-messages with the corresponding

period (PRE, DURING, POST) 3.  Apply it to data on the fly

Page 29: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Disaster-related Tweets

•  [PRE] New #tropical storm forms in the West #Pacific. #Nesat may hit the #Philippines & #China as a #typhoon next week

•  [DURING] @Yahoo News: Powerful #typhoon with winds up to 106 mph makes landfall in #Philippines as 100,000 odered to fless homes

•  [POST] News5 Action center is now accepting donations for the victims of Typhoon “pedring. Drop boxes are located @ TV5 Office :)

Page 30: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Datasets Datasets collected using the Twitter Streaming API with appropriate hashtags: those announced by the crisis management authorities at the time of an incident. 1.  Joplin Tornado on May 22, 2011: 1500 tweets 2.  Haiti Earthquake on Jan 12, 2010: 1500 tweets

3.  Nesat Typhoon in Philipines on Sep 27, 2011: 500 tweets

Page 31: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Tweet4Act

Collection -> Filtering -> Period Classification

Page 32: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

1. Filtering Process

•  Cleaning: up to 13% are not incident-related •  Outlier detection

–  Normalize message text: remove the “RT @username ” and “@username ” prefixes.

–  Remove duplicate messages after normalization. –  Remove all terms that appear in less than a fraction s = 0.05 of messages. –  Run the k-medoid clustering algorithm on each dataset. –  Discard clusters having a negative number or zero as silhouette coefficient. –  Select from each cluster the fraction m of messages closer to the medoid.

•  Result = top-m fraction of the most representative messages from each cluster

•  1,198, 1,167 and 373 unique messages in Joplin, Haiti and Nesat datasets respectively

Page 33: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Validation on CrowdFlower

•  Precision –  Tweets identified as crisis-related by our method: 498 tweets

from Joplin, 250 from Haiti, 200 from Nesat. –  Each task, which also consists of a set of correctly labeled

tweets (i.e., golden data), asked workers to choose if a tweet is crisis-related or not.

–  3 votes/tweet.

•  Recall –  Random samples of 231, 220, and 244 tweets from Joplin,

Haiti and Nesat (before applying the filter). –  Manually labeled those messages as crisis-related or not.

Page 34: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Filtering Process Validation

Using CrowdFlower

Page 35: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

2. Dictionary-based Period Classification

•  Compares the words in each message against a dictionary of words known to be present in specific periods of a crisis-incident

•  Most frequent words across datasets •  “warning” & “alert” in PRE •  “now”, “sweeps” in DURING •  “aftermath”, “donate” in POST

Page 36: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

3. NLP-Based Period Classification

Tense of verbs help identify period (A. Iyengar et al., 2011) 1. If the word is listed in the dictionary, add +1 to the period it is listed under and stop processing that word (i.e., if a verb is in the dictionary, we ignore it below). 2. If the word is an auxiliary verb, add +1 to the period it is associated (e.g., could-PRE, are-DURING, did-POST). 3. If the word is a verb in future/present/past tense, add +0.5 to pre/during/post period, respectively. 4. Sum up scores of each period across all words in the phrase and pick the period with the largest sum.

Page 37: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Simple Scoring Example

NFL teams gathering supplies aid for tornado victims in Kansas Missouri (Morning Call) … •  In this message, both words “aid ” and “victim ” are

matched in the dictionary for the POST period. •  The verb “gathering ” is in continuous form and

contributes to the DURING period. •  In total, the message has +2 score for POST and +0.5

for DURING; hence, it is classified as POST .

Page 38: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Manual Period Classification (labeling)

CrowdFlower period labeling

! ! !

Page 39: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

Performance of Tweet4Act Period Tweet4act SVM MaxEnt Tree RF

P R F1 P R F1 P R F1 P R F1 P R F1 Joplin Tornado

PRE 0.33 0.85 0.48 0.00 0.00 0.00 0.43 0.21 0.28 0 0 0 0 0 0

DURING

0.88 0.89 0.89 0.32 0.91 0.47 0.35 0.55 0.43 0.3 0.73 0.43 0.32 0.1 0.48

POST 0.61 0.84 0.71 0.67 0.20 0.31 0.55 0.6 0.57 0.57 0.4 0.47 1.00 0.1 0.18

AVG. 0.61 0.86 0.69 0.33 0.37 0.39 0.44 0.45 0.42 0.29 0.38 0.45 0.66 0.37 0.33

Haiti Earthquake

PRE 0.63 1.00 0.77 1.00 0.67 0.80 1.00 1.00 1.00 1 0.67 0.8 1.00 0.33 0.5

DURING

0.72 0.97 0.83 0.75 0.6 0.67 0.67 0.80 0.73 0.6 0.6 0.6 1.00 0.4 0.57

POST 0.46 0.82 0.59 0.92 0.97 0.94 0.97 0.95 0.96 0.92 0.95 0.93 0.88 1.00 0.94

AVG. 0.60 0.87 0.71 0.89 0.74 0.80 0.88 0.91 0.89 0.84 0.74 0.78 0.96 0.58 0.67

Nesat Typhoon

PRE 0.36 1.00 0.53 1.00 0.50 0.67 1.00 0.50 0.67 0.33 0.25 0.28 1.00 0.5 0.67

DURING

0.94 0.94 0.94 0.79 1.00 0.88 0.81 1.00 0.90 0.71 0.77 0.74 0.79 1 0.88

POST 0.52 0.85 0.65 1.00 0.2 0.33 1.00 0.40 0.57 0 0 0 1.00 0.2 0.33

AVG. 0.61 0.93 0.71 0.93 0.57 0.62 0.94 0.63 0.71 0.35 0.34 0.51 0.93 0.57 0.63

PRE

PRE

PRE

DURING

DURING

DURING

POST

POST

POST

AVG

AVG

AVG

v  Tweet4act provides better recall over baseline algorithms for all datasets

v  But, it misses some precision v  Haiti (0.60 against 0.96 by RF) v  Nesat (0.61 against 0.94 by

MaxEnt)

Page 40: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Talk outline

1.  Quick overview of existing crowdsourcing 2.  An integrated architecture for automatic task

assignment 3.  Online tweet monitoring

–  Classified tweets into PRE/DURING/POST

4.  Summary and future work

Page 41: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Summary and Future Work

1.  Crowdsourcing is a powerful paradigm to help in crisis reporting during and after it happens

2.  Implicit reporting serves task awareness 3.  Explicit reporting with recurring crowds opens new

research opportunities for effective task assignment: to report on crises, to participate in task evaluation

4.  Task assignment is effective when skill learning and task evaluation are possible

5.  All that is only possible with a general-purpose crowdsourcing platform

Page 42: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

Workers

Research Center for Natural Hazards & Disaster Recovery

Niigata University

Better Task Design & Algorithms

Task Results Real Data on Natural Disasters (Currently, data on Typhoon Wipha in 2013)

Algorithms to Help in Future Natural Disasters

Page 43: Task Assignment Optimization in Crowdsourcing (and its ... · Task Assignment Optimization in Crowdsourcing (and its applications to crisis management) Sihem Amer-Yahia ... Tweet4act

SAGEO 2014

References [Boim et. Al] R. Boim, O. Greenshpan, T. Milo, S. Novgorodov, N. Polyzotis, and W. C. Tan. Asking the right questions in crowd data sourcing. In ICDE, pages 1261–1264, 2012.

[Krager et. Al] D. R. Karger, S. Oh, and D. Shah. Budget-optimal task allocation for reliable crowdsourcing systems. CoRR, abs/1110.3564, 2011.

[Ramesh et. al.] A. Ramesh, A. Parameswaran, H. Garcia-Molina, and N. Polyzotis. Identifying reliable workers swiftly. Technical report, Stanford University. 2012.

[Ho et. al.] C.-J. Ho and J. W. Vaughan. Online task assignment in crowdsourcing markets. In AAAI, 2012. Senjuti Basu Roy, Ioanna Lykourentzou, Saravanan Thirumuruganathan, Sihem Amer-Yahia, Gautam Das: Crowds, not Drones: Modeling Human Factors in Interactive Crowdsourcing. DBCrowd 2013: 39-42 Senjuti Basu Roy, Ioanna Lykourentzou, Saravanan Thirumuruganathan, Sihem Amer-Yahia, Gautam Das : Optimization in Knowledge-Intensive Crowdsourcing. CoRR abs/1401.1302 (2014) Senjuti Basu Roy, Saravanan Thirumuruganathan, Sihem Amer-Yahia, Gautam Das, Cong Yu: Exploiting group recommendation functions for flexible preferences. ICDE 2014: 412-423 Atsuyuki Morishima, Sihem Amer-Yahia, Senjuti Basu Roy. “Crowd4U: An Initiative for Constructing an Open Academic Crowdsourcing Network”, HCOMP 2014. Senjuti Basu Roy, Sihem Amer-Yahia,”It Takes a Village” - A Raging Vision to Enable Collaboration in Human Intensive Applications (under review) Habibur Rahman, Senjuti Basu Roy, Ioanna Lykourentzou, Saravanan Thirumuruganathan, Sihem Amer-Yahia, Gautam Das, Optimization in Collaborative Crowdsourcing (under review) A. Iyengar, T. Finin, and A. Joshi (2011) Content-based prediction of temporal boundaries for events in Twitter. In Proceedings of the Third IEEE International Conference on Social Computing. K. Starbird, L. Palen, A. Hughes, and S. Vieweg (2010) Chatter on the red: what hazards threat reveals about the social life of microblogged information. In Proceedings of the 2010 ACM conference on Computer supported cooperative work, pages 241–250. ACM. Latonero, Mark, and Irina Shklovski. "“Respectfully Yours in Safety and Service”: Emergency Management & Social Media Evangelism.” Proceedings of the 7th International ISCRAM Conference– Seattle. Vol. 1. 2010.