Skip to content

Latest commit

 

History

History
50 lines (37 loc) · 2.1 KB

README.md

File metadata and controls

50 lines (37 loc) · 2.1 KB
Raw

Code for "Prevalence and prevention of large language model use in crowd work"

./experiments

Here we share the HTML + JS files we used to create the human intelligence task.

  • ./html_study1.html: summarization task for study #1. Note that it only works with a url parameter PROLIFIC_ID.
  • ./html_study2.html: summarization task for study #2. Note that it only works with a url parameter PROLIFIC_ID.
  • ./html_survey.html: post-task survey for study #2.
  • ./images_better/: these are the images used for study #2.

./data

The original data collected by Prolific can be found here. in this folder, we share this very data augmented with metrics related to LLM use.

Study-1-augmented.csv.gz

Rows kept from the original file:

  • original_text: original summary
  • key_strokes: Key strokes, separated by spaces.
  • log_of_what_they_did: Key strokes. Broken up into [keystroke, timestamp].
  • datetime: Time when completed.
  • text: Text summary.
  • started_at: When task was started
  • completed_at: When task was completed.
  • time_taken: How many seconds it took to complete the task.
  • total_approvals: Total number of times the workers has had a task approved.
  • Demographic-related columns: age, sex, ethnicity_simplified, country_of_birth, Nationality, language, student_status, employment_status.

Rows added:

  • y_cc: classifier prediction (real|synthetic).
  • cal_logit: classifier logits.
  • cal_prob: classifier probabilities [0,1].
  • copied: whether the worker copy-pasted, (0|1).
  • character_pred: heuristic: contains line-breaks typical of ChatGPT (real|synthetic).
  • speed_pred: heuristic: over 400 words per minute reading and 80 wpm writting (real|synthetic).

Test-set-model.csv.gz

This is the data from the test set, unseen by the model and used for calibration. Contains synthetic summaries and real summaries from the 2019 paper.

  • text: Text summary.
  • target: whether the summary is real or synthetic (real|synthetic).
  • y_cc: classifier prediction (real|synthetic).

./analyses