Suggestion Box

Spot an error or have suggestions for improvement on these notes? Let us know!

Week 11 · Modularizing and Scaling Up

0 · Overview

Up to this point, we have worked with individual participant files and written code step by step to import, score, filter, and summarize the data. Now it is time to bring everything together into a modular, reproducible workflow. We will use a combination of scripts, functions, and Quarto that work together to process all participants automatically and to keep the project organized.

Our goals:

Refactor our working code into small, readable functions.
Keep checks that help us catch data problems early.
Apply our functions to all participants in data/raw/.
Save outputs we can reuse later (a combined study file and per‑participant cleaned files).

Why this matters
Modular code lets us reuse logic and scale up from one file to many. Light‑weight checks give us confidence that our results are based on well‑formed data, which is good scientific practice.

1 · Modularization and Project Organization

In earlier weeks, all code was written directly inside a Quarto document. That made sense while we were learning each concept, but now it is time to separate our analysis into distinct, modular parts.

1A) Quarto vs. Script Files

Component	Purpose	Example Contents
Quarto Report (`.qmd`)	Readable narrative that explains the analysis and displays results; sources scripts and shows outputs.	YAML header, code chunks calling your functions, markdown text interpreting results.
R Scripts (`.R`)	Reusable, clearly defined functions. Each script should focus on one logical task.	`score_questionnaire.R` for scoring, `summarize_behavior.R` for behavioral summaries, `process_participant.R` to glue them for one file.

When you render your Quarto document, it will source these scripts to access the functions they contain.

1B) Why Modularization Matters

Clarity: Each script/function has a single, well‑named purpose.
Reusability: The same functions process new data without rewriting code.
Scalability: Apply functions across many files with lapply().
Reproducibility: The Quarto report narrates the pipeline and re‑runs it consistently.

1C) Project Setup and Assumptions

Project: npt_project.Rproj
Raw data: data/raw/ with files like npt-experiment-<timestamp>.csv
Cleaned outputs (new):

data/cleaned/study_level.csv and data/cleaned/study_level.rds
data/cleaned/participants/ (one cleaned CSV per participant)

We’ll assume all files in data/raw/ share the same format. If other file types or naming schemes live in that folder, they can cause problems. If that’s your situation, either clean the folder first or use a filename filter (we’ll show how below).

1D) Step‑by‑Step Setup

Create a subfolder for cleaned participant data (Console):

dir.create("data/cleaned/participants", recursive = TRUE)

Rename our existing script files:
- Open scripts/process_participant.R (from the last section).
- Save As → scripts/summarize_behavior.R (it currently holds behavioral summaries).
- Clear out the old process_participant.R so we can write a new one in this note.
Keep scripts/score_questionnaire.R — we’ll convert it into a function now.
Folder tree (target):

   npt_project/
    ├── data/
    │   ├── raw/
    │   │   ├── npt-experiment-2025-11-05-12-34-56.csv
    │   │   ├── npt-experiment-2025-11-05-12-36-10.csv
    │   │   └── ...
    │   └── cleaned/
    │       └── participants/
    ├── scripts/
    │   ├── score_questionnaire.R
    │   ├── summarize_behavior.R
    │   └── process_participant.R   # will write in Section 4
    └── reports/
        └── npt_import.qmd          # Quarto narrative

1E) Using a Quarto Report

We already have an empty reports/npt_import.qmd file from earlier, and now it's time to use it. Open reports/npt_import.qmd and add this YAML header:

---
title: "Processing Individual Participant"
author: "Your Name"
format: html
execute:
  echo: true
  warning: true
  message: false
---

Intro line for the report we can place below the header:

This report documents how we import and process each participant’s raw data file, summarize it using our functions, and combine the results into a single study-level dataset.

2 · Score the Questionnaire (Turn Our Snippet into a Function)

We already wrote the questionnaire-scoring code earlier. Now we’ll make four small edits to turn that script into a function:

2A) Conceptual Plan

Read the questionnaire JSON string.
Parse the JSON into an R object.
Flatten to a numeric vector.
Reverse‑score specified items.
Compute a mean score using na.rm = TRUE.
Return one numeric value.

We’re turning this into a function because the scoring rule is one idea. We write it once, test it once, and reuse it for every participant.

2B) Convert Our Existing Script into a Function

Four small edits in scripts/score_questionnaire.R:

Add the function header at the top:

score_questionnaire <- function(json_string) {

Add a closing brace } at the very end.
Our function is now expecting to work on an object called json_string, so we need to replace the line that grabbed the JSON from a data frame with the parameter the function expects:
```
responses <- fromJSON(json_string)
```
Since mean_score is defined locally to the function, we need to explicitly return the final score with return(mean_score) so we can use it later.

The logic stays the same as before, but now we can reuse this code on all of our participant files.

2C) Add Flexibility with Arguments

We may want to keep our questionnaire_scoring() function for use elsewhere. However, it can currently only process data on a 0 to 4 Likert scale and will always reverse score items 2, 4, and 7, so we might want to make it more flexible. To do this, we’ll add arguments that:

specify which items should be reverse-scored (reverse), and
specify the scale bounds (scale_min, scale_max).

This lets us handle different Likert scales (e.g., 0–4 or 1–5) without editing the function body.

Reverse formula for a bounded scale:

reversed_value = (scale_max + scale_min) - original_value

Default behavior: no reversed items (reverse = integer(0)).

Then we want to modify our current reverse scoring code to incorporate the passed argument reverse and only reverse score items when items are specified by adding an if statement.

if (length(reverse) > 0) {
  responses[reverse] <- (scale_max + scale_min) - responses[reverse]
}

2D) Simple, Clear Checks

Sometimes the JSON string we pass in may be missing or empty. We’ll guard against that and return a numeric NA when there’s nothing to score. This will make our function more robust to errors while also not missing any missing or empty data.

What these helpers do:

is.null(json_string): checks whether the JSON object we’re trying to score is NULL (non-existent).
is.na(json_string): checks whether the value is NA.
nzchar(json_string): returns TRUE if a character string has non-zero length. We negate it to check if the string is empty (!nzchar(json_string)).
NA_real_: the numeric version of NA (keeps the function’s output numeric when missing).

We can wrap these into one simple guard at the start of the function:

# If the JSON string is missing or empty, return a numeric NA
if (is.null(json_string) || is.na(json_string) || !nzchar(json_string)) {
  return(NA_real_)
}

We also want a hard stop if someone specifies a reverse index that doesn’t exist (e.g., 12 when there are only 10 items). This prevents an incorrect mean score.

# After parsing + flattening to numeric 'responses':
# responses <- as.numeric(unlist(fromJSON(json_string)))

# If reverse is provided, it must reference valid item positions
if (length(reverse) > 0) {
  if (any(reverse < 1 | reverse > length(responses))) {
    stop("One or more 'reverse' item indices are out of range for this questionnaire response.")
  }
}

if (length(reverse) > 0) checks whether the reverse vector actually contains any item numbers.
- If reverse is empty (for example, the user didn’t specify any reversed items), the code inside is skipped.
if (any(reverse < 1 | reverse > length(responses))) checks whether any of the numbers in reverse fall outside the valid range of item positions.
- length(responses) is how many items we actually have in the questionnaire.
- So if reverse includes a number smaller than 1 or larger than that total, this condition becomes TRUE.
stop("One or more 'reverse' item indices are out of range...") will cause R to throw an error and stops running the function if the invalid index check is TRUE
- That prevents it from continuing with an incorrect reverse specification (which could silently produce wrong mean scores).

In short: The code checks whether any of the reverse indices you provided are invalid, and if so, it stops immediately with an error message so you can fix the input.

This approach keeps the function honest: if the reverse-item specification is wrong, we fix the specification rather than silently producing a misleading score.

3 · Summarize Behavioral Data (Turn Our Snippet into a Function)

3A) Conceptual plan

Our behavioral scoring steps are a bit longer, but follow the same idea: we want to encapsulate them in a function we can call on each participant’s data.

Conceptually, the function should:

Read one participant’s data frame.
Convert the correct column to logical.
Separate practice and experiment blocks.
Filter reaction times between 250 and 900 ms.
Compute mean RT and accuracy for each subset.
Return a one-row data frame summarizing that participant.

We’ll name this function summarize_behavior() and save it in scripts/summarize_behavior.R.

3B) Convert Our Existing Script into a Function

Just like before, we can start from the working code and make a few quick edits:

Add the header at the top:
```
summarize_behavior <- function(df) {
```
Add a closing brace } at the end.
Replace any hard-coded variable names (like participant_data) with the parameter df, which stands for data frame.

Note: df is not a good variable name for broader code where we may forget what df contains, but within a function, a simple variable name that is specified as an argument is okay because within a function, the argument name is only used inside that function, so we know what it is (because we have to explicitly specify it in the arguments) and there’s no risk of confusing it with other data frames elsewhere in your code.

Explicitly return the resulting data frame using return(participant_summary) or just the final object name.

The goal is the same: take our one-participant script and make it reusable for every file.

3C) Add Flexibility with Arguments

We’ll add arguments for the reaction time filter window, since the 250–900 ms range might change in future analyses if we determine we were too strict or too lenient with our RT exclusion criteria.

summarize_behavior <- function(df, rt_min = 250, rt_max = 900) {
  ...
}

This will make it easy to adjust our filtering criteria later without editing the function body.

3D) Add Simple Checks

Like before, we can add lightweight checks to catch problems early and keep output consistent:

Required columns present (block, trialType, trial_type, rt, correct):

  # Ensure all expected column names are there
   if (!all(c("block", "rt", "correct") %in% names(df)) ||
       !any(c("trial_type", "trialType") %in% names(df))) {
     stop("Input data frame is missing required columns.")
   }

Coerce rt to numeric if needed:

  ## Check if rt column is numeric 
   if (!is.numeric(df$rt)) {
     df$rt <- as.numeric(df$rt)
     warning("'rt' column was not numeric. Coerced with as.numeric().")
   }

Coerce correct to logical:

  ## Change correct column to logical
  if (!is.logical(df$correct)) { 
    df$correct <- as.logical(df$correct) 
  }

After computing means/accuracies, add small range checks (warnings only):

# Accuracy 0..1
acc_cols <- c("practice_acc", "magnitude_acc", "parity_acc")
for (col in acc_cols) {
  val <- participant_summary[[col]]
  if (!is.na(val) && (val < 0 || val > 1)) {
    warning(paste(col, "is outside [0, 1]. Check 'correct' coding."))
  }
}
# Mean RTs within [rt_min, rt_max]
rt_cols <- c("practice_mean_rt", "magnitude_mean_rt", "parity_mean_rt")
for (col in rt_cols) {
  val <- participant_summary[[col]]
  if (!is.na(val) && (val < rt_min || val > rt_max)) {
    warning(paste(col, "is outside [", rt_min, ", ", rt_max, "]."))
  }
}

This doesn’t stop the function—it just prints a warning if something looks off (for example, if accuracy values fall outside 0–1 or reaction times weren’t properly filtered). A warning in R is different from an error:

An error stops the function immediately and prevents it from returning anything.
A warning still lets the function finish running but alerts you that something might be wrong. This is useful when we want to keep our code running but still know if a participant’s data needs a closer look. For example:
If accuracy is greater than 1, the correct column might not have been converted to logical (TRUE/FALSE).
If mean reaction times fall outside the expected range (250–900 ms), our filtering step might not have worked correctly.

Using small range checks like these is a simple way to build good data habits. They don’t change the results, but they help us trust our pipeline by flagging issues early—before they show up in the final dataset.

4 · Process a Participant File (Combine Questionnaire + Behavior)

4A) Conceptual plan

Now that we have two working functions, we can build one more that ties them together. This new function, process_participant(), should:

Derrive a subject_id from the file name and then read a single participant’s .csv file.
Extract and score the questionnaire using score_questionnaire().
Summarize the behavioral data using summarize_behavior().
Save a cleaned copy of the participant’s data for reference.
Return a one-row data frame for that participant.

4B) Function Scaffold (`scripts/process_participant.R`)

process_participant <- function(file_path) {
  ## Derive a subject id from the filename (no extension)

  ## Read the raw CSV

  #### Questionnaire score ------------------------------------------------
  ## Score questionnaire with our defaults (reverse 2,4,7 on 0–4 scale)

  #### Behavioral summary -------------------------------------------------
  ## Filter and summarize behavioral data (250–900 ms)

  #### Save participant summary -------------------------------------------
  dir.create("../data/cleaned/participants", recursive = TRUE, showWarnings = FALSE)

  ## Combine into a single-row participant summary

  ## Save summary CSV to cleaned/participants
  write.csv(
    df_clean,
    file = file.path("../data/cleaned/participants", paste0(subject_id, ".csv")),
    row.names = FALSE
  )

  #### Return output ------------------------------------------------------
  stopifnot(nrow(df_clean) == 1)  # one row per participant
  return(df_clean)
}

stopifnot(nrow(df_clean) == 1) confirms that the function really returns one row per participant. If something went wrong earlier and it accidentally produced more (or zero) rows, R will stop and print an error. It’s a small safeguard we can include without making the function more complex — a final “sanity check” before returning the result.

Checking Our Functions

Let's check to make sure the functions all work as expected before we move on.

First, save all your function scripts!

Then, you can copy this into your Console:

## Clear environment
rm(list = ls())

## Source scripts to load functions
source("scripts/score_questionnaire.R")
source("scripts/summarize_behavior.R")
source("scripts/process_participant.R")

## Test the functions on one participant file
process_participant("data/raw/npt-experiment-2025-11-05-10-33-05.csv")

5 · Scaling Up to All Participants

5A) Conceptual Plan

Once your process_participant() function works for one file, you can scale it to the full dataset.

The steps are:

Collect all file paths using list.files(), specifying the folder that contains your participant .csv files (usually data/raw/).
Apply your function to each file using lapply(). This applies the same logic across participants without writing a loop manually.
Combine the outputs into one data frame using do.call(rbind, ...).
Save the combined data to both .csv and .rds for later use.

5B) Implementation

Now that our functions are saved in the scripts/ folder, we’ll call them from our Quarto report. This part of the pipeline uses our functions to process all the data — it doesn’t define any new functions.

This time, our Quarto report acts as the narrative record of our analysis. It ties everything together:

It sources our scripts so the functions are available.
It runs those functions on all participant files.
It saves the combined outputs.
And it displays summaries or previews (like head(study_level)).

Because this section performs the actual data processing, rather than defining a new resusable function, it belongs in the Quarto file rather than an R script. The scripts hold the tools, and the Quarto report shows how those tools are used and what they produce.

#### Load packages ------------------------------------------------------------
```{r}
if (!require("pacman")) {install.packages("pacman"); require("pacman")}
p_load("jsonlite", "ggplot2")
```

#### Load functions ------------------------------------------------------------
```{r}
source("../scripts/score_questionnaire.R")
source("../scripts/summarize_behavior.R")
source("../scripts/process_participant.R")
```

#### 1) Find files -------------------------------------------------------------
```{r}
# Prefer a pattern to avoid accidentally pulling other CSVs
file_list <- list.files("../data/raw", pattern = "^npt-experiment-.*\\.csv$", full.names = TRUE)
```

#### 2) Apply our participant processor ---------------------------------------
```{r}
participant_rows <- lapply(file_list, process_participant)
```

#### 3) Combine into one study-level data frame --------------------------------
```{r}
study_level <- do.call(rbind, participant_rows)
```

#### 4) Save combined outputs --------------------------------------------------
```{r}
dir.create("../data/cleaned", recursive = TRUE, showWarnings = FALSE)
write.csv(study_level, "../data/cleaned/study_level.csv", row.names = FALSE)
saveRDS(study_level, "../data/cleaned/study_level.rds")
```

#### 5) Quick sanity check -----------------------------------------------------
```{r}
stopifnot(nrow(study_level) == length(file_list))
head(study_level)
```

Notes:

The filename pattern helps ensure we only read the intended task files (for example, files starting with "npt-experiment-").
Keeping this in the Quarto file lets us reproduce the entire pipeline later. Each time we render the document, and it will re-run all processing steps automatically.
The R scripts remain simple and reusable. You could reuse the same scripts in another Quarto report, or in a different experiment, without editing their internal logic.

New functions we’re using

Before running the code, let’s introduce two base R functions we haven’t used yet:

Function	What it does	How we use it here
`lapply()`	Applies a function to each element in a list and returns the results in a new list.	We use it to run `process_participant()` on each file path in `file_list`.
`do.call()`	Calls a function (like `rbind`) on a list of objects.	We use it to combine the list of participant summaries (from `lapply`) into one data frame.

Think of it like this:

lapply() → “run this function for every file.”
do.call(rbind, ...) → “stack all the results together into one big data frame.”

6 · Wrap‑Up

At this point, we’ve fully modularized our scoring pipeline:

score_questionnaire() handles text‑based survey data.
summarize_behavior() handles behavioral metrics.
process_participant() ties them together for one file.
A short base‑R pipeline processes all participants and creates a combined dataset.

Together, these steps form a reproducible, scalable workflow for transforming individual participant files into a single, ready-to-analyze dataset.

If you'd like to check out all the finial files we made this week to compare to yours, you can download them here and click "View raw".