In the previous article I provided a short walkthrough of resource files, and how they are handled in jaysire and jsPsych. However, the only form of resource file we looked at in that article were image files, and trials were handled using the
trial_image_button_response() function. In this article I’ll extend this discussion, introducing audio files and video files. To make our lives a little easier, this time around I won’t create the resource files from scratch. Instead, I’ll use resource files that come bundled with the jaysire package:
We have one image file (“heart.png”), two versions of the same video in different formats (“heart.mpg” and “heart.webm”), and a single audio file (“lukewarm_banjo.mp3”).
First, let’s write an trial using
trial_image_button_response() to handle display the image file and ask people to rate its pleasantness on a 3-point scale:
This is familiar to us from the previous article. I’ve specified the
stimulus itself using the
insert_resource() function, listed the
choices available to participants, and some ancillary information about the size of the image.
Embedding videos in an experiment is – in principle – exactly the same as embedding images. Instead of using
trial_image_button_response() we use
trial_video_button_response(). However, there are a few complexities to note here. First, video files often require codecs to run, not every browser supports every form of video, and so on. As a consequence, it is often a good idea to have multiple versions of your video file, and let the browser find a version that it can display. As a consequence, instead of having a
stimulus argument, we specify the video file using a vector of
sources, like so:
trial_video_button_response() will end the trial as soon as the participant makes a response or the video ends. However, there are arguments that you can specify that allow you to make this decision yourself, as this example illustrates.
The structure of the audio trial is essentially the same as what we’ve seen previously.
However, there is one nuance to audio files if you are playing them locally (i.e., on your own computer, rather than from a remote server), which I’ll explain in the next section.
To build the experiment, I’ll start by specifying where the experiment is going to live on my computer. As with previous articles, I’ll just put it in a temporary folder:
To build the experiment into the
exp_path folder, I’ll call the
Most of this should look familiar. I’ve specified the
timeline that describes the experiment, the
path to the folder where it should be located, I’ve used the
build_resources() function to tell jaysire how to handle each of the resource files, and I’ve indicated that when the experiment finishes we should
save_locally() into the data folder (see the first article for more information about the data folder).
There is one thing that is new. I’ve specified the
use_webaudio argument and set it to
FALSE. The reason for this is that when you’re running the experiment locally, the web browser treats locally stored audio files as though they were being hosted on a different server to the webpage itself, and that triggers a “cross origin request error” (it is treated as a security risk). Fortunately, jsPsych includes a method to disable this and by setting
use_webaudio = FALSE you can get around this issue and run your audio files even when the experiment is running locally!
There is one last thing worth discussing, namely the file structure of the experiment that we have build. To start with, let’s take a look at what
build_resources(resources) #> # A tibble: 4 x 4 #> name type from to #> <chr> <chr> <chr> <chr> #> 1 heart.mpg video /tmp/RtmpJ7a2M8/temp_libpath335d2e79f18… resource/video/he… #> 2 heart.png image /tmp/RtmpJ7a2M8/temp_libpath335d2e79f18… resource/image/he… #> 3 heart.webm video /tmp/RtmpJ7a2M8/temp_libpath335d2e79f18… resource/video/he… #> 4 lukewarm_ba… audio /tmp/RtmpJ7a2M8/temp_libpath335d2e79f18… resource/audio/lu…
Now take a look at where all the files have ended up:
list.files(exp_path, recursive = TRUE) #>  "experiment/experiment.js" #>  "experiment/index.html" #>  "experiment/resource/audio/lukewarm_banjo.mp3" #>  "experiment/resource/image/heart.png" #>  "experiment/resource/script/jspsych-audio-button-response.js" #>  "experiment/resource/script/jspsych-image-button-response.js" #>  "experiment/resource/script/jspsych-video-button-response.js" #>  "experiment/resource/script/jspsych.js" #>  "experiment/resource/script/xprmntr.js" #>  "experiment/resource/style/jspsych.css" #>  "experiment/resource/video/heart.mpg" #>  "experiment/resource/video/heart.webm"
Here is the entire experiment:
library(jaysire) resources <- system.file("extdata", "resources", package = "jaysire") list.files(resources) #>  "heart.mpg" "heart.png" "heart.webm" #>  "lukewarm_banjo.mp3" image_trial <- trial_image_button_response( stimulus = insert_resource("heart.png"), stimulus_height = 400, stimulus_width = 400, choices = c("Unpleasant", "Neutral", "Pleasant") ) video_trial <- trial_video_button_response( sources = insert_resource(c("heart.mpg", "heart.webm")), choices = c("Unpleasant", "Neutral", "Pleasant"), trial_ends_after_video = FALSE, response_ends_trial = TRUE ) audio_trial <- trial_audio_button_response( stimulus = insert_resource("lukewarm_banjo.mp3"), choices = c("Unpleasant", "Neutral", "Pleasant"), trial_ends_after_audio = FALSE, response_ends_trial = TRUE ) build_experiment( timeline = build_timeline(image_trial, video_trial, audio_trial), path = temporary_folder(), resources = build_resources(resources), use_webaudio = FALSE, on_finish = save_locally() )
You can check out a working version of the experiment here.