Research Continuity

Dear SRCD community, 

We have been developing guides and materials for running developmental studies via scheduled video chats (using Zoom). We now have a Github repository to share the materials we've developed so far. Thank you for those who expressed interest in these materials - as many of us are transitioning to online research while also trying to adjust to many other changes in our lives, we hope you can save some time and effort to create similar materials for your research.

Rather than a makeshift method during this covid-19 situation, we see this as a promising source of high-quality data to complement traditional in-person testing. As many of our studies require real-time interaction with an experimenter, our goal is to: (1) provide families with fun, enjoyable experience, (2) recreate in-person testing experience as much as possible, and (3) make it easy to train students and research assistants to ensure standardized, consistent testing procedures.

***** Details for those who are interested: *****

Visit this Github repository: https://github.com/sociallearninglab/online_testing_materials

The front page (README.md) provides detailed instructions for downloading materials and descriptions of different files. These materials are organized into folders "before_testing", "during_testing", and "after_testing".

The majority of the files are keynote slides that you can directly edit/adapt for your study session (including scripts so you know what our researchers say to parents/children in each slide) or export as PDFs (instructions for parents or training manual for research assistants). These files contain various tips/settings we've decided to use to streamline the testing process, and we tried to make them as detailed and self-explanatory as we can.

Also included here are our consent information and email templates for communication with parents. Our IRB approved Waiver of Documentation for online testing, so parents can easily consent via email or verbally during testing. You will also find a Databrary link to a video recording of a "mock testing session" so you can see what an example study looks like from the participant's view. This video will also give you a sense of how we are using these slides (especially calibration phase). We plan to upload more example videos for a range of studies.

We hope you find these materials useful. The repository is under CC-BY-NC license; you are free to share and adapt these materials in ways that fit your lab's needs. If you do, we would very much appreciate credit to the original source, as many lab members worked hard to create these materials (doi: 10.5281/zenodo.3762737, instructions on README).

*****Some FAQ we've received so far *******

1. Do you video-record? How do you store data?
We run studies via screen-sharing, and video-record using Zoom's "recording on cloud" feature. Zoom does not save the exact setup of the screen, so for studies where this is necessary for later coding, we are considering using Quicktime screen recording feature as an extra measure.

The video clips are then saved and stored on our primary data storage (physically in our lab space). Our main computer in lab works as a "server" so we can remotely access the hard disk (through VPN) and upload the data to the lab computer from our testing laptops. Another great solution is to use Databrary (you can create secure volumes on Databrary to store your data and manage access levels from private to public). 

2. What's your consent process like? 
Our IRB granted Waiver of Documentation so parents do not need to sign a hard copy. We try to get their consent via email before the session starts; in case we don't, we ask parents to verbally consent while we video-record the session. At the end of the session we explain various levels of video use permissions and ask them to choose (e.g., research-only, academic use, public). We are still experimenting to figure out the easiest way to do this! 

3. Any tips on how to transition my study to be run online?
Studies that involve presenting videos/pictures/storybooks via computer are the most easily adaptable. We cannot use pointing or other ways to direct children's attention as we normally do in in-person testing, so we've been (1) using calibration slides to create a context where experimenters can point or gaze towards a particular position on the screen, and (2) modifying our studies to build in animations/sound effects to direct children's attention to appropriate places. Our studies are usually in Keynote or PPT presentations but you can use anything you want.

4. What do you mean "calibration"?
It's nothing fancy -- because Zoom offers various layouts of different screens (shared screen, thumbnails of videos from participants) and we have no way of knowing exactly what the setup looks like on the participants' end, we need to initialize the screen setup so everyone's screen will look the same. We do this via "calibration" slides; this involves the parent using the computer first to check a few options, click a few things, and place the researcher's face in the correct spot on the screen. This takes a couple of minutes, but very useful! 

5. How do you recruit participants?
We updated our lab website to create a page dedicated to online research (more updates to come). We also created a "flyer" (you can find it on our repository). The flyer is specifically for our lab, but soon we will be replacing this with a flyer that directs parents to a website open to all researchers. We will be providing more information on SRCD Commons about this soon. Please stay tuned for an exciting cross-institution collaborative project for connecting parents to online researchers!

------------------------------
Hyowon Gweon
Assistant Professor
Department of Psychology
Stanford University
Stanford CA 94305
------------------------------

Thanks for posting these great resources Hyo! I hope people will consider adding their own materials-- maybe as file attachments to replies in this thread. (We will organize those attachments in the library associated with this community-- we're not github, but we're trying).

One question: I know there have been issues of security with Zoom. U of Wisconsin is recommending other platforms for sensitive data. Have you had any problems/push-back about Zoom?

------------------------------
Chuck Kalish
SRCD
Washington DC
------------------------------

Original Message:
Sent: 04-23-2020 19:35
From: Hyowon Gweon
Subject: Online Testing via Zoom: Startup Guides and Materials

Dear SRCD community, 

We have been developing guides and materials for running developmental studies via scheduled video chats (using Zoom). We now have a Github repository to share the materials we've developed so far. Thank you for those who expressed interest in these materials - as many of us are transitioning to online research while also trying to adjust to many other changes in our lives, we hope you can save some time and effort to create similar materials for your research.

Rather than a makeshift method during this covid-19 situation, we see this as a promising source of high-quality data to complement traditional in-person testing. As many of our studies require real-time interaction with an experimenter, our goal is to: (1) provide families with fun, enjoyable experience, (2) recreate in-person testing experience as much as possible, and (3) make it easy to train students and research assistants to ensure standardized, consistent testing procedures.

***** Details for those who are interested: *****

Visit this Github repository: https://github.com/sociallearninglab/online_testing_materials

The front page (README.md) provides detailed instructions for downloading materials and descriptions of different files. These materials are organized into folders "before_testing", "during_testing", and "after_testing".

The majority of the files are keynote slides that you can directly edit/adapt for your study session (including scripts so you know what our researchers say to parents/children in each slide) or export as PDFs (instructions for parents or training manual for research assistants). These files contain various tips/settings we've decided to use to streamline the testing process, and we tried to make them as detailed and self-explanatory as we can.

Also included here are our consent information and email templates for communication with parents. Our IRB approved Waiver of Documentation for online testing, so parents can easily consent via email or verbally during testing. You will also find a Databrary link to a video recording of a "mock testing session" so you can see what an example study looks like from the participant's view. This video will also give you a sense of how we are using these slides (especially calibration phase). We plan to upload more example videos for a range of studies.

We hope you find these materials useful. The repository is under CC-BY-NC license; you are free to share and adapt these materials in ways that fit your lab's needs. If you do, we would very much appreciate credit to the original source, as many lab members worked hard to create these materials (doi: 10.5281/zenodo.3762737, instructions on README).

*****Some FAQ we've received so far *******

1. Do you video-record? How do you store data?
We run studies via screen-sharing, and video-record using Zoom's "recording on cloud" feature. Zoom does not save the exact setup of the screen, so for studies where this is necessary for later coding, we are considering using Quicktime screen recording feature as an extra measure.

The video clips are then saved and stored on our primary data storage (physically in our lab space). Our main computer in lab works as a "server" so we can remotely access the hard disk (through VPN) and upload the data to the lab computer from our testing laptops. Another great solution is to use Databrary (you can create secure volumes on Databrary to store your data and manage access levels from private to public). 

2. What's your consent process like? 
Our IRB granted Waiver of Documentation so parents do not need to sign a hard copy. We try to get their consent via email before the session starts; in case we don't, we ask parents to verbally consent while we video-record the session. At the end of the session we explain various levels of video use permissions and ask them to choose (e.g., research-only, academic use, public). We are still experimenting to figure out the easiest way to do this! 

3. Any tips on how to transition my study to be run online?
Studies that involve presenting videos/pictures/storybooks via computer are the most easily adaptable. We cannot use pointing or other ways to direct children's attention as we normally do in in-person testing, so we've been (1) using calibration slides to create a context where experimenters can point or gaze towards a particular position on the screen, and (2) modifying our studies to build in animations/sound effects to direct children's attention to appropriate places. Our studies are usually in Keynote or PPT presentations but you can use anything you want.

4. What do you mean "calibration"?
It's nothing fancy -- because Zoom offers various layouts of different screens (shared screen, thumbnails of videos from participants) and we have no way of knowing exactly what the setup looks like on the participants' end, we need to initialize the screen setup so everyone's screen will look the same. We do this via "calibration" slides; this involves the parent using the computer first to check a few options, click a few things, and place the researcher's face in the correct spot on the screen. This takes a couple of minutes, but very useful! 

5. How do you recruit participants?
We updated our lab website to create a page dedicated to online research (more updates to come). We also created a "flyer" (you can find it on our repository). The flyer is specifically for our lab, but soon we will be replacing this with a flyer that directs parents to a website open to all researchers. We will be providing more information on SRCD Commons about this soon. Please stay tuned for an exciting cross-institution collaborative project for connecting parents to online researchers!

------------------------------
Hyowon Gweon
Stanford CA
617-324-2887
------------------------------

Yes, thanks for sharing all these resources, Hyo!

Oddly enough, although UW is emailing us regularly about concerns with Zoom, the UW IRB had no trouble with it.  It may be because they don't consider the data we intend to collect to be particularly sensitive (???).  They did ask us to include the following in our consent:

"This study will be conducted using a third-party application called Zoom. Zoom uses software to encrypt video calls to aid in the protection of users' privacy. Although researchers will make every effort to maintain confidentiality, it cannot be guaranteed when using Zoom because the researchers cannot control the security of the software.  The study session will be recorded using Zoom's recording feature. Following completion of the study session, videos will be uploaded and stored on secure servers."

The only significant concern our IRB raised was about how we would handle it if an individual who was not on the consent form came into the camera's view during the study. They had us add this to the consent and protocol:  

"We will only collect data from about those listed on the consent form. If someone not on the consent form enters the frame of the study, we will not collect data from them. Further, we will either edit them out of the recording of the session (if possible) or not retain the recording of the session (if it is not possible to edit them out)."

------------------------------
Kristin Shutts
Madison WI
608-265-8949
------------------------------

Original Message:
Sent: 04-24-2020 08:36
From: Chuck Kalish
Subject: Online Testing via Zoom: Startup Guides and Materials

Thanks for posting these great resources Hyo! I hope people will consider adding their own materials-- maybe as file attachments to replies in this thread. (We will organize those attachments in the library associated with this community-- we're not github, but we're trying).

One question: I know there have been issues of security with Zoom. U of Wisconsin is recommending other platforms for sensitive data. Have you had any problems/push-back about Zoom?

------------------------------
Chuck Kalish
SRCD
Washington DC

Original Message:
Sent: 04-23-2020 19:35
From: Hyowon Gweon
Subject: Online Testing via Zoom: Startup Guides and Materials

Dear SRCD community, 

We have been developing guides and materials for running developmental studies via scheduled video chats (using Zoom). We now have a Github repository to share the materials we've developed so far. Thank you for those who expressed interest in these materials - as many of us are transitioning to online research while also trying to adjust to many other changes in our lives, we hope you can save some time and effort to create similar materials for your research.

Rather than a makeshift method during this covid-19 situation, we see this as a promising source of high-quality data to complement traditional in-person testing. As many of our studies require real-time interaction with an experimenter, our goal is to: (1) provide families with fun, enjoyable experience, (2) recreate in-person testing experience as much as possible, and (3) make it easy to train students and research assistants to ensure standardized, consistent testing procedures.

***** Details for those who are interested: *****

Visit this Github repository: https://github.com/sociallearninglab/online_testing_materials

The front page (README.md) provides detailed instructions for downloading materials and descriptions of different files. These materials are organized into folders "before_testing", "during_testing", and "after_testing".

The majority of the files are keynote slides that you can directly edit/adapt for your study session (including scripts so you know what our researchers say to parents/children in each slide) or export as PDFs (instructions for parents or training manual for research assistants). These files contain various tips/settings we've decided to use to streamline the testing process, and we tried to make them as detailed and self-explanatory as we can.

Also included here are our consent information and email templates for communication with parents. Our IRB approved Waiver of Documentation for online testing, so parents can easily consent via email or verbally during testing. You will also find a Databrary link to a video recording of a "mock testing session" so you can see what an example study looks like from the participant's view. This video will also give you a sense of how we are using these slides (especially calibration phase). We plan to upload more example videos for a range of studies.

We hope you find these materials useful. The repository is under CC-BY-NC license; you are free to share and adapt these materials in ways that fit your lab's needs. If you do, we would very much appreciate credit to the original source, as many lab members worked hard to create these materials (doi: 10.5281/zenodo.3762737, instructions on README).

*****Some FAQ we've received so far *******

1. Do you video-record? How do you store data?
We run studies via screen-sharing, and video-record using Zoom's "recording on cloud" feature. Zoom does not save the exact setup of the screen, so for studies where this is necessary for later coding, we are considering using Quicktime screen recording feature as an extra measure.

The video clips are then saved and stored on our primary data storage (physically in our lab space). Our main computer in lab works as a "server" so we can remotely access the hard disk (through VPN) and upload the data to the lab computer from our testing laptops. Another great solution is to use Databrary (you can create secure volumes on Databrary to store your data and manage access levels from private to public). 

2. What's your consent process like? 
Our IRB granted Waiver of Documentation so parents do not need to sign a hard copy. We try to get their consent via email before the session starts; in case we don't, we ask parents to verbally consent while we video-record the session. At the end of the session we explain various levels of video use permissions and ask them to choose (e.g., research-only, academic use, public). We are still experimenting to figure out the easiest way to do this! 

3. Any tips on how to transition my study to be run online?
Studies that involve presenting videos/pictures/storybooks via computer are the most easily adaptable. We cannot use pointing or other ways to direct children's attention as we normally do in in-person testing, so we've been (1) using calibration slides to create a context where experimenters can point or gaze towards a particular position on the screen, and (2) modifying our studies to build in animations/sound effects to direct children's attention to appropriate places. Our studies are usually in Keynote or PPT presentations but you can use anything you want.

4. What do you mean "calibration"?
It's nothing fancy -- because Zoom offers various layouts of different screens (shared screen, thumbnails of videos from participants) and we have no way of knowing exactly what the setup looks like on the participants' end, we need to initialize the screen setup so everyone's screen will look the same. We do this via "calibration" slides; this involves the parent using the computer first to check a few options, click a few things, and place the researcher's face in the correct spot on the screen. This takes a couple of minutes, but very useful! 

5. How do you recruit participants?
We updated our lab website to create a page dedicated to online research (more updates to come). We also created a "flyer" (you can find it on our repository). The flyer is specifically for our lab, but soon we will be replacing this with a flyer that directs parents to a website open to all researchers. We will be providing more information on SRCD Commons about this soon. Please stay tuned for an exciting cross-institution collaborative project for connecting parents to online researchers!

------------------------------
Hyowon Gweon
Stanford CA
617-324-2887
------------------------------

Hi Chuck and Kristin, 

So far we haven't gotten any pushback from our IRB about Zoom; the concern about people in the background is reasonable though. Our IRB protocol and consent form is going to go through a few revisions/updates as we continue testing. 

One issue related to your questions is the parental permission for video use; currently our Level1 (default) consent is research-use only (videos stay in lab server), Level 2 consent is "academic/educational use" (involves us presenting example clips at conferences etc., but not distributing them), and Level 3 consent is "public" (involves distribution on our websites or as supplemental information for  journal publications). But now with many conferences going online, more talks/presentations will be posted online, so we're discussing how best to incorporate various ways of using videos in our consent form. Lookit has a two-tiered permission (Lookit team only vs. share on Databrary) with similar categories (private, scientific/educational, public),  but puts some forms of distribution under "scientific" instead of "public" . We will likely implement something like this going forward.  

My hope is that many labs will start using our materials -- the more labs use a standardized way of setting up the testing environment, the easier it'll be for parents and researchers to start testing and minimize the efforts for setting things up. We will be hosting online tutorials to help people utilize our materials, so stay tuned! :) 

Hyo

------------------------------
Hyowon Gweon
Assistant Professor
Department of Psychology
Stanford University
Stanford CA 94305
------------------------------

Original Message:
Sent: 04-24-2020 10:30
From: Kristin Shutts
Subject: Online Testing via Zoom: Startup Guides and Materials

Yes, thanks for sharing all these resources, Hyo!

Oddly enough, although UW is emailing us regularly about concerns with Zoom, the UW IRB had no trouble with it.  It may be because they don't consider the data we intend to collect to be particularly sensitive (???).  They did ask us to include the following in our consent:

"This study will be conducted using a third-party application called Zoom. Zoom uses software to encrypt video calls to aid in the protection of users' privacy. Although researchers will make every effort to maintain confidentiality, it cannot be guaranteed when using Zoom because the researchers cannot control the security of the software.  The study session will be recorded using Zoom's recording feature. Following completion of the study session, videos will be uploaded and stored on secure servers."

The only significant concern our IRB raised was about how we would handle it if an individual who was not on the consent form came into the camera's view during the study. They had us add this to the consent and protocol:  

"We will only collect data from about those listed on the consent form. If someone not on the consent form enters the frame of the study, we will not collect data from them. Further, we will either edit them out of the recording of the session (if possible) or not retain the recording of the session (if it is not possible to edit them out)."

------------------------------
Kristin Shutts
Madison WI
608-265-8949

Original Message:
Sent: 04-24-2020 08:36
From: Chuck Kalish
Subject: Online Testing via Zoom: Startup Guides and Materials

Thanks for posting these great resources Hyo! I hope people will consider adding their own materials-- maybe as file attachments to replies in this thread. (We will organize those attachments in the library associated with this community-- we're not github, but we're trying).

One question: I know there have been issues of security with Zoom. U of Wisconsin is recommending other platforms for sensitive data. Have you had any problems/push-back about Zoom?

------------------------------
Chuck Kalish
SRCD
Washington DC

Original Message:
Sent: 04-23-2020 19:35
From: Hyowon Gweon
Subject: Online Testing via Zoom: Startup Guides and Materials

Dear SRCD community, 

We have been developing guides and materials for running developmental studies via scheduled video chats (using Zoom). We now have a Github repository to share the materials we've developed so far. Thank you for those who expressed interest in these materials - as many of us are transitioning to online research while also trying to adjust to many other changes in our lives, we hope you can save some time and effort to create similar materials for your research.

Rather than a makeshift method during this covid-19 situation, we see this as a promising source of high-quality data to complement traditional in-person testing. As many of our studies require real-time interaction with an experimenter, our goal is to: (1) provide families with fun, enjoyable experience, (2) recreate in-person testing experience as much as possible, and (3) make it easy to train students and research assistants to ensure standardized, consistent testing procedures.

***** Details for those who are interested: *****

Visit this Github repository: https://github.com/sociallearninglab/online_testing_materials

The front page (README.md) provides detailed instructions for downloading materials and descriptions of different files. These materials are organized into folders "before_testing", "during_testing", and "after_testing".

The majority of the files are keynote slides that you can directly edit/adapt for your study session (including scripts so you know what our researchers say to parents/children in each slide) or export as PDFs (instructions for parents or training manual for research assistants). These files contain various tips/settings we've decided to use to streamline the testing process, and we tried to make them as detailed and self-explanatory as we can.

Also included here are our consent information and email templates for communication with parents. Our IRB approved Waiver of Documentation for online testing, so parents can easily consent via email or verbally during testing. You will also find a Databrary link to a video recording of a "mock testing session" so you can see what an example study looks like from the participant's view. This video will also give you a sense of how we are using these slides (especially calibration phase). We plan to upload more example videos for a range of studies.

We hope you find these materials useful. The repository is under CC-BY-NC license; you are free to share and adapt these materials in ways that fit your lab's needs. If you do, we would very much appreciate credit to the original source, as many lab members worked hard to create these materials (doi: 10.5281/zenodo.3762737, instructions on README).

*****Some FAQ we've received so far *******

1. Do you video-record? How do you store data?
We run studies via screen-sharing, and video-record using Zoom's "recording on cloud" feature. Zoom does not save the exact setup of the screen, so for studies where this is necessary for later coding, we are considering using Quicktime screen recording feature as an extra measure.

The video clips are then saved and stored on our primary data storage (physically in our lab space). Our main computer in lab works as a "server" so we can remotely access the hard disk (through VPN) and upload the data to the lab computer from our testing laptops. Another great solution is to use Databrary (you can create secure volumes on Databrary to store your data and manage access levels from private to public). 

2. What's your consent process like? 
Our IRB granted Waiver of Documentation so parents do not need to sign a hard copy. We try to get their consent via email before the session starts; in case we don't, we ask parents to verbally consent while we video-record the session. At the end of the session we explain various levels of video use permissions and ask them to choose (e.g., research-only, academic use, public). We are still experimenting to figure out the easiest way to do this! 

3. Any tips on how to transition my study to be run online?
Studies that involve presenting videos/pictures/storybooks via computer are the most easily adaptable. We cannot use pointing or other ways to direct children's attention as we normally do in in-person testing, so we've been (1) using calibration slides to create a context where experimenters can point or gaze towards a particular position on the screen, and (2) modifying our studies to build in animations/sound effects to direct children's attention to appropriate places. Our studies are usually in Keynote or PPT presentations but you can use anything you want.

4. What do you mean "calibration"?
It's nothing fancy -- because Zoom offers various layouts of different screens (shared screen, thumbnails of videos from participants) and we have no way of knowing exactly what the setup looks like on the participants' end, we need to initialize the screen setup so everyone's screen will look the same. We do this via "calibration" slides; this involves the parent using the computer first to check a few options, click a few things, and place the researcher's face in the correct spot on the screen. This takes a couple of minutes, but very useful! 

5. How do you recruit participants?
We updated our lab website to create a page dedicated to online research (more updates to come). We also created a "flyer" (you can find it on our repository). The flyer is specifically for our lab, but soon we will be replacing this with a flyer that directs parents to a website open to all researchers. We will be providing more information on SRCD Commons about this soon. Please stay tuned for an exciting cross-institution collaborative project for connecting parents to online researchers!

------------------------------
Hyowon Gweon
Assistant Professor
Department of Psychology
Stanford University
Stanford CA 94305
------------------------------

Original Message------

Hi Chuck and Kristin, 

So far we haven't gotten any pushback from our IRB about Zoom; the concern about people in the background is reasonable though. Our IRB protocol and consent form is going to go through a few revisions/updates as we continue testing. 

One issue related to your questions is the parental permission for video use; currently our Level1 (default) consent is research-use only (videos stay in lab server), Level 2 consent is "academic/educational use" (involves us presenting example clips at conferences etc., but not distributing them), and Level 3 consent is "public" (involves distribution on our websites or as supplemental information for  journal publications). But now with many conferences going online, more talks/presentations will be posted online, so we're discussing how best to incorporate various ways of using videos in our consent form. Lookit has a two-tiered permission (Lookit team only vs. share on Databrary) with similar categories (private, scientific/educational, public),  but puts some forms of distribution under "scientific" instead of "public" . We will likely implement something like this going forward.  

My hope is that many labs will start using our materials -- the more labs use a standardized way of setting up the testing environment, the easier it'll be for parents and researchers to start testing and minimize the efforts for setting things up. We will be hosting online tutorials to help people utilize our materials, so stay tuned! :) 

Hyo

------------------------------
Hyowon Gweon
Stanford CA
6504981194
------------------------------