Appendix E: Pilot Study Kit¶
This kit is for a student or teacher who wants to answer a simple question: do people learn something from reading a few chapters of this book? It is built for a science club, an advisory period, or a class that can spend two or three sessions on the book.
A pilot study is a small, first try at a study. It helps you find problems with your plan and gives you a rough idea of what might happen in a bigger, better-designed study later. It cannot settle the question by itself.
No pilot has been run yet
As of this writing, no pilot study of Gut Check has been carried out, and there are no results to report. If you run one, you will be collecting the first data. Please report what you find honestly, including results that are disappointing.
E.1 Purpose and Research Question¶
Purpose: to check whether students score higher on a short knowledge quiz after reading and exploring selected chapters of Gut Check, and to learn what readers found clear or confusing.
Research question: After reading Chapters 1, 5, 10, and 20 (or a subset of them), do students answer more quiz questions correctly than they did before reading?
Secondary question: How do students rate the chapters for clarity, interest, and usefulness?
What you might expect: average quiz scores will be higher after reading than before. Write your prediction down before you collect any data. Fixing your plan in advance is a small version of preregistration, which you can read about in Chapter 20.
E.2 Ethics and Permissions¶
Studying people, even with a short quiz, comes with responsibilities. Some of your readers may live with a gut condition or a mental health condition, and none of them should feel pressured, judged, or exposed.
Check the rules first. Before you collect any data from students under 18, check your school's rules and your district's rules. Talk to a teacher, club advisor, or principal. Many schools require permission from a parent or guardian before minors take part in any research activity, even an anonymous one. Follow whatever your school requires, and get approval in writing if you can.
If you plan to publish or present results. If you hope to share your results outside the club (in a science fair, a student journal, a conference poster, or online), you may need review by an ethics committee before you start. This might be an Institutional Review Board (IRB), a science fair Scientific Review Committee, or a similar group. Rules differ by school, fair, and journal, so ask early. Getting review after the data are collected is often not allowed.
Ground rules for every pilot:
- Voluntary. Anyone can choose not to take part, skip any question, or stop at any time, with no effect on grades, club standing, or anything else. Say this out loud at the start.
- Anonymous. Do not collect names, student ID numbers, email addresses, or anything else that could identify a person. Use random code numbers instead (see Step 1 below).
- No health information. No one is asked about their own health, diagnoses, symptoms, medicines, or diet. The quiz tests knowledge from the book, and the feedback form asks only about the book. If a participant brings up their own health, thank them and move on without recording it.
- Not a grade. Quiz scores are never used for any grade.
- Respectful discussion. If a group discussion happens, remind everyone that people with gut conditions may be in the room, and that conditions like IBS and Crohn's disease are real and not anyone's fault.
- Store data safely. Keep paper forms in a folder only the study leader can access. When the results are summarized, follow your school's guidance on how long to keep the forms and how to dispose of them.
E.3 Procedure¶
The study uses a single-group pre/post design: the same group takes the same quiz before and after reading.
| Step | What happens | Time |
|---|---|---|
| 1. Set up | Hand out code cards. Explain the study, that it's voluntary and anonymous, and that it isn't graded. | 5 min |
| 2. Pre-quiz | Everyone takes the 10-question quiz without any help. | 10 to 15 min |
| 3. Read and explore | Read the chosen chapters and try their MicroSims (see the session plan). | 2 to 3 sessions |
| 4. Post-quiz | Everyone takes the same quiz again. | 10 to 15 min |
| 5. Feedback form | Everyone fills out the anonymous feedback form. | 5 min |
Step 1: code cards. Before the first session, write random three-digit numbers (like 417 or 862) on small cards, one per person. Each participant keeps their card and writes the same code on the pre-quiz, the post-quiz, and the feedback form. This lets you match a person's two quizzes without knowing who they are. Don't make a list linking codes to names.
Session plan (choose one):
| Session | Two-session version | Three-session version |
|---|---|---|
| 1 | Pre-quiz. Chapter 1 and the Peristalsis Builder MicroSim. Chapter 5. | Pre-quiz. Chapter 1 and its MicroSim. |
| 2 | Chapter 10 and Chapter 20. Post-quiz. Feedback form. | Chapter 5 and Chapter 10. |
| 3 | Chapter 20. Post-quiz. Feedback form. |
Keep the test fair:
- Don't show anyone the answer key until after the post-quiz.
- Don't go over the quiz questions during the reading sessions, and don't teach directly to them. Let the chapters do the teaching.
- Record, for each session, the date, how long it lasted, which sections and MicroSims were used, and how many people attended (a count only, no names).
E.4 The Pre/Post Knowledge Quiz¶
Print this section on its own page. Use the same quiz for the pre-quiz and the post-quiz.
Code number: ______
Circle the best answer. This is not graded. It's fine to guess.
1. About how many neurons does the enteric nervous system (the nerve network in the gut wall) contain?
- A. About 86 billion
- B. About 400 to 600 million
- C. About 38 trillion
- D. About 5,000
2. When food stretches part of the gut wall, the peristaltic reflex causes:
- A. The muscle behind the food to contract and the muscle ahead of it to relax
- B. The whole gut to relax at once
- C. The brain to send a signal that starts every squeeze
- D. The muscle ahead of the food to contract so the food stops moving
3. About how much of the body's serotonin is made in the gut?
- A. About 10%
- B. About half
- C. More than 90%
- D. None; all serotonin is made in the brain
4. Which statement about serotonin made in the gut is correct?
- A. It travels to the brain and directly improves mood.
- B. It is made by gut bacteria only.
- C. It is the main source of the brain's serotonin.
- D. It does not cross the blood-brain barrier; the brain makes its own serotonin from tryptophan.
5. SSRI medicines for depression sometimes cause nausea or loose stools when someone first starts them. Why?
- A. SSRIs kill gut bacteria.
- B. SSRIs block the serotonin transporter in the gut as well as in the brain.
- C. SSRIs are made from dairy.
- D. SSRIs stop the gut from making any serotonin.
6. Which statement best describes irritable bowel syndrome (IBS)?
- A. It is caused by visible damage to the bowel that a colonoscopy can always see.
- B. It is not a real condition, because tests are often normal.
- C. It is a type of inflammatory bowel disease, like Crohn's disease.
- D. It is a disorder of gut-brain interaction: real symptoms caused by changes in gut sensitivity, gut movement, and gut-brain signaling.
7. Which of these is an alarm feature that means a doctor should look for a cause other than IBS?
- A. Bleeding from the rectum or unplanned weight loss
- B. Belly pain that gets better after a bowel movement
- C. Symptoms that get worse during stressful weeks
- D. Bloating after meals
8. About what share of highly cited animal studies were later confirmed in human randomized trials?
- A. Almost all of them
- B. About 9 in 10
- C. About one in three
- D. None of them
9. A rare side effect happens to 1 in 10,000 people who don't take a supplement and 2 in 10,000 people who do. A headline says the supplement "doubles your risk." What is the absolute difference?
- A. 50 extra cases per 100 people
- B. 2 extra cases per 100 people
- C. 1 extra case per 10,000 people
- D. 10,000 extra cases
10. Which phrase could legally appear on a US dietary supplement label without FDA approval (as long as it carries the required disclaimer)?
- A. "Treats IBS"
- B. "Supports digestive comfort"
- C. "Cures anxiety"
- D. "Relieves depression"
Answer key (for the study leader only)¶
| # | Answer | Where it's taught |
|---|---|---|
| 1 | B | Chapter 1, Section 1.5 |
| 2 | A | Chapter 1, Section 1.7 |
| 3 | C | Chapter 5, Section 5.6 |
| 4 | D | Chapter 5, Section 5.7 |
| 5 | B | Chapter 5, Section 5.9 |
| 6 | D | Chapter 10, Section 10.3 |
| 7 | A | Chapter 10, Section 10.6 |
| 8 | C | Chapter 20, Section 20.4 |
| 9 | C | Chapter 20, Section 20.8 |
| 10 | B | Chapter 20, Section 20.10 |
Short explanations:
- The enteric nervous system contains hundreds of millions of neurons, about 400 to 600 million by common estimates, roughly as many as the spinal cord. The brain has about 86 billion, far more.
- Stretch triggers contraction behind the food and relaxation ahead of it. This local reflex works without a command from the brain.
- More than 90% of the body's serotonin is made in the gut, mostly by enterochromaffin cells (some estimates say about 95%).
- Serotonin made in the gut does not cross the blood-brain barrier. The brain makes its own serotonin from tryptophan, an amino acid that can cross.
- SSRIs block the serotonin transporter in both the brain and the gut, which is why early stomach side effects are common.
- IBS is a disorder of gut-brain interaction. It is real even though the usual tests are often normal, and it is different from inflammatory bowel disease.
- Alarm features such as rectal bleeding, unplanned weight loss, anemia, or certain family history always need further evaluation.
- Only about one in three highly cited animal studies were later confirmed in human randomized trials.
- Going from 1 to 2 cases per 10,000 doubles the relative risk, but the absolute difference is only 1 extra case per 10,000 people.
- "Supports digestive comfort" is a structure-function claim. Claims to treat, cure, or relieve a disease need drug-level approval.
E.5 Anonymous Feedback Form¶
Give this right after the post-quiz.
Code number: ______
Please don't write your name. There are no right or wrong answers. You may skip any question.
Circle one number for each statement. 1 = Strongly disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly agree.
| # | Statement | |||||
|---|---|---|---|---|---|---|
| F1 | The chapters explained ideas in a way I could understand. | 1 | 2 | 3 | 4 | 5 |
| F2 | The everyday comparisons helped me understand the science. | 1 | 2 | 3 | 4 | 5 |
| F3 | The MicroSims (interactive simulations) helped me understand how the gut works. | 1 | 2 | 3 | 4 | 5 |
| F4 | The Hype Meter helped me think about whether a health claim is believable. | 1 | 2 | 3 | 4 | 5 |
| F5 | The chapters kept my interest. | 1 | 2 | 3 | 4 | 5 |
| F6 | I feel more confident checking a gut-health claim I see online than I did before. | 1 | 2 | 3 | 4 | 5 |
| F7 | I would recommend this book to a friend. | 1 | 2 | 3 | 4 | 5 |
Q1. What was the most helpful part of these sessions (a chapter, a MicroSim, a case, a Hype Meter card, or something else)? Why?
Q2. What was confusing, or what would you change to make the book better?
E.6 Scoring and Summarizing¶
Score the quizzes. Give 1 point for each correct answer, so each quiz has a score from 0 to 10. Only include people who have both a pre-quiz and a post-quiz with the same code number. Report how many people you had to leave out and why.
For each person, find the change: post-quiz score minus pre-quiz score. A positive number means the score went up.
For the whole group, calculate:
- Average pre-quiz score: add up all the pre-quiz scores and divide by the number of people.
- Average post-quiz score: the same, for the post-quiz.
- Average change: add up everyone's change and divide by the number of people. (This equals the average post-quiz score minus the average pre-quiz score.)
- Share improving: the number of people whose score went up, divided by the total number of people, as a percentage. Also report how many stayed the same and how many went down.
- Question by question: for each question, the percentage of people who got it right before and after. This shows which ideas the chapters taught well and which ones need work.
For the feedback form, report the average rating (1 to 5) for each statement, and the number of people who answered it. For the open questions, read all the answers, group similar comments into a few themes, and count how many comments fit each theme. Quote short comments only if they can't identify anyone.
What a pilot like this can and can't show
A small pre/post study without a comparison group can't prove that the book caused any learning. Scores might go up because people saw the same questions twice, because they talked about the topics with friends, because they learned the same ideas in another class, or because they tried harder the second time. The study leader also knows the goal of the study, so it isn't blinded. And a club group is not a random sample of all students. A fair conclusion sounds like: "In this small group, average scores rose from X to Y after reading. A larger study with a comparison group would be needed to know whether the book caused the change." Grade your own result with the Hype Meter: a single small pilot is Preliminary evidence at best.
Ideas for a stronger follow-up study: add a comparison group that reads something else (or the same chapters later), use a different but equally hard version of the quiz for the post-quiz, check whether people remember the ideas a few weeks later, and include more than one school.
E.7 Results Template¶
Copy this table and fill it in. Leave cells blank rather than guessing.
| Item | Result |
|---|---|
| Dates of sessions | |
| Chapters and MicroSims used | |
| Number of sessions and total time | |
| Number who took the pre-quiz | |
| Number who took the post-quiz | |
| Number with matched pre- and post-quizzes (n) | |
| Average pre-quiz score (out of 10) | |
| Average post-quiz score (out of 10) | |
| Average change (points) | |
| Number (and %) whose score went up | |
| Number (and %) whose score stayed the same | |
| Number (and %) whose score went down | |
| Question with the biggest gain | |
| Question with the smallest gain | |
| Average feedback rating, F1 to F7 | |
| Main themes from open questions | |
| Problems with the procedure | |
| Limitations | Single group, no comparison group, not blinded, same quiz used twice, small sample |
Question-by-question results:
| Question | % correct before | % correct after | Change (percentage points) |
|---|---|---|---|
| 1 | |||
| 2 | |||
| 3 | |||
| 4 | |||
| 5 | |||
| 6 | |||
| 7 | |||
| 8 | |||
| 9 | |||
| 10 |
If you run a pilot and want to share what you learned, the author would be glad to hear about it, especially about questions or chapters that confused readers. Never share forms or any information that could identify a participant.