Some of you, probably most of you, will recognize the little critter to the right. Yep, he's the "talking frosted mini-wheat" as my son likes to say. Apparently not only is he talkative for a ball of spun wheat and sugar but he's attentive! Who knew!

Kellogg's has covered the most recent incarnation of their Frosted Mini-Wheats box with copy stating that eating their cereal can "Improve Kids Attentiveness by nearly 20%". As a scientist I was interested to know what they based that claim on. It would seem any filling cereal with some carbs and fiber would do the trick. It didn't take long to find the junk science in their claim.

They do actually make a reference on the box to a tiny footnote at the bottom of the box. Unfortunately they do not cite the study in the footnote, I doubt any reputable journal would take it, so there is no citation. Instead they state:

Based upon independent clinical research, kids who ate Kellogg’s® Frosted Mini-Wheats® cereal for breakfast had up to 18% better attentiveness three hours after breakfast than kids who ate no breakfast. For more information, visit www.frostedminiwheats.com.
So...kids who eat do better than those who don't. Was there really any question in anyone's mind about that fact? I'd be willing to bet that mice exposed to air live longer than those who aren't, but is that a valid scientific question...? No, it isn't.

The point here is that they picked a terrible negative control (one that renders the data meaningless), used no neutral or positive controls (pure glucose, long or intermediate chain carbohydrate shakes perhaps), and make no comparisons to similar products (oatmeal, pop-tarts, corn flakes, etc...). Despite ignoring such vital experimental procedures they trumpet the results as a major finding. In effect they present a scientific straw man in order to make their product look good. Beyond that they work hard at obscuring the data while giving the impression of openness.

The information page about the study is fairly well done. It explains the study in layman's terms and makes it easy to understand. They openly state that the only comparison is between kids given water or kids given Frosted Mini-Wheats. Of course they fail to mention that this narrow view limits the study to such a degree that there is little to no chance that Frosted Mini-Wheats could come out looking bad. The fact that there is a high likelihood that no significant difference could ever be proved between, say, Frosted Mini-Wheats and a bagel with cream cheese is not mentioned. In any reasonable scientific paper a limitation this large would be mentioned.

The funniest thing is that they mention other studies showing that high fiber, complex carb type foods help with morning attentiveness in children. The part they gloss over is that these studies are legitimate academic studies, most likely didn't use Mini-Wheats, and most likely show a similar increase in attentiveness versus not eating. In fact there is no data presented.

They go over the high level, "what tests were done and what do they measure" stuff, but never say how they came to an overall conclusion of a 20% increase or present any data. They could have averaged the results of the two tests, or they could have cherry picked the highest. My guess from their approach is that they took the highest % increase and used that for their claim. For all we know one of the tests showed 0% increase.
We have no idea what the statistical significance of the results are. They state that the test was performed on "children" from 8-12 years old. This implies more than two children but n=3 is hardly statistically significant, especially if the three children were gathered not by random sampling but by the number of kids they could round up during "Bring Your Kids to Work Day" at Kellogg's HQ. We simply don't know how they did their study.
Further, the distributions of the two groups might have overlapped. What is to say that the attentiveness of the starved group wasn't 17.5% (They round up their result for the fed group from 18% to 20% on the box). For all we know the acceptable P value for their study was 1, ensuring that any result they found would be significant. I doubt that, but without the data who's to say what they did?
The sad thing is that I worked quite hard on an analytical method to detect phytosterols at General Mills as a part of the design of a clinical trial to determine if phytosterols decreased serum cholesterol. It is possible to do good clinical work in the food industry, Kellogg's just didn't try. So, how is it that they can claim this is a clinical study? One of the pre-requisites for a good clinical study is excellent study design and meticulous data analysis and reporting. Kellogg's fails at both.

This is a perfect example of a pseudo-scientific snowjob. Luckily it has very little import on peoples lives other than possibly an increased out of pocket expense for a name brand cereal. It isn't, for example, the vice president editing or deleting scientific conclusions from government papers or the congressional record.

In a way I wish I was teaching high school science. This is the perfect example to get kids thinking about the scientific method, understanding how to interpret it, and learning how to structure their own experiments in order to avoid these flaws. It also makes the all too important point that critical scientific reasoning is necessary in daily life, much to the chagrin of most kids that age.

Unfortunately I'm not teaching high school. Fortunately, I am headed to graduate school in a couple months to pursue my dream of teaching/researching aquatic ecology at the college level. Unfortunately, there are numerous bad science teachers out there.
A colleague of mine just told me a story about her daughters science teacher. He does not have a degree in science and has refused to educate himself on the topic despite teaching it to kids.

Luckily it seems that if he has religious convictions he keeps them to himself. His problem is instead that he is unable to help the children with the assignments he gives from the book! His stock answer is, "I'm not an expert in that field, you should ask...", and referring questions to other teachers in the building who actually have science backgrounds. This is for high school level science assignments that he assigns straight from the textbook! Apparently he is so well known for this that the kids often write that phrase under his picture in the yearbook.

My colleague's daughter was struggling and she did the right thing by contacting the teacher in order to facilitate helping her daughter. The teacher was unable to help her and so she had to hire a tutor at an exorbitant price to help her daughter. Her daughter is doing fine now and is planning to move to an excellent technology focused charter school in the area (In part to get away from the teachers complete incompetence).

Wow!!! The ignorance and stubborn ineptitude that it must take not to be willing to brush up on the curriculum you are teaching enough to even be able to help kids with their homework questions is so incredible it takes my breath away. Every teacher I know (and I was raised by two of them) is constantly learning and relearning the material. To accept a job in a field you know nothing about, and then refuse to even attempt to educate yourself despite having learned much of the same stuff in high school yourself, is the height of stupidity, arrogant incompetence, and lack of respect for education and the lives of students. To do that you must ignore the fact that by teaching inferior material you are dooming every class you teach to a lifetime of ignorance, or several semesters of remedial education later on.

So, in conclusion, write to Kellogg's and let them know how dishonest their Attentiveness advertising campaign is. Meanwhile, I will try to learn that one should never treat the information on the back of cereal boxes with less skepticism than you would the rantings of a street preacher. I will also make sure that no children I know attend science classes at Maple Grove Senior High in Maple Grove, Minnesota without a warning as to the quality of the education.