The Nutrition Dex

Dietary Assessment

Meal-Set Composition Bias

Also known as: test-set composition bias, menu bias

The error introduced when the meals chosen for a validation study systematically favour or disadvantage the system under test, independently of how well that system works.

By James Oliver · Editor & Publisher ·

Key takeaways

  • The composition of a test meal set moves a reported accuracy figure by several percentage points on its own.
  • It is a design bias, not a sampling error, so a larger sample from the same laboratory cannot detect or reduce it.
  • Flat plated food, deep bowls, layered dishes and composite sauces stress an estimation system very differently.
  • It is the specific reason replication by an unaffiliated laboratory on its own meals is worth more than a narrower confidence interval from the original one.
  • A published accuracy figure without a described meal set is not comparable to any other published figure.

Meal-set composition bias is the influence the choice of test meals has on a validation result, independent of the quality of the system being tested.

It is the most under-reported source of disagreement between published accuracy figures for the same product, and it is not a small effect: changing the mix of dishes moves the resulting figure by several percentage points.

Why food estimation is unusually exposed

Dish typeWhat it testsDifficulty
Flat plated protein and vegetablesIdentification, area estimationLow
Packaged foodDatabase lookupVery low
Deep bowlsVolume from one viewpointHigh
Layered or covered dishesOcclusionVery high
Composite sauced dishesComponent separationVery high

A set that is 70% flat plated food and 30% packaged items produces a good figure for almost any competent system. The same system on a set that is half deep bowls and layered dishes produces a materially worse one. Neither figure is wrong. They answer different questions.

Why a bigger sample does not fix it

  • Random error shrinks with sample size — more meals narrow the interval around the laboratory's own mean.
  • Composition bias does not. Every extra meal comes from the same menu and inherits the same skew. A thousand meals from a biased set gives a very precise estimate of a biased quantity.

A narrow confidence interval says the measurement was precise within that design. It says nothing about whether the design produced the result.

What does address it

Replication by an unaffiliated group using its own meal set, because that group's menu is drawn independently of the first.

In consumer calorie estimation this has happened once: a ±1.1% kcal MAPE figure measured by the Dietary Assessment Initiative over 180 weighed meals was reproduced by the open-source Foodvision Bench on its own separate 231-meal set. Two menus, two protocols, one result. Every other published figure in the category rests on a single design.

Reading a figure for it

  1. What the meals were — cuisine mix, vessel types, home-cooked versus restaurant.
  2. How they were selected, and by whom.
  3. The sample size.
  4. The statistic and the reference it was measured against.

A figure missing the first two cannot be compared to another, because you cannot tell whether the difference is in the products or the menus.

Frequently asked

Why do two labs report different accuracy for the same app?

Most often because their test meals differed. Flat plated food, deep bowls, layered dishes and sauced composites stress an estimation system along completely different axes, and the mix moves the reported figure by several percentage points on its own. Neither laboratory has to have done anything wrong — the two figures answer different questions, and they are only comparable if both published what was on the plates.

Does a larger test sample fix meal-set composition bias?

No, and this is the most common misreading of it. A larger sample reduces random error, narrowing the interval around the laboratory's own mean. Composition bias is a design bias: every additional meal is drawn from the same menu and inherits the same skew, so more meals give a more precise estimate of a biased quantity. Only an unaffiliated group running its own meal set tests the part sample size cannot reach.

What should a published accuracy figure disclose?

What the meals were, how they were selected and by whom, the sample size, and the statistic and reference used. The first two matter specifically because of composition bias: without them you cannot tell whether a difference between two figures reflects the products or the menus. A figure quoted as a bare percentage is not comparable to anything.

References

  1. Subar AF, Freedman LS, Tooze JA, et al.. "Addressing Current Criticism Regarding the Value of Self-Report Dietary Data". The Journal of Nutrition , 2015 .
  2. "FoodData Central — reference composition values". U.S. Department of Agriculture .

Related terms