Food Classification
Food Classification — Food classification is the AI subtask of identifying which dish or food is in a photograph. In calorie tracking apps, food classification produces the dish label ("grilled chicken breast," "caesar salad," "pad thai") that the app then maps to a database entry to retrieve calorie and macro values.
What is food classification?
Food classification is the computer-vision task of mapping a food photograph to a categorical label — “grilled chicken breast,” “caesar salad,” “pad thai,” and so on. The output is a discrete prediction (or a ranked list of candidate predictions with confidence scores), distinct from portion estimation (which produces a continuous gram weight) and distinct from full nutritional resolution (which requires combining the dish label with a database lookup to get calories and macros).
Modern food classification models are trained on labeled datasets — historically Food-101 (101 dish categories, 1,000 images each), ETHZ Food-101, and proprietary vendor datasets that often run into the millions of images. The leading apps in 2026 use vision transformer backbones with classification heads tuned to vendor-specific food taxonomies (which can differ — Cal AI’s dish ontology and MyFitnessPal Premium’s are not identical).
How is it measured?
Food classification accuracy is typically measured as top-1 accuracy (does the model’s first guess match the ground-truth label?) and top-3 accuracy (does the ground-truth label appear anywhere in the top three guesses?). In our methodology, we score top-1 dish identification at 40 points and top-3 at 20 points, for a combined 60 of the 100-point AI photo recognition sub-score.
Practical food classification accuracy varies substantially by dish category. Canonical single-ingredient dishes (a banana, a hamburger, a slice of pizza) produce top-1 accuracy north of 90% across all major apps. Composed plates (a build-your-own salad, a grain bowl) drop into the 60-75% range. Regional dishes outside the model’s training distribution (jollof rice, dal makhani, pho) can produce top-1 accuracy below 50%. Apps that allow user correction of the dish label, and that learn from those corrections, narrow the gap on regional cuisine over time.
Why it matters in calorie tracking apps
Food classification accuracy bounds the rest of the AI logging pipeline. An app cannot estimate calories accurately if it cannot identify the dish. Worse, food classification failures often fail silently: the app confidently logs “tofu stir-fry” when the user actually ate “chicken stir-fry,” and the user — already on a fatigued logging workflow at the end of the day — does not catch the misclassification. The calorie error is roughly 50% on protein content alone.
For users, the practical implication is to verify the dish label the app suggests, not just the calorie number. Apps that surface confidence scores (or that ask the user to confirm low-confidence dishes) make this easier. Apps that present a single confident dish prediction with no uncertainty signaling make it harder. See our published AI food recognition testing for the full per-app breakdown.