Deep-based Ingredient Recognition forCooking Recipe RetrievalJingjing ChenCity University of HongKongKowloon, HongKongjingjchen9-c@my.city.edu.hkChong-Wah NgoCity University of HongKongKowloon, HongKongcscwngo@cityu.edu.hkABSTRACTRetrieving recipes corresponding to given dish pictures facilitatesthe estimation of nutrition facts, which is crucial to various healthrelevantapplications. Thecurrentapproachesmostlyfocusonrecog-nition of food category based on global dish appearance withoutexplicit analysis of ingredient composition. Such approaches areincapable for retrieval of recipes with unknown food categories,a problem referred to as zero-shot retrieval. On the other hand,content-based retrieval without knowledge of food categories isalso diff icult to attain satisfactory performance due to large visualvariations in food appearance and ingredient composition. As thenumber of ingredients is far less than food categories, understand-ing ingredients underlying dishes in principle is more scalable thanrecognizing every food category and thus is suitable for zero-shotretrieval. Nevertheless, ingredient recognition is a task far harderthan food categorization, and this seriously challenges the feasi-bility of relying on them for retrieval. This paper proposes deeparchitectures for simultaneous learning of ingredient recognitionand food categorization, by exploiting the mutual but also fuzzyrelationship between them. The learnt deep features and semanticlabels of ingredients are then innovatively applied for zero-shot re-trievalofrecipes. ByexperimentingonalargeChinesefooddatasetwith images of highly complex dish appearance, this paper demon-strates the feasibility of ingredient recognition and sheds light onthis zero-shot problem peculiar to cooking recipe retrieval.KeywordsFoodcategorization; ingredientrecognition; zero-shotretrieval; multi-task deep learning1. INTRODUCTIONWhile there is a large number of cooking recipes posted on theInternet, f inding a right recipe given a picture of dish remains achallenge yet to be fully explored. The major problem underly-ing this challenge is the recognition of food categories as well astheir ingredients. Indeed, the problem is commonly shared amonghealth-related applications. For example, food-log management[1], which records daily food intake for dietary habit monitoring,Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor prof it or commercial advantage and that copies bear this notice and the full cita-tion on the f irst page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-publish, to post on servers or to redistribute to lists, requires prior specif ic permissionand/or a fee. Request permissions from permissions@acm.org.MM ’16, October 15-19, 2016, Amsterdam, Netherlandsc ? 2016 ACM. ISBN 978-1-4503-3603-1/16/10...$15.00DOI: http://dx.doi.org/10.1145/2964284.2964315PepperPepperPepperBlack beansChopped garlicCeleryMinced porkYubaYubaYubaCeleryParsleyParsleyFungusEggEggEggChopped green onionChopped green onionMusselsMinced porkCornkernelsPeasCarrotsFigure 1: Variations in visual appearance and composition ofingredients show the challenges of predicting ingredients evenfor dishes within the same food category. The f i rst row showsthreeexamplesofdishesforthecategory“friedgreenpeppers”,followed by “yuba salad” ad “steam egg custard” in second andthird rows respectively.often requires manual input of food intake. In addition to time-consuming, the process is error-prone. As investigated in [11], self-reporting data obtained from unfriendly acquired process tends tounderestimate the actual food intake. These concerns motivate theuse of mobile devices as a convenient means in capturing picturesof food intake for automatic recognition [24] [14] [25] [3] [16].This paper studies the recognition of ingredients for recipe re-trieval in the domain of Chinese dishes. Different from food cate-gorization, which is to identify the name of a dish (e.g., fried greenpepper shown in Figure 1), ingredient recognition is to uncover theingredients inside a dish (e.g., green pepper, black bean, choppedgarlic). In the literature, associating food categories to their re-spective recipes is regarded as a general pipeline that facilitates theestimation of calories and nutrition facts [35] [14]. The pipeline iseffective for recognizing restaurant dishes and the food categorieswith standardized cooking method (e.g., fast food) that often havesimilar visual appearance with the same ingredients. However,most dishes in Chinese food have no standardized cooking method,food presentation and ingredient composition. Direct mapping be-tween dishes and recipes, by using the names of food categories,is not likely to attain satisfactory retrieval rate, not mentioning theimperfect performance in food recognition. The diff iculty of thistask is probably alleviated, nevertheless, with the presence of GPSand restaurant menus as utilized by Im2Calories [24] and Menu-32
