Building a Multimodal Game-Box Scanner With VisionKit, IGDB, and GPT-5.6 Luna

Wait 5 sec.

September 25. I pointed my app at the Pokémon Red box on an Amazon page. It found Pokémon Red Version. Under "Read from the photo," it says "EAME BOYI." On September 17, a photo of the Japanese box, ポケットモンスター赤 (Pocket Monsters Red), came out as 「新トレスター」 ("New Trestar," a game that does not exist).Duo Queue is an iOS app I built for Shipaton 2026. Two people list the games they own and decide what to play tonight. One feature: point the camera at a game box, and the app finds the game. I added it on September 16. On September 22, the fourth version went to App Review.September 16: read the text on the boxThe first version used Apple's VisionKit. The camera reads the text on the box, and the app searches IGDB, the game database I use for titles and cover art. The next morning I added a lock: pick the biggest line of text, hold still for about a second, and it takes the shot.My own note that night: "Right now it just reads whatever text is written."September 17: 新トレスター ("New Trestar")I tried a photo of the Pokémon Red box from the web. The app read it as 「新トレスター」 ("New Trestar"). There is no game called that. It did not search anything.Then the bigger problem. IGDB has almost no Japanese names. ポケモン青 (Pokémon Blue), ポケットモンスター赤 (Pocket Monsters Red), スマブラ (Sumabura, the Japanese nickname for Super Smash Bros.): zero results each. "Pokemon Blue" in English works. 大乱闘スマッシュブラザーズ (Dairantō Smash Bros., the Japanese title) found only Ultimate, because that one entry happens to have its Japanese name stored.So I wrote a dictionary on the server that turns Japanese and English nicknames into the English official title. スマブラ (Sumabura) becomes Super Smash Bros., botw becomes The Legend of Zelda: Breath of the Wild. 452 lines. The scanner also tries title plus subtitle, then the title alone, then the next line, up to four times. The title it read became editable.That afternoon I wrote: "From a photo I took or a saved image, it still can't find the game."September 17, night: the cover fingerprintAn AI model would read the box better. But every scan would cost money, and I wanted a one-time purchase. So I tried matching the cover art instead: cut the box out of the photo, straighten it, make a 256-bit fingerprint, compare it with 3,054 real covers from IGDB. I could not collect real box photos, so I made damaged copies of the covers to test with.Cutting out the box worked 60 times out of 60. The right game in the top 5: about half. Without the cut-out: 1%. Not good enough on its own. It became a tiebreaker for the text results.September 22, morning: バイオ ("Bio") ate バイオレット ("Violet")On September 21 I decided to use the OpenAI API after all. The cheapest model, text only.Before switching, I measured the dictionary. 30 strings, the way OCR should read them: 23 correct at the top.One of the misses was ポケモン バイオレット, Pokémon Violet. The dictionary turned it into "Pokémon Resident Evil." In Japan, Resident Evil is バイオハザード (Biohazard), and people call it バイオ (Bio). The dictionary found バイオ (Bio) inside バイオレット (Violet) before it looked at the subtitle list. Zero results. The fix: put all the words in one list and match the longest first.Then 14 strings with OCR-style mistakes, like 「セルダの伝説 フレス オフ サ ワイルド」 (a misread of The Legend of Zelda: Breath of the Wild) and 「エルテンリング」 (a misread of Elden Ring). The dictionary got 0 of 14. Two came back with something. 「マリオカ一ト8 テラックス」 (a misread of Mario Kart 8 Deluxe), with the long vowel read as the kanji for "one", came back as Super Mario 64.With gpt-5.6-luna fixing the string before the dictionary: 43 of 44. Dictionary only: 27 of 44. The one miss was Momotaro Dentetsu, without its subtitle. These 44 are strings, not photos. The 14 mistakes were written to look like OCR mistakes.Same input, same answer, so there is a cache of 500. 2,000 calls a day at most. If the model is not sure, it falls back to the dictionary. The API key stays on the server.September 22, afternoon: send the photoAt 14:02 I tried the new build on my iPhone. Not bad. Then I scanned Pokémon Gold and got a different game. The server found Gold from the text "Pokemon 金" (金 means gold). So the mistake was on the phone side, before the search. I asked for the model to read the box too.At 14:14 the app sent the photo itself. Now it sends the whole frame as a 640px JPEG at the moment of the shutter. OCR runs alongside, only as a hint. With reasoning_effort set to low, a Tears of the Kingdom cover took 1.42, 1.44 and 1.56 seconds. With the default, 1.60 to 2.13, and 3.50 on the first call. On the phone, from the shutter to the result, it is about six and a half seconds, like in the video at the top.I tested it on three cover images. Tears of the Kingdom: right. Taiko no Tatsujin: Rhythm Festival: right. Dragon Quest XI S came back as Dragon Quest XI, without the S.The first scan is free. The second one opens the paywall.In the review video, I pointed the phone at a Yahoo! Shopping page for Pokémon Scarlet. Under "Read from the photo" it says Yahoo!ショッピング (Yahoo! Shopping). The top result was the DLC, The Indigo Disk. Scarlet itself was second.Duo Queue’s Current StageVersion 1.1 with the photo scan went to App Review on September 22 at 18:06. Four versions in six days: text, text with a dictionary, text with a model, the photo with a model. The dictionary is still there for the search box. バイオ (Bio) no longer eats バイオレット (Violet).