Embedding model can be used to search text inside image

Wait 5 sec.

Just something I've been testing over few week since Embedding Gemma 2 released which is about 330MB at Q8. Not sure if this would count as 'resource' flair... Image shows cosine similarity changes as I refine the text query (aka text embedding vector) over time. Shows how well it work & quirks it has at same time. I'll leave tl;dr version for now, 3:30 am here. If anyone is interested I'll do few more tests and make example query image like above - still have 7 more sets totaling 22448 screenshots with mix of eng / kor / jpn already indexed, but I'm out of test ideas. TAKE WITH GRAIN OF SALT because this was tested only on few sets that isn't really scientific. TL;DR for EmbeddingGemma2 at least: For it's size & embedding dim (330MB Q8 / 768 embedding dim) this is very much usable Requires careful wordings (e.g. blond vs blonde) - maybe SLM's limitation or embedding dim's limitation (at least eng & kor) Text seemingly DOMINATE cosine similarity more than visual traits sometimes adding visual trait HARM (refer chair above) the similarity not sure if it's due to text being more informationally dense (aka more 'surprising') or just model being small and not recognizing such visual traits, idk regardless, visual traits are less useful in this test set since like 90% of 3003 imgs would have blonde girl in it. query cos sim. delta to next candidate image with dialog '삼각건이 머리에 얹어졌다' 0.75960 +0.01774 image with blonde girl 0.68208 +0.00250 image with blonde girl making >< expression 0.70416 +0.00340 (at least kor) Word's existence dominate over whole sentence e.g. for above image's example "삼각건"(cleaning hat thing) dominates more than other words around it fyi only 2 image out of 3003 contains world "삼각건", which you can see via delta gap drops query cos delta image with word 삼각건 0.69251 +0.01211 / +0.00766 / +0.00008 image with word 삼각건 & 머리 0.72881 +0.01331 / +0.02728 / +0.00203 image with sentence 삼각건이 머리에 얹어졌다 0.71156 +0.00810 / +0.01660 / +0.00041 Test setup: Visual novel screenshots of a single route 3003 screenshots from hanasaki workspring 3078 scs from irotoridori no hikari EmbeddingGemma2 via llama cpp model = KV = Q8_0 nb = b = ctx = 4096 (embedding model restriction, all need to be same) RX 7900GRE via vulkan (f u rocm, so annoying to build) Sample output for anyone curious about processing time Index mode: $ py embed_img_▽.py "./하나사키워크스프링 히카리" Batch: 8 Resize width: 1280 Resize threshold: 1536 [VectorDB] open w/ 0 rows removing 0 missing records from DB Checking new/modified images in dir Got 3003 new/modified imgs [EmbedServer] open for http://127.0.0.1:8081 100%|██████████| 376/376 [08:30 &#32; submitted by &#32;