The problem
A utility contractor photographs every job: trenches being dug, cables being spliced, cabinets being installed, the site put back together afterwards. Thousands of photographs pile up, and finding anything later means someone remembers roughly when it was taken.
Tagging them would fix that, but the company does not want generic tags like construction or outdoors. They have their own vocabulary: 29 precise terms they already use in their paperwork. The question this thesis asks is simple to state: can an AI model that knows nothing about this company learn to use their words correctly?
Their vocabulary
The 29 terms are not a flat list. They are organised into eight categories, and some categories are exclusive: a photograph shows one work phase, not three.
The taxonomy. Categories marked with a dot are mutually exclusive. Several terms are company-specific: a sonde is an orange ball or an orange-and-black cylinder, buried so the pipe can be found again from above ground. Not something a general model would guess from the word.
How the experiment works
We show the model the same photograph several times, giving it a little more help each time, and measure what each addition is worth. Because each step adds exactly one thing, a change in the score can be traced to that one thing.
- No help. “Describe this photograph with short tags.” The model invents its own words.
- The word list. We give it the 29 terms and say: use only these.
- The words explained. We add one sentence per term saying what the company means by it.
- Training. Instead of explaining, we show the model 1155 already-tagged photographs and let it learn from them.
What we found
Explaining the words is worth far more than merely listing them, but neither comes close to training the model on real examples.
Accuracy (micro-F1) on 283 photographs the models never saw during training. Higher is better.
It is not one lucky model
We repeated the experiment on four different models, from three different research groups and at three different sizes. Explaining the words helps every one of them. Simply listing the words does not.
Each line is one model. Every line rises at the last step. The middle step is flat or even slightly down for two of them. Giving a model a list of words it does not understand is not reliably useful.
What did not work, and why that matters
Three attempts to push the prompt further all made the results worse: adding guidance about how many tags to expect, adding physical descriptions of trench depths, and restructuring the answer so the company's categories became the shape of the output.
The last one is the most interesting. Forcing the model to fill in one slot per category made it answer every category, including ones that rarely apply: image quality is genuinely worth mentioning in about 3% of photographs, and the model mentioned it in 100% of them. When we scrambled the categories at random as a control, the damage vanished, so it was the company's own sensible grouping causing the problem, not the format. A tidy category reads to the model as a question that must have an answer.
These negative results are a real part of the contribution. They say something practical: you can tell a model what your words mean, but you cannot talk it into seeing a distinction it never learned. For that, it needs examples.
What you can try here
The Try the models tab runs the real thing on a photograph of your choosing. You can tick more than one model and run them together on the same picture, and the page outlines the tags only one of them produced. The most revealing pairing is a trained model against the untrained one: the untrained model tends to add plausible-sounding tags that are not there, sometimes including terms that never occur in this company's photographs at all.