AI & Research
Coding agents have SWE-bench. Fashion production has nothing: no shared test for whether an AI-generated spec can actually be built. Here is how we evaluate the AI that writes production specs at Silhouet, and where we think the industry needs to go.
The numbers on AI projects are brutal. RAND puts the failure rate of AI initiatives above 80 percent, and an MIT study found 95 percent of enterprise GenAI pilots produce no measurable value. The root cause those studies point to is not the models. It is the data underneath them: unstructured, fragmented, and unlabeled. Fashion production is close to the worst case. The knowledge that decides whether a garment can be built lives in PDFs, email threads, Illustrator files, and someone's memory of what the factory said last season.
There is a second, quieter cause that gets less attention: nobody measures the output. A generated tech pack can look complete and still be unbuildable. A missing seam allowance, a size curve that does not grade, a BOM without interfacing. The document reads as plausible, and plausible is not factory-ready. Those are different bars, and only one of them costs you a sample round.
Without a score, demos win. A vendor demo shows the flattering example. The production run finds the gaps, and the brand eats the cost. The AI conversation in fashion runs on vibes because almost nobody grades outputs against what a factory actually needs.
When coding agents took off, the field converged on a simple discipline. SWE-bench grades an agent by taking a real repository issue, letting the agent write a patch, and then running the repository's own test suite against it. The agent does not get style points. Either the tests pass or they do not. That one design choice made progress measurable, comparable across labs, and honest.
Fashion manufacturing has no equivalent. There is no shared answer to the question that matters: can a factory build from this spec on the first read? Every AI tool in the category claims accuracy. None can prove it against a common standard, because the standard does not exist.
We think the make side deserves the same discipline the software world built for itself. That starts with treating factory-readiness as something you can test, not something you assert.
Our technical lead, Chris Lee, wrote the internal rule we build by: every agent ships with an eval, and you pick the eval by its oracle, meaning the thing you grade against. How many right answers exist determines how you test.
When there is one right answer, you grade against a golden set of verified values. Sizing, garment details, and construction fields work this way: a medium either matches the graded spec or it does not. When there are many right answers but one testable property, you run a check instead, which is the SWE-bench pattern. There are many valid tech packs for the same hoodie, but whether the measurements grade cleanly across sizes is computable. And when no computable answer exists, only then do we use an AI judge, always graded against a reference, and validated against human labels before we trust it. An unvalidated judge is a random number you have chosen to believe.
Scoring works the same way everywhere: binary pass or fail per field, tallied to a percentage. Many small checks instead of one big grade, so we see where the AI fails, not just how often. And coverage is tracked separately from correctness, because a field that got filled is not the same as a field that got filled right.
Every AI task in Silhouet ships with an eval. The auto-fills that draft details, BOMs, sizing, and construction, the synthesis that turns a sketch and a Shopify product into a full spec, and the factory-readiness check that reads the result the way a factory would: each one is graded, field by field, against verified ground truth.
The checks are deterministic and code-graded wherever possible, which makes them reproducible, cheap to run, and debuggable. Results are tracked over time, so when a model or a prompt changes, we see the number move before a brand ever could. Accuracy that is not tracked drifts, and drift in this domain means wrong samples.
The factory-readiness check deserves a special mention, because it is an eval pointed at your work instead of ours. It reads a tech pack the way a factory production manager would, and flags what a factory would ask about before you send it. Fewer clarifying questions, fewer surprise sample rounds.
The metric is the easy part. The scarce, valuable thing is trustworthy ground truth. A general-purpose AI can grade itself against the internet. An AI that writes production specs has to be graded against specs that were actually built: graded measurement sets, BOMs that were costed and ordered, construction that produced an accepted sample.
That ground truth is exactly what Silhouet's structured production data is. Every product a brand builds in the platform adds verified values to the record: what the factory accepted, what graded cleanly, what a real production run confirmed. Production memory is not only how the next tech pack gets easier. It is how the AI gets honestly graded.
This is why we keep saying the structured data is the point. Whoever holds real, verified production data can build real, verifiable production AI. Nobody else can.
The longer arc is a benchmark suite for the category, the make-side equivalent of what SWE-bench did for code. The tasks are already clear from daily production work: draft a factory-ready tech pack from a sketch and a catalog. Catch the spec gap a factory would flag. Grade a size curve. Size a reorder against real sell-through. Each is testable, and none has a public standard today.
We are building these evals for our own agents first, and we will share what we learn as the work matures. The research posture is real: Chris first-authored a paper presented at the ICML 2026 Workshop on Human-AI Co-Creativity and accepted to VL/HCC 2026, on how people work with what AI generates. That question, how a human stays in control of AI output, is the same one a founder faces when an agent drafts their spec. It is why nothing in Silhouet ships to a factory without the founder reviewing and approving it.
If you are a researcher or an operator working on evaluation for AI in fashion production and supply chain, we want to compare notes. Email brandon@silhouetapp.com.
You do not need to be technical to use this. When any vendor shows you AI for production work, ask three questions. How do you measure accuracy? What ground truth do you measure against? What happens to quality when the model underneath you changes?
A team that grades its AI will answer in numbers and will tell you where the AI is still weak, because they can see it. A team that cannot answer is showing you a demo, and the demo is the answer. Treat measurement as the buying signal, whoever you buy from, including us.
An eval is an automated test that grades an AI system's output against ground truth or a testable property, the way a test suite grades code. Good evals score many small checkpoints as pass or fail and tally the results, so you can see exactly where the system fails and catch regressions when models or prompts change.
Research points at the data, not the models: RAND measures AI project failure above 80 percent and MIT found 95 percent of GenAI pilots produce no measurable value, with unstructured and fragmented data as the root cause. Fashion production runs on PDFs, spreadsheets, and email, so most AI built on top of it has nothing reliable to stand on, and without evals the quality problems stay invisible until a sample comes back wrong.
Yes, with two conditions: structured inputs and a human review gate. Silhouet's AI drafts the spec, BOM, measurements, and construction from your Shopify data or a sketch, you review and approve every value, and a factory-readiness check flags gaps before anything goes out. The claim that fails is full autonomy, an AI sending unreviewed specs to a factory.
A golden set is a collection of verified correct answers used to grade an AI system. In fashion production that means graded measurement sets, BOMs that were actually costed and ordered, and construction details that produced accepted samples. Building trustworthy ground truth is the hardest and most valuable part of evaluation, which is why real production data matters more than any model choice.
We are building the evals for our own agents first and tracking them over time. As the work matures we plan to share findings, and we are open to collaborating with researchers and operators working on evaluation for AI in fashion production. Reach out at brandon@silhouetapp.com.
Silhouet generates factory-ready tech packs and checks them the way a factory would, with an eval behind every agent. Free to start.