Figured it's worth being upfront about the state of things rather than pretending the platform is further along than it is.
The ranking is the hard part. Right now a big chunk of our lab data comes from COAs that shops published themselves. Those are real third-party results, but the shop picked which sample got sent, and that's a different thing from an independent purchase off the shelf. We label tests by source for exactly that reason, but labeling it doesn't fully solve it.
The actual fix is commissioning our own testing - buying vials like a normal customer and sending them in blind. We're not there yet, mostly because it costs money we don't have. It's the plan, not a promise with a date on it.
Review volume is the other gap. Plenty of shops here have solid lab data and barely any reviews, which means their score is sitting mid-range because half the formula has nothing to work with. That's not the shop being bad, it's us not having enough data yet. It sorts itself out as more people leave reviews, but it takes time.
So why bother with it now? Because even incomplete, the information helps. Knowing a shop has 14 verifiable Janoshik reports at 99%+ is worth something. Knowing another one has zero independent testing on file is also worth something. Knowing a COA comes from a lab whose company registration doesn't resolve is definitely worth something. None of that is the finished picture, but it beats guessing.
One thing we want to do as this grows: actually ask shop owners what they think of the ranking. A few have already pushed back on things - one pointed out that a "latest tests" feed rewards whoever tests most often rather than whoever's actually good, which was a fair hit. Another asked why a strong reputation elsewhere doesn't carry over here. Those are the kind of criticisms that make the system better, and we'd rather hear them than not.
We're not going to let shops set their own scores, obviously. But there's a difference between shops influencing their ratings and shops telling us where the methodology is unfair, and the second one is genuinely useful.
If you run a shop and think something about how this works is wrong, say so. Same if you're a buyer and the scores don't match what you're seeing. It's early enough that changing things is still easy.