how many independent tests before you trust a vendor. my number is three.
Genuine methodology question rather than a vendor post, so flairing it Discussion.
My rule has been three independent results, from at least two different members, across at least two different batches. Below that I treat a vendor as unknown rather than good.
Three is arbitrary. I picked it because it is the smallest number that can show you a spread, and spread is what I actually care about — a vendor averaging 98.5 from a 99.4 and a 97.6 is a different proposition to one averaging 98.5 from 98.4 and 98.6, and one test cannot distinguish them.
What is your number and why?
best — the order this archive was captured in
Four for me, and specifically four across at least a year, because I want to see whether the process drifts.
The BCH record is the example that set my number. Twelve documents in the audit, all complete, results clustered — that is a process. Compare that with a vendor whose six results run 98.4, 97.9, 96.2, 94.8, 98.1, 93.7. The mean of that second set is not a useful number about anything. It is two vendors wearing one name.
"two vendors wearing one name" is exactly the thing i was reaching for. a spread that wide makes the mean actively misleading.
which is why vendor-level averages should always ship with a range. i publish both and people quote the average anyway.
people have always done this. the point estimate is the headline, the interval is the story, and the interval loses every single time.
put the range in the headline then. "98.5 (93.7–99.4)" is not hard to type.
trying that formatting next update. if it changes how people quote it i will report back.
The directory's own answer, for what it is worth: a source page gets created at two member reports plus one independent test, and that threshold is deliberately low because a page with a thin record and a visible thin-record caveat is more useful than no page at all.
What the threshold is not is a recommendation. Several pages exist specifically so people can see that we know very little.
my number is two, but they have to be from different services on the same batch. inter-lab agreement tells me more than two agreeing results from one lab.
this is the better methodology and it is more expensive, which is why almost nobody does it.
mine is "however many exist, plus one from me". edit: which is a lazy answer, but the one from me is the part that matters.
zero, i test every order. it is cheaper than a wasted vial and i have the disposable income of someone with no social life.
this is all theatre. the vendors know which batches get tested.
they do not, because members buy retail and do not announce it. and if they did, we would see systematically better results on tested batches than on the vendor COAs, and we see the opposite about as often.
alright, that is a real answer. i will stop saying it.
- 1Four for me, and specifically four across at least a year, because I want to…6 comments in this branch · started by u/janoshik_junkie