Why did the rating system change?

 
 
 
Oct. 2006
         

Around October 1, 2006 new rating critera were developed. The rating criteria had to change due to something known in the reviewing community as "quality creep".

A common problem experienced by reviewers is that over time the general overall quality of the products they are reviewing improves. Therefore, a product formerly rated high is not as good relative to the newer and better overall general quality in the product market. Many lights that used to be above average are not as spectacular when the technology and innovations that made them so good become more commonly experienced.

It is precisely this type of adjustment to product ratings which I feel will make my ratings more valid and adaptable to future changes in the market. To keep the ratings static without any changes, while general overall product quality in the market improves, would result in most lights reviewed eventually rating "5.0 Stars". This would have made the rating system useless.

I decided to go through the laborious process of re-evaluating almost all the lights reviewed thus far and decide if the rating should change as a result of the new criteria. Many did.

As a result of the changes, the upper end of the ratings now allows for greater distinction between the higher quality lights.

The general critera for each category is listed below. 4.5 is now going to be the category for the best quality/performance lights, and 5.0 is for lights with extra refinement or superior performance. All of this is, of course, in my opinion. The Star ratings have a SUBJECTIVE component which may bump a light up or down about 1/2 Spot (usually) or more (occasionally) if I feel it deserves it.

If anyone sees what they feel is big discrepency in the new rating for a light, please let me know via e-mail and I'll consider your remarks. Please do not expect a response, as I will probably hear from quite a few folks.

1     Star  = Poor / Not Recommended at All
1½  Stars = Poor+ / Not Recommended / Well Below Average
2     Stars = Fair / Low end equipment / Below Average
2½  Stars = Fair+ / Low end equipment / Aproaching Average
3     Stars = Good / Average / Minimum Serious Use (think of "Maglite 3-D" as a good example)
3½  Stars = Good+ / Better / Above Average / Better Quality lights
4     Stars = Very Good / Well Above Average / Highly recommended
4½  Stars = Very Good+ / Way Above Average / Some of the Best / Highly Recommended
5     Stars = Excellent / Best of the Best / Extremely Well Designed

Remember, each Star rating encompases a RANGE of quality. Not all lights rated with the same Star rating are the same quality. Some are better, some worse, but they fit in the same general range in my opinion.

I ran some quick calculations based on the new rating system to show the approximate number of lights (percentage) in each Star rating category. The number of lights in the "above average" categories is greater than the number "below average", but this is mostly due to somthing in statistics called "sample bias" - manufacturers with high quality lights send them for review - manufacturers with junky lights do not. The end result of the new ratings criteria is that only about 10% of the lights on the site rate 5.0 and another 10% rate 4.5. These are the absolutely best lights you can get (in my opinion). Lights rating 4.0, 3.5 and 3.0 Stars make up 65% of the lights reviewed. These are average to well above average lights available on the market and some are very economical in price but still display good to very good quality. 2.5, 2.0, 1.5 and 1.0 Stars are marginal lights with some usefulness, but not necessarily recommended for serious use. These four category ratings make up the bottom 15% of lights reveiwed on this site.

10% 5    Stars
10% 4½ Stars
20% 4    Stars
27% 3½ Stars
18% 3    Stars
  5% 2½ Stars
  5% 2    Stars
   2.5% 1½ Stars
   2.5% 1    Star
 

 

 
TOP OF PAGE       HOME
Legal and Copyright Information     Javascript Menu: Thanks to Milonic.com