Whats your view on Magazine Tool Reviews please vote in the poll.

Do you find magazine performed tool reviews to have credibility


  • Total voters
    107

Rob Keeble

Member
Messages
12,633
Location
GTA Ontario Canada
Hi All

I just recieved my latest copy of a popular woodworking magazine and this is the second month in a row that I am upset by what seems to be semi blatant bias in their reviews. (of course based on I presume, who pays the most for the exposure). Do they really think we are dummies. There are times when they insult our intelligence.

So I thought maybe we could get a poll going and hopefully just hopefully one of these editors (who do not have the time or decency to respond to letters you take the time and effort to send to them) might get to see what the general woodworkers opinion of these magazine led tool reviews are.

Also you may wish to add your ideas as to the process that should be undertaken when performing a tool review.

Just for the record I used to rely on these tests for new tool purchases now I would rather ask my fellow woodworkers on a forum.

rant rant rant :bang:
 
Magazine reviews are a 'part' of my decision making process. I do not rely on them totally. Other WW's, friends, manufacturer's reputation, and hands on experience (when I can get it) all come into play.
 
I voted yes, but I too don't rely on them alone and I make sure they are comparing similar products. Sometimes they saw "mid range" and some of the tools are "high range" and some are "low range" and then some in the middle. A recent lathe review comes to mind. Don't remember who did it, but they included very old models for some companies (even though some had introduced newer models) and brand new models for others.
 
My vote would be for "Partially". I've seen evidence of bias in the past, and also seen reviews that didn't include the particular brand or model I'm interested in. (I've run into the "missing model" scenario with Consumer Reports quite a bit, too.)
 
So what do you think the criteria should be for going about a review. Surely there should be more openess would it not be cool to have them ask their readers what tool they would like to see reviewed. There is no excuse for not soliciting input from the readers today.
What about the use of external evaluation personnel where tests are done in a more scientifically agreed manner. Here I think of University Labs or approval bodies such as UL or something of the like.
What about disclosure of the motivation for the criteria that were selected to be evaluated and why. So we get to understand why a feature is there and why it is important that it work.

I really got the bit by the teeth on this one when I saw a review last month of sanders in two different magazines and how they differed. I wont mention the names but there was a new palm sander that has been launched on the market by a very well known brand that spends significant money on advertising in all the magazines. So up pops this review. Then in the one mag they very obviously slant the criteria to lean towards the motivation for this new tool and in the other the criteria were completely different. The results were different. So how scientific was that and how reliable was that. A good publication should have the courage of its conviction to stand up and provide authentic comment which in the end will aid a product that is deficient in improving in order to secure a place in the market.
 
I voted 'Yes', because typically they do a good job of describing what features are desirable, what aren't, etc... So I look at them as being a good guide.

I do keep in mind which of the Mfg are buying full page ads in the magazine however, as I'm sure that plays into to a certain degree.

The only magazine out there where I think you can believe they are objective is consumer reports because they don't take ads.
 
My vote would be for "Partially". I've seen evidence of bias in the past, and also seen reviews that didn't include the particular brand or model I'm interested in. (I've run into the "missing model" scenario with Consumer Reports quite a bit, too.)

Exactly Vaughn. So what should be included (even if it is done by reference to an online report that was too lengthy to publish) is to demonstrate that every available brand in a particular market place, was solicited to participate. Then one knows that it was up to the manufacturer to decide to participate. If there were fees to participate these should also be disclosed along with whatever other conditions apply. Under these circumstances should a manufacturer decide not to participate the consumer or reader of the report than then draw their own conclusions from this. Why would you not? Unless you had something to hide or felt you were not going to get a fair hearing.
 
I could not vote because the poll does not have enough options. My answer is sometimes and it depends both on the magazine and the reviewer. Magazine reviews are one of many types of reasearch that I do before purchasing a tool.
 
Hi All

I just received my latest copy of a popular woodworking magazine and this is the second month in a row that I am upset by what seems to be semi blatant bias in their reviews. (of course based on I presume, who pays the most for the exposure). Do they really think we are dummies. There are times when they insult our intelligence.

Just for the record I used to rely on these tests for new tool purchases now I would rather ask my fellow woodworkers on a forum.

Rob, I am curious as to what you feel the specific bias was in this instance. I am also curious to which magazine you are referring to. When I first read your post I thought you were talking about Popular Woodworking magazine and was trying to figure out what the problem could be, but then I saw the "a popular woodworking magazine" (my bad for reading too fast). I take the magazine reviews with a grain of salt, but not because I feel they have a bias but more because I do not agree with what value they put on certain tools aspects (I may value sharpness over toughness in a chisel review). Maybe I am naive, but I feel that the magazines that I read (Popular Woodworking, Woodworking, Woodwork and Fine Woodworking) are not intentionally bias (we are only human so there is always some bias at work) in their reviews.

I put less much faith in fellow woodworkers. Few of us have worked on enough brands to have an informed opinion of what is best. We also tend to have more bias. It is hard to spend that money on a tool and be honest about it. There is a lot of justification that we need to do to feel good about the purchase. We also tend to be brand loyal. Just look at the Lie Nielson vs Vertias handplane posts in SMC & Knots.
 
Even in a totally "honest" review, the person reviewing the tool may have different things that are important to him/her than would be important to you. So if you did the review, tool A might have come out on top but the magazine reviewer gave tool B the top review.

Not biased beyond the normal differences in people. And having the opinion of someone who's actually had their hands on the tool has value compared to buying "blind".

Mike
 
I take reviews with a grain of salt and try desperately to not let those views that agree with mine seem more credible than others. I use the parts that help me like specifications, hands-on details mentioned and such. I try to ignore the hype, if present.

Wood magazine openly corrected themselves when they found they had goofed up a 10" TS blade test. . . that was cool. Others have given top honors to tools that the reviewer's own words in the article and the raw numbers show clearly was not the best choice.

Some are obvious, others are confusing and some seem pretty objective. As far as methods go, I favor a fixed, repeatable, quantified process applied to each tool and a mathematically achieved value assigned the result. That way no matter how someone "feels" about something, you at least have those values to compare.

I do appreciate the review's comments unless they are blatantly false. We all have different opinions but awarding a "best buy" to a drill that bursts into flames is a little wrong (this is a fictitious example).
 
Last edited:
i generally take any review as a subjective thing......unless there are several different folks with different work habits testing the gizmo for more than a few minutes any results must be somewhat biased......if not toward a "brand" then toward a feature or a feel..
 
I put less much faith in fellow woodworkers. Few of us have worked on enough brands to have an informed opinion of what is best. We also tend to have more bias. It is hard to spend that money on a tool and be honest about it. There is a lot of justification that we need to do to feel good about the purchase. We also tend to be brand loyal.

Ain't that the truth! Most woodworkers make terrible tool reviewers. Too much bias! :rofl:

Besides, sometimes I think we ask too much of the magazines. All the ones that come to mind are trying to do their best. We're a tough crowd, after all, and most of us are pretty cheap! ;) I know of one that asks for constant feedback, even letting readers vote on their cover. And another devised a bandsaw test that pulled a weighted workpiece through various bandsaws, timing them to measure cutting power. Most of them try hard, because they know how grumpy we are... ;)

The other thing: longterm profit margins for magazines are razor slim. I get a lot of them at work, and its easy to tell the state of the economy by how thick the mags are. Right now, they're on a starvation diet, and things haven't even gotten bad yet. It's likely that we're going to see a shake out in this sector, so revel in the abundance of mags, good and bad, while you can. We'll be seeing far fewer in the not too distant future... :doh:

Thanks,

Bill
 
Even in a totally "honest" review, the person reviewing the tool may have different things that are important to him/her than would be important to you. So if you did the review, tool A might have come out on top but the magazine reviewer gave tool B the top review

That is one of the things I like about reviews in Canadian Home Workshop magazine -- they have three reviewers. Each of the three tries out ALL the tools, puts them through the same paces, and then writes up their thoughts. They never totally agree. And you get different things like "I have small hands and this tool fit my hands better", and so on.

Do other magazines do this?

(Full Disclosure: I write articles on occasion for CHW magazine. But I've never done a tool review.)
 
I didn't vote, because like Frank said, not enough options. Some I have read have been pretty good, and some plain stunk. I will say that FEW mags have improved the "Testing Methods" they use over the past couple of years while others have not. I totally agree that some Testers/Reviewers place the importance of some features much too high when that feature may be the LEAST important feature of the tool, (which can be the individual doing the Testing, that is to blame for this). I also like to see the procedural Parameters used for the testing, and again, a few have done this more often in the recent past.

In one fairly recent test of 14" Bandsaws, a lot of Hype was given to one saw about it's "Great minimum blade deflection", when there were three others with exactly the same blade deflection in the 6" Red Oak Resaw test, while another saw from the same MFR was kinda chastised for having the "Most blade deflection" during the same test, BUT it cut Twice as fast as the others, 60"/min vs 30"/min for the others. What I would have like to have known was if that Fast saw would ALSO have a low Blade Deflection if the feed rate was slowed to the same as the others, AND maybe switched blades between the two saws of the same Mfr. that showed such a large difference. Even if the blades were all of the same brand, they don't all necessarily perform the same. It's things like this that always makes me feel like they didn't quite finish the test properly.

I do read the reviews to get a general overview and compare specs, (when they include the important ones, which they sometimes don't), and to see if there are any Glaring flaws that I should look for if I'm considering one of the tools being tested. I DO NOT put heavy weight on any of these tool reviews however in my tool selection process, but they ARE a part of the process for me.
 
Last edited:
I give low credibility to these reviews because of first hand, personal experience with outdoors and gun magazines. A bad review of a product made by a regular advertiser simply will not be published.
Now, before you jump on me for the comparison with gun magazines. Do check ownership of the publication(s). There are several publishing companies that put out many magazines dealing with a variety of subjects. All depend on advertising for their bottom line.
Chuck's suggestion would be the ideal way to go but it won't happen in the real world.
These reviews do have some value but the reader must use his own careful judgement. I am particularly sceptical of those reviews that do not include a popular brand. e.g. They review brands A; B; D; and E. But, notably brand C is not included. The reason is invariably because brand C is not an advertiser and it is a good machine that would overshadow the brand(s) advertised.
 
Last edited:
I voted "Yes" because there was no "Sometimes" option. I have been very pleased with purchases I've made based on most magazine reviews. Over the past few years, I've learned to rely on forums such as this for much more accurate input on specific equipment.
 
Top