Click through to see the images.
I was scheduled to give a presentation to a reef aquarium club some time ago, and I figured I'd try to do something different after the talk. I asked to have everyone coming to the talk bring all of their water testing kits, expired or not, just to see what would happen if a bunch of hobbyists tested the same water samples with different kits. With the help of my colleague and organic chemist Dr. David Flanigan, I got some water samples ready, and the club members brought in 14 pH kits, 9 nitrate kits, 14 calcium kits, and 12 alkalinity kits. They brought in 10 refractometers/floating hydrometers, too.
I gave out some slips of paper that asked for the brand/type, expiration date, result, and any comments, and asked to have them filled out accordingly. I also asked them, time allowing, to give their kit to someone else when they finished with it, so we could see what happened when two people tried using the same kit. So, I'll give more than one result for some kits, but only one for many others.
Note that no one was told where the water samples came from before testing in order to help reduce any bias in findings. In fact, they were told that some of the water wasn't from an aquarium (it wasn't). The results of these tests were quite surprising to say the least, and are certainly worth a look. Make sure that you look at the whole article (for reasons you'll see below).
Refractometers/Hydrometers
To test the refractometers/hydrometers, we used a calibrated MA877 Milwaukee Seawater Digital Refractometer to measure the specific gravity of a few gallons of water from one the aquariums I maintain at school. The specific gravity at 23°C was 1.024.
A Milwaukee MR100ATC refractometer used in this study.
Instant Ocean Hydrometer.
Refractometer Result Notes
Instant Ocean
1.027
Hydrometer #5 measured the specific gravity at 1.027, and the comment was "easy, but off".
Instant Ocean
1.024
Hydrometer #4 measured the specific gravity at 1.024, and the comment was "easy to use".
Instant Ocean
1.024
Hydrometer #3 measured the specific gravity at 1.024, and there was no comment.
Instant Ocean
1.024
Hydrometer #2 measured the specific gravity at 1.024, and there was no comment.
Instant Ocean
1.024
Hydrometer #1 measured the specific gravity at 1.024, and there was no comment.
Unbranded, of Chinese manufacture
1.040
Refractometer #5 measured the specific gravity at 1.040, and the comment was "easy to use - cheap e-bay tester".
Unbranded, of Chinese manufacture
1.024
Refractometer #4 measured the specific gravity at 1.024, and there was no comment.
Premium Blue RHS-10ATC
1.024
Refractometer #3 measured the specific gravity at 1.024, and there was no comment.
Milwaukee MR100ATC
1.024
Refractometer #2 measured the specific gravity at 1.024, and the comment was "easy".
Milwaukee MR100ATC
1.024
Refractometer #1 measured the specific gravity at 1.024, and the comment was "easy to read".
pH Kits
To test pH kits, we used the same water that was taken from an aquarium. David used a triple-calibrated Oakton Instruments Testr-2 digital pH meter and found that the pH of the sample was 8.2.
A typical Salifert pH test kit utilized in this study.
pH Kit Result Notes
Reef Master
8.2
Kit #14 was expired by 18 months. The result was 8.2, and the comment was "easy instructions, easy to use".
Sera
8.5
Kit #13 had no expiration date indicated. The result was 8.5, and there was no comment.
Nutrafin
8.2, 8.5
Kit #12 was expired by 1 month and the results were 8.2 and 8.5. The comments, respectively, were "easy" and "easy to use".
Seachem
8.2
Kit #11 had no expiration date indicated. The result was 8.2, and the comment was "fast, palette read easily".
Red Sea
8.6
Kit #10 was expired by 6 months. The result was 8.6, and the comment was "color hard to read".
Red Sea
8.2, 8.4
Kit #9 was expired by 6 months, and the results were 8.2 and 8.4. The only comment was "easy to understand".
Instant Ocean
8.2
Kit #8 was expired by 22 months. The result was 8.2, and the comment was "simple instructions, easy to use".
Instant Ocean
8.2
Kit #7 was expired by 18 months. The result was 8.2, and the comment was "easy/fast".
Aquarium Pharmaceuticals
8.2, 8.2
Kit #6 had no expiration date indicated, and the results were 8.2 and 8.2. The comments, respectively, were "very simple to use and understand" and "easy to read and use".
API
8.2
Kit #5 had no expiration date indicated, and the result was 8.2. The comment was "color scale is hard to read".
API
8.2, 8.2
Kit #4 had no expiration date indicated, and the results were 8.2 and 8.2. There were no comments.
Salifert
8.0
Kit #3 had no expiration date indicated, and the result was 8.0. The comment was "old kit".
Salifert
8.0
Kit #2 was expired by 10 months, and the result was 8.0. The comment was "super easy".
Salifert
8.0, 8.0
Kit #1 was expired by 5 months. The results were 8.0 and 8.0. The comments, respectively, were "easy to use" and "easy to read and match results".
Nitrate Kits
To test nitrate kits, I did not want to use tank water as I expected it to have a very low concentration. So, David made up a standard for us to use, which had a nitrate concentration of 25ppm.
A Red Sea Nitrate Pro test kit.
Nitrate Kit Result, ppm Notes
Reef Master
100
Kit #9 was expired by 18 months. The result was 100ppm, and the comment was "easy instructions".
Nutrafin
110
Kit #8 was not expired. The result was 110ppm, and the comment was "very easy".
Red Sea
100
Kit #7 was expired by 20 months. The result was 100ppm, and the comment was "hard to pick a color".
Red Sea
50, 50
Kit #6 was expired by 21 months. The results were 50ppm and 50ppm. The comments were "not sure how to interpret" and "colors match, easy directions".
API
50
Kit #5 had no expiration date indicated. The result was 50ppm, and the comment was "always easy - never trusted".
API
5, 20
Kit #4 had no expiration date indicated. The results were 5ppm and 20ppm. The comments were "results depend on how hard shaken" and "results depend on shaking."
Instant Ocean
1, 1
Kit #3 was expired by 22 months. The results were 1ppm and 1ppm. The comments were, oddly enough, "easy" and "very long".
Instant Ocean
20, 20
Kit #2 was expired by 10 months. The results were 20ppm and 20ppm. The comments were "easy to use" and "easy to read".
Salifert
25, 100
Kit #1 was expired by 16 months. The results were 25ppm and 100ppm. The comments, respectively, were "easy to read instructions" and "instructions easy to follow".
Calcium Kits
To test calcium kits, we again used the aquarium water. I took a sample and sent it to a local water quality lab and they reported that the concentration of calcium was 338ppm.
API Calcium Test Kit.
Calcium Kit Result, ppm Notes
Elos
350
Kit #14 was expired by 3 months. The result was 350ppm, and the comment was "easy test".
Reef Master
225
Kit #13 was expired by 15 months. The result was 225ppm, and the comment was "easy instructions".
Sera
400
Kit #12 had no expiration date indicated. The result was 400ppm, and there was no comment.
Nutrafin
240
Kit #11 was expired by 1 month. The result was 240ppm, and the comment was "easy to read".
Red Sea
380
Kit #10 was not expired. The result was 380ppm, and the comment was "simple to use".
Nature Reef
410, 450
Kit #9 was not expired. The results were 410ppm and 450ppm. The comments were (almost comically) "difficult to understand" and "pretty simple directions".
Seachem
100, 350
Kit #8 had no expiration date indicated. The results were 100ppm and 350ppm. The comments were "wasn't hard to use" and "easy to understand".
API
340, 390
Kit #7 had no expiration date indicated. The results were 340ppm and 390ppm. The comments were "too many drops - easy to lose count" and none were given from the second tester. Oddly enough, despite the comments about it being easy to lose count, this was the closest anyone got to the lab-measured concentration. Of course, that could also be pure luck after looking at the rest of the results...
API
480
Kit #6 was expired by 18 months. The result was 480ppm, and the comments were "took a long time to add drops one by one".
Instant Ocean
495, 975
Kit #5 was expired by 24 months. The results were 495ppm and 975ppm. The comments were "difficult to tell when the correct color was achieved" and "the testing was easy, but obtaining the results is time consuming".
Instant Ocean
240, 300
Kit #4 was expired by 5 months. The results were 240ppm and 300ppm. The comments were "hard to tell color shift" and "very confusing, not sure when exactly it is blue".
Salifert
320
Kit #3 was not expired. The result was 320ppm, and there was no comment.
Salifert
250, 350
Kit #2 was expired by 5 months. The results were 250ppm and 350ppm. The comments, respectively, were "instructions were difficult to understand" and "confusing, pain in the ass".
Salifert
200
Kit #1 had no expiration date indicated, and the result was 200ppm. The comment was "very old kit - directions are complex".
Alkalinity Kits
To test alkalinity kits, we used the same aquarium water. And again, I took a sample and sent it to the lab, which reported that the alkalinity was 3.3meq/l. Note that different kits may report results in either ppm, meq/l, or dKh, but I have converted all results to meq/l for ease of comparison.
Seachem MultiTest for pH and Alkalinity.
Alkalinity Kit Results, meq/l Notes
Marineland
3.5
Kit #12 was expired by 8 months. The result was 3.5meq/l, and the comments were "easy to read".
LaMotte
3.2
Kit #11 was expired by 7 months. The result was 3.2meq/l, and the comments were "easy test".
Sera
3
Kit #10 had no expiration date indicated. The result was 3meq/l, and the comment was "easy to read and understand".
Elos
3.2
Kit #9 was not expired. The result was 3.2meq/l, and the comment was "easy to use, a little hard to read".
Nutrafin
2.2, 2.4
Kit #8 was expired by 1 month. The results were 2.2meq/l and 2.4meq/l. The comments were "easy" and "pretty easy".
Seachem
2.5
Kit #7 had no expiration date indicated. The result was 2.5meq/l, and the comment was "easy!".
Red Sea
2.25, 2.25
Kit #6 was expired by 6 months, and the results were 2.25meq/l and 2.25meq/l. The only comment was "tough to distinguish".
API
3.6
Kit #5 was expired by 17 months. The result was 3.6meq/l, and the comment was "easy".
API
2.5, 3.2
Kit #4 had no expiration date indicated, and the results were 2.5meq/l and 3.2meq/l. There were no comments.
API
3.2, 3.9
Kit #3 had no expiration date indicated, and the results were 3.2meq/l and 3.9meq/l. There were no comments.
Instant Ocean
2.5
Kit #2 was expired by 5 months. The result was 2.5meq/l, and the comment was "super easy".
Instant Ocean
2.5, 4.5
Kit #1 had no expiration date indicated, and the results were 2.5meq/l and 4.5meq/l. The comments, respectively, were "easy to use" and "simple to read/use".
Comments
First, I'm just the messenger reporting results turned in by a group of reef aquarium hobbyists. So please don't sue/shoot me if you don't like the findings! Other than that, what a mess. The results for many types and brands of kits are obviously all over the place, and I guess the question is why that's the case.
Well, most of the kits were expired, which could obviously cause problems. Although, in some cases the expired kits performed as well as or even better than the current ones. Of course, that could have simply been random chance. So, it would seem that either (many of) the kits are simply inaccurate, or the method of use produces unreliable results, or (many) hobbyists just aren't good at following directions and using the kits properly - or any combination of these.
As far as user error goes, this could occur for many reasons and some kits had easier to follow instructions than others. Likewise, some kits are simply easier to use and/or produce results that are easier to read. And, in the case of one of the hydrometers being a little off on the high side, all it takes is a single tiny bubble stuck on the back-side of the swing arm to give such a result. In other words, there are plenty of opportunities to mess things up. So, I'll take things a step further to show that user error very likely plays a significant role in the apparent poor performances of many of the kits above, especially in cases where a kit was tested twice and gave significantly different results.
Alkalinity Kits - Part II
At a different time I also wanted to see what sorts of results would be produced if the same person took their time and used the same kit, on the same sample, repeatedly. So, David and I used our own kits and our own funds to go out and purchase a few more from a couple of local shops. We only got one of each brand of kit available to us (did I mention it was our own money?), but we figured that even testing one kit of each brand could be very informative. Besides, any of the kits we purchased could have been the kit that the next hobbyist needing one would have bought.
Depending on the type of dye used by the manufacturers of these kits, some can apparently be used equally well for freshwater and saltwater. However, others may give results that vary as much as 10% between the two (Holmes-Farley 2002). I wouldn't consider this to be significant, though. So, we figured we'd do both to see what happens.
David made a freshwater standard in the lab, which had an alkalinity of 3.5meq/l, and also made up a fresh batch of seawater using DI water and a popular salt mix, which was aerated for a few hours afterwards then allowed to sit for several more. This sample was then tested 6 times (3 times by each of us) with a Hach alkalinity kit, which coincidentally and consistently indicated an alkalinity of 3.5meq/l. Note that while we are comparing results of kits with the results of a kit, the point here is to look at consistency of results.
After preparation, each test kit was used 6 times to determine the alkalinity of the freshwater standard and 6 times to determine the alkalinity of the seawater sample. Note that rather than have one person perform all 6 tests on each sample, we decided to do 3 tests each on both samples for two reasons.
First of all, this could shed some light on how two people may do things differently even when following the same set of instructions. For example, when any given set of instructions says "fill to the line" two people may have different ideas of what this means. You've probably seen water form a meniscus when put in a glass container, and one person's idea of filling to a line may mean that the top of the meniscus touches the line, while another may think that the bottom of the meniscus should be at the line, etc.
Secondly, I'm partially colorblind. We thought this might be significant in some cases, as these kits require users to compare a colored liquid (the test medium) to a colored chart or scale of some sort to determine test results. Approximately 8% of males (but only 0.5% of females) are at least partially colorblind (Wikipedia 2012), meaning many hobbyists may have trouble with various colorimetric test kits due to a visual deficiency. So, we both performed the tests to see if I would have any particular difficulties not experienced by David.
Lastly, as noted above, different kits report results in either ppm, meq/l, or dKh, but I have converted all results to meq/l for ease of comparison.
Freshwater, meq/l
JF DF
Test Kit Test 1 Test 2 Test 3 Test 1 Test 2 Test 3 Average
Nutrafin
2.80
2.80
2.80
2.80
2.80
2.80
2.80
Salifert
4.00
4.00
4.00
4.00
4.00
4.00
4.00
Seachem
4.00
4.00
4.50
4.00
4.00
4.50
4.17
Red Sea
3.60
3.60
3.60
3.60
3.60
3.60
3.60
API
3.60
3.60
3.60
3.90
3.90
3.90
3.75
Instant Ocean
3.50
3.50
3.50
3.50
3.50
3.25
3.46
Saltwater, meq/l
JF DF
Test Kit Test 1 Test 2 Test 3 Test 1 Test 2 Test 3 Average
Nutrafin
2.80
2.80
2.80
2.80
2.80
2.60
2.77
Salifert
3.60
3.60
3.70
3.60
3.60
3.60
3.62
Seachem
4.00
3.50
3.50
4.00
4.00
4.00
3.83
Red Sea
2.80
2.70
2.70
2.90
2.80
2.70
2.77
API
3.90
3.60
3.60
3.90
3.60
3.60
3.70
Instant Ocean
3.50
3.50
3.50
3.25
3.25
3.25
3.38
Notes:
Instant Ocean: This kit was not expired, but oddly enough an expiration date could not be found on the outside of the package.
API: This kit had no expiration date indicated.
Note that the instructions said to cap the test vial with the included cap and to shake the sample after each drop added. However, the cap did not fit well and shaking the sample allowed some sample water to escape the test vial. So, while we started with the seawater sample, the remaining tests performed by both of us were done without the cap, by swirling the sample in the vial.
Red Sea: This kit was not expired.
Freshwater notes: We agreed during all six tests that the final color was a little darker than the darkest part of the scale that comes with this kit. The scale stops at 3.6, so the results may have been slightly higher than 3.6.
Saltwater notes: Note that Riddle (2007) tested Red Sea's alkalinity kit using a seawater sample and found the results to be approximately 19% low.
Seachem: This kit had no expiration date indicated.
Riddle (2007) also tested Seachem's alkalinity kit using a seawater sample and found the results to be within about 10% of expected.
Salifert: This kit was not expired.
Riddle (2007) also tested Salifert's alkalinity kit using a seawater sample and found the results to be about 11% high.
Nutrafin: This kit was not expired.
More Comments
Now we see a very different picture. As you can see, consistency was actually surprisingly good for any given test when tested by one of us, although in some cases David and I got somewhat different results for the same kit. For example, when I used the Instant Ocean kit for the seawater sample I got 3.5meq/l all three times, but David got 3.25meq/l all three times. When it comes down to it, I personally find a variance of only 0.25meq/l to be very good for an easily affordable hobbyist test kit. For that matter, all results for all kits on each of the samples were within 0.5meq/l.
So, with all this in mind, here are some suggestions: Always check expiration dates on kits and make sure they are up to date. Don't test when you're in a hurry. Sit down, take your time, and thoroughly read the instructions. I think it would also be a good idea to use any new kit at least three times on a single sample to see if you/the kit can provide consistent results. If you get different answers, try to figure out if it's you doing something a little different each time, or if it's the kit. If you're convinced it's the kit, try a different brand. And, as far as overall accuracy goes, it seems that lately there are more and more kits of various sorts that come with their own standards. These should allow you to check a kit's accuracy yourself.
For the manufacturers of the kits, the instructions on many of these (but certainly not all) were not easy to understand/follow, especially for inexperienced hobbyists. For best results, adding some pictures would likely be very helpful. For example, a picture of what "fill to the line" for a given kit means, etc. would be great and easily clear up any confusion. With respect to alkalinity kits, it would also be nice if everyone would use meq/l.
References
Holmes-Farley, R. 2012. Chemistry and the Aquarium: What is Alkalinity?. Advanced Aquarist 1(2). URL: http://www.advancedaquarist.com/2002/2/chemistry.
Riddle, D. 2007. Product Review: Alkalinity Test Kit Showdown. Advanced Aquarist 6(8). URL: http://www.advancedaquarist.com/2007/8/review
Wikipedia, 2012. Color blindness. URL: http://en.wikipedia.org/wiki/Color_blindness
View the full article
How strong are the links in food webs? An experiment demonstrates that they're strong enough for a disturbance to propagate across four trophic levels and two ecosystems. The experiment demonstrates that invasive species such as purple loosestrife could have broad effects on surrounding plant and animal communities, many of them cryptic. View the full article