Measurement Errors and Challenges for Mechanics

“Measure in a way which makes sense relative to part function” 

  Introduction

There is no absolute accuracy in measuring, nor is it required. Measurement variation is present every time a part is measured so it is expected that sometime someone may believe that your measurement is in error. Be it your boss, another employee, an inspector, or a customer, they will be absolutely certain you made a mistake. The first thing to remember: “The intensity of the conviction has no bearing on whether it is true or not.” But how do you resolve the problem? The following principles of conflict resolution are what I used for my repair business. They might not be adaptable to everyone’s situation, but they should point the way toward resolving measurement conflicts. 

“Excessive accuracy is a sign of poor breeding” - Socrates.

Principles of Measurement Conflict Resolution

Caliper Master used to check calipers. Periodic checks between calibration assures quality measurements

1. A measurement that does not follow the agreed upon process shall be considered invalid.
2. Measurements taken with instruments clearly damaged, abused, or malfunctioning shall be considered invalid.
3. Measurements taken using improper technique or improper usage of the instrument shall be considered invalid.
4. An uncalibrated measurement cannot challenge a calibrated measurement.
5. When two or more valid repeatable measurements are made, they are considered equal and the results averaged and the average considered the reported measurement (some exceptions apply).

1. A measurement that does not follow the agreed upon process shall be considered invalid.

The measurement process is what makes a measurement repeatable and reproducible. Repeatability is the range of values obtained when a single operator measures the same part several times at the same location. If you do not measure at the same spot, using the same measurement tool and fixture, then you might not be able to duplicate your measurement.

Reproducibility is the range of values obtained when two or more people measure the same item at the same location. All need to agree on what they are measuring, and how they are measuring the part. If one uses a micrometer and the other a tape measure, then the results will not agree. If one measures crankshaft run-out with the crankshaft installed in the engine and you measure using a granite flat plate and vee-blocks, the measurements may not agree.

It is expected and normal that repeated measurements yield different values even when the measurement process is careful defined. I might use a slightly different pressure on my micrometer than you. All measurements contain a level of uncertainty as to the true value. The goal is to reduce this uncertainty to an amount that is not important. Without an agreement on what is being measured and how it is to be measured there is no accuracy in the measurement.

The most serious process error resulted in the loss of Alaska Airlines on January 31,2000 with 83 passengers. Excessive thread wear on the jackscrew assembly resulted in loss of the horizontal stabilizer. The mechanic work card stated that thread wear was "within allowable limits" . In fact, the threads on the jackscrew nut were almost completely worn away. The process (fixtures) used by the mechanic were not what Boeing specified and therefore the measurement results were different and 83 people+crew lost their lives! 

You can have several types of process errors:

1. Location error. Measuring at a different location.
2. Condition error. Measuring under different conditions.
3. Instrument error. Using an incorrect instrument.
4. Operator Error

Location Error

Measurements occur at finite points on an infinite surface. For the measurements to be repeatable, the agreed on process must specify the measurement location. If I measure the diameter of a shaft at one point and you measure at another point then it’s expected that the shaft diameter might be different at different locations on the shaft.

Location Error Example: Exhaust Temperature (EGT) Probe.

A temperature probe is placed into each exhaust stack. If each probe is not at the same position in the exhaust stack then comparative temperature differences among exhausts are caused not only by differences in the combustion process but also differences in probe position. Each probe needs to be installed at the same distance and the same orientation for measurements to be comparable.

Location Error Example: Cylinder Head Temperature (CHT) Probe
Normally, CHT is taken using a bayonet probe into a boss at the cylinder head. Some installations will use instead a thermocouple spark plug gasket to measure CHT. “Red line” CHT is typically based on the temperature at the bayonet boss and not at the spark plug gasket. Due to a location these temperature measurements will be different.

Condition Error

The Conditions in which a measurement is taken affects your measurement. The weight of an aircraft depends on the local gravity constant. The diameter of a cylinder barrel depends on its temperature. Six thousandths of an inch clearance between the piston and barrel at room temperature might shrink to no clearance at redline temperature. Aircraft control cable tension depends upon aircraft temperature.

Instrument error

Six inch thick granite surface plate provides a basic reference surface. Accurate 

measurements and descriptions of geometric form must begin with an established 

relationship to some known position; the surface plate is that reference position.

Rule 1 for surfacee plates: Don't put your coffee cup on the surface plate.
Rule 2 for surface plates: If anyone does so and it doesn't cause a fight -- then you don't have a metrology lab.

This includes the measurement instrument and fixtures or tools used in the measurement. For example, shaft run-out performed on a vee block and a dial indicator will yield incorrect results if the table sags from the weight of the crankshaft. 

Have you ever had this happen to you? I have. A part measures beyond wear limits and the customer states that "it doesn't look worn to me". Like he has micrometer eyeballs! Eyeballs are less precise than a dial caliper. A Dial caliper is less precise than a micrometer. Part of the measurement process is selecting the proper measurement instrument for the part and the tolerances in question.

Less Precise each division is 200 psi

More Precise each division is 50 psi

Operator Error

That human influence can be corrupted by the goals of the interrogation where a source inspection seeks to prove that the measurement passes and stops when the objective is achieved while the buyers inspection seeks to discover what's wrong and stops when his objectives are achieved.

When you are personally affected by the outcome of a measurement, the measurement result may be biased toward the desired outcome. Often repeated measurements are taken until the desired measurement is achieved. This is the Placebo affect and was at work in the Alaska Airlines accident discussed earlier. Science knows that bias influences results at the unconscious level. Rigorous double-blind experiments attempt to remove bias. Bias is not the fault of the scientist or mechanic - it is part of human nature. Employer's and inspector's efforts to stamp out bias through sanctions have the opposite result of increasing bias.

2. Measurements taken using improper technique or improper instrument usage shall be considered invalid.

Alaska airlines used a different fixture to pull the jackscrew into position than what the aircraft manufacturer specified. Accident analysis revealed that the Alaska Airlines fixture did not pull as tightly as the factory fixture. This resulted in less measured wear than their actually was. Instrument error also occurs when one uses a different measurement tool than the one prescribed. Using a less accurate dial caliper instead of a more accurate micrometer is an obvious example.

Corollary: Random procedures produce random results.

Below is an example of a .002 difference in measuring a rod using two slightly different measurement techniques. This shows how sensitive measurements are to the technique used in measurement. The first photograph the rod is located at the ends of the caliper mandrel. This method produces an error as the mandrel flexes

More Abbe error

Less Abbe error

Abbe error: Measurement errors are increased as the item being measured is moved away from the axis of the measurement device. Dial calipers are more accurate (have less bias) if you place the work piece close to the scale. This reduces the Abbe error. The Abbe error on this caliper is .002 inch. This occurs because the caliper jaws are forced to flex outward. By placing the work piece closer to the center axis of the caliper, the bending force is reduced and more accurate measurements are taken.

3. Measurements taken with instruments clearly damaged, abused, or malfunctioning shall be considered invalid.

Such instruments are considered retired or out of service and cannot be used. Measurements using such instruments cannot be made.

4. An uncalibrated measurement instrument cannot challenge a calibrated measurement instrument.

Checking the accuracy of a dial caliper with a length standard.

The uncalibrated instruments results are not verified. Accuracy, precision and function are unknown.

5. When two or more valid repeatable measurements are made, they are considered

equal and the results averaged and the average considered the reported measurement.

Valid repeatable measurements can be averaged and the average value is the reported value. For example, the calibration readings you get back for your torque wrench are the average of 10 measurements at each setting. Reproducible measurements (measurements separated by time, place, equipment or operator) are not averaged.  Deciding which measurement is the correct measurement is an example of Segal's Law "A man with a watch knows what time it is. A man with two watches is never sure."

Averaging data for reporting purposes seems to violate my suggestion about not throwing out measurement data. The result of an average is a representation of the data and not the data itself. For example, the average US household has 3.5 family members. Nobody counted any half family members. 3.5 is a representation of the original data. However, for reporting purposes, many systems require one number. What number should be entered? Without guidelines from the measurement specification, one can choose any of the valid measurements. By averaging them you give equal treatment to all the measurements. But remember, averaging is a summary of the data and not the observed measurement.

 

Important exception to this rule. If the part is critical for flight safety then use the worst case measurement, assuming that measurements are all equally valid. Using this practice, the bias is toward safety at the expense of economics. A bias that protects the mechanic and the public.

 

Measurement Uncertainty

“Good enough for the Women that I date…”

The picture below shows a rod that measures more than 3-3/8 inch and less than 3-1/2 inch. We are uncertain as to exactly how much. The uncertainty due to the limit of resolution is 1/8 inch. Problems arise when measurements are close to, or straddle the go/no-go airworthiness criteria.

All measurements are possibilities and nothing more

 There is no way of knowing for certain which measurement is the true measurement. If you are in charge of aircraft airworthiness, you cannot express or state uncertainty in your maintenance release. Either the aircraft is airworthy or it's not. Yet all measurements are uncertain. Your best protection is to have your employer write a procedure to deal with measurement uncertainty. Good luck! However, you can only

make the best decision with the tools at hand. Lack of sufficient resources, including policy resources, does influence the outcome of measurements. One method reducing mechanic uncertainty is the greater use of fixed limit gages, otherwise known as go/no-go gages, or attribute gages. 

Ball gages used by the author to check inside diameter of hose assemblies.

Two thickness gages attached together make a convenient go/no-go gage. Shown here checking for minimum and maximum clearance.

Limit gages provide unambiguous yes or no decisions regarding acceptability of the inspected part. The mechanic is in familiar territory when asked to compare attributes rather than manipulating numeric data.

Measurement Statement

"It's a hair under"

A conclusion based on measurement data should contain the measurement data. "Worn Beyond Service Limits" is hard to defend at a later date if you don't have the data. Always include the measurement with the conclusion. Never throw out data that supports your finding. 

Specification uncertainty

"He who does not specify what he wants, deserves what he gets."

How do you properly measure something when the specification is unclear? If the specification is subject to several different interpretations about its true meaning, then several different actions will follow. Random processes produce random results; and possibly random engine failures.  It's ineffective to have the true meaning of the specification divulged during the accident investigation.

This is a FAA Approved torque specification for an aircraft engine is subject to several different interpretations. How would you interpret the torque value: 90-110 Foot Pounds or 7.5 - 9.2 Inch pounds

Interpretation #

1. I can use either an inch pound or foot pound wrench and set it anywhere between the two numbers.

2. I can use either an inch pound or foot pound wrench and set it in the middle of the two numbers. Why do I have to do the math - couldn't the specification just state what I should set my wrench too?

3. If I use a foot pound wrench it must have tenth's of a pound resolution otherwise I use my inch pound wrench.

4. I use my foot pound wrench and set it in the middle, but I really can't because the middle is 8.35 and my foot pound wrench doesn't read in hundredth's of a pound.

5. This is the minimum and maximum applied torque that the joint can tolerate and still perform correctly. The chart above out of an engine engine overhaul manual illustrates the design flaw on the 590-610 value. No torque wrench can achieve this kind of accuracy.

6. The range of numbers must be a limit tolerance showing the minimum and maximum allowable torque. My foot pound wrench is not accurate enough so I use my inch pound wrench and it has an accuracy of 5%. I set it at 100 inch pounds. 5% of 100 is 95 to 105 so I am within tolerance. BUT, I cannot use my wrenches on the 590-610 (49.2-50.8) torque because they require a wrench with an accuracy of 1.7%! If this is a limit tolerance, then this torque is impossible since no torque wrench is accurate to 1.7%. (NASA uses a "preload uncertainty" of +-25% for torquing lubricated bolts on the space shuttle.) So this must not be a limit tolerance.
7. Mechanic sets his wrench at the middle of the range (assembly torque) and quality control uses the minimum and maximum as an inspection torque.

Either these torques are a range of admissible torques, where the mechanic can select any value. True torque may fall outside the range due to torque wrench errors; or it is a tolerance limit where the values are boundaries and applied torque must be accurate enough to stay within the boundary. The problem with interpreting this chart as a limit is that the limits are impossibly narrow for a torque wrench. The tolerance limits must always be bigger than the capability of the selected process. The design establishes the tolerance limits but the tolerance specifications are not based on what the design requires or what the engineer wants; but what he can get. If the tolerance specification cannot be met, then the design is flawed.

Its also unclear what the mechanic should set as the "target" or nominal value. The target torque should be clearly defined. The chart should tell the guy pulling the wrench what to set it to, rather than have him do the math. This reduces math and interpretation errors.

 There are times where a range of admissible torque values are necessary. The most common example is the torque for castellated nuts where the hole in the bolt must be lined up with the gap in the nut. You can use whatever torque within the range that lines up with the hole.

FAA and NTSB Torque Confusion

The FAA and NTSB seem confused about the difference between a torque value, range, and limit. A "limit" or "tolerance limit" is a boundary within which measurement variation must stay within. In other words an accuracy statement. A range of torque, or values, is a bunch of admissible torque values with no expressed tolerance or accuracy statement. But notice how both the FAA and NTSB muddle these two important concepts.

• AC43.13B Table 7-1 Recommended Torque Values, but the column is named "TORQUE LIMITS". 
• FAA Docket No. 2004-CE-11-AD; AD 2004-22-28 uses the term "range" to describe how to apply a torque value "...tighten the "B" nuts to specified torque ranges..." 
• AD2005-12-51 introduces the term "torque values" to express a single bolt torque -adding further confusion as to intent.
• NTSB Safety Recommendation A93-43 thru 45 dated March 18, 1993. "These manuals state that the adapter nut is to be tightened against the oil pump housing to a preload torque range between 50 to 60 ft.- lb. using a Cessna wrench adapter..." 

For the Alaska Airlines Flight 261 Disaster, one organization, The Aircraft Mechanic's Fraternal Organization (AMFA), recognized that using a range of torque (250- 300 inch pounds) might produce less accurate measurements.

"Amend the End Play check procedure to apply only one specific torque value, rather than a value range such as 250-300 inch lbs. This would help in constant readings during the End Play check." The FAA and NTSB have ignored this safety suggestion.

The use of the word "range" implies that the mechanic could set his torque wrench at any value within the range (250 to 300). This would produce a "range" of actual torque among mechanics. Whereas if everyone used a single specific torque value (target), there would have been less variability.

The idea of a torque range goes back in history before "digital" settings on torque wrenches. The first wrenches were of the beam style with a scale. When you apply torque with a beam wrench your muscles twitch causing the reading to vary slightly. For example, a 80 foot pound torque, the best you could hope for is for the reading to wobble between 75 and 85. By the engineer stating a range of acceptable torque, he is stating that try to keep the wrench between these two values. With the arrival of the snap style and the digital wrench, the mechanic sets the wrench to a number rather than shooting for a range.

When Measurements Go Wrong

"They don't check too bad if you don't check em to good"

 

The only claim a mechanic can make in regards to a measurement is that it is as good as any reasonable mechanic could make using the tools and working conditions present at the time of measurement. Unfortunately, this might not be good enough. A failure to meet specification may result in physical injury to someone. The mechanic knows that he must act on the basis of such measurements and that his action will reveal his mistakes, and what is more important, he knows that such mistakes may carry with them serious consequences. It is the responsibility of any reasonable mechanic to be "sure" of his measurements and to make sure the conditions meet the requirements set forth:

• Specifications and instructions are complete and without any confusion in their meaning or implantation.
• There is adequate time to perform the task.
• Working conditions are standardized.
• Proper tooling is used.

The above requirements for good measurement involve others in the measurement process who share the responsibility for the outcome, whether good or bad.

"Mechanic's aren't perfect, we have a tolerance."

Beware of Tight Tolerances

"I admit it isn't in tolerance but you gotta admit it's pretty darn close!"

 

Tolerances serve to define the maximum deviation from the ideal value that will still allow proper function. Tighter (narrower) tolerances do NOT mean a higher quality part, just a more expensive one. A part that is designed to accommodate a wider range of variation is less likely to fail due to things like contamination, thermal expansion, wear, and repair methods. Tight torque tolerances in a torque specification should be a warning to the mechanic that the joint is torque critical. It should be a warning to other interested parties that there might be a design problem.

Rounding and Estimating

If a measurement falls between the increments (lines), should you estimate between the lines? Is this dial indicator above indicating 36-1/2 or 36 or 37? A rule of metrology states that an instrument can never perform better than its resolution and any number between the lines is uncertain. Therefore, record measurements to the same resolution as the instrument. If the measurement falls between lines, then round your measurement to the nearest number (36 or 37). Now that we've read the "text-book" recommendation. Back to the real world. Do not round measurements! If you are a machinist or cabinet maker, you will make parts and cabinets that won't fit. If you are a mechanic, you will pass unairworthy parts.

Round only where specification or industry standard accepts the practice. In our dial indicator example, report your measurement as 36-1/2 or your best observed estimate of where you believe the needle is at. Why not to round:

1. Rounding produces a value that you didn't observe.
2. Rounding is arbitrary and involves "guess work"
3. Rounding introduces a measurement error between the reported value and the true value.

You can never be faulted for describing an observation to the best of your ability. You can be faulted for recording a number you didn't observe. The practice of "not rounding" is called "reading between the lines" and is common practice with any machinist. There are words to describe the area between the lines that engineers and purist might find quaint, but they work. Words such as "a little fat" or 'a hair over" or if the measurement falls exactly on the line, "dead-nuts on". The area between the lines might not yield an exact measurement, but it still produces useful information that the experienced mechanic uses to his advantage. Kinda sorta.

"A hair under 3-1/2 inch"

Instrument Accuracy -- Torque Wrenches

"There is no sense in being precise, when you don't even know what you're talking about.''

A torque wrench that is accurate to 5% does not mean that your torques are accurate to 5%. Accuracy is the closeness of a measurement to the exact value. It has two components:

1. the accuracy of the instrument. 5% in this example.

2. the accuracy of your use of it. An analogy is the rated life of a light bulb is 100 hours. On an airplane it lasts only 20 hours. The 100 hours is under lab conditions - our 20 hours are under a vibrating aircraft cowling. Lab life is useful for comparison shopping, but of little use in predicting performance under actual conditions. Same holds true for torque wrenches: The calibration accuracy of a torque wrench is made under lab conditions where:

1. Temperature is controlled at 68 degrees F.

2. The wrench is mounted horizontally to eliminate gravity effects.

3. The wrench is aligned carefully so that no side-loads are applied to the wrench.

4. A mechanical puller smoothly and slowly pulls the wrench until it snaps.

5. Five warm-up pulls are made before measurements begin to condition the wrench.

6. Ten measurements are made at each wrench setting and averaged.

When you pull the wrench it's often mounted vertically; temperature is never 68 degrees F.; side-loading and pulling speed varies no matter how careful we are. This is why NASA gives a torque wrench an accuracy of +- 25% in typical in-use conditions.


"3.3.1 Although this is probably the most popular method of determining the preload in a bolt, its accuracy may be as poor as ±30%, even though the torque applied may be accurate to within ±1%".  British DEF STAN 00-970 DESIGN AND AIRWORTHINESS REQUIREMENTS FOR SERVICE AIRCRAFT VOLUME 2 - ROTORCRAFT

Mechanic's take measurements under some awful conditions. For the Alaska Airlines accident mentioned previously, conditions were such that a large error could be expected. It's the responsibility of employer's to provide the best possible measurement conditions, and the responsibility of engineer's to allow for measurement errors because of poor conditions or limited access. Engineering is responsible to develop a measurement specification where the best accuracy that can be achieved in actual conditions, and is good enough for the appliance to operate satisfactorily. 

When Disaster Strikes

For 30 years I have watched as, in a totally random manner, cylinders break loose from crankcases. Preventing this is the reason I have written for years about the importance of proper torque. In every case I am aware of the NTSB and FAA "chocks it up to the mechanic not tightening the nuts." This philosophy of conjecture rather than analysis results in a conclusion based on ignorance rather than facts. Could it be that the joint was designed to fail? In other words, statistically, the joint is expected to fail whenever a torque falls on the tail of the probability curve. A rare event but with a large enough population an event that would appear from time to time (randomanly!). Afterall, a torque specification results in a wide range of outcomes (NASA's figures +-25% torque preload uncertainty).

The other possibility is based on "random procedures produce random results". The random nature of these failures point toward mechanic's using random torque procedures - in other words not following the manual! Follow the manual as closely as possible - especially the type of lubricant specified, the placement of the lubricant, the use of calibrated torque wrenches, and the proper torque sequence. Do not accept alternative processes without engineering approval.

Suspect Data

The discussion has been on conflicting data. Suspect data occurs when the customer doesn't believe your measurements or your conclusions. Take for example aircraft engine horsepower comparisons. One shop advertises that their engines produce X more horsepower than the factory. Is this an accurate comparison? Is the data suspect? Slight differences in one engine measurement test and another have a large affect on horsepower output. Change the accessory loading, oil viscosity, air intake, exhaust; whether the engine is driving a propeller or driving a loaded shaft, and the measured horsepower changes. There is a violation of Principle #1. "The measurement does not follow an agreed on process." Both horsepower measurements may be correct, but the comparison is not. The same engine produces different horsepower when tested in a slightly different manner. There are other situations where the data itself is suspect.

First Impressions Always Matter 

If I walk into an aircraft maintenance shop and see that they do not use torque wrenches, an instant impression is formed that is not good. Another shop uses torque wrenches and measurement tools, but they look abused; no calibration stickers, no evidence of measurement control or testing; my impression is not so good. In either shop I might doubt their data and their recommendations. If I walk into a shop and see calibration stickers, proper storage, a measurement area complete with devices to do spot tests on measurement equipment, then I know I'm in a professional shop and I am willing to accept their data - even though I might not like what they are telling me.

For you young mechanics, you're judged by the impression you make more than the quality of your work. Not only customers, but inspector's, don't have a deep enough understanding of the repair process to judge your workmanship. But they easily judge your appearance. You might reject a very expensive part based on a wear measurement that the customer can't see. It takes a lot of belief in your capability and judgment to accept your data. The more expensive the part, the more faith is required. Whether the customer or inspector believes you or not depends upon your appearance and your presentation. Present data in a professional manner. Clearly state your measurement, the limit, and then your conclusion. This is much better than a statement that simply says "worn out or worn beyond limits."

• Use calibrated instruments and use calibration stickers. Display calibration reports or stickers so that they are visible to customers.
• Use measurement training certificates and awards in the customer area.
• Perform in-house measurement tests.
• Talk in a professional manner.
• Dress in a professional manner.

Measurement Records

Measurement data that supports your decisions should be recorded. By regulation or industry standards, other data may be recorded, who's purpose is to assist in quality audits, regulatory audits, and legal actions against you. Extreme care needs to go into the planning on the form and content of measurement records so that records assist you and not convict you. First step in planning measurement records is to decide what measurements need not be recorded:

• Measurement data that is not used for quality control should NOT be recorded. This type of data is subject to misinterpretation at a later data. “Data has no meaning apart from its context."  This includes:
• Set-up measurements used for adjusting or testing the measurement fixture.
• Test measurements to establish if the measurement test fixtures are working to your satisfaction.
• In-process measurements taken to judge the progress of your work.
•  Also included in this list are any measurements that do not follow a process.. If a measurement record doesn't contain the process then it's impossible to know what was measured and how it was measured.· By not recording these measurements you avoid someone in the future incorrectly assigning some significance or meaning to the measurement.

Several aircraft accidents caused by Measurement Errors

Mars Climate Orbiter - Conversion Error - Managers refused to accept early indicates of error

Air Canada Flight 143 - Ran out of fuel - Conversion Error

Alaska Airlines Flight 261 - Measurement Error

Air France Flight 447 - Airspeed Inconsistencies

Other Articles you might be interested in

What is Crankshaft Run-Out?

Lobing and other Low Frequency Form Error In Aircraft Piston Engines