Significance and Use

American National Standards Institute Inc.

4.1 Using best practices for SGPE ensures that decisions made will be based on scientific principles, and the outputs obtained will be more objective than those evaluation sessions conducted without this planning, structure, focus, and best practices. These small group evaluations contrast with more formal product tests that include a prequalified participant sample, hypothesis testing, and statistical analysis. Without best sensory practices and procedures, SGPE may be unstructured, unsystematic, difficult to manage, and may lead to outputs that are unclear, not credible, or ignored. Additionally, the use of proper sensory practices reduces bias among participants with specific sample knowledge or a desire to advance an agenda. This guide provides a framework for conceptualizing, organizing, and executing these SGPE.

4.2 SGPE are used in situations in which formal, hypothesis-driven product evaluations are not required. These include situations in which the decision risk is small or stakeholders feel comfortable in making a decision with the attendant risks, or both. Examples of these situations may include limited availability of samples or other resources, potential patent exposure, or low incidence of target population. The SGPE could be an initial screening step or a precursor test before a more formal product test. In the proper context, SGPE can also be a decision-making tool in and of itself. Using the framework presented here provides a degree of rigor that may be absent when a few people evaluate a product without controlled conditions. A poster presented at the 2009 Pangborn Sensory Science Symposium (1)3 reported the results of a survey on SGPE. 59 % of respondents (N = 92) stated that, at their place of employment, typically, non-sensory professionals organized SGPE. Table 1 summarizes key differences between a typical unstructured product evaluation with a small group not following best practices and an SGPE that follows the best practices outlined in this guide.

Scope

1.1 This guide covers those occasions in which a small group of individuals (generally between three and ten) with potentially different functional roles and degrees of training in sensory and product evaluation, evaluates a product or series of products for a specific objective, with a pre-identified decision to be made, but without the use of formal hypothesis testing or statistics. In the product testing industry, these are often referred to as “benchings,” “cuttings,” or “bench screenings” or, in the case of food products, “tastings,” “informal tastings,” “team tastings,” or “technical tastings.” In this guide, the term “Small Group Product Evaluation” (SGPE) is used.

1.2 The aim of this guide is to provide best practices to ensure that SGPE are conducted with sufficient rigor to enable the most appropriate decision or to yield the needed learning while considering the risk. Because the participants may be heterogeneous with respect to functional role, knowledge of the issue at hand, sensory sensitivity, and degree of sensory or product evaluation training, the likelihood of agreement on a path forward is not assured. Additionally, participants may have certain biases with respect to the issue to be decided, because of prior knowledge or their role within the organization. These potential derailers can be addressed through proper planning and execution of an SGPE. When SGPE are unstructured, unfocused and experimental error and biases uncontrolled, the outputs of SGPEs do not inform decisions or deliver the desired learning in a scientific manner. The goal of this document is to elevate the practice of small group product evaluations by outlining a structure, defining decision criteria in advance, and providing guidelines for implementation, drawing upon existing sensory theory and methods. Outputs from these SGPE are used to inform decisions and determine next steps including the risks involved with each of these.

SGPE are widely used, and when properly conducted, are an option in the sensory professional’s toolbox. SGPE should be conducted only when the risks are known, stated, and shared. Limited timing and resources alone are not adequate reasons to utilize SPGE testing and forgo formal sensory testing. Risks in doing so must be clearly communicated and agreed to by all involved parties.

The proper uses of SGPE are several: to screen variables, to establish hypotheses, to gain information about a product set or category, to take a course of action where a low risk product decision is needed or for product learning throughout a development program. In all of these cases, the team must accept the risks that come with having SGPE outputs to inform a decision. One risk involved in SGPE is missing small differences among products (beta risk), when the goal of the evaluation is to find such differences, particularly those differences that might be important to the consumer. An SGPE failure to find differences does not mean that product similarity or equivalence is established, since much larger sample sizes than are common to SPGE’s are required to establish similarity/equivalence.

1.3 This guide covers the planning and implementation processes, including objective setting, method determination, number and types of participants, ballots, sample preparation, decision criteria, products to be included, review of information collected, and management of the post-product evaluation discussion to arrive at a decision within the small group. Documenting and communicating SGPE outputs are also covered, as well as next steps if a decision cannot be reached. Worked examples across industries including food, household, and personal care are included. The different types of SGPE covered include those commonly executed but is not exhaustive.

1.4 This guide does not cover the use of small group evaluations to pilot research or test protocols before implementation in larger scale testing. In addition, the use of small group evaluations to substitute for larger evaluations that incorporate formal hypothesis testing and statistical analysis or to replace hedonic testing are neither recommended nor included within this guide. SGPE that are regular activities of a quality function and product reviews that are done for demonstration or informative purposes with no defined decision criteria are also not covered in this guide.

1.5 See 5.2 for a best practice recommendation for the role of the sensory professional or trained delegate in the planning, designing, conducting, or oversight of structured SGPE.

1.6 This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety, health, and environmental practices and determine the applicability of regulatory limitations prior to use.

1.7 This international standard was developed in accordance with internationally recognized principles on standardization established in the Decision on Principles for the Development of International Standards, Guides and Recommendations issued by the World Trade Organization Technical Barriers to Trade (TBT) Committee.

Go to ASTM E3093 at ASTM.org

Add your thoughts about the standard.

1375 - Demolition Materials (549) 1377 - Cartridge and Propellant Actuated Devices and Components (524) 4720 - Hose and Flexible Tubing (552) 4730 - Hose, Pipe, Tube, Lubrication, and Railing Fittings (1823) 5120 - Hand Tools, Nonedged, Nonpowered (1124) 5305 - Screws (781) 5306 - Bolts (996) 5310 - Nuts and Washers (864) 5330 - Packing and Gasket Materials (583) 5340 - Hardware, Commercial (953) 5905 - Resistor (753) 5910 - Capacitors (783) 5930 - Switches (1012) 5935 - Connectors, Electrical (4357) 5940 - Lugs, Terminals, and Terminal Strips (564) 5945 - Relays and Solenoids (733) 5950 - Coils and Transformers (856) 5960 - Electron Tubes and Associated Hardware (1630) 5961 - Semiconductor Devices and Associated Hardware (707) 6145 - Wire and Cable, Electrical (1629) 6240 - Electric Lamps (662) 6505 - Drugs and Biologicals (1343) 6515 - Medical and Surgical Instruments, Equipment, and Supplies (1907) 6520 - Dental Instruments, Equipment, and Supplies (790) 6530 - Hospital Furniture, Equipment, Utensils, and Supplies (745) 6610 - Flight Instruments (540) 6625 - Electrical and Electronic Properties Measuring and Testing Instruments (1241) 6640 - Laboratory Equipment and Supplies (1187) 6810 - Chemicals (1114) 8010 - Paints, Dopes, Varnishes, and Related Products (2046) 8030 - Preservative and Sealing Compounds (554) 8140 - Ammunition and Nuclear Ordnance Boxes, Packages and Special Containers (797) 8305 - Textile Fabrics (910) 8415 - Clothing, Special Purpose (660) 8455 - Badges and Insignia (3093) 8915 - Fruits and Vegetables (531) 9330 - Plastics Fabricated Materials (759) FACR (1647) ISDA (1776) ISDD (786) ISDF (1827) ISDN (581) MISC (1045) PACK (539) SESS (581)
Aerospace Material (225) Aircraft Air Conditioning, Heating, and Pressurizing Equipment (334) Aircraft Hydraulic, Vacuum, and De-icing System Components (263) Ammunition, 75mm through 125mm (192) Ammunition, over 30mm up to 75mm (208) Ammunition, through 30mm (349) Analytical Chemistry (180) Bombs (192) Building (414) Bulk Explosives (261) Cartridge and Propellant Actuated Devices and Components (524) Consumer Product Evaluation (267) Copper (192) Demolition Materials (549) Electrical and Magnetic Conductor (162) Electronics (153) Environmental Toxicology (187) Fire Control Computing Sights and Devices (338) Fire Control Radar Equipment, except Airborne (197) Fuzes and Primers (454) Geotechnical Engineering (217) GUN (640) Land Mines (276) Medical Device and Implant (427) Military Chemical Agents (156) Miscellaneous Aircraft Accessories and Components (231) Miscellaneous Fire Control Equipment (216) Nondestructive Testing (246) Nonferrous Metal and Nonferrous Alloy (367) Nuclear Technology (274) Optical Sighting and Ranging Equipment (451) Paint and Related Coating (456) Parachutes; Aerial Pick Up, Delivery, Recovery Systems; and Cargo Tie Down Equipment (324) Petroleum (378) Plastic Pipe (248) Pyrotechnics (175) Road and Paving (177) Rockets, Rocket Ammunition and Rocket Components (368) Rubber (166) Security System Pedestrian and Walkway Safety (157) Specialized Test and Handling Equipment, Nuclear Ordnance (150) Sports and Recreation (316) Steel (908) Torpedos and Components, Inert (258) (621)