After we have cataloged the elements of reasoning, we must evaluate texts and our own reasoning for clarity, accuracy, precision, relevance, depth, breadth, significance, logic, and fairness. When making a decision with incomplete information, it is critical to recognize that truth is often a degree of belief based on our evaluation of the quality of the information and reasoning.

1. Evaluate point of view

• Strive to be fair-minded in evaluating all points of view and identify their strengths and weaknesses
  • Playing the devil's advocate by arguing from a different point of view is a powerful exercise
• After reading a text, examine how much influence the author's point of view had on you

 Critically evaluate the reliability of an author (and publisher):

• What qualifications does the author have for writing on this subject? (Or what are the qualifications of the people the author quotes?)
• Based on your research on the author's background, what factors may have influenced his or her point of view?
• When and where was the article first published? Does this information affect the credibility of the article?

 Compare and contrast points of view to reveal how related material is presented by different authors and different purposes of their writing. After reading two texts on the same topic, ask yourself:

• What is the author's point of view in each of these articles?
• Why do you think that the points of view presented are so different?
• How much influence did each author's point of view have on you?

 

1A. Evaluate a Scientific Author's Qualifications

• Examine the primary source of information. ls there a reference to the source of information? If not, it cannot be verified. If so, is the source reputable?

• Examine the reputation of the author. Do the author(s) have training in science? If so, have they had formal training leading to an advanced degree such as a Master's degree or doctorate, and have they published widely in reputable journals? If not, then are they working with a reputable scientist(s) to evaluate the data?
• Does the discoverer say that a powerful establishment is trying to suppress his or her work? Often, the discoverer describes mainstream science as part of a larger conspiracy that includes industry and government. The idea is that the establishment will presumably stop at nothing to suppress discoveries that might shift the balance of wealth and power in society. This is not how science actually works. Science is an open and international enterprise focused on uncovering true descriptions of reality.

• Determine if the work was published in a peer-reviewed journal. Peer review is the standard process for scientific publications. Peer-reviewed manuscripts have been read by several scholars in the same field (called peers), and these peers have indicated that the experiments and conclusions meets the standards of their discipline and are suitable for publication. In the absence of peer-review the significance and quality of the data cannot be assessed.
• Has the discovery been pitched directly to the media? The integrity of science rests on the willingness of scientists to expose new ideas and findings to the scrutiny of other scientists. Thus, scientists expect their colleagues to reveal new findings to them initially. An attempt to bypass peer review by taking a new result directly to the media, and thence to the public, suggests that the work is unlikely to stand up to close examination by other scientists.

• Check if the journal has a good reputation for scientific research. If a peer-reviewed paper is cited, where was it published? Is the journal widely respected? One tool that is commonly used for ranking, evaluating, categorizing, and comparing journals is the frequency with which the "average article" in a journal has been cited in a particular year or period. The frequency of citation reflects acknowledgment of importance by the scientific community. High-impact and widely respected journals include Science and Nature. Therefore, a citation in Science generally suggests scholarly acceptance, whereas publication in a nonscientific or little-known journal does not.

• Determine if there is an independent confirmation by another published study. Even if a study is peer-reviewed and published in a reputable journal, independent assessment is critical to confirm or extend the findings. Even the best journals or scientists will occasionally make mistakes and publish papers that are later retracted. Sometimes there may be outright fabrication that is overlooked by the reviewers and not detected until later. In other cases, the scientific report may be accurate but its significance may be misrepresented by the media. Although it is a slow process ro establish a scientific "truth," a particular scientific conclusion will eventually either gain broad acceptance or be discarded.

• Assess whether a potential conflict of interest exists. Most of the high-impact journals require a conflict of interest statment on the first page of an article.

• Assess the quality of institution or panel. Does the report emanate from a University accredited by the U.S. Department of Education or equivalent society? Such information is generally more reliable than that issued from a single individual putting information out on the web. In the United States, government research arms such as the National Science Foundation and the National Institute of Health and professional scientific societies generally provide up-to-date, high-quality information.

 

2. Evaluate of Degree of Truth in Information

After analyzing to identify the different kinds of information, we must be explicit about the quality of each piece of information used in the text or our own thinking. Using the highest quality information in arguments increases the degree of belief in the truth of the argument. We must acknowledge when poor quality information is used in an argument and clearly state that we have low confidence in the truth of the argument.

• Search for information that opposes your position as well as information that supports it
• Make sure that all information used is clear, accurate, and relevant to the question at issue
• Make sure you have gathered sufficient information
• We can have the most confidence in facts that have been confirmed by many different independent observers.
  

A scientist's perspective on facts

In everyday language most of us consider a confirmed fact to be truth. However, scientists consider all truth to be provisional, the current facts serve as description of truth only for the time being. Scientists assume that all knowledge has the potential to be overturned if new information suggests that it should be. Scientists use the uncertainty and percent confidence to describe the statistical likelihood that a fact is true.

Physicist Richard P. Feynman once said, "In physics and in human affairs... whatever is not surrounded by uncertainty, cannot be the truth." He said this in reference to a newspaper article that asserted absolute belief in a scandalous rumor regarding a colleague. He observed that a responsible reporter should have referred to an "alleged incident." With no reference to a process that had first evaluated the quality of the truth, he considered accusation to be opinion, not fact.

Examples:

• Is a particular measurement 78 ± 50 or 78 ± 1 meters? As you can see, the uncertainty deeply affects how you will use that information.
• It is a scientific formalism that any measurement missing a stated uncertainty has an uncertainty of ±1 in a last significant digit. Therefore, 78 seconds is understood to be 78 ± 1 seconds and 78.0 seconds is 78.0 ± 0.1 seconds.
• "The crash test results indicate a 98% chance that a head-on collision will kill you. As a professional scientist I cannot say that a head-on collision will kill you."

This last example highlight the property that all scientific information is actually a statement probability. Nothing in science is ever "proven" or "100% certain." Always avoid saying that science has proven something. This is a discipline-specific error in reasoning commonly made by non-scientists. Non-scientists sometimes misinterpret when scientists attach uncertainty to every fact. If there is 95% confidence that climate change is being caused by human activity, people with a psychological bias to avoid taking action around this crisis may focus on the 5% uncertainty in the truth value. On the other hand, people who are convinced of this fact and want to take action get frustrated that scientists refuse to say that it has been proven, we are certain. In practice, 95% confidence in science is the gold standard for a complex phenomenon being "as good as proven," but scientists always keep open the possibility that they don't have all the data and keep open the possibility that this fact may be more nuanced or simply wrong in the future.

Comparing and Contrasting Information

By comparing and contrasting information, you can identify facts, make inferences, and draw conclusions that would not otherwise be possible. After reading two texts, ask yourself:

• How do the articles differ in the information each one presents?
• Are the articles different in how they present information?
• Does the information appear to be complete and accurate? Why or why not?

 

2. Evaluate assumptions

[Unfinished]

Contrasting Assumptions

If two sides are arguing from different assumptions, it is very effective to focus on these in critical evaluation. Controversies generally rest on different sides interpreting the same information through different assumptions.

Assumptions, can be unjustified or justified, depending upon whether we do or do not have good reasons for them. Likewise, if two sides of a controversy share assumptions that are found faulty, both arguments become invalid.

Example:

• Ethan Nadelmann, founder and executive director of the Drug Policy Foundation, argues that law enforcement officials are overzealous in prosecuting individuals for marijuana possession citing that 87% of marijuana arrests are for possession of small amounts.
• The Office of National Drug Control Policy (ONDCP) contends that marijuana is not a harmless drug and must remain restricted. Besides causing physical problems, marijuana affects academic performance and emotional adjustment.
• Underlying both of their arguments is the assumption that adults cannot be permitted to make their own decisions about the use of particular drugs as they choose. A libertarian who worries about governmental restrictions on personal liberty would immediately recognize this shared deep assumption and challenge it. If convincingly challenged, both arguments lose validity.

 

3. Evaluate reasoning

When an argument doesn't "feel" right, first analyze it as follows. Write down the information that forms each premise of the argument and categorize them. Write down the conclusion and label it. Write your best general description of the reasoning that links them. The mechanics of the reasoning are usually found in a "therefore" type statement. To unmask the logic, replace the premise statements with letters that represent concepts and properties. Example: "It's raining and the sun is shining, therefore it's raining." The logical form is "X has property Q and P, therefore X has property Q". The logic is sound. [I will link some more examples later.]

3A. Logical Fallacies

Fallacies are faulty reasoning used in the construction of an argument. This topic is so vast that I have created a separate fallacies of reasoning page. The identification of fallacious reasoning invalidates an argument and we then forced to formulate our own arguments to uncover truth.

3B. Evaluate Propaganda

Propaganda is information that is not objective and is used primarily to influence an audience to reach a specific conclusion. Propaganda attempts to arouse emotions to short-circuit rational judgment. It is not by definition "good" or "bad." However, it's use indicates possible intent to, at worst mislead, or at best persuade without the use of reasoning. The techniques of propaganda are utilized in some logical fallacies and you will find some conceptual overlap. The following is a list of common propaganda techniques:

• Hacking Identity: The Pride, Fear, Outrage, Hatred Formula. Critically examine when identity categories become significant to an argument. In some cases it may be appropriate, in others it may be an emotionally manipulative red herring. Example: In recent years, the Russian government has planted appeals to pride to amplify difference and strengthen online social communities. This is then followed by stories designed to invoke fear and outrage. The effort is apparently designed to "hack" the minds of people in democratic nations into feeling disillusioned with social and political institutions.
• Stereotyping. People or objects are lumped together under simplistic labels, also called labeling. Example: Blonde women are beautiful, but dumb.
• Overgeneralizations. Treating a complex general thing as if it were a concrete thing. Example: " The UN's bureaucracy has forsaken its commitment... " or " The City extends strike deadline."

 

3C. Evaluate Cognitive Biases

A cognitive bias is a cognitive shortcut that leads to a loss of objectivity. Cognitive biases can lead to irrational thought through distortions of perceived reality, inaccurate judgment, or illogical interpretation. By learning about some of the most common biases, you can learn and how to avoid falling victim to them. The identification of cognitive biases at work in an argument should make you skeptical. Like fallacies, this topic is so vast that I have created a separate cognitive biases page to explain them.

4. Evaluate Judgments and Conclusions

After you read an article, you should be able to answer these questions:

• What judgments and conclusions were drawn by the author of this article?
• Why do you agree or disagree with the author's drawn conclusions?
  • Are their faults of reasoning that make the drawn conclusion unjustified?
  • Does the drawn conclusion challenge your assumptions?
• What other drawn conclusions are possible to draw using the same information?
• What other information might be important to know before making any judgment on the value and importance of this text?

 

5. Predict future Implications and Consequences

The alignment of reasonable future implications and consequences of a conclusion or judgment with your values should inform your reasoning.

• Trace the implications and consequences that follow from your reasoning
• Search for negative as well as positive implications
• Attempt to consider all possible consequences