Basic Considerations for Argument and Evidence Evaluation

This post considers a set of useful mental tools for evaluating reasons and evidence.

We will begin with a Bayesian treatment of missing evidence. The central question is straightforward: if a thorough search fails to uncover evidence that we would expect to find if a claim were true, how should that absence affect our confidence in the claim?

On this page

1. Part I — Missing Evidence and Bayesian Reasoning
   1. Definitions
   2. A Preliminary Probability Point
   3. Implications of the Bayesian Result
   4. Critical Questions for Evaluating Evidence
2. Part II — Limits, Counterexamples, and Explanatory Reasoning
   1. When Confirmation Can Mislead: Black Swans and the Turkey
   2. The Raven Paradox and Evidential Relevance
   3. Abduction and Alternative Explanations
   4. Evidence as Argument and as Story
   5. Evaluating Statistical and Research Evidence
   6. A Working Conception of Evidence
3. Part III — Analyzing Arguments
   1. Step 1: Clarify and Define Terms
   2. Step 2: Formulate the Argument
   3. Step 3: Determine whether the conclusion follows (if the premises are true)
   4. Step 4: Examine Each Premise’s Justification
   5. Step 5: See if there are reasons to think the premises are false
   6. Step 6: Practice Intellectual Virtue
   7. Additional Tips
4. Part IV — Reference Toolkit
   1. Dialogue Types
   2. Argumentation Schemes and Critical Questions

Part I — Missing Evidence and Bayesian Reasoning

This first part develops the narrow probabilistic question before turning to the complications that arise in real-world evidential reasoning.

Definitions

1. Let $X$ be an arbitrary event, or combination of arbitrary events.
2. Let $\neg X$ denote "NOT X"
3. Let $E$ represent the existence of a set of positive evidence that would indicate the reality of X
4. Let $\neg E$ denote "NOT E", or the total absence of positive evidence.
5. Let $P(x)$ denote the probability of event $X$.
6. Let $P(X \mid E)$ denote the conditional probability of X given E

By definition, this conditional probability is the joint probability of X AND E divided by the probability of E:

$P(X \mid E) = \frac{P(X \cap E)}{P(E)}$

A Preliminary Probability Point

7. Let X be the intersection of two statistically independent events $X = A \cap B$.

• $P(A \cap B) \le P(A \cup B)$
• This follows from the laws of probability and accords with the common-sense point that adding conjunctive requirements cannot make an event more probable.
• The more “AND” conjuncts a claim contains, necessarily the lower the probability. See the Conjunction Fallacy for further discussion.

Assumptions

1. Event $X$ will very likely leave some type of evidence if the event occurs. The probability of E (observing evidence), given X, is greater than the probability of NOT E (missing evidence), given X. A significant event will leave evidence, if it does not leave evidence then how are we to determine whether it happened? $P(X \mid E) > P(\neg X \mid E)$

• $P(X \mid E) > P(\neg X \mid E)$
• $1- P(X \mid E) > P(\neg X \mid E)$
• $P(\neg X \mid E) < \frac{1}{2}$

2. If the event X is extraordinary, then all else equal, the probability of event X is very small: $P(X) << 1$

• The probability of no evidence existing is high, the more we search for evidence but fail to find any, the probability approaches 1.

3. After an exhaustive search for evidence, we have not found any. In the places we have expected to find evidence, we have not found any.

• $P(\neg E) \approx 1$

Applying Bayes’ Theorem

$P(\neg E \mid X) = \frac{P( X \mid \neg E)P(\neg E)}{P(X)}$

From Assumption (1):

$\frac{1}{2} > \frac{P( X \mid \neg E)P(\neg E)}{P(X)}$

$P( X \mid \neg E) < (\frac{1}{2})(\frac{P(X)}{P(\neg E)})$

• Consider the numerator $P(X)$ approaching zero and denominator $P(\neg E)$ approaching 1. Plug in $\frac{.01}{.99} = 0.01$
• $This\; should\; make\; sense;\; the\; more\; unlikely\; X,\; the\; more\; likely\; probability\; of\; NOT\; E.$
• $The\; ratio\; will\; yield\; a\; small\; number,\; and\; the\; conditional\; probability\; will\; be\; smaller.$

$P( X \mid \neg E) < (\frac{1}{2})(\frac{P(X)}{P(\neg E)}) < 1$

$P( X \mid \neg E) < (\frac{1}{2})$ implies $P( \neg X \mid \neg E) > (\frac{1}{2})$

Combining the terms yields:

$P( \neg X \mid \neg E) > P( X \mid \neg E)$

• This means that given a lack of evidence for X, it is likely that X did not occur. This shows that the probability of NOT X given NOT E will be greater than the probability of X given NOT E.

Implications of the Bayesian Result

These criteria can help narrow the search among competing explanations. When X already has a low likelihood and a thorough search produces no expected evidence for X, NOT X becomes more likely than X. Of course, someone may dispute the prior probability of X or argue that particular artifacts should count as relevant evidence. Those questions must be settled—or at least made explicit—before the probability assignment can do useful work.

Within a positive relevance framework, anything that increases the probability of a proposition counts as positive evidence; anything that decreases its probability counts as negative evidence; and evidence that leaves the probability unchanged is irrelevant. In Bayesian confirmation theory, the standard positive-relevance account says that E is evidence for H if and only if Pr(H|E) > Pr(H). E is evidence against H when Pr(H|E) < Pr(H), and neutral with respect to H when Pr(H|E) = Pr(H).

In principle, this framework lets us compare competing claims by asking how each item of evidence changes their relative probability. See Theories of Evidential Relation for a more thorough discussion.

On this framework, missing expected evidence for a grand claim is a reason to lower our confidence in that claim—or, at minimum, to give serious consideration to alternative explanations with greater probabilistic merit. This provides a lower bound for responsibly adopting extraordinary claims.

I think this is a natural extension of the Sagan Standard aphorism: “Extraordinary claims require extraordinary evidence.” In Bayesian terms, an extraordinary claim begins with a low prior probability relative to more ordinary alternatives. There will therefore have to be a significant amount of evidence—or evidence whose likelihood approaches P(X) = 1—in order for the posterior probability to move substantially toward accepting the extraordinary claim. Intuitively, this is why extraordinary claims carry a higher burden of proof; that is almost part of what it means for a claim to be extraordinary. Otherwise, how would we distinguish competing claims if both required the same burden of proof? Put differently, improbable claims generally require stronger evidence—evidence that significantly increases the posterior probability—than claims already probable given the background conditions.

Critical Questions for Evaluating Evidence

Some critical questions to consider when confronted with evidence for some claim X:

1. If there is evidence for X, is the evidence strong? Does it significantly increase the conditional probability P(X|E)? Evidence strength need not necessarily correlate with evidence amount; a single piece of strong evidence can outweigh multiple pieces of weak evidence.

2. If there is strong evidence for X, is the prior probability of X so low that it renders the posterior probability of X to be fairly low as well, even after taking the evidence into account?

3. If there is evidence for X, is it overpowered by the counterevidence for NOT X?

4. If there appears to be evidence for X, is it understated evidence? This happens when there is generic evidence in favor of some proposition, but after consideration of more specific pieces of evidence, the generic evidence seems to understate the total evidence. Definition: This fallacy is committed when one uses some relatively general known fact about X to support a hypothesis when a more specific fact about X (that is also known to obtain) fails to support that hypothesis.

For example, a prosecutor might mislead a jury by pointing out the defendant bought a knife days before the victim was stabbed, neglecting to mention that the knife purchased was a butter knife. (Paul Draper). Another Example: You see your neighbor bought a new safe car, so you conclude they must care about the safety of their children. You peer through the windows and see there are no seatbelts in the car. The initial general evidence is overridden by the more specific evidence.

A few further considerations follow. Some facts can raise the probability of a hypothesis even while the hypothesis remains unlikely overall. Likewise, a true hypothesis may still have evidence against it. We therefore have to distinguish evidential direction from overall probability and consider the strength of the evidence, the total evidence, the prior and posterior probabilities of the hypothesis, and any alternative or unexplored hypotheses that may turn out to be more likely.

Back to contents ↑

Part II — Limits, Counterexamples, and Explanatory Reasoning

The Bayesian result is useful, but its force depends on what evidence we should expect, how complete the search has been, and which alternative explanations remain available.

When Confirmation Can Mislead: Black Swans and the Turkey

There is another aphorism that might come to mind in response to the latter: "Absent evidence is not evidence of absence". Let’s consider the parable of the Turkey from Nassim Taleb's book The Black Swan. Black Swan events are incredibly rare, improbable, events with significant consequences. However, in hindsight we see them as more predictable due to a variety of reasons such as the Lucretius Problem, Survivorship Bias, Narrative Fallacy, Ludic Fallacy, Hindsight Bias and information asymmetry.

The parable of the turkey shows us that, complex systems that appear stable (lack volatility), are subject to regime shifts due to black swan events, despite all of the evidence confirming the hypothesis that the system is stable. "A Turkey is fed for a thousand days by a butcher; every day confirms to its staff that the butchers love turkeys 'with increased statistical confidence'. The butcher will keep feeding the turkey until a few weeks before Thanksgiving. Then comes that day when it is really not a good idea to be a Turkey" (Taleb, Antifragile, p.93).

As the turkey gains confidence in the claim "the butcher loves turkeys", the butcher surprises the turkey with an abrupt belief revision. This is a Black Swan event from the perspective of the turkey. You can think of a variety of scenarios where confirming evidence can lead you to an incorrect conclusion.

Related concepts in the original post

confirmation · evidence · falsifiability · confirmation bias · Raven Paradox

The Raven Paradox and Evidential Relevance

Hempel describes the paradox in terms of the hypothesis:^[3][4]

(1) All ravens are black. In the form of an implication, this can be expressed as: If something is a raven, then it is black.

Via contraposition, this statement is equivalent to:

(2) If something is not black, then it is not a raven.

In all circumstances where (2) is true, (1) is also true—and likewise, in all circumstances where (2) is false (i.e., if a world is imagined in which something that was not black, yet was a raven, existed), (1) is also false.

Given a general statement such as all ravens are black, a form of the same statement that refers to a specific observable instance of the general class would typically be considered to constitute evidence for that general statement. For example,

(3) My pet raven is black.

is evidence supporting the hypothesis that all ravens are black.

The paradox arises when this same process is applied to statement (2). On sighting a green apple, one can observe:

(4) This green apple is not black, and it is not a raven.

By the same reasoning, this statement is evidence that (2) if something is not black then it is not a raven. But since (as above) this statement is logically equivalent to (1) all ravens are black, it follows that the sight of a green apple is evidence supporting the notion that all ravens are black. This conclusion seems paradoxical because it implies that information has been gained about ravens by looking at an apple.

Abduction and Alternative Explanations

Despite these problems, absence of evidence can still count as evidence of absence when the relevant conditions hold. Suppose a theory predicts that, under a particular scenario, evidence of type X should appear with high probability. You test the theory and fail to find that evidence. The failure does not deductively prove that the initial conditions were absent, but it does count against them. This is closely related to abductive reasoning: we compare explanations of what we observe and prefer those that best account for both the evidence that is present and the evidence that is missing.

Consider the hypothesis that a burglar is hiding in your home. You search the entire house, verify that entry points show no signs of tampering, and perhaps even use a trained dog to detect traces you might otherwise miss. Someone can always preserve the burglar hypothesis by inventing an additional possibility—perhaps the burglar has an invisibility cloak—but treating every logically possible rescue as equally serious makes ordinary reasoning impractical.

A lack of confirming evidence therefore does not guarantee that H is false; rather, its force depends on whether we had good reason to expect evidence if H were true. The following argumentation scheme for abductive reasoning captures that structure; the premises can be negated or adapted to fit the case above:

• F is a finding or given set of facts.
• E is a satisfactory explanation of F. 
• No alternative explanation E' given so far is as satisfactory as E. 
• Therefore, E is plausible, as a hypothesis. 

And the associated critical questions:

• CQ 1: How satisfactory is E itself as an explanation of F, apart from the alternative explanations available so far in the dialogue?
• CQ2: How much better an explanation is E than the alternative explanations available so far in the dialogue?
• CQ3: How far has the dialogue progressed? If the dialogue is an inquiry, how thorough has the search been in the investigation of the case?
• CQ4: Would it be better to continue the dialogue further, instead of drawing a conclusion at this point?

Evidence as Argument and as Story

There is a paper called "Arguments, Stories, and Evidence: Critical Questions for Fact-Finding" by argumentation theorists Floris Bex and Bart Verheij, that I find to be incredibly useful with the analysis of evidence. Both of these scholars study the overlap between Evidence, Argumentation, Law, and AI. As indicated by the title, evidence is usually presented to us within the frame of a story; evidence is crafted in story format similar to the distinct argumentation schemes structuring common patterns of reasoning. Naturally, we can formulate a set of critical questions associated with the story, argument, and evidence to assess its merit. The stories lay the facts out in a structured order to elicit cause-effect relationships among the actors in the story; this allows the storyteller to provide counterfactual accounts of the evidence and assign motive.

The authors describe this framework in the context of a criminal investigation, but it generalizes to other contexts as well where stories are constructed around a body of evidence. "Key questions in a narrative approach include how to establish the coherence and quality of stories (the search for plausibility criteria), when to believe a story (the issue of justification of the belief in a story) and how to choose between alternative stories (the issue of story comparison)." (Verheij p.1).

Like Argumentation schemes, Story Schemes are patterns of common instantiated stories; argument schemes are commonly instantiated patterns of argument (such as analogy, goal-oriented reason, abduction, etc.), the story schemes have a structure and associated set of critical questions relevant to the story scheme being instantiated. Bex and Verheij offer a hybrid argumentative-narrative approach; the model combines elements from both frameworks, (story schemes and argumentation schemes).  In the narrative approach such knowledge takes the form of general scenarios that can be seen as story schemes (Bex 2009), standard general event-patterns that act as a background for particular instantiated stories.

In the argumentative approach, evidence is organized into premise → conclusion structures, with warrants in the Toulmin sense guiding the generalizations that connect them. I list additional schemes later in the post; for now, consider the following argument from witness testimony:

• Witness w is in a position to know whether a is true or not.
• Witness w asserts that a is true (false).
• Therefore, a may plausibly be taken to be true (false). 

"In the argumentative approach, the individual facts at issue are supported by the evidence in the case through arguments. The argument-based approach is inherently dialectical: not only evidence supporting the probanda but also evidence against them should be considered, and any sources of doubt in the arguments should be made explicit." (Verheij p.3). Critical questions can be used to assess the particular instantiation. Each argument scheme has a different set of questions. For witness testimony:

• Was w in a position to know whether a is true?
• Is w truthful?
• Is w biased? 
• Is w’s statement that a internally consistent? 
• How plausible is w’s statement that a? 
• Is a consistent with what other witnesses say?
• Have all of the relevant witnesses been considered? 

In the narrative approach, facts are arranged into sequences of events about factual/counterfactual scenarios (called stories); the evidence is used to causally explain the possible alternative hypotheticals. The evidence is explained abductively; "The basic idea of abductive inference (see e.g. Walton 2001) is that if we have a general rule ‘c is a cause for e’ and we observe e, we are allowed to infer c as a possible hypothetical explanation of the effect e. This cause c which is used to explain the effect can be a single state or event, but it can also be a sequence of events, a story." (Verheij p.4).

So in the narrative approach, we will have a set of data/evidence, and causal links represented as a directed graph, showing the sequence of events leading to the outcome, justified with abduction. Evidence by itself is atomic, it does not indicate any direction of causality; we must discern the direction of causality and this is typically done within a story schema.

In the case of a criminal trial, you will be likely wanting to invoke the scheme for intentional actions presented by Pennington and Hastie (1993) "motive → goal → action → consequences"; the evidence is usually explained by reference to this story schema, but there are other variants in the literature. The idea is that the evidence will "fit" into this broader explanatory/story schema; this gives meaning to the isolated pieces of evidence, and directs our attention to the search for additional/contrary evidence.

There are many criteria by which we evaluate abductive schemas " The choice between these alternative stories depends on how well the individual stories explain the evidence and how coherent (Thagard 2004) each of them is. The coherence of a story largely depends on whether the story conforms to our general commonsense knowledge of the world, that is, whether we deem the story to be inherently plausible (i.e. without considering the evidence in the case). Here, story schemes play an important role (see Bex 2009)." (Verheij p.5).

The key point is that in the narrative approach, evidence is explained causally, based on hypothetical stories that "make sense" of the data better than others. It differs from the argumentative approach in that we aren't constructing arguments in support of individual pieces of evidence, rather we are holistically connecting them into a story. Many people reason about masses of evidence exactly this way.

The hybrid approach connects these two forms of reasoning about facts and evidence. In the argumentative approach, facts are justified through arguments based on evidence; in the narrative approach, facts are assessed partly by how well they fit into a larger explanatory story. The hybrid model combines them: story schemes represent causal relationships among events, while argument schemes anchor the evidential claims used within those stories. Unanchored evidence weakens a story because the causal narrative then rests on claims that have not themselves been adequately supported. Conversely, when an argument supporting an evidential claim is successfully attacked, part of the story’s evidential anchor is weakened. This hybrid framework gives rise to a further set of critical questions:

• (CQ1) Are the facts of the case made sufficiently explicit in a story? A case should contain a clearly phrased, sufficiently specified and coherent story detailing “what happened”. 
• (CQ2) Does the story conform to the evidence? a. Is the story sufficiently supported by the evidence in the case? b. Is the story contradicted by evidence in the case? In general, not all elements of a story can be supported by evidence. This does not need to be a problem, and is in fact unavoidable as certain story elements must by their nature be indirectly justified. When an element of a story is not supported by a piece of evidence (in a given argument), we speak of an 'evidential gap'. Some gaps can be inferred from other facts, depending on the strength of the other evidence. 
• (CQ3) Is the support that the evidence gives to the story sufficiently relevant and strong?  a. Are the reasoning steps from evidence to events in the story justified by warranting generalizations and argument schemes that are sufficiently strong and grounded? b. Are there exceptions to the use of the generalizations and schemes that undermine the connection between evidence and fact (based on the Toulmin framework, assessed by critical questions)? 
• (CQ4) Has the story itself been sufficiently critically assessed? a. Is the story sufficiently coherent? Are there required elements missing? Are there implausible events or causal relations? Is the story inconsistent? Here coherence has a specific meaning, namely that the story fits our knowledge and expectations about the world we live in. In other words, a story should be complete (i.e. have all its essential parts) and plausible (i.e. have plausible causal relations) b. Have story consequences been used to test the story? In other words, are there implications from the story we should expect to observe if the story were true?
• (CQ5) Have alternative stories been sufficiently taken into account? a. Has a sufficient search for alternative explanations been performed, not only in the investigative phase, but also in court? b. Are there good reasons to choose one story over the alternatives? Have the alternatives been sufficiently refuted?
• (CQ6) Have all opposing reasons been weighed? Have all considerations that are used to weigh opposing reasons been made explicit? Has this been done both at the level of individual facts and events and at the level of stories? 

If you are interested in this type of research check out Analyzing Stories Using Schemes by F.J. Bex,  Anchored Narratives in Reasoning about Evidence, and Towards a Formal Account of Reasoning about Evidence: Argumentation Schemes and Generalizations.

Evaluating Statistical and Research Evidence

So far, the discussion has treated evidence in general terms. The next section narrows the focus to statistical and research evidence at a high level.

What happens when we face a large body of statistical evidence produced by studies of varying strength and methodology? Conflicting evidence is difficult to reason through, and the answer we want may not be available with confidence. Evidence hierarchies can provide useful guidance because study designs differ in the kinds of inferences they can support. But the hierarchy should not be applied mechanically. The best design for supporting a warrant is the one that most closely matches the claim being made; the strength and generality of the claim should correspond to an appropriate research design. Comparisons among studies therefore take place within a broader process of critical questioning, rebuttal, and scrutiny of assumptions and design choices. The Cochrane Handbook for Systematic Reviews of Interventions, the PRISMA  statements , and other reporting guidelines frameworks provide useful starting points for that kind of appraisal.

• What is the current state of knowledge on this topic? Remember that evidence is often incomplete.
• Is reliable information available from the research literature? If so, what kinds of studies are available?
• How extensive is the literature?
• Is there a critical appraisal guideline?
• How much of the argument depends on factual claims—what is the case—and how much depends on normative claims—what one therefore ought to do?
• How well supported is the inference from fact to value or action?

From these considerations, several general rules follow:

• When assessing an argument that relies on empirical claims, first determine the type of study being cited.
• Once the study design has been identified, consult the relevant evidence hierarchy. In general, stronger designs warrant greater credence when they are well executed and appropriately matched to the claim. Thus, a strong systematic review will ordinarily be more reliable than a single randomized trial, and a cohort study will ordinarily be more reliable than a case-control study. There are important exceptions, so design type should guide rather than replace critical appraisal.
• The absence of a randomized trial does not mean that no good evidence supports a warrant. Match the standard of proof to the question, and do not demand evidence that cannot reasonably exist.
• The context of the argument can often indicate the strength of evidence required.
• Always assess arguments for unstated assumptions that relate factual claims to action or belief claims.

A Working Conception of Evidence

This is just a starting point. Evidential reasoning and evaluation is a rich topic. I will end this topic with a quote and working definition of evidence from The Evidential Foundations of Probabilistic Reasoning by David Schum:

"In any inference task, our evidence is always incomplete, rarely conclusive, and often imprecise or vague; it comes from sources having any gradation of credibility. As a result, conclusions reached from evidence having these attributes can only be probabilistic (and fallibilistic , my insertion) in nature. Probabilistic reasoning requires many difficult judgements in the process of establishing the credentials of evidence in terms of its relevance, credibility, and inferential force. No evidence comes to us with these credentials already established".

"Achinstein provides several interpretations of evidence including one he favors (1982, 322-336). He says that e is potential evidence on hypothesis H if and only if (1) e is true, (2) e does not make H necessary, (3) the probability of H on evidence e is substantial, and (4) the probability of an explanatory connection between H and e is substantial. Achinstein's characterization is valuable because it summarizes several points about evidence that we must examine quite carefully. First, we will often have uncertainty about whether or not the evidence e 'is true', and we have to be able to represent the nature of this uncertainty. In part, this will involve specifying exactly what evidence we believe we have. Second, if evidence does not entail (or make necessary) hypothesis H, this simply means that e is inconclusive; this is essentially what I (and Hacking) have stipulated so far. But assumptions 3 and 4 raise issues concerning relevance and the force of e on H and the nature of the argument or chain of reasoning that links e and H" (p. 15).

I like this conception because it emphasizes that evidence does not simply “stand on its own.” We have to explain why a piece of evidence bears on a hypothesis and how much inferential force it should carry. With that foundation in place, we can turn from the evaluation of evidence to the evaluation of arguments.

Back to contents ↑

Part III — Analyzing Arguments

The preceding sections focused on evidence. This section shifts from evidence itself to the arguments built from it: how to reconstruct those arguments, test their inferential structure, and assess the support for their premises.

Step 1: Clarify and Define Terms

• Get clear on terms, concepts, and distinctions operative in the argument.
• Providing clean necessary and sufficient conditions for every term may not be feasible. When such conditions are available, however, they give you something explicit to test.
• Clarification helps avoid ambiguity and equivocation.
• Definitions should illuminate rather than deepen vagueness, and they should avoid circularity.
• Be aware of definition techniques.
  • Circular definition
  • Stipulative Definition
  • Precising Definition
  • Persuasive Definition
  • Ostensive Definition
  • Operational Definition
  • Lexical Definitions
  • Genus-Differentia definition
  • Extensional and Intensional Definitions
  • Coordinative Definition

Step 2: Formulate the Argument

• Structure the argument in premise → conclusion form so that what you are trying to demonstrate is explicit.
• Make the argument’s assumptions as explicit as possible.

Step 2.1: Determine the Kind of Argument

• Deductive: truth of premises is supposed to guarantee the truth of the conclusion. The premises are supposed to logically entail the conclusion. If all of the premises are true, then the conclusion must be true. You can evaluate these arguments by checking if they commit Syllogistic Fallacies (Formal Fallacies)

• Non-Deductive: The truth of the premises does not guarantee the truth of the conclusion, but is meant merely to probabilistically (or plausibly) support the truth of the conclusion. It is logically possible for all premises to be true while the conclusion is false. Most of our arguments fall into this domain. They are to be considered defeasible (subject to exceptions).

  • Inductive Arguments
  • Abductive or Inference to Best Explanation
  • Case Based Reasoning
  • Argument from Precedent
  • Argument from Analogy
  • Counterfactual conditionals
  • Conductive Argument (Pro/Con)
  • Cost Benefit Analysis
  • Anchored Narratives
  • etc...

• These fall within the domain of "informal logic". This overlaps with argumentation theory as well. The logical form of the argument is insufficient to guarantee its soundness. These types of arguments are typically embedded in a dialogue; a goal-directed, collaborative communicative exchange between one-many parties. Douglas Walton suggests several forms of dialogue:

  • The "critical discussion" occurs where the goal is to resolve a conflict of opinions.
  • The "persuasion dialogue", broader than the critical discussion, occurs where one side attempts to prove a thesis using premises accepted by the other side.
  • The "negotiation type of dialogue" occurs where the goal is to make a deal.
  • The "quarrel" occurs where the goal is a better personal relationship between the parties.
  • The "information seeking type of dialogue" occurs where the goal is to transfer information from one party to another.
  • The "deliberation" occurs where two parties are trying to decide what action to take when prompted by a practical problem.
  • The "inquiry" occurs where the goal is to prove something to a high standard.

• Evaluating an argument requires understanding the intended relationship between its premises and conclusion. Identifying the structure helps determine whether those relationships are strong enough to justify accepting the conclusion.

Step 2.2: Find the Main Conclusion

• Identifying the conclusion is sometimes straightforward. In other cases, look for conclusion indicators; in informal dialogue, the conclusion may remain implicit and need to be reconstructed.

Indicator
So
Hence
Accordingly
Thus
It follows that
As a result
For this reason
Consequently
This means that
Which implies/entails/shows that
We can conclude that

Step 2.3: Find the Premises

• Premises are the reasons on which a speaker bases a conclusion. Premise indicators, like conclusion indicators, often signal that a reason is being offered in support of a claim.

Indicator
Since
Because
For
As shown by / As indicated by
Assuming that / Given that / Seeing that
Owing to
The reason is that / considering that
Due to the fact that
In light of / However
But/And (At the beginning of sentence)

Step 2.4: Put the Argument in Premise-by-Premise Form

• Take the premises you have identified.
• List them in numbered order.
• Note the logical connections among them, where applicable, and identify any sub-conclusions on which the main conclusion depends.
• List the main conclusion and state how it is supposed to follow from the premises.
• The form of the reconstructed argument will depend on the type of argument identified in Step 2.1.
• The content of each premise must be consistent; the meaning of terms must remain stable throughout the argument.
• When an argument has no obvious structure, apply the principle of charity: reconstruct the premises in the most plausible form consistent with the speaker’s intent, and avoid creating a straw man.
• Some arguments remain implicit; reformulate them in the most faithful way while remaining consistent with authorial intent.

Step 3: Determine whether the conclusion follows (if the premises are true)

• Assuming the premises are true, does the conclusion follow?
• For deductive arguments, check for logical validity. If the structure is valid, then you can move on to assessing the truth of the premises contained in the body of the argument.
• For non-deductive arguments, assess the strength of the inference: do the premises probabilistically support the conclusion? Do they make the conclusion more likely than not, all else equal? In ordinary dialogue we rarely assign precise probabilities, so this stage often reduces to a judgment of plausibility. The degree of plausibility assigned to a premise typically depends on background assumptions, so make explicit the warrants that connect the premises.
• At this stage, we are assessing the formal or structural features of the argument rather than the content of the premises. It is possible for an argument to be valid yet only insignificantly increase the likelihood of the conclusion being true in cases of weak evidence.

Step 4: Examine Each Premise’s Justification

• Now begin posing critical questions about the argument and the justification for each premise. This includes identifying informal fallacies, checking for inconsistencies, assessing whether the relevant burden of proof has been met, evaluating the strength and relevance of the evidence, and looking for defeaters.
• Remain charitable throughout this process. Ask for a premise to be restated when it is unclear, ask clarifying questions, and remain intellectually curious. Do not rush to accuse an interlocutor of committing a fallacy; misapplying fallacy labels is itself poor reasoning and undermines charitable interpretation.
• Remember that the objective is to use argumentation to move toward the truth of a claim. Overusing or underusing these tools can pull the exchange away from that goal.
• See this Master List of Fallacies

Step 4.1: Assess the Epistemic Merits of the Reasons

• Are the reasons justified or are they simply assertions?
• Are they plausible?
• Do they rest on implicit assumptions needing justification?
• Do the reasons conflict with other propositions we are justified in accepting?

Step 4.2: Assess the Connection Between the Reasons and Premises

• Do the reasons actually lend credence to the premise? Are they genuinely supportive, or are they irrelevant?
• Would the reasons only support the premise with the addition of an implicit linking claim? If so, is that claim justified? Is there an implicit warrant needing explication and scrutiny?
• If the reasons are supposed to only probabilistically support the premise, do they actually make the premise more probable?

Step 4.3: Assess the Dialectical Merits of the Reasons

• Are the reasons question-begging? That is, do the reasons presuppose the truth of the very premise they are supposed to support (or the truth of the ultimate conclusion of the main argument)? Would one already have to accept the relevant premise or conclusion in order to be justified in accepting the reason offered on behalf of that premise?
• Ask yourself: In order to accept one of the premises, would you have had to accept the conclusion beforehand?
• In question-begging reasoning, one assumes the truth of the very claim one is trying to support.
• Do the reasons illicitly shift the burden of proof? Is the person advancing the argument implying the detractor ought to retain the burden of proof after they question the truth of a premise?
• Do the reasons equally support a claim incompatible with the premise they are meant to support? Do they support a claim incompatible with something else in the argument?

Step 5: See if there are reasons to think the premises are false

• Even justified premises can be false. At this stage, directly test them by constructing counterarguments and asking whether there are good reasons to reject them.
• Look for counterexamples, rebutting defeaters, etc.
• For a non-deductive argument, the relevant question is whether it is cogent: strong and supported by true premises.
• Consider how the reasons work together. Ask whether the conjunction of the premises provides enough support to accept the conclusion, then consider the inverse: taken collectively, do the counterexamples or counterarguments provide enough weight to reject the conclusion or favor its negation? The task is to assess the combined force of the reasons rather than treating each one in isolation.
• Apply the same method to each individual premise. If the justification for a premise is considerably weak in light of counterevidence and counterexamples, then we have reason to believe the premise is false. If the premise is false, this logically transfers to the conclusion of the main argument being false.

Step 6: Practice Intellectual Virtue

• Argumentation should be a mutual endeavor among parties jointly trying to discover the truth.
• Doing this well requires intellectual virtues: stable, habitual, excellent character traits that orient us toward truth.
• These are dispositions that are conducive to the exchange of ideas.
• Here we are referring to behavioral traits of the individual needed to foster effective exchange.

Additional Tips

1. Act as if you are on the same team. This humanizes the other person and keeps the shared commitment to discovering truth in view.
2. Be willing to learn from your interlocutor. This is how you grow and better understand your own position.
3. Do not be afraid to say “I don’t know.” Be willing to live with uncertainty.
4. Put truth and love at the center of the exchange. Orient yourself away from “winning the debate” and from egocentric tendencies that can distort the pursuit of truth.
5. Diversify your information sources. Do not rely on sources that only serve to reaffirm your beliefs. Be open to multiple fresh perspectives. Seek out objections to your views. Do not lock yourself in an echo chamber. Make friends with people who have different worldviews. Really try to empathize with their position and background information and circumstances.
6. Explore rather than expose; reject the “game” mindset. Treating discussion as a contest encourages tribalism rather than inquiry.
7. You are not your ideas and beliefs. The moment you conflate your identity with your position on certain topics, any counter-argument (or questioning) to your position will necessarily be seen as an attack. You then get defensive and put up barriers. Your value as an individual is independent of the beliefs/ideas you allege to have.
8. Put away the boxes, labels, and caricatures of the "kind of person" you are interacting with. No one fits perfectly into these preconceived boxes; and we certainly belong to multiple overlapping categories. We each have individual perspectives we can bring to the conversation. Do not presuppose everything there is to know about your interlocutor simply based on their group membership. All your preconceived filtering will do is obstruct the exchange of ideas. We are all a work in progress.
9. Go slowly when analyzing arguments. Prefer methodical, systematic reflection to quick “gotcha” responses.
10. Steelman your interlocutor rather than constructing a straw man. Doing so reduces tribalism and signals a commitment to truth.
11. Do not psychologize or over-pathologize your interlocutor.
12. De-weaponize argument. Use arguments as tools for illuminating reality and exploring it together, not as weapons for “defeating” an opponent.

Some additional intellectual virtues:

Back to contents ↑

Part IV — Reference Toolkit

The material below is best used as a reference: dialogue types help identify the goal of an exchange, while argumentation schemes provide recurring patterns of reasoning and critical questions for evaluating them.

Now some basic argument patterns, critical questions, and considerations from the Douglas Walton paradigm. Douglas Walton proposes a classification of dialogue types that characterize common types of interactions (source). I am not sure if this is an exhaustive list, but nevertheless it's important to consider the type of dialogue you are in before applying a set of standards to the exchange. You can think of these dialogue types as broad structural features that characterize different goals and objectives for coming together with an interlocutor to speak about something. The types of arguments someone proposes will likely vary conditional on the type of dialogue instantiated. It is also important to note that Walton never claims these to be mutually exclusive; rather they can occur at different stages of a much larger dialogue and rapidly shift between different one type or another. Dialogue types can also be embedded in another dialogue type; you can imagine that within a debate there might also be information seeking. 

Dialogue Types

Type of Dialogue	Initial Situation	Individual Goals of Participants	Collective Goal of Dialogue	Benefits
Persuasion	Difference of opinion	Persuade other party	Resolve difference of opinion	Understand positions
Inquiry	Ignorance	Contribute Findings	Prove or Disprove Conjecture	Obtain Knowledge
Deliberation	Contemplation of future consequences	Promote personal goals	Act on a Thoughtful Basis	Formulate Personal Priorities
Negotiation	Conflict of Interest	Maximize Gains (self-interest)	Settlement (without undue inequity)	Harmony
Information Seeking	One Party lacks information	Obtain information	Transfer of Knowledge	Help in Goal Activity
Quarrel (Eristic)	Personal Conflict	Verbally hit out and humiliate opponent	Reveal deeper conflict	Vent Emotions
Debate	Adversarial	Persuade Third Party	Air strongest arguments for both sides	Spread Information
Pedagogical	Ignorance of One Party	Teaching and Learning	Transfer of Knowledge	Reserve Transfer

Being aware of the dialogue type helps focus critical questions on the features most relevant to the exchange. The following argumentation schemes—stereotypical patterns of reasoning—are included for reference:

Argumentation Schemes and Critical Questions

The point of this post is to consider some very basic critical thinking patterns when prompted with evidence and reasons for accepting a claim. Courtesy of Philosophical Disquisitions, here are some more argument schemes, taken from Walton's Fundamentals of Critical Argumentation:

Defeasible Modus Ponens

1. Data: P.
2.  Warrant: As a rule, if P, then Q. Therefore, . . . 
3. Qualifier: presumably, . . . 
4. Claim: . . . Q. 

Critical Questions: 

1. Backing: What reason is there to accept that, as a rule, if P, then Q? 
2. Rebuttal: Is the present case an exception to the rule that if P, then Q?

Argument from an Established Rule

1. Major Premise: If carrying out types of actions including A is the established rule for x, then (unless the case is an exception), x must carry out A. 
2. Minor Premise: Carrying out types of actions including A is the established rule for a. 
3. Conclusion: Therefore, a must carry out A.

Critical Questions:

1. Does the rule require carrying out types of actions that include A as an instance? 
2. Are there other established rules that might conflict with or override this one? 
3. Is this case an exceptional one, that is, could there be extenuating circumstances or an excuse for noncompliance? 

Practical Inference

1. Major Premise: I have a goal G. 
2. Minor Premise: Carrying out this action A is a means to realize G. 
3. Conclusion: Therefore, I ought (practically speaking) to carry out this action A.

Critical Questions:

1. What other goals that I have that might conflict with G should be considered? 
2. What alternative actions to my bringing about A that would also bring about G should be considered? 
3. Among bringing about A and these alternative actions, which is arguably the most efficient? 
4. What grounds are there for arguing it is practically possible for me to bring about A? 
5. What consequences of my bringing about A should also be taken into account?

Argument from sign: Observation is evidence of existence of an event or property.

1. What is the strength of the correlation of the sign with the event signified?
2. Are there other events that would more reliably account for the sign? Argument from example. An example is used to support a generalization. 1. Is the proposition presented by the example in fact true?
3. Does the example support the general claim it is supposed to be an instance of?
4. Is the example typical?
5. How strong is the generalization?
6. Are there special circumstances in the example that impair its generalizability?

Evidence to a hypothesis. If A then B, B is observed, therefore A is true.

1. Is it the case that if A is true then B is true?
2. Has B been observed to be true (or false)?
3. Could there be some reason why B is true, other than it being because A is true?

Correlation to cause. There is a correlation between A and B, therefore A causes B?

1. Is there a large number of instances?
2. Is there a reverse causal relationship?
3. Can a common cause be ruled out?
4. Are there mediating variables?
5. Are changes in B due to how defined?

Cause to effect. A tends to cause B, and A has occurred; therefore B.

1. How likely is B, given A?
2. What’s evidence for generalization?
3. Are there counteracting factors?

Waste. Don’t stop trying to realize A or all your previous efforts will be wasted?

1. Could past efforts still payoff?
2. Is A possible?
3. Does the value of realizing A outweigh the cost of continuing?

Ethos. If a is a person of good moral character, then what a contends (A) is more plausible.

1. Is “a” of good moral character?
2. Is a’s character relevant?
3. How strong a weight of presumption in favor of A is warranted?

Bias. Arguer (“a”) is biased, so likely has not taken evidence on both sides of issue into account.

1. Is the dialogue of the type that requires participants to take both sides into account?
2. What is the evidence that a is biased? Established rule. If everyone is expected to do A, then you must do so too. 1. Is doing A in fact what the rule states?
3. Does the rule apply to this case?
4. Are other rules involved?
5. Are there reasons for an exception?

Gradualism. If you take a first step (A), you will eventually be caught up in bad consequences.

1. Is A what’s being proposed?
2. Do any of the casual links in the sequence lack solid evidence?
3. Does the outcome plausibly follow, and is it as bad as suggested? Types:
   • Causal slippery slope.
   • Precedent slippery slope.
   • Classification vagueness.
   • Arbitrary classification.
   • Verbal slippery slope.

Need for help. Person y should help person x if x needs help, and y can help without it being too costly for him or her to do so.

1. Would the proposed action A really help x?
2. Is it possible for x to really carry out A?
3. Would there be negative effects of carrying out A that would be too great?

Resources:

1. Virtuous Virtues

2. How to Analyze Arguments Like a Philosopher

3. Dr. Graham Oppy on the Nature of Arguments

4. Pursuing the Truth: A Guide to Critical Thinking

5. Other Models of Argumentation

    - Toulmin Model of Argumentation

    - Scriven Model of Argumentation

    - The Dialectical Method of Evaluating Argument

6. Argumentation, Dialogue Theory, and Probability Modeling: Alternative Frameworks for Argumentation Research in Education