Preface

Why another book on probability?

This book has two titles. The subtitle, ‘Fundamentals for the Empirical Sciences’, reflects the intentions and the motivation of the first author for writing this book. He received his academic training in psychology but considers himself a methodologist. His scientific interest is in explicating fundamental concepts of empirical research (such as causal effects and latent variables) in terms of a language that is precise and at the same time compatible with the statistical models used in the analysis of empirical data. Applying statistical models aims at estimating and testing hypotheses about parameters such as expectations, variances, covariances, and so on (or of functions of these parameters, such as differences between expectations, ratios of variances, regression coefficients, etc.), all of which are terms of probability theory. Precision is necessary for securing logical consistency of theories, whereas compatibility of theories about real-world phenomena with statistical models is crucial for probing the empirical validity of theoretical propositions via statistical inference.

Much empirical research uses some kind of regression in order to investigate how the expectation of one random variable depends on the values of one or more other random variables. This is true for analysis of variance, regression analysis, applications of the general linear model and the generalized linear model, factor analysis, structural equation modeling, hierarchical linear modeling, and the analysis of qualitative data. Using these methods, we aim at learning about specific regressions. A regression is a synonym for what, in probability theory, is called a factorization of a conditional expectation, provided that the regressor is numerical. This explains the main title of this book, ‘Probability and Conditional Expectation’.

What is it about?

Since the seminal book of Kolmogoroff (1933–1977), the fundamental concepts of probability theory are considered to be special concepts of measure theory. A probability measure is a special finite measure, random variables are special measurable mappings, and expectations of random variables are integrals of measurable mappings with respect to a probability measure. This motivates Part I of this book with three chapters on the measure-theoretical foundations of probability theory. Although at first sight this part seems to be far off from practical applications, the contrary is true. This part is indispensable for probability theory and for its applications in empirical sciences. This applies not only to the concepts of a measure and an integral but also, in particular, to the concept of a measurable mapping, although we concede that the full relevance of this concept will become apparent only in the chapters on conditional expectations. The relevance of measurable mappings is also the reason why chapter 2 is more detailed than the corresponding chapters in other books on measure theory.

Part II of the book is fairly conventional. The material covered – probability, random variable, expectation, variance, covariance, and some distributions – is found in many books on probability and statistics.

Part III is not only the longest; it is also the core of the book that distinguishes it from other books on probability or on probability and statistics. Only a few of these other books contain detailed chapters on conditional expectations. Exceptions are Billingsley (1995), Fristedt and Gray (1997), and Hoffmann-Jørgensen (1994). Our book does not cover any statistical model. However, we treat in much detail what we are estimating and which the hypotheses are that we test or evaluate in statistical modeling. How we are estimating is important, but what we are estimating is of most interest from the empirical scientist point of view, and this point is typically neglected in books on statistics and in books on probability theory such as Bauer (1996) or Klenke (2013). A simple example is the meaning of the coefficient β1 in the equation E(Y | X, Z) = β0 + β1X + β2Z + β3ZX. Oftentimes, this coefficient is misinterpreted as the ‘main effect’ of X. However, sometimes β1 has no autonomous meaning at all, for example if P(Z = 0) = 0. In general, this coefficient is just a component of the function g1(Z) = β1 + β3Z that can be used to compute the conditional effects of X on Y for various values z of Z (see chapter 15 for more details). The crucial point is that such concepts can be treated most clearly within probability theory, without referring to a statistical model, sample, estimation, or testing.

This also includes exemplifying the limitations of conditional expectations. Simple examples show that conditional expectations do not necessarily serve the purpose of the empirical researcher, which often is to evaluate the effects of an intervention on an outcome variable. But even in these situations, conditional expectations are indispensable for the definition of parameters and other terms of substantive interest (see, e.g., chapter 14).

There is much overlap of Parts II and III with Steyer (2003). However, that book is written in German, and the mathematics is considerably less rigorous. Aside from mathematical precision, the two books also differ in the definition of an important concept: In Steyer (2003), the term regression is used as a synonym of a conditional expectation, whereas in this book we use it as a synonym for the factorization of a conditional expectation , provided that the codomain of X is .

In chapter 9, the first chapter of Part III, we gently introduce conditional expectation values and discrete conditional expectations. In chapter 10, we then present the general theory of conditional expectations that has been introduced by Kolmogoroff (1933–1977) and since that time has been treated in many books on probability theory – although much too briefly in order to be intelligible to researchers in empirical sciences. Our chapter on conditional expectations contains many more details and is supplemented by a number of other chapters on important special aspects and special cases.

Such a special aspect is the concept of a residual with respect to a conditional expectation (see chapter 11). Residuals have many interesting properties, and they are used in order to introduce the concepts of conditional variance and covariance, as well as the notion of a partial correlation. We then turn to specific parameterizations of a conditional expectation, including the concepts of a linear regression (chapter 12) and a linear logistic regression (chapter 13). Note that these concepts are introduced as probabilistic concepts. As mentioned, they are what we aim at estimating in applying the corresponding statistical models.

Chapters 14 to 16 provide the probabilistic foundations of the analysis of conditional and average effects of treatments, interventions, or expositions to potentially harmful or beneficial environments. To our knowledge, this material is not found in any other textbook. Note, however, that although these two chapters provide important concepts, they do not cover the theory of causal effects, which is another book project of the first author.

Part IV uses conditional expectations in order to introduce conditional independence (chapter 16) and conditional distributions (chapter 17). Although these two chapters are more extensive than comparable chapters or sections in other books, the material is found in other books on probability theory as well.

For whom is it?

This book has been written for two kinds of readers. The first are applied statisticians and empirical researchers who want to understand in a proper language (i.e., in terms of probability theory) what they estimate and test in their empirical studies. The second kind of readers are mathematicians who want to understand in terms of probability theory what applied statisticians and empirical researchers estimate and test in their research. Both kinds of readers are potential contributors to the methodology of empirical sciences.

Many exercises and their solutions provide extensive material for assignments in courses, but they also facilitate independent learning. At the same time, these exercises and their solutions help streamline the main text.

Note that we do not provide all proofs, in particular in the chapters on measure, integral, and distributions. In these cases, we refer to other textbooks instead. We decided to include only those proofs that may help to increase understanding of the background and to learn important mathematical procedures. Of course, we provide proofs of all propositions for which we did not find an appropriate reference.

Prerequisites

We assume that the reader is familiar with the elementary concepts of logic, sets, functions, sequences, and matrices, as presented for example in chapters 1 and 2 of Rosen (2012). We try to stick to his notation as closely as possible.

One of the exceptions is the symbol for the implication, for which we use ⇒ instead of →. Another exception is the symbol for the equivalence, for which we use ⇔ instead of ↔.

Box 0.1 summarizes the most important notation to start with. The concepts referred to by these symbols are defined, for example, in Rosen (2012) or in Ellis and Gulick (2006). For a rich...