Synergy, Redundancy, and Independence in Population Codes

Approximate conditional stimulus distributions

In a recent study, Nirenberg et al. (2001) studied the importance of noise correlations for how information is encoded by pairs of ganglion cells in the retina. Noise correlations can be ignored explicitly by assuming that the joint response distribution for two neurons is given by Equation 7. Bayes' rule can be used to find the stimulus distribution conditioned on the neural response for that case:

17

Nirenberg et al. (2001) denoted this quantity by p_ind(s|r₁, r₂), but we use p_shuffle to avoid confusion between different kinds of independence. They suggested using the KL divergence between the true decoding dictionary p(s|r₁, r₂) and the approximate dictionary p_shuffle(s|r₁, r₂) to quantify the amount of information that is lost by using a decoder that assumes conditional independence. Averaged over the real, correlated responses, r₁ and r₂, one obtains:

18

This measure does not refer to any specific algorithm for estimating the stimulus or errors made by that algorithm but, instead, is meant to be a general characterization of the ability of any decoder to make discriminations about the stimulus, if knowledge of the noise correlations is ignored. Nirenberg et al. (2001) argued that it is appropriate to consider an approximate decoding dictionary combined with the real spike trains, because the brain always automatically has access to the real, correlated spike trains but may make simplifying assumptions about how to decode the information that those spike trains contain. They state that $\widehat{D}$ measures the loss in information that results from ignoring correlations in the process of decoding and, thus, refer to this measure as ΔI.

Nirenberg and Latham (2003) make a connection between the KL divergence and the encoded information by using an argument about the number of yes/no questions one must ask to specify the stimulus (see below). Although this argument may initially seem reasonable, closer consideration reveals that it is flawed. This can be demonstrated by the direct contradiction that results from assuming that the KL divergence measures an information loss, as well as the contradictory implications of this argument. In particular, there are situations in which this putative information loss can be greater than the amount of information present. Furthermore, interpreting the measure $\widehat{D}$ as a general test of the importance of noise correlations for encoding information about a stimulus is problematic, because of the highly counterintuitive results that one finds when applying the measure to toy models.

Contradiction. The central claim made by Nirenberg et al. (2001) is that $\widehat{D}$ measures the amount of information about the stimulus that is lost when one ignores noise correlations. If this were true, then the information that such a decoder can capture would be given by:

19

This expression for “I_no-noise” is unusual. It does not obviously have the form of a mutual information, as is evident from the fact that the probability distribution inside the logarithm is not the same as that multiplying the logarithm. The fact that Equation 19 is not a mutual information term can be demonstrated by specific examples. Figure 4 shows one such case for a pair of model neurons that can generate three different responses (0, 1, or 2 spikes) to each of three, equally likely stimuli. The joint response distribution, p(r₁, r₂|s) is shown in Figure 4A. For this toy model, $\widehat{D}$ exceeds the total information encoded by both neurons, and, consequently, Equation 19 is negative. This example demonstrates that if one assumes that $\widehat{D}$ is an information loss, then one would sometimes lose more information than was present by ignoring noise correlations. Because the mutual information between the output of a decoder and the input stimulus cannot be negative, this is a clear contradiction. Therefore, $\widehat{D}$ is not an information loss.

Open in a separate window

Figure 4.

$\widehat{D}$ can be larger than the information that the cells encode about the stimulus. A, The conditional joint response distribution p(r₁, r₂|s), of two neurons responding to three stimuli. Each of the neurons responds with either zero, one, or two spikes. p(r₁, r₂) is the average of p(r₁, r₂|s) over the stimuli. The a priori probability of each of the stimuli equals 1/3. B, The conditional stimulus distribution for the cell pair, p(s|r₁, r₂), obtained using Bayes' rule. C, The conditional stimulus distribution that assumes no noise correlation, p_shuffle(s|r₁,r₂), obtained by inverting p(r₁|s)p(r₂|s) using Bayes' rule; see text for details. For this case, the information that both cells carry about the stimulus, I(R₁, R₂; S) equals 0.0140 bits, whereas $\widehat{D}$ equals 0.0145 bits.

Counter-intuitive properties of $\widehat{D}$ Because $\widehat{D}$ is always positive, one might wonder whether it sets a useful upper bound on the importance of noise correlations. Again rewriting:

20

where:

21

We see that both terms are non-negative, because they both have the form of a KL divergence. The first term, in fact, is , which is a measure of the strength of noise correlations and an upper bound on the synergy. Because the second term is non-negative, . Therefore, $\widehat{D}$ does not constitute an upper bound on the importance of noise correlations as is also demonstrated by specific examples in Figure 5. Even so, perhaps $\widehat{D}$ constitutes a tighter upper bound on the synergy than ? This turns out not to be the case, as shown below.

In Figure 5, we imagine a simple situation in which a pair of neurons can only generate two responses, spike or no spike, and they are only exposed to two different, equally likely stimuli. In both of these examples, the neurons fire a spike with p = 0.5 for the first stimulus, but neither fires a spike for the second stimulus. As such, they are sparse in a manner similar to many real neurons. In example A, the response to the first stimulus is perfectly anticorrelated, meaning that if one cell fires a spike, the other stays silent, and vice versa. Knowledge of this noise correlation resolves any ambiguity about the stimulus, such that the joint mutual information is one bit. Because each cell mostly remains silent in this stimulus ensemble, the individual mutual information of each cell is considerably lower, and the synergy of the cell pair is +0.377 bits. Using Bayes' rule to find the real conditional stimulus distribution and the one that ignores noise correlations, one finds that $\widehat{D}$ = 0.161 bits, or about 2.3 times smaller than the synergy. This is a strange result, because synergy can only arise from noise correlations. Thus, one naively expects that all synergy is lost when one ignores the noise correlations. Consistent with this expectation, the upper bound on the synergy, is 0.5 bits, and information lost by using shuffled spike trains is ΔI_noise = 0.451 bits.

In example B, the two cells have a complete positive correlation for the first stimulus, meaning that they either both fire a spike or both remain silent, each with p = 0.5. As before, they both remain silent for the second stimulus. In this case, the two neurons always have exactly the same response. As a result, the synergy equals -0.311 bits, which is a redundancy of 100%. However, they still have quite strong noise correlations, and $\widehat{D}$ = 0.053 bits or 16.9% of the joint mutual information, which is virtually the same fraction as in example A. This comparison indicates that $\widehat{D}$ cannot distinguish between noise correlations that lead to redundancy and those that lead to synergy. In this example, shuffling the spike trains breaks the complete redundancy of the two cells and actually increases the encoded information. Correspondingly, ΔI_noise = -0.238 bits or -76% of the joint mutual information (negative values imply that a shuffled set of responses would have more information than the original spike trains).

Figure 6 shows an example with three stimuli and neurons capable of three responses. Here, the neurons have anticorrelations that allow all three stimuli to be perfectly resolved, and the synergy equals +0.415 bits or 26.2% of the joint mutual information. However, the correlations between these cells are such that $\widehat{D}$ = 0. Interestingly, p_shuffle(s|r₁, r₂) is not equal to p(s|r₁, r₂) for all joint responses, but in all cases in which they are unequal, the joint response probability p(r₁, r₂) = 0 (for related examples and discussion, see Meister and Hosoya, 2001). This example is an extreme illustration, in which the measure $\widehat{D}$ implies that there is no cost to ignoring noise correlations, when, in fact, observing the responses of the two cells together provides substantially more information about the stimulus than expected from observations on the individual neurons in isolation. Clearly, the measure $\widehat{D}$ cannot be relied on to detect the impact of interesting and important noise correlations on the neural code.

Open in a separate window

Figure 6.

Cells may be synergistic but $\widehat{D}$ = 0. The conditional joint response distribution p(r₁, r₂|s) of two neurons responding to three stimuli, each with probability 1/3. In this case, the cells are synergistic but $\widehat{D}$ is zero: I(R₁, R₂; S) = 1.585 bits; Syn(R₁, R₂) = 0.415 bits; $\widehat{D}$ = 0 bits.

Problematic implications of $\widehat{D}$ Although Nirenberg and Latham (2003) do not attempt to explore all of the consequences of interpreting their KL divergence as a general measure of information loss, we show here that this argument leads to further contradiction. One corollary of their claim comes from its extension to cases other than assessing the impact of ignoring noise correlations (Nirenberg and Latham, 2003). Hence, one can also ask how much information is lost by a decoder built from any approximate version $\tilde{p}$(s|r₁, r₂) of the conditional stimulus distribution, and the answer, if we follow the arguments of Nirenberg and Latham (2003), must be D_KL[p(s|r₁, r₂)||$\tilde{p}$(s|r₁, r₂)]. However, in general, the KL divergence between p(s|r₁, r₂) and $\tilde{p}$(s|r₁, r₂) can be infinite, if for some value of s, r₁ and r₂, $\tilde{p}$ = 0 and p ≠ 0. This result is clearly impossible to interpret.

Another corollary is that the information loss resulting from ignoring noise correlations is defined for every joint response (r₁,r₂). This means that we can also use the formalism to determine how much information the decoder loses when acting on the shuffled spike trains. This expression is:

22

However, we have shown above that the mutual information that a pair of neurons conveys about the stimulus under the assumption of conditional independence is I_shuffle and the consequent difference in mutual information is ΔI_noise. Equation 22 is not identical to ΔI_noise. In particular, ΔI_noise can be either positive or negative, because the assumption of conditional independence sometimes implies a gain of information rather than a loss (Abbott and Dayan, 1999). This typically occurs when the neurons have positive correlations (Fig. 5, example B) (Petersen et al., 2001). In this case, shuffling the spike trains actually reduces their joint noise and, therefore, can increase the information conveyed about the stimulus. In contrast, Equation 22 is never negative, implying that there is always a loss of information. Thus, another contradiction results.

What does $\widehat{D}$ measure? Nirenberg et al. (2001) argue that their average KL divergence measures the number of additional yes/no questions that must be asked to determine the stimulus when a decoder uses the dictionary p_shuffle(s|r₁, r₂) instead of p(s|r₁, r₂). They identify this number of yes/no questions with a loss in mutual information about the stimulus. However, this identification is mistaken. The KL divergence is not equivalent to entropy or an entropy difference (Cover and Thomas, 1991). Any information theoretic quantity that has units of bits can, intuitively, be thought of as representing the number of yes/no questions needed to specify its random variable. However, this does not imply that all such quantities are equivalent. For instance, both the entropy and the mutual information have units of bits. However, they are conceptually different: a neuron firing randomly at a high rate has lots of entropy but no information, whereas a neuron firing at a low rate, but locked precisely to a stimulus, has less entropy but more information.

The precise information theoretic interpretation of the KL divergence comes in the context of coding theory (Cover and Thomas, 1991). If signals x are chosen from a probability distribution p(x), then there exists a way of representing these signals in binary form such that the average “code word” has length equal to the entropy of the distribution. Each binary digit of this code corresponds to a yes/no question that must be asked about the value of x, and, hence, the code length can be thought of as representing the total number of yes/no questions that must be answered, on average, to determine the value of x. Achieving this optimal code requires a strategy matched to the distribution p(x) itself; in particular, the code length for each value x should be chosen to be -log₂p(x). The KL divergence between two distributions, p(x) and q(x) and D_KL[p(x)∥q(x)], measures the average extra length of code words for signals x drawn from p(x) using a code that was optimized for q(x). It is not an information loss in any sense. Instead, one might think of D_KL as measuring a form of coding inefficiency. In the present context, however, this loss of coding efficiency does not refer to the code of the neuron but, rather, to a nonoptimal code that would be constructed by a hypothetical observer for the conditional stimulus distribution p(s|r₁, r₂).

The KL divergence is commonly used in the literature simply as a measure that quantifies the difference between two probability distributions, without reference to its precise information theoretic interpretations. In this sense, $\widehat{D}$ is a sensible measure of the (dis)similarity between p_shuffle(s|r₁, r₂) and p(s|r₁, r₂), but it does not assess how much information about the stimulus can be obtained by using one distribution or the other. Moreover, as a general measure of the dissimilarity of probability distributions, the KL divergence is one of several common choices. Other sensible measures include the L₂ norm and the Jensen-Shannon divergence. Each of these measures is the answer to a specific question about the dissimilarity of two distributions.

Because $\widehat{D}$ is a KL divergence between approximate and real decoding dictionaries and because it cannot be interpreted as a loss of encoded information, this quantity should be thought of as a measure related to the problem of decoding and not to the problem of encoding. One important consequence of this distinction is that one cannot reach very general conclusions using any decoding-related measure. As noted above, there are many possible decoding algorithms, and the success of any algorithm is dependent on the choice of an error measure. Thus, the conclusions one reaches about the problem of decoding must always be specific to a given decoding algorithm and a particular error measure.

In the case of $\widehat{D}$, one is implicitly assuming that the decoding dictionary is represented by a code book that is optimized for p_shuffle(s|r₁, r₂). This is not the only possible code book that ignores noise correlations. Another possibility is to use one optimized for p(s), which completely ignores the neural response altogether. This counter-intuitive choice does explicitly ignore the noise correlations and, in some circumstances (e.g., the example in Fig. 4), it actually is more efficient than the one optimized for p_shuffle(s|r₁, r₂).

Another source of confusion is that $\widehat{D}$ is expressed in units of bits, rather than reconstruction error. This is highly misleading, because the encoded information is also expressed in bits. Although the encoded information provides a completely general bound concerning the performance of any possible decoder, it is important to keep in mind that $\widehat{D}$ does not have this level of generality, despite its suggestive units.

What does it mean to ignore noise correlations? The most obvious sense in which one can ignore noise correlations is to combine spike trains from two different stimulus trials. As described above, shuffling the spike trains changes the joint response distribution p(r₁, r₂|s) into p_shuffle(r₁, r₂|s) (Eq. 12) and consequently changes the probability of finding any joint response to p_shuffle(r₁, r₂) (see Eq. 21). Finally, the information that the shuffled spike trains encode about the stimulus is I_shuffle(S; R₁, R₂) (Eq. 13). However, the measure $\widehat{D}$ refers to a different circumstance: it assumes a decoding dictionary that ignores noise correlations but combines this with the real, correlated spike trains. For some purposes, this may be an interesting scenario. If $\widehat{D}$ does not assess the impact of this assumption on the information encoded about the stimulus, then what is the answer to this question?

In general, this question is ill-defined. The obvious approach is to construct the new joint probability distribution q(s, r₁, r₂) = p_shuffle(s|r₁, r₂) p(r₁, r₂) which combines a decoding dictionary that ignores noise correlations with the real, correlated spike trains. Then, the mutual information between stimulus and responses under the joint distribution q is given by:

23

where and . However, this scenario is strange, because simultaneously assuming p_shuffle(s|r₁, r₂) and p(r₁, r₂) implies (through Bayes' rule) that the distribution over the stimuli q(s) is different from the original p(s).

It is also worth noting that this formalism can be extended to the case of assuming any approximate decoding dictionary, $\tilde{p}$(s|r₁, r₂), by again forming the joint distribution, $\tilde{q}$(s, r₁, r₂) = $\tilde{p}$(s|r₁, r₂) p(r₁, r₂). Similarly, a different distribution over the joint responses, $\tilde{p}$(r₁, r₂), can be inserted. However, the distribution over stimuli $\tilde{p}$(s) will, in general, not be equal to the actual distribution, p(s). This can lead to contradictory results; for instance, the apparent mutual information can exceed the original stimulus entropy, I_q > H(S), because the new distribution over stimuli $\tilde{p}$(s) might have larger entropy than p(s).

Nirenberg and Latham (2003) discuss a special case of comparing two neural codes, in which one code is a reduced code or subset of the first (Nirenberg and Latham, 2003). One example of a reduced neural code would be a code that counts spikes in a large time window versus one that keeps many details of spike timing by constructing “words” using spike counts in a smaller time bin (Strong et al., 1998). In this case, the joint response of the reduced code, r′, can always be found from the joint response of the full code, r, by a deterministic function, r′= F[r]. Because R′ is a reduced code, it always conveys less information about the stimulus than the full code: I(S; R′) ≤ I(S; R). This difference in information can be rewritten as an average of KL divergences in a form that is suggestive of the measure $\widehat{D}$. However, it is important to keep in mind that a neural code that ignores noise correlations by combining spike trains from shuffle trials is not a reduced version of the real neural code. The shuffled responses, R_shuffle, include the entire set of responses found in the simultaneous responses, R, but they occur with different probabilities. There is no deterministic function that can act on r on every trial to produce r_shuffle. Therefore, the information lost by constructing a reduced neural code is not directly relevant to the case of ignoring noise correlations.

The subtlety of what it means to “ignore correlations” can be seen in a simple example. Suppose that we observe a set of signals {y₁, y₂,... y_N}, all of which are linearly related to some interesting signal x. There are many simple situations in which our best estimate of x (e.g., the estimate that makes the smallest mean square error) is just a linear combination or weighted sum of the y_i, that is . Such a “decoder” obviously does not detect correlations among the y_i in any explicit way, because there is no term approximate to y_iy_j that would be analogous to detecting synchronous spikes form different neurons. In contrast, the optimal values of the weights w_i depend in detail on the signal and noise correlations among the y_i (as is relevant for the linear decoding of spike trains discussed below). Is this implicit dependence sufficient to say that the linear decoder makes use of correlations? Or does it ignore correlations because it does not explicitly detect them? Even if we can resolve these ambiguities in simple linear models, would our definitions of what it means to ignore correlations be sufficiently general that they could be applied to arbitrary neural responses? These difficulties simply do not arise in the discussion of synergy and redundancy from an information theoretic point of view.