Outputs of Generative Diffusion Models are Often Unattributable
Zheng Dai1, David K Gifford1
1Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology
Contact: *@mit.edu (replace "*" with "zhengdai")
Abstract
Background
Modern generative diffusion models work by replicating the statistical patterns of large training datasets. Thus, developing a method that is able to attribute generated outputs to influential training data would greatly advance our understanding of and ability to regulate these models. This has led to much work towards this goal.
Contributions
In this work we investigate whether attribution is actually possible. We find that models trained with enough data often generate samples that are unattributable. We establish this through a large-scale analysis of what-if scenarios, revealing that we can often omit any sample or creator from the training data without affecting a generated sample. Our study focuses on diffusion models, which has become the dominant model for generating audiovisual media, and is also prevalent in many scientific applications including protein structure modeling and therapeutic discovery. Central to our analysis is a model ablation methodology that allows efficient removal of training examples from a trained model without the need to retrain.
Methods
Counterfactual Analysis
To discover what the effects of a unit*We use the term unit to generically denote a part of the training data, generalizing the notion of a single image to a collection of one or more images. of training data was on a generated sample, we consider the counterfactual question: "what would have been generated if the unit of data in question was missing?". This can be computed by going back to the training set, removing the unit of data in question, and training a new model on incomplete dataset. We can regenerate the sample using the newly trained model to produce a counterfactual sample, a sample that would have been generated if the model had been trained on the incomplete dataset. The counterfactual can then be compared with the factual sample generated by the original model trained on the full dataset to assess the effect of that unit of training data. We describe this type of analysis as counterfactual analysis.
Figure 1. A cartoon illustrating of the causal relationship between the training data and the final generated sample. The training data (left) is used to train a model (middle), which is then used to generate a sample (right). Try hovering over the training set to omit samples from the training data. Omitting samples leads to changes in the final generated sample. The changes in the final generated sample can be used to infer what the model learned from the training data. Note how the generated sample changes significantly when the training sample in the center is omitted, while omission of other samples only induce subtle changes, indicating that the training sample in the center has the strongest causal influence on the generated sample.
Performing counterfactual analysis by retraining models on incomplete datasets is extremely computationally expensive. Exploring an entire counterfactual universe (i.e. all possible counterfactual samples that can be produced by leaving out one unit of training data), which we need to do in order to fully understand the contributions of all parts of the training dataset, is effectively impossible.
Computing Counterfactual Samples Efficiently via Ablation
Instead of training a single model on the entire training dataset, we train sets of model parameters independently on different subsets of the training dataset. These independently trained parameters are then combined into a single model. If we then use the overall model to generate a sample, we can remove the causal influence of any given unit of training data by removing all parameters of the model that were trained on it. Since this ablated model has effectively never seen the training data that we removed, we can use it to generate counterfactual samples for counterfactual analysis.
Figure 2. Using ablation to compute counterfactuals with a digit generator trained on 384 digits (only 10 training digits are shown on the left, click here to show all 384). Hover over the training set on the left to highlight in red the flow of the causal influence from the training data (left) to the independently trained model parameters (small circles) to the combined model (large circle) to the generated sample (right). Click a member of the training set on the left to remove it via albation, and click it again to unablate. Note how the flow of causal influence is broken, and how it is not broken for any other member of the training dataset. Click here to see another example.
Ensembles of Diffusion Models are Viable and Ablatable Image Generators
Ablation requires bespoke architecture, for which we developed the diffusion ensemble, a model architecture that can withstand ablation and operates similarly to standard diffusion models with similar performance.Diffusion ensembles are formed by aggregating the outputs of independently trained diffusion models via arithmetic averaging. A diffusion model can be viewed as a predictor that outputs a prediction of the denoised sample upon receiving a noisy sample, so the arithmetic average of multiple predictions within an ensemble can be treated as an aggregate prediction. These predictions are then used to generate a sampling trajectory in the same way they are generated for normal diffusion models, which then produces the final generated sample.
Figure 3. Examples of images that were sampled by diffusion ensembles. Each image was generated by a different ensemble. Click here to view more images.
Results
Attributability Decays with Training Dataset Size
If the counterfactual universe consists of samples that are all similar to the generated sample, then we must conclude that no training sample contributed significantly to the generation of that sample. We operationalize this notion of attribution decay by defining the counterfactual radius: the largest possible distance between a factual sample and a counterfactual sample. We can alternatively define the counterfactual radius as the radius of the smallest ball centered at the factual sample that contains the entire counterfactual universe. A low counterfactual radius corresponds with low attributability.
We compute the counterfactual radius of generated samples for 24 different models that were trained on varying datasets, and find that there is a clear negative relationship between the training set size and the counterfactual radius.
Figure 4. The counterfactual radius is negatively correlated with the training data size. Given a model, we plot the training set size on the x-axis. We plot the arithmetic mean of the counterfactual radii of the samples generated by the model on the y-axis. Hover over a point to expand it into a box and strip plot of the individual samples*Note that variations in x coordinates in strip plots are meaningless and to see the size of the samples' counterfactual universe and the dataset that was used to train it. The size of the counterfactual universe does not always match the training set size, since for some datasets we remove multiple samples at once (for example, for CelebA datasets we remove all samples that correspond to a single individual at once). Counterfactual radii are measured in Euclidean distance, and all images were scaled to 256-by-256 and 3 channels to ensure consistency between datasets.
Samples can be Unattributable
It is possible for the counterfactual radius to be zero. This can occur when we sample from discrete spaces, such as handwritten digits where each pixel value is either black or white. We refer to a sample with a zero counterfactual radius as unattributable, since for every training sample, we can produce a counterfactual sample that certifies that the outcome of the sampling process would be entirely unchanged if that training sample had not existed.
Figure 5. Examples of unattributable samples generated from a model that generates binary images of handwritten digits.
Citation
"Dai, Z. & Gifford, D. Outputs of Generative Diffusion Models are Often Unattributable. Nature Communications 17, 6974 (2026)."