Our piece, What Is ‘Life Cycle Analysis’? dealt with a topic that is all the rage, having studies done to scientifically determine the burden of a product or process on the environment. We had doubts about the reliability of all such studies, and those doubts brought us this stern letter:
The LCA of canning vs. freezing green beans that you referred to was sub-contracted through the Network for Business Innovation and Sustainability. The scientific integrity of Rita Schenck, PhD, Executive Director of IERE, who conducted the study, is respected worldwide. It was also peer reviewed by independent scientists.
The study is not, as you framed your review, presented with a “veneer of science;” rather it was conducted meticulously with all assumptions clearly stated. Perhaps because of your desire to create a zesty article and because of a superficial understanding of LCA, you have misconstrued statements.
For example, the study located the hypothetical bean-freezing plant in the same spot, as the canning plant; this was a methodological procedure for setting a level playing field, a way of setting the conditions for “other things being equal’ (such as, grid source of electricity), this is the way that processing impacts of canning vs. freezing could be compared. But you missed the point and said, “what if the frozen bean factory was located somewhere else?” Also, you seem to lack the knowledge that this study’s use of LCA indicators for pointing to broader generalizations is grounded in very extensive research; (e.g., whether or not a particular kind of emission is carcinogenic or not).
When making generalizations, the study took into account the research-grounded robustness of indicators utilized and used rigorous statistical methods to indicate which differences were likely to be significantly significant. In this time of urgency for addressing climate change it is just one small piece of the pie but nonetheless important to note that, “The greenhouses gases from the lifecycle of frozen green beans were 39 percent greater than for canned beans.
The model for climate change and the inventory information for climate change are quite robust and have relatively low variability and so this difference is probably statistically significant.”
It’s always fun to play with “what if’s” but it’s ridiculous to use such mind play about ice cream to discount a very carefully conducted scientific study regarding the environmental impacts of our manufacturing processes.
Mr. Prevor, to balance your presentation, may I suggest that you please publish these comments; or, submit a retraction that shows a more balanced perspective of your own. If you would like to deepen your understanding of LCA, let me know and I will forward information regarding a planned workshop series on the topic that will be forthcoming soon.
— Karl Ostrom, PhD, Co-Director
Network for Business Innovation and
Sustainability (NBIS)
There is always room at the Pundit for differences of opinion, so we very much welcome Dr. Ostrom’s letter. In private correspondence, we assured him of our desire to learn and requested the information he references on upcoming seminars on the topic of Life Cycle Analysis (LCA). When we receive it, we will not only try to attend but will publicize the schedule for any Pundit readers who are interested.
Although we are happy to publish his letter, Dr. Ostrom will have to be significantly more persuasive before he gets that retraction he is looking for.
Although there is no question that Dr. Schenk and her organization, The Institute for Environmental Research and Education (IERE), are well respected among those who are interested in such things, and we have little doubt that Dr. Schenk is meticulous in her calculations. It is inherent in the nature of what was attempted in this case that such a study is bound to confuse with false accuracy.
This saddens us because we are actually exceedingly sympathetic to the goals of IERE. The analysis on its web site strikes us as spot on:
“…Unfortunately, environmental decisions are rarely based on the facts of the matter, but rely simply on value judgments. That means we are ignoring information about potential environmental damage, and it means that our decisions are just as likely to cause damage to the environment as to benefit it, even where environmental protection was the goal of the decision. Many millions of dollars have been spent to protect the environment, while simply causing the environmental damage to another environmental system.
In a world where resources are limited, and where decisions not based on facts are likely to cause further environmental degradation, better access to facts is an urgent need. IERE was formed to address this need.
Among the factors that lead to decision making without concerns for the facts are:
- The facts often don’t exist, because no one has spent the time and money to develop the relevant data
- Often, the individuals making the decisions are not aware of the relevant facts, or do not have the skills needed to make use of these facts.
- Some individuals prefer to make decisions based on their values rather than facts.
While little can be done to change decision making approaches in individuals, significant efforts can be made to provide facts and skills to decision-makers. IERE was formed to do just that…”
Providing facts and skills to decision-makers is close to the Pundit’s heart, and so we applaud the effort.
We did link to the complete study, so anyone interested could study it for themselves and we do so again here.
Yet the number of variables, their meaning and import are so vast and so frequently changing that it is difficult to see how one of these studies could provide a good basis for decision-making.
Dr. Ostrom has five specific objections to our response to the study:
-
“The study is not, as you framed your review, presented with a “veneer of science;” rather it was conducted meticulously with all assumptions clearly stated.”
We do not disagree that the study is meticulous nor claim that the assumptions underlying the study were not clearly stated. It is simply that stating the assumptions and being meticulous does not necessarily produce a correct outcome.
Since the invention of the spreadsheet, anyone looking to buy a company gets meticulous projections of sales and profits for the next two decades with a lot of assumptions clearly identified. That doesn’t mean these numbers projected for sales or profit in the year 2027 are accurate.
Equally this study can be meticulous, but if the assumptions are incorrect, its results will not occur. And the very meticulousness of the numbers — the results purport to demonstrate the effect of canning vs. freezing green beans in terms of ozone depletion out to the fifth decimal place — should make us wary of false accuracy.
It is precisely this meticulousness that we refer to when we point to a veneer of science. Just as those spreadsheet projections don’t guarantee accuracy, neither do projections to the fifth decimal place on ozone depletion guarantee any specific impact on global warming.
-
“For example, the study located the hypothetical bean-freezing plant in the same spot, as the canning plant; this was a methodological procedure for setting a level playing field, a way of setting the conditions for “other things being equal’ (such as, grid source of electricity), this is the way that processing impacts of canning vs. freezing could be compared. But you missed the point and said, “what if the frozen bean factory was located somewhere else?”
Exactly correct. This was a “methodological procedure” to demonstrate what would happen “other things being equal” — but other things are rarely equal. And by assuming conditions that create an equivalency which may not exist in the real world, one can draw a deceptive picture of reality.
This strikes at the purpose of this study. If someone owned a piece of land and was considering a plant on that piece of land and so commissioned a study of the environmental impact of building a canning vs. a freezing plant on that specific piece of land, well, more power to him. If he wants to use his land so as to minimize his impact on the environment, bravo for him.
This study, though, has a different agenda. This study clearly has marketing implications, and the reason it was publicized to begin with is because its sponsor wants to persuade consumers to switch to canned from frozen for environmental reasons and to persuade trade buyers that they should start to promote canned and reduce the size of frozen food aisles for environmental reasons.
A study that assumes that frozen food facilities will be built exactly where canning facilities are built cannot be used in this way.
In other words, if you want to trumpet a study as having implications for people’s behavior, you need to study each side of the comparison as it actually is or as it optimally could be — you can’t “norm” to one of the variable methods being studied.
-
“Also, you seem to lack the knowledge that this study’s use of LCA indicators for pointing to broader generalizations is grounded in very extensive research; (e.g., whether or not a particular kind of emission is carcinogenic or not).”
We will have to let our readers judge if we lack knowledge or not. What we note is that while accusing us of lacking knowledge, the letter doesn’t actually mention any specific research to justify the relationship between the indicators the study uses and broad generalizations.
Our point was, and remains, that the study, including footnotes, does not persuade on these points. Since the study is published on websites open to laymen and distributed to the media, and is not an article in a professional journal, it is a reasonable expectation that if the authors are able to defend their findings, they will do so in the study.
Presumably this report is trying to persuade someone of something. To claim that there is some “very extensive research” that proves something but it is not in the report is to make that persuasion impossible except by people willing to simply accept assertions as truth.
Yes, we need a lot more evidence before we buy into a conclusion that canning green beans represents a 150% improvement on “Human Health Cancer” over freezing green beans.
-
“In this time of urgency for addressing climate change, it is just one small piece of the pie but nonetheless important to note that, “The greenhouses gases from the lifecycle of frozen green beans were 39 percent greater than for canned beans.
The model for climate change and the inventory information for climate change are quite robust and have relatively low variability and so this difference is probably statistically significant.”
Is this a good time to mention that we are not actually certain of the urgency of addressing climate change? Even assuming global warming to be true, bad and caused by humans, we would still have to prove that we can do something about it before we necessarily felt much urgency in this area.
As we mentioned, we are not sure that in a “fair fight,” frozen would calculate at 39% greater than canned. This number may have been distorted by various assumptions — including, as we mentioned above, plant location.
Then to leap from that number to the impact on climate change is quite a leap. First it is based on a model — there are many models that all come out with wildly different results. Why should we believe that the one model used by people who do green bean studies is the accurate one?
Second, what percentage of greenhouse gasses out there could possibly come from one green bean plant, much less from the difference in emissions between a canning and freezing plant? Surely it is such a tiny, insignificant number it has to get caught up in the margin of error of any study.
-
“It’s always fun to play with “what if’s” but it’s ridiculous to use such mind play about ice cream to discount a very carefully conducted scientific study regarding the environmental impacts of our manufacturing processes.”
Dr. Ostrom is referring to our half tongue-in-cheek ending, which went as such:
We could go on but let us end with this: If you want to judge the impact on human health of the two processes, don’t we have to consider what people will actually eat? Presumably people who buy frozen green beans prefer them to canned.
If, due to environmental guilt, people now buy green beans they like less, perhaps they will eat them less and eat more ice cream to compensate for their sadness at not enjoying their vegetables. The ice cream will make them fat and add weight to each vehicle they ride, which will increase carbon emissions. Add in the additional carbon footprint of manufacturing, transporting and storing all that ice cream and we may wind up hurting the environment more than if we didn’t rely on such studies.
Sound far-fetched? No more so than these life-cycle studies.
Yes, we were making a joke — but only half a joke. This study was not a comparison between two ways of canning green beans that resulted, in the end, in identical products.
This is a comparison between two treatments that result in products with different taste and texture profiles.
Now if one plant or another happens to shift from canned to frozen or frozen to canned but demand does not change, then, to use Dr. Ostrom’s phrase, other things being equal, there will be no impact on the environment or human health. In total, the exact same number of frozen green beans will be produced and the exact same number of canned green beans will be produced. Just the factories they are made in will be reshuffled.
The only way there can be any change — positive or negative — related to the subject of this study is if demand shifts from frozen to canned or vice a versa, or if total demand falls or increases.
The point — and it is an important one — of our little joke is that you cannot make an informed judgment about the environmental and health impact of two different products without also making judgments about patterns of consumption and substitution.
Let us assume that based on this study, Congress passed a law banning frozen green beans. It seems reasonable to think that people who buy frozen rather than canned have a reason for preferring it. So, perhaps, everyone will not switch to canned. Some people will buy more fresh green beans; others will more frequently order them in restaurants where they have fresh rather than canned and, in many cases, consumers will purchase other frozen vegetables or if they are banned also, just eat fewer vegetables. Perhaps they will eat more of other products. Dr. Ostrom didn’t like our ice cream example, fine, make it beef, Peking duck, whatever he wants.
The key is that this study is, in effect, being used to urge a particular course of action or policy, and we can’t judge the effect of that behavior or policy without knowing the change of consumption likely to ensue.
It reminds us very much of those who advocate only organic produce to increase public health, without any thought to the health impact that may come about if produce consumption goes down because the organic product is priced higher.
**
The Pundit is hardly the first one to point out that Life Cycle Assessment or Analysis depends on hundreds of variables:
“Personally, I believe it will never be possible to solve controversial discussions about products with an LCIA [life cycle inventory assessment] method that is based solely on mathematical relations between interventions and protection areas. There are simply too many uncertainties, there is too much ignorance, and they can only be overcome by all kinds of subjective, subtle, and basically value-laden choices. …
Arnold Tucker, Ph.D.
Life Cycle Impact Assessment — Some Remarks.
Life Cycle Impact Assessment of SETAC-Europe (Second Working Group — WIA-2).
In this case, we are confronted with an LCA created for marketing purposes and that purpose makes what would be an interesting intellectual exercise into a program that should be held to a higher standard. That was the purpose of our piece.
We do appreciate Dr. Ostrom’s taking the time to write us and share his understanding of the situation.
