Over the last four years, California Energy Commission has rolled out its HERSII rating system, which was designed as a way to provide a score to homeowners and prospective buyers that compares the energy performance of a house to similar homes (without occupant behavior). The HERSII system is based on the same engine that drives California Title 24 Energy Code and was identified as a key part of the State’s approach to driving energy efficiency in existing buildings. 

The goal of the HERSII system is to label all California homes in order to equate energy performance with building value at time of sale, and as a way for a homeowner to get third-party recommendations as to the most cost effective options to save energy.

Like puppies and apple pie... or is it?

HERSII sounds great and asset ratings and third-party raters make for really nice policy white papers, but when put into practice on existing buildings, the results speak for themselves ― and don't necessarily agree with the theory. The HERSII ratings system is extremely costly, has low consumer demand, and does not result in significant conversion to energy efficiency projects.

In an attempt to develop a one-size-fits-all solution to provide an asset rating for use at time of sale and in the appraisal process and provide an actionable workscope and savings projection all in one system, we have instead created a solution that does not work well for either use.  

So the big question is, will the HERSII strategy work? 

The good news is that we now have results back from our initial tests of the system. However, the unfortunate reality is that the test phase has demonstrated serious shortcomings in the current theory, which should at a minimum call the HERSII system into question and hopefully lead to thoughtful evaluation of the shortcomings that emerge to avoid costly missteps for the marketplace and government programs.

Based on results from HERSII tests over the last four years, the HERSII rating system does not appear to be catching on with homeowners, nor does it seems to be driving market transformation in terms of conversion to energy savings or discernible asset value improvements (though the latter is harder to know with the current dataset). Admittedly, some of our data is a little sparse, but that is a function of how hard it is to get real numbers out of these pilots. 

What can we learn from the Sonoma County HERSII Pilot?

Sonoma County ran the largest HERSII pilot in the state, paying a subsidy of $700 per HERSII rating, which resulted in 100 percent free audits and ratings to customers, and unsustainably high margins for raters. 

In Sonoma, we see a clear jump in the number of HERSII ratings while this lucrative incentive was on the table. However, shortly after the massive subsidy went away, the number of ratings drop immediately back to virtually the same number of ratings as were occurring before the incentive program. It would appear that even after this very expensive attempt to seed the market, there is little to no actual consumer demand in the market for ratings and one is left to wonder what happened to all the HERSII raters who got trained in Sonoma for this brief pilot, now that there is no more work.

Clearly, California does not have the funds to pay $700 or more per rating out of public funds on every home in California, so the fact that Sonoma’s 100 percent rebate pilot seemingly did nothing to jumpstart actual consumer demand is very concerning, and means an attempt to roll this system out would force California homeowners to shell out for an expensive service they are not valuing.

The fact that the Sonoma pilot program did not result in any discernable market transformation, either in terms of sustainable businesses conducting ratings or demand for ratings by homeowners, should be of serious concern.

Do homeowners value HERSII ratings?

When we look at the adoption curve of HERSII ratings during the recovery act period, we find that ratings are highly correlated to rating incentive programs and appear to have little organic consumer demand.  

In fact, as is apparent in the chart on the right, it appears that these incentive programs had little lasting impact on demand in the marketplace.

When you couple this with the emerging facts related to the propensity of the HERSII system to overpredict potential savings by as much as a factor of 3x  (see:  Ex Ante Tool Review Findings Disposition for Energy Upgrade California Custom (‘Advanced’) Measure Savings, 1 Mar 2013), one realizes that not only is this system not working in terms of driving customers to take action, but it is failing to do so while massively inflating the level of savings and ROI being sold to the customer.  

The propensity of the HERSII system to project significantly higher savings than what is delivered means that if HERSII in its present form was applied as a mandate, there would be many millions of California homeowners who would see only a small fraction of the savings they were told to expect.  

This fundamental issue is based on the CEC’s attempt to create a system that is based on code, and then apply it to operational predictions of savings.  These two uses are nearly opposite in terms of how a model is constructed, and speaks to the need for systems designed for the specific purpose being asked - one size fits all is not cutting it (learn more about this issue here).

The CEC may have the legal authority to regulate ratings in California, and AB758 may give the Commission the legal basis to attempt to implement this approach, however it is highly unlikely that they have the political capital to make it stick once we start subjecting real people to these outcomes.

Asset ratings may in fact have a place, but they are really not compatible with delivering actionable workscopes in the real world, and the one size fits all approach has resulted in a system that really fits nobody very well.  HERSII is an expensive and complex system to deliver ratings that have significant accuracy issues, don't work for industry, and consumers don't seem to value.

Is HERSII worth the cost?

Each HERSII rating costs at least $500 and often more, based on the fact that conducting a rating consumes half a day in the field and then more hours back in the office inputting data into the California Energy Commission (CEC) mandated software tool. There is little room for substantial economies of scale.

When you consider that we have upward of ten million homes in California, the cost of having a HERSII rating done for all California homes will require California homeowners to invest $5 billion dollars in ratings - and that is just for ratings, not actual energy efficiency.

Do HERSII Ratings turn into energy efficiency projects?

California home performance contractors have a very hard time converting HERSII third-party audits into viable leads that result in customers who are ready to make energy efficiency upgrades. 

HERSII Ratings conducted in Sonoma were cross-referenced with Energy Upgrade California™ (EUC) projects and it was found that conversion from rating to energy efficiency retrofit was under 10 percent (this analysis was conducted prior to 100 percent completion of the program, and until it is redone more accurately should be considered a clear directional indicator). This is especially surprising given that in the Sonoma program, unlike elsewhere in the state, home performance contractors were allowed to provide the rebate to customers for their diagnostic test-in as long as the project included a HERSII rating upon completion. Given that contractor close rates are typically 30 percent or greater, their participation in the Sonoma pilot would have been expected to inflate the actual conversion rates when compared to HERSII on the open market by third-party raters.

At a 10 percent conversion rate, a $700 rating incentive translates into $7,000 of public funds in rating incentive per closed retrofit in addition to Utility incentives, and other Energy Upgrade California program spending.  Clearly this was not a cost effective way to drive demand for retrofitting.

This trend is borne out in PG&E figures for its entire territory, which shows that of the 582 homeowners who obtained HERSII ratings, only 59 projects have been completed energy upgrades through Energy Upgrade California, which pencils out to a similar 10% conversion rate.  This of course compared to conversion rates of audits to retrofits that are typical of integrated home performance contractors of over 30%.

Why is it so hard to get homeowners to act on their ratings?

The HERSII process is convoluted, complex, and expensive.  From a homeowner perspective, it goes something link this:

First, a third-party rater does a HERSII rating for a homeowner who shells out something like $500 for the audit and report. If that customer is interested in doing work, they bring the rating report to a home performance contractor to get the upgrades implemented. However, here is where it starts to gets tricky and the real world collides with theory.

Contractors are ultimately responsible for a project’s outcome, including its energy performance, and cost. When presented with a third-party HERSII rating, a contractor must determine if the report recommendations match the conditions he/she finds in the home. This means the contractor always must go back to the customer and do what amounts to another audit of the building to confirm recomendations, code compliance, and pricing, but — because the customer has already paid for a rating audit — the contractor has to conduct this work for free.

On a very large percentage of these third-party HERSII ratings, the contractor will end up not agreeing with either the estimated price coded into the rating (which drives the Return on Investment and ranking in the software), or they will disagree with the solution being recommended once someone with construction experience visits the site and creates an actionable workscope.

This puts the homeowner in a bind. On one hand, they have the third-party rater sporting a CEC logo on their report telling them one thing, and a contractor telling them another thing, and far too often the homeowner becomes uncertain as to how to proceed and who to trust, so no work gets done. The third-party model ends up with very low conversion rates, and is generally not a profitable source of customers for home performance contractors.

How do we move forward from here?

It is time that we focus our energies in California on moving to a market that relies on actual performance, not complex regulatory structures translated into software and a numerical scales. The web of regulation we have created has many unintended consequences, and with the advent of smart meters and big data, we have the opportunity to move beyond regulatory proxies and instead harness private markets and sources of capital.  We should embrace this change as a great success of the California system, not a failure.  It is time for State policy to advance and lead the nation towards a smarter more efficient model to deliver the deep energy efficiency required if we are going to hit our climate, energy, and economic goals.

In light of the results from HERSII to date, the California Energy Commission strongly considers adopting simpler approaches to providing homeowners with ratings, that are substantially lower in cost to homeowners and do not pretend to be of investment grade accuracy. One such solution is the national DOE Home Energy Score.  A very simple model can be input through an API from a variety of software tools, and is simple enough to be part of a home inspection, resulting in a 1 to 10 score for the homeowner.

These simpler systems work as a  way for homeowners to gauge a home's performance but will not be confused with real energy audits capable of delivering solutions to homeowner that contractors can actually build and that will deliver reliable results. We should separate upfront simple asset ratings from the more complex needs of a diverse contractor marketplace who can deliver the level of quality auditing necessary to develop an actionable workscope and a real price. Instead of applying an expensive and inaccurate comprehensive HERSII rating on every house in the State, we instead would only do an investment grade audit when there is pathway to getting a project built.

We need to do more than tweak the knobs on the current model.  We need to be open to new ideas and substantial change in our approach.  Solutions are out there, but we need to be willing to admit that many current ideas are not working, and minor changes will not be sufficient to reverse the trend.  It is time for California to step back into the lead and help the country move towards a sustainable market for energy efficiency.

This process should start today by admitting that the HERSII system needs to be reevaluated based on evidence and feedback from the market.  It does not matter how much we have invested, we need to look at empirical results and change course.  We cannot afford to keep marching forward with the same basic theories, while hoping for different results.  

With many initiatives in full swing to put ratings on homes and buildings across the country, there may be a lesson to be learned from recent cases where consumers have prevailed in suing car companies for promising an MPG that is not based on real-world driving.

Before you read any farther, I want to make clear that these models are not the problem, it is actually how they are being applied.  Applying any predictive model to an individual building is the problem.  It is up to us to use these tools as a way to manage this risk in pools, and avoid driving that risk down to homeowners.  Great examples of this can be seen in the success of solar PPA and Leasing models.  

Most labels, including the National RESNET HERS and California HERSII labeling system and the DOE Home Energy Score, are in fact asset scores that, similar to an MPG, score a house based on a set of average users.  These scores have potentially wide variance for any particular building, and have a tendency in many climate zones to over-predict savings.

A recently released study called Modeled vs. Actual Savings for Energy Upgrade California Retrofits, analyzes predicted savings based on the CEC’s required energy modeling software, against actual results from customer bills.  The study shows that California HERSII is over-predicting by and average of 50%, with a huge amount of variance between winner and losers.  Resulting in more than 78% of homeowners not achieving the savings being predicted.

A recent LBNL report on the Home Energy Score called "Accuracy of the Home Energy Saver Energy Calculation Methodology" was announced with an email headline declaring that HES is now within 1% accuracy on average, which is a great result.  Of course on closer examination it is clear that there is wide variance for any individual project - perhaps even wider than other tools in the study.  Which of course goes back to the original issue that homeowners are truly not interested in the average, especially if they get the bad end of the stick.  

This issue is not exclusive to the United States.  The Green Deal in the UK has also suffered from estimated savings that are outpacing reality.  In an April 11, 2012 article in The Telegraph, called “Green measures for homes 'save less on fuel bills than forecast”

"An official pilot study of 67 homes in Sutton, South London, found all of those who took the 25-year payback package rather than ten years - a third of the owners - faced repayments higher than the savings.

Another study of 139 council houses in Sunderland found savings on energy bills were just 12 per cent rather than the expected 19 per cent.

Luciana Berger, Labour's climate change spokesman, said the proposals were "complex, confusing and leave customers exposed to mis-selling".

"Before anyone takes out the Green Deal they have a right to know how much it will cost them and how much they will save. Relying on guesswork just isn't good enough. If people are promised savings which never arrive, they will think the Green Deal is a con."

Both the CA and UK rating systems are based on a faulty notion that relative scores are more important than accuracy.  In striving to achieve a relative indicator of performance, and removing behavior (how people actually use their homes) we are left with a system that is really more about policy and theory then what matters to real people - which generally boils down to, how much does it cost, and what will I save.

The idea that we are going to put MPG stickers on every home in CA, or the US, at great expense (It will cost $5B to label home in CA alone) is a mistake.  At least with a real MPG on a car it is based on actually testing the vehicle, and you are not having to test every single vehicle on the road, vs. labeling buildings where we are attempting to derive energy use from physics calculations and every house needs an expensive custom test.  

Here is an article on the topic of MPG from the Huffington Post, Hyundai Fuel Economy Lawsuit Filed Alleging Misleading Ads, that I think should have us all worried:

"For carmakers, the new trendy thing is to have a vehicle in the lineup that gets 40 mpg. One huge problem is that fuel efficiency figures are not based on real-world driving. And automakers opt to advertise with the fuel economy figures that are most impressive -- for highway driving -- rather than lower city or average mileage calculations, which would make their cars look less efficient.

But the Hyundai lawsuit is the second one in recent months to challenge automakers over lofty fuel economy claims. In February, California attorney Heather Peters sued Honda in small claims court over the fuel economy claims for her 2006 Honda Civic hybrid. She said she never got anything close to the 50 mpg she was promised. A judge awarded her $9,867 in the case, which Honda is appealing."

In the end our goal is to save energy and drive consumer adoption.  There is a distinct risk that all of our efforts may backfire when consumers come to understand how imperfect our ratings are, and when the financial community tries to underwrite investments based on energy savings that don’t really exist.

It is time that we start focusing on real data and actual savings, rather than more complicated regulatory schemes.  All this is not to say that energy efficiency does not work, instead it should tell us that we need to move from energy efficiency expressed through code and complicated ratings or scores, and instead focus on turning savings into a resource that can be valued and traded.

We are in a unique moment in time where we can move past regulatory frameworks, to engage markets that can finance our long-term goals, which are simply too expensive to achieve driven primarily with public dollars.  

This article is not to say that modeling is worthless or wrong.  In point of fact, it is far better at predicting savings than just simple one size fits all deemed savings, and can be very good predictors of a large pool of buildings.  However, we need to rapidly move from a model where we have policy anointed solutions based on long regulatory process and instead start measuring and valuing savings predictions based on actual results.  Once we have a system that can measure real savings versus predictions, markets can step in, invest, and manage savings risk so that building owners and households don't have to.

If you look at the Solar industry as a guide, where in CA residential Energy Service Contracts (Leases and PPAs) are currently 75% of the market, building owners on completely insulated from risk through performance guarantees, private capital is flowing, and quality has become a function of industry.  

A system built on real proven savings at the meter will drive innovation and investment, which is the path towards a real and sustainable solution that can achieve the promise of energy efficiency in the build environment.

Enough talk and theory... get the actuarial data and the market will follow!