[Editor's note: A version of this story appears in the May 2019 edition of Oil and Gas Investor. Subscribe to the magazine here.]

The fortunes of the oil and gas industry are inextricably linked to the price of crude. The sector outperformed the S&P 500 Index from December 2007 through mid-2014, tracking the commodity price rally. But the weakness in the oil price that began in mid-2014 and continued through 2017 was a major headwind, causing the sector to underperform the index during this period.

In our inaugural report on value creators in the oil and gas industry, “Value Creation in Oil and Gas 2018: Managing the Price Cycle,” we found that median total shareholder return (TSR) over the downturn was flat for the sector but 12% for the S&P 500 Index. Because of the weak crude price, the oil sector was also near the bottom of the 33 industries in the Boston Consulting Group (BCG) 2018 Value Creators Rankings.

But on closer inspection, a more nuanced picture emerged. We found that downstream players performed well, benefiting from stronger crack spreads driven by refined product demand. However, companies with large upstream operations or those focused on upstream, including the majors and E&P companies, were hurt by the “lower-for-longer” price environment.

When oil prices rise, E&Ps’ exposure to the cost of crude means they offer greater potential upside than other peer groups. Indeed, E&P stocks rallied in the second and third quarters of 2018, though they lagged the rise in the price of crude. But as crude dipped again in the fourth quarter, the peer group fell even more sharply. The quarterly price drop also erased share price gains that the oil sector made earlier in 2018, leaving its performance flat for the year.

Current oil market dynamics point to an extended period of price uncertainty. E&P companies face challenging times ahead, with recent share price declines suggesting the process of regaining investors’ trust may be a lengthy one.

RELATED VIDEO: What's Affecting Oil Prices This Week?

For the report, we selected 64 organizations spanning five peer groups along the value chain: the majors, refining and marketing, E&P, national oil companies (NOCs) and integrated oil companies smaller than the majors. Companies had to be valued at more than $8 billion, have a free float of at least 20% and had to have existed before 2014.

BCG ranks companies using TSR as its value creation metric because it reflects the true bottom line for a company’s shareholders. It measures the return from a stock investment, assuming all dividends are reinvested in the stock.

Our approach deconstructs TSR into a number of underlying drivers, using a combination of revenue growth and margin change to assess changes in fundamental value. We then factored in the change in the valuation multiple arising from investor expectations. These two factors determine the capital gain (or loss) for investors. Finally, dividends, share repurchases and debt repayments were tracked to determine the contribution of free-cash-flow distributions on TSR.

Our primary focus was the recent crude price downturn from mid-2014 through 2017. But we also examined TSR performance over the 10 years from December 2007 through December 2017, marking a full oil price cycle, to see how TSR drivers changed. Using BCG analytical tools, we discovered that the relative importance of these drivers differed significantly by peer group.

E&Ps: Pure play vs. diversified

Without downstream operations limiting the impact of the crude price downturn, North American E&P companies were exposed to the full effects of the drop. They were hit by a combination of revenue and margin contraction from mid-2014 through 2017, resulting in negative TSR for the peer group. E&P was the oil sector’s best TSR performer for the period from December 2007 through mid-2014, led by pure-play companies. But it has been one of the sector’s laggards since the downturn.

We included in our analysis both small pure-play E&P companies and larger diversified E&Ps. Pure plays are typically focused on one geographic area and a single type of upstream asset. Diversified E&Ps are larger and operate across multiple regions and upstream asset types. Spurred by the shale boom, however, both pure-play and diversified companies are heavily invested in the shales—unconventional oil and gas reserves centered largely in the U.S.—to boost returns.

Our study found a clear distinction between the performances of pure-play E&Ps and diversified companies. As a whole, the two groups delivered a negative median TSR during the downturn. But some pure-play E&P companies fared significantly better, with a handful producing positive TSR in double digits. These small players, which typically concentrate on shale oil assets, were able to increase sales by driving an uplift in production volumes. As a result, they ranked as TSR leaders both within their peer group and among the 64 companies in our sample.

Diversified results

By contrast, diversified E&P companies, despite having an ostensibly more resilient business model, were too large to compete solely by increasing revenues through a rapid boost in production from unconventional assets. Instead, they suffered revenue declines and delivered negative TSRs across the board. Diversified players also delivered negative TSR during the full oil price cycle between 2007 and 2017.

For diversified E&P companies, multiple expansions were the principal drivers of value creation, partly offsetting declining revenues. These expansions, often observed in a downturn, reflected investor expectations of an eventual rebound in earnings. But another factor was at play: Many diversified companies were themselves M&A targets, which also bolstered valuation multiples and positively impacted TSR performance.

Investors in pure-play E&Ps are typically oriented toward growth. However, diversified companies’ shareholder registers have changed as they have grown and are now dominated by value and income investors.. Consequently, for this group of companies, we found that increasing the dividend was paramount to achieving TSR leadership over other diversified players.

The leading diversified companies, such as Occidental Petroleum Corp., provided shareholders with generous dividend yields without diluting the share base. Instead, they raised debt to meet dividend payments. Those diversified companies that opted to pay dividends by issuing new shares or that raised capital through dilutive secondary stock offerings were punished by investors with a weaker TSR performance, a theme that was true for many companies in our sample.

In contrast, investors in pure-play E&Ps delivering good revenue growth were willing to accept stock offerings that raised funds for production growth, but also diluted the share base.

As commodity prices moved upward in the first half of 2018, several diversified E&P companies increased dividends or initiated share buyback programs to reignite investor interest. For example, Anadarko Petroleum Corp. raised its dividend fivefold, cut debt and expanded its share repurchase program to $4 billion. However, following the price dip later in the year, companies will likely put buybacks and dividend hikes on hold in 2019.

A new era for oil prices

The price decline since October 2018 has brought the cost of Brent crude back to range between $60 and $70 a barrel. Market fundamentals suggest that there is potential for oversupply well into 2019 and possibly 2020, depending on the impact of the International Maritime Organization’s ruling to reduce sulfur emissions from marine fuel. This long-term oversupply will likely keep downward pressure on prices and stop them from moving substantially above the band. However, the impact of trade wars, slowing global growth, sanctions and OPEC decision-making are creating an uncertain outlook. In short, oil company boards face the prospect of making investment decisions without a clear price signal, potentially for the next two years.

In this more volatile price environment, pure-play E&P companies realize that they can’t simply grow production without taking into account shareholder returns. Many are basing their investment decisions on anticipated returns and the ability of projects to deliver free cash flow. By maximizing cash flows and controlling spending, companies could in theory deliver self-funding investment portfolios and use their specialized expertise in distinct shale basins to deliver leading profitability and earnings growth.

However, we believe that most pure-play E&P companies will need to access debt financing to achieve their investment objectives. In addition, they lack the financial firepower to deliver the level of payouts offered by larger diversified companies, leaving growth as this group’s primary TSR driver.

Diversified E&Ps face a different set of challenges. In general, these companies don’t possess the financial strength and resilience of the majors or their ability to cope with volatility—though they face a similar imperative to maintain a steady flow of competitive dividends if they are to meet the expectations of income-oriented investors. And, unlike pure-play E&P companies, they can’t rely on revenue growth as their main driver of returns.

What’s more, given the TSR underperformance by E&Ps as a whole, many investors may increasingly be inclined to create their own portfolios rather than rely on a diversified E&P company to manage a portfolio of different assets for them. However, by retaining their diverse approach to geographies, capabilities and asset types, they may still represent a more sustainable investment choice over the long term as the economics of shale production change.

For example, ConocoPhillips Co. was one of the leading TSR performers among diversified players. Its presence across different asset classes has underpinned the company’s dividend program, allowing it to make the payout its top capital allocation priority and meet investors’ heightened payout expectations from diversified players.

TSR action steps

During the lower-for-longer oil price period that started in mid-2014, the operational and strategic focus of most players was on cost control and capital efficiency improvements to maximize the value of existing production and new barrels brought onstream. The industry’s mantra was value over volume. Shale players were able to leverage short project cycles to dial up and dial down spending according to oil price signals.

A key question for E&P companies that have reduced their exposure to non-shale assets is how to allocate capital within the shale space so that they maximize TSR performance while minimizing their exposure to volatile commodity prices.

The need for capital discipline, especially among diversified E&P companies, will become more pressing if they are to meet the goals of a competitive dividend payout, reducing debt taken on during the downturn and coping with oil price uncertainty.

Companies will need to take steps in the following areas so that they can maximize TSR while also ensuring the long-term future of the business.

Upstream margin improvement

In 2019, E&P investors would seem to be more concerned about returns than growth. They appear reluctant to support increased capex, particularly if it is not covered by free cash flows. We believe this investor focus on returns, combined with volatile oil prices, is creating an uncertain outlook for drilling and production activity in unconventional shale assets. Indeed, following a slow start to the year, most U.S. onshore shale players face lower rates of production growth in 2019.

This difficult environment is likely to benefit best-in-class diversified E&P companies more than most pure-play operators. Their differing asset bases with distinct cash-flow profiles provide greater immunity to the issues in an individual geography. And because they are more focused on shareholder payouts, the diversifieds are better placed to meet investor expectations.

For pure-play E&P companies, the task of becoming a future TSR leader is more challenging. Companies that can tap higher-margin assets at the lowest cost will see the greatest improvement in TSR. But to achieve this, they will need to leverage top-tier positions in leading shale basins and find ways to extract additional cost savings. Cost inflation and infrastructure bottlenecks in the Permian Basin, the largest shale area in the U.S., threaten to jeopardize margin improvement in the short term, however.

Today, new digital technologies enable companies to increase operating efficiencies and maximize free cash flow while containing costs and maintaining competitiveness with new energy solutions. Companies that successfully deploy these technologies in the field will have an advantage in delivering bottom-line earnings.

Consolidation ahead?

As the U.S. shale industry matures, consolidation among players is set to gather pace. While some of these deals will be paid for with surplus cash flow, others will be funded by raising debt or through the issue of new shares. Though share dilution and higher debt costs can hurt TSR in the near term, transactions that give companies a stronger market position can add value in the longer term.

We see a continued role for E&P players focused on growth, but consolidation will gradually shift the ownership of unconventional shale resources to larger companies. There has already been a spate of deals between E&P players operating within single shale basins. In addition, some of the majors have recently purchased unconventional resource plays, particularly in the Permian. The majors’ size means they can extract value across the value chain, while also negotiating more favorable terms from service companies, which brings new competitive pressures for E&P players.

Those E&P companies enlarged through consolidation will also need to deal with changing shareholder expectations. Investors have different sustainability and resilience requirements from large players than from smaller ones. A leading competitive position in one basin or one asset type will likely be insufficient to ensure competitiveness over a longer time period. Companies may have to revise their investment strategies as a result.

Shareholder expectations

Companies across oil industry peer groups have initiated dividend increases and share buyback programs to reward their investors following the downturn. BCG research shows that players that can maintain and increase a high dividend command a higher valuation multiple. This is because investors view dividends as a promise of future payouts and a sign of faith in the business over the long term. On the flip side, companies that cut the dividend risk being severely punished by shareholders. Consequently, industry players should carefully analyze their sources and uses of cash and optimize payout plans so that dividends are sustainable even in the event of a significant deterioration in the market.

For E&Ps, it is important to ensure that investors are aligned with company goals. Otherwise, an expectations gap will emerge between management and shareholders, and investors will quickly flee if the external environment deteriorates. North American shale players also need to understand that TSR drivers, particularly the determinants of their valuation multiples, will evolve as their industry matures. By using analytical tools, they can quantify the impact of these drivers and anticipate relative TSR positioning.

Portfolio and growth decisions

In an evolving energy landscape, companies across the oil industry will need to take steps to secure their future—whether by reinvesting in upstream assets or by diversifying across business segments or into new energy investments. Given the high proportion of income-oriented investors that value sustainable dividend payouts in the short term, it may take longer for companies to see the effects of these decisions on their share price and TSR performance. This is particularly the case for companies with larger market capitalizations.

Companies today benefit from a generally more favorable, if more volatile, commodity price environment, but in the future, competition between fuels may depress product prices across the board. Regardless of how these forces play out, we expect price volatility to test oil and gas companies’ ability to make critical decisions.

Players will need to develop a strategy that is resilient in this changing energy landscape, having a plan that creates the right balance among various TSR drivers, and one that signals to investors a clear value creation path for both the near term and long term. They will need to consider the role M&A can play in redefining their portfolios, the impact and sustainability of increasing dividends, the fundamental drivers of valuation multiples, the investor reaction to potential strategy shifts and how reallocating spending among business areas will impact their TSRs. Companies that can successfully build resilient strategies will be the oil and gas value creators during the next price cycle.

Rebecca Fitz, from BCG’s Washington, D.C., office, is a senior director at the firm’s Center for Energy Impact. Clint Follette, from BCG’s Houston office, is a partner and managing director and head of the firm’s North America oil and gas sector. Sarah Burns, from BCG’s Calgary office, is a principal in the firm’s energy and corporate finance and strategy practices.