When Total, the French energy group, began in the 1990s to consider the tricky task of extracting extra-heavy crude oil from central Venezuela’s Orinoco belt, the company’s technicians would tell their bosses that all they needed to make the venture profitable was an extra $10 a barrel on the international oil price, recounts Thierry Desmarest, the company’s chairman and chief executive officer.

That wish has been granted. But it is not just higher oil prices that have made possible Total’s venture into what the industry calls “unconventional” oil – difficult to extract and requiring complicated processing. Advances in technology have played a part.

“When we and ConocoPhillips [the US company that also invested around the same time] looked at Venezuela we were convinced we could bring the cost of production very low,” Mr Desmarest says of the venture, which began producing oil in August 1998. “New technology really changed the picture.”

Some companies are exploring even more difficult locations. In the massive Athabasca oil sands of Canada, companies including ExxonMobil, Royal Dutch/Shell and ChevronTexaco, three of the world’s largest energy groups, extract sticky, tar-like bitumen from sand in a mining operation so messy and expensive that some of their competitors still dismiss it as foolish. Roughly two tonnes of the sands must be dug to produce one barrel of oil.

Are such projects the answer to the “capacity crunch” that has sent oil prices soaring this year? The shrinking gap between oil supply and demand is bringing fresh urgency to the question of whether new sources can be located rapidly enough to ensure a balance over the medium and long term.

David O’Reilly, ChevronTexaco’s chairman and chief executive officer, says: “There is a lack of spare capacity . . consumers are legitimately uneasy about how reliable oil supplies are, given the somewhat narrower margin for error.”

Canada and Venezuela combined possess twice the oil reserves of Saudi Arabia. But Saudi Arabia’s “conventional” oil would be much more easily accessible and relatively cheap for foreign companies to extract – if they could get access to the kingdom. Meanwhile recent exploration for new reserves has proved relatively unsuccessful: global reserves found since 2000, when Kazakhstan’s giant Kashagan oilfield was discovered, have fallen 40 per cent compared with the previous four-year period.

This failure to find easily exploitable oil leaves the industry with a stark choice. Either companies pursue challenging, unconventional ventures with high overhead costs – which could prove unprofitable if large volumes of cheaper oil come to the market – or they continue to push to invest in countries such as Russia and Saudi Arabia, where the political climate for investment is, at best, uncertain.

How men such as Mr Desmarest balance these alternatives will have a profound effect, not only on their companies but also on whether the world’s growing thirst for oil can be met.

The decline of oil was most famously predicted by M. King Hubbert of the US, who was largely correct when he forecast, 14 years before the event, that oil production in the contiguous US (excluding Alaska and Hawaii) would peak around 1970. He believed the peak for world oil would come around 2000. But companies continued to find oil reserves.

The US Department of Energy now believes a production peak will come between 2030 and 2075. But supporters of the “peak oil” theory have gained a following again as the oil price has risen to record nominal highs of nearly $50 a barrel.

They point out that no big oilfield has been discovered since Kashagan in 2000. They argue that reserve estimates given by countries such as Saudi Arabia are overblown by governments keen to preserve the political power their oil affords them. And they quote Ali Naimi, Saudi Arabia’s energy minister, who last week put the annual global decline of older fields, such as those in the North Sea and US that came into production after the oil shocks of the 1970s, at 8-10 per cent.

“People have been crying wolf for the last 30 years, but now they may be right and no one is listening to them,” said Peter Thiel, a former internet entrepreneur and now hedge fund manager whose $250m portfolio is heavily weighted with energy equities and commodities. He believes future energy prices are too low accurately to reflect the world’s fast dwindling reserves of oil. He points out that the last independent audit of Saudi Arabia’s reserves was 20 years ago.

Oil companies have found it difficult to locate new oil reserves. Shell this year became the industry’s most dramatic example, having to cut its proved reserves estimate by more than 20 per cent after struggling to find oil and providing the US Securities and Exchange Commission with false data from 1994 to 2002. But the problem is industry-wide. Between 2001 and 2003 only six of the world’s 15 leading oil companies managed to replace all the reserves of oil they pumped, according to Deutsche Bank.

Most oil executives reject the notion that oil will soon run out. They say it is the lack of investment in infrastructure to tap and transport reserves that has created fears of a capacity crunch.

In the past, periods of higher oil prices have always stimulated investment, which in turn has often led to technological advances. People continuously underestimate the impact of technology, says Daniel Yergin, chairman of Cambridge Energy Research Associates. “Every time people think it’s over you have a technological revolution,” he says.

But this time, analysts say, companies appear to have other priorities. Rather than emphasising production goals, many are adhering to strict targets for returns, while handing free cash back to shareholders.

“There is a serious danger that short-termism driven by the demands of the stock market . . . may prove to be seriously damaging to oil consumers,” Paul Stevens, professor at the University of Dundee, wrote recently in the Middle East Economic Survey. He pointed out that BP is expected to return about $18bn to shareholders in the next three years as long as oil prices stay high and that, in this decade, Deutsche Bank estimates major oil companies have cut back their exploration budgets by 27 per cent.

“The natural response to higher prices is to see more investment, but our investment criteria are based on good returns,” Mr O’Reilly says. High returns depend not only on high oil prices but also on a project’s cost and the share of profits that oil-rich countries are willing to give companies.

For many international oil companies the preferred option is to have better access to countries that have previously been hostile to foreign investment.

About two-thirds of the world’s easily-tapped oil reserves lie in the Middle East. But despite years of negotiation leaders such as those in Saudi Arabia and Kuwait remain unwilling to reopen the doors to international energy groups on the terms oil companies want.

Leading oil company executives were in Vienna last week to meet the energy ministers of the Organisation of Petroleum Exporting Countries, the cartel with many Middle Eastern members that controls more than three-quarters of the world’s oil reserves. Total’s Mr Desmarest says: “It is important to have the opportunity to invest in Opec countries. If you want to increase [world oil production] capacity, it is key.”

While oil-rich countries have enjoyed three years of high prices, many have not reinvested the profits in finding and producing more oil. Instead, they have often spent the money on keeping powerful minorities on their side and on social programmes that are becoming increasingly expensive as their populations grow and unemployment rises. An Opec report last month shows a 6.5 per cent decrease in well completions in 2003 compared with the year before.

Russia is another hope for foreign investors. BP has a partnership with TNK, at the time of the joint venture the country’s third-largest oil company, and other oil company executives are pushing hard to expand there. Big projects are under way in eastern Siberia. Impressive double-digit oil production growth has made Russia the world’s largest producer and second largest oil exporter (behind Saudi Arabia).

But opportunities in Russia are also limited by political uncertainty, especially following the government’s judicial assault on Yukos, once the country’s largest oil company. Another big challenge to production and export growth has been Moscow’s indecision over which of many pipeline and export terminal upgrades to support. Analysts at Sanford Bernstein expect Russian production to peak in 2007/08 and growth to slow to 7.5 per cent in 2004, 6 per cent in 2005, 4 per cent in 2006 and 2 per cent in 2007.

The continued uncertainty over access to easily exploitable oil is leading other companies, including Total and Royal Dutch/Shell, to pour money into high-cost ventures such as extraction of extra-heavy crude oil, liquefied natural gas (LNG) and “gas-to-liquids” (GTL) – a high-cost yet commercially unproved process that produces cleaner-burning diesel, among other fuels.

In the past year Qatar alone has attracted $33bn worth of proposed investments in GTL.

Analysts at Deutsche Bank believe that, after 2008, almost half the opportunities for oil companies will be in LNG, GTL and the Venezuelan and Canadian oil sands, with deep water projects making up another quarter of the expected openings.

But oil companies diverge on the need to get into such projects. Lord Browne, BP’s chief executive, rules out any big investment by BP in Canada’s oil sands and GTL. “There are tons of opportunities and if there aren’t today, there will be tomorrow,” he says.

Whatever options companies do choose, decisions must be reached quickly in an industry where lead times are inevitably long.

Today’s tight oil supply, for example, may be partly due to excessive optimism in the late 1980s and early 1990s about how quickly companies would be able to negotiate investments in countries such as Angola, Azerbaijan or Georgia. Any lack of investment today would be felt about 10 years from now.

“We are in an industry where we have to be patient. Total had 10 years of technical co-operation with Pdvsa [Venezuela’s state oil company], before being allowed to invest in Venezuela,” Mr Desmarest says.

IHS Energy, an industry consulting firm, calculates that until 2008 enough projects will come to fruition for production to meet demand. World oil demand in 2004 is expected to grow by 2.5m barrels a day, more than twice the rate of the past four years, with growth of 1.8m b/d expected in 2005. Further ahead, analysts warn that demand projections are a “fools’ game”.

On the supply side, few can guess how fast Iraq will be able to overcome its security problems and return to full output. Other shocks to supply remain possible. The assumption that Saudi Arabia would be an infallible supplier was shaken with this year’s attacks on international oil workers.

But new regions are being opened up. Libya is expected to welcome US investment after Washington lifted sanctions on the country this week. Mr Yergin says: “One of the biggest positives is the reconnection with Libya, which will have the effect of expanding capacity. A year ago Libya was not on the map as a growth area, today it is.” Many – though not Mr Yergin – believe Washington’s desire to see more oil investment played a part in its entente with Tripoli.

If a number of other Opec countries were to allow greater foreign invest ment, interest in greater exploitation of unconventional oil sources would surely wane quickly. But so far there are few indications that they will do so, leaving oil companies stuck with difficult choices.

As Lord Browne says, oil companies “cannot be bureaucratic institutions. The fact is that this is a risk business and you have to make big calls.”