Whenever I raise doubts about optimism, the eyebrows go up and it's not hard to tell what people are thinking.

‘Are you trying to suggest we should be miserable instead?’

Surely optimism is a good thing?

Yes, optimism is a good thing. It gives us hope. It keeps us motivated and resilient. It even keeps us healthy and we live longer.

It's healthy (and normal) to be more optimistic about our success, popularity, influence and abilities than we can justify with hard facts. Ask an audience to raise their hands if they think they are above-average drivers. Expect about 80 per cent of the hands to go up.

Think of a line stretched between hopeless on the left and total confidence that everything will work out on the right. Most of us live a little to the right of the midpoint. We may be a bit unrealistic, but the facts don't matter as much as the benefits.

Go further to the right of the centre point and realism does start to matter. We are likely to deny reality and tell ourselves that everything will come right, even if we do nothing. We are likely to set goals with no thought of the action it takes to get there, or what we might do when we encounter a setback.

A more realistic view of optimism: the Stockdale Paradox

The story of James Stockdale, illustrates the value of a more realistic view of optimism.

Admiral James Stockdale survived eight years as a prisoner of war in Vietnam and was tortured more than 20 times. He says that the key to surviving and growing from the experience was to retain his faith that he would be released, while at the same time having the discipline to face the brutal facts of his reality. (These days, that combination of hope and realism is known as the 'Stockdale Paradox'.)

When the author Jim Collins asked him who were most likely to die in captivity, Stockdale replied, ‘Oh that’s easy, the optimists. They were the ones who said, ‘”We’re going to be out by Christmas”. Christmas would pass and they would say, “We’re going to be out by Easter”. They died of a broken heart.’

Too much optimism is just fantasy, or denial. The solution is not hopelessness, but balance.

Two useful tips based on the research

1. Apply some healthy pessimism

Ask yourself, or your team, what might go wrong as you strive for your goal. Come up with a strategy to handle that setback.

2. Ignore the pressure to be relentlessly positive

One research team calls the social pressure to be relentlesly positive tyranny. It leads to over-confidence and denial. Encourage the skeptics in your team. They are team players too.