Last Wednesday evening the House of Lords held a 90-minute debate on a Deloitte research paper, published last summer, entitled Technology and people: The great job-creating machine ( http://www2.deloitte.com/uk/en/pages/finance/articles/technology-and-people.html). Written by myself and my colleagues, Debapratim De and Alex Cole, the paper examining census data for England and Wales since 1871 to track the relationship between technological innovation and the destruction and creation of jobs.

Though limited in scope and focussing on lessons from the past, the conclusions of our report run counter to today's prevailing mood of pessimism about the effects of technology.

Last week's House of Lord's debate was tabled by Lord Borwick, a Conservative peer with extensive experience in automotive and low-carbon technologies.  In opening the debate Lord Borwick observed that fears about the job destroying properties of technology were not new. He noted how, in 1589, the inventor William Lee applied for a patent for a knitting machine that could produce stockings at speed and at far higher quality than could be done by hand. In denying him the patent Elizabeth I said, "Consider thou what the invention could do to my poor subjects. It would assuredly bring to them ruin by depriving them of employment, thus making them beggars".

Lord Borwick argued that while the "minus side of the ledger" - the job-destroying capacity of machines - was clear, the way in which technology creates jobs was less obvious and less well understood. He drew on the Deloitte paper to show how technology has changed the nature and composition of employment. Machines have freed society from often dangerous tasks requiring the repetitive application of muscle power to those involving brain power, creativity and empathy. In the process work has become less arduous and dangerous. And while technology has destroyed jobs in huge numbers, for instance in agriculture and manufacturing, it has created far more jobs. The net effect is that despite breakneck technological innovation, the number of jobs in the UK has more than doubled in the last century and a half.

The general mood of the debate that followed was somewhat less positive.

Lord Rees, the scientist and Astronomer Royal, pointed out that the jobs that have multiplied most since the 1990s are socially valuable but poorly paid—nursing auxiliaries, teaching assistants and care workers. He worried that this trend will accelerate, with robots taking over areas of accountancy, medical diagnostics and surgery, generating wealth for a capitalist elite and little for the rest. Sustaining social cohesion would, in such circumstances, require a massive redistribution of wealth and an expansion of public service jobs.

Lord Giddens, the sociologist and former director of the London School of Economics, agreed that future technology may well overturn past relationships – "The digital revolution has not yet transformed the world as profoundly as the original industrial revolution did, but it is moving far, far faster...we are in new territory...We simply do not know whether the theorems advocated in the [Deloitte] report will hold over the next 20 or 30 years, although some aspects of them may do".

A number of speakers talked of how technology could erode the quality of work and weaken communities. The Bishop of Derby talked of a "dark underside" to technology, creating insecure work requiring greater flexibility on the part of the employee. He cited how the work of a friend, a community nurse, had changed. Her schedule was now determined by a programme on her iPad, requiring her to be at a particular place at a particular time and requiring the filing of a report before moving to the next meeting. She could no longer decide who to visit and how long to spend with them, writing up reports at the end of the day. As the Bishop described it, "This technological efficiency totally cuts across the face-to-face pastoral engagement that people need for healthcare to flourish".

The tone of much of last week's Lord's debate fits with a wider mood of concern about the affects technology. Fears that machines could render humans obsolete, fuelling inequality and social exclusion, are, as Lord Borwick observed, as old as machines themselves. Today the concern is that a wave of new technologies, from artificial intelligence, to advanced robots and driverless cars, will shrink the pool of work in the economy forever.

No one can deny that most change, particularly technological change, has negative effects. History is littered with examples of communities and lives damaged by the loss of jobs in a once-dominant sector. Coal, steel, textiles and shipbuilding have suffered this fate in many Western nations and today China faces similar wrenching change. But on balance, I am an optimist about the effects of technology. Here's why.

First, technological change is the essential driver of human welfare. The application of technology increases the outputs from a given set of inputs – it raises productivity, which, in turn, drives incomes. The slow pace of innovation before the Industrial Revolution meant that, for hundreds of years, living standards scarcely rose from one generation to another. Progress requires the application of technology and changes to the nature of work.

Second, today's emerging technologies may seem to us awe-inspiring, but previous generations have felt the same way. The arrival of the telephone, the car, air travel and computers were, in their day, profoundly transformational and disruptive. Those technologies were seen as different from all that had come before, and indeed they were. Only history will tell whether the balance of advantage to be obtained from today's emerging innovations will be less than in the past. What is clear is that welfare-enhancing innovations have always been disruptive (think, for instance, of the devastating effect of the car on the vast industry of farriers, grooms, stables and vets that made up horse-drawn transport).

Third, history shows that the number of jobs in an economy is not fixed. When a machine comes along and replaces a person the result, paradoxically, is increased demand, new industries and, in time, more jobs. The view that technology is a net destroyer of work has, at least in the past, proved to be a fallacy of composition. Yes, jobs were destroyed and communities suffered, often permanently. But elsewhere, often indirectly, new jobs were created. The net effect was what we have today, a world which is more technologically advanced than ever, and where more people are in work than ever.

Fourth, we can forecast the jobs of the future with no more precision than we can forecast the winning products and fashions of the future. Who would have guessed in the 1970s that gyms, coffee shops, discount airlines or mobile phones would be such a prominent part of our lives today? Technology pessimists have always assumed, and continue to assume, that jobs lost to machines will not be replaced elsewhere. The insatiability of human desire, and the manifest deficiencies of provision in all sorts of areas of modern life in even rich countries, from health care to education and social services, points to enormous unmet demand.

My hunch is that technological change will remain unpredictable, erratic, creative and destructive, as it has always been.

Although the lot of the average person has been improved by technology, the distribution of gains are uneven, and are likely to remain so. There are remedies. In the first half of the twentieth century democratic governments in the West embarked on a massive redistribution of income from the well off to those on low incomes through government spending, taxation and welfare spending. This was a political response to the inequality that grew out of the Industrial Revolution. The growing debate in the UK and the US about the need for a higher minimum wage reflects contemporary concerns about inequality.

On one aspect of the House of Lords debate, the importance of skills and education, there was clear consensus. Lord Fox highlighted the importance of STEM degrees and the massive shortfall in the number of engineers being produced by the education system. For every two engineers recruited in the UK today, three retire. Baroness Rock pointed out that the World Economic Forum has estimated that 65% of today's primary school children will work in occupations that do not exist currently. She said that the government should be focussing on future-proofing education, giving children the strongest possible foundation for this new world.

We cannot predict how today's technology will shape tomorrow's society, or which goods and services its citizens will demand. But history points to two challenges in navigating this new world. If the pace of adoption of technology is accelerating, society will need to prepare for higher levels of unemployment in industries that are transformed. And the way in which technological change tends to rewards high-level education and skills suggests that income inequality may yet widen. Education, training and the distribution of income are likely to be central to the political debate for many years to come.

The content of this article is intended to provide a general guide to the subject matter. Specialist advice should be sought about your specific circumstances.