The question of intelligent machines has always fascinated me. And obviously not just me. There is a huge number of sci-fi books and movies built on the idea that machines surpasses humans as the most intelligent 'species' on earth. It is of course a wonderful idea to develop since it opens up for an array of possible futures. As someone who works with technology and the design of technology, the question of intelligent machines is of course not only entertaining--it is a serious question that deserves serious treatments.
In the book "Superintelligence--paths, dangers, strategies", Nick Bostrom a philosopher, among other things, presents a serious attempt to explore the notion of 'superintelligence', that is, machines that are more intelligent than humans. Bostrom discusses how we can define and understand superintelligence, the possible ways for it to become real, and what the consequences would be if it actually happened. It is one of the most thoughtful treatments of these questions as far as I know. Bostrom approaches the questions in a truly philosophical manner, that is, he does not necessarily judge if a particular development is 'good' or 'bad' but try to explore what it means and how it can be understood and how likely it is to happen. The writing and 'method' of analysis in the book is impressive, even though I have seen reviews that see the book as 'hard to read'. There are some technical parts of the book but they are not too difficult for an informed reader.
It is obvious that Bostrom is warning us. Superintelligence, if realized, poses a definite threat to humankind. Bostrom explores several strategies that humans could take in trying to 'control' the consequences of a superintelligence, but it is clear that they all have their issues and problems.
Overall, anyone who is involved in the development of intelligence inside machines should read this book. I think there are many aspects of Bostrom's reasoning that is applicable on much less intelligent machines. Many of the issues that he discusses can be seen in simpler and more contained forms as soon as we have machines doing things that we humans can't do and are not able to understand, such as highly sophisticated algorithms or extremely powerful data analysis. As soon as we do not have full transparency of what the machines are doing to reach their results we are to some extent in the same situation as Bostrom describes as dealing with a superintelligence. There is of course an extra level when you reach the 'super', and that is of course what Bostrom is primarily examining. So, I think this book deserves reading by anyone who is engaged with enhancing the ability of machine 'intelligence' and especially by those who believe that superintelligence is not possible and those who believe that it is a 'solution' to problems we human experience.