This particular question has probably been debated since the the advent of the computer.. and maybe even before. However, this may become more of a heated topic in the near future. As rapid strides are made into the realms of AI it’s going to be come more important. We are going to want more and more from our digital personal assistants and it won’t be long before they start anticipating our wants and needs. Amazon and Facebook probably know more about what I want then I do. Siri scans my emails and suggests contacts and calendar appointments. Our smart home thermostats anticipate our arrival to regulate our temperature. Soon driverless cars will anticipate where and when we’ll need to be. If a single entity, such as Siri, becomes the central point for our managing much of our lives it may well be that we establish some form of emotional connection. However, going beyond a digital assistant things get a little less clear. Take for instance the simple command “Make me happy”. As Nick Bostrom points out in his book “Superintelligence” the AI may very well realize that happiness in the human brain is correlated to chemicals and the next thing you know you’ve got a wire running into your brain and an unending grin on your face. Or how about “Keep me safe”. You would almost certainly be locked in metal box and fed through tubes. Maybe even worse, a truly powerful entity could start destroying anything that it deems as a threat.
So, what’s the solution? You can’t possibly program or even train an AI for every possible scenario. Even the simplest task could be misconstrued with catastrophic consequences. If you fed it every law created by humanity.. it would be unusable. I would argue that the only really safe way would be for this superintelligence to fully understand what it is to be human.. is for it to be human itself. Of course, it would face numerous temptations.. such as “I could solve all humanity’s problems with force”. Or, “I should probably wipe out most of humanity and start over”. Our only hope would be that is laid down it’s power long enough to fully grasp the human condition.