The Customer-Centric Paperclip Maximizer
Imagine that you work at a hypothetical company with one core goal: customer satisfaction. A company that’s obsessed with pleasing the customer. It's your prime goal, to the exclusion of all others. To the point that your company actually advertises itself as "Earth's most customer-centric company".
So... imagine that you're working for Amazon. And today, you're faced with some decisions.
To start with, let's discuss that small open-source company whose service we copied, and are now selling in direct competition with them under their trademarked name. Understandably, the company has asked us to stop using their trademark. But changing the name would confuse our customers, and we can't have that – not if we're Earth's most customer-centric company! Let's just leave this one to the lawyers. Besides, with us competing with them, it's not like they'll be able to fund the lawsuit for long.
Ok, next problem: one of our suppliers has recently released a new product that is selling like hotcakes. Do we release a copycat product that seriously undercuts their price, preventing them from recouping their development costs, and possibly causing layoffs? Of course we do! After all, the customer loves lower prices, and the customer is always right!
You know another thing the customer loves? Receiving things on time. And that makes this one easy: we've just discovered that we're in the midst of a once-in-a-century pandemic, and the fulfillment center staff are asking us to guarantee their safety, even if doing so may cause delays. But what, are they crazy? Do they not understand that we can always replace dead staff? But to disappoint a customer‽ We may never get them back!
Okay quick, snap out of it! You don't work for Amazon! It was just a bad dream. We're safe. They're maximizing for customer satisfaction, not paperclips.
But wait a minute... what happens if their AI interprets zero customers as infinite satisfaction?