Let me tell you a story. Many years ago I worked for big banks and insurance companies. One day I was tasked with a project. It was an amazing, from the tech point of view, project. It was something like this: a user navigates to a bank website looking for information about some product. The website presents the user a simple contact form - first name, last name, phone number and/or email. Based on provided data bank would use it to update user data (if there was no official account it would update the "ghost" account, aka "I know about you, but you don't know about me"). Next the bank would scrape all publicly available social media accounts and build the "hidden" profile (I'll get to this later). Based on all that data, user would be assigned a score based on which all future interaction with a bank would be determined. For a regular person this would mean that "I'm sorry but according to our system we cannot give you a loan".
Now, about the "hidden" profile. It's a thing that all big companies (including banks and insurance companies) hold. It's all the data collected from all publicly available profiles (and sometimes from the shady sites), used to create a profile that's not visible to a frontline workers and it's referenced as a "system decided based on your data".
Now, to make this more scary. This happened 10-15 years ago. Way before the so called AI. Imagine how much more data those companies have about you in today's world and how good they are in processing it.
Hey guys, this right here is a super valuable point to address and really strikes straight to the heart of the ability of a system like this to give the illusion of choice. People absolutely will still think, despite this, they are still in control and we need to address it not dismiss it.
I'm undoing the downvote on this comment, it absolutely is a big part of the conversation, even if you think it's naive.
It’s naive to think you can’t be influenced into buying things you wouldn’t otherwise.
Also there’s the matter of pricing: they’ll get you to pay as much as possible, either by pushing more expensive versions or by actually changing the price you see on websites like Amazon.
What’s the issue if they’re ONLY using this info to improve my experience
Suppose they start out entirely benevolent. That commitment must be perpetually renegotiated in upheld over time. As the landscape changes, as the profit motive applies pressure, as new data and technologies become available, as new people on the next step of their careers get handed the reigns, the consistency of intention will drift over time.
The nature of data and privacy is such that it's perpetually subjected to these dynamic processes. The fabric of any pact being made, is always being rewoven, first with little compromises and then with big ones.
They don't use it only for improving user experience. Based on a user profile they can bump your premiums just because you posted a photo on a snowboard (risky activity) or they can deny you a loan because someone posted on your timeline that you own someone some money.
Also based on your profile you are manipulated to buy products/services you don't really need.