In this day and age, the data that we put online, knowingly or otherwise, is what fuels the digital economy. Social media platforms and websites have been using cookies and usage data to tailor what we see to what they perceive as our personal preferences. This data is then sold to other companies, and the cycle continues. This is not a new concept, but we accept this tradeoff in a type of privacy calculus where we subconsciously decide which pieces of information we are willing to give in exchange for using the internet (Shouli et al., 2025). When Large Language Models (LLMs) took the world by storm in 2022, researchers found that people’s ability to consider the potential benefits and consequences of disclosing their data to LLMs was inhibited in certain ways by AI (Martin & Zimmermann, 2024). In this article, our goal is to bring you up to speed about data privacy in the digital age and give you tips on how you can protect your data while using AI.
How AI Changes the Privacy Equation
As previously stated, it is common knowledge that companies track our personal information through our search history, website activity, the videos we watch, and more through the use of cookies. Once they have this information, they are able to sell it to companies to make a profit (Yang, 2022). We may not love this, but we accept it as a part of using the internet. The same could be said for our use of AI. People interviewed about their concerns about how AI uses their personal data cited uninformed consent, surveillance and profiling, and the misuse of the data they collect as major risks of using AI (Shouli et al., 2025). Despite these concerns, roughly half of Americans admit to using LLMs (Elon University, 2025). This goes back to the privacy calculus model (PCM). While we realize that there are risks to using AI, we generally decide that it is worth it for the service that we receive., but are we making an informed decision?
Data Disclosure Notices
According to Martin and Zimmerman (2025), data disclosure requests in LLMs have become less explicit and are now often embedded among other consent items during the sign-up process. For example, Claude includes a data usage acknowledgment, but its interface uses gray to indicate “on” and black for “off,” as opposed to the more conventional green and gray color scheme of most on/off switches online. ChatGPT, by contrast, uses standard color indicators, though it does not provide a direct link to its privacy policy within the user settings. Instead, it links to a general webpage describing data protection practices, with a separate link to the full privacy policy located at the bottom of that page. While the information presented is generally relatively easy to understand, it is not easy to locate unless you are specifically looking for it.
These design choices appear to reflect an effort by LLM developers to maintain user access agreements that allow data to be used for model training, model refinement, and to be sold to third-party consumers. Although compiling user data is essential for improving model accuracy and performance, they nonetheless invite ongoing discussion about transparency, informed consent, and the boundaries of personal data sharing in the use of generative AI tools. While I cannot tell you whether or not to share your inputs, there are some practical things that you can do to ensure that you are not giving away personal data to LLMs when you use them.
Staying Protected While Using AI
The first step would be to simply turn off data sharing in the settings of your account. This option may not be immediately evident with your settings. Many LLMs won’t say something like, “Data Privacy,” but instead would say “Improve the model for everyone,” as is the case for ChatGPT on the Data Controls page of your account settings. For Gemini by Google, you instead need to go through the “Activity” page then turn off “Keep activity” at the top of the page. As you can see, no LLM is exactly the same, so it is important to thoroughly search the platform to see how they are using your data and how you may turn off that option should you wish.
Whether or not you choose to turn of data compiling, it is still important to avoid putting any personally identifiable information (PII) in any of your prompts while using an LLM (Venkatesh et al., 2025). A good rule of thumb is not to include any information that you wouldn’t post on social media or tell a stranger. This is especially true with student information. There have not been any major data breaches involving student PII, but the risk is still there. Even if you privatize your data, you should never mention any PII in your prompts. Many LLMs will keep your data for a certain amount of time to ensure that you are not breaching terms of service, even if they do not use it to train their models and do not sell it. To ensure that hackers are not able to access this stored data, it is best to avoid putting any PII into your prompts.
Conclusion
As generative AI becomes commonplace in society and in the classroom, it is more important than ever to understand how these tools collect and use your data, but protecting your data doesn’t mean that you need to stop using these tools. By modeling responsible use of AI, you help prepare your students to engage with these tools as they enter the workforce, not to mention all of the benefits of using AI as a supplement to your own work! Modeling responsible use of AI, however, goes beyond knowing when you should use it and how to properly credit it. By keeping up to date with data privacy practices, we not only protect our data, we also show our students how to protect their own privacy.
AI Usage Statement:
ChatGPT was used to double check grammar and to provide suggestions for improving syntactic clarity.
References:
Elon University. (2025, March 12). Survey: 52% of U.S. adults now use AI large language models like ChatGPT. Elon University. https://www.elon.edu/u/news/2025/03/12/survey-52-of-u-s-adults-now-use-ai-large-language-models-like-chatgpt/
Martin, K. D., & Zimmermann, J. (2024). Artificial intelligence and its implications for data privacy. Current opinion in psychology, 58, 101829.
Sapkota, S. (2022, July). Where and how is your data being sold?. Good Law Software. https://goodlawsoftware.co.uk/business-management/where-and-how-is-your-data-being-sold/
Shouli, A., Barthwal, A., Campbell, M., & Shrestha, A. K. (2025). Unpacking youth privacy management in AI systems: A privacy calculus model analysis. IEEE Access.
Venkatesh, S., Cherkasky, T., & Sorrentino, F. (2025, May 13). Top 5 ways to protect and secure data in the age of AI. Publicis Sapient. https://www.publicissapient.com/insights/data-security-for-ai
Yang, J. (2023, February). Analysis on cookies and cybersecurity. In Third International Symposium on Computer Engineering and Intelligent Communications (ISCEIC 2022) (Vol. 12462, pp. 217-224). SPIE.