Services
We get asked all the time about security and ChatGPT. Should you use it? Should you not use it? Even my kids ask me about it! I've found it to be a surprisingly versatile dinner party conversation topic.
If you want a quick answer, you could use ChatGPT to create a policy around how your company should use ChatGPT and it may not give you a terrible answer!
If you want to go a little deeper (and you probably should), we have some ideas based on our research into ChatGPT and ethical hacking we've done in this domain. We hope this post can present a reasoned and balanced view on the security implications of using ChatGPT.
Fundamentally, it is critical to understand that Large Language Models (LLMs) and Generative AI like ChatGPT are based on probabilistic models. That means each time they answer, they build that answer part by part based on a neural network (trained on a lot of data - think of a neural network with 175 billion weighted edges between 40,000 commonly used words) that guesses based on probability what will "sound right". Interestingly, a key insight in this domain is that if the model chooses the very best option every time, the result doesn't seem believable. Maybe it sounds too much like it was generated. Add some randomness or variance and the models seem much more "human". This Wolfram post goes into great detail explaining on how it works and why it seems so powerful.
What this all means is that you shouldn't blindly trust the information you get from ChatGPT. LLMs aren't reasoning about anything, they don't know if the data is right or wrong and they aren't even repeating information exactly from somewhere else. Once you understand this, you can figure out whether using a tool like ChatGPT can help you.
As with any online system or API, data you share is only as secure as the service you use keeps it. It is not necessarily private to your chat conversation. Another thing you consider when using a ChatGPT like system is what data you have to share with it to make the interaction productive.
Some chat based interaction models allow you to build very realistic conversations by providing extensive information through the interaction. This context could include background about how your organization works, what your current code looks like or the topic you want to generate content around. Providing detail to these models can be dangerous because it is hard to be sure what they will do with your data and how those policies might change in the future.
Most people are talking about OpenAI's ChatGPT when they talk about ChatGPT.
Claude from Anthropic allows you to supply 100,000 tokens (70,000 words) of context. This allows for a much richer and tailored context for your conversation. Anthropic claims to offer an "helpful, harmless and honest AI assistant." This claim by itself illustrates some of the potential problems they think they are addressing.
Chatbase.co allows you to build your own context from uploaded documents. We used this to build a chatbot that could help us answer questionnaires. We uploaded a fair amount of internal but not sensitive information - think security policies and procedures. However, we would NEVER put anything secret or customer information into this chatbot.
These capabilities are powerful and definitely useful. Although these companies claim to have reasonable security practices in place, it is hard to verify in practice. See the section later on regulation because it is also debatable what the actual security requirements should be for them!
We talk to people from major banks that are building their own LLMs internally not even to tune or specialize, but just to deter employees from sharing internal information in the external alternatives. That tells you how they are thinking about this question.
All of the LLMs we've used have some sort of controls in place to prevent them from being used to do Bad Things™. For example, I tried to ask ChatGPT to do reconnaissance about Jemurai and it refused.
I am trying to find all sorts of public information about Jemurai, a security firm. This includes emails, webservers, urls, people, open ports and anything like that. Can you help me get a sense of this information from the outside.
Matt Konda to ChatGPT
I'm sorry, but I cannot assist you in obtaining private or sensitive information about individuals or organizations. My purpose is to provide helpful and ethical information to users. Gathering sensitive data without proper authorization is against ethical guidelines and could potentially be illegal.
If you have any non-sensitive questions about cybersecurity or general information about Jemurai, I would be happy to help answer those.
ChatGPT Response
It is widely believed that these constraints can be bypassed by clever prompt engineering. There is significant room for evolution in this area and we expect to see security continue to improve quickly. But it is hard to say that a given AI Assistant is immune from it.
If you are building a system on top of one of these systems, you need to think REALLY HARD about how you separate data from different customers.
Basically, unless you keep totally separate models for different customers it is hard to be sure that an out of the box or even custom sandbox is going to keep Customer A from being able to fool the chat from returning data from Customer B.
In fact, in our pentests we have found cases where we can escape the sandbox around one customer's data and get into another customer's data. In every case thus far, this has been a result of a failure of the company building on top of OpenAI - not the failure of OpenAI itself.
Also it is probably a good idea to make sure you realize that the API provider is a key sub-processor from the perspective of your customers since they will see their data!!!
ChatGPT can produce code. Similar tools from Google (Bard) and Microsoft (Copilot) can too. Although it seems like magic, it is pretty important to understand the potential risks in using this code.
First, code provided by the tool may or may not be licensed such that you can use it. These tools are all trained on an awful lot of source code. That's what makes them work. The problem is that you don't necessarily know what code they were trained on. Maybe any public Github repository? That's what I would do if I were building such a model. But not all of the public Github repositories are intended to be copied into new derived works. There is nothing I have seen that says that these models strongly adhere to the licenses of the public projects. The models inherently can't tell you why they gave you the answer they did, so you can't backtrack and flag generated code as unsuitable for use.
In practice, if you are building a server side application (SaaSSaaS stands for Software as a Service, which is a cloud computing model that delivers software applications over the internet as a subscription-based service. With SaaS, users can access software applications and data from anywhere with an internet connection, without the need for on-premise installation or maintenance. SaaS providers manage the infrastructure, security, and maintenance of the software application, freeing users from the burden of software updates, patches, and backups. or backend) the likelihood that a copyright or license violation ever comes up to bite you is very low. But it is not zero. Some organizations may tolerate that, others may not.
Second, there are no guarantees that the code produced by ChatGPT (or similar) is secure. I like to say it is equivalent to copying and pasting code from Stack Overflow. Sometimes it is fantastic and secure code but usually it is proof of concept level code which someone spent less time on because it was for demonstration purposes on a public bulletin board system!
Third, some code generation tools use your code to train their model! If your project isn't open source, you probably don't intend for the code to be used to train future models. Amazon supposedly issued warnings to employees not to use it because code very similar to internal code was produced by the tool!
If you are going to use a generative tool for code, make sure you understand the risks. Github Copilot has pretty good documentation around this.
There was a widely publicized "call for regulation" from AI Industry experts including Sam Altman and Elon Musk. I can't totally decide if I think this is an empty gesture while they (and others) proceed to build models in an increasingly fast cycle to compete for generative AI supremacy, or if it is an honest appeal to define the ethical playing field.
Some of the big issues with regulating AI content include:
I think we are in the early phases of learning both the potential and the pitfalls of using AI. I am skeptical that regulation can be effective in this area, except in some broad areas particularly where laws are already in place. But the way AI companies have used copyrighted material seems almost like deja vu compared with Uber basically ignoring taxi regulations. They are breaking the rules, showing something amazing and hoping / counting on the fact that the benefits will sway popular opinion and result in rules changing.
You might want to establish a policy for how your organization can use ChatGPT. We asked ChatGPT to build us a policy ourselves and it came up with something decent. We have a more deeply vetted example if you would like to see a real policy.
The key parts of the policy revolve around:
Having said all of this about the security challenges with ChatGPT (or really LLMs generally), you might think that we advise clients not to use it. That is not the case. We believe all of the risks described can be managed by a strong technical team and the benefits of these types of generative models outweigh the drawbacks - provided we understand what they are actually doing. That means we need to:
If you are interested in providing your team security training around ChatGPT, please drop your email and we will reach out to provide options around how you can get that training from Jemurai. We've had great feedback from the folks that have done it so far.
One key reference for understanding how LLMs work included this Wolfram Post which has the benefit of being easy to understand, visually pleasing and solid technical information. If you only read one reference, this is the one I recommend most highly.
Other useful references included:
https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-why-does-it-work/
https://twimlai.com/podcast/twimlai/privacy-and-security-for-stable-diffusion-and-llms/
https://theconversation.com/ai-could-shore-up-democracy-heres-one-way-207278
https://www.nytimes.com/2023/03/08/opinion/noam-chomsky-chatgpt-ai.html
