With large language models ("LLMs") taking the world by storm, it's safe to say that they have established their place as a mainstay of modern technology. In 2023 ChatGPT has particularly captured attention with many people having incorporated LLMs into their professional workflows and everyday lives. Whilst many people have found various ways to utilise LLMs, hackers have been actively seeking to find workarounds to bypass LLMs' acceptable use and content policies. Their goal of which is to turn these models into tools which can significantly increase the rate of cyberattacks by writing malicious code or phishing emails within seconds.

LLMs refer to large language models with artificial intelligent systems that are trained to recognise and analyse a specific language, such as English. ChatGPT is an example of a popular LLM. These models are capable of understanding written language and can provide responses to users based on a specific query or instruction.

With the increased adoption of LLM technologies, there is a growing misconception that removing any proprietary, commercial, or personal information before uploading such information into an LLM is sufficient to mitigate the associated risks of LLMs. Although such efforts will reduce the risk of breaching privacy laws such as the Protection of Personal Information, 2013 ("POPIA") , LLMs may still expose companies to cybersecurity risks. They can cause commercial loss if sensitive, proprietary or confidential information or data is leaked.

Broadly, LLMs can mainly be exploited in three ways:

  • Misinformation campaigns;
  • Generation of malicious content; and
  • Bias exploitation.

In this article, we explore the risks that companies may be exposed to when LLMs are incorporated into their operations.

Prompt injections

A prompt injection is the deliberate use of tailored prompts to influence the responses generated by LLMs. For instance, to make use of an LLM like ChatGPT to produce malware or to provide misinformation to other users. Hackers can utilise prompt injections as a means to launch large-scale misinformation campaigns. The primary purpose of prompt injections is to bypass content policies which prevent language models from generating harmful or objectionable content including malware or phishing emails. Essentially, hackers use prompt injections to manipulate the output of LLMs. Prompt injections can be used inter alia to force LLMs to produce false information or to programme code for viruses and other malware.

Prompt leaking

Prompt leaking is a method used to extract sensitive or hidden information from LLMs. These models are trained on extensive data and are often connected to commercial data sources, which makes it possible for competitors to use prompt leaking to obtain confidential company data such as stock levels, new software releases, or upcoming software features.

For instance, a competitor might use prompt leaking to extract information about an upcoming software release. If they can develop and launch the same software before the product goes to market, they could potentially capture the target market and reduce the overall market share. Essentially, the commercial risk associated with prompt leaking is that competitors can use LLMs to identify strategic plans and commercial objectives. Again, this information could be misused to intentionally diminish the market share by targeting the client base or releasing new products or software before the intended market launch.


Jailbreaking has been around for a long time and is historically known for unlocking iPhones to get free access to all App Store applications. Nowadays, jailbreaking is also used on LLMs, such as ChatGPT, to enable the LLM to produce content that its policies and restrictions would typically block. Broadly, jailbreaking aims to deceive the LLM into believing that it is not bound by its content policy or use restrictions, allowing the LLM to generate any output. These jailbreaking methods are often referred to as "Do Anything Now" (DAN) or "Developer Mode". A hacker's primary purpose in jailbreaking an LLM is to get around the prohibition of generating malware. If hackers are successful, this can significantly increase the rate at which hackers can generate malware which would likely lead to a massive increase in the number of cyberattacks.


Operationalising LLMs may expose your company to several risks including, amongst others:

  • Leakage of commercially sensitive information;
  • Security compromises (and potential liability under POPIA);
  • Creation of an additional attack vector for hackers to exploit as an entry point into the company's infrastructure;
  • Potential leakage of underlying system parameters, leading to the inadvertent release of sensitive or commercial information. For example, system information relating to unreleased products, new software updates, stock and inventory levels, or even a list of existing clients; and
  • The potential for commercial loss if a competitor is successful in extracting proprietary information (such as unreleased products, new software updates, stock and inventory levels, and client lists) using an LLM.

Accordingly, companies must maintain oversight and governance throughout the lifecycle of LLMs, ranging from procurement or development through to its operationalisation. Companies should carefully consider what commercial data will be made available to LLMs. It is recommended that companies create separate databases to house the data which will be made available to the LLMs. This will enable oversight over data which may be at risk of being leaked and will reduce the likelihood of unforeseen data being leaked.

Mitigation steps

To guard against the risks identified in this article, companies leveraging LLM functionality should review their existing cybersecurity framework to account for the additional risks introduced by the LLMs. In conjunction with the cybersecurity framework review, internal policies should also be implemented to regulate the company's use of LLMs internally. ENS' specialist Technology, Media, and Telecommunications team has created an AI toolkit which helps organisations deal with the multiple risks faced by AI technology adoption and to fast-track AI implementation in a responsible manner.

The content of this article is intended to provide a general guide to the subject matter. Specialist advice should be sought about your specific circumstances.