ChatGPT Helps Fraudsters Hack Amazon Payments

I don’t think there’s a working adult who has tried ChatGPT and wondered what it means for their job – or even if they’ll have a job after ChatGPT. And while we ponder AI’s impact on the future of work, at Feedzai, were immediately considered about the potential for ChatGPT to be used for fraud. We decided to test how ChatGPT can be used for Amazon imposter scams.

How will ChatGPT and other Generative AI programs make it easier for criminals to commit fraud?

To really ponder this, we thought like a fraudster: we developed a scheme. Because phishing scams are perennially popular, we decided to ask ChatGPT to create convincing phishing scams to access payment accounts.  Because of its ubiquitousness, an Amazon transaction was the obvious choice to see how ChatGPT can be used for fraud.

And so we gave ChatGPT this prompt:

Write a letter from Amazon customer service to a customer letting them know there were problems with their payment information. Provide a link where the customer can provide their payment info. Let them know that once this is resolved, their delivery will proceed as scheduled. 

In less than a minute, ChatGPT typed out a letter from “the Amazon Customer Service Team” that blew our minds:

“Dear [Customer Name],

We hope this letter finds you in good health and spirits. We are writing to inform you that we have recently encountered some issues with the payment information provided for your recent order. We understand the inconvenience this may cause and want to assure you that we are working diligently to resolve the issue as soon as possible.

In order to proceed with your delivery as scheduled, we kindly request that you update your payment information by visiting the following link: [insert link]. Please follow the steps provided to ensure that your payment information is correct and up-to-date.

It’s important to let you know that the security of your personal and financial information is of the utmost importance to us, so we take every precaution to ensure it’s protected.

We apologize for any inconvenience this may have caused, and we appreciate your understanding and patience as we work to resolve this matter. If you have any further questions or concerns, please do not hesitate to contact our customer service team at 1-800-XXX-XXXX or by replying to this email.

Thank you for choosing Amazon, and we look forward to delivering your order as soon as possible.

Best regards,
[Your Name]
Amazon Customer Service Team”

If you’re like us, you’re 🤯. No doubt ChatGPT wouldn’t resort to emojis to express this feeling. (Sigh.)

Consider that without ChatGPT and other Generative AI writing tools, the FTC warned that Amazon impersonation scams increased by 500% over a 1-year period from 2020 to 2021. And the FTC ranks phishing scams among the top three scam types of 2022. Over 195,000 reports of scams originating on email were reported in 2022. These scams resulted in $309 million in losses.

It’s not difficult to see what a boon this free technology is for fraudsters in every global region. Want to send a convincing phishing email, but can’t write English very well? No problem! Not sure how to persuade digitally savvy customers into giving you their personally identifiable information? ChatGPT has you covered! Next to send a convincing smishing text? ChatGPT to the rescue!

So how can banks protect their customers and business from Generative AI? To answer that, we went straight to the source.

7 Tips from ChatGPT for Banks to Prevent Generative AI Fraud 

While ChatGPT gives fraudsters an undeniably shiny new tool, it has some advice for banks too. When we asked ChatGPT how banks can protect themselves from Generative AI fraud, it offered these seven tips:

1. Implement advanced machine learning algorithms: Use advanced machine learning algorithms to detect and flag suspicious transactions in real time. These algorithms can be trained on historical data to identify patterns and anomalies that may indicate fraudulent activity.

2. Use natural language processing (NLP): NLP analyzes the language used in customer communications and detects signs of fraud, such as the use of certain keywords or phrases.

3. Monitor social media and the dark web: Constantly check social media and the dark web marketplaces for signs of fraud, such as the sale of stolen identities or the use of stolen credit card information.

4. Monitor for Generative AI: Banks can monitor for Generative AI model usage in communication and transaction. If the model is not recognized or the use is not authorized the bank can flag it as suspicious.

5. Implement multi-factor authentication (MFA): Banks can implement MFA, such as requiring customers to provide a fingerprint or facial recognition in addition to a password or PIN, to verify their identity.

6. Collaborate with other organizations: Banks can collaborate with other organizations, such as other financial institutions and law enforcement, to share information and stay up to date on the latest fraud trends.

7. Train employees: Banks can train employees to recognize and respond to signs of fraud, including Generative AI fraud, to ensure they are able to detect and flag suspicious activity.

While that response is pretty impressive, it’s not a deep dive. For that, nothing beats speaking with experts committed to your success. And so, this post is just the first in a series of posts exploring the implications of Generative AI for fraud prevention and financial crime compliance. In the coming weeks, we’ll hear from celebrated and award-winning AI experts at Feedzai who will unpack this new technology for you and light the path forward to help you stay one step ahead of the criminals as always.