AI nudification apps raise concerns

Growing concern over widespread use of intimate AI generated imagery among teenagers has resulted in hefty responses from teachers and policymakers alike. They are calling for a ban on certain apps. The ban on all social media for certain age groups is also in discussion.

Generative AI apps like, but not exclusively, Grok can be used to produce “deepfake” nude images of people, by replacing the head on a naked body with another person’s image, by removing clothes or making it perform a sexual act. The results look highly realistic, often not discernable from real imagery.

“Once they have a mobile phone, it’s not an exception. They have access to pornographic and intimate materials,” says April Anna Barker, chairperson of the board of the CMETB, outlining the issues: “The exposure has rapidly increased. There is nudification, there is abuse. These images are really bad for their mental health, their well-being and their safety.”

Mick Moran, a former garda frontline officer in Monaghan and now CEO of Hotline – Ireland’s foremost service to combatting illegal online content, including child sexual abuse material (CSAM), paints a grim picture: “We’ve seen for the last 12 months that nudification apps cause extraordinary harm to people.”

Hotline dealt with more than 29,000 cases of verified CSAM last year, twice the figure for 2022 and 10 times more than 2020.

As parent and public representative, Minister for AI and Digital Transformation, Niamh Smyth, says the development is horrifying, while not surprising, given the pace of digital developments. However: “Under Ireland’s Online Safety Framework, there is a clear obligation on online platforms to act on reports of illegal content.”

In light of the recent controversies relating to Grok, she wrote directly to X to request a meeting to discuss what steps the platform is taking to address these issues and to ensure compliance with Irish and EU law. This has been to no avail, she said, describing the challenge of getting hold of the big tech companies in the first place.

Yet, she makes clear: “The sharing of non-consensual intimate images is illegal, and the generation of child sexual abuse material is illegal. These are serious offences, both offline and online.”

Minister Smyth encourages users to report illegal content to the online platform where they encountered it and specifically the regulator Coimisiún na Meán, assuring: “We have robust laws.”

In her opinion, nudification apps should be outright banned, given the “harmful content”.

This proposal is ardently supported by Eoghan Cleary, Educational Expert with The Sexual Exploitation Research and Policy Institute. He sees a clear connection between the type of porn available and real life incidents: “90% of mainstream porn is sexually violent. We know peer-on-peer sexual abuse is increasing and we know that sexually violent crime for under 18 years olds in Ireland has multiplied 6.5 times in the 15 years since the majority of Irish adolescents got independent access to the internet through smart phone ownership.”

Double-edged sword

For Seán Stack, principal of St Patrick’s College, Cavan, AI is a double-edged sword: “Technology has been a powerful tool in enhancing learning and teaching for many years now. Of course, as with any powerful tool, it has the potential to be misused and there can be challenges that need to be met by children, their guardians and schools alike.”

As principal he is responsible for 850 students in an all boys schools Cavan. “It is crucial, I would feel, that the link between school and home is on the same page about monitoring the use of this technology for the students. Being fully aware what is going on in this digital world is not something that can be left on schools alone, nor can the piece on educating proper use be left solely to homes.”

CMETB Board Chair April Anna Barker agrees there needs to be a “multi-layered response” coming from home, teachers, but mainly the policy-makers and industry: “Social media giants are the ones disseminating the information, they need to be accountable and regulated better.”

Education key

At the same time, she believes teenagers must be given the tools to have the capacity to deal with the exposure appropriately. A ban, she feels, would not solve the problem: “The ETB recognises that they live in a digital world. In fact, they are streets ahead of us.”

Instead she believes “we must equip them with confidence and teach critical learning. What is real, what is fake.”

In her opinion, classes like Social, Personal and Health Education enable students to understand the implications of social media and show them to be mindful. They foster a collaborative approach and good relationships between staff and students.

A shift in attitude is helping to tackle the issue: “The issue is widely dealt with. Intimate imagery doesn’t involve the same stigma it used to, students aren’t too embarrassed to tell anyone, there is always a trusted adult they can talk to.” In addition to that, every ETB school has a strong safeguarding procedure in place.

St Pat’s have a similar approach: “AI can be dealt with in the very early stages by educating what misuse can look like and the impacts it can have,” says Seán Stack.

Noting that any misuse can be taking place over breaks from school, the principal implores all parents and guardians “to take a very active role in the knowledge and oversight of what their children are engaging with on their phones.”

He added that a school’s response to these behaviours can be to engage bullying procedures with potential disciplinary aspects.

Ultimately: “Anything that would rise to the level of manipulation of sexual content may be a matter for Gardaí and/or TUSLA.”