Stanford researchers find Mastodon has a massive child abuse material problem
Stanford researchers find Mastodon has a massive child abuse material problem
Not a good look for Mastodon - what can be done to automate the removal of CSAM?
https://stacks.stanford.edu/file/druid:vb515nd6874/20230724-fediverse-csam-report.pdf
I'd suggest that anyone who cares about the issue take the time to read the actual report, not just drama-oriented news articles about it.
If I can try to summarize the main findings:
- Computer-generated (e.g.., Stable Diffusion) child porn is not criminalized in Japan, and so many Japanese Mastodon servers don't remove it
- Porn involving real children is removed, but not immediately, as it depends on instance admins to catch it, and they have other things to do. Also, when an account is banned, the Mastodon server software is not sending out a "delete" for all of their posted material (which would signal other instances to delete it)
Problem #2 can hopefully be improved with better tooling. I don't know what you do about problem #1, though.
Such a signal exists in the ActivityPub protocol, so I wonder why it's not being used.
4.1 Illustrated and Computer-Generated CSAM
Stopped reading.
Child abuse laws "exclude anime" for the same reason animal cruelty laws "exclude lettuce." Drawings are not children.
Drawings are not real.
Half the goddamn point of saying CSAM instead of CP is to make clear that Bart Simpson doesn't count. Bart Simpson is not real. It is fundamentally impossible to violate Bart Simpson's rights, because he doesn't fucking exist. There is nothing to protect him from. He cannot be harmed. He is imaginary.
This cannot be a controversial statement. Anyone who can't distinguish fiction from real life has brain problems.
You can't rape someone in MS Paint. Songs about murder don't leave a body. If you write about robbing Fort Knox, the gold is still there. We're not about to arrest Mads Mikkelsen for eating people. It did not happen. It was not real.
If you still want to get mad at people for jerking off to the wrong fantasies, that is an entirely different problem from photographs of child rape.
Mastodon is a piece of software. I don't see anyone saying "phpBB" or "WordPress" has a massive child abuse material problem.
Has anyone in the history ever said "Not a good look for phpBB"? No. Why? Because it would make no sense whatsoever.
I feel kind of a loss for words because how obvious it should be. It's like saying "paper is being used for illegal material. Not a good look for paper."
What is the solution to someone hosting illegal material on an nginx server? You report it to the authorities. You want to automate it? Go ahead and crawl the web for illegal material and generate automated reports. Though you'll probably be the first to end up in prison.
I get what you're saying, but due to federated nature, those CSAMs can easily spread to many instances without their admins noticing them. Having even one CSAM in your server is a huge risk for the server owner.
I've thought about building a truly decentralized app similar to lemmy, but the question if how to prevent things like CSAM from ending up on unwitting users' devices is the main thing stopping me.
Lemmy has exactly the same problem, and the solution seems to be to defederate from instances that host that kind of content. That works, but it's a lot of work for an admin, so we absolutely need better moderation tools to help detect unwanted content and block the source of it.
I just wish people wouldn't post such nonsense.
Thats a dumb argument, though.
phpbb is not the host or the provider. Its just something you download and install on your server, with the actual service provider (You, the owner of the server and operator of the phpbb forum) being responsible for its content and curation.
Mastadon/Twitter/social media is the host/provider/moderator.
According to corporate news everything outside of the corporate internet is pedophiles.
Well, terrorists became boring, and they still want the loony wing of the GOP's clicks, so best to back off on Nazis and pro-Russians, leaving pedophiles as the safest bet.
Nazis not being the go-to target for a poisoning the well approach worries me in many different levels
Hasn’t Twitter had the same problem for years?
Seems odd that they mention Mastodon as a Twitter alternative in this article, but do not make any mention of the fact that Twitter is also rife with these problems, more so as they lose employees and therefore moderation capabilities. These problems have been around on Twitter for far longer, and not nearly enough has been done.
The actual report is probably better to read.
It points out that you upload to one server, and that server then sends the image to thousands of others. How do those thousands of others scan for this? In theory, using the PhotoDNA tool that large companies use, but then you have to send the every image to PhotoDNA thousands of times, once for each server (because how do you trust another server telling you it's fine?).
The report provides recommendations on how servers can use signatures and public keys to trust scan results from PhotoDNA, so images can be federated with a level of trust. It also suggests large players entering the market (Meta, Wordpress, etc) should collaborate to build these tools that all servers can use.
Basically the original report points out the ease of finding CSAM on mastodon, and addresses the challenges unique to federation including proposing solutions. It doesn't claim centralised servers have it solved, it just addresses additional challenges federation has.
Yeah I was mostly just complaining about the poor quality of mainstream tech articles. The original report is a much pretty read and brings up some great points.
These articles are written by idiots, serving the whims of a corporate stooge to try and smear any other than corporate services and it isn't even thinly veiled. Look at who this all comes from
Its weird how this headline shows up only when other headlines start covering how popular Mastadon is now.
Coincidence? Sure smells like it. God, I love astroturfing in the morning.
The article written by WaPo and regurgitated by The Verge is crap, but the study from Stanford is solid. However, it's nowhere near as doom and gloom as the articles, and suggests plenty of ways to improve things. Primarily they suggest better tools for moderation.
better tools for moderation
Where have I heard that before?
The study from Stanford conflates pencil drawings of imaginary characters with actual evidence of child rape.
Half the goddamn point of saying CSAM instead of CP is to make that difference blindingly obvious. Somehow, they still missed it. Somehow they are talking about sexual abuse as if it's something that can happen to pixels.
Nothing you can do except go after server owners like usual. Has nothing to do with the fedi. Mastodon has nothing to do with either because anyone can pop up their own alternative server. This is one of many protocols they have or will use to distribute this stuff.
This just in: criminals are using the TCP protocol to distribute CP!!! What can the internet do to stop this? Oh yeah, go after server owners and groups like usual.
Things are a bit complicated in the fediverse. Sure, your instance might not host any pedo community, but if a user on your instance subscribe/interact with those community, the CSAMs might get federated into your instance without you noticing. There are tools to help you combat this, but as an instance owner you can't just assume it's not your problem if some other instance host pedo stuff.
That is definitely alarming, and a downside of the fedi, but seems like a necessary evil. Unfortunately admins and mods of small communties in the fedi will be the ones exposed to this. There has been better methods if handling this though. There are shared block lists out there and they already have lists that block out undesirable stuff like that, so it at least minimizes the amount of innocent eyes of mods, who are just regular unpaid people, from seeing disgusting stuff. Also, obviously those instances should be reported to the police, fbi, or whatever the heck
There is a database of known files of CSAM and their hashes, mastodon could implement a filter for those at the posting interaction and when federating content
Shadow banning those users would be nice too
They are talking about AI generated images. That’s the volume part.
This is one of the reasons I'm hesitant to start my own instance - the moderation load expands exponentially as you scale, and without some sort of automated tool to keep CSAM content from being posted in the first place, I can only see the problem increasing. I'm curious to see if anyone knows of lemmy or mastodon moderation tools that could help here.
That being said, it's worth noting that the same Standford research team reviewed Twitter and found the same dynamic in play, so this isn't a problem unique to Mastodon. The ugly thing is that Twitter has (or had) a team to deal with this, and yet:
“The investigation discovered problems with Twitter's CSAM detection mechanisms and we reported this issue to NCMEC in April, but the problem continued,” says the team. “Having no remaining Trust and Safety contacts at Twitter, we approached a third-party intermediary to arrange a briefing. Twitter was informed of the problem, and the issue appears to have been resolved as of May 20.”
Research such as this is about to become far harder—or at any rate far more expensive—following Elon Musk’s decision to start charging $42,000 per month for its previously free API. The Stanford Internet Observatory, indeed, has recently been forced to stop using the enterprise-level of the tool; the free version is said to provide read-only access, and there are concerns that researchers will be forced to delete data that was previously collected under agreement.
So going forward, such comparisons will be impossible because Twitter has locked down its API. So yes, the Fediverse has a problem, the same one Twitter has, but Twitter is actively ignoring it while reducing transparency into future moderation.
If you run your instance behind cloudlare, you can enable the CSAM scanning tool which can automatically block and report known CSAMs to authorities if they're uploaded into your server. This should reduce your risk as the instance operator.
https://developers.cloudflare.com/cache/reference/csam-scanning/
I think the common sense solution is creating instances for physically local communities (thus keeping the moderation overhead to a minimum) and being very judicious about which instances you federate your instance with.
That being said, It's only a matter of time before moderation tools are created for streamlining the process.
My instance is for members of a certain group, had to email the owner a picture of your card to get in. More instances should exist like that. General instances are great but it's nice knowing all the people on my local are in this group too.
Nah, not intimidated. More that I ran a sizeable forum in the past and I know what what a pain in the ass this kind of content can be to deal with. That's why I was asking about automated tools to deal with it. The forum I ran got targeted by a bunch of Turkish hackers, and their one of their attack techniques involved a wave of spambot accounts trying to post crap content. I wasn't intimidated (fought them for about two years straight), but by the end of it I was exhausted to the point where it just wasn't worth it anymore. An automated CSAM filter would have made a huge difference, but this was over a decade ago and those tools weren't around.
Direct link to the (short) report this article refers to:
https://stacks.stanford.edu/file/druid:vb515nd6874/20230724-fediverse-csam-report.pdf
https://purl.stanford.edu/vb515nd6874
After reading it, I’m still unsure what all they consider to be CSAM and how much of each category they found. Here are what they count as CSAM categories as far as I can tell. No idea how much the categories overlap, and therefore no idea how many beyond the 112 PhotoDNA images are of actual children.
- 112 instances of known CSAM of actual children, (identified by PhotoDNA)
- 713 times assumed CSAM, based on hashtags.
- 1,217 text posts talking about stuff related to grooming/trading. Includes no actual CSAM or CSAM trading/selling on Mastodon, but some links to other sites?
- Drawn and Computer-Generated images. (No quantity given, possibly not counted? Part of the 713 posts above?)
- Self-Generated CSAM. (Example is someone literally selling pics of their dick for Robux.) (No quantity given here either.)
Personally, I’m not sure what the take-away is supposed to be from this. It’s impossible to moderate all the user-generated content quickly. This is not a Fediverse issue. The same is true for Mastodon, Twitter, Reddit and all the other big content-generating sites. It’s a hard problem to solve. Known CSAM being deleted within hours is already pretty good, imho.
Meta-discussion especially is hard to police. Based on the report, it seems that most CP-material by mass is traded using other services (chat rooms).
For me, there’s a huge difference between actual children being directly exploited and virtual depictions of fictional children. Personally, I consider it the same as any other fetish-images which would be illegal with actual humans (guro/vore/bestiality/rape etc etc).
If we took this to its logical conclusion, most popular games would be banned. How many JRPGs have underage protagonists? How many of those have some kind of love story going on in the background? What about FPS games where you're depicted killing other people? What about fantasy RPGs where you can kill and control animals?
Things should always be legal unless there's a clear victim. And communities should absolutely be allowed to filter out anything they want, even if it's 100% legal. So the lack of clear articulation of the legal issues is very worrisome since it implies a moral obligation to remove legal but taboo content.
Just replace Mastodon with Internet.
There's really no way to completely stop it without giving up all privacy.
I'm not actually going to read all that, but I'm going to take a few guesses that I'm quite sure are going to be correct.
First, I don't think Mastodon has a "massive child abuse material" problem at all. I think it has, at best, a "racy Japanese style cartoon drawing" problem or, at worst, an "AI generated smut meant to look underage" problem. I'm also quite sure there are monsters operating in the shadows, dogwhistling and hashtagging to each other to find like minded people to set up private exchanges (or instances) for actual CSAM. This is no different than any other platform on the Internet, Mastodon or not. This is no different than the golden age of IRC. This is no different from Tor. This is no different than the USENET and BBS days. People use computers for nefarious shit.
All that having been said, I'm equally sure that this "research" claims that some algorithm has found "actual child porn" on Mastodon that has been verified by some "trusted third part(y|ies)" that may or may not be named. I'm also sure this "research" spends an inordinate amount of time pointing out the "shortcomings" of Mastodon (i.e. no built-in "features" that would allow corporations/governments to conduct what is essentially dragnet surveillance on traffic) and how this has to change "for the safety of the children."
How right was I?
Halfway there. The PDF lists drawn 2D/3D, AI/ML generated 2D, and real-life CSAM. It does highlight the actual problem of young platforms with immature moderation tools not being able to deal with the sudden influx of objectional content.
The content in question is unfortunately something that has become very common in recent months: CSAM (child sexual abuse material), generally AI-generated.
AI is now apparently generating entire children, abusing them, and uploading video of it.
Or, they are counting "CSAM-like" images as CSAM.
Of course they're counting "CSAM-like" in the stats, otherwise they wouldn't have any stats at all. In any case, they don't really care about child abuse at all. They care about a platform existing that they haven't been able to wrap their slimy tentacles around yet.
I'm always suspicious if someone argues pro Contents Filter with "protection of children" as the main argument...
As a parent, I'm always worried about any policy with "protect the children" as the main argument. There are lots of stupid policies proposed and sometimes implemented that are justified this way, such as:
- facial recognition to prevent underage kids from playing certain video games
- proof of ID to access social media and porn
- complicated parental controls on devices and services
And so on.
Most of these have easy ways to circumvent these rules and absolutely violate privacy, so I will be teaching my kids how to do that. In fact, once our home Internet gets fast enough, I may route all traffic through a VPN just to avoid most of these stupid rules and instead rely on trust with my kids to keep them safe on the Internet.
So what im reading is they didnt actually look at any images, they found hashtags, undisclosed hashtags at that. So basically we've no idea what they think they found, for all we know cartoon might've been one of the tags
Don't bother with the bullshit clickbait article. Honestly, don't give them the views.
The underlying study is good though, and worth reading.
This seems like a very normal thing with all social media. Now if the server isn't banning and removing the content within a reasonable amount of time then we have major issues.
Seems like if you talk about Mastodon but not Twitter or Facebook in the same post it makes it feel like one is greater than the others. This article seems half banked to get clicks.
Is this Blahaj.zone admin "child abuse material" or actual child abuse material?
Or maybe it's better to err on the side of caution when it comes to maybe one of the worst legal offences you can do?
I'm tired of people harping on this decision when it's a perfectly legitimate one from a legal standpoint. There's a reason tons of places are very iffy about nsfw content.
I know that people like to dump on Cloudflare, but it's incredibly easy to enable a built-in CSAM scanner with CloudFlare.
On that note, I'd like to see built-in moderation tools using something like PDQ and TMK+PDQF and a shared hashtable of CSAM and other material that may be outlawed or desirable to filter out in different regions (e.g. terrorist content, Nazi content in Germany, etc.)
I don't want much, I just want deletion to be propagated reliably across the fediverse. If someone got banned for CSAM and their contents purged, I want those action propagated across all federated instances. I can't even delete my comment reliably here on Lemmy since many instances doesn't seem to get the deletion requests.
Wait, why do people like to dump on CloudFlare? I must be out of the loop.
People are wary about how internet got more and more centralized behind cloudlare. If you're ever getting caught in cloudlare's captcha hell because they flag your IP as suspicious, you'll get wary too because you suddenly realized how big cloudlare now when half of the internet suddenly ask you to solve cloudlare captcha.
Lemmy.world had to start using CloudFlare because some script kiddies were DDOSing it. Some people were complaining that it encourages centralization, etc.
Personally, I love it. The service you get even at the lowest level of payment ($20/mo) is great. And what you get for free can't be compared.
Does the CSAM scanner hook into lemmy properly though?
It looks like it scans and flags on the outbound (user download of the image), so as long as it sits in front of your instance, it should work just fine.
You're still responsible for removing the material, complying with any preservation requirements, and any other legal obligations, and notifying CloudFlare that it's been removed.
It would be ideal if it could block on upload, so the material never makes it to your instance, but that would likely be something else like integration with PhotoDNA or something similar.
I bet theres more CP hosted by Bing.
Tell them to look into their government and politicians.
Isn't Peter Thiel the evil one?
This is a serious problem... True or false either way....
The article points out that the strength of the Fediverse is also it’s downside. Federated moderation makes it challenging to consistently moderate CSAM.
We have seen it even here with the challenges of Lemmynsfw. In fact they have taken a stance that CSAM like images with of age models made to look underage is fine as long as there is some dodgy ‘age verification’
The idea is that abusive instances would get defederated, but I think we are going to find that inadequate to keep up without some sort of centralized reporting escalation and ai auto screening.
The problem with screening by AI is there's going to be false positives, and it's going to be extremely challenging and frustrating to fight them. Last month I got a letter for a speeding infraction that was automated: it was generated by a camera, the plate read in by OCR, the letter I received (from "Seat Pleasant, Maryland," lol) was supposedly signed off by a human police officer, but the image was so blurry that the plate was practically unreadable. Which is what happened: it got one of the letters wrong, and I got a speeding ticket from a town I've never been to, had never even heard of before I got that letter. And the letter was full of helpful ways to pay for and dispense with the ticket, but to challenge it I had to do it it writing, there was no email address anywhere in the letter. I had to go to their website and sift through dozens of pages to find one that had any chance of being able to do something about it, and I made a couple of false steps along the way. THEN, after calling them up and explaining the situation, they apologized and said they'd dismiss the charge--which they failed to do, I got another letter about it just TODAY saying a late fee had now been tacked on.
And this was mere OCR, which has been in use for multiple decades and is fairly stable now. This pleasant process is coming to anything involving AI as a judging mechanism.
THEN, after calling them up and explaining the situation, they apologized and said they'd dismiss the charge--which they failed to do
That sounds about right. When I was in college I got a speeding ticket halfway in between the college town and the city my parents lived in. Couldn't afford the fine due to being a poor college student, and called the court and asked if an extension was possible. They told me absolutely, how long do you need, and then I started saving up. Shortly before I had enough, I got a call from my Mom that she had received a letter saying there was a bench warrant for my arrest over the fine
How do you plan to train the AI to recognise CP?