Crashdoom @pawb.social Network Administrator for Pawb.Social, furry, and a programmer
Mastodon: @[email protected]
[RFC] Maintenance Windows
Given the size the community has grown to and concerns around our current approach to maintenance, I’d like to get some feedback on when we should look to schedule maintenance going forward.
There’s two cases covered by the poll, so please choose the all options that fit your preferences.
Case 1: Non-interactive Maintenance This covers any maintenance where we don’t actively need to monitor the upgrade process; Copying data from one location to another, etc. (Usually, this could be left overnight and brought back up in the morning)
Case 2: Interactive Maintenance This covers any maintenance that requires us to perform a series of actions and continually monitor the process to ensure no issues; Upgrading Mastodon or Lemmy, database migration, or upgrading the Kubernetes cluster. (Usually, this would need to occur during the day due to time constraints)
To vote, please find the corresponding pinned comments and upvote them!
If you’ve got any questions, concerns, etc. please leave them below and we’ll get back to them asap!
Apologies for the delay. We're working on this over the course of the weekend trying to restore the emojis.
So, the issue lays in that there's no technical way to notify the remote user (someone not on furry.engineer or pawb.fun) that they've been suspended on our end, without sending a message to them directly. If we suspend them on our end, that doesn't per se suspend them on their end and they wouldn't know that their messages were no longer reaching our users; They would still be able to message other users on their instance, and users on other instances, but not to our users.
We're apprehensive about notifying remote accounts specifically because we don't often know the moderation practices of the remote instance (to know if they'll deal with it, or if they have open-registration allowing anyone to join without approval) and it may encourage further abusive behavior through ban evasions (creating new accounts on that instance or elsewhere to continue messaging) from the user being made aware that we're no longer receiving their messages.
Appreciate the feedback so far, let me try to see if I can answer most / many of the questions:
What are the risks of #4?
Many users are worried about the risk of automated actions going wrong and not knowing what we mean with "pattern of spammy behavior."
For how we would identify the pattern of behavior that would allow for automated actions, we would review any major spam wave, such as the one we've been experiencing over the past few days:
We would then identify any indicators we could use that are indicative of the known spam, and create a heuristic ruleset that would limit or suspend those accounts while targeting only those accounts actively engaging in the spam, not just referring to it. There are additional safeguards we can add, such as preventing rules being applied to users where the user is followed by someone on our instances.
For the risk of automated actions going wrong, if we were using a limit (not a suspend) then the account would be hidden from public view but could still be viewed if specifically searched by name, it would also suppress all notifications from that user unless they are followed by you. (e.g. if they messaged you out of the blue, you wouldn't see it if you weren't following them.)
If a suspend was used, the account would be marked for deletion from our instances but all follower relationships would immediately break (e.g. if you were following them, the system would automatically unfollow when they are suspended). Typically, we can restore data within 30 days, but follower relationships are typically unrecoverable. So long as rules are appropriately limited in scope to only target those with a lot of spam indicators, no false-positives should occur.
What about appeals?
For local users (anyone registered on furry.engineer and pawb.fun), all actions against your account (except reports) can be appealed. If you have a post removed or are suspended, all actions can be appealed directly to the admin team.
For remote users, we can remove restrictions on remote accounts if we receive an appeal from any of our users, or by the affected account directly. This can be done via email, or just through a DM to one of the admins who can pass it to the team.
Would the AI model have oversight?
Yes. Where the team believe the filter has flagged sufficient content appropriately and maintains no false-positives, we may promote a model or ruleset to allowing automated actions (limit / suspend).
We'll keep an eye on the actions of each ruleset by reviewing the daily / weekly actions taken to ensure they meet the criteria and have not misidentified any users or content, and we'll also start publicly tracking the statistics of the models / rulesets we create and use, including a count of false-positives or reversed decisions.
Will you notify users?
Due to limitations in Mastodon, we can only notify local users (users on furry.engineer or pawb.fun) when actions are taken against their account; This process happens automatically when your post is removed, or your account is warned, limited, or suspended.
There's no easy way to notify remote users other than sending them a DM, but doing so could be seen as spammy or lead to inciting further abusive behavior by informing them of our activity. While we can have transparency with our users due to having an invite-only platform, other instances are frequently open-registration which can allow the abusive user to re-create an account to continue to harass our users. BUT, I'm open to suggestions on this.
#2: I've had some light experience before specifically with TensorFlow Lite models during my degree program. For the Coral Edge TPU, we wanted to off-load the processing to try to get the speed as near to zero latency as possible, though admittedly, it would potentially be superfluous. I'm also looking into some existing models I could potentially use but hadn't found any that particularly stood out, but if anyone has any recommendations I'd love to check them out!
#3: Good question; If the system flags a post automatically as potentially spam, and the team determine it's not spam, I would probably like to be able to train on that message as "ham" / not-spam to avoid future false positives. But, that would be an extension of the scope of what we'd train on, so I'd very much like feedback on that too.
#4: Yes, when a user is limited the profile will show a content warning before the contents of the profile. I believe the prompt is something like "This user has been hidden by the moderators of [instance name]". For repeated mis-identifications, yes, edge-cases like this we could approve the user and exempt them from future automated reports.
[RFC] Use of Automated Moderation Tools
Due to the recent spam waves affecting the Fediverse, we'd like to open requests for comment on the use of automated moderation tools across Pawb.Social services.
We have a few ideas on what we'd like to do, but want to make sure users would feel comfortable with this before we go ahead with anything.
For each of these, please let us know if you believe each use-case is acceptable or not acceptable in your opinion, and if you feel like sharing additional info, we'd appreciate it.
---
1. Monitoring of Public Streaming Feed
We would like to set up a bot that monitors the public feed (all posts with Public visibility that appears in the Federated timeline) to flag any posts that meet our internally defined heuristic rules.
Flagged posts would be reported per normal from a special system-user account, but reports would not be forwarded to remote instances to avoid false-positives.
These rules would be fixed based on metadata from the posts (account indicators, mentions, links, etc.), but not per-se the content of the posts themselves.
2. Building of a local AI spam-detection model
Taking this a step further, we would like to experiment with using TensorFlow Lite and Google Coral Edge TPUs to make a fully local model, trained on the existing decisions made by our moderation team. To stress, the model would be local only and would not share data with any third party, or service.
This model would analyze the contents of the post for known spam-style content and identifiers, and raise a report to the moderation team where it exceeds a given threshold.
However, we do recognize that this would result in us processing posts from remote instances and users, so we would commit to not using any remote posts for training unless they are identified as spam by our moderators.
3. Use of local posts for non-spam training
If we see support with #2, we'd also like to request permission from users on a voluntary basis to provide as "ham" (or non-spam / known good posts) to the spam-detection model.
While new posts would be run through the model, they would not be used for training unless you give us explicit permission to use them in that manner.
I'm hoping this method will allow users who feel comfortable with this to assist in development of the model, while not compelling anyone to provide permission where they dislike or are uncomfortable with the use of their data for AI training.
4. Temporarily limiting suspected spam accounts
If our heuristics and / or AI detection identify a significant risk or pattern of spammy behavior, we would like to be able to temporarily hide / suppress content from the offending account until a moderator is able to review it. We've also suggested an alternative idea to Glitch-SOC, the fork we run for furry.engineer and pawb.fun, to allow hiding a post until it can be reviewed.
Limiting the account would prevent anyone not following them from seeing posts or mentions by them, until their account restriction is lifted by a moderator.
In a false-positive scenario, an innocent user may not have their posts or replies seen by a user on furry.engineer / pawb.fun until their account restriction is lifted which may break existing conversations or prevent new ones.
---
We'll be leaving this Request for Comment open-ended to allow for evolving opinions over time, but are looking for initial feedback within the next few days for Idea #1, and before the end of the week for ideas #2 through #4.
We're aware of this issue and are working to resolve it. Yesterday, the last of the files from Cloudflare R2 transferred and we're working on getting the files backed up before removing unnecessary files, and restoring the missing files to the local storage. We're needing to do the removal of unnecessary files as there are ~6TB of stored media, but only 1TB of used media between furry.engineer and pawb.fun.
When it next occurs, could you email [email protected] and attach a screenshot of the error page? If it happens in an app, could you let me know what action you're trying to do and I'll see if I can pin it down in the logs! :3
@[email protected] @[email protected] @[email protected] Does this still appear to be occurring for you? We were having technical issues over the past day but they appear to be resolved now.
Is this working again? We were experiencing some networking issues, but they appear to be resolved (https://phiresky.github.io/lemmy-federation-state/site?domain=pawb.social).
Edit: Does appear that we're lagging a little on out-going federation activities due to the volume of incoming requests. I'll look into setting up horizontal scaling for Lemmy in the coming days once we've finished setting up the final node in the Kubernetes cluster! We're currently hamstrung for resources on the two servers we have.
[Feb 14] Pawb.Social Migration Bugs / Issues Megathread
On Feb 14th we migrated Lemmy from its standalone Docker setup to the same Kubernetes cluster operating furry.engineer and pawb.fun, discussed in https://pawb.social/post/6591445.
As of 5:09 PM MT on Feb 14th, we are still transferring the media to the new storage, which may result in broken images. Please do still reply to this thread if your issue is media related, but please check again after a few hours and edit your comment to say "resolved" if it's rectified by the transfer.
As of 11:02 AM MT on Feb 15th, we have migrated all media and are awaiting the media service coming back online and performing a hash check of all files. Once this is completed, uploads should work per normal.
----
To make it easier for us to go through your issues, please include the following information:
- Time / Date Occurred
- Page URL where you encountered the issue
- What you were trying to do at the time you encountered the issue
- Any other info you think might be important / relevant
Status update
Both instances are back online again! We're currently transferring cached media from remote instances to the local storage, so avatars, emojis, and older attachments may currently appear as broken images.
As of Feb 7th at 10:45 PM Mountain Time, pawb.fun has re-generated all feeds, while furry.engineer is continuing with an estimated 25 minutes to go. We're also re-generating the ElasticSearch indicies which power the full-text search system and expect that to continue through the night.
[Maintenance] Feb 7 - Mastodon Data Migration
tl;dr summary furry.engineer and pawb.fun will be down for several hours this evening (5 PM Mountain Time onward) as we migrate data from the cloud to local storage. We'll post updates via our announcements channel at https://t.me/pawbsocial.
----
In order to reduce costs and expand our storage pool, we'll be migrating data from our existing Cloudflare R2 buckets to local replicated network storage, and from Proxmox-based LXC containers to Kubernetes pods.
Currently, according to Mastodon, we're using about 1 TB of media storage, but according to Cloudflare, we're using near 6 TB. This appears to be due to Cloudflare R2's implementation of the underlying S3 protocol that Mastodon uses for cloud-based media storage, which is preventing Mastodon from properly cleaning up no longer used files.
As part of the move, we'll be creating / using new Docker-based images for Glitch-SOC (the fork of Mastodon we use) and hooking that up to a dedicated set of database nodes and replicated storage through Longhorn. This should allow us to seamlessly move the instances from one Kubernetes node to another for performing routine hardware and system maintenance without taking the instances offline.
We're planning to roll out the changes in several stages:
-
Taking furry.engineer and pawb.fun down for maintenance to prevent additional media being created.
-
Initiating a transfer from R2 to the new local replicated network storage for locally generated user content first, then remote media. (This will happen in parallel to the other stages, so some media may be unavailable until the transfer fully completes).
-
Exporting and re-importing the databases from their LXC containers to the new dedicated database servers.
-
Creating and deploying the new Kubernetes pods, and bringing one of the two instances back online, pointing at the new database and storage.
-
Monitoring for any media-related issues, and bringing the second instance back online.
We'll be beginning the maintenance window at 5 PM Mountain Time (4 PM Pacific Time) and have no ETA at this time. We'll provide updates through our existing Telegram announcements channel at https://t.me/pawbsocial.
During this maintenance window, furry.engineer and pawb.fun will be unavailable until the maintenance concluded. Our Lemmy instance at pawb.social will remain online, though you may experience longer than normal load times due to high network traffic.
----
Finally and most importantly, I want to thank those who have been donating through our Ko-Fi page as this has allowed us to build up a small war chest to make this transfer possible through both new hardware and the inevitable data export fees we'll face bringing content down from Cloudflare R2.
Going forward, we're looking into providing additional fediverse services (such as Pixelfed) and extending our data retention length to allow us to maintain more content for longer, but none of this would be possible if it weren't for your generous donations.
Lemmy v0.19.3
We've updated to Lemmy v0.19.3!
For a full change log, see the updates below:
Major changes
Improved Post Ranking
There is a new scaled sort which takes into account the number of active users in a community, and boosts posts from less-active communities to the top. Additionally there is a new controversial sort which brings posts and comments to the top that have similar amounts of upvotes and downvotes. Lemmy’s sorts are detailed here.
Instance Blocks for Users
Users can now block instances. Similar to community blocks, it means that any posts from communities which are hosted on that instance are hidden. However the block doesn’t affect users from the blocked instance, their posts and comments can still be seen normally in other communities.
Two-Factor Auth Rework
Previously 2FA was enabled in a single step which made it easy to lock yourself out. This is now fixed by using a two-step process, where the secret is generated first, and then 2FA is enabled by entering a valid 2FA token. It also fixes the problem where 2FA can be disabled without passing any 2FA token. As part of this change, 2FA is disabled for all users. This allows users who are locked out to get into their account again.
New Federation Queue
Outgoing federation actions are processed through a new persistent queue. This means that actions don’t get lost if Lemmy is restarted. It is also much more performant, with separate senders for each target instance. This avoids problems when instances are unreachable. Additionally it supports horizontal scaling across different servers. The endpoint /api/v3/federated_instances contains details about federation state of each remote instance
Remote Follow
Another new feature is support for remote follow. When browsing another instance where you don’t have an account, you can click the subscribe button and enter the domain of your home instance in the popup dialog. It will automatically redirect you to your home instance where it fetches the community and presents a subscribe button. Here is a video showing how it works.
Moderation
Reports are now resolved automatically when the associated post/comment is marked as deleted. This reduces the amount of work for moderators. There is a new log for image uploads which stores uploader. For now it is used to delete all user uploads when an account is purged. Later the list can be used for other purposes and made available through the API.
[Defederation] pisskey.io
- Instance: pisskey.io
- Type: Defederation
- Affects: Pawb.Social, furry.engineer, pawb.fun
- Reason: Nazi imagery, affiliated with poa.st and other known abusive instances
- Fediseer Action: Censured
[Defederation] the.asbestos.cafe
- Instance: the.asbestos.cafe
- Type: Defederation
- Affects: Pawb.Social, furry.engineer, pawb.fun
- Reason: Homophobia, harassment, trolling, admin involved abuse
- Fediseer Action: Censured
Block has been applied to the entire domain.
Evidence
- https://the.asbestos.cafe/objects/14decba5-b6fb-4c20-b8d7-8cf68374ec58
- https://the.asbestos.cafe/objects/094cfc21-7182-471a-90cc-a493fb126635
- https://the.asbestos.cafe/objects/764af9ee-288c-4151-aa41-b2e295dc95cc
- https://the.asbestos.cafe/objects/3f5960bb-8fc1-41b0-b180-7748cd0e15b3
- https://the.asbestos.cafe/objects/bee81dc2-85ee-45ae-b1b6-39f2d2d4dd6f
[Defederation] nyanide.com
- Instance: lab.nyanide.com
- Type: Defederation
- Affects: Pawb.Social, furry.engineer, pawb.fun
- Reason: Trolling, harassment, homophobia, nazi imagery, admin / mod engaged abuse
- Fediseer Action: Censured
Block has been applied to the entire domain.
cunnyborea.space
cross-posted from: https://pawb.social/post/3393854
> - Instance: cunnyborea.space > - Type: Defederation > - Affects: Pawb.Social, furry.engineer, pawb.fun > - Reason: Racism, antisemitism, homophobia, abusive admin, nazi imagery > - Fediseer Action: Censured > > ::: spoiler Evidence > - https://i.imgur.com/HgvMVOh.png > - https://union.place/@inquiline/111247251661793670 > :::
[Defederation] cunnyborea.space
- Instance: cunnyborea.space
- Type: Defederation
- Affects: Pawb.Social, furry.engineer, pawb.fun
- Reason: Racism, antisemitism, homophobia, abusive admin, nazi imagery
- Fediseer Action: Censured
Evidence
- https://i.imgur.com/HgvMVOh.png
- https://union.place/@inquiline/111247251661793670
bv.umbrellix.org
cross-posted from: https://pawb.social/post/3337642
> - Instance: bv.umbrellix.org > - Type: Defederation > - Affects: Pawb.Social, furry.engineer, pawb.fun > - Reason: Toxicity, abusive admin, death threats, emotional abuse > - Fediseer Action: Censured > > ::: spoiler Evidence > - (CW: Abuse) https://bv.umbrellix.org/notice/AZwi9Is6mxT9vo5TSi > - https://sunbeam.city/@[email protected]/111142509643025163 > ::: > > Admin Note: This block applies to the root domain (umbrellix.org) and all associated sub-domains.
[Defederation] bv.umbrellix.org
- Instance: bv.umbrellix.org
- Type: Defederation
- Affects: Pawb.Social, furry.engineer, pawb.fun
- Reason: Toxicity, abusive admin, death threats, emotional abuse
- Fediseer Action: Censured
Evidence
- (CW: Abuse) https://bv.umbrellix.org/notice/AZwi9Is6mxT9vo5TSi
- https://sunbeam.city/@[email protected]/111142509643025163
Admin Note: This block applies to the root domain (umbrellix.org) and all associated sub-domains.
[Maintenance] Oct 8th Uplink infrastructure change
Currently, we’re running the Ubiquiti Dream Machine directly as the modem via PPPoE, but there appears to be an intermittent issue with the software implementation that results in periodic downtimes of a few minutes while it reconnects.
We’re looking at switching this back out with the ISP provided router in pass through mode to negate the PPPoE connectivity drop.
We don’t expect this to take longer than 1 hour to switch over and test for reliability before bringing the services back up.
We’ll be performing this maintenance around 11 AM US Mountain Time, and will provide updates via the Telegram channel at https://t.me/pawbsocial.
liberdon.com
cross-posted from: https://pawb.social/post/3077174
> - Instance: liberdon.com > - Type: Defederation > - Affects: Pawb.Social, furry.engineer, pawb.fun > - Reason: Harassment, trolling, misinformation, sexism, homophobia > - Fediseer Action: Censured > > ::: spoiler Evidence > - https://liberdon.com/@ghast > - https://liberdon.com/@ghast/111178732016951128 > - https://liberdon.com/@rimugu/111178647646581863 > - https://liberdon.com/@aristoh4ck8r/111177188127718387 > - https://liberdon.com/@gunkslinger/111175171054509579 > - https://liberdon.com/@TylerAbeoJordan/111174105820298526 > - https://liberdon.com/@TylerAbeoJordan/111174078685171464 > - https://liberdon.com/@TylerAbeoJordan/111173743376687580 > - https://liberdon.com/@medicineman9/111173062104385765 > - https://liberdon.com/@spinneria/111169619274938220 > - https://liberdon.com/@nosat/111165212240469156 > ::: >
[Unexpected Downtime Postmortem] Oct 6th
One of the data storage systems (CEPH) encountered a critical failure when Proxmox lost connection to several of its nodes, ultimately resulting in the CEPH configuration being cleared by the Proxmox cluster consensus mechanism. No data, except ElasticSearch, was stored on CEPH.
When the connection was lost to the other nodes, a split-brain occurred (when nodes disagree on which changes are authoritative and which should be dropped). As we tried to recluster all of the nodes, a resolution occured that resulted in the ceph.conf file being wiped and the data on CEPH being unrecoverable.
Thankfully, we’ve suffered no significant data loss, with the exception of having to rebuild the Mastodon ElasticSearch indexes from 6 AM this morning to present.
I’d like to profusely apologize for the inconvenience, but we felt it necessary at the time to offline all services as part of our disaster recovery plan to ensure no damage occurred to the containers while we investigated.
[Defederation] liberdon.com
- Instance: liberdon.com
- Type: Defederation
- Affects: Pawb.Social, furry.engineer, pawb.fun
- Reason: Harassment, trolling, misinformation, sexism, homophobia
- Fediseer Action: Censured
Evidence
- https://liberdon.com/@ghast
- https://liberdon.com/@ghast/111178732016951128
- https://liberdon.com/@rimugu/111178647646581863
- https://liberdon.com/@aristoh4ck8r/111177188127718387
- https://liberdon.com/@gunkslinger/111175171054509579
- https://liberdon.com/@TylerAbeoJordan/111174105820298526
- https://liberdon.com/@TylerAbeoJordan/111174078685171464
- https://liberdon.com/@TylerAbeoJordan/111173743376687580
- https://liberdon.com/@medicineman9/111173062104385765
- https://liberdon.com/@spinneria/111169619274938220
- https://liberdon.com/@nosat/111165212240469156
Migration of Pawb.Social media
We've migrated all of the Pawb.Social (Lemmy) media from within the container to the new local NAS.
We'll be performing the same updates for pawb.fun and furry.engineer over the course of the next week, minimizing downtime where possible.
We will need to take the services offline once the initial transfer is completed to perform a final check for any new data, and transition to the local NAS, but we'll post an announcement prior to the final maintenance for each instance.
These changes are to allow us longer and cheaper retention of remote and local media across all of our services, and to allow us to begin backing up media in case of a catastrophic failure.
[Defederation] asimon.org
- Instance: asimon.org
- Type: Defederation
- Affects: Pawb.Social, furry.engineer, pawb.fun
- Reason: Racism, homophobia, transphobia, trolling, abusive admin
- Fediseer Action: Censured
Evidence
https://asimon.org/@coin/posts/Aa8tebnT4kYwTVwrwW https://asimon.org/@coin/posts/AZxmCGq8YxrIx5B6UC https://asimon.org/objects/eda7faae-454b-46f7-a309-94066c261ce6 https://asimon.org/@coin/posts/AZKA8C9YIF3zLb2WLA
Changes to Mastodon search and you
With the upgrade to Mastodon 4.2.0, the way the search system works is changing to allow more broad searches across posts made across the fediverse.
Cutting to the chase: Full-text search across opted-in posts will be possible in the next few days.
----
In more depth, the system will allow you to search across your own posts and posts you've interacted with from others irrespective of their opt-in status for full-text search, but will additionally allow you to search ANY posts made by users that opt-in to allow their posts to be indexed.
To that effect, a new setting was added to the Public Profile > Privacy and Reach page called "Include public posts in search results" which is disabled by default. Turning this on will allow any user across the Fediverse to search and find your posts using full-text search.
Shortcuts:
- furry.engineer users click here
- pawb.fun users click here
- For everyone else: There's Mastercard
(Sorry not sorry xD)But, just take your instance's domain and add/settings/privacyto the end (e.g.https://INSTANCE.TLD/settings/privacy)
To allow our users to decide if this is a feature they want to use on their posts going forward, we won't be enabling ElasticSearch until Friday Sept. 22.
----
Additionally, new keywords will be added to the search system:
from:mebefore:2022-11-01,after:2022-11-01,during:2022-11-01language:frhas:pollin:libraryfor searching only in posts you have written or interacted with
These will be enabled on Friday along with the rest of the changes to the search system.
----
If you've got any questions, please let us know!