meta
-
Server maintenance today, Friday April 26th 2024 @ ~11 AM EST
I'll be doing some server maintenance today around 11 AM EST. You will be unable to upload images to fanaticus until the maintenance is finished.
For the technical, I'll be (finally) migrating the pict-rs data to object storage. I was lazy last fall and simply upgraded the server size (and therefore the volume size) instead of doing the proper thing and migrating. This is bad for two reasons:
- It's not scalable -- volume storage is limited and requires regular updates to increase the size of the volume. Object storage automatically scales. You pay for what you use and that's it
- Our server is way beefier than it needs to be, soley so our volume size can be large enough to store all our images. That means I pay more for server power we just don't need. I knew this was an issue when I kicked the can down the road last fall but my life got busy and I needed to just buy myself some time.
I'll be following these docs:
- https://git.asonix.dog/asonix/pict-rs/#user-content-filesystem-to-object-storage-migration
I'm not sure how long the migration will take, I'm guessing a long time considering how large the volume is. With any luck, we'll be back up and running within the hour.
-
Fanaticus Added to Lemmy.World's Blocklist
Hi all, it was brought to my attention that Lemmy.World is not federating with Fanaticus as of about a week ago. I did a little digging and found that we were recently added to Lemmy.World's blocklist.
I've reached out to the admins on multiple channels and am hoping to resolve this soon. Sorry for the inconvenience!
-
Maintenance Upgrade at 1:30 EST - 0.19.0 Released
Fanaticus will be upgrading to 0.19.0 at 1:30 EST. According to the release post, this will likely take 30 minutes.
Check out the new features here.
-
How do game bots work?
Not a tech guy, can't code or anything. Just created a Real Madrid community here and would love to deploy one (pls join if you’re a football fan or curious).
Idk if I can do it but i'd love to hear how it's done.
-
How defederated is this instance?
Wondering how many instances fanaticus has defederated from or has been defederated from simce I haven't found fanaticus communities in my main account and similarly don't see many other here.
-
Script for Users to Block Fanaticus's Game Bot Threads
We understand that some users in the fediverse don't want to see sports content and are frustrated with Fanaticus's game bots' daily posts.
Currently, lemmy doesn't support blocking entire instance's at a user level. Because of that limitation, I am providing a list of the current communities on Fanaticus that have active game bots and a little python script so users can block those communities.
Here's a list of the communities that currently have a game bots running:
[email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected]
If you're comfortable with coding, there's an API endpoint to block a community. I threw together a little python script that would block all the above (I think 🙂).
``` from plemmy import LemmyHttp
must include protocol e.g. https://lemmy.world
pl = LemmyHttp("INSTANCE_NAME_HERE")
username = "USERNAME_HERE" password = "PASSWORD_HERE" pl.login(username, password)
fanaticus_communities = [ "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", "[email protected]", ]
for comm in fanaticus_communities: print("Getting community: ", comm) commRes = pl.get_community(name=comm).json() if commRes and commRes['community_view']: commId = commRes['community_view']['community']['id'] print("Blocking community: ", comm, " communityId: ", commId) pl.block_community(True, commId) print("Successfully blocked community: ", comm) else: print("Failed to find community: ", comm) ```
---
In the future, we are planning on adding the game bots for the other major sports as well (they’re not in season now) so I will update this script as those bots come online.
-
How can I set up the game thread bot?
I’d love to set that up for football games for /c/scgamecocks if possible!
-
Game Bots Get an Upgrade
Today is my first free Saturday since mid summer. With that time, I was able to fully migrate our game bots to their dev server and upgrade them to our latest version of code.
This should improve the stability of fanaticus as well as the bots.
-
Braves bot seems to be broken
@[email protected] - are you able to reboot it? Ive been MIA for a bit since lemmy.world was having issues connecting but I made a new account which seems to be working.
-
Federation issue between lemmy.world and fanaticus
Hi, everyone! A few of you have brought to my attention that lemmy.world is not receiving new posts from fanaticus and vice versa.
I've been debugging it and haven't been able to resolve the issue. Fanaticus' nginx logs are reporting that all requests incoming from lemmy.world are returning a 400. Additionally, we've found that outbound requests to lemmy.world from fanaticus are being responded to with a CloudFlare page.
I am reaching out to the lemmy.world admins to see if we can get fanaticus's IP whitelisted on their CF config. I'll update this thread with any news.
-
Maintenance Upgrade at 20:00 EST - 0.18.4 Released
Well we just updated to 0.18.3 and a new one was released today. We'll update tonight.
-
Maintenance Upgrade At 8:15 EST
Sorry for the delay! It's been a wild month, lots of vacation :)
We're going to be updating to 0.18.3 in 15 minutes. The upgrade should take 5 minutes or so.
-
Missing gameday threads
Not the end of the world, but I don't see a pre or live game thread for twins v mariners on either side (7/26)
-
old.reddit-look for Lemmy
@headie_sage, I don’t know if you’ve seen this and I don’t know if we could/would want to implement this on fanaticus.social, but I thought this was an interesting post.
cross-posted from: https://lemmy.world/post/1941511
> As requested by some users: 'old' style now accessible via https://old.lemmy.world > > Code can be found here: https://github.com/rystaf/mlmym , created by Ryan (Is he here?) (Yes he appears to be! @[email protected] ! Thanks for this awesome front-end!)
-
Suggestion - daily MLBPickle/Immaculate Grid posts
For a little bit there was someone regularly posting their Pickle/IG results each day, but it seems like that's fallen off. Would it be possible or desirable to have a bot post whenever those roll over?
-
Maintenance Upgrade At 10:00 EST
We'll be upgrading to 0.18.2. Downtime should be minimal. There's no nginx configuration changes in this one.
-
Early July Game Bot Update
Hello everyone! I'm back from vacation (was gone the past two weeks) and am ready to dive back into work on the game bots.
I just wanted to give everyone a quick update. You can check out my previous update thread here.
Status
- All 30 teams have their own game bots configured and activated
- 29 of the game bots run on the same server as fanaticus' lemmy code (big beefy box) with Cerevant running the phillies bot.
- The post limit has been increased in 0.18.1 to 50k from 10k so we no longer have a truncation issue
- The sidebar bot has been ported to lemmy but unfortunately is a bit more limited that we are used to (it only updates the standings of a single division) -- check out what it looks like here
- Cerevant has [identified an issue](Reported: https://github.com/LemmyNet/lemmy/issues/3544) where pinned posts are not being federated correctly on foreign instances
- There is an issue with the game bots that has occasionally taken down the fanaticus servers (I'll discuss this more below)
Current Issue
Game Bots Out of Control
> There is an issue with the game bots that has occasionally taken down the fanaticus servers
Game bot will periodically hit a 503 while updating one of its game threads and enter a loop where it continuously attempts to update the post and fail. A bot that enters this state almost immediately spikes one of the server's four CPUs to 100%, then very quickly fills up the shared disk with the same log (GBs of logs). The GBs of logs produced by these rogue bots fill up the server and because the game bots share the same server as fanaticus, they cause the lemmy server to crash.
While I'm not entirely sure what causes this issue, my current hunch is that it's related to a rate limit either with our lemmy or nginx configuration.
When we first started bumping up to the max CPU & memory on our server I upgraded it to a beefy dedicated machine with a quad core and 8GB of RAM. I did not upgrade the shared disk space at that time. The server size was total overkill but I wanted to run the game bots on the same machine.
I've gotten a lot of questions regarding why the game bots were turned off last week and this is the reason. I had to basically babysit them and restart any that ran off the rails and because I was on vacation, I couldn't investigate the bug to fix it or babysit the bots.
Possible Fixes
These are some options I'm tossing around to fix this issue. They're not necessarily going to happen but I'm writing them here publicly so you all know that I'm working on it and to get some public comment.
- Fix the bug, obviously :)
- Move the game bots to another server, separate from fanaticus1
- Create some sort of system that detects the bots are in a runaway state, alerts me, and/or programmatically restarts the bad bots
This is a top priority for me now that I'm back from vacation and I will be working on this starting today.
Future Work
- Add more features to the sidebar bot to achieve feature parity with the former Reddit game bots (team/league leaders, multiple division & playoff standings, live league scoreboards)
- Customizable periodic threads (e.g. a daily "Around the Horn")
- Closer to NBA/NFL/NHL season, we'll port those game bots to lemmy
Contributing
First off, huge shout-out to Cerevant who has been working with me porting the bot to lemmy. He's done amazing work, added new features, fixed multiple bugs, and has been overall a great person to work with! Thank you!!
Another shout out to todrob99 for creating the original redball bots. He's been very responsive and encouraging as Cerevant and I work on the bots.
For any of you who want to contribute, whether it's coding, creating features requests, reporting bugs, or reviewing PRs, you can find my fork of the redball app here.
We are also discussing the work in the #lemmy channel of todrobb's Discord channel here.
Finally, another way to contribute is to take over your community's bot and run it yourself! The code is open source and I can help you set it up on your own server. That would make it easier on me so I don't have to run all the bots and you can configure it to your community's liking.
---
TL;DR bots are live, a little buggy, but are being improved daily!
1 This is definitely the correct thing to do but I'm hesitant to do this right away. I'm not a server admin by trade and this means 1) Extra overhead ($$) and maintenance 2) That I have to take time away from improving and extending the bots' functionality to do server admin work
-
Fanaticus Updated to 0.18.1 🎉 🎉
Thank you to everyone's patience while we waited for the official release of 0.18.1. I know it was tough seeing other instances upgrading to RCs the long wait is over now!
Please report here if you're having any issues with the upgrade. I've tested login, posting, commenting, uploading images, and password reset.
Note: Try Hard Refreshing Your Browser!!
I've heard a lot of comments in the admin chats that a hard refresh will fix a lot of the issues you may encounter so try that first.
-
Maintenance Upgrade At 10:30 EST
cross-posted from: https://fanaticus.social/post/147327
> Fanaticus will be upgrading to 0.18.1 at 10:30 AM EST.
-
Release 0.18.1 🎉 🎉
github.com Release v0.18.1 Release · LemmyNet/lemmyWhat's Changed Remove actix_rt & use standard tokio spawn by @cetra3 in #3158 Add TLS support for diesel-async database connections by @sunaurus in #3189 after 30 days, replace comment.content and...
We'll be updating later today.
-
Lemmy v0.18.1 RC10 looks like a big improvement
lemmy.world Lemmy.world status update 2023-07-05 - LemmyWorldAnother day, another update. More troubleshooting was done today. What did we do: - Yesterday evening @phiresky@[email protected] [https://lemmy.world/u/phiresky] did some SQL troubleshooting with some of the lemmy.world admins. After that, phiresky submitted some PRs to github. - @[email protected]...
Lemmy.world has submitted PRs to fix things that seem to be creating more stability on their instance. I don't know if RC10 will become the official release or how long till the official release is made but v0.18.1 looks like a huge win.
-
Game bots and vacation
Hi all,
Quick update on the game bots. There is some sort of bug where the game bots gets stuck in a broken loop and spike the server CPU (b/c of unlimited requests) until the server crashes. I am on vacation now and unable to debug the issues. For the rest of the weekend, I'll be turning off the game bots in order to ensure the stability of the fanaticus servers.
When I have time, I'll debug and fix the issue but for now, the game bots will go quiet. I apologize for the inconvenience.
-
Lemmy v0.18.0 Release
cross-posted from: https://lemmy.ml/post/1465740
> ## What is Lemmy? > > Lemmy is a self-hosted social link aggregation and discussion platform. It is completely free and open, and not controlled by any company. This means that there is no advertising, tracking, or secret algorithms. Content is organized into communities, so it is easy to subscribe to topics that you are interested in, and ignore others. Voting is used to bring the most interesting items to the top. > > ## Major Changes > > ### HTTP API instead of Websocket > > Until now Lemmy-UI used websocket for all API requests. This has many disadvantages, like making the code harder to maintain, and causing live updates to the site which many users dislike. Most importantly, it requires keeping a connection open between server and client at all times, which causes increased load and makes scaling difficult. That's why we decided to rip out websocket entirely, and switch to HTTP instead. This change was made much more urgent by the sudden influx of new users. @CannotSleep420 and @dessalines have been working hard for the past weeks to implement this change in lemmy-ui. > > HTTP on its own is already more lightweight than websocket. Additionally it also allows for caching of server responses which can decrease load on the database. Here is an experimental nginx config which enables response caching. Note that Lemmy doesn't send any cache-control headers yet, so there is a chance that private data gets cached and served to other users. Test carefully and use at your own risk. > > ### Two-Factor Authentication > > New support for two-factor authentication. Use an app like andOTP or [Authenticator Pro]( > https://f-droid.org/packages/me.jmh.authenticatorpro/) to store a secret for your account. This secret needs to be entered every time you login. It ensures that an attacker can't access your account with the password alone. > > ### Custom Emojis > > Instance admins can add different images as emojis which can be referenced by users when posting. > > ### Other changes > > #### Progressive Web App > > Lemmy's web client can now be installed on browsers that support PWAs, both on desktop and mobile. It will use an instance's icon and name for the app if they are set, making it look like a given instance is an app. > > Note for desktop Firefox users: the desktop version of Firefox does not have built in support for PWAs. If you would like to use a Lemmy instance as a PWA, use use this extension. > > #### Error Pages > > Lemmy's web client now has error pages that include resources to use if the problem persists. This should be much less jarring for users than displaying a white screen with the text "404 error message here". > > #### Route Changes > > Pages that took arguments in the route now take query parameters instead. For example, a link to lemmy.ml's home page with a few options used to look like this: > >
> https://lemmy.ml/home/data_type/Post/listing_type/All/sort/Active/page/1 >
> > The new route would look like this: > >> https://lemmy.ml?listingType=All >
> Note that you now only have to specify parameters you want instead of all of them. > > #### Searchable select redesign > The searchable selects, such as those used on the search page, have a new look and feel. No more inexplicable green selects when using the lightly themes! > > #### Share button > > Posts on the web client now have a share button on supported browsers. This can be used to share posts to other applications quickly and easily. > > #### Lemmy-UI Overall look and feel > > lemmy-ui is now upgraded to bootstrap 5, and every component is now much cleaner. > > Special thanks to sleepless, alectrocute, jsit, and many others for their great work on improving and re-organizing lemmy-ui. > > #### Database optimizations > > Special thanks to johanndt, for suggesting improvements to Lemmy's database queries. Some of these suggestions have already been implemented, and more are on the way. > > Query speed is Lemmy's main performance bottleneck, so we really appreciate any help database experts can provide. > > #### Captchas > > Captchas are not available in this version, as they need to be reimplemented in a different way. They will be back in 0.18.1, so wait with upgrading if you rely on them. > > ## Upgrade instructions > > Follow the upgrade instructions for ansible or docker. > > If you need help with the upgrade, you can ask in our support forum or on the Matrix Chat. > > ## Support development > > We (@dessalines and @nutomic) have been working full-time on Lemmy for almost three years. This is largely thanks to support from NLnet foundation. > > If you like using Lemmy, and want to make sure that we will always be available to work full time building it, consider donating to support its development. No one likes recurring donations, but they've proven to be the only way that open-source software like Lemmy can stay independent and alive. > -
Federation with lemmy.world is broken
https://fanaticus.social/u/fanaticus_admin, https://fanaticus.social/u/headie_sage
I'm unable to subscribe to https://fanaticus.social/c/[email protected] from my account on fanaticus.social
I'm unable to subscribe to https://lemmy.world/c/[email protected] from my account on lemmy.world
You can see that https://lemmy.world/c/hockey is active, but https://fanaticus.social/c/[email protected] shows no activity.
-
Game Bot Bug Crashed Servers
Sorry all, I was at my co-ed softball game and the game day bots filled 25GB of disc in 3 hours.
I'll do a post mortem tomorrow. First thought is that I should absolutely not host the bots on the game servers as the app. I apologize for any inconvenience I caused. It's unacceptable and it won't happen again.
-
Slow Federation
As I am trying to spread awareness of our instance and communities to others, I am running into issues with other instance users being unable to find certain communities. Example:
> https://lemmy.one/comment/281356
While I am able to search communities on their instance just fine from fanaticus.social (in this case, lemmy.one), they are unable to find us and I am also unable to find some of our communities when I use the lemmy.one search function. @headie_sage I don’t know if you have more knowledge in how to resolve this.
!No results for the Nationals community
There are no results for our Nationals community.
!No results for the NFL community, but there is a post to [email protected]
There is not a result for our NFL community, just one of my posts to [email protected].
!There is both a result for [email protected] and a post to [email protected]
The [email protected] community does appear in the results.
-
Game Day Bot Update
Some of you may have noticed there's a game day bot that's been posting to some of your communities. I forked the popular redball bot that's widely used on reddit and have been working to get it compatible with lemmy.
That project is not complete but I plan on putting in a lot of work this weekend in order to squash the bugs and add in the remaining missing features.
Once complete, I'll roll out a standard version of it to all of the baseball communities on fanaticus. That being said, I don't want to be the guy running every baseball community on fanaticus. Anyone who wants to run a community can ask to be a mod and I'll work with them to set up and configure their own baseball bot.