Requirements

• [X] Is this a bug report? For questions or discussions use https://lemmy.ml/c/lemmy_support
• [X] Did you check to see if this issue already exists?
• [X] Is this only a single bug? Do not put multiple bugs in one issue.
• [X] Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Summary

We have a user with a few hundred comments who has crashed the site twice today while trying to delete his account.

This query ends up running for a long time and locks subsequent updates to comment:

UPDATE "comment" SET "content" = $1, "deleted" = $2, "updated" = $3 WHERE ("comment"."creator_id" = $4) RETURNING "comment"."id", "comment"."creator_id", "comment"."post_id", "comment"."content", "comment"."removed", "comment"."published", "comment"."updated", "comment"."deleted", "comment"."ap_id", "comment"."local", "comment"."path", "comment"."distinguished", "comment"."language_id"

This was running for 8 minutes before I killed it. The user in question has 352 comments and 3073 entries in comment_like. This doesn't seem like such a large amount that there should be significant impact from a user deletion.

Steps to Reproduce

I haven't been able to reproduce this with a test user, so far only this one external user keeps causing it on our site.

I've had to disable the /api/v3/user/delete_account URL for now.

Technical Details

Logs are too noisy but this is triggered by a post to /api/v3/user/delete_account from Jerboa

Version

0.18.2

Lemmy Instance URL

lemmy.ca

Requirements

• [X] Is this a bug report? For questions or discussions use https://lemmy.ml/c/lemmy_support
• [X] Did you check to see if this issue already exists?
• [X] Is this only a single bug? Do not put multiple bugs in one issue.
• [X] Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Summary

When on the page of a user on a remote instance, the linked RSS feed returns "Record not found"

Steps to Reproduce

1. Go to the page of a remote user, on a local instance (https://lemmy.world/u/[email protected])
2. Click the RSS button
3. The linked page returns "Record not found" (https://lemmy.world/feeds/u/[email protected])

Technical Details

For example, https://lemmy.world/u/[email protected] links to https://lemmy.world/feeds/u/[email protected] and returns "Record not found"

The page of a local user correctly returns an RSS feed (https://lemmy.ml/u/dessalines links to https://lemmy.ml/feeds/u/dessalines.xml which returns an RSS record)

The RSS feed for a remote community on a local instance returns the same "Record not found". (https://lemmy.world/feeds/c/[email protected])

See lemmy-ui issue (https://github.com/LemmyNet/lemmy-ui/issues/1954) for a related user interface bug.

Version

BE 0.18.2

Lemmy Instance URL

lemmy.ml, lemmy.world, lemmy.ca, etc

Requirements

• [X] Is this a bug report? For questions or discussions use https://lemmy.ml/c/lemmy_support
• [X] Did you check to see if this issue already exists?
• [X] Is this only a single bug? Do not put multiple bugs in one issue.
• [X] Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Summary

When calling api/v3/comment/list you get a list of all comments on that post. Including any removed ones. The only change removed ones have is that the "removed" field is set to true. This is a massive problem because it delegates the obscuring of removed content to the front end. You can view these despite not being being logged in. When I used a mobile app that didn't take into account this removed flag, I was met with some disgusting NSFL imagery I'd rather not have seen.

I strongly recommend obscuring the content of removed comments from the API if the user is not logged in or if the user is not a mod of the community/ not an instance owner. I understand the need to keep this information in the case of reversing moderation decisions and the modlog, but there is zero reason for non-mods and non-admins to have access to it in the normal endpoints. Furthermore, the baton should not be passed to front end developers either. The source of truth should be the backend, and the backend should enforce it.

Finally this should happen with other places where comments are listed and posts are viewed. Comments deleted by the user should not be visible to anyone in the API besides the user and maybe mods/admins. I'm not sure what other endpoints it'll apply to, but in my opinion this is paramount.

Steps to Reproduce

1. Create a post in a community you moderate
2. Create a comment on that post
3. Remove that comment
4. Open up dev tools
5. Go to that post again
6. Look at the http response.

The JSON response has all identifying info removed, and the "removed" flag circled.

!image

Technical Details

n/a

Version

0.18.2

Lemmy Instance URL

No response

Credit to @phiresky for this idea, originally posted in comments of #2994

This PR adds community_id to post_aggregates (& a new index on post_aggregates) to enable joining community directly to post_aggregates when querying posts.

On lemm.ee, this optimization speeds up the query for front page of subscribed posts ~1000x, from several seconds to to just milliseconds. You can check a before/after of query plans here: https://gist.github.com/sunaurus/856e03165bb0c0010505afeebde45230

Requirements

• [X] Is this a feature request? For questions or discussions use https://lemmy.ml/c/lemmy_support
• [X] Did you check to see if this issue already exists?
• [X] Is this only a feature request? Do not put multiple feature requests in one issue.
• [X] Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Is your proposal related to a problem?

at first glance, with lemmy URLs I can't tell the context of a URL off the bat. (for example, which community it is in, which post a comment is a part of).

with the URLs for any community post just being {lemmy domain}/post/{post id} and the URLs for any comment being {lemmy domain}/comment/{comment id}

Describe the solution you'd like.

Changing these URLs to include:

• the community that they were posted in
• the post that they are a comment on

The result would be:

• Posts looking like \{lemmy domain\}/c/\{community name\}/post/\{post id\}
• Comments looking like \{lemmy domain\}/c/\{community name\}/post/\{post id\}/comment/\{comment id\}

Describe alternatives you've considered.

I don't have other ideas for URL formats. This seemed the neatest to me.

Additional context

No response

Requirements

• [X] Is this a feature request? For questions or discussions use https://lemmy.ml/c/lemmy_support
• [X] Did you check to see if this issue already exists?
• [X] Is this only a feature request? Do not put multiple feature requests in one issue.
• [X] Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Is your proposal related to a problem?

Some admins want to manage the amount of storage needed for image uploads and so have instituted limits on image sizes. Examples include lemmy.ee at 100kB (https://lemm.ee/post/25065) and beehaw.org at 4000x4000 (https://github.com/LemmyNet/lemmy/issues/3473#issuecomment-1620520547).

There does not seem to be a way to set, enforce or expose these rules via the API, so these appear to be implemented in NGINX, giving a 413 Payload Too Large error in the case of lemm.ee.

As a result, when a client app is attempting to upload an offending image, all it can currently tell the user is that the image is somehow too large and that they must find out for themself what the rules are.

Describe the solution you'd like.

The ideal would be that the instance handles the required resizing for the user.

If this is not seen as an attractive approach, it would then be extremely helpful for client apps if this information could be exposed via the API, so that they could query this information before attempting an upload and automatically handle the resizing to the instance's requirements on the user's behalf.

My initial inclination is that the bounding-box limit would generally be easier for developers to work with.

Describe alternatives you've considered.

The lemm.ee link above suggests that users should use other image hosts if their files are too large, but this is not an attractive option for app developers or end users, as if either requires the developers to make a choice of third-party host on behalf of their users (which some will doubtless have reasons for disliking) or it requires the users to make choices or take actions which they may not understand. Most users in most cases would rather just see their images resized, perhaps with a note in the app to inform them that this had happened in case they wanted to deal with this differently.

Additional context

No response

This is an urgent test addition to highlight the problem with comment deletes not replicating when a remote-server creates the comment, the home server has no code to replicate delete of comment to all the downstream subscribe servers. Gamma serves as an example of the downstream servers subscribed who are not getting the delete in 0.18.2 version.

The intention here is to put more developer eyes on https://github.com/LemmyNet/lemmy/issues/3625

Requirements

• [X] Is this a feature request? For questions or discussions use https://lemmy.ml/c/lemmy_support
• [X] Did you check to see if this issue already exists?
• [X] Is this only a feature request? Do not put multiple feature requests in one issue.
• [X] Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Is your proposal related to a problem?

Just a lack of the ability to show silent appreciation

Describe the solution you'd like.

Lemmy is not a commercial project, nor should it be. Gifts as designed by other social media are a toxic money dump that prey on users' good will. Gifts to other users should not be "purchased" from some central authority (though a case could be made that server maintainers could be the distributors of gifts, and at that point it's up to them?).

I think theres a much more wholesome way to actually have a "gift economy" - if a post or comment of mine receives a "gift" from someone, that gift goes into my inventory. The only use for that gift in my inventory is to gift it to another post.

Describe alternatives you've considered.

more or less mused upon below

Additional context

I think this can create a positive feedback loop where users can show their appreciation to each other and feel motivated to do so

But there are a couple of issues:

• Where do gifts come from? Does every user just have one of each by default? Do servers distribute them based on their own rules? Are they earned via engagement?
  • If servers determine this, this could help differentiate communities from one another, but also increase "choice paralysis" when choosing a server.
    • This could also allow servers to opt out of gifting entirely if they choose to keep things simpler
• What of people who don't give back? They become gift leeches and all the gifts from the community disappear into black holes.
  • Auto-regen periods defined by the server? I.e. after one week, if a user has no "thank you" gift, they will be granted one in their inventory
• What would the gifts be, and who decides?
  • Do servers decide? Does lemmy only support certain gifts? If servers decide, how do we limit or support gifting between users of X server on posts of Y servers?

Requirements

• [X] Is this a bug report? For questions or discussions use https://lemmy.ml/c/lemmy_support
• [X] Did you check to see if this issue already exists?
• [X] Is this only a single bug? Do not put multiple bugs in one issue.
• [X] Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Summary

If you browse with show read posts enabled, and read every post on page 1 of content, when you navigate to page 2, it will show you page 2 of "unread" content, and you need to go back to page one.

Steps to Reproduce

1. Disable "Show Read Posts"
2. Browse a community

• I suggest a community with consistent content but a relatively static sort - e.g. [email protected] w/TopMonth

3. "Read" each post on the 1st page (e.g. upvote all of them)
   • Pay attention to top couple of posts on page
   • Notice vote counts at bottom of page
4. Browse to Page 2
   • Notice large drop in post vote count between end of page 1 and page 2
   • Pay attention to top couple of posts on page
5. Browse back to page 1
   • Notice vote counts at top of page and bottom of page aligns between the end of the original page 1 and the viewed page "2"
   • Notice Page 1 content is different from original page 1 content

Technical Details

I believe this is a pagination issue when constructing the offsets used for pagination, there may need to be a mechanism to deduct or track the state of read posts.

Version

BE: 18.

Lemmy Instance URL

lemmy.fmhy.ml

Requirements

• [X] Is this a bug report? For questions or discussions use https://lemmy.ml/c/lemmy_support
• [X] Did you check to see if this issue already exists?
• [X] Is this only a single bug? Do not put multiple bugs in one issue.
• [X] Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Summary

If you receive an abusive DM there is no way to remove it from your inbox without admin help.

Steps to Reproduce

Ideally a person would be able to delete all private messages in their inbox, regardless of whether they created them or not.

But if this is not possible, blocking the abuse account should hide all DMs from them.

Reproduction:

1. Person B send messages to person A
2. Person A block person B
3. See person B messages still show up in inbox of person A
4. Observe person A also has no way to delete person B's messages. Their stuck it person A's inbox forever unless an admin intervenes.

Technical Details

N/A

Version

0.18.2

Lemmy Instance URL

No response

Requirements

• [X] Is this a feature request? For questions or discussions use https://lemmy.ml/c/lemmy_support
• [X] Did you check to see if this issue already exists?
• [X] Is this only a feature request? Do not put multiple feature requests in one issue.
• [X] Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Is your proposal related to a problem?

allow for .mp3 .ogg etc. uploads

Describe the solution you'd like.

uploading audio files

Describe alternatives you've considered.

n/a

Additional context

n/a

Requirements

• [X] Is this a feature request? For questions or discussions use https://lemmy.ml/c/lemmy_support
• [X] Did you check to see if this issue already exists?
• [X] Is this only a feature request? Do not put multiple feature requests in one issue.
• [X] Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Is your proposal related to a problem?

As a moderator I need to know when there are new posts in the communities I moderate so I can fulfill my role.

https://lemmy.world/post/1320681

Describe the solution you'd like.

A new option in settings to receive a notification for every new post only in the communities I moderate. Current notifications for new posts

Describe alternatives you've considered.

Creating a second account subscribed only to the communities I moderate and enable new post notification on that account.

https://lemmy.world/comment/1323243

Additional context

No response

Requirements

• [X] Is this a feature request? For questions or discussions use https://lemmy.ml/c/lemmy_support
• [X] Did you check to see if this issue already exists?
• [X] Is this only a feature request? Do not put multiple feature requests in one issue.
• [X] Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Is your proposal related to a problem?

Many videos from external sites fail to properly embed on Lemmy clients. Video detection from a URL is based on opengraph tags (specifically og:video), which a lot of sites intentionally don't send, and should be made more intelligent to ensure better content sharing. This is a major sore point for Lemmy at the moment.

Describe the solution you'd like.

To facilitate better video embedding, the fetch_site_metadata function needs to be improved - this way, no UI changes will be required. One easy and cheap way to do this might be to use something like yt-dlp, which can extract direct video URLs from indirect URLs. There are Rust wrappers available for the library.

For example, consider the imgur URL https://imgur.com/gallery/hOPRxdR. From this, yt-dlp is able to extract the direct MP4: ~ ❯ yt-dlp https://imgur.com/gallery/hOPRxdR -g https://i.imgur.com/txMlHj7.mp4

This process should fail fast. If any error is encountered, simply ditch the approach and fall back to opengraph.

Considerations

1. This can be too aggressive: for example, YouTube and v.reddit links can be resolved, but they are m3u8 streams that can't be played by the UI in a <video> tag. Only three formats are supported: mp4, webm, and ogg. This can be tuned in the command: ~ ❯ yt-dlp https://imgur.com/gallery/hOPRxdR -g -f 'best[ext=webm]/best[ext=mp4]/best[ext=ogg]' https://i.imgur.com/txMlHj7.mp4
2. Error handling is required in case a URL can't be selected: ~ ❯ yt-dlp https://v.redd.it/s2426qje27cb1 -g -f 'best[ext=webm]/best[ext=mp4]/best[ext=ogg]' ERROR: [Reddit] s2426qje27cb1: Requested format is not available. Use --list-formats for a list of available formats This is a non-zero exit code, so the normal fail-fast behavior might be enough.

Describe alternatives you've considered.

I considered using yt-dlp to also do the downloading of a video to pictrs, but that seems like unnecessary storage use. I'm sure there are alternative solutions out there.

Additional context

Sample Rust code (I am not a Rust programmer so don't judge)

```rust fn run_ytdl(input: &str) -> Result<YoutubeDlOutput, youtube_dl::Error> { YoutubeDl::new(input) .download(false) .format("best[ext=webm]/best[ext=mp4]/best[ext=ogg]") .socket_timeout("10") .run() }

fn extract_direct_url(input: &str) -> Option<String> { let output = run_ytdl(input).ok()?;

match output { YoutubeDlOutput::Playlist(playlist) => playlist.entries?.get(0)?.clone().url, YoutubeDlOutput::SingleVideo(video) => video.url } } ```

Requirements

• [x] Is this a feature request? For questions or discussions use https://lemmy.ml/c/lemmy_support
• [X] Did you check to see if this issue already exists?
• [X] Is this only a feature request? Do not put multiple feature requests in one issue.
• [X] Is this a backend issue? Use the lemmy-ui repo for UI / frontend issues.

Is your proposal related to a problem?

When I open Lemmy as an average user, I want to see posts that:

• have active discussions
• are from within the last day
• are highly upvoted

However, the Hot sort type shows me extremely recent posts with few votes or comments, and Active mostly shows me posts that are between one and two days old.

Describe the solution you'd like.

Create additional sort types:

• Active Six Hours
• Active Twelve Hours
• Active Day
• Active Two Days

With Active Six Hours, the latest comment time no longer updates after six hours past the post published time. With Active Twelve Hours, the latest comment time no longer updates past twelve hours after the post published time, and so on an so forth.

Active Two Days would be the current default behavior of the Active sorting algorithm.

Describe alternatives you've considered.

• Set the default sorting algorithm to Hot
• Adjust the Gravity value until the average age of posts on the front page of lemmy.ml is twelve hours.

Additional context

Posts on the topic:

• YSK: Feel like you only see the same 2-day old content? At least on lemmy.world, you can change your homepage’s default sorting type & scope to “Hot” and “All”
• FYI sort by “Top Day” if you’re tired of seeing the same top posts from 2-3 days ago

This PR is not complete (missing items detailed below), but I am submitting it already as a draft to get some early feedback. Please check the description below before checking code - I would really appreciate feedback on the overall design which is included in the description. But comments on the partially complete code are of course welcome as well.

----

Introduction

This PR contains an overhaul of Lemmy authentication. It introduces three new authentication tokens: access tokens, refresh tokens, and api tokens (more details below).

The changes are intended to be backwards compatible - the existing /login endpoint will become deprecated but will remain operational until we are ready to remove it in a future version.

What is wrong with our current authentication?

1. Auth tokens never expire: https://github.com/LemmyNet/lemmy/issues/3364
2. Auth sessions can't be revoked by users
3. There is no support for httpOnly cookie based auth: https://github.com/LemmyNet/lemmy-ui/issues/1252
4. There is no support for api token based auth - all 3rd party apps require user passwords
5. All auth tokens have full access to everything, their scope can't be limited

This PR contains intends to solve all these issues.

Proposed solution

This PR proposes to replace the existing auth token with 3 new types of tokens:

Access token

This token can be acquired with either a refresh token or an API token.

The new access token is intended to be a backwards compatible drop-in replacement for the existing auth token, with a few key differences:

• It expires within 5 minutes (so even if it leaks, it can only be abused within 5 minutes of the leak)
• It contains a method claim, which can be used later to limit certain activities to specific methods (for example, disallow password changes if the access token was obtained via an API token)

Refresh token

This token can be acquired using username + password (+ 2fa).

It lives in a secure httpOnly cookie (can't be read from browser js), which is limited only to the /api/v3/get_access_token path.

This is intended only for trusted web interfaces (such as lemmy-ui) and can be used to create access tokens with full access to the user. Each refresh token can be considered a separate "session". Each token records its last use time, as well as last use ip address - these values can be displayed to users in some new security UI so they get an overview of their active sessions. Each refresh token expires 2 weeks after it was last used, or when revoked manually by a user.

API token

This token must be manually created by users with a specific label and expiry date.

This is intended for 3rd party apps to avoid users from entering their passwords directly into untrusted code. The api token can be used similarly to refresh tokens to request access tokens, but the created access tokens would have limited access. Each API token will also record their last use time as well as last use ip address. API tokens expire after their user defined expiry date, or when revoked manually.

----

To summarize the general flow:

1. Acquire either a refresh token (if trusted web ui) or an API token (if 3rd party app)
2. Request access token using the token from step 1
3. Make all API requests with access token from step 2
4. If access token is close to expiry (or last request failed due to token), get a new access token (and retry last request)
5. If getting access token fails due to a token error, assume the (refresh or api) token has expired and go back to step 1

Rollout plan

1. Release the new logic in a minor Lemmy version
2. Add a migration guide to release notes to allow app developers to migrate to the new APIs
3. Update Lemmy-ui to use the new endpoints
4. After some time has passed, remove the old /login endpoint in a backwards-incompatible Lemmy update

TODO in this PR

• Add refresh token list & revoke endpoints
• Add api token create & list & revoke endpoints
• Disallow some actions (new api token creation + password change + reading user e-mail?) when access token method is Api
• Add some tests

TODO in future PRs

• Switch lemmy-ui to use new authentication
• Add security page to lemmy-ui, where users can see and revoke their sessions (refresh tokens), as well as see/revoke/create API tokens
• Add method for 3rd party apps to redirect users to an API token creation page (with a potential return_url to automatically get back to the app with the created token)

When re-running the first cargo clippy command in fix-clippy.sh, the build time of db_views is now 311.1s instead of 1281.9s

Helps with #3610

When re-running the first cargo clippy command in fix-clippy.sh, the build time of db_views is now 311.1s instead of 1281.9s

Helps with #3610