Does Lemmy content get indexed by search engines?
Does Lemmy content get indexed by search engines?
One of the side effects of the reddit meltdown is that many search results were unavailable because of communities going private. It would be great if we could fill in the void with lemmy content instead.
Yes it does!
I heard that reddit has a dedicated cdn each for Microsoft and Google scraping. That's why they work so well to search reddit posts. It will probably take some effort to feed data so we'll from the fediverse.
On that note, perhaps we should have some per-community as well as per-post scrape/noscrape toggle. Might be difficult to get buy-in from all parties.
Whether a community gets to opt out of being scraped depends on the scraper respecting robots.txt and/or the meta tag of the page.
Not all do, particularly the ones scraping for SEO purposes, so instances might to add IP bans for scrapers that refuse to respect restrictions in those places.
I've just tried a quick test using some popular queries and it looks as though communities are indexed but individual posts aren't? I agree, it would be nice to replace Reddit in this regard.
Maybe the above is only a temporary measure to help maintain server load?
One way to check is to do a search. e.g. for lemmy.world, google site:https://lemmy.world/
Already 906 results, with minimal SEO!
Some google searches already give me Lemmy posts, so it seems to work. I think indexing Lemmy posts takes more time, as I couldn't find my 'blog article' about hosting Lemmy on a Raspberry Pi or the community where it was posted yet trough Google yet. But I was able to find older communities on Feddit.nl, So most of the posts probably can't be found yet, as they simply are too new.
At the same time, I see people making “news articles” using people’s Reddit posts. More people making money on our content.
I used "F1 lemmy" multiple times in google already and the top post(s) were links to the F1 lemmy.
It works.
Which instance are you reading for F1?
Oh I want to know this too! Can't wait to get a bustling f1 community around here. I'm in [email protected] and @lemmy.ml
The first 3 hits google gives me as of now ;)
https://lemmy.ml/c/formula1
https://lemmy.world/c/formula1
https://lemmy.eus/c/[email protected]
Probably but you’re not seeing it lemmy is not large enough to have the same tracktion as Reddit