I was using it’s APIs. But new restrictions have effectively been put in place that prevent me from using them for what I need. Similar API calls were being made that were causing DDOS attacks on lemmy.world.
As for running a lemmy instance itself. That’s a thought but I need the data in a different format to do efficient searches. It’s a tricky problem.
Yep that’s the new idea. The sad part is that with this method there’s no way to get historical data. Only new posts. So if a server goes down, gets DDOSd etc… I’ll lose posts forever.
Also building an ActivityPub implementation from scratch isn’t trivial either. So that’ll take some time.
I’ve got a few other ideas I’m playing with as well. Like just assuming that internal post IDs are all sequential and literally fetching them one by one. Or maybe some combination of both?
Thanks for the shout-out.
But FYI I’ve run into some bugs that’s preventing new content from being indexed. So you won’t see anything new (from about a week ago) until I can find a new method to fetch new posts.
Ya if it’s not federated already, I’m thinking the link will just remain as-is? Then comes the question of what happens after federation is complete? Ideally I’m thinking to do these rewrites as the comment containing the link gets federated so it only happens once, but that prevents updates… If I do it during someone looking at the post, then that becomes a lot of network traffic…
Still a lot to plan out initially and then propose to the devs for feedback… So it’ll be awhile still.
I just got a PR merged today that might help with this. I’ll start experimenting with it more over the next week or so.
Basic theory is that I can detect at least lemmy posts in the comment bodies and then rewrite those to your local instance. Primarily question is going to be performance, as remote network calls will be necessary.
Not saying a tool like this isn’t helpful for you but personally I don’t think I’d ever trust a service to analyze my code and provide accurate comments. There’s so many little quirks and ultimately nuance in what is written vs what and why it’s doing it that I’d never expect anyone but the original author of the code to get it right.
Think of Lemmy as email. Each post or comment is just an email sent to a distribution group (a community). If your email server goes down, all of those users and distribution groups are gone. Now I’ll still have the emails I sent to you in my email box but you won’t be able to see them as your email server is offline. Sure you could create a new account on a new server but you’d have to tell everyone about your new address (federate) but there’s nothing to associate your old user with your new one and there’s no way to backfill data. I could reply-all or forward (comment) on to your new address but there’s still no way to associate those old posts with your new account.
If an instance goes down (permanently), federation of all of the communities hosted by that instance essentially stop. The content that has already been posted remains but anything new added to those communities only remain on your home instance. The only way for federation to resume is for that instance to come back online with the same domain it started with.
Same. Honestly I need to create a community just for this tool IMO. But I don’t have the time to moderate it.
https://www.search-lemmy.com/ ?
I’m open to feedback though if the search results seem out of order etc…
With ActivityPub all of the primary ids contain the domain of the hosting server. So if you lose your domain none of the other instances know that you’re the authority on those communities, posts, comments or users. So essentially federation breaks with all of the old data.
It searches the title and body. It also automatically searches for similar words like. Like ‘bike’, ‘biking’, ‘bikes’ (aka stemming). Granted though, I’m still improving the page ranking as time goes on.
Lookup Overseerr. https://overseerr.dev/
There is a public API now. While I won’t support sorting, you can process and do what you will with the results as-is. Currently I only support Posts and Communities for now.
When you search for posts you’re just matching against the title or body. For communities it’s searching the posts within that community.
There’s also more filters now with: instance/community/author/since/until and a safe-search option.
So I’m not sure how close this comes to your idea but I thought I’d share.
Yep. The idea is that the instance you select is the instance that all links will open up in. This way once you find a post you’re looking for you don’t have to find it on your home instance and can immediately start replying, save it for later, subscribe to the community, etc…
Edit: I do plan on “fixing” this eventually, but I’m waiting on a bugfix on lemmy itself. You can see more here: https://github.com/marsara9/lemmy-search/issues/20
Just pushed a new update, that should fix an issue with the home-instance selector. Since you’re on lemmy.world I’d suggest selecting that instance from the drop-down and trying your search again…
Bummer. That’ll slow down Kbin’s inclusion into this. Well once it is available I’m sure I’ll start digging into it. But thanks for the info.
I’ve already started to abstract away Lemmy from the search engine itself. So the first steps are in place. Once I get the kinks of the 0.4.x release knocked out then I plan on reading up on Kbin’s API and I’ll start working on the crawler. I can’t promise anything but that should give you a rough timeline.
If you have any programming skills I could always use a hand.
Check out my post history.
But https://www.search-lemmy.com. It has a few bugs but it should work for you. Especially if you set your home instance to something large like Lemmy.world.
Edit: if you want to help contribute: https://www.github.com/marsara9/lemmy-search
Correct. As I can only provide links to posts that are on your selected home instance. Eventually I’ll change this but you’ll get a 404 page for links that aren’t on your home instance, but see my P.S. below.
P.s. there have been changes to the Lemmy API that have prevented me from getting updates for about a month now. So most of the results you’re seeing are from old posts only. Until I can rebuild the crawler or find a new API there won’t be any new content.