lndb.info is basically dead at this point and does not work correctly at the moment. Novelupdates might be useful though for getting the japanese stuff.
Never used MAL myself so I can't comment.
As for scraping together the info. Yen Press and J-Novel Club went pretty fine tbh. I have their stuff already. The others ones are bit more problematic. Cross Infinite World has a site but they don't really include all the info I would have liked and scraping their site seems like quite the undertaking since scraping the correct stuff seems difficult.
Seven Seas also is missing quite some info on their site I will have to figure something out for these two.
Ultimately I would love to get the japanese stuff as well but me lacking japanese skills will make that one very difficult. Though I guess Novelupdates might prove useful here.
Oh my it actually posted 🎉 Had some issues posting since well lots of people joined ^^"
Building a proper Datasource for Light Novels (rebuild lndb.info like database with open library)
First time looking at lemmy. But I hope this will be a fun place to use and the community grows here a bit.
Lets come to topic I guess. I would like to build up a proper datasource for light novels which can be used in different ways. Basically my idea is to collect as much info on light novel data as I can (stuff like authors, illustrators, release date, etc) and then contribute all of it to open library (a project of the internet archives)
On top of the open library other people then can built cool interfaces and stuff.
Right now I am working on gathering the info using some scraping techniques and just hit my head against the wall until it eventually crumbles lol. I had some success but I will only be able to get data on english and german releases of light novel, since I don't speak japanese ^^"
If someone want (or can) help me out gathering data let me know. Also if you happen to know websites that contain japanese infos let me know.