Author Topic: FT links: a suggested method of checking for these automatically  (Read 64 times)

John

  • Administrator
  • Newbie
  • *****
  • Posts: 7
    • View Profile
Hassan Khalid Siddiqi sent me the following private message.


"You mentioned that you needed help with finding/monitoring full text links for research papers. I don't know if you have any experience with command line but have you checked out fulltext?

https://github.com/ropensci/fulltext

"It supports quite a few sources/backends. You can get full links by searching against the paper DOI as mentioned here

https://books.ropensci.org/fulltext/links.html

"So basically if we maintain a list of DOI, we should be able to set up a daily automated scripts to check if a fulltext link is available or not and then notify you so that you can update the website. Even though I am a programmer, I don't know the R language that's used, but can be learned. I managed to get fulltext running on my phone using the Termux app (after some effort on compiling its code) but it should be installable on a pc/laptop as well. I also managed to find the full text link for one of papers from the wiki, so it does work.

"Also scihub seems to be missing from the supported list of sources, could set up separate stuff checks for it."


Laurel, what do you think about Hassan's suggestion?
« Last Edit: March 06, 2021, 04:22:58 AM by John »

John

  • Administrator
  • Newbie
  • *****
  • Posts: 7
    • View Profile
Re: FT links: a suggested method of checking for these automatically
« Reply #1 on: March 06, 2021, 04:06:12 AM »
When Hassan explained this in simpler terms, it became clear that his suggestion could be very helpful to us, and set up quite easily, as he explains in his first paragraph here:


"My solution was that when you add a new abstract to the site, you add its DOI to a list, this list will be checked daily with fulltext to see if a full text link has been published or not. If yes, then just notify you with the link, so that you can add it to the site and remove its DOI from the DOI list.

"This could be made even better, you don't need to maintain a DOI list, we can just check the research page with a script to see which paper doesn't have a full text link tag, and we use search for it directly. However, for this, website will need to be better formatted, i.e full text link of each paper must ideally be under the same title.

"We can also check for expired links with a script, but there are likely throttles in these research paper sites that will prevent too many requests to be sent to them at the same time."

I'm thinking that what is suggested in the first paragraph, here, could be very helpful, but I'm not sure about the value to us of the rest. What do you think, Laurel?
« Last Edit: March 06, 2021, 04:27:50 AM by John »

hassan.siddiqi.14

  • Newbie
  • *
  • Posts: 1
    • View Profile
Re: FT links: a suggested method of checking for these automatically
« Reply #2 on: March 07, 2021, 12:19:46 PM »
So I have joined the forum as well. Let's see how we can do this.