FEATURE: Select which links should be used in web scraper nodes (cheerio, puppeteer, and playwright)

[0xi4o]

• Adds a Manage Links button to Web Scraper nodes like Cheerio, Puppeteer, and Playwright. Until now, users can't choose which links to scrap. The node just picks the first n links (configurable in Additional Parameters) and scraps their content. Now users can pick which links they want to scrap.
• Clicking on the Manage Links button will open a dialog where you can fetch all the links from the given URL, and remove the ones you don't need. During upsert, only the selected links will be scraped. Selecting links is optional. The nodes will fallback to the old behavior when no links are selected.
• If Relative Links Method is set in Additional Parameters before selecting links, the manage links dialog will use the selected method to fetch the links.
• If Relative Links Limit is set and is smaller than the number of selected links, the limit will apply to the selected links and the first n links are chosen. If it's larger, then all the selected links are scraped.

[HenryHengZJ]

Bug:

1.) Enter https:flowiseai.com into the Cheerio node:

2.) Open Manage Links and modified to https://docs.flowiseai.com:

3.) Fetch links

4.) Click save

5.) On the canvas, the Cheerio Node URL is still https:flowiseai.com, same as when you click manage links:

[0xi4o]

@HenryHengZJ Fixed the bug. The URL field in the node will be updated if we change it in the manage links dialog.

[HenryHengZJ]

awesome work @0xi4o !

[HenryHengZJ]

@0xi4o one side topic - right now whenever users restart the apps, and ask a question, it runs through the webscraping stuff again because of the init() function. Can we skip this if the request is not the upserting request? maybe a flag that passes to the options. Because we only need to process the scraping when user is hitting the vector/upsert API

[0xi4o]

@HenryHengZJ Sure! I'll update it 👍

[0xi4o]

@HenryHengZJ There's already an isUpsert flag in the buildLangchain() in server/src/index.ts. It's set to true only during upsert operations. When calling the prediction endpoints, it's not set so it won't do upsert. This means the scraping will also only happen during upsert and not during prediction.

[HenryHengZJ]

@HenryHengZJ There's already an isUpsert flag in the buildLangchain() in server/src/index.ts. It's set to true only during upsert operations. When calling the prediction endpoints, it's not set so it won't do upsert. This means the scraping will also only happen during upsert and not during prediction.

it wont upsert, but it will still go through the init of Cheerio node, scraping all the links, which might take significant portion of the time

[0xi4o]

@HenryHengZJ Added a condition to skip initializing web scraper nodes during prediction.

[luc4t]

Could you add the option to exclude links ending with ".jpg, .jpeg, .png, .pdf or anything similar?

Cheerio will scrape jpegs and upsert jpeg headers and everything after it into the vector database. I haven't found an option to prevent this.

Apify's Cheerio web scraper uses:

[
{
"glob": "/**/*.{png,jpg,jpeg,pdf}"
}
]