I have quite an extensive history of scraping web sites for various data over the years, I’d be happy to help you out but I can’t really know how to help without knowing what website your trying to scrape, different sites have their own challenges (maybe behind a login, or using JavaScript to load content - in which case a http response won’t give you what you’re after, or any number of things really).

If you give me a link to a book you want to download as an example I can take a look and help guide you through it

Depending on what you want to scape, that’s a lot of overkill and overcomplication. Full website testing frameworks may not be necessary to scrape. Python with it’s tooling and package management may not be necessary.

I’ve recently extracted and downloaded stuff via Nushell.

1. Requirement: Knowledge of CSS Selectors
2. Inspect Website DOM in Webbrowser web developer tools
   1. Identify structure
   2. Identify adequate selectors; testable via browser dev tools console document.querySelectorAll()
3. Get and query data

For me, my command line terminal and scripting language of choice is Nushell:

let $html = http get 'https://example.org/'
let $meta = $html | query web --query '#infobox .title, #infobox .tags' |  | { title: $in.0.0 tags: $in.1.0 }
let $content = $html | query web --query 'main img' --attribute data-src
$meta | save meta.json

or

1..30 | each {|x| http get $'https://example.org/img/($x).jpg' | save $'($x).jpg'; sleep 100ms }

Depending on the tools you use, it’ll be quite similar or very different.

Selenium is an entire web-browser driver meaning it does a lot more and has a more extensive interface because of it; and you can talk to it through different interfaces and languages.

It needs a driver and the web-browser to be executed in headless mode. For Chrome that’s chrome-driver. You can get it here.

To make a script for it, I recommend talking to a LLM. I have asked it to build scrapers before, so it does the job.

If you want a practical use of Selenium being demonstrated, you can see it in LucidWebSearch plugin for Oobabooga.

There is no simplification that you’re looking for. It seems you don’t have a programing background. If you really need to scrape something, you need to learn a programing language, HTTP, HTML, and maybe javascript. AFAIK, there is no easy way or point and click scrapper building tool. You will need to invest time and learn. Don’t worry, you should be able to get it done in 2-3 months if you do invest your time in.

We use node.js with puppeteer for some of our web crawling at work. It’s pretty straightforward once you have a basic script to launch it. If you havent already I’d highly suggest installing vs code. You install node.js, then using npm (node package manager) install puppeteer and whatever other dependencies you might have. Someone out there probably has a basic js file out there that will open chrome, or just ask an LLM (I just use ChatGPT, they’re all the same shit). From there you just need to navigate to your pages, then use a queryselector and .click() to click on your elements. It’s all javascript from there.

Pro tip: write your queryselectors in your browser using the inspect element/console tab, then put it in your JS file. Nothing is worse than being 10 minutes into a crawl and you’ve got a queerselector.

Ask dr gpt