Everyone loves the idea of scraping, no one likes maintaining scrapers that break once a week because the CSS or HTML changed.
I loved scraping until my ip was blocked for botting lol. I know there’s ways around it it’s just work though
I successfully scraped millions of Amazon product listings simply by routing through TOR and cycling the exit node every 10 seconds.
That’s a good idea right there, I like that
This guy scrapes
lmao, yeah, get all the exit nodes banned from amazon.
That’s the neat thing, it wouldn’t because traffic only spikes for 10s on any particular node. It perfectly blends into the background noise.
You guys use IP’s?
I’m coding baby’s first bot over here lol, I could probably do better
Or in the case of wikipedia, every table on successive pages for sequential data is formatted differently.
Just use AI to make changes ¯_(ツ)_/¯
Here take these: \\
¯(ツ)/¯\\ Thanks
I’m down with scraping, but “parses HTML with regex” has got me fucked up.
13 years ago my god. I wonder what Jon Skeet is doing these days.
I remember when he passed me in the reputation ranking back in the early days and thinking that I needed to be a little bit more active on the site to catch him lol.
That was a great read. Thanks!
This is the way
What’s wrong with parsing HTML with regex?
In short, it’s the wrong tool for the job.
In practice, if your target is very limited and consistent, it’s probably fine. But as a general statement about someone’s behavior, it really sounds like someone is wasting a lot of time and regularly getting sub-par results.
Just a heads up for companies thinking it’s wrong to scrap: if you don’t want info to be scraped, don’t put it on the internet.
But, but, norobots.txt!
Chad doesn’t care!
The sad part is that scrapping is often easier then using the api.
Much less beholden to arbitrary rules also. Way too many times companies will just up and lift their API access or push through restrictions. No ty, I’ll just access it myself then
deleted by creator
API starter kit
- Outdated and unsupported and hasn’t been replaced yet but is the standard way to use the service.
- Lots of authorization tokens.
- The example in the docs doesn’t work (if there is one).
- You have no idea where the online tutorial got the information because it doesn’t have links to resources and the docs have barely anything even though its giant.
- Uses asynchronous programming to make it faster but its still much much slower then scrapping without asynchronous programming.
So true that it hurta
I scrape with bash lord help me.
you scrape WITH BASH?
Awk all the things!
pipe sed pipe grep pipe tr pipe grep… I would say I am a bit of a plumber
there’s literally dozens of us!
or maybe just 2 idk
as a windows user i say kindly on our behalf thank you for pushing the envelope ✉
someone’s never used a good api. like mastodon
Hold on, I thought it was supposed to be realism on the virgin’s behalf and ridiculous nonsense on the chad behalf:
All I see is realism on both sides lol
I used Twitter Scraper to get twitter data for my thesis. Shortly after, it became obsolete.
https://github.com/taspinar/twitterscraper/issues/368 rip twitter scraper
I’ve just discovered selenium and my life has changed.
I created a shitty script (with ChatGPT’s help) that uses Selenium and can dump a Confluence page from work, all its subpages and all linked Google Drive documents.
How so?
When a customer needs a part replaced, they send in shipping data. This data has to be entered into 3-4 different web forms and an email. This allows me to automate it all from a single form that has built in error checking so human mistakes are limited.
Company could probably automate this all in the backend but they won’t :shrug:
Using Selenium for this is probably overkill. You might be better off sending direct HTTP requests with your form data. This way you don’t actually have to spin up an entire browser to perform that simple operation for you.
That said, if it works - it works!
It’s all fun and games until you have to support all this shit and it breaks weekly!
That being said, I do miss the simplicity of maintaining selenium projects for work
I wanted to build a Discord bot that would check NIST for new CVEs every 24 hours. But their API leaves quiiiiiiite a bit to be desired.
Their pages, however…
Just use this https://github.com/CVEProject/cvelistV5/tree/main/cves
Oh yeah, that’s much more robust
Ok then make a spotify scraper
Removed by mod
ROFL, Chad only thinks that shit works
I have totally no idea what these are about…
Websites and services create APIs for programmers to use them. So Spotify has code that let’s you build a program that can use its features. But you need a token they give you after you sign up. The token can be revoked and used to monitor how much of their service you’re using. That way they can restrict if its too much.
Scraping is raw dogging the web slut you met at the cougar ranch who went home with you because you reminded her of her dog
Put like that I want to learn everything about it.
‘Scraping’ is the process of anonymously and programmatically collecting data from a webpage(s), often without the website’s permission and only limited to the content made publicly available. This is in contrast to using an API provided by the database owner which is limited by tokens, access volume, available end points etc.
deleted by creator
