There are four bottles on my list that I only buy on offer. Checking by hand is exactly the kind of chore you keep up for two weeks. A script that does it every morning is twenty lines of code, which is the problem: it is so easy that almost nobody stops to ask whether they may.
This piece is mostly about that question, and about the mistake I made in it while thinking I was being careful.
The job
A list of URLs, one check a day, and a note when a price drops. Runs on my machine, sends nothing anywhere, and behaves on someone else’s server the way I would want someone else’s script to behave on mine.
My actual prompt
Write a Node script that fetches the price for a list of URLs and compares
it against last time.
Before fetching a page: read that host's robots.txt and check whether the
path is allowed for our user agent. If it is not, skip the URL with a clear
message. Do not fetch it anyway.
At least five seconds between two requests.
Use a recognisable user agent, not a browser pretending to be a person.
Take the price from the structured data, not from the markup.
Finding no price is not an error, just report it.
Store results so a later run can show the difference.The script is at sparkone.nl/tools/prijswacht.mjs.
Taken apart
Put the permission check before the fetch, not after. It sounds obvious, and in almost every example online it is the other way round: fetch, then filter. The difference is that in the second case you have already loaded the server and already hold the data.
Ask for a recognisable user agent. A script disguised as Chrome is a script that knows it is doing something it shouldn’t. Put your own name in it and an administrator can block you without any fuss, which is the point.
Structured data over markup. Pulling the price out of a class="price-large" works until the first restyle. Pulling it from the structured data lasts longer, because that is there for search engines and changes less often.
Keep the previous value. The interesting thing is not the price, it is the change. That is two lines of code and the only thing that justifies a notification.
What broke
I tested with two URLs: a Google search page and a bol.com account page. The first was refused properly. The second my script fetched without complaint.
That looked right until I read bol.com’s robots.txt myself. It says, in as many words, Disallow: /*/rnwy/account/, which covers exactly the URL I was using. My script should have refused it.
The cause was that my robots parser compared paths with a simple “starts with” test. The standard has two wildcards: an asterisk for any run of characters, and a dollar sign for the end of the URL. Large shops use almost nothing else. Of the 136 lines in that one robots.txt, dozens contain an asterisk, and my script silently ignored every one of them.
That is the most dangerous kind of bug in this sort of work. The script did nothing visibly wrong. No error, nothing blocked, and the output looked exactly as intended. I would never have noticed if I had not checked by hand.
After the fix, fifteen lines that turn a pattern into a regular expression, it refuses both URLs. And that is the wider point: with code that goes onto the internet on your behalf, it is not enough that the output looks right. You have to verify that the rule you thought you were following was actually applied.
Ethics note
Robots.txt is not the only boundary, and honestly not the most important one. It is a technical signal. On top of it sit terms of service, and at many shops those forbid automated fetching regardless of what robots.txt says. Nobody is going to pursue you over four pages a day for personal use, and it is still worth knowing that “robots.txt allowed it” is not a licence.
Then scale. Four URLs, once a day, five seconds apart, is less traffic than a single browsing visitor. The same code with two hundred URLs every fifteen minutes is something else, especially for a small importer running their shop on cheap hosting. The code barely changes, the impact does, and that is the kind of boundary no error message exists for.
And the wine caveat: a price watcher makes you a better hunter, not a better drinker. I gave myself a rule that nothing goes on the list unless I have already bought it deliberately once. Otherwise you buy on discount instead of on taste, which is the exact habit I ran into in the piece about profiles.
Take it
Write a script that fetches [what you want to track] for a list of URLs
and compares it against last time.
First read that host's robots.txt and check whether the path is allowed for
our user agent, including the wildcards * and $. Not allowed means skip with
a message, not fetch anyway.
At least five seconds between requests, and a recognisable user agent.
Take the value from the structured data, not the markup.
Store results so the next run can show the difference.A small exercise for today: open the robots.txt of a site you visit often (add /robots.txt to the domain) and find the lines with an asterisk. That is the half most home-made scripts ignore.