I’m spending my free time creating projects because I want to change from mechanical engineering to a job involving data. But the website’s API I’m using doesn’t give me the information I need, even though I can see it on the site.
Is there a program that can automatically run some code I write in Python every few hours? This code would use Beautiful Soup to scrape the data, organize it, and put it into a SQL table. If not, can I somehow write a script to do this using Python or another language?
Yes, you may use the scheduling modules in Python (such as “schedule” or “apscheduler”) to automate this job by running your scraping application regularly. Your script can also be run at predetermined intervals using tools like Task Scheduler on Windows or Cron jobs on Unix-based computers. Beautiful Soup for Python is a good tool for web scraping, and SQLAlchemy is a useful library for managing SQL database transactions. You may easily automate data storage and retrieval with these technologies.
Given that modern websites frequently separate the front-end and back-end, I would strongly advise analysing the traffic in Chrome Dev tools before utilising Selenium, Beautiful Soup, and so on.
I know in Windows you can set a scheduled task to wake your computer to run the task and not to run if it’s on battery. If you leave it plugged in overnight (I assume this is the case since you’re counting hours instead of days/weeks) you should be good to go. The Raspberry pi is a good option though.