How to deploy python crawler scripts on Linux and set up scheduled tasks

How to deploy python crawler scripts on Linux and set up scheduled tasks

Last year, due to project needs, I wrote a crawler in Python. The crawled data needs to be stored in the PG database of the production environment. Therefore, you need to deploy the script to the CentOS server and set up a scheduled task to automatically start the script.

The implementation steps are as follows:

1. Install pip (the operating system comes with python2.6 which can be used directly, but there is no pip)

# Download pip installation package wget "https://pypi.python.org/packages/source/p/pip/pip-1.5.4.tar.gz#md5=834b2904f92d46aaa333267fb1c922bb" --no-check-certificate
# Unzip the installation package and install tar -xzvf pip-1.5.4.tar.gz
cd pip-1.5.4
python setup.py install

2. Install third-party libraries with pip

pip install PyGreSQL==5.0.3
pip install requests==2.18.3

3. Set up scheduled tasks

# Start the scheduled task service service crond start
# View the scheduled task service status service crond status
# Open the scheduled task editing window crontab -e 
# Add two scheduled tasks, which will be executed once at 0:00 and 12:20 every day, and written to the log 0 0 * * * /usr/bin/python /home/longrise/psrd/collect.py > /home/longrise/psrd/collect.log 2>&1 &

20 12 * * * /usr/bin/python /home/longrise/psrd/collect.py > /home/longrise/psrd/collect.log 2>&1 &

The syntax of scheduled tasks is as follows:
# For details see man 4 crontabs

# Example of job definition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12) OR jan,feb,mar,apr ...
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# | | | | |
# * * * * * user-name command to be executed

The above method of deploying python crawler scripts in Linux and setting scheduled tasks is all the content that the editor shares with you. I hope it can give you a reference. I also hope that you will support 123WORDPRESS.COM.

You may also be interested in:
  • Jenkins configuration python script timing task process diagram
  • Python BlockingScheduler timing tasks and other implementations
  • How to use APScheduler, a Python timed task tool
  • Python scheduled task APScheduler example example detailed explanation
  • Several common methods for Python crawler scheduled tasks (recommended)

<<:  Detailed explanation of JS WebSocket disconnection reasons and heartbeat mechanism

>>:  Detailed explanation of MySQL user rights verification and management methods

Recommend

CSS achieves the effect of aligning multiple elements at both ends in a box

The arrangement layout of aligning the two ends o...

Deployment and Chinese translation of the docker visualization tool Portainer

#docker search #docker pull portainer 1. Download...

Vue+element ui realizes anchor positioning

This article example shares the specific code of ...

Linux uses if to determine whether a directory exists.

How to use if in Linux to determine whether a dir...

The difference between html empty link href="#" and href="javascript:void(0)"

# contains a location information. The default anc...

JavaScript implements asynchronous submission of form data

This article example shares the specific code of ...

Detailed explanation of common methods of JavaScript arrays

Table of contents Common array methods pop() unsh...

A Different Kind of "Cancel" Button

The “Cancel” button is not part of the necessary ...

CSS to implement QQ browser functions

Code Knowledge Points 1. Combine fullpage.js to a...

VMware ESXi 5.5 deployment and configuration diagram process

Table of contents 1. Installation requirements 2....

How to use VirtualBox to simulate a Linux cluster

1. Set up HOST on the host Macbook The previous d...

Sample code for JS album image shaking and enlarging display effect

The previous article introduced how to achieve a ...

Summary of some points to note when registering Tomcat as a service

Here are some points to note when registering Tom...

Linux type version memory disk query command introduction

1. First, let’s have a general introduction to th...