Demand scenario: The boss asked me to use the crawler to write or update the data into the MySQL database. Baidu found two methods 1 is to use pymysql to connect to mysql, and add, delete, modify and query data by operating native sql statements; 2 is to use sqlalchemy to connect to mysql, build tables and operate databases through ORM models, without writing native sql statements, which is relatively simple; The following is my experience of using sqlalchemy. Implementation process: Connect to the database > Create a table through the model class > Establish a session > Execute the create table statement > Add, delete, modify and query through the session from sqlalchemy import exists, Column, Integer, String, ForeignKey, exists from sqlalchemy.ext.declarative import declarative_base from sqlalchemy import create_engine from sqlalchemy.orm import sessionmaker # Created database engine engine = create_engine("mysql+pymysql://user:pwd@ip/database name?charset=utf8") #Create session type DBSession = sessionmaker(bind=engine) # Instantiate the official model - Base is the ORM model Base = declarative_base() # Create a service order table class ServiceOrder(Base): __tablename__ = 'serviceOrderTable' id = Column(Integer, primary_key=True, autoincrement=True) serviceOrderId = Column(String(32), nullable=False, index=True, comment='Service Order ID') serviceDesc = Column(String(268), comment='Service Description') oneLevelName = Column(String(32), comment='C Category') twoLevelName = Column(String(32), comment='T subclass') threeLevelName = Column(String(32), comment='I project') fourLevelName = Column(String(32), comment='S sub-item') transferTimes = Column(String(32), comment='Number of transfers') overDueStatus = Column(String(32), comment='Expired status') serviceTimeLimit = Column(String(32), comment='Service time limit') serTimeLimitTypeName = Column(String(16), comment='Time limit type') # One-to-many: # serviceWorkOrder = relationship("ServiceWorkOrder", backref="serviceorder") # Many-to-one: Multiple service orders can belong to the service order class ServiceWorkOrder(Base): __tablename__ = 'serviceWorkOrderTable' id = Column(Integer, primary_key=True, autoincrement=True) serviceWorkOrderId = Column(String(32), nullable=False, index=True, comment='Service Work Order ID') workOrderName = Column(String(268), comment='Work Order Name') fromId = Column(String(32), comment='Service Order ID') createUserSectionName = Column(String(32), comment='Create User Section') createUserName = Column(String(32), comment='creator') handlerName = Column(String(32), comment='Handler') statusName = Column(String(32), comment='Work order status') createTime = Column(String(32), comment='Creation time') # The book table on the "many" side is linked to the user table via a foreign key: # serviceOrder_id = Column(Integer, ForeignKey('serviceOrderTable.id')) # Create a database. If the database already exists, it will not be created. It will directly connect to the existing library according to the library name. def init_db(): Base.metadata.create_all(engine) def drop_db(): Base.metadata.drop_all(engine) def insert_update(): # all_needed_data_lists is the data format that needs to be inserted into the database [{key: value, ... }, { }, { }...] for item in all_needed_data_lists: ServiceOrderRow = ServiceOrder(serviceOrderId=item['serviceOrderId'], serviceDesc=item['serviceDesc'], oneLevelName=item['oneLevelName'], twoLevelName=item['twoLevelName'], threeLevelName=item['threeLevelName'], fourLevelName=item['fourLevelName'], transferTimes=item['transferTimes'], overDueStatus=item['overDueStatus'], serviceTimeLimit=item['serviceTimeLimit'], serTimeLimitTypeName=item['serTimeLimitTypeName'], ) try: # Use exists to determine whether the target object exists and return True or Faults it_exists = session.query( exists().where(ServiceOrder.serviceOrderId == item['serviceOrderId'] ) ).scalar() except Exception as e: self.log.error(e) break try: # If it does not exist, add it; if it does exist, update the existing data if not it_exists: session.add(ServiceOrderRow) else: session.query(ServiceOrder).filter(ServiceOrder.serviceOrderId == item['serviceOrderId'])\ .update(item) except Exception as e: self.log.error(e) break try: session.commit() self.log.info('Data updated successfully!') except: session.rollback() self.log.info('Data update failed!') if __name__ == "__main__": # Create a database. If the database already exists, it will not be created. It will directly connect to the existing library according to the library name. init_db() # Create a session object to add, delete, modify and query: session = DBSession() # Use session to add or modify data. Remember to submit insert_update() The above is the full content of this article. I hope it will be helpful for everyone’s study. I also hope that everyone will support 123WORDPRESS.COM. You may also be interested in:
|
>>: How to set the user root password and remote connection method for Alibaba Cloud ECS instance
Today I will share with you a breathing carousel ...
This article shares the specific code for JavaScr...
Preface In the previous article, I shared with yo...
How can you forget lazy loading of routes that al...
1. Error details Once when manually performing a ...
After a lot of trouble, I finally figured out the...
System environment: Windows 7 1. Install Docker D...
Table of contents 1. The origin of tomcat 1. Tomc...
After getting used to VM, switching to BOX is a l...
1. Installation Environment Computer model: Lenov...
Table of contents 1. Introduction 2. Actual Cases...
1. Log in to MySQL database mysql -u root -p View...
Recently, I have a need to list all host names in...
As a super rookie, I just started learning MySQL ...
Preface When we forget the MySQL database passwor...