How to insert 10 million records into a MySQL database table in 88 seconds

How to insert 10 million records into a MySQL database table in 88 seconds

The database I use is MySQL database version 5.7

First prepare the database table yourself

Actually, I encountered some problems when inserting 10 million data. Now I will solve them first. At the beginning, I got an error when inserting 1 million data. The console information is as follows:

com.mysql.jdbc.PacketTooBigException: Packet for query is too large (4232009 > 4194304). You can change this value on the server by setting the max_allowed_packet' variable.

The above error occurs because the max_allowed_packet configuration of the database table is not large enough, because the default is 4M. Later, I adjusted it to 100M and no error was reported.

set global max_allowed_packet = 100*1024*1024*

Remember, you need to log in to the database again after setting it to see the set value

show VARIABLES like '%max_allowed_packet%'

The code is as follows:

package insert;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.Date;
import com.mysql.jdbc.PreparedStatement;
public class InsertTest {
   public static void main(String[] args) throws ClassNotFoundException, SQLException {
     final String url = "jdbc:mysql://127.0.0.1/teacher" ; 
     final String name = "com.mysql.jdbc.Driver"; 
     final String user = "root" ; 
     final String password = "123456" ; 
     Connection conn = null; 
     Class.forName(name); //Specify the connection type conn = DriverManager.getConnection(url, user, password); //Get the connection if (conn!= null ) {
       System.out.println("Connection obtained successfully");
       insert(conn);
     } else {
       System.out.println("Failed to obtain connection");
     }
   }
   public static void insert(Connection conn) {
     // Start time Long begin = new Date().getTime();
     // sql prefix String prefix = "INSERT INTO t_teacher (id,t_name,t_password,sex,description,pic_url,school_name,regist_date,remark) VALUES ";
     try {
       // Save sql suffix StringBuffer suffix = new StringBuffer();
       // Set the transaction to non-auto commit conn.setAutoCommit( false );
       // PST is better than ST PreparedStatement pst = (PreparedStatement) conn.prepareStatement( "" ); //Prepare to execute statement// Outer loop, total number of committed transactions for ( int i = 1 ; i <= 100 ; i++) {
         suffix = new StringBuffer();
         // j-th submission step for ( int j = 1 ; j <= 100000 ; j++) {
           // Build SQL suffix suffix.append( "('" + uutil.UUIDUtil.getUUID()+ "','" +i*j+ "','123456'" + ",'Male'" + ",'Teacher'" + ",'www.bbk.com'" + ",'XX University'" + ",'" + "2016-08-12 14:43:26" + "','Note'" + ")," );
         }
         // Build complete SQL
         String sql = prefix + suffix.substring( 0 , suffix.length() - 1 );
         // Add execution SQL
         pst.addBatch(sql);
         // Execute operations pst.executeBatch();
         // Commit transaction conn.commit();
         // Clear the last added data suffix = new StringBuffer();
       }
       //First-class connection pst.close();
       conn.close();
     } catch (SQLException e) {
       e.printStackTrace();
     }
     // End time Long end = new Date().getTime();
     // Time consuming System.out.println( "Time taken to insert 10 million records: " + (end - begin) / 1000 + " s" );
     System.out.println("Insert completed");
   }
}

Summarize

The above is the operation method introduced by the editor to insert 10 million data into the MySQL database table in 88 seconds. I hope it will be helpful to everyone. If you have any questions, please leave me a message and the editor will reply to you in time. I would also like to thank everyone for their support of the 123WORDPRESS.COM website!

You may also be interested in:
  • MySql quick insert tens of millions of large data examples
  • Example code for inserting millions of data into MySQL using JDBC in Java
  • How to quickly insert millions of test data in MySQL
  • Teach you how to insert 1 million records into MySQL in 6 seconds

<<:  ThingJS particle effects to achieve rain and snow effects with one click

>>:  JavaScript offset implements mouse coordinate acquisition and module dragging within the window

Recommend

10 skills that make front-end developers worth millions

The skills that front-end developers need to mast...

A collection of information about forms and form submission operations in HTML

Here we introduce the knowledge about form elemen...

Workerman writes the example code of mysql connection pool

First of all, you need to understand why you use ...

How to implement horizontal bar chart with percentage in echarts

Table of contents Example Code Rendering Code Ana...

A detailed discussion on detail analysis in web design

In design work, I often hear designers participati...

Vue achieves the top effect through v-show

html <div class="totop" v-show="...

Detailed example of IOS database upgrade data migration

Detailed example of IOS database upgrade data mig...

A case study on MySQL optimization

1. Background A sql-killer process is set up on e...

Example analysis of interval calculation of mysql date and time

This article uses an example to describe the inte...

Docker builds CMS on-demand system with player function

Table of contents text 1. Prepare the machine 2. ...

Nginx service 500: Internal Server Error one of the reasons

500 (Internal Server Error) The server encountere...

Solution to the cross-domain problem of SpringBoot and Vue interaction

Table of contents Browser Same Origin Policy 1. V...

MySQL Full-text Indexing Guide

Full-text indexing requires special query syntax....