Implementation of inserting millions of records into MySQL database within 10 seconds

Implementation of inserting millions of records into MySQL database within 10 seconds

First, let’s think about a question:

To insert such a huge amount of data into the database, it must be accessed frequently under normal circumstances, and no machine equipment can handle it. So how can we avoid frequent access to the database? Can we access it once and then execute it?

Java has actually given us the answer.

Two key objects are used here: Statement and PrepareStatement

Let's look at the characteristics of the two:

BaseDao tool class to be used (jar package/Maven dependency) (Maven dependency code is attached at the end of the article) (packaged for easy use)

Note: (Important) rewriteBatchedStatements=true, insert multiple data at a time, and only insert them once! !

public class BaseDao { // Static tool class, used to create database connection objects and release resources for easy calling // Import the driver jar package or add Maven dependencies (Maven is used here, and the Maven dependency code is attached at the end of the article)
    static {
        try {
            Class.forName("com.mysql.cj.jdbc.Driver");
        } catch (ClassNotFoundException e) {
            e.printStackTrace();
        }
    }
 
    // Get the database connection object public static Connection getConn() {
        Connection conn = null;
        try {
            // rewriteBatchedStatements=true, insert multiple data at a time, insert only onceconn = DriverManager.getConnection("jdbc:mysql://localhost:3306/million-test?rewriteBatchedStatements=true", "root", "qwerdf");
        } catch (SQLException throwables) {
            throwables.printStackTrace();
        }
        return conn;
    }
 
    // Release resources public static void closeAll(AutoCloseable... autoCloseables) {
        for (AutoCloseable autoCloseable : autoCloseables) {
            if (autoCloseable != null) {
                try {
                    autoCloseable.close();
                } catch (Exception e) {
                    // TODO Auto-generated catch block
                    e.printStackTrace();
                }
            }
        }
    }
}

Next are the key codes and comments:

/* Because the processing speed of the database is very fast, the single throughput is very large and the execution efficiency is very high. addBatch() loads several SQL statements together and sends them to the database for execution at once. The execution takes a very short time. preparedStatement.executeUpdate() sends them to the database one by one for execution, and the time is consumed in the transmission of the database connection.*/
public static void main(String[] args) {
    long start = System.currentTimeMillis(); // Get the current time of the system and record it before the method starts executing Connection conn = BaseDao.getConn(); // Call the static tool class just written to get the connection database object String sql = "insert into mymilliontest values(null,?,?,?,NOW())"; // SQL statement to be executed PreparedStatement ps = null;
    try {
        ps = conn.prepareStatement(sql); // Get PreparedStatement object // Continuously generate sql
        for (int i = 0; i < 1000000; i++) {
            ps.setString(1, Math.ceil(Math.random() * 1000000) + "");
            ps.setString(2, Math.ceil(Math.random() * 1000000) + "");
            ps.setString(3, UUID.randomUUID().toString()); // The UUID class is used to randomly generate a string that will not be repeated ps.addBatch(); // Add a set of parameters to the batch command of this PreparedStatement object.
        }
        int[] ints = ps.executeBatch(); // Submit a batch of commands to the database for execution. If all commands are executed successfully, an array of update counts is returned.
        // If the array length is not 0, it means that the SQL statement is executed successfully, that is, one million data are added successfully!
        if (ints.length > 0) {
            System.out.println("One million records have been added successfully!!");
        }
    } catch (SQLException throwables) {
        throwables.printStackTrace();
    finally
        BaseDao.closeAll(conn, ps); // Call the static tool class just written to release resources}
    long end = System.currentTimeMillis(); // Get the system time again System.out.println("Time taken: " + (end - start) / 1000 + "seconds"); // Subtracting the two times is the time taken to execute the method}

Finally, let's run and see the effect:

Hey, the duration here is more than 10 seconds, the equipment is not good enough, I hope you understand~

<!--mysql-connector-java dependency used to connect to the database-->
<dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
    <version>8.0.27</version>
</dependency>

PS: It will be faster after adding threads, and examples will be given in subsequent articles.

This is the end of this article about how to insert one million records into MySQL database within 10 seconds. For more information about how to insert one million records into MySQL, please search for previous articles on 123WORDPRESS.COM or continue to browse the following related articles. I hope you will support 123WORDPRESS.COM in the future!

You may also be interested in:
  • MySQL automatically inserts millions of simulated data operation code
  • 4 ways to optimize MySQL queries for millions of data
  • MySQL single table million data records paging performance optimization skills
  • How to quickly insert millions of test data in MySQL

<<:  Two ways to remove the 30-second ad code from Youku video

>>:  Example of using CSS3 to create Pikachu animated wallpaper

Recommend

Automatic backup of MySQL database using shell script

Automatic backup of MySQL database using shell sc...

JavaScript deshaking and throttling examples

Table of contents Stabilization Throttling: Anti-...

Tutorial on setting up scheduled tasks to backup the Oracle database under Linux

1. Check the character set of the database The ch...

How to check PCIe version and speed in Linux

PCIE has four different specifications. Let’s tak...

Deploy Confluence with Docker

1. Environmental requirements 1. Docker 17 and ab...

About Generics of C++ TpeScript Series

Table of contents 1. Template 2. Generics 3. Gene...

Example analysis of mysql user rights management

This article describes the MySQL user rights mana...

MySQL 8.0.20 Installation Tutorial with Pictures and Text (Windows 64-bit)

1: Download from mysql official website https://d...

MySQL scheduled full database backup

Table of contents 1. MySQL data backup 1.1, mysql...

Example of how to configure nginx to implement SSL

Environmental Description Server system: Ubuntu 1...

MySQL 8.0.18 installation tutorial under Windows (illustration)

Download Download address: https://dev.mysql.com/...

JavaScript to implement simple carousel chart most complete code analysis (ES5)

This article shares the specific code for JavaScr...

MySQL online log library migration example

Let me tell you about a recent case. A game log l...