How to use Maxwell to synchronize MySQL data in real time

How to use Maxwell to synchronize MySQL data in real time

About Maxwell

Maxwell is a daemon written in Java that can read MySQL binlog in real time and write row updates in JSON format to Kafka, RabbitMq, Redis, etc. With MySQL incremental data stream, there are many usage scenarios, such as real-time synchronization of data to cache, synchronization of data to ElasticSearch, data migration, etc.

maxwell official website: http://maxwells-daemon.io
Maxwell source code: https://github.com/zendesk/maxwell

Configuration and use of Maxwell

Maxwell depends on Java SDK, so you need to configure the JDK environment first.

1. Download the Maxwell installation package

root@xxx maxwell]# pwd
/usr/local/maxwell
[root@xxx maxwell]# wget https://github.com/zendesk/maxwell/releases/download/v1.19.5/maxwell-1.19.5.tar.gz
[root@xxx maxwell]# tar zxvf maxwell-1.19.5.tar.gz 
[root@xxx maxwell]# cd maxwell-1.19.5

2. Configure MySQL and open MySQL binlog log

[root@xxx mysql]# vi /usr/local/mysql/my.cnf 

[mysqld]
log-bin=mysql-bin #add this line binlog-format=ROW #select row mode server_id=1 #randomly specify a string that cannot be the same as the name of other machines in the cluster. If there is only one machine, you can specify it at will

Restart the mysql service, log in to mysql, and view the binlog log mode

mysql> show variables like '%log_bin%'
+---------------------------------+-------------------------------+
| Variable_name | Value |
+---------------------------------+-------------------------------+
| log_bin | ON |
| log_bin_basename | /data/mysqldb/mysql-bin |
| log_bin_index | /data/mysqldb/mysql-bin.index |
| log_bin_trust_function_creators | OFF |
| log_bin_use_v1_row_events | OFF |
| sql_log_bin | ON |
+---------------------------------+-------------------------------+
6 rows in set (0.11 sec)

Maxwell needs permission to store state in the database specified by the schema_database option (the default database name is maxwell), so you need to give it permission in advance:

#Create a user yhrepl with synchronized data
mysql> create user 'yhrepl'@'*' identified by 'scgaopan'; 
Query OK, 0 rows affected (0.10 sec)

#This user yhrepl must have the permission to operate the database table that needs to be synchronized mysql> grant all privileges on test.* to 'yhrepl'@'%' identified by 'scgaopan'; 
Query OK, 0 rows affected (0.13 sec)

#Give yhrepl the permission to synchronize datamysql> grant select,replication client,replication slave on *.* to 'yhrepl'@'%' identified by 'scgaopan';
Query OK, 0 rows affected (0.10 sec)
# Maxwell needs permission to store state in the database specified by the schema_database option (the default database name is maxwell)
mysql> grant all privileges on maxwell.* to 'yhrepl'@'%' identified by 'scgaopan';
Query OK, 0 rows affected (0.09 sec)

3. Start Maxwell, mainly introducing the actual combat of writing data to rabbitmq:

[root@xxx maxwell-1.19.5]# vi /usr/local/maxwell/maxwell-1.19.5/config.properties
#Log level log_level=DEBUG

producer=rabbitmq
daemon=true

#Monitored database, mysql user must have permission to read binlog and create new database tables host=47.105.110.xxx
user=yhrepl
password=scgaopan

output_nulls=true
jdbc_options=autoReconnet=true

#Which tables in the database are monitored? filter=exclude: *.*,include: test.AA

#replica_server_id and client_id unique identifier, used for cluster deployment replica_server_id=64
client_id=test-id

#metrics_type=http
#metrics_slf4j_interval=60
#http_port=8111
#http_diagnostic=true # default false

#rabbitmq
rabbitmq_host=47.105.110.xxx
rabbitmq_port=5672
rabbitmq_user=guest
rabbitmq_pass=guest
rabbitmq_virtual_host=/
rabbitmq_exchange=maxwell
rabbitmq_exchange_type=topic
rabbitmq_exchange_durable=false
rabbitmq_exchange_autodelete=false
rabbitmq_routing_key_template=%db%.%table%
rabbitmq_message_persistent=false
rabbitmq_declare_exchange=true

Launch Maxwell:

[root@xxx maxwell-1.19.5]# ./bin/maxwell
#Can be started in the background [root@xxx maxwell-1.19.5]# nohub ./bin/maxwell &

If the startup is successful, the maxwell library will be automatically generated. The library records the status of maxwell synchronization, the ID of the last synchronization, and other information. After the main library fails or the synchronization is abnormal, as long as the maxwell library exists, the next synchronization will be based on the ID of the last synchronization. If the maxwell library is not generated or an error is reported, the mysql user permissions configured in config.properties may be insufficient.

RabbitMQ Operation

RabbitMQ operation, after starting Maxwell, a Maxwell exchange is generated

However, the binding of the corresponding queue, exchange and queue needs to be implemented by the user.

Create a maxwell-test queue:

Bind the queue to the exchange:

Note that the Routing key here is case sensitive.

Modify a record in the database and you can see that there is a record in the maxwell-test queue.

Full synchronization

Use the maxwell-bootstrap command

./bin/maxwell-bootstrap --database xhd --table xhd-sso --host 127.0.0.1 --user xiehd --password xiehd2018 --client_id maxwell_dev

Synchronize all data in the xhd.xhd-sso table and specify maxwell with client_id maxwell_dev to perform synchronization

The previous command is open first, and then start maxwell with client_id=maxwell_dev

./bin/maxwell --client_id maxwell_dev

Wait for the execution to complete

The above is the details of how to use Maxwell to synchronize MySQL data in real time. For more information about using Maxwell to synchronize MySQL data, please pay attention to other related articles on 123WORDPRESS.COM!

You may also be interested in:
  • Python implements a script to synchronize MySQL specified table incremental data to clickhouse
  • Detailed explanation of the actual process of master-slave synchronization of MySQL database
  • Two ways of storing scrapy data in MySQL database (synchronous and asynchronous)
  • Mysql master/slave database synchronization configuration and common errors
  • KTL tool realizes the method of synchronizing data from MySQL to MySQL
  • Tutorial on how to synchronize MySQL data to ElasticSearch using Python
  • Master-slave synchronization configuration and read-write separation of MySQL database
  • Steps to synchronize MongoDB data to MySQL using node.js
  • Master-slave synchronous replication configuration of MySQL database under Linux
  • PHP uses SWOOLE extension to synchronize MySQL data

<<:  The difference between shtml and html

>>:  How to use CSS media query aspect-ratio less

Recommend

Linux remote login implementation tutorial analysis

Linux is generally used as a server, and the serv...

JavaScript imitates Jingdong magnifying glass special effects

This article shares the specific code of JavaScri...

Detailed explanation of Nginx access restriction configuration

What is Nginx access restriction configuration Ng...

Introduction to Linux common hard disk management commands

Table of contents 1. df command 2. du command 3. ...

Best Practices Guide for Storing Dates in MySQL

Table of contents Preface Do not use strings to s...

Detailed explanation of data type issues in JS array index detection

When I was writing a WeChat applet project, there...

A brief discussion on the mysql execution process and sequence

Table of contents 1:mysql execution process 1.1: ...

How to use JavaScript to get the most repeated characters in a string

Table of contents topic analyze Objects of use So...

Solution to prevent caching in pages

Solution: Add the following code in <head>: ...

Solution to the problem of failure to insert emoji expressions into MySQL

Preface I always thought that UTF-8 was a univers...

Specific use of Linux dirname command

01. Command Overview dirname - strip non-director...

A complete record of a Mysql deadlock troubleshooting process

Preface The database deadlocks I encountered befo...

Summary of Vue's monitoring of keyboard events

Key Modifiers When listening for keyboard events,...

Solve the problem of docker container exiting immediately after starting

Recently I was looking at how Docker allows conta...