Skip to main content

Striim for BigQuery Documentation

Set up your MySQL Source

You must perform all setup tasks appropriate for your source environment before you can create a pipeline. If any of these tasks are not complete, the corresponding prerequisite checks will fail.

For all MySQL environments

An administrator with the necessary privileges must create a user for use by the adapter and assign it the necessary privileges:

CREATE USER 'striim' IDENTIFIED BY '******';
GRANT REPLICATION SLAVE ON *.* TO 'striim'@'%';
GRANT REPLICATION CLIENT ON *.* TO 'striim'@'%';
GRANT SELECT ON *.* TO 'striim'@'%';

  • The caching_sha2_password authentication plugin is not supported in this release. The mysql_native_password plugin is required.

  • The REPLICATION privileges must be granted on *.*. This is a limitation of MySQL.

  • You may use any other valid name in place of striim. Note that by default MySQL does not allow remote logins by root.

  • Replace ****** with a secure password.

  • You may narrow the SELECT statement to allow access only to those tables needed by your application. In that case, if other tables are specified in the source properties for the initial load application, Striim will return an error that they do not exist.

On-premise MySQL setup

Striim reads from the MySQL binary log. If your MySQL server is using replication, the binary log is enabled, otherwise it may be disabled.

For on-premise MySQL, the property name for enabling the binary log, whether it is one or off by default, and how and where you change that setting vary depending on the operating system and your MySQL configuration, so for instructions see the binary log documentation for the version of MySQL you are running.

If the binary log is not enabled, Striim's attempts to read it will fail with errors such as the following:

2016-04-25 19:05:40,377 @ -WARN hz._hzInstance_1_striim351_0423.cached.thread-2 
com.webaction.runtime.Server.startSources (Server.java:2477) Failure in Starting 
Sources.
java.lang.Exception: Problem with the configuration of MySQL
Row logging must be specified.
Binary logging is not enabled.
The server ID must be specified.
Add --binlog-format=ROW to the mysqld command line or add binlog-format=ROW to your 
my.cnf file
Add --bin-log to the mysqld command line or add bin-log to your my.cnf file
Add --server-id=n where n is a positive number to the mysqld command line or add 
server-id=n to your my.cnf file
        at com.webaction.proc.MySQLReader_1_0.checkMySQLConfig(MySQLReader_1_0.java:605) ...

Amazon Aurora for MySQL setup

See How do I enable binary logging for my Amazon Aurora MySQL cluster?.

Amazon RDS for MySQL setup

  1. Create a new parameter group for the database (see Creating a DB Parameter Group).

  2. Edit the parameter group, change binlog_format to row and binlog_row_image to full, and save the parameter group (see Modifying Parameters in a DB Parameter Group).

  3. Reboot the database instance (see Rebooting a DB Instance).

  4. In a database client, enter the following command to set the binlog retention period to one week:

    call mysql.rds_set_configuration('binlog retention hours', 168);

Azure Database for MySQL setup

You must create a read replica to enable binary logging. See Read replicas in Azure Database for MySQL.

In this section: