low loading performance while batch inserting rows into Spanner using jdbc - google-cloud-spanner

Question

Background: I am trying to load TSV-formatted data files (dumped from MySQL database) into a GCP Spanner table.
client library: the official Spanner JDBC dependency v1.15.0
table schema: two string-typed columns and ten int-typed columns
GCP Spanner instance: configured as multi-region nam6 with 5 nodes
My loading program runs in GCP VM and is the exclusive client accessing the Spanner instance. Auto-commit is enabled. Batch insertion is the only DML operation executed by my program and the batch size is around 1500. In each commit, it fully uses up the mutation limit, which is 20000. And at the same time, the commit size is below 5MB (the values of two string-typed columns are small-sized). Rows are partitioned based on the first column of the primary key so that each commit can be sent to very few partitions for better performance.
With all of the configuration and the optimization above, the insertion rate is only around 1k rows per second. This really disappoints me because I have more than 800million rows to insert. I did notice that the official doc mentioned the approx. peak write (QPS total) is 1800 for the multi-region Spanner instance.
So I have two questions here:
Considering such low peak write QPS, does it mean GCP doesn't expect or doesn't support customers to migrate large datasets to the multi-region Spanner instance?
I was seeing the high read latency from Spanner monitoring. I don't have any read requests. My guess is that whiling writing rows Spanner needs to first read and check whether a row with the same primary key exists. If my guess is right, why it takes so much time? If not, could I get any guidance on how these read operations happen?

Answer 1

It's not quite clear to me exactly how you are setting up the client application that is loading the data. My initial impression is that your client application may not be executing enough transactions in parallel. You should normally be able to insert significantly more than 1,000 rows/second, but it would require that you do execute multiple transactions in parallel, possibly from multiple VM's. I used the following simple example to test the load throughput from my local machine to a single node Spanner instance, and that gave me a throughput of approx 1,500 rows/second.
A multi-node setup using a client application running in one or more VM's in the same network region as your Spanner instance should be able to achieve higher volumes than that.
import com.google.api.client.util.Base64;
import com.google.common.base.Stopwatch;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.util.Random;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
public class TestJdbc {
public static void main(String[] args) {
final int threads = 512;
ExecutorService executor = Executors.newFixedThreadPool(threads);
watch = Stopwatch.createStarted();
for (int i = 0; i < threads; i++) {
executor.submit(new InsertRunnable());
}
}
static final AtomicLong rowCount = new AtomicLong();
static Stopwatch watch;
static final class InsertRunnable implements Runnable {
#Override
public void run() {
try (Connection connection =
DriverManager.getConnection(
"jdbc:cloudspanner:/projects/my-project/instances/my-instance/databases/my-db")) {
while (true) {
try (PreparedStatement ps =
connection.prepareStatement("INSERT INTO Test (Id, Col1, Col2) VALUES (?, ?, ?)")) {
for (int i = 0; i < 150; i++) {
ps.setLong(1, rnd.nextLong());
ps.setString(2, randomString(100));
ps.setString(3, randomString(100));
ps.addBatch();
rowCount.incrementAndGet();
}
ps.executeBatch();
}
System.out.println("Rows inserted: " + rowCount);
System.out.println("Rows/second: " + rowCount.get() / watch.elapsed(TimeUnit.SECONDS));
}
} catch (SQLException e) {
throw new RuntimeException(e);
}
}
private final Random rnd = new Random();
private String randomString(int maxLength) {
byte[] bytes = new byte[rnd.nextInt(maxLength / 2) + 1];
rnd.nextBytes(bytes);
return Base64.encodeBase64String(bytes);
}
}
}
There are also a couple of other things that you could try to tune to get better results:
Reducing the number of rows per batch could yield better overall results.
If possible, using InsertOrUpdate mutation objects is a lot more efficient than using DML statements (see example below).
Example using Mutation instead of DML:
import com.google.api.client.util.Base64;
import com.google.cloud.spanner.Mutation;
import com.google.cloud.spanner.jdbc.CloudSpannerJdbcConnection;
import com.google.common.base.Stopwatch;
import com.google.common.collect.ImmutableList;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.Random;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicLong;
public class TestJdbc {
public static void main(String[] args) {
final int threads = 512;
ExecutorService executor = Executors.newFixedThreadPool(threads);
watch = Stopwatch.createStarted();
for (int i = 0; i < threads; i++) {
executor.submit(new InsertOrUpdateMutationRunnable());
}
}
static final AtomicLong rowCount = new AtomicLong();
static Stopwatch watch;
static final class InsertOrUpdateMutationRunnable implements Runnable {
#Override
public void run() {
try (Connection connection =
DriverManager.getConnection(
"jdbc:cloudspanner:/projects/my-project/instances/my-instance/databases/my-db")) {
CloudSpannerJdbcConnection csConnection = connection.unwrap(CloudSpannerJdbcConnection.class);
CloudSpannerJdbcConnection csConnection =
connection.unwrap(CloudSpannerJdbcConnection.class);
while (true) {
ImmutableList.Builder<Mutation> builder = ImmutableList.builder();
for (int i = 0; i < 150; i++) {
builder.add(
Mutation.newInsertOrUpdateBuilder("Test")
.set("Id")
.to(rnd.nextLong())
.set("Col1")
.to(randomString(100))
.set("Col2")
.to(randomString(100))
.build());
rowCount.incrementAndGet();
}
csConnection.write(builder.build());
System.out.println("Rows inserted: " + rowCount);
System.out.println("Rows/second: " + rowCount.get() / watch.elapsed(TimeUnit.SECONDS));
}
}
} catch (SQLException e) {
throw new RuntimeException(e);
}
}
private final Random rnd = new Random();
private String randomString(int maxLength) {
byte[] bytes = new byte[rnd.nextInt(maxLength / 2) + 1];
rnd.nextBytes(bytes);
return Base64.encodeBase64String(bytes);
}
}
}
The above simple example gives me a throughput of approx 35,000 rows/second without any further tuning.
ADDITIONAL INFORMATION 2020-08-21: The reason that mutation objects are more efficient than (batch) DML statements, is that DML statements are internally converted to read queries by Cloud Spanner, which are then used to create mutations. This conversion needs to be done for every DML statement in a batch, which means that a DML batch with 1,500 simple insert statements will trigger 1,500 (small) read queries and need to be converted to 1,500 mutations. This is most probably also the reason behind the read latency that you are seeing in your monitoring.
Would you otherwise mind sharing some more information on what your client application looks like and how many instances of it you are running?

Answer 2

With more than 800million rows to insert, and seeing that you are a Java programmer, can I suggest using Beam on Dataflow?
The spanner writer in Beam is designed to be as efficient as possible with its writes - grouping rows by a similar key, and batching them as you are doing. Beam on Dataflow can also use several worker VMs to execute multiple file reads and spanner writes in parallel...
With a multiregion spanner instance, you should be able to get approx 1800 rows per node per second insert speed (more if the rows are small and batched, as Knut's reply suggests) and with 5 spanner nodes, you can probably have between 10 and 20 importer threads running in parallel - whether using your importer program or using Dataflow.
(disclosure: I am the Beam SpannerIO maintainer)

Answer 3

Cloud Spanner has launched a new feature that greatly improves the performance of the use case here and enables more efficient data updates.
If the batch of DML queries have the same SQL text and are parameterized, similar to PreparedStatement(s) generated by JDBC client in this post, the queries in the batch are combined to execute a single server-side action to generate rows followed by another single server-side write action.
This reduces the number of server-side actions linearly by batch size leading to much improved latency and better throughput.
The improvement in latency ranges where better performance improvement is seen with bigger batch sizes. The feature is applied automatically in Batch DML APIs.
Official documentation of this performance optimization can be found here: https://cloud.google.com/spanner/docs/dml-best-practices#batch-dml