[Question posted by a user on YugabyteDB Community Slack]
I deployed a 3 master + 3 tserver cluster on 3 different regions. I am trying to set a preferred region.
Even though each master is in a different region, I had to use https://docs.yugabyte.com/latest/admin/yb-admin/#modify-placement-info so that is appeared in the config.
I then used : https://docs.yugabyte.com/latest/admin/yb-admin/#set-preferred-zones to set the preferred zone.
But the tablets are not rebalancing the leaders. Is there anything else to do ?
sample yb-tserver.conf
/home/yugabyte/bin/yb-tserver \
--fs_data_dirs=/mnt/disk0 \
--tserver_master_addrs={yb-master-0.yb-masters.yugabyte1.svc.cluster.local:7100,nyzks901i:29600},ldzks449i:29605,sgzks449i:29601 \
--placement_region=nyz-core-prod \
--use_private_ip=never \
--server_broadcast_addresses=nyzks902i:29500 \
--metric_node_name=yb-tserver-0 \
--memory_limit_hard_bytes=3649044480 \
--stderrthreshold=0 --num_cpus=0 \
--undefok=num_cpus,enable_ysql \
--rpc_bind_addresses=yb-tserver-0.yb-tservers.yugabyte1.svc.cluster.local \
--webserver_interface=0.0.0.0 \
--enable_ysql=true \
--pgsql_proxy_bind_address=0.0.0.0:5433 \
--cql_proxy_bind_address=yb-tserver-0.yb-tservers.yugabyte1.svc.cluster.local
sample yb-master.conf
/home/yugabyte/bin/yb-tserver \
--fs_data_dirs=/mnt/disk0 \
--tserver_master_addrs={yb-master-0.yb-masters.yugabyte1.svc.cluster.local:7100,nyzks901i:29600},ldzks449i:29605,sgzks449i:29601 \
--placement_region=nyz-core-prod \
--use_private_ip=never \
--server_broadcast_addresses=nyzks902i:29500 \
--metric_node_name=yb-tserver-0 \
--memory_limit_hard_bytes=3649044480 \
--stderrthreshold=0 \
--num_cpus=0 \
--undefok=num_cpus,enable_ysql \
--rpc_bind_addresses=yb-tserver-0.yb-tservers.yugabyte1.svc.cluster.local \
--webserver_interface=0.0.0.0 \
--enable_ysql=true \
--pgsql_proxy_bind_address=0.0.0.0:5433 \
--cql_proxy_bind_address=yb-tserver-0.yb-tservers.yugabyte1.svc.cluster.local
Even creating a new table results in errors:
cur.execute(
... """
... CREATE TABLE employee (id int PRIMARY KEY,
... name varchar,
... age int,
... language varchar)
... """)
Traceback (most recent call last):
File "<stdin>", line 7, in <module>
psycopg2.errors.InternalError_: Invalid argument: Invalid table definition: Timed out waiting for Table Creation
And following yb-tserver logs after the error above:
W0806 15:00:55.004124 63 catalog_manager.cc:7475] Aborting the current task due to error: Invalid argument (yb/master/catalog_manager.cc:7623): An error occurred while selecting replicas for tablet 0a503b0b9196425c956a8b9939b2c370: Invalid argument (yb/master/catalog_manager.cc:7623): Not enough tablet servers in the requested placements. Need at least 3, have 1: Not enough tablet servers in the requested placements. Need at least 3, have 1
E0806 15:00:55.004171 63 catalog_manager_bg_tasks.cc:142] Error processing pending assignments, aborting the current task: Invalid argument (yb/master/catalog_manager.cc:7623): An error occurred while selecting replicas for tablet 0a503b0b9196425c956a8b9939b2c370: Invalid argument (yb/master/catalog_manager.cc:7623): Not enough tablet servers in the requested placements. Need at least 3, have 1: Not enough tablet servers in the requested placements. Need at least 3, have 1
I0806 15:00:55.139647 2352 ysql_transaction_ddl.cc:46] Verifying Transaction { transaction_id: bb93b749-6b57-41b6-8f50-7461a07dc254 isolation: SNAPSHOT_ISOLATION status_tablet: 4069e18783a747bea31895b3ab6c69f6 priority: 1756854571847405073 start_time: { physical: 1628261964493502 } }
I0806 15:00:55.295994 53 ysql_transaction_ddl.cc:77] TransactionReceived: OK : status: PENDING
status_hybrid_time: 6669361378174894079
propagated_hybrid_time: 6669361378174910464
I0806 15:00:55.296051 53 ysql_transaction_ddl.cc:97] Got Response for { transaction_id: bb93b749-6b57-41b6-8f50-7461a07dc254 isolation: SNAPSHOT_ISOLATION status_tablet: 4069e18783a747bea31895b3ab6c69f6 priority: 1756854571847405073 start_time: { physical: 1628261964493502 } }: status: PENDING
status_hybrid_time: 6669361378174894079
propagated_hybrid_time: 6669361378174910464
W0806 15:00:55.310122 66 master_service_base-internal.h:39] Unknown master error in status: Invalid argument (yb/master/catalog_manager.cc:7623): An error occurred while selecting replicas for tablet 0a503b0b9196425c956a8b9939b2c370: Invalid argument (yb/master/catalog_manager.cc:7623): Not enough tablet servers in the requested placements. Need at least 3, have 1: Not enough tablet servers in the requested placements. Need at least 3, have 1
I0806 15:00:55.496171 2352 ysql_transaction_ddl.cc:46] Verifying Transaction { transaction_id: bb93b749-6b57-41b6-8f50-7461a07dc254 isolation: SNAPSHOT_ISOLATION status_tablet: 4069e18783a747bea31895b3ab6c69f6 priority: 1756854571847405073 start_time: { physical: 1628261964493502 } }
I0806 15:00:55.652542 67 ysql_transaction_ddl.cc:77] TransactionReceived: OK : status: PENDING
In addition to Dorian's answer: please show and validate the full placement info. You can do this in the following way:
The master placement info can be found using curl http://MASTER-ADDRESS:7000/api/v1/masters | jq.
The tserver placement info can be found on http://MASTER-ADDRESS:7000/tablet-servers (via a browser).
The preferred zones overview using curl http://MASTER-ADDRESS:7000/cluster-config | jq
That way, it should be easy to see and validate if everything is set as it should.
You also need the --placement_cloud=cloud and --placement_zone=rack1 arguments to your yb-tserver processes (to match what you passed to modify_placement_info step). Not just the --placement_region gflag.
Otherwise, the create table step doesn't find matching TServers to meet the desired placement for the table.
I have two tables finance.movies and finance.dummytable_3.
Both has been created using Spark SQL and their meta information is same
Both has two files
The total file size of finance.movies is 312363 while that of finance.dummytable_3 is 1209
But when I execute a simple "select *" on both the tables, finance.dummytable_3 whose total file size is less execute 2 jobs while finance.movies executes only one job. Screenshot follows.
hive> describe formatted finance.movies;
OK
# col_name data_type comment
movieid int
title_movie string
genres string
# Detailed Table Information
Database: finance
Owner: root
CreateTime: Thu Apr 16 02:18:51 EDT 2020
LastAccessTime: UNKNOWN
Protect Mode: None
Retention: 0
Location: hdfs://test.com:8020/GenNext_Finance_DL/Finance.db/movies
Table Type: MANAGED_TABLE
Table Parameters:
numFiles 2
spark.sql.create.version 2.3.0.2.6.5.0-292
spark.sql.sources.provider parquet
spark.sql.sources.schema.numParts 1
spark.sql.sources.schema.part.0 {\"type\":\"struct\",\"fields\":[{\"name\":\"movieid\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"title_movie\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"genres\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}
totalSize 312363
transient_lastDdlTime 1587017931
# Storage Information
SerDe Library: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe
InputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat
Compressed: No
Num Buckets: -1
Bucket Columns: []
Sort Columns: []
Storage Desc Params:
path hdfs://test.com:8020/GenNext_Finance_DL/Finance.db/movies
serialization.format 1
Time taken: 3.877 seconds, Fetched: 35 row(s)
hive> describe formatted finance.dummytable_3;
OK
# col_name data_type comment
colname1 string
colname2 string
colname3 string
# Detailed Table Information
Database: finance
Owner: root
CreateTime: Wed Apr 15 02:40:18 EDT 2020
LastAccessTime: UNKNOWN
Protect Mode: None
Retention: 0
Location: hdfs://test.com:8020/GenNext_Finance_DL/Finance.db/dummytable_3
Table Type: MANAGED_TABLE
Table Parameters:
numFiles 2
spark.sql.create.version 2.3.0.2.6.5.0-292
spark.sql.sources.provider parquet
spark.sql.sources.schema.numParts 1
spark.sql.sources.schema.part.0 {\"type\":\"struct\",\"fields\":[{\"name\":\"colName1\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"colName2\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}},{\"name\":\"colName3\",\"type\":\"string\",\"nullable\":true,\"metadata\":{}}]}
totalSize 1209
transient_lastDdlTime 1586932818
# Storage Information
SerDe Library: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe
InputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat
Compressed: No
Num Buckets: -1
Bucket Columns: []
Sort Columns: []
Storage Desc Params:
path hdfs://test.com:8020/GenNext_Finance_DL/Finance.db/dummytable_3
serialization.format 1
Time taken: 0.373 seconds, Fetched: 35 row(s)
When I execute the following query the number of jobs being executed is 2
spark-sql --master yarn --name spark_load_test -e "select * from finance.dummytable_3 limit 2;"
While when I execute the following query, the number of job being executed is just one. Despite both the tables having two files and identical struture.
spark-sql --master yarn --name spark_load_test -e "select * from finance.movies limit 2;"
Configurations:-
3 node cluster
Node 1 --> 172.27.21.16(Seed Node)
Node 2 --> 172.27.21.18
Node 3 --> 172.27.21.19
cassandra.yaml paramters for all the nodes:-
1) seeds: "172.27.21.16"
2) write_request_timeout_in_ms: 5000
3) listen_address: 172.27.21.1(6,8,9)
4) rpc_address: 172.27.21.1(6,8,9)
This is my code.yaml file:-
keyspace: prutorStress3node
keyspace_definition: |
CREATE KEYSPACE prutorStress3node WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 2};
table: code
table_definition: |
CREATE TABLE code (
id timeuuid,
assignment_id bigint,
user_id text,
contents text,
save_type smallint,
save_time timeuuid,
PRIMARY KEY(assignment_id, id)
) WITH CLUSTERING ORDER BY (id DESC)
columnspec:
- name: assignment_id
population: uniform(1..100000) # Assignment IDs from range 1-100000
- name: id
cluster: fixed(100) # Each assignment id will have atmost 100 ids(maximum 100 codes allowed per student)
- name: user_id
size: fixed(50) # user_id comes from account table varchar(50)
- name: contents
size: uniform(1..2000)
- name: save_type
size: fixed(1) # Generated values between 1 to 4
population: uniform(1..4)
- name: save_time
insert:
partitions: fixed(1) # The number of partitions to update per batch
select: fixed(1)/100 # The ratio of rows each partition should insert as a proportion of the total possible rows for the partition (as defined by the clustering distribution columns
batchtype: UNLOGGED # The type of CQL batch to use. Either LOGGED/UNLOGGED
queries:
query1:
cql: select id,contents,save_type,save_time from code where assignment_id = ?
fields: samerow
query2:
cql: select id,contents,save_type,save_time from code where assignment_id = ? LIMIT 10
fields: samerow
query3:
cql: SELECT contents FROM code WHERE id = ? # Create index for this query
fields: samerow
Following is my script which I am running from 172.27.21.17(A node on the same network but in a different cluster):-
#!/bin/bash
echo '***********************************'
echo Deleting old data from prutorStress3node
echo '***********************************'
cqlsh -e "DROP KEYSPACE prutorStress3node" 172.27.21.16
sleep 120
echo '***********************************'
echo Creating fresh data
echo '***********************************'
/bin/cassandra-stress user profile=./code.yaml n=1000000 no-warmup cl=quorum ops\(insert=1\) \
-rate threads=25 -node 172.27.21.16
sleep 120
Issue:-
The issue is that when I run the script just after deleting previous data and after bringing up those nodes, it works fine, but when I run it further, for each run, I hit the following error:-
Unable to find prutorStress3node.code
at org.apache.cassandra.stress.StressProfile.maybeLoadSchemaInfo(StressProfile.java:306)
at org.apache.cassandra.stress.StressProfile.maybeCreateSchema(StressProfile.java:273)
at org.apache.cassandra.stress.StressProfile.newGenerator(StressProfile.java:676)
at org.apache.cassandra.stress.StressProfile.printSettings(StressProfile.java:129)
at org.apache.cassandra.stress.settings.StressSettings.printSettings(StressSettings.java:383)
at org.apache.cassandra.stress.Stress.run(Stress.java:95)
at org.apache.cassandra.stress.Stress.main(Stress.java:62)
In the file StressProfile.java ,I saw that table metadata is being populated to NULL. I tried to make sense of the stack trace, but was not able to make anything of it. Please give me some directions as to what might have gone wrong?
Giving keyspace and table name in small case solved the issue for me. Previously i was giving mixed case and i was getting the same error.
I'm using the spark-cloudant library 1.6.3 that is installed by default with the spark service.
I'm trying to save some data to Cloudant:
val df = getTopXRecommendationsForAllUsers().toDF.filter( $"_1" > 6035)
println(s"Saving ${df.count()} ratings to Cloudant: " + new Date())
println(df.show(5))
val timestamp: Long = System.currentTimeMillis / 1000
val dbName: String = s"${destDB.database}_${timestamp}"
df.write.mode("append").json(s"${dbName}.json")
val dfWriter = df.write.format("com.cloudant.spark")
dfWriter.option("cloudant.host", destDB.host)
if (destDB.username.isDefined && destDB.username.get.nonEmpty) dfWriter.option("cloudant.username", destDB.username.get)
if (destDB.password.isDefined && destDB.password.get.nonEmpty) dfWriter.option("cloudant.password", destDB.password.get)
dfWriter.save(dbName)
However, I hit the error:
Starting getTopXRecommendationsForAllUsers: Sat Dec 24 08:50:11 CST 2016
Finished getTopXRecommendationsForAllUsers: Sat Dec 24 08:50:11 CST 2016
Saving 6 ratings to Cloudant: Sat Dec 24 08:50:17 CST 2016
+----+--------------------+
| _1| _2|
+----+--------------------+
|6036|[[6036,2503,4.395...|
|6037|[[6037,572,4.5785...|
|6038|[[6038,1696,4.894...|
|6039|[[6039,572,4.6854...|
|6040|[[6040,670,4.6820...|
+----+--------------------+
only showing top 5 rows
()
Use connectorVersion=1.6.3, dbName=recommendationdb_1482591017, indexName=null, viewName=null,jsonstore.rdd.partitions=5, + jsonstore.rdd.maxInPartition=-1,jsonstore.rdd.minInPartition=10, jsonstore.rdd.requestTimeout=900000,bulkSize=20, schemaSampleSize=1
Name: org.apache.spark.SparkException
Message: Job aborted due to stage failure: Task 2 in stage 642.0 failed 10 times, most recent failure: Lost task 2.9 in stage 642.0 (TID 409, yp-spark-dal09-env5-0049): java.lang.RuntimeException: Database recommendationdb_1482591017: nothing was saved because the number of records was 0!
at com.cloudant.spark.common.JsonStoreDataAccess.saveAll(JsonStoreDataAccess.scala:187)
I know there is data because I also save it to files:
! cat recommendationdb_1482591017.json/*
{"_1":6036,"_2":[{"user":6036,"product":2503,"rating":4.3957030284620355},{"user":6036,"product":2019,"rating":4.351395783537379},{"user":6036,"product":1178,"rating":4.3373212302468165},{"user":6036,"product":923,"rating":4.3328207761734605},{"user":6036,"product":922,"rating":4.320787353937724},{"user":6036,"product":750,"rating":4.307312349612301},{"user":6036,"product":53,"rating":4.304341611330176},{"user":6036,"product":858,"rating":4.297961629128419},{"user":6036,"product":1212,"rating":4.285360675560061},{"user":6036,"product":1423,"rating":4.275255129149407}]}
{"_1":6037,"_2":[{"user":6037,"product":572,"rating":4.578508339835482},{"user":6037,"product":858,"rating":4.247809350206506},{"user":6037,"product":904,"rating":4.1222486445799404},{"user":6037,"product":527,"rating":4.117342524702621},{"user":6037,"product":787,"rating":4.115781026855997},{"user":6037,"product":2503,"rating":4.109861422105844},{"user":6037,"product":1193,"rating":4.088453520710152},{"user":6037,"product":912,"rating":4.085139017248665},{"user":6037,"product":1221,"rating":4.084368219857013},{"user":6037,"product":1207,"rating":4.082536396283374}]}
{"_1":6038,"_2":[{"user":6038,"product":1696,"rating":4.894442132848873},{"user":6038,"product":2998,"rating":4.887752985607918},{"user":6038,"product":2562,"rating":4.740442462948304},{"user":6038,"product":3245,"rating":4.7366090605162094},{"user":6038,"product":2609,"rating":4.736125582066063},{"user":6038,"product":1669,"rating":4.678373819044571},{"user":6038,"product":572,"rating":4.606132758047402},{"user":6038,"product":1493,"rating":4.577140478430046},{"user":6038,"product":745,"rating":4.56568047928448},{"user":6038,"product":213,"rating":4.546054686400765}]}
{"_1":6039,"_2":[{"user":6039,"product":572,"rating":4.685425482619273},{"user":6039,"product":527,"rating":4.291256016077275},{"user":6039,"product":904,"rating":4.27766400846558},{"user":6039,"product":2019,"rating":4.273486883864949},{"user":6039,"product":2905,"rating":4.266371181044469},{"user":6039,"product":912,"rating":4.26006044096224},{"user":6039,"product":1207,"rating":4.259935289367192},{"user":6039,"product":2503,"rating":4.250370780277651},{"user":6039,"product":1148,"rating":4.247288578998062},{"user":6039,"product":745,"rating":4.223697008637559}]}
{"_1":6040,"_2":[{"user":6040,"product":670,"rating":4.682008703927743},{"user":6040,"product":3134,"rating":4.603656534071515},{"user":6040,"product":2503,"rating":4.571906881428182},{"user":6040,"product":3415,"rating":4.523567737705732},{"user":6040,"product":3808,"rating":4.516778146579665},{"user":6040,"product":3245,"rating":4.496176019230939},{"user":6040,"product":53,"rating":4.491020821805015},{"user":6040,"product":668,"rating":4.471757243976877},{"user":6040,"product":3030,"rating":4.464674231353673},{"user":6040,"product":923,"rating":4.446195112198678}]}
{"_1":6042,"_2":[{"user":6042,"product":3389,"rating":3.331488167984286},{"user":6042,"product":572,"rating":3.3312810949271903},{"user":6042,"product":231,"rating":3.2622287749148926},{"user":6042,"product":1439,"rating":3.0988533259613944},{"user":6042,"product":333,"rating":3.0859809743588706},{"user":6042,"product":404,"rating":3.0573976830913203},{"user":6042,"product":216,"rating":3.044620107397873},{"user":6042,"product":408,"rating":3.038302525994588},{"user":6042,"product":2411,"rating":3.0190834747311244},{"user":6042,"product":875,"rating":2.9860048032439095}]}
This is a defect with spark-cloudant 1.6.3 that is fixed with 1.6.4. The pull request is https://github.com/cloudant-labs/spark-cloudant/pull/61
The answer is to upgrade to spark-cloudant 1.6.4. See this answer if you are trying to do that on the IBM Bluemix Spark Service: Spark-cloudant package 1.6.4 loaded by %AddJar does not get used by notebook