I need to schedule repair to run on the existing Cassandra twice a week, once I added the first schedule I can't add the second schedule due to the following error, is there any way to run the repair twice a week in reaper?
You cannot create multiple schedules for a table because running multiple repair sessions on the same table is not recommended.
If you want to setup different schedules for different keyspaces/tables, you need to explicitly specify the keyspace and/or table when setting up the schedule.
As a side note, the recommended repair interval is once every gc_grace_seconds. The default GCGS is 10 days (864000) so running repairs once a week is sufficient. Cheers!
Related
I have to test functionality manually where , if a background job fails then a record is added in a table having columns Exception Id, Job Id, Job Name, Exception, Method Name, Service Name, etc. columns. At present there are around 25 background jobs in the application and i want to test the impact of adding maximum records in the table. How should I do that? Or i have to manually fail jobs to add maximum record in the table (which is practically not possible, as the we can add unlimited records in the table)Is there any way?
I tried to manually fail the job, so that records can be added in the table, however, adding thousands of records manually is practically impossible.
How should i add maximum records in the table manually/ How should i test this scenario?
Can you try doing the following :
Create various Windows batch jobs covering the 25 background jobs.
Create Windows task scheduler such that it is continuously triggered. You may try scheduling the job execution on the same day or spread across multiple days.
What this will do is you no longer need to run the background jobs manually as the task scheduler will take care of this.
Currently, I'm running once a week manually nodetool repair (while no action is happening to the cassandra nodes (nothing is inserted, e.t.c.)). Just wondering, if I can run nodetool repair, while data is being inserted?
Secondly - Can I create a script in crontab, that automatically runs every week nodetool repair, and is ocne a week enough to run nodetool repair?
yes, if running one repair at a time you shouldn't be impacting normal usage. Instead of a cron job i would recommend using reaper (free and open source) for automating it. Would give you a bit more visibility and it handles things a bit better than just the default that running from nodetool provides.
Yes you can run repair while data is inserting it may impact your traffic so avoid this you can run repair table or key space wise.
Yes, you can run repair and schedule a crone for the same.
I am currently running Cassandra 3.0.9 in a 18 node config. We loaded quite a bit of data and now are running repairs against each node. My nodetool command is scripted to look like:
nodetool repair -j 4 -local -full
Using nodetool tpstats I see the 4 threads for repair but they are repairing very slowly. I have 1000's of repairs that are going to take weeks at this rate. The system log has repair items but also "Redistributing index summaries" listed as well. Is this what is causing my slowness? Is there a faster way to do this?
Repair can take a very long time, sometime days, sometime weeks. You might improve things with the following:
Run primary partition range repair (-pr) This will repair only the primary partition range of each node, which overall, will be faster (you still need to run a repair on each node, one at a time).
Using -j is not necessarily a big winner. Sure, you will repair multiple tables at a time, but you put much more load on your cluster, which can damage your latency.
You might want to prioritize repairing the keyspaces / tables that are most critical to your application.
Make sure you keep your node density reasonable. 1 to 2TB per node.
Focus repairing in priority the nodes that went down for more than 3 hours (assuming max_hint_window_in_ms is set to it's default value)
Focus repairing in priority the tables for which you create tombstones (DELETE statements)
I have a Cassandra 2.2.9 repair that has failed, and in this state the Cassandra metrics show about 70 repair tasks still pending. Cassandra should take care of retrying these failed tasks itself, but for whatever reason this time it has not.
The repairs take a long time, instead of running the whole repair again, can I see the chosen token ranges Cassandra uses for the repair so I can manually run the last few tasks instead?
One way I found is to search the logs at the time you started the repair - Cassandra spits out the ID and token range of each repair task it will attempt at the time you run the repair.
With Cassandra 2.2.9 I found grepping the logs for new session: will sync did the trick :)
We have a mysql server running which is serving application writes. To do some batch processing we have written a sync job to migrate data into cassandra cluster.
1. A daily sync job which transfers by updated timestamp for that day.
2. A complete sync job which transfers complete data, overriding existing ones.
Now there may be a possibility that the row was deleted from mysql, in that case using the above approach it will lie forever in cassandra.
To solve that problem we have given a TTL of 15 days for every row. So eventually it will get deleted, if it was not deleted then in next full sync the TTL will be over written again.
Its working fine as far as the use case is concerned but the issue is that in full sync complete data is over written and sstable is generated continuously with compactions happenning all the time, load averages shoot up with slowness and backup size increases (which could have been avoided).
Essentially we would want to replace the existing table data by new data but we dont want to truncate before starting the job but only after job completes.
Is there any way by which this can be solved other than creating a new table altogether and dropping past table when data is generated?
You can look at the double-run migration strategy I presented here: http://www.slideshare.net/doanduyhai/from-rdbms-to-cassandra-without-a-hitch
It has the advantage of allowing 100% uptime and possible rollback if things go wrong. The downside is the amount of work required in term of releases & codes