I have GPU slurm cluster , but , I want limit run time on specific node to all users in partition that include all servers (general partition for all nodes and users) , but I can't found how to change config file for this:
..
NodeName=beta4 CPUs=64 Gres=gpu:2080ti:8 Boards=1 SocketsPerBoard=2 CoresPerSocket=16 ThreadsPerCore=2 RealMemory=257825
PartitionName=public Nodes=beta1,beta2,beta4 Default=YES State=UP MaxTime=INFINITE PriorityTier=1 TRESBillingWeights="CPU=1.0,gres/gpu=10.0"
So , the quetion if I can set to all "public" users ,use node beta4 only 2 days and beta1,beta2 unlimited time?
Related
cassandra service (3.11.5) stops automatically after it starts/restart on AWS linux.
I have fresh installation of cassandra on new instance of AWS linux (t3.xlarge) and
sudo service cassandra start
or
sudo service cassandra restart
after 1 or 2 seconds, the service stop automatically. I looked into logs and I found these.
I am not sure, I havent change configs related to snitch and its always SimpleSnitch. I dont have any multiple cassandras. Just only on single EC2.
Logs
INFO [main] 2020-02-12 17:40:50,833 ColumnFamilyStore.java:426 - Initializing system.schema_aggregates
INFO [main] 2020-02-12 17:40:50,836 ViewManager.java:137 - Not submitting build tasks for views in keyspace system as storage service is not initialized
INFO [main] 2020-02-12 17:40:51,094 ApproximateTime.java:44 - Scheduling approximate time-check task with a precision of 10 milliseconds
ERROR [main] 2020-02-12 17:40:51,137 CassandraDaemon.java:759 - Cannot start node if snitch's data center (datacenter1) differs from previous data center (dc1). Please fix the snitch configuration, decommission and rebootstrap this node or use the flag -Dcassandra.ignore_dc=true.
Installation steps
sudo curl -OL https://www.apache.org/dist/cassandra/redhat/311x/cassandra-3.11.5-1.noarch.rpm
sudo rpm -i cassandra-3.11.5-1.noarch.rpm
sudo pip install cassandra-driver
export CQLSH_NO_BUNDLED=true
sudo chkconfig --levels 3 cassandra on
The issue is in your log file:
ERROR [main] 2020-02-12 17:40:51,137 CassandraDaemon.java:759 - Cannot start node if snitch's data center (datacenter1) differs from previous data center (dc1). Please fix the snitch configuration, decommission and rebootstrap this node or use the flag -Dcassandra.ignore_dc=true.
It seems that you started the cluster, stopped it and renamed the datacenter from dc1 to datacenter1.
In order to fix:
If no data is stored, delete the data directories
If data is stored, rename the datacenter back to dc1 in the config
I had the same problem , where cassandra service immediately stops after it was started.
in the cassandra configuration file located at /etc/cassandra/cassandra.yaml change the cluster_name to the previous one, like this:
...
# The name of the cluster. This is mainly used to prevent machines in
# one logical cluster from joining another.
cluster_name: 'dc1'
# This defines the number of tokens randomly assigned to this node on the ring
# The more tokens, relative to other nodes, the larger the proportion of data
...
Similar to slurm: use a control node also for computing.
I would like to use the frontend as an compute node. I made the following entries in slurm.conf
NodeName=gisc RealMemory=63000 Sockets=1 CoresPerSocket=8 ThreadsPerCore=2 State=UNKNOWN Weight=2
NodeName=c[0-2] RealMemory=126000 Sockets=1 CoresPerSocket=16 ThreadsPerCore=2 State=UNKNOWN Weight=1
PartitionName=normal Nodes=gisc,c[0-2] Default=YES MaxTime=INFINITE State=UP
And restarted both slurmd and slurmctld.
However, I always get no response from the frontend node which is proved by an asterix in the status.
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
normal* up infinite 1 idle* gisc
normal* up infinite 2 alloc c[0-1]
normal* up infinite 1 idle c2
Also, I cannot start slurmd on the frontend node. The logs do not help.
Could it be that slurmd and slurmctld are conflicting on the frontend node?
My /etc/hosts looks as follows
192.168.1.1 gisc.localdomain gisc gisc-eth0.localdomain gisc-eth0
### ALL ENTRIES BELOW THIS LINE WILL BE OVERWRITTEN BY WAREWULF ###
#
# See provision.conf for configuration paramaters
# Node Entry for node: c0 (ID=22)
192.168.1.2 c0.localdomain c0 c0-eth0.localdomain c0-eth0
# Node Entry for node: c1 (ID=23)
192.168.1.3 c1.localdomain c1 c1-eth0.localdomain c1-eth0
# Node Entry for node: c2 (ID=24)
192.168.1.4 c2.localdomain c2 c2-eth0.localdomain c2-eth0
facepalm The slurm-client library was missing on the frontend. Only the slurm-server library was installed...
Adding a new single token per nodes to existing datastax cluster and data transfer is not working. Process followed is mentioned below. Please update me if the process i followed is wrong.Thanks
We have 3 Single token range datastax nodes in our AWS EC2 Datacenter, both Search and Graph enabled. We are planning to add 3 more nodes into into our datacenter. We are currently using DseSimpleSnitch and Simple network topology for our keyspace.Also our current replication factor is 2.
Node 1 : 10.10.1.36
Node 2 : 10.10.1.46
Node 3 : 10.10.1.56
cat /etc/default/dse | grep -E 'GRAPH_ENABLED=|SOLR_ENABLED='
GRAPH_ENABLED=1
SOLR_ENABLED=1
Datacenter : SearchGraph
Address Rack Status State Load Owns Token
10.10.1.46 rack1 Up Normal 760.14 MiB ? -9223372036854775808
10.10.1.36 rack1 Up Normal 737.69 MiB ? -3074457345618258603
10.10.1.56 rack1 Up Normal 752.25 MiB ? 3074457345618258602
Step (1) For adding 3 new node into our datacenter first we changed our keyspace topology and snitch to network aware.
1)Changed the snitch.
cat /etc/dse/cassandra/cassandra.yaml | grep endpoint_snitch:
endpoint_snitch: GossipingPropertyFileSnitch
cat /etc/dse/cassandra/cassandra-rackdc.properties |grep -E 'dc=|rack='
dc=SearchGraph
rack=rack1
2)
(a) Shut down all the nodes, then restart them.
(b) Run a sequential repair and nodetool cleanup on each node.
3)Changed keyspace topology.
ALTER KEYSPACE tech_app1 WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'SearchGraph' : 2};
ALTER KEYSPACE tech_app2 WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'SearchGraph' : 2};
ALTER KEYSPACE tech_chat WITH REPLICATION = {'class' : 'NetworkTopologyStrategy', 'SearchGraph' : 2};
Reference : http://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsChangeKSStrategy.html , http://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsSwitchSnitch.html
Step (2) For updating token range and setting up new cassandra node, we follow below process.
1) Recalculate token range
root#ip-10-10-1-36:~# token-generator
DC #1:
Node #1: -9223372036854775808
Node #2: -6148914691236517206
Node #3: -3074457345618258604
Node #4: -2
Node #5: 3074457345618258600
Node #6: 6148914691236517202
2) Installed Datastax enterprise same version on new nodes.
3) Stopped the node service and and cleared the data.
4) (a) Assigned token range in following manner to new node.
Node 4: 10.10.2.96 Range: -2
Node 5: 10.10.2.97 Range: 3074457345618258600
Node 6: 10.10.2.86 Range: 6148914691236517202
4) (b) Configured cassandra.yaml on each new node:
Node 4 :
cluster_name: 'SearchGraph'
num_tokens: 1
initial_token: -2
parameters:
- seeds: "10.10.1.46, 10.10.1.56"
listen_address: 10.10.2.96
rpc_address: 10.10.2.96
endpoint_snitch: GossipingPropertyFileSnitch
Node 5 :
cluster_name: 'SearchGraph'
num_tokens: 1
initial_token: 3074457345618258600
parameters:
- seeds: "10.10.1.46, 10.10.1.56"
listen_address: 10.10.2.97
rpc_address: 10.10.2.97
endpoint_snitch: GossipingPropertyFileSnitch
Node 6 :
cluster_name: 'SearchGraph'
num_tokens: 1
initial_token: 6148914691236517202
parameters:
- seeds: "10.10.1.46, 10.10.1.56"
listen_address: 10.10.2.86
rpc_address: 10.10.2.86
endpoint_snitch: GossipingPropertyFileSnitch
5) Changed the snitch.
cat /etc/dse/cassandra/cassandra.yaml | grep endpoint_snitch:
endpoint_snitch: GossipingPropertyFileSnitch
cat /etc/dse/cassandra/cassandra-rackdc.properties |grep -E 'dc=|rack='
dc=SearchGraph
rack=rack1
6) Start DataStax Enterprise on each new node in two minutes intervals with consistent.rangemovement turned off:
JVM_OPTS="$JVM_OPTS -Dcassandra.consistent.rangemovement=false
7) After the new nodes are fully bootstrapped, used nodetool move to assign the new initial_token for existing nodes as per token recalculation done at step 4(a). Process done on each node one at a time.
On Node 1(10.10.1.36) : nodetool move -3074457345618258603
On Node 2(10.10.1.46) : nodetool move -9223372036854775808
On Node 3(10.10.1.56) : nodetool move 3074457345618258602
Datacenter: SearchGraph
Address Rack Status State Load Owns Token
10.10.1.46 rack1 Up Normal 852.93 MiB ? -9223372036854775808
10.10.1.36 rack1 Up Moving 900.12 MiB ? -3074457345618258603
10.10.2.96 rack1 UP Normal 465.02 KiB ? -2
10.10.2.97 rack1 Up Normal 109.16 MiB ? 3074457345618258600
10.10.1.56 rack1 Up Moving 594.49 MiB ? 3074457345618258602
10.10.2.86 rack1 Up Normal 663.94 MiB ? 6148914691236517202
Post Updated:
But we are getting following error while joining nodes.
AbstractSolrSecondaryIndex.java:1884 - Cannot find core chat.chat_history
AbstractSolrSecondaryIndex.java:1884 - Cannot find core chat.history
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.business_units
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.feeds
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.feeds_2
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.knowledegmodule
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.userdetails
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.userdetails_2
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.vault_details
AbstractSolrSecondaryIndex.java:1884 - Cannot find core search.workgroup
AbstractSolrSecondaryIndex.java:1884 - Cannot find core cloud.feeds
AbstractSolrSecondaryIndex.java:1884 - Cannot find core cloud.knowledgemodule
AbstractSolrSecondaryIndex.java:1884 - Cannot find core cloud.organizations
AbstractSolrSecondaryIndex.java:1884 - Cannot find core cloud.userdetails
AbstractSolrSecondaryIndex.java:1884 - Cannot find core cloud.vaults
AbstractSolrSecondaryIndex.java:1884 - Cannot find core cloud.workgroup
Node joining failed with following error :
ERROR [main] 2017-08-10 04:22:08,449 DseDaemon.java:488 - Unable to start DSE server.
com.datastax.bdp.plugin.PluginManager$PluginActivationException: Unable to activate plugin com.datastax.bdp.plugin.SolrContainerPlugin
Caused by: java.lang.IllegalStateException: Cannot find secondary index for core ekamsearch.userdetails_2, did you create it?
If yes, please consider increasing the value of the dse.yaml option load_max_time_per_core, current value in minutes is: 10
ERROR [main] 2017-08-10 04:22:08,450 CassandraDaemon.java:705 - Exception encountered during startup
java.lang.RuntimeException: com.datastax.bdp.plugin.PluginManager$PluginActivationException: Unable to activate plugin
Has anyone encountered these errors or warnings before?
Token Assign Issue ::
1) I had wrongly assigned token range in Step 4) (a). Assign token which
bisect or trisect the value which are generated using
"token-generator"
Node 4: 10.10.2.96 Range: -6148914691236517206
Node 5: 10.10.2.97 Range: -2
Node 6: 10.10.2.86 Range: 6148914691236517202
Note : We don't need to change the token range of existing nodes in data
center.No need to follow procedure in Step 7 which i have mentioned
above.
Solr Issue resolved : Cannot find cor ::
Increased load_max_time_per_core value in dse.yaml configuration file,
still i was receving the error.Finalys solved the issue
by following method
1) Started the new nodes as non-solr and wait for all cassandra data
to migrate to joining nodes.
2) Add the parameter auto_bootstrap: False directive to the
cassandra.yaml file
3) Re-start the same nodes after enabling solr. Changed parameter
SOLR_ENABLED=1 in /etc/default/dse
3) Re-index in all new joined nodes. I had to reloaded all core
required with the reindex=true and distributed=false parameters in
new joined nodes.
Ref : http://docs.datastax.com/en/archived/datastax_enterprise/4.0/datastax_enterprise/srch/srchReldCore.html
I'm trying to make glusterfs cluster with Heketi for Kubernetes persistent volumes. I have 3 nodes in gluster cluster:
heketi-cli node list
Id:242e801e6eeb7ec10acda60a409b5d98 Cluster:fd539c5d13b6229498c6c67ac491163d
Id:439fb090888a745633f9db6ac4d243b8 Cluster:fd539c5d13b6229498c6c67ac491163d
Id:5e9b7e5f3ec33c77c42437e89ca857a3 Cluster:fd539c5d13b6229498c6c67ac491163d
But when I try to provision a volume for Heketi database by using command:
heketi-cli setup-openshift-heketi-storage
I get an error:
Error: No space
But I have enough free space on my volumes:
Devices:
Id:931b4f87e3675368a4f737ed6862e0cf Name:/dev/sdb State:online Size (GiB):29 Used (GiB):0 Free (GiB):29
Devices:
Id:3a2a30b22ade4efca7949e9cc082b685 Name:/dev/sdb State:online Size (GiB):29 Used (GiB):0 Free (GiB):29
Devices:
Id:5d1b5c7b258c52569bff1e1c720015c5 Name:/dev/sdb State:online Size (GiB):29 Used (GiB):0 Free (GiB):29
What can be the reason for this strange behavior?
I'm sorry, I have found the reason. It's the count of gluster node, it should be equal to count of gluster instances in kubernetes. In previous turn I had only 3 gluster nodes and 4 gluster instances in kubernetes.
There can be a number of problems that lead to this error message. The 2 most common ones are:
You do not have the minimum of 3 nodes in your gluster cluster
The heketi-cli setup-openshift-heketi-storage command needs to create a volume for heketi's database. That volume is now 2GB by default but it used to 32GB(!) (see heketi issue #639). So depending on your heketi-cli version it may be trying to create a 32GB volume on your 29GB bricks. Nasty.
I suggest you look at the logs of heketi:
$ kubectl get pod -l name=heketi
NAME READY STATUS RESTARTS AGE
heketi-703226055-7g3hb 1/1 Running 0 18h
$ kubectl logs heketi-703226055-7g3hb -f
Heketi v3.0.0-111-gc5f0f58
[heketi] INFO 2017/02/14 22:17:53 Loaded kubernetes executor
...
I am restoring to a fresh new Cassandra 2.2.5 cluster consisting of 3 nodes.
Initial cluster health of the NEW cluster:
-- Address Load Tokens Owns Host ID Rack
UN 10.40.1.1 259.31 KB 256 ? d2b29b08-9eac-4733-9798-019275d66cfc uswest1adevc
UN 10.40.1.2 230.12 KB 256 ? 5484ab11-32b1-4d01-a5fe-c996a63108f1 uswest1adevc
UN 10.40.1.3 248.47 KB 256 ? bad95fe2-70c5-4a2f-b517-d7fd7a32bc45 uswest1cdevc
As part of the restore instructions in Datastax docs, i do the following on the new cluster:
1) cassandra stop on all of the three nodes one by one.
2) Edit cassandra.yaml for all of the three nodes with the backup'ed token ring information. [Step 2 from docs]
3) Remove the contents from /var/lib/cassandra/data/system/* [Step 4 from docs]
4) cassandra start on nodes 10.40.1.1, 10.40.1.2, 10.40.1.3 respectively.
Result:
10.40.1.1 restarts back successfully:
-- Address Load Tokens Owns Host ID Rack
UN 10.40.1.1 259.31 KB 256 ? 2d23add3-9eac-4733-9798-019275d125d3 uswest1adevc
But the second and the third nodes fail to restart stating:
java.lang.RuntimeException: A node with address 10.40.1.2 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.
at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:546) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:766) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:693) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:585) ~[apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:300) [apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:516) [apache-cassandra-2.2.5.jar:2.2.5]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:625) [apache-cassandra-2.2.5.jar:2.2.5]
INFO [StorageServiceShutdownHook] 2016-08-09 18:13:21,980 Gossiper.java:1449 - Announcing shutdown
java.lang.RuntimeException: A node with address 10.40.1.3 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.
...
Eventual cluster health:
-- Address Load Tokens Owns Host ID Rack
UN 10.40.1.1 259.31 KB 256 ? 2d23add3-9eac-4733-9798-019275d125d3 uswest1adevc
DN 10.40.1.2 230.12 KB 256 ? 6w2321ad-32b1-4d01-a5fe-c996a63108f1 uswest1adevc
DN 10.40.1.3 248.47 KB 256 ? 9et4944d-70c5-4a2f-b517-d7fd7a32bc45 uswest1cdevc
I understand that the HostID of a node might change after system dirs are removed.
My question is:
Do i need to explicitly state during the start to replace itself? Are the docs incomplete or am i missing something in my steps?
Turns out there were stale directories commit_log and saved_caches which i missed to delete earlier. The instructions work correctly with those directories deleted.
Usually on a situation like this, after i do a
$ systemctl stop cassandra
It i will run the
$ ps awxs | grep cassandra
will notice cassandra still has some features up.
I usually do a
$ kill -9 cassandra.pid
and
$ rm -rf /var/lib/cassandra/data/* && /var/lib/cassandra/commitlog/*
java.lang.RuntimeException: A node with address 10.40.1.3 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.
If you are still facing this above error, that means your cassandra process is running on that node. Login to 10.40.1.3 node firstly. Then follow the following steps-
$ jps
You see some processes running. For example:
9107 Jps
1112 CassandraDaemon
Then kill the CassandraDaemon process by the process id you see after executing jps. In my example, here process id 1112 for CassandraDaemon.
$ kill -9 1112
Then check processes again after a while-
$ jps
You will see CassandraDaemon will no longer be available.
9170 Jps
Then remove your saved_caches and commitlog and start cassandra again.
Do this for all nodes you are suffering with above error you mentioned.