I am having issues create a build with cassandra as a service with a version that is higher than 2.X (let x be higher than 1).
I have verified that
services:
- cassandra
produce a cassandra 2.0.9 as i put in my .travis.yml
$ cqlsh --execute="show version" 127.0.0.1
[cqlsh 4.1.1 | Cassandra 2.0.9 | CQL spec 3.1.1 | Thrift protocol 19.39.0]
However my project requires 2.2.4 as a minimum.
when i tried doing as per travis suggested:
before_install:
- sudo rm -rf /var/lib/cassandra/*
- wget https://archive.apache.org/dist/cassandra/2.2.4/apache-cassandra-2.2.4-bin.tar.gz && tar -xvzf apache-cassandra-2.2.4-bin.tar.gz && sudo sh apache-cassandra-2.2.4/bin/cassandra
- sleep 30
It fails to boot the cassandra as it waits at the following line:
Connection error: Could not connect to 127.0.0.1:9160
While dumping the raw_log it gets stuck here:
INFO 16:01:31 Loading org.apache.cassandra.config.CFMetaData#2716f853[cfId=5f2fbdad-91f1-3946-bd25-d5da3a5c35ec,ksName=system_auth,cfName=resource_role_permissons_index,cfType=Standard,comparator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.UTF8Type,org.apache.cassandra.db.marshal.UTF8Type),comment=index of db roles with permissions granted on a resource,readRepairChance=0.0,dcLocalReadRepairChance=0.0,gcGraceSeconds=7776000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata=[ColumnDefinition{name=role, type=org.apache.cassandra.db.marshal.UTF8Type, kind=CLUSTERING_COLUMN, componentIndex=0, indexName=null, indexType=null}, ColumnDefinition{name=resource, type=org.apache.cassandra.db.marshal.UTF8Type, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}],compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={sstable_compression=org.apache.cassandra.io.compress.LZ4Compressor},bloomFilterFpChance=0.01,memtableFlushPeriod=3600000,caching={"keys":"ALL", "rows_per_partition":"NONE"},defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false]
INFO 16:01:31 Initializing system_auth.resource_role_permissons_index
Does anyone have any idea how to get travis to successful build a cassandra 2.X higher than its default?
NOTES:
My project is PHP base.
I have tried this in Container mode and as
sudo (as per travis instructions also
Related
I'm trying to set up hawkBit update server as per https://www.eclipse.org/hawkbit/gettingstarted/ but a lot of issues that I do not know how to resolve.
The following is summary setup steps:
Prerequisites
sudo apt install -y default-jre default-jdk
java -version
*openjdk version "11.0.9.1" 2020-11-04*
*OpenJDK Runtime Environment (build 11.0.9.1+1-Ubuntu-0ubuntu1.18.04)*
*OpenJDK 64-Bit Server VM (build 11.0.9.1+1-Ubuntu-0ubuntu1.18.04, mixed mode, sharing)*
sudo apt install maven
sudo apt-get -y install mariadb-server
Download build source code
git clone https://github.com/eclipse/hawkbit.git
cd hawkbit
mvn clean install
*Start hawkBit update server*
*Access http://localhost:8080/UI/login to check*
java -jar ./hawkbit-runtime/hawkbit-update-server/target/hawkbit-update-server-0.3.0-SNAPSHOT.jar
Customize hawkBit
3.1. Set up data base
sudo mysql -u root
*Welcome to the MariaDB monitor. Commands end with ; or \g.*
*Your MariaDB connection id is 31*
*Server version: 10.1.47-MariaDB-0ubuntu0.18.04.1 Ubuntu 18.04*
*Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.*
*Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.*
MariaDB [(none)]> USE mysql;
*Reading table information for completion of table and column names*
*You can turn off this feature to get a quicker startup with -A*
*Database changed*
MariaDB [mysql]> CREATE USER 'test'#'localhost' IDENTIFIED BY '123456a#';
*Query OK, 0 rows affected (0.00 sec)*
*MariaDB [mysql]> UPDATE user SET authentication_string=password('123456a#') where user='test';*
*Query OK, 0 rows affected (0.00 sec)*
*Rows matched: 1 Changed: 0 Warnings: 0*
MariaDB [mysql]> GRANT ALL PRIVILEGES ON *.* TO 'test'#'localhost';
*Query OK, 0 rows affected (0.00 sec)*
MariaDB [mysql]> UPDATE user SET plugin='mysql_native_password' WHERE User='test';
*Query OK, 1 row affected (0.01 sec)*
*Rows matched: 1 Changed: 1 Warnings: 0*
MariaDB [mysql]> FLUSH PRIVILEGES;
*Query OK, 0 rows affected (0.00 sec)*
MariaDB [mysql]> CREATE DATABASE hawkbit;
*Query OK, 1 row affected (0.00 sec)*
MariaDB [mysql]> ALTER DATABASE hawkbit COLLATE latin1_bin;
Query OK, 1 row affected (0.00 sec)
MariaDB [mysql]> grant all privileges on hawkbit.* TO 'test'#'localhost' identified by '123456a#';
*Query OK, 0 rows affected (0.01 sec)*
MariaDB [(none)]> exit
*Bye*
sudo service mysql restart
Append the following code to hawkbit-runtime/hawkbit-update-server/pom.xml
<dependency>
<groupId>org.mariadb.jdbc</groupId>
<artifactId>mariadb-java-client</artifactId>
<scope>compile</scope>
</dependency>
Override the following code to hawkbit-runtime/hawkbit-update-server/src/main/resources/application-mysql.properties:
spring.jpa.database=mysql
spring.datasource.url=jdbc:mysql://localhost:3306/hawkbit
spring.datasource.username=test
spring.datasource.password=123456a#
spring.datasource.driverClassName=org.mariadb.jdbc.Driver
Application.properties is appended with new URL:
server.hostname=http://<my_ip_addr>
server.port=<my_port>
Rebuild and run:
cd hawkbit
mvn clean install
java -jar ./hawkbit-runtime/hawkbit-update-server/target/hawkbit-update-server-0.3.0-SNAPSHOT.jar --spring.profiles.active=mysql
Open http://<my_ip_addr>:<my_port>, I cannot log in by "admin" username and "admin" password.
Run hawkBit without --spring.profiles.active=mysql, I succeed to log in by admin, admin.
Even the application.properties is kept unchanged, the results are the same.
I try to comment out all User Security:
# spring.security.user.name=admin
# spring.security.user.password={noop}admin-pwd
# spring.main.allow-bean-definition-overriding=true
and uncomment Define own users instead of default "admin" user:
hawkbit.server.im.users[0].username=test
hawkbit.server.im.users[0].password={noop}123456a#
hawkbit.server.im.users[0].firstname=Huong
hawkbit.server.im.users[0].lastname=Ha
hawkbit.server.im.users[0].permissions=ALL
the build failed with the following log:
[INFO] hawkBit :: Runtime :: Update Server ................ FAILURE [ 7.579 s]
[INFO] hawkBit :: Test Report ............................. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14:59 min
[INFO] Finished at: 2020-12-26T16:47:05+07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.1:test (default-test) on project hawkbit-update-server: There are test failures.
[ERROR]
[ERROR] Please refer to /home/huong/software-update-server/hawkbit/hawkbit-runtime/hawkbit-update-server/target/surefire-reports for the individual test results.
So I have some questions and hope someone could help:
The exact steps to setup hawkBit update server. Any experiences would be helpful
The hawkbit database (is created as above) is an empty database, do you have an database which is compatible with hawkBit?
Thanks,
Huong Ha
The procedure you are following is quite correct. However, there are couple of remarks regarding your approach:
You can omit running all the tests while building Hawkbit by specifying skipTests maven argument: mvn clean install -DskipTests. It will drastically decrease the build time.
You don't need to modify the application.properties in order to use MySQL database. Instead just specify the --spring.profiles.active=mysql program argument for activating the "mysql" Spring profile and adapt the application-mysql.properties as needed (in your case changing the username and password datasource config).
You need to either use the spring.security.user.name and spring.security.user.password or hawkbit.server.im.users, not both. Spring security user should work out of the box. Additionally, you should NOT comment out spring.main.allow-bean-definition-overriding=true, it is responsible for the bean overriding mechanism of Spring and has nothing to do with Spring Security.
Hawkbit is using Flyway DB version control (https://flywaydb.org) meaning that it will execute all of the required DB migration scripts during the application start.
And yes, you can run Hawkbit with java11 (see https://github.com/eclipse/hawkbit/commit/ed9a4b1bb31f4cd996e8af8867d67cdcda682d00)
Hope that brings you further ;)
cassandra service (3.11.5) stops automatically after it starts/restart on AWS linux.
I have fresh installation of cassandra on new instance of AWS linux (t3.xlarge) and
sudo service cassandra start
or
sudo service cassandra restart
after 1 or 2 seconds, the service stop automatically. I looked into logs and I found these.
I am not sure, I havent change configs related to snitch and its always SimpleSnitch. I dont have any multiple cassandras. Just only on single EC2.
Logs
INFO [main] 2020-02-12 17:40:50,833 ColumnFamilyStore.java:426 - Initializing system.schema_aggregates
INFO [main] 2020-02-12 17:40:50,836 ViewManager.java:137 - Not submitting build tasks for views in keyspace system as storage service is not initialized
INFO [main] 2020-02-12 17:40:51,094 ApproximateTime.java:44 - Scheduling approximate time-check task with a precision of 10 milliseconds
ERROR [main] 2020-02-12 17:40:51,137 CassandraDaemon.java:759 - Cannot start node if snitch's data center (datacenter1) differs from previous data center (dc1). Please fix the snitch configuration, decommission and rebootstrap this node or use the flag -Dcassandra.ignore_dc=true.
Installation steps
sudo curl -OL https://www.apache.org/dist/cassandra/redhat/311x/cassandra-3.11.5-1.noarch.rpm
sudo rpm -i cassandra-3.11.5-1.noarch.rpm
sudo pip install cassandra-driver
export CQLSH_NO_BUNDLED=true
sudo chkconfig --levels 3 cassandra on
The issue is in your log file:
ERROR [main] 2020-02-12 17:40:51,137 CassandraDaemon.java:759 - Cannot start node if snitch's data center (datacenter1) differs from previous data center (dc1). Please fix the snitch configuration, decommission and rebootstrap this node or use the flag -Dcassandra.ignore_dc=true.
It seems that you started the cluster, stopped it and renamed the datacenter from dc1 to datacenter1.
In order to fix:
If no data is stored, delete the data directories
If data is stored, rename the datacenter back to dc1 in the config
I had the same problem , where cassandra service immediately stops after it was started.
in the cassandra configuration file located at /etc/cassandra/cassandra.yaml change the cluster_name to the previous one, like this:
...
# The name of the cluster. This is mainly used to prevent machines in
# one logical cluster from joining another.
cluster_name: 'dc1'
# This defines the number of tokens randomly assigned to this node on the ring
# The more tokens, relative to other nodes, the larger the proportion of data
...
I am using Spark to do some computation over some data and then push to Hive. The Cloud Dataproc versions is 1.2 with Hive 2.1 included. The Merge command in Hive is only support by version 2.2 onwards. So I have to use preview version for dataproc cluster. When I use version 1.2 for dataproc cluster, I can create the cluster without any issue. I got this error "Failed to bring up Cloud SQL Metastore" when using preview version.
The initialisation script is here. Has anyone every met this problem before?
hive-metastore.service is not a native service, redirecting to systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install is-enabled hive-metastore
mysql.service is not a native service, redirecting to systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable mysql
insserv: warning: current start runlevel(s) (empty) of script `mysql` overrides LSB defaults (2 3 4 5).
insserv: warning: current stop runlevel(s) (0 1 2 3 4 5 6) of script `mysql' overrides LSB defaults (0 1 6).
Created symlink /etc/systemd/system/multi-user.target.wants/cloud-sql-proxy.service → /usr/lib/systemd/system/cloud-sql-proxy.service.
Cloud SQL Proxy installation succeeded
hive-metastore.service is not a native service, redirecting to systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install is-enabled hive-metastore
[2018-06-06T12:43:55+0000]: Failed to bring up Cloud SQL Metastore
I believe the issue may be that your metastore was initialized from an older version of Dataproc and thus has outdated schema.
If you have the failed cluster (if not, please create a new one as before, you can use --single-node option to reduce cost), then SSH to master node and upgrade schema:
$ gcloud compute ssh my-cluster-m
$ /usr/lib/hive/bin/schematool -dbType mysql -info
Hive distribution version: 2.3.0
Metastore schema version: 2.1.0 <-- you will need this
org.apache.hadoop.hive.metastore.HiveMetaException: Metastore schema version is
not compatible. Hive Version: 2.3.0, Database Schema Version: 2.1.0
*** schemaTool failed ***
$ /usr/lib/hive/bin/schematool -dbType mysql -upgradeSchemaFrom 2.1.0
Unfortunately this cluster cannot be returned to running state, so please delete and recreate it.
I have created this PR to make issue more discoverable:
https://github.com/GoogleCloudPlatform/dataproc-initialization-actions/pull/278
I am looking for a set of instructions to create a local multi-node Cassandra 2.x cluster on a Window 7 64 bit PC.
It should preferably use CCM “Cassandra Cluster Manager” and allow management using DataStax OpsCenter
I followed the instructions in “Getting Started with Apache Cassandra on Windows the Easy Way” but they are for a single node cluster.
EDIT: I got stuck on deploying OpsCenter agents on each node using CCM, any ideas?
Articles used for this tutorial:
CCM 2.0 and Windows
Cassandra Wiki - Windows Development
Setting up a multi-node Cassandra cluster on a single Windows
machine
See also:
Getting Started with Apache Cassandra on Windows the Easy Way
Cassandra DevCenter (free registration required)
Prerequisites:
The following tools are assumed to be already installed:
JDK 7 or older
ANT build tool
Step 1: Install Python
Download and install latest version of Python 2.x from here
e.g. https://www.python.org/ftp/python/2.7.11/python-2.7.11.amd64.msi
Note: This will also install “pip” tool
The following directories need to be added to the PATH
<PYTHON_INSTALL_DIR>\Python
<PYTHON_INSTALL_DIR>\Python\Scripts
<ANT_INSTALL_DIR>\bin
Step 2: Install CCM “Cassandra Cluster Manager”
In a new Command Prompt/Powershell window (login as yourself)
type “pip install ccm” – which will automatically download and install ccm
> pip install ccm
Collecting ccm
Downloading ccm-2.0.6.tar.gz (56kB)
100% |################################| 57kB 1.8MB/s
Collecting pyYaml (from ccm)
Downloading PyYAML-3.11.tar.gz (248kB)
100% |################################| 249kB 1.7MB/s
Collecting six>=1.4.1 (from ccm)
Downloading six-1.10.0-py2.py3-none-any.whl
Installing collected packages: pyYaml, six, ccm
Running setup.py install for pyYaml
Running setup.py install for ccm
Successfully installed ccm-2.0.6 pyYaml-3.11 six-1.10.0
Step 3: Install “psutil (python system and process utilities)”
In the same window as for Step 2:
type “pip install psutil” – – which will automatically download and install psutil
> pip install psutil
Collecting psutil
Downloading psutil-3.3.0-cp27-none-win_amd64.whl (92kB)
100% |################################| 94kB 1.4MB/s
Installing collected packages: psutil
Successfully installed psutil-3.3.0
Note: This window can now be closed
Step 4: Set-ExecutionPolicy Unrestricted
In a new Powershell window (login as local admin), type “Set-ExecutionPolicy Unrestricted”
*Note: You must set the execution policy of Windows Powershell to allow CCM to launch instances of Cassandra. An unrestricted execution policy will also allow CCM to run on the regular command prompt (cmd) as well as Windows Powershell
PS C:\Windows\system32> Set-ExecutionPolicy Unrestricted
Execution Policy Change
The execution policy helps protect you from scripts that you do not trust. Changing the execution policy might expose
you to the security risks described in the about_Execution_Policies help topic. Do you want to change the execution
policy?
[Y] Yes [N] No [S] Suspend [?] Help (default is "Y"): Y
Step 5: Register PY extension
Note: Add .PY extension to environment variable $PATHEXT, to allow ccm to be executed from any location (run on PowerShell as administrator):
In the same window as for Step 4 type
[Environment]::SetEnvironmentVariable("PATHEXT", "$env:PATHEXT;.PY", "MACHINE")
Note: This window can now be closed
Step 6: Check if CCM is up and running
In a new Command Prompt window (login as yourself) type:
>ccm status
No currently active cluster (use ccm cluster switch)
Step 7: Update hosts file
Open Notepad as Administrator and the following lines to the C:\Windows\System32\drivers\etc\hosts file:
#cassandra nodes
127.0.0.1 127.0.0.2
127.0.0.1 127.0.0.3
127.0.0.1 127.0.0.4
127.0.0.1 127.0.0.5
127.0.0.1 127.0.0.6
Step 8: Create and populate a 3 node cluster using Cassandra v2.1.2
Note: This will download version 2.1.2 of Cassandra, build it and then use it to create a new CCM cluster called “mytestcluster”.
Cassandra installation path %USERPROFILE%.ccm\repository\2.1.2
“test” cluster path %USERPROFILE%.ccm\test
C:\Users\myusername>ccm create mytestcluster -v 2.1.2
Downloading http://archive.apache.org/dist/cassandra/2.1.2/apache-cassandra-2.1.2-bin.tar.gz to c:\users\myusername\appdata\local\temp\ccm-qwauvs.tar.gz (21.735MB)
22790390 [100.00%]
Extracting c:\users\myusername\appdata\local\temp\ccm-qwauvs.tar.gz as version 2.1.2 ...
Current cluster is now: mytestcluster
C:\Users\myusername>ccm status
Cluster: 'mytestcluster'
------------------------
No node in this cluster yet
C:\Users\myusername>ccm populate -n 3
C:\Users\myusername>ccm status
Cluster: 'mytestcluster'
------------------------
node1: DOWN (Not initialized)
node3: DOWN (Not initialized)
node2: DOWN (Not initialized)
C:\Users\myusername>ccm start
Started: node1 with pid: 17432
Started: node3 with pid: 6308
Started: node2 with pid: 22484
C:\Users\myusername>ccm status
Cluster: 'mytestcluster'
------------------------
node1: UP
node3: UP
node2: UP
C:\Users\myusername>ccm jconsole
C:\Users\myusername>ccm node1 show
node1: UP
cluster=mytestcluster
auto_bootstrap=False
thrift=('127.0.0.1', 9160)
binary=('127.0.0.1', 9042)
storage=('127.0.0.1', 7000)
jmx_port=7100
remote_debug_port=0
initial_token=-9223372036854775808
pid=17432
C:\Users\myusername>ccm node2 show
node2: UP
cluster=mytestcluster
auto_bootstrap=False
thrift=('127.0.0.2', 9160)
binary=('127.0.0.2', 9042)
storage=('127.0.0.2', 7000)
jmx_port=7200
remote_debug_port=0
initial_token=-3074457345618258603
pid=22484
C:\Users\myusername>ccm node3 show
node3: UP
cluster=mytestcluster
auto_bootstrap=False
thrift=('127.0.0.3', 9160)
binary=('127.0.0.3', 9042)
storage=('127.0.0.3', 7000)
jmx_port=7300
remote_debug_port=0
initial_token=3074457345618258602
pid=6308
How am I supposed to bot a new Cassandra node when I get this error?
INFO [SSTableBatchOpen:1] 2014-02-25 01:51:17,132 SSTableReader.java (line 223) Opening /var/lib/cassandra/data/system/local/system-local-jb-5 (5725 bytes)
ERROR [main] 2014-02-25 01:51:17,377 CassandraDaemon.java (line 237) Fatal exception during initialization
org.apache.cassandra.exceptions.ConfigurationException: Saved cluster name Test Cluster != configured name thisisstupid
at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:542)
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:233)
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:462)
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:552)
Name of cluster in the cassandra.yaml file is:
cluster_name: 'thisisstupid'
How do I resolve?
You can rename the cluster without deleting data by updating it's name in the system.local table (but you have to do this for each node...)
cqlsh> UPDATE system.local SET cluster_name = 'test' where key='local';
# flush the sstables to persist the update.
bash $ ./nodetool flush
Finally you need to rename the cluster to the new name in cassandra.yaml (again on each node)
The above commands with UPDATE SET cluster_name does not work for me. I found the working is to follow the instructions from the DataStax documentation on Initializing a multiple node cluster:
sudo service cassandra stop
sudo rm -rf /var/lib/cassandra/data/system/*
sudo vi /etc/cassandra/cassandra.yaml, setup the proper parameters
sudo service cassandra start
nodetool status
For some good cluster node setups, I found this blog post to be very useful.
I had same issue with Datastax4.7 enterprise edition. I resolved it with above instructions:
Change the cluster name back to "test Cluster"
start the cluster:
cqlsh> UPDATE system.local SET cluster_name = 'Cassendra Cluster' where key='local';
cqlsh> exit;
$ nodetool flush system
Stop the cluster:
$sudo service dse stop;sudo service datastax-agent stop
Edit the file:
$ sudo vi /etc/dse/cassandra/cassandra.yaml
cluster_name: 'Cassendra Cluster'
Start the cluster:
$sudo service dse start;sudo service datastax-agent start
Check Installation log:
$ cat /var/log/cassandra/output.log
For Cassandra 3.0.5, I tired the answer by Lyuben Todorov but it was missing a key part:
bash $ ./nodetool flush
should be:
bash $ ./nodetool flush system
as in here
For Cassandra 3.7 you should replace the ./nodetool flush with ./nodetool flush system. Otherwise it won't work.
If cassandra is not running, you can wipe the data folder.
rm -rf /usr/lib/cassandra/data/*
This will clear the system tables which had the wrong cluster name. \
When your restart cassandra with the correct cluster name in your cassandra.yaml everything will be regenerated and you should be good.
For Bitnami certified Cassandra - 3.11.5
Here's the solution that worked for me:
Stop the Cassandra service
sudo /opt/bitnami/ctlscript.sh stop cassandra
Remove the system data
sudo rm -rf /opt/bitnami/cassandra/data/data/system/*
Start the Cassandra Service
sudo /opt/bitnami/ctlscript.sh start cassandra