I have an RDS Postgres database, currently sitting at version 14.3.
I want to schedule a major version upgrade to 14.5 to happen during the maintenance window.
I want to do this manually via the console, because last time I did a major version of the Postgres version by changing the CDK definition, the deploy command applied the DB version change immediately, resulting in a few minutes downtime of the database (manifesting as connection errors in the application connecting to the database).
When I go into the AWS RDS console, do a "modify" action and select the "DB Engine Version" - it only shows one option, which is the current DB version: "14.3".
According to the RDS doco 14.4, 14.5. and 14.6 are all valid upgrade targets: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.PostgreSQL.html#USER_UpgradeDBInstance.PostgreSQL.MajorVersion
Also, when I do aws rds --profile raido-prod describe-db-engine-versions --engine postgres --engine-version 14.3 it shows those versions in the ValidUpgradeTarget collection.
Using CDK version 2.63.0
Database CDK code:
// starting with 14.3 to test the manual upgrade process
const engineVersion14_3 = PostgresEngineVersion.VER_14_3;
const dbParameterGroup14_3 = new ParameterGroup(this, 'Postgres_14_3', {
description: "RaidoDB postgres " + engineVersion14_3.postgresFullVersion,
engine: DatabaseInstanceEngine.postgres({
version: engineVersion14_3,
}),
parameters: {
// none we need right now
},
});
/* Note that even after this stack has been deployed, this param group
will not be created, I guess it will only be created when it's attached
to an instance? */
const engineVersion14_5 = PostgresEngineVersion.VER_14_5;
// CDK strips out underbars from name, hoping periods will remain
const dbParameterGroup14_5 = new ParameterGroup(this, 'Postgres.14.5.', {
description: "RaidoDB postgres " + engineVersion14_3.postgresFullVersion,
engine: DatabaseInstanceEngine.postgres({
version: engineVersion14_5,
}),
parameters: {
// none we need right now
},
});
this.pgInstance = new DatabaseInstance(this, 'DbInstance', {
databaseName: this.dbName,
instanceIdentifier: this.dbName,
credentials: Credentials.fromSecret(
this.raidoDbSecret,
this.userName,
),
vpc: props.vpc,
vpcSubnets: {
subnetGroupName: props.subnetGroupName,
},
publiclyAccessible: false,
subnetGroup: dbSubnetGroup,
multiAz: false,
availabilityZone: undefined,
securityGroups: [props.securityGroup],
/* Should we size a bigger instance for prod?
Plan is to wait until its needed - there will be some downtime for
changing these. There's also the "auto" thing. */
allocatedStorage: 20,
instanceType: InstanceType.of(InstanceClass.T4G, InstanceSize.SMALL),
engine: DatabaseInstanceEngine.postgres({
version: engineVersion14_3,
}),
parameterGroup: dbParameterGroup14_3,
/* Not sure what this does, changing it to true didn't allow me to change
the version in the console. */
allowMajorVersionUpgrade: true,
/* 14.3.x -> 14.3.x+1 will happen automatically in maintenance window,
with potential downtime. */
autoMinorVersionUpgrade: true,
// longer in prod
backupRetention: Duration.days(30),
/* This enables DB termination protection.
When stack is destroyed, db will be detached from stack but not deleted.
*/
removalPolicy: RemovalPolicy.RETAIN,
// explain and document the threat model before changing this
storageEncrypted: false,
/* "Enhanced monitoring"."
I turned this on while trying to figure out how to change the DB version.
I still don't think we should have it enabled until we know how/when we'll
use it - because it costs money in CloudWatch Logging, Metric fees and
performance (execution and logging of the metrics from the DB server).
*/
monitoringInterval: Duration.minutes(1),
monitoringRole: props.monitoringRole,
/* Useful for identifying expensive queries and missing indexes.
Retention default of 7 days is fine. */
enablePerformanceInsights: true,
// UTC
preferredBackupWindow: '11:00-11:30',
preferredMaintenanceWindow: 'Sun:12:00-Sun:13:00',
});
So, the question: What do I need to do in order to be able to schedule the DB version upgrade in the maintenance window?
I made a lot of changes during the day trying to diagnose the issue before I posted this question, thinking I must be doing something wrong.
When I came back to work the next day, the modify screen DB Engine Version field contained the upgrade options I was originally expecting.
Below is my documentation of the issue (unfortunately, our CDK repo is not public):
Carried out by STO, CDK version was 2.63.0.
This page documents my attempt to manually schedule the DB version upgrade
using the console for application during the maintenance window.
We need to figure out how to do this since the DB upgrade process results in a
few minutes of downtime, so we'd prefer to avoid doing it in-hours.
It's preferred that we figure out how to schedule the upgrade - if the choice
comes down to asking team members to work outside or hours or accept
downtime - we will generally choose to have the downtime during business hours.
Note that that Postgres instance create time is:
Sat Jan 28 2023 10:34:32 GMT+1000
Action plan
in the RDS console
Modify DB instance: raido
DB engine version: change 14.3 to 14.5
save and select to "schedule upgrade for maintenance window" (as
opposed to "apply immediately")
Actions taken
2023-02-02
When I tried to just go in and change it manually in the AWS console, the only
option presented on the "modify" screen was 14.3 - there was nothing to
change the version to.
I tried creating a 14.5 param group in CDK, but it just ignored me, didn't
create the param group I'm guessing because it's expected to actually be
attached to a DB.
Tried copying and creating a new param group to change the version, but there's
no "version" param in the param group.
Tried manually create a db of version 14.5 "sto-manual-14-5", but after the
DB was reported successfully created (as 14.5) - "14.3" is still the only
option in the "modify" screen for raido-db.
Tried creating a t3.micro in case t4g ws the problem - no good.
Tried disabling minor version auto-upgrade - no good.
Note that 14.6 is the current most recent version, both manually created 14.3
and 14.5 databases presented no other versions to upgrade to - so this problem
doesn't seem to be related to the CDK.
List available upgrades: aws rds --profile raido-prod describe-db-engine-versions --engine postgres --engine-version 14.3
Shows 14.4, 14.5 and 14.6 as valid target versions.
This page also shows the the versions should be able to be upgraded to, as at
2023-02-02: https://docs.amazonaws.cn/en_us/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.PostgreSQL.html#USER_UpgradeDBInstance.PostgreSQL.MajorVersion
After all this, I noticed that we had the instance declared as
allowMajorVersionUpgrade: false, so I changed that to true and deployed,
but still can't select any other versions.
Also tried aws rds --profile raido-prod describe-pending-maintenance-actions
but it showed no pending actions.
I found this SO answer talking about RDS problems with T3 instances (note, I
have previously upgraded major versions of T3 RDS postgres instances):
https://stackoverflow.com/a/69295017/924597
On the manually created t3.micro instance, I tried upgrading the instance size
to a standard large instance. Didn't work.
Found this SO answer talking about the problem being related to having
outstanding "recommended actions" entries:
https://stackoverflow.com/a/75236812/924597
We did have an entry talking about "enhanced monitoring".
Tried enabling "enhanced monitoring" via the CDK, because it was listed as
a "recommended action" on the RDS console.
After deploying the CDK stack, the console showed the enhanced monitoring was
enabled, but the "recommended action" to enable it was still listed.
At this point, the console still showed 14.3 as the only option in the list on
the modify screen.
Posted to StackOverflow: Can't change RDS Postgres major version from the AWS console?
Posted to AWS repost: https://repost.aws/questions/QU4zuJeb9OShGfVISX6_Kx4w/
Stopped work for the day.
2023-02-03
In the morning, the RDS console no longer shows the "recommended action" to
enabled enhanced monitoring.
The modify screen now shows "14.3, 14.4, 14.5 and 14.6" as options for the
DB Engine Verison (as expected and orginally desired).
Given the number of changes I tried above, I'm not sure what, if any of them
may have caused the console to start displaying the correct options.
It may have been a temporary issue with RDS, or AWS support may have seen my
question on the AWS repost forum and done something to the account.
Note that I did not raise a formal support request via the AWS console.
I wanted to try and confirm if the enhanced monitoring was the cause of the
issue, so I changed the CDK back (there is no "enhance monitoring" flag, I
just commented out the code that set the monitoring role and interval).
After deploying the CDK stack, there was no change to the RDS instance -
enhanced monitoring was still enabled.
I did a manual modify via the RDS console to disable enhanced monitoring.
The change did apply and was visible in the consle, but the "recommended
actions" list did not have any issues.
At this point I had to attend a bunch of meetings, lunch, etc.
When I came back after lunch, the "recommended actions" list now shows an
"enhanced monitoring" entry.
But the modify console page still shows the 14.3 - 14.6 DB engine options, so
I don't think "enhanced monitoring" was the cause of the problem.
I scheduled the major version upgrade (14.3 -> 14.5, because 14.6 is not yet
supported by the CDK) for the next maintenance window.
Analysis
My guess is that the issue was caused by having allowMajorVersionUpgrade set
to false. I think changing it to true is what caused the other
version options to eventually show up on the modify page. I think the
reason the options didn't show up on the modify page after depoloying the
CDK change is down to waiting for an eventual consistency conflict to converge.
We are planning to migrate from HMC to Backoffice. I want all the data in HMC to be transferred to Backoffice. when I ran SOLR Full indexing but I am below getting Issue.
ERROR [Thread-6784] (00011GS2) [BackofficeIndexerStrategy] Executing indexer worker as an admin user failed:
ERROR [Thread-645] (00011HJD) [BackofficeIndexerStrategy] Executing indexer worker as an admin user failed:
INFO | jvm 1 | main | 2021/01/13 09:54:59.310 | de.hybris.platform.solrfacetsearch.indexer.exceptions.IndexerException: de.hybris.platform.solrfacetsearch.solr.exceptions.SolrServiceException: Could not check for a remote solr core [master_backoffice_backoffice_product_flip] due to Server refused connection at: http://Dev:8983/solr
Every time I get error. I had to restart my solr server.
Can someone please help me on this.
Thanks
This might be an issue about SSL checks. You can try using disabling the SSL by adding
solrserver.instances.default.ssl.enabled=false
to your local.properties file. This should best only be used on your local development environment.
Or, you can try using https in your default solr server like https://localhost:8983/solr
I am using sap hybris 2205, products displayes in front but, solrserver not working. i try this one also (backoffice.facet-facet conf.->siteindex->enter->finish) getting error.
I'm trying to connect to the OpenEdge database so that I can perform queries on it. However, I'm having trouble figuring out how to connect to it to make these queries.
So far I've tried making a config file, but I'm unsure how to establish a connection to the server using it:
"HMMv10": {
"dbConfig": {
"connectionString": "DRIVER={Progress OpenEdge 11.3 Driver}",
"UID": "SYSPROGRESS",
"pwd": "***",
"host": "host.local",
"port": "18210",
"db": "hmm10"
}
}
I don't know what that config file goes with or how it works but before trying to setup 3rd party software to connect to an OpenEdge database it is a really good idea to use the provided OE tools to verify that you have a properly configured connection available to connect to.
Progress provides a command line tool called "sqlexp" that you can use to test connections. The easiest way to access that tool is via a "proenv" command shell. On Windows just open "proenv" (it is installed with the other Progress commands in the "Progress" program group). If you have a Linux or UNIX install you run $DLC/bin/proenv.
Then start sqlexp like this (I used my local dbname and port number):
proenv> sqlexp -db s2k -H localhost -S 9500 -user sysprogress -password sysprogress
OpenEdge Release 11.7.2 as of Tue Oct 24 18:20:59 EDT 2017
Connecting user "sysprogress" to URL "jdbc:datadirect:openedge://localhost:9500;databaseName=s2k"... (8920)
SQLExplorer>
If there are no errors and you get the SQLExplorer> prompt that is sufficient to show that there is a working connection available.
If you would like to go the extra mile and prove that you have been granted permission to fetch data try a simple select statement:
SQLExplorer> select count(*) from pub.customer;
count(*)
--------------------
1117
SQLExplorer> quit;
proenv>
If this does not work then the local DBA has not configured SQL access, or you have incorrect configuration/credentials and no amount of working with your other tool will fix that.
OpenEdge SQL notes:
If you are not already aware - all OpenEdge data is variable length. Many SQL tools expect that the width of a field is known use metaschema data to get a default value. If the local DBA has not been maintaining that data via "dbtool" then you are likely to have errors due to data being wider than expected.
sqlexp expects commands to end with ";".
The "pub" schema is the schema where OpenEdge application tables live. Table names need to have "pub." pre-pended.
Table and field names that contain "-" will need to be quoted.
I'm putting up what I found to be the answer just in case anyone else is looking for this. Using a REST API in NodeJS I used npm-odbc and the OpenEdge odbc drivers to connect to the database. Even though the npm-odbc package only mentions Unix it does in fact work on windows.
I am having trouble being able to accessing a table in the Glue Data Catalog using pySpark in Hue/Zeppelin on EMR. I have tried both emr-5.13.0 and emr-5.12.1.
I tried following https://github.com/aws-samples/aws-glue-samples/blob/master/examples/data_cleaning_and_lambda.md
but when trying to import the GlueContext it errors saying No module named awsglue.context.
Another note is that when doing a spark.sql("SHOW TABLES").show() it comes up empty for Hue/Zeppelin but when using the pyspark shell on the master node I am able to see and query the table from the Glue Data Catalog.
Any help is much appreciated, thanks!
Ok, I spent some time to simulate the issue, so I spinned up an EMR, with "Use AWS Glue Data Catalog for table metadata" enabled. After enabling web connections, and in zeppelin I issued a show databases command, and it worked fine. Please find herewith the command & output from Zeppelin:
%spark
spark.sql("show databases").show
+-------------------+
|airlines-historical|
| default|
| glue-poc-tpch|
| legislator-new|
| legislators|
| nursinghomedb|
| nycitytaxianalysis|
| ohare-airport-2006|
| payments|
| s100g|
| s1g|
| sampledb|
| testdb|
| tpch|
| tpch_orc|
| tpch_parquet|
+-------------------+
As far as your other issue of "No module named awsglue.context", I think it may not be possible with an EMR commissioned Zeppelin. I think the only way, an awsglue.context can be accessed / used, is via a Glue Devendpoint that you may need to set up in AWS Glue, and then, use an glue jupyter notebook or a locally setup Zeppelin notebook connected to glue development endpoint.
Am not sure if glue context can be directly accessed from an EMR commissioned Zeppelin notebook, maybe am wrong.
You can still access the glue catalog, since EMR provides you with an option for the same, so you can access the databases and do your ETL jobs.
Thanks.
Please check out the details in this link from AWS, and see if the EMR is configured correctly as recommended (Configure Glue Catalog in EMR). Also ensure that appropriate permissions are granted to access AWS Glue catalog. Details are in the attached link. Hope this helps.
you can use the below function to check the availability of databases in glue
def isDatabasePresent(database_name):
"""
check if the glue database exists
:return: Boolean
"""
client = get_glue_client()
responseGetDatabases = client.get_databases()
databaseList = responseGetDatabases['DatabaseList']
for databaseDict in databaseList:
if database_name == databaseDict['Name']:
return True
I am using HAProxy to for AWS RDS (MySQL) load balancing for my app, that is written using Flask.
The HAProxy.cfg file has following configuration for the DB
listen mysql
bind 127.0.0.1:3306
mode tcp
balance roundrobin
option mysql-check user haproxy_check
option log-health-checks
server db01 MASTER_DATABSE_ENDPOINT.rds.amazonaws.com
server db02 READ_REPLICA_ENDPOINT.rds.amazonaws.com
I am using SQLALCHEMY and it's URI is:
SQLALCHEMY_DATABASE_URI = 'mysql+pymysql://USER:PASSWORD#127.0.0.1:3306/DATABASE'
but when I am running an API in my test environment, the APIs that are just reading stuff from DB are executing just fine but the APIs that are writing something to DB are giving me errors mostly that:
(pymysql.err.InternalError) (1290, 'The MySQL server is running with the --read-only option so it cannot execute this statement')
I think I need to use 2 URLs now in this scenario, one for read-only operation and one for writes.
How does this work with Flask and SQLALCHEMY with HAProxy?
How do I tell my APP to use one URL for write operations and other HAProxy URL to read-only operations?
I didn't find any help from the documentation of SQLAlchemy.
Binds
Flask-SQLAlchemy can easily connect to multiple databases. To achieve
that it preconfigures SQLAlchemy to support multiple “binds”.
SQLALCHEMY_DATABASE_URI = 'mysql+pymysql://USER:PASSWORD#DEFAULT:3306/DATABASE'
SQLALCHEMY_BINDS = {
'master': 'mysql+pymysql://USER:PASSWORD#MASTER_DATABSE_ENDPOINT:3306/DATABASE',
'read': 'mysql+pymysql://USER:PASSWORD#READ_REPLICA_ENDPOINT:3306/DATABASE'
}
Referring to Binds:
db.create_all(bind='read') # from read only
db.create_all(bind='master') # from master