MySQL 8 ST_GeomFromGeoJSON is throwing longitude out of range error - geometry
I have a table with polygons stored as geojson.
CREATE TABLE `location_boundaries` (
`id` INT UNSIGNED NOT NULL,
`name` VARCHAR(255) NULL DEFAULT NULL COLLATE 'utf8mb4_unicode_ci',
`geo_json` JSON NULL DEFAULT NULL,
`geom` GEOMETRY NULL DEFAULT NULL,
PRIMARY KEY (`id`)
)
COLLATE='utf8mb4_unicode_ci'
ENGINE=InnoDB
I have geojson multipolygon for asia as follows in the table:
{"type": "MultiPolygon", "coordinates": [[[[-168.25, 77.7], [-180, 77.7], [-180, 58.1], [-168.25, 58.1], [-168.25, 77.7]]], [[[39.6908, 84.52666], [180, 84.38487], [180, 26.27883], [142.084541, 22.062707], [130.147, 3.608598], [141.1373, -1.666358], [141.0438, -9.784795], [130.2645, -10.0399], [118.2545, -13.01165], [102.7975, -8.388008], [89.50451, -11.1417], [61.62511, -9.103512], [51.62645, 12.54865], [44.20775, 11.6786], [39.78016, 16.56855], [31.60401, 31.58641], [33.27769, 34.00057], [34.7674, 34.85347], [35.72423, 36.32686], [36.5597, 37.66439], [44.1053, 37.98438], [43.01638, 41.27191], [41.28304, 41.41274], [36.26378, 44.40772], [36.61315, 45.58723], [37.48493, 46.80924], [38.27497, 47.61317], [39.56164, 48.43141], [39.77264, 50.58891], [39.6908, 84.52666]]]]}
When I run the following
UPDATE location_boundaries SET geom = ST_GeomFromGeoJSON(geo_json) where id = 6255147
I'm getting the following error:
Longitude -180.000000 is out of range in function st_geomfromgeojson. It must be within (-180.000000, 180.000000].")
What's going on. All this was working fine in mysql 5.7 yet in mysql 8 everything has messed up?
Related
pg node (postgres) UPDATE statement returns column does not exist when it does exist
In pgAdmin/the cli, the following query:
UPDATE wq SET l_id = NULL, v_id = NULL WHERE w_id = 'cf93bc71-88c1-4bba-9e5c-fdc58d0ed14e';
works fine. However, when calling the same with the pg package in node:
const w_id_val = 'cf93bc71-88c1-4bba-9e5c-fdc58d0ed14e';
/*
(here client is the result of calling const pool = pg.Pool({...}) ,
then let client = await pool.connect());
*/
const result = await client.query(
`UPDATE wq
SET l_id = null,
v_id = null
WHERE w_id = $1`,
[w_id_val]
);
I get the following error:
{
"message":"column \"w_id\" does not exist",
"stack":"error: column \"w_id\" does not exist\n at Connection.parseE (/Users/lukasjenks/Documents/Work/socrative-nodejs/node_modules/pg/lib/connection.js:569:11)\n at Connection.parseMessage (/Users/lukasjenks/Documents/Work/socrative-nodejs/node_modules/pg/lib/connection.js:396:17)\n at Socket.<anonymous> (/Users/lukasjenks/Documents/Work/socrative-nodejs/node_modules/pg/lib/connection.js:132:22)\n at Socket.emit (events.js:314:20)\n at Socket.EventEmitter.emit (domain.js:483:12)\n at addChunk (_stream_readable.js:297:12)\n at readableAddChunk (_stream_readable.js:272:9)\n at Socket.Readable.push (_stream_readable.js:213:10)\n at TCP.onStreamRead (internal/stream_base_commons.js:188:23)\n at TCP.callbackTrampoline (internal/async_hooks.js:126:14)",
"name":"error",
"length":112,
"severity":"ERROR",
"code":"42703",
"position":"68",
"file":"parse_relation.c",
"line":"3514",
"routine":"errMissingColumn"
}
I can confirm the column exists with this query:
SELECT table_schema, table_name, column_name, data_type
FROM information_schema.columns
WHERE table_name = 'wq';
public wq id uuid
public wq w_id uuid
public wq l_id uuid
public wq v_id uuid
I can also confirm that the column (w_id) should be recognized by pg as when using pg to query the table with a SELECT statement, I get this back in the fields property in the result object returned:
fields: [
Field {
name: 'id',
tableID: 26611,
columnID: 1,
dataTypeID: 2950,
dataTypeSize: 16,
dataTypeModifier: -1,
format: 'text'
},
Field {
name: 'w_id',
tableID: 26611,
columnID: 3,
dataTypeID: 2950,
dataTypeSize: 16,
dataTypeModifier: -1,
format: 'text'
},
...
I've also confirmed this isn't a case issue; i.e. the column name is all lowercase and using double quotes around the column name has no effect.
Azuresql-Invalid character value for cast specification
Im trying to import data from flat file(.dat) into azure sqldb for first time (new to azure) using below command.i'm getting error as shown below. command used: bcp NAV_MO_MB in /Users/n1234/Documents/Tickets/Tickets/NEXT/SGC_test.dat" -f -S stage-nonprod-5f188055.database.windows.net -d amlstage -U aml_user -P -q -t"|" -c -e /Users/n1234/Documents/Tickets/Tickets/NEXT/err_log.txt -F2. SQLState = 22005, NativeError = 0 Error = [Microsoft][ODBC Driver 17 for SQL Server]Invalid character value for cast specification SQLState = 22001, NativeError = 0 Error = [Microsoft][ODBC Driver 17 for SQL Server]String data, right truncation``` data sample : SGC | ?| 4762| 7001297| 20/03/10|15:38:00 | 91| 1| 331| 0| 1| | -99.71| 100| 37| 353.71|OLGA |SILVA | |613 CAMINO |WALNUT |CA |0 |9095697291 |CA |US | 53/06/10|CAN4431594 | 1| 0 | 0 table created using bewlo query in sqldb** ```SET ANSI_NULLS ON GO SET QUOTED_IDENTIFIER ON GO CREATE TABLE [dbo].[NAV_MO_MB]( [SOURCE] [varchar](6) NOT NULL, [VENDOR] [nvarchar](35) NULL, [store] [INT] NOT NULL, [visit] [int] NOT NULL, [tran_date] [date], [tran_time] [time](7) null, [reg_nbr] [smallint] null, [trans_nbr] [int] null, [op_nbr] [int] null, [vo_ind] [tinyint] null, [seq_nbr] [int] null, [acc_nbr] [nvarchar](40) null, [amt] [money] null, [code] [smallint] null, [tender] [tinyint] null, [vamt] [money] null, [first_name] [nvarchar](35) null, [last_name] [nvarchar](35) null, [middle_init] [nchar](1) null, [address_line1] [nchar](30) null, [city_name] [nvarchar](30) null, [state_code] [nchar](2) null, [postal_code] [nchar](10) null, [phone_nbr] [nchar](18) null, [photo_id] [nchar](2) null, [photo_id_cnt] [nchar](2) null, [birth_date] [date] null, [id1_nbr] [nvarchar](40) null, [id1_type_code] [smallint] null, [national_id] [nvarchar](32) null, [govt_id_code] [smallint] null ) ON [PRIMARY]``` **Please help me to understood the error and the resolution?.**
You have some trailing invalid characters. The last line : [PRIMARY]'''' Remove the '''' at the end. Sometimes this happens when you copy and paste code.
Azure Cosmos Bulk-Executor Library - {\"Errors\":[\"The input content is invalid because the required properties - 'id; ' - are missing\"]}
I am trying to bulk insert documents/Json using Azure Bulk Executor library. I used the approach as described in Azure Documentation here: https://learn.microsoft.com/en-us/azure/cosmos-db/bulk-executor-java
Below is the DocumentClient creation:
public static DocumentClient documentClientFrom(BulkExecutionConfiguration cfg) throws DocumentClientException {
ConnectionPolicy policy = new ConnectionPolicy();
RetryOptions retryOptions = new RetryOptions();
retryOptions.setMaxRetryAttemptsOnThrottledRequests(0);
policy.setRetryOptions(retryOptions);
policy.setConnectionMode(cfg.getConnectionMode());
policy.setMaxPoolSize(cfg.getMaxConnectionPoolSize());
return new DocumentClient(cfg.getServiceEndpoint(), cfg.getMasterKey(), policy, cfg.getConsistencyLevel());
}
When I try to do a bulk import as below:
BulkImportResponse bulkImportResponse = bulkExecutor.importAll( documents, false, true, null );
I am getting the Error:
com.microsoft.azure.documentdb.DocumentClientException: Message: {"Errors":["Encountered exception
while executing function. Exception = Error: {\"Errors\":[\"The input content is invalid because the
required properties - 'id; ' - are missing\"]}\r\nStack trace: Error: {\"Errors\":[\"The input
content is invalid because the required properties - 'id; ' - are missing\"]}\n at createCallback
(script.js:6223:26)\n at Anonymous function (script.js:686:29)"]}
ActivityId: 3b6ca789-508d-40a2-bda1-b4e424dac88f, Request URI: /apps/6ec7164c-b528-494b-b40e-
249832f34bb1/services/221058a8-0299-4341-ad5a-0ea3d0a49cc1/partitions/ebae0619-6126-453c-b6a1-
774739a52a71/replicas/132378420031633670p/, RequestStats:
RequestStartTime: 2020-07-08T10:55:11.7899079Z, RequestEndTime: 2020-07-08T10:55:11.7999008Z, Number
of regions attempted:1
ResponseTime: 2020-07-08T10:55:11.7999008Z, StoreResult: StorePhysicalAddress: rntbd://cdb-ms-prod-
southcentralus1-fd11.documents.azure.com:14397/apps/6ec7164c-b528-494b-b40e-
249832f34bb1/services/221058a8-0299-4341-ad5a-0ea3d0a49cc1/partitions/ebae0619-6126-453c-b6a1-
774739a52a71/replicas/132378420031633670p/, LSN: 105, GlobalCommittedLsn: 105, PartitionKeyRangeId:
0, IsValid: True, StatusCode: 400, SubStatusCode: 400, RequestCharge: 4.38, ItemLSN: -1,
SessionToken: -1#105, UsingLocalLSN: False, TransportException: null, ResourceType: StoredProcedure,
OperationType: ExecuteJavaScript
, SDK: Microsoft.Azure.Documents.Common/2.11.0, StatusCode: BadRequest
at com.microsoft.azure.documentdb.internal.ErrorUtils.maybeThrowException(ErrorUtils.java:74)
at com.microsoft.azure.documentdb.internal.GatewayProxy.performPostRequest(GatewayProxy.java:284)
at com.microsoft.azure.documentdb.internal.GatewayProxy.doExecute(GatewayProxy.java:108)
at com.microsoft.azure.documentdb.internal.GatewayProxy.processMessage(GatewayProxy.java:360)
at com.microsoft.azure.documentdb.DocumentClient$5.apply(DocumentClient.java:3037)
at com.microsoft.azure.documentdb.internal.RetryUtility.executeDocumentClientRequest(RetryUtility.java:73)
at com.microsoft.azure.documentdb.DocumentClient.doCreate(DocumentClient.java:3042)
at com.microsoft.azure.documentdb.DocumentClient.executeStoredProcedure(DocumentClient.java:1571)
at com.microsoft.azure.documentdb.bulkexecutor.internal.BatchInserter$1.call(BatchInserter.java:186)
at com.microsoft.azure.documentdb.bulkexecutor.internal.BatchInserter$1.call(BatchInserter.java:158)
at com.microsoft.azure.documentdb.repackaged.com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.microsoft.azure.documentdb.repackaged.com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
at com.microsoft.azure.documentdb.repackaged.com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Here is json for the object I am trying to insert:
.........
"grocinvoiceInd": "N",
"_id": {
"transactionId": {
"$numberLong": "<xxxxxx>"
},
"txnSeqNbr": 0
},
"sourceInfo": "<xxxxxxx>"
}
Apparently it looks like it has a _id field but its not inserting the record. Any help is highly appreciated.
it is necessary for the document to have an id field, because you've created a collection with a partition key, if you create a collection with a partition key, all the documents in that collection must have that property https://learn.microsoft.com/en-us/azure/cosmos-db/partitioning-overview Also as described in Azure documentation https://learn.microsoft.com/en-us/azure/cosmos-db/bulk-executor-java "Currently, the bulk executor library is supported only by Azure Cosmos DB SQL API and Gremlin API accounts." Considering this we bulk executor may not work with MongoDB storage accounts at this time.
Sequelize "Connection closed" on Azure SQL
I have a Node v10.17.0 app on Ubuntu 16.04 using Sequelize 4.41.1 calling Azure SQL.
On every start up, the node app runs sync on each database model for any migrations. Sequelize sends queries to the database IF OBJECT_ID('[Shop]', 'U') IS NULL CREATE TABLE [Client] ... etc to create the tables if they don't exist. Then EXEC sys.sp_helpindex #objname = N''[Shop]. The first couple of tables sync successfully (there aren't any changes).
Then we get to the query that returns the error SequelizeDatabaseError message":"Connection closed before request completed." "code":"ECLOSE"
IF OBJECT_ID('[Client]', 'U') IS NULL CREATE TABLE [Client] ([id] NVARCHAR(255) , [name] NVARCHAR(255) NOT NULL, [timezone] VARCHAR(255) CHECK ([timezone] IN(N'Africa/Abidjan', N'Africa/Accra', N'Africa/Addis_Ababa', N'Africa/Algiers', N'Africa/Asmara', N'Africa/Asmera', N'Africa/Bamako', N'Africa/Bangui', N'Africa/Banjul', N'Africa/Bissau', N'Africa/Blantyre', N'Africa/Brazzaville', N'Africa/Bujumbura', N'Africa/Cairo', N'Africa/Casablanca', N'Africa/Ceuta', N'Africa/Conakry', N'Africa/Dakar', N'Africa/Dar_es_Salaam', N'Africa/Djibouti', N'Africa/Douala', N'Africa/El_Aaiun', N'Africa/Freetown', N'Africa/Gaborone', N'Africa/Harare', N'Africa/Johannesburg', N'Africa/Juba', N'Africa/Kampala', N'Africa/Khartoum', N'Africa/Kigali', N'Africa/Kinshasa', N'Africa/Lagos', N'Africa/Libreville', N'Africa/Lome', N'Africa/Luanda', N'Africa/Lubumbashi', N'Africa/Lusaka', N'Africa/Malabo', N'Africa/Maputo', N'Africa/Maseru', N'Africa/Mbabane', N'Africa/Mogadishu', N'Africa/Monrovia', N'Africa/Nairobi', N'Africa/Ndjamena', N'Africa/Niamey', N'Africa/Nouakchott', N'Africa/Ouagadougou', N'Africa/Porto-Novo', N'Africa/Sao_Tome', N'Africa/Timbuktu', N'Africa/Tripoli', N'Africa/Tunis', N'Africa/Windhoek', N'America/Adak', N'America/Anchorage', N'America/Anguilla', N'America/Antigua', N'America/Araguaina', N'America/Argentina/Buenos_Aires', N'America/Argentina/Catamarca', N'America/Argentina/ComodRivadavia', N'America/Argentina/Cordoba', N'America/Argentina/Jujuy', N'America/Argentina/La_Rioja', N'America/Argentina/Mendoza', N'America/Argentina/Rio_Gallegos', N'America/Argentina/Salta', N'America/Argentina/San_Juan', N'America/Argentina/San_Luis', N'America/Argentina/Tucuman', N'America/Argentina/Ushuaia', N'America/Aruba', N'America/Asuncion', N'America/Atikokan', N'America/Atka', N'America/Bahia', N'America/Bahia_Banderas', N'America/Barbados', N'America/Belem', N'America/Belize', N'America/Blanc-Sablon', N'America/Boa_Vista', N'America/Bogota', N'America/Boise', N'America/Buenos_Aires', N'America/Cambridge_Bay', N'America/Campo_Grande', N'America/Cancun', N'America/Caracas', N'America/Catamarca', N'America/Cayenne', N'America/Cayman', N'America/Chicago', N'America/Chihuahua', N'America/Coral_Harbour', N'America/Cordoba', N'America/Costa_Rica', N'America/Creston', N'America/Cuiaba', N'America/Curacao', N'America/Danmarkshavn', N'America/Dawson', N'America/Dawson_Creek', N'America/Denver', N'America/Detroit', N'America/Dominica', N'America/Edmonton', N'America/Eirunepe', N'America/El_Salvador', N'America/Ensenada', N'America/Fort_Nelson', N'America/Fort_Wayne', N'America/Fortaleza', N'America/Glace_Bay', N'America/Godthab', N'America/Goose_Bay', N'America/Grand_Turk', N'America/Grenada', N'America/Guadeloupe', N'America/Guatemala', N'America/Guayaquil', N'America/Guyana', N'America/Halifax', N'America/Havana', N'America/Hermosillo', N'America/Indiana/Indianapolis', N'America/Indiana/Knox', N'America/Indiana/Marengo', N'America/Indiana/Petersburg', N'America/Indiana/Tell_City', N'America/Indiana/Vevay', N'America/Indiana/Vincennes', N'America/Indiana/Winamac', N'America/Indianapolis', N'America/Inuvik', N'America/Iqaluit', N'America/Jamaica', N'America/Jujuy', N'America/Juneau', N'America/Kentucky/Louisville', N'America/Kentucky/Monticello', N'America/Knox_IN', N'America/Kralendijk', N'America/La_Paz', N'America/Lima', N'America/Los_Angeles', N'America/Louisville', N'America/Lower_Princes', N'America/Maceio', N'America/Managua', N'America/Manaus', N'America/Marigot', N'America/Martinique', N'America/Matamoros', N'America/Mazatlan', N'America/Mendoza', N'America/Menominee', N'America/Merida', N'America/Metlakatla', N'America/Mexico_City', N'America/Miquelon', N'America/Moncton', N'America/Monterrey', N'America/Montevideo', N'America/Montreal', N'America/Montserrat', N'America/Nassau', N'America/New_York', N'America/Nipigon', N'America/Nome', N'America/Noronha', N'America/North_Dakota/Beulah', N'America/North_Dakota/Center', N'America/North_Dakota/New_Salem', N'America/Ojinaga', N'America/Panama', N'America/Pangnirtung', N'America/Paramaribo', N'America/Phoenix', N'America/Port-au-Prince', N'America/Port_of_Spain', N'America/Porto_Acre', N'America/Porto_Velho', N'America/Puerto_Rico', N'America/Punta_Arenas', N'America/Rainy_River', N'America/Rankin_Inlet', N'America/Recife', N'America/Regina', N'America/Resolute', N'America/Rio_Branco', N'America/Rosario', N'America/Santa_Isabel', N'America/Santarem', N'America/Santiago', N'America/Santo_Domingo', N'America/Sao_Paulo', N'America/Scoresbysund', N'America/Shiprock', N'America/Sitka', N'America/St_Barthelemy', N'America/St_Johns', N'America/St_Kitts', N'America/St_Lucia', N'America/St_Thomas', N'America/St_Vincent', N'America/Swift_Current', N'America/Tegucigalpa', N'America/Thule', N'America/Thunder_Bay', N'America/Tijuana', N'America/Toronto', N'America/Tortola', N'America/Vancouver', N'America/Virgin', N'America/Whitehorse', N'America/Winnipeg', N'America/Yakutat', N'America/Yellowknife', N'Antarctica/Casey', N'Antarctica/Davis', N'Antarctica/DumontDUrville', N'Antarctica/Macquarie', N'Antarctica/Mawson', N'Antarctica/McMurdo', N'Antarctica/Palmer', N'Antarctica/Rothera', N'Antarctica/South_Pole', N'Antarctica/Syowa', N'Antarctica/Troll', N'Antarctica/Vostok', N'Arctic/Longyearbyen', N'Asia/Aden', N'Asia/Almaty', N'Asia/Amman', N'Asia/Anadyr', N'Asia/Aqtau', N'Asia/Aqtobe', N'Asia/Ashgabat', N'Asia/Ashkhabad', N'Asia/Atyrau', N'Asia/Baghdad', N'Asia/Bahrain', N'Asia/Baku', N'Asia/Bangkok', N'Asia/Barnaul', N'Asia/Beirut', N'Asia/Bishkek', N'Asia/Brunei', N'Asia/Calcutta', N'Asia/Chita', N'Asia/Choibalsan', N'Asia/Chongqing', N'Asia/Chungking', N'Asia/Colombo', N'Asia/Dacca', N'Asia/Damascus', N'Asia/Dhaka', N'Asia/Dili', N'Asia/Dubai', N'Asia/Dushanbe', N'Asia/Famagusta', N'Asia/Gaza', N'Asia/Harbin', N'Asia/Hebron', N'Asia/Ho_Chi_Minh', N'Asia/Hong_Kong', N'Asia/Hovd', N'Asia/Irkutsk', N'Asia/Istanbul', N'Asia/Jakarta', N'Asia/Jayapura', N'Asia/Jerusalem', N'Asia/Kabul', N'Asia/Kamchatka', N'Asia/Karachi', N'Asia/Kashgar', N'Asia/Kathmandu', N'Asia/Katmandu', N'Asia/Khandyga', N'Asia/Kolkata', N'Asia/Krasnoyarsk', N'Asia/Kuala_Lumpur', N'Asia/Kuching', N'Asia/Kuwait', N'Asia/Macao', N'Asia/Macau', N'Asia/Magadan', N'Asia/Makassar', N'Asia/Manila', N'Asia/Muscat', N'Asia/Nicosia', N'Asia/Novokuznetsk', N'Asia/Novosibirsk', N'Asia/Omsk', N'Asia/Oral', N'Asia/Phnom_Penh', N'Asia/Pontianak', N'Asia/Pyongyang', N'Asia/Qatar', N'Asia/Qyzylorda', N'Asia/Rangoon', N'Asia/Riyadh', N'Asia/Saigon', N'Asia/Sakhalin', N'Asia/Samarkand', N'Asia/Seoul', N'Asia/Shanghai', N'Asia/Singapore', N'Asia/Srednekolymsk', N'Asia/Taipei', N'Asia/Tashkent', N'Asia/Tbilisi', N'Asia/Tehran', N'Asia/Tel_Aviv', N'Asia/Thimbu', N'Asia/Thimphu', N'Asia/Tokyo', N'Asia/Tomsk', N'Asia/Ujung_Pandang', N'Asia/Ulaanbaatar', N'Asia/Ulan_Bator', N'Asia/Urumqi', N'Asia/Ust-Nera', N'Asia/Vientiane', N'Asia/Vladivostok', N'Asia/Yakutsk', N'Asia/Yangon', N'Asia/Yekaterinburg', N'Asia/Yerevan', N'Atlantic/Azores', N'Atlantic/Bermuda', N'Atlantic/Canary', N'Atlantic/Cape_Verde', N'Atlantic/Faeroe', N'Atlantic/Faroe', N'Atlantic/Jan_Mayen', N'Atlantic/Madeira', N'Atlantic/Reykjavik', N'Atlantic/South_Georgia', N'Atlantic/St_Helena', N'Atlantic/Stanley', N'Australia/ACT', N'Australia/Adelaide', N'Australia/Brisbane', N'Australia/Broken_Hill', N'Australia/Canberra', N'Australia/Currie', N'Australia/Darwin', N'Australia/Eucla', N'Australia/Hobart', N'Australia/LHI', N'Australia/Lindeman', N'Australia/Lord_Howe', N'Australia/Melbourne', N'Australia/NSW', N'Australia/North', N'Australia/Perth', N'Australia/Queensland', N'Australia/South', N'Australia/Sydney', N'Australia/Tasmania', N'Australia/Victoria', N'Australia/West', N'Australia/Yancowinna', N'Brazil/Acre', N'Brazil/DeNoronha', N'Brazil/East', N'Brazil/West', N'CET', N'CST6CDT', N'Canada/Atlantic', N'Canada/Central', N'Canada/Eastern', N'Canada/Mountain', N'Canada/Newfoundland', N'Canada/Pacific', N'Canada/Saskatchewan', N'Canada/Yukon', N'Chile/Continental', N'Chile/EasterIsland', N'Cuba', N'EET', N'EST', N'EST5EDT', N'Egypt', N'Eire', N'Etc/GMT', N'Etc/GMT+0', N'Etc/GMT+1', N'Etc/GMT+10', N'Etc/GMT+11', N'Etc/GMT+12', N'Etc/GMT+2', N'Etc/GMT+3', N'Etc/GMT+4', N'Etc/GMT+5', N'Etc/GMT+6', N'Etc/GMT+7', N'Etc/GMT+8', N'Etc/GMT+9', N'Etc/GMT-0', N'Etc/GMT-1', N'Etc/GMT-10', N'Etc/GMT-11', N'Etc/GMT-12', N'Etc/GMT-13', N'Etc/GMT-14', N'Etc/GMT-2', N'Etc/GMT-3', N'Etc/GMT-4', N'Etc/GMT-5', N'Etc/GMT-6', N'Etc/GMT-7', N'Etc/GMT-8', N'Etc/GMT-9', N'Etc/GMT0', N'Etc/Greenwich', N'Etc/UCT', N'Etc/UTC', N'Etc/Universal', N'Etc/Zulu', N'Europe/Amsterdam', N'Europe/Andorra', N'Europe/Astrakhan', N'Europe/Athens', N'Europe/Belfast', N'Europe/Belgrade', N'Europe/Berlin', N'Europe/Bratislava', N'Europe/Brussels', N'Europe/Bucharest', N'Europe/Budapest', N'Europe/Busingen', N'Europe/Chisinau', N'Europe/Copenhagen', N'Europe/Dublin', N'Europe/Gibraltar', N'Europe/Guernsey', N'Europe/Helsinki', N'Europe/Isle_of_Man', N'Europe/Istanbul', N'Europe/Jersey', N'Europe/Kaliningrad', N'Europe/Kiev', N'Europe/Kirov', N'Europe/Lisbon', N'Europe/Ljubljana', N'Europe/London', N'Europe/Luxembourg', N'Europe/Madrid', N'Europe/Malta', N'Europe/Mariehamn', N'Europe/Minsk', N'Europe/Monaco', N'Europe/Moscow', N'Europe/Nicosia', N'Europe/Oslo', N'Europe/Paris', N'Europe/Podgorica', N'Europe/Prague', N'Europe/Riga', N'Europe/Rome', N'Europe/Samara', N'Europe/San_Marino', N'Europe/Sarajevo', N'Europe/Saratov', N'Europe/Simferopol', N'Europe/Skopje', N'Europe/Sofia', N'Europe/Stockholm', N'Europe/Tallinn', N'Europe/Tirane', N'Europe/Tiraspol', N'Europe/Ulyanovsk', N'Europe/Uzhgorod', N'Europe/Vaduz', N'Europe/Vatican', N'Europe/Vienna', N'Europe/Vilnius', N'Europe/Volgograd', N'Europe/Warsaw', N'Europe/Zagreb', N'Europe/Zaporozhye', N'Europe/Zurich', N'GB', N'GB-Eire', N'GMT', N'GMT+0', N'GMT-0', N'GMT0', N'Greenwich', N'HST', N'Hongkong', N'Iceland', N'Indian/Antananarivo', N'Indian/Chagos', N'Indian/Christmas', N'Indian/Cocos', N'Indian/Comoro', N'Indian/Kerguelen', N'Indian/Mahe', N'Indian/Maldives', N'Indian/Mauritius', N'Indian/Mayotte', N'Indian/Reunion', N'Iran', N'Israel', N'Jamaica', N'Japan', N'Kwajalein', N'Libya', N'MET', N'MST', N'MST7MDT', N'Mexico/BajaNorte', N'Mexico/BajaSur', N'Mexico/General', N'NZ', N'NZ-CHAT', N'Navajo', N'PRC', N'PST8PDT', N'Pacific/Apia', N'Pacific/Auckland', N'Pacific/Bougainville', N'Pacific/Chatham', N'Pacific/Chuuk', N'Pacific/Easter', N'Pacific/Efate', N'Pacific/Enderbury', N'Pacific/Fakaofo', N'Pacific/Fiji', N'Pacific/Funafuti', N'Pacific/Galapagos', N'Pacific/Gambier', N'Pacific/Guadalcanal', N'Pacific/Guam', N'Pacific/Honolulu', N'Pacific/Johnston', N'Pacific/Kiritimati', N'Pacific/Kosrae', N'Pacific/Kwajalein', N'Pacific/Majuro', N'Pacific/Marquesas', N'Pacific/Midway', N'Pacific/Nauru', N'Pacific/Niue', N'Pacific/Norfolk', N'Pacific/Noumea', N'Pacific/Pago_Pago', N'Pacific/Palau', N'Pacific/Pitcairn', N'Pacific/Pohnpei', N'Pacific/Ponape', N'Pacific/Port_Moresby', N'Pacific/Rarotonga', N'Pacific/Saipan', N'Pacific/Samoa', N'Pacific/Tahiti', N'Pacific/Tarawa', N'Pacific/Tongatapu', N'Pacific/Truk', N'Pacific/Wake', N'Pacific/Wallis', N'Pacific/Yap', N'Poland', N'Portugal', N'ROC', N'ROK', N'Singapore', N'Turkey', N'UCT', N'US/Alaska', N'US/Aleutian', N'US/Arizona', N'US/Central', N'US/East-Indiana', N'US/Eastern', N'US/Hawaii', N'US/Indiana-Starke', N'US/Michigan', N'US/Mountain', N'US/Pacific', N'US/Pacific-New', N'US/Samoa', N'UTC', N'Universal', N'W-SU', N'WET', N'Zulu')), [clipTemplateId] NVARCHAR(255) NULL, [shopId] NVARCHAR(255) NULL, [createdAt] DATETIMEOFFSET NOT NULL, [updatedAt] DATETIMEOFFSET NOT NULL, PRIMARY KEY ([id]), FOREIGN KEY ([clipTemplateId]) REFERENCES [ClipTemplate] ([id]) ON DELETE NO ACTION, FOREIGN KEY ([shopId]) REFERENCES [Shop] ([id]) ON DELETE NO ACTION);
Same format as the other queries, except it's got a list of timezones. I can see the previous queries in the audit log, but not this one.
I've run SELECT OBJECT_ID('[Client]', 'U') and it returns 1218103380, so the query shouldn't actually be doing anything, given the IF at the top?
Reproducable every time against multiple Azure SQL databases in different development environments with the node app. Other SQL clients send the query with success. App last worked yesterday. No app or infra changes since.
When putting the query into some online SQL validators like https://www.eversql.com/sql-syntax-check-validator/, it says the query is invalid, but MS SQL Server Management Studio thinks it's fine, and I can run it without issue.
Sequelize Config:
database: {
database: process.env.DATABASE_NAME,
dialect: "mssql",
host: process.env.DATABASE_HOST,
loggingEnabled: false,
password: process.env.DATABASE_PASSWORD,
pool: {
acquire: 30000,
idle: 30000,
max: 10,
min: 0,
},
port: parseInt(process.env.DATABASE_PORT, 10) || 1433,
transactionTimeout: parseInt(process.env.DB_TRANSACTION_TIMEOUT, 10) || 30000,
username: process.env.DATABASE_USERNAME,
}
It seems 10.17.0 broke my DB connection for long queries. Rolling back to Node v10.16.3 has resolved the issue. On start up, my Node program was pulling the latest 10.x version. It started using Node v10.17.0, released 21st October 2019, which was when the issue started.
Here is a solution for timeout error while running long queries. connection timeout or already too many clients
Convert HQL to SparkSQL
I'm trying to convert HQL to Spark.
I have the following query (Works in Hue with Hive editor):
select reflect('java.util.UUID', 'randomUUID') as id,
tt.employee,
cast( from_unixtime(unix_timestamp (date_format(current_date(),'dd/MM/yyyy HH:mm:ss'), 'dd/MM/yyyy HH:mm:ss')) as timestamp) as insert_date,
collect_set(tt.employee_detail) as employee_details,
collect_set( tt.emp_indication ) as employees_indications,
named_struct ('employee_info', collect_set(tt.emp_info),
'employee_mod_info', collect_set(tt.emp_mod_info),
'employee_comments', collect_set(tt.emp_comment) )
as emp_mod_details,
from (
select views_ctr.employee,
if ( views_ctr.employee_details.so is not null, views_ctr.employee_details, null ) employee_detail,
if ( views_ctr.employee_info.so is not null, views_ctr.employee_info, null ) emp_info,
if ( views_ctr.employee_comments.so is not null, views_ctr.employee_comments, null ) emp_comment,
if ( views_ctr.employee_mod_info.so is not null, views_ctr.employee_mod_info, null ) emp_mod_info,
if ( views_ctr.emp_indications.so is not null, views_ctr.emp_indications, null ) employees_indication,
from
( select * from views_sta where emp_partition=0 and employee is not null ) views_ctr
) tt
group by employee
distribute by employee
First, What I'm trying is to write it in spark.sql as follow:
sparkSession.sql("select reflect('java.util.UUID', 'randomUUID') as id, tt.employee, cast( from_unixtime(unix_timestamp (date_format(current_date(),'dd/MM/yyyy HH:mm:ss'), 'dd/MM/yyyy HH:mm:ss')) as timestamp) as insert_date, collect_set(tt.employee_detail) as employee_details, collect_set( tt.emp_indication ) as employees_indications, named_struct ('employee_info', collect_set(tt.emp_info), 'employee_mod_info', collect_set(tt.emp_mod_info), 'employee_comments', collect_set(tt.emp_comment) ) as emp_mod_details, from ( select views_ctr.employee, if ( views_ctr.employee_details.so is not null, views_ctr.employee_details, null ) employee_detail, if ( views_ctr.employee_info.so is not null, views_ctr.employee_info, null ) emp_info, if ( views_ctr.employee_comments.so is not null, views_ctr.employee_comments, null ) emp_comment, if ( views_ctr.employee_mod_info.so is not null, views_ctr.employee_mod_info, null ) emp_mod_info, if ( views_ctr.emp_indications.so is not null, views_ctr.emp_indications, null ) employees_indication, from ( select * from views_sta where emp_partition=0 and employee is not null ) views_ctr ) tt group by employee distribute by employee")
But I got the following exception:
Exception in thread "main" org.apache.spark.SparkException: Job
aborted due to stage failute: Task not serializable:
java.io.NotSerializableException:
org.apache.spark.unsafe.types.UTF8String$IntWrapper
-object not serializable (class : org.apache.spark.unsafe.types.UTF8String$IntWrapper, value:
org.apache.spark.unsafe.types.UTF8String$IntWrapper#30cfd641)
If I'm trying to run my query without collect_set function its work, It's can fail because struct column types in my table?
How can I write my HQL query in Spark / fix my exception?