I need to load the data in psql db time-to-time via a node app.
I'm using node-postgres package, which seems to be working fine with INSERT statements.
Since my db_dum is huge, I need to move to COPY statement in pg_dump (for better performance), but getting all kinds of error while trying to load the data with pg package in Node - This works find if I use command line psql
the psql dump file I have is huge which includes COPY statements like this:
COPY hibernate.address (id, create_timestamp,
update_timestamp, street_address_line_1, street_address_line_2, city, state_code, postal_code, postal_code_suffix, country_code, latitude, longitude, updated_by_base_user_id) FROM stdin;
379173 2017-02-20 02:34:17.715-08 2018-01-20 08:34:17.715-08 3 Brewster St \N Phoenix AZ 17349 \N US \N \N 719826
/.
Here's a pseudo code for the node app running the sql dump file:
const sqlFile = fs.readFileSync('data_dump.sql').toString();
const connectionString = `postgres://<user>:${PgPassword}#${pgIpAndPort}/<table>`;
const client = new pg.Client(connectionString);
lient.connect();
client.query(sqlFile);
Here's a sample pg_dump command I use (which is for data-only - no schema):
pg_dump -U <user> --data-only --disable-triggers -n hibernate <table_name> > <dump_file.sql>
but it doesn't work when I'm trying to load the data via node app
I know --column-inserts would solve the problem, but that decreases the performance drastically.
So I'm looking for possible solutions for loading the data with COPY tbl FROM stdin; statement in the node app
Any suggestion/comments is appreciated.
Related
This is in reference to rather comment (not answer), I added here: MLflow: INVALID_PARAMETER_VALUE: Unsupported URI './mlruns' for model registry store
I extracted files from here
train.py MLproject wine-quality.csv
These are in directory:feb24MLFLOW
I am in directory feb24MLFLOW with following contents
:memory mlruns train.py wine-quality.csv
When I run following command
mlflow server --backend-store-uri sqlite:///:memory --default-artifact-root ./mlruns
The UI loads but does not show any data in it neigther does database as below. see screenshot.
I am using --default-artifact-root ./mlruns flag because, when I print print(mlflow.get_tracking_uri()), I get the current directory
file:///<mydirectorylocations>/feb24MLFLOW/mlruns
For some reason I see my database is not updating (or inserting). I checked that with in terminal.
$ sqlite3
sqlite> .open :memory
sqlite> .tables
alembic_version metrics registered_model_tags
experiment_tags model_version_tags registered_models
experiments model_versions runs
latest_metrics params tags
sqlite> select * from runs;
sqlite>
As you can see there is no data after running select * from runs above.
Please note that I have following contents in
./mlruns
d6db5cf1443d49c19971a1b8b606d692 meta.yaml
Can somebody suggest I show results in the UI? or insert in databse? or what am I doing wrong?
Please note that when I run mlflow ui, I see data in the UI but I get:
error_code: "INVALID_PARAMETER_VALUE"
message: " Model registry functionality is unavailable; got unsupported URI './mlruns' for model registry data storage. Supported URI schemes are: ['postgresql', 'mysql', 'sqlite', 'mssql']. See https://www.mlflow.org/docs/latest/tracking.html#storage for how to run an MLflow server against one of the supported backend storage locations."
I have an excel sheet with employee data. My task is to store the data from excel file in MongoDB database > employee collection(a row from excel sheet in mongodb document). I'm doing all this in a react application. I thought of using mongoimport. Since I need it in a CSV or Json format, I converted my excel file to CSV using SheetJs npm package and created a blob file of type csv. And then using the below command I was able to import that CSV file to my mongoDB database.
mongoimport --db demo --collection employees --type csv --headerline --file /path/to/myfile.csv
But I did this from mongo shell by giving a path of my local disk. Now I'm trying to implement this within my react app. Initially I proceeded with this idea - as soon as I upload an excel file, I will convert that to CSV file and I will call a post api with that CSV file in body. Upon sending that request, I will call the "mongoimport" command in my nodejs backend/server so that the data from that CSV file will be stored in my mongoDb collection. Now I can't find any solution to usmongoimport command programmatically. How can I call the "Mongoimport" command in my nodejs server code? I couldn't find any documentation regarding it.
If that is not the right way of doing this task, please suggest me any entirely other way of achieving this task.
In Layman's terms I want to import data from an excel file to MongoDb database using Reactjs/nodejs app.
how are you?
First of all, mongoimport also allows you to import TSV files (same command as for CSV but putting --type tsv), many times friendlier to use with Excel.
Regarding mongoImport, I regret to report that mongoImport cannot be used by any means other than the command line.
What you can do from NodeJs is execute commands in the same way that they are executed by a terminal. For this you can use the child_proccess Module or Exec Function.
I hope that helps even a little.
Regards!
I need to record temperatures to a SQLite-DB on a linux system (using bash)
My problem is that I get the temperature readings in an individual file.
How can I get that reading into the SQLite command
sqlite3 mydb "INSERT INTO readings (TStamp, reading) VALUES (datetime(), 'xxx');"
The file contains just one line with the value "45.7" and should replace the xxx.
Using fix data the SQL command works pretty well.
You can simply echo commands to the sqlite3, just like this:
temp=`cat file_with_temperature_value`
echo "INSERT INTO readings (TStamp, reading) VALUES (datetime(), '$temp');" | sqlite3 mydb
or do it like in your example:
temp=`cat file_with_temperature_value`
sqlite3 mydb "INSERT INTO readings (TStamp, reading) VALUES (datetime(), '$temp');"
I am using the psycopg2 library with Python3 on a linux server to create some temporary tables on Redshift and querying these tables to get results and write to files on the server.
Since my queries are long and takes about 15 minutes to create all these temp tables that I ultimate pull data from, how do I ensure that my connection persists and I don't lose the temp tables that I later query? Right now I just do a cursor() before the execute(), is there a default timeout for these?
I have noticed that whenever I do a
Select a,b from #results_table
or
select * from #results_table
the query just freezes/hangs, but
select top 35 from #results_table
returns the results (select top 40 fails!). There are about a 100 rows in #results_table, and I am not able to get them all. I did a ps aux and the process just stays in the S+ state. If I manually run the query on Redshift it finishes in seconds.
Any ideas?
Re-declaring a cursor doesn't create new connection while using psycopg2.
I have table with more than 3 000 000 rows. I have try to export the data from it manually and with SQL Server Management Studio Export data functionality to Excel but I have met several problems:
when create .txt file manually copying and pasting the data (this is several times, because if you copy all rows from the SQL Server Management Studio it throws out of memory error) I am not able to open it with any text editor and to copy the rows;
the Export data to Excel do not work, because Excel do not support so many rows
Finally, with the Export data functionality I have created a .sql file, but it is 1.5 GB, and I am not able to open it in SQL Server Management Studio again.
Is there a way to import it with the Import data functionality, or other more clever way to make a backup of the information of my table and then to import it again if I need it?
Thanks in advance.
I am not quite sure if I understand your requirements (I don't know if you need to export your data to excel or you want to make some kind of backup).
In order to export data from single tables, you could use Bulk Copy Tool which allows you to export data from single tables and exporting/Importing it to files. You can also use a custom Query to export the data.
It is important that this does not generate a Excel file, but another format. You could use this to move data from one database to another (must be MS SQL in both cases).
Examples:
Create a format file:
Bcp [TABLE_TO_EXPORT] format "[EXPORT_FILE]" -n -f "[ FORMAT_FILE]" -S [SERVER] -E -T -a 65535
Export all Data from a table:
bcp [TABLE_TO_EXPORT] out "[EXPORT_FILE]" -f "[FORMAT_FILE]" -S [SERVER] -E -T -a 65535
Import the previously exported data:
bcp [TABLE_TO_EXPORT] in [EXPORT_FILE]" -f "[FORMAT_FILE] " -S [SERVER] -E -T -a 65535
I redirect the output from hte export/import operations to a logfile (by appending "> mylogfile.log" ad the end of the commands) - this helps if you are exporting a lot of data.
Here a way of doing it without bcp:
EXPORT THE SCHEMA AND DATA IN A FILE
Use the ssms wizard
Database >> Tasks >> generate Scripts… >> Choose the table >> choose db model and schema
Save the SQL file (can be huge)
Transfer the SQL file on the other server
SPLIT THE DATA IN SEVERAL FILES
Use a program like textfilesplitter to split the file in smaller files and split in files of 10 000 lines (so each file is not too big)
Put all the files in the same folder, with nothing else
IMPORT THE DATA IN THE SECOND SERVER
Create a .bat file in the same folder, name execFiles.bat
You may need to check the table schema to disable the identity in the first file, you can add that after the import in finished.
This will execute all the files in the folder against the server and the database with, the –f define the Unicode text encoding should be used to handle the accents:
for %%G in (*.sql) do sqlcmd /S ServerName /d DatabaseName -E -i"%%G" -f 65001
pause