How to use pgloader in postgres/pgadmin platform?
How can I upload fixed width length column value from text file?
Example:
**column_name position length**
given_name 1 20
mobile 22 15
This is what i did in a hurry, using the table.load file:
LOAD FIXED FROM stdin
WITH ENCODING utf8
(
given_name from 0 for 20
, mobile from 21 for 15
)
INTO postgresql://user:pwd#host:port/database?schema.table
(
given_name
, mobile
);
Using the command:
cat yourfile.txt | pgloader table.load
I got it loaded.
Regards
Related
I'm running some simulation code using ubuntu and I keep running into the same error. I am trying to read data from the .dat file. But there is some error which I could not find.
This is error message:
At line 1939 of file CompoundMPIBSC20200823.f90 (unit = 11, file = 'C-340120b.dat')
Fortran runtime error: Bad value during floating point read"
And C-340120b.dat file looks like this:
C-340120b.dat file
6 10 1.531581196563372e-15
0.0014553174 0.0055615333 0.0119703978 0.0203850084 0.0305528957 0.0422600997 0.0553257997 0.0695976542 0.0849475255 0.1012676622
0.1184670631 0.1364683308 0.1552047081 0.1746171362 0.1946516651 0.2152575030 0.2363847713 0.2579826431 0.2799978445 0.3023733948
0.3250475021 0.3479528735 0.3710162901 0.3941586119 0.4172949742 0.4403351781 0.4631846022 0.4857450840 0.5079162267 0.5295967855
0.5506862041 0.5710862773 0.5907027347 0.6094469646 0.6272378474 0.6440030470 0.6596808652 0.6742212377 0.6875870601 0.6997550335
0.7107163254 0.7204769965 0.7290581527 0.7364958286 0.7428406051 0.7481569620 0.7525223874 0.7560262383 0.7587683705 0.7608575997
0.7624099924 0.7635469966 0.7643935146 0.7650758792 0.7657198055 0.7664483527 0.7673799150 0.7686263022 0.7702908977 0.7724669898
0.7752362202 0.7786672785 0.7828147402 0.7877181912 0.7934015601 0.7998727305 0.8071233638 0.8151290525 0.8238496666 0.8332299862
0.8432005800 0.8536788860 0.8645704950 0.8757706624 0.8871659719 0.8986361128 0.9100557539 0.9212965992 0.9322292652 0.9427254733
0.9526599682 0.9619125838 0.9703700425 0.9779277777 0.9844915839 0.9899791031 0.9943211058 0.9974625834 0.9993636185 1.0000000000
6 11 1.475893077189510e-15
0.0016525844 0.0062956494 0.0135059282 0.0229208176 0.0342303569 0.0471704147 0.0615165460 0.0770787515 0.0936966304 0.1112350678
0.1295802432 0.1486359799 0.1683205560 0.1885633307 0.2093018003 0.2304793480 0.2520427451 0.2739399970 0.2961185348 0.3185236648
0.3410972443 0.3637767806 0.3864948806 0.4091790904 0.4317517598 0.4541305840 0.4762291287 0.4979578467 0.5192252758 0.5399391683
0.5600082344 0.5793434896 0.5978599757 0.6154784463 0.6321271026 0.6477428598 0.6622731054 0.6756767045 0.6879251243 0.6990032865
0.7089101737 0.7176591771 0.7252781911 0.7318094543 0.7373091004 0.7418464594 0.7455031054 0.7483717148 0.7505546359 0.7521623630
0.7533118077 0.7541244615 0.7547244701 0.7552366540 0.7557845046 0.7564881945 0.7574626361 0.7588156121 0.7606460387 0.7630423235
0.7660809513 0.7698252039 0.7743241221 0.7796116798 0.7857062078 0.7926100427 0.8003094738 0.8087748795 0.8179612002 0.8278085409
0.8382431405 0.8491784407 0.8605164079 0.8721490905 0.8839601234 0.8958267320 0.9076214149 0.9192140226 0.9304737122 0.9412709632
0.9514795610 0.9609785827 0.9696542131 0.9774014787 0.9841259295 0.9897450178 0.9941894123 0.9974040480 0.9993489837 1.0000000000
...
...
...
6 30500 2.203435261320421e-18
0.5647132406 0.8435296561 0.9197993603 0.9501219587 0.9657979424 0.9751478483 0.9812026747 0.9853454006 0.9882967561 0.9904674356
0.9921063590 0.9933715119 0.9943670960 0.9951635956 0.9958099778 0.9963412928 0.9967830203 0.9971540150 0.9974684566 0.9977371653
0.9979685074 0.9981690312 0.9983439172 0.9984973073 0.9986325426 0.9987523421 0.9988589360 0.9989541675 0.9990395697 0.9991164266
0.9991858197 0.9992486651 0.9993057428 0.9993577205 0.9994051720 0.9994485928 0.9994884125 0.9995250049 0.9995586968 0.9995897744
0.9996184895 0.9996450644 0.9996696956 0.9996925575 0.9997138054 0.9997335777 0.9997519982 0.9997691778 0.9997852163 0.9998002034
0.9998142198 0.9998273389 0.9998396267 0.9998511432 0.9998619429 0.9998720754 0.9998815859 0.9998905154 0.9998989017 0.9999067791
0.9999141790 0.9999211303 0.9999276594 0.9999337906 0.9999395461 0.9999449463 0.9999500100 0.9999547546 0.9999591958 0.9999633483
0.9999672254 0.9999708395 0.9999742019 0.9999773228 0.9999802118 0.9999828775 0.9999853278 0.9999875699 0.9999896101 0.9999914544
0.9999931080 0.9999945754 0.9999958609 0.9999969680 0.9999978997 0.9999986585 0.9999992466 0.9999996655 0.9999999164 1.0000000000
The 3 dots in the above data file are just to tell you that there are many more entries in the file. These dots are not there in the original file.
And program :
file CompoundMPIBSC20200823.f90
open(11,file=fname(mel),status='old',form='formatted')
open(12,file=fname1(mel),status='old',form='formatted')
do men=1,nen !nen=75:energy intervals Energy split number cycle
!!write(iw,*) 'mel,men:',mel,men
read (11,'(i2,I7,d22.15/(10f13.10))') na,nenerg,tcrpc,(rpw(i),i=1,ith)
read (12,'(i2,I7,d22.15/(10f15.10))') na,nenerg,tcrpc,(rpw1(i),i=1,ith)
!!write(iw,*) 'mel,men,na,nenerg:',mel,men,na,nenerg,tcrpc
ftcs(men)=tcrpc ! corresponds to the total elastic scattering cross section at the energy
penergy(men)=nenerg/1000.
rpw(ith)=1.
!---------------------------
line 1939 is
read (11,'(i2,I7,d22.15/(10f13.10))') na,nenerg,tcrpc,(rpw(i),i=1,ith)
I've tried different modifications of the code but didn't get any results.
Any help would be greatly appreciated!
You appear to have spaces padding the fields in your data file, but not in your read format. The first line of your file (with column labels) is
000000000111111111122222222223333
123456789012345678901234567890123
6 10 1.531581196563372e-15
so splitting this into i2,I7,d22.15 gives
i2 | I7 | d22.15 |
00 | 0000000 | 1111111111222222222233 | 33
12 | 3456789 | 0123456789012345678901 | 23
6 | 1 | 0 1.531581196563372e- | 15
which is clearly not as intended.
There are two ways around this problem:
As Ian Bush points out, you can forego the read format entirely, and used list-directed input, as
read (11,*) na,nenerg,tcrpc,(rpw(i),i=1,ith)
This will parse your file token by token rather than relying on column widths, and is usually a much better option for parsing data files.
If you must use a read format, you need to add space padding to it, e.g.
'(i2,X,I7,X,d22.15/10(X,f13.10))', which will then split the input string as
i2 | X | I7 | X | d22.15
00 | 0 | 0000001 | 1 | 1111111122222222223333
12 | 3 | 4567890 | 1 | 2345678901234567890123
6 | | 10 | | 1.531581196563372e-15
How do I fix USER FATAL MESSAGE 740? This error is generated by Nastran when I try to run a BDF/DAT file of mine.
*** USER FATAL MESSAGE 740 (RDASGN)
UNIT NUMBER 5 HAS ALREADY BEEN ASSIGNED TO THE LOGICAL NAME INPUT
USER ACTION: CHANGE THE UNIT NUMBER ON THE ASSIGN STATEMENT AND IF THE UNIT IS USED FOR
PARAM,POST,<0 THEN SPECIFY PARAM,OUNIT2 WITH THE NEW UNIT NUMBER.
AVOID USING THE FOLLOWING UNIT NUMBERS THAT ARE ASSIGNED TO SPECIAL FILES IN MSC.NASTRAN:
1 THRU 12, 14 THRU 22, 40, 50, 51, 91, 92. SEE THE MSC.NASTRAN INSTALLATIONS/OPERATIONS
GUIDE SECTION ON MAKING FILE ASSIGNMENTS OR MSC.NASTRAN QUICK REFERENCE GUIDE ON
ASSIGN PHYSICAL FILE FOR REFERENCE.
Below is the head of my BDF file.
assign userfile='SUB1_PLATE.csv', status=UNKNOWN, form=formatted, unit=52
SOL 200
CEND
ECHO = NONE
DESOBJ(MIN) = 35
set 30=1008,1007,1015,1016
DESMOD=SUB1_PLATE
SUBCASE 1
$! Subcase name : DefaultLoadCase
$LBCSET SUBCASE1 DefaultLbcSet
ANALYSIS = STATICS
SPC = 1
LOAD = 6
DESSUB = 99
DISPLACEMENT(SORT1,PLOT,REAL)=ALL
STRESS(SORT1,PLOT,VONMISES,CORNER)=ALL
BEGIN BULK
param,xyunit,52
[...]
ENDDATA
Below is the solution
Correct
assign userfile='SUB1_PLAT.csv', status=UNKNOWN, form=formatted, unit=52
I shortened the name of CSV file to SUB1_PLAT.csv. This reduced the length of the line to 72 characters.
Incorrect
assign userfile='SUB1_PLATE.csv', status=UNKNOWN, form=formatted, unit=52
The file management section is limited to 72 characters, spaces included. The incorrect line stretches 73 characters. The nastran reader ignores the 73rd character and on. Instead of reading "unit=52" the reader reads "unit=5" which triggers the error.
|<--------------------- 72 Characters -------------------------------->||<- Characters are ignored truncated ->
assign userfile='SUB1_PLATE.csv', status=UNKNOWN, form=formatted, unit=52
References
MSC Nastran Reference Guide
The records of the first four sections are input in free-field format
and only columns 1 through 72 are used for data. Any information in
columns 73 through 80 may appear in the printed echo, but will not be
used by the program. If the last character in a record is a comma,
then the record is continued to the next record.
I am trying to write Chinese characters to a CSV file based on their Unicode code points found in a text file in unicode.org/Public/zipped/13.0.0/Unihan.zip. For instance, one example character is U+9109.
In the example below I can get the correct output by hard coding the value (line 8), but keep getting it wrong with every permutation I've tried at generating the bytes from the code point (lines 14-16).
I'm running this in Python 3.8.3 on a Debian-based Linux distro.
Minimal working (broken) example:
1 #!/usr/bin/env python3
2
3 def main():
4
5 output = open("test.csv", "wb")
6
7 # Hardcoded values work just fine
8 output.write('\u9109'.encode("utf-8"))
9
10 # Comma separation
11 output.write(','.encode("utf-8"))
12
13 # Problem is here
14 codepoint = '9109'
15 u_str = '\\' + 'u' + codepoint
16 output.write(u_str.encode("utf-8"))
17
18 # End with newline
19 output.write('\n'.encode("utf-8"))
20
21 output.close()
22
23 if __name__ == "__main__":
24 main()
Executing and viewing results:
example $
example $./test.py
example $
example $cat test.csv
鄉,\u9109
example $
The expected output would look like this (Chinese character occurring on both sides of the comma):
example $
example $./test.py
example $cat test.csv
鄉,鄉
example $
chr is used to convert integers to code points in Python 3. Your code could use:
output.write(chr(0x9109).encode("utf-8"))
But if you specify the encoding in the open instead of using binary mode you don't have to manually encode everything. print to a file handles newlines for you as well.
with open("test.txt",'w',encoding='utf-8') as output:
for i in range(0x4e00,0x4e10):
print(f'U+{i:04X} {chr(i)}',file=output)
Output:
U+4E00 一
U+4E01 丁
U+4E02 丂
U+4E03 七
U+4E04 丄
U+4E05 丅
U+4E06 丆
U+4E07 万
U+4E08 丈
U+4E09 三
U+4E0A 上
U+4E0B 下
U+4E0C 丌
U+4E0D 不
U+4E0E 与
U+4E0F 丏
cpxSR2bnPUihaNxIFFA8Sc+8gUnWuJxJi8ywSW5ju0npWrFJHW2MSZAeMklcZ71IjrBySF2ci0gdecRI0vD/SM4ZF0m1ZSJJBY8bSZJl/0intaxIlQJBSPdY3EdBLM9Hp4wLSOK8Nki8L1pIoglxSAvNbkjHg0VIDlv7R6B2Y0elCqVGFWuVRgagAkdxHTdHELxRR9i2VkdyEUlHU84kRzTS2kalKFxG
This is a string from an XML file from my mass spectrometer. I am trying to write a program to load two such files, subtract one set of values from another, and write the results to a new file. According to the specification file for the .mzML format, the encoding of the numerical data is alleged to be base64. I can't convert this data string to anything legible using any of the many online base64 converter or using NotepaD++ and the MIME toolkit's base64 converter.
The string, in the context of the results file, looks like this:
<binaryDataArray encodedLength="224">
<cvParam cvRef="MS" accession="MS:1000515" name="intensity array" unitAccession="MS:1000131" unitName="number of counts" unitCvRef="MS"/>
<cvParam cvRef="MS" accession="MS:1000521" name="32-bit float" />
<cvParam cvRef="MS" accession="MS:1000576" name="no compression" />
<binary>cpxSR2bnPUihaNxIFFA8Sc+8gUnWuJxJi8ywSW5ju0npWrFJHW2MSZAeMklcZ71IjrBySF2ci0gdecRI0vD/SM4ZF0m1ZSJJBY8bSZJl/0intaxIlQJBSPdY3EdBLM9Hp4wLSOK8Nki8L1pIoglxSAvNbkjHg0VIDlv7R6B2Y0elCqVGFWuVRgagAkdxHTdHELxRR9i2VkdyEUlHU84kRzTS2kalKFxG</binary>
I can't proceed until I can work out what format this encoding is meant to be!
Thanks in advance for any replies.
You can use this trivial program to convert it to plaintext:
#include <stdio.h>
int main(void)
{
float f;
while (fread(&f, 1, 4, stdin) == 4)
printf("%f\n", f);
}
I compiled this to "floatdecode" and used this command:
echo "cpxSR2bnPUihaNxIFFA8Sc+8gUnWuJxJi8ywSW5ju0npWrFJHW2MSZAeMklcZ71IjrBySF2ci0gdecRI0vD/SM4ZF0m1ZSJJBY8bSZJl/0intaxIlQJBSPdY3EdBLM9Hp4wLSOK8Nki8L1pIoglxSAvNbkjHg0VIDlv7R6B2Y0elCqVGFWuVRgagAkdxHTdHELxRR9i2VkdyEUlHU84kRzTS2kalKFxG" | base64 -d | ./floatdecode
Output is:
53916.445312
194461.593750
451397.031250
771329.250000
1062809.875000
1283866.750000
1448337.375000
1535085.750000
1452893.125000
1150371.625000
729577.000000
387898.875000
248514.218750
285922.906250
402376.906250
524166.562500
618908.875000
665179.312500
637168.312500
523052.562500
353709.218750
197642.328125
112817.929688
106072.507812
142898.609375
187123.531250
223422.937500
246822.531250
244532.171875
202255.109375
128694.109375
58230.625000
21125.322266
19125.541016
33440.023438
46877.441406
53692.062500
54966.843750
51473.445312
42190.324219
28009.101562
14090.161133
Yet another Java Base64 decode with options to uncompress should you need it.
Vendor spec indicated "32-bit float" = IEEE-754 and specified little-endian.
Schmidt's converter shows the bit pattern for IEEE-754.
One more Notepad++ step to look at the hex codes:
Notepad++ TextFX plugin (after the Base64 decode you already did)
select the text
TextFX > TextFX Convert > Convert text to Hex-32
lets you look at the hex codes:
"000000000 72 9C 52 47 66 E7 3D 48- ... 6E 63 BB 49 |rœRGfç=H¡hÜHP
Little-endian: 47529C72 converts (via Schmidt) as shown above by David.
You can access such data from mzML files in Python through pymzML, a python interface to mzML files.
http://pymzml.github.com/
I made a matrix and I want to export it to Excel. The matrix looks like this:
1 2 3 4 5 6 7
2 0.4069264
3 0.5142857 0.2948718
4 0.3939394 0.4098639 0.3772894
5 0.3476190 0.3717949 0.3194444 0.5824176
6 0.2809524 0.3974359 0.2222222 0.3388278 0.3974359
7 0.2809524 0.5987654 0.3933333 0.4188713 0.4711538 0.3429487
8 0.4675325 0.4855072 0.4523810 0.4917184 0.3409091 0.4318182 0.4128788
9 0.3896104 0.5189594 0.4404762 0.2667549 0.5471429 0.3604762 0.3081502
10 0.4242424 0.4068878 0.3484432 0.2708333 0.4766484 0.3740842 0.4528219
11 0.3476190 0.3942308 0.2881944 0.3228022 0.4711538 0.2147436 0.3653846
12 0.6060606 0.3949830 0.2971612 0.3541667 0.5022894 0.3484432 0.4466490
13 0.4675325 0.5972222 0.6060606 0.3670635 0.4393939 0.3939394 0.3695652
14 0.4978355 0.4951499 0.4480952 0.4713404 0.3814286 0.3147619 0.4629121
15 0.4632035 0.4033883 0.4508929 0.3081502 0.4728571 0.3528571 0.4828571
16 0.3766234 0.5173993 0.4771825 0.4734432 0.5114286 0.3514286 0.4214286
17 0.3939394 0.5289116 0.3260073 0.3333333 0.5663919 0.2330586 0.3015873
18 0.3939394 0.3708791 0.2837302 0.4102564 0.3392857 0.2559524 0.4123810
19 0.3160173 0.5727041 0.4885531 0.3056973 0.4725275 0.3827839 0.3346561
20 0.3333333 0.5793651 0.4257143 0.4876543 0.4390476 0.2390476 0.3131868
21 0.5281385 0.3762755 0.4052198 0.2997449 0.4180403 0.2898352 0.4951499
22 0.3593074 0.3784014 0.4075092 0.2423469 0.4908425 0.3113553 0.3430335
23 0.5281385 0.5875850 0.4404762 0.4634354 0.6071429 0.3763736 0.3747795
24 0.3549784 0.6252381 0.5957341 0.4328571 0.4429563 0.4429563 0.3422619
25 0.4242424 0.4931973 0.5054945 0.2142857 0.4670330 0.4285714 0.4312169
26 0.3852814 0.5671769 0.4954212 0.4073129 0.3736264 0.4890110 0.4523810
27 0.5238095 0.3269558 0.5187729 0.4051871 0.5412088 0.5155678 0.5859788
28 0.3160173 0.1904762 0.3205128 0.3384354 0.3429487 0.3173077 0.5123457
29 0.2380952 0.4468537 0.5196886 0.4536565 0.4491758 0.4491758 0.4634039
30 0.4545455 0.4295635 0.4080087 0.4791667 0.3474026 0.3019481 0.4627329
31 0.2857143 0.3988095 0.3397436 0.3443878 0.4294872 0.2756410 0.3456790
32 0.3636364 0.3027211 0.3772894 0.3452381 0.4413919 0.3388278 0.3818342
33 0.3333333 0.4482402 0.4080087 0.4275362 0.2888199 0.4047619 0.4301242
34 0.5411255 0.4825680 0.4043040 0.4417517 0.4748168 0.3850733 0.3708113
35 0.3160173 0.5476190 0.4230769 0.3979592 0.3653846 0.3397436 0.2283951
36 0.4603175 0.4653209 0.4778912 0.5170807 0.3928571 0.4508282 0.4254658
37 0.3939394 0.1955782 0.2490842 0.4047619 0.2490842 0.3516484 0.4559083
38 0.3463203 0.4660494 0.4300000 0.4157848 0.3833333 0.2233333 0.2788462
39 0.5844156 0.4668367 0.3809524 0.3843537 0.4803114 0.3008242 0.5026455
40 0.5454545 0.4902211 0.3740842 0.2946429 0.5279304 0.2971612 0.3293651
41 0.5800866 0.3758503 0.5073260 0.5136054 0.3598901 0.5393773 0.4823633
42 0.4458874 0.3937390 0.3785714 0.4686949 0.3768315 0.3127289 0.4954212
43 0.6536797 0.5740741 0.5533333 0.4453263 0.4866667 0.5400000 0.4358974
44 0.5887446 0.5548469 0.4308608 0.3949830 0.5462454 0.3411172 0.5136684
45 0.4069264 0.4357993 0.4308608 0.3830782 0.4308608 0.3795788 0.4025573
46 0.5974026 0.3826531 0.3672161 0.3954082 0.4441392 0.3159341 0.5141093
47 0.2554113 0.4196429 0.4262821 0.4961735 0.2788462 0.3301282 0.3055556
I tried the command:
WriteXLS("my matrix after i converted it to data.frame", "test.xls")
but I got this error:
The Perl script 'WriteXLS.pl' failed to run successfully.
I googled it but I couldn't find a solution.
Thanks in advance.
Any reason why you can't just use write.csv?
write.csv(mymatrix, "test.csv")
Import it in Excel and you're set!
PS: I assume you're not putting quotes around your variable name in the WriteXLS call, right?
One other option on Windows (which seems a reasonable assumption given that you are using Excel):
You can write a matrix (or data frame) to the clipboard using a command like:
write.table(mymat, 'clipboard', sep='\t')
Then just go into Excel, click in the cell that you want to be the top left cell, then do a paste and your matrix is there (the sep='\t' is important for Excel to interpret it correctly).
This is similar to other answers, but you don't need an intermediate file on disk.
You could also check xlsx if you do not mind the Excel 2007 format, as xlsx does not depend on Perl (though depends on rJava).
After loading the packge via library(xlsx) just try the following:
write.xlsx(USArrests, "/usarrests.xlsx")
It's hard to see what is going on here exactly. Might be several things.
I think the easiest way to write a matrix to excell is by using write.table() and importing the data in excell. It takes an extra step but it also keeps your data in a nice format.
If foo is your matrix:
write.table(foo,"foo.txt")
If you get an error maybe trie coercing the object to a matrix:
write.table(as.matrix(foo),"foo.txt")
Does the matrix contain values in the upper triangle as well? Perhaps making a full matrix works:
foo<-foo+t(foo)
write.table(as.matrix(foo),"foo.txt")
But these are all just random shots in the dark since I don't have a matrix to work with.
EDIT: In response to the other answer, you can remove the column and rownames with col.names=FALSE and row.names=FALSE in both write.table() and write.csv() (which are the same function with different default values).
I have met the same problem, after reinstalling strawberry perl : after debugging the WriteXLS function in R, I found out the the perl module Text::CSV_XS was missing from my fresh new install. I installed this module from the DOS command line :
perl -MCPAN -e shell
install Text::CSV_XS
After this, WriteXLS was working fine.
upper # matrix name
write.xlsx2(upper,file = "File.xlsx", sheetName="Sheetname",col.names=TRUE, row.names=TRUE, append=TRUE, showNA=TRUE)