I have got a series of geopackages which represent every single packet of land across England. Each geopackage is one local area and i want to combine them to create one single file for all of england.
Each geopackage is between 50 and 1500 MB and there are ~300 of them
I would like to combine the to create one single MSSQL Spatial
I think that ogr2ogr is a best bet for this, but I am having a little trouble
I have tried this code below as a .bat, which i confess i'm not that familiar with and has been copied from a previous colleague who is no longer available
set "MSSQLSPATIAL_USE_GEOMETRY_COLUMNS=NO"
ogr2ogr -f "MSSQLSpatial" "MSSQL:server=x;database=y;trusted_connection=yes" "File1.gpkg" -update -append -nln "SQLTableName" -gt 1000 -t_srs "EPSG:27700" -s_srs "EPSG:27700"
for %%a in ("Folder\*.gpkg") Do (
ogr2ogr -f "MSSQLSpatial" "MSSQL:server=x;database=y;trusted_connection=yes" "%%a" -update -append -nln "SQLTableName" -gt 100000 -t_srs "EPSG:27700" -s_srs "EPSG:27700"
)
But this creates the table in SQL but doesnt fill it with any data
The gpkg are all fine as I can open them in GIS and see the content is all there
Related
I created a script to run a function that takes a specific file and compares it to several others. The files that I have to scan for the matches are more than 900 and incrementally named like this:
A001-abc.txt A001-efg.txt [..] A002-efg.txt A002-hjk.txt [..] A120-xwz.txt (no whitespaces)
The script looks like this:
#!/bin/bash
for f in *.txt; do
bedtools function /mydir/myfile.txt *.txt> output.txt
done
Now, the computation is extremely intensive and can't be completed. How can I group the files according to their incremental name and perform the computation on files A001-abc A001-def then go to A002-abc, A002-def.., to A123-xyz and so on?
I was thinking that by finding a way to specify the name with an incremental variable and then grouping them, I could divide the computation in many smaller ones. Is it reasonable and doable?
Many thanks!
The problem might be that you're not using $f. Try using this line in the for-loop instead. Also, use >> so the output appends instead of overwriting the output file.
bedtools function /mydir/myfile.txt "$f" >> output.txt
But if you still want to break it up, you might try a double loop like this:
for n in $(seq --format="%03.f" 1 120); do
for f in "A$n"*; do
bedtools function /mydir/myfile.txt "$f" >> "${n}output.txt"
done
done
The values of $n from the output of the seq command is 001 002 .. 120.
So the inner loop loops through A001* A002* .. A120*.
I created two arrays in shell script
I create an array in shell script
arr1=(manoj 724409 suharsha 298427 koti 702984 sudhakar 283789 chaitanya 769235)
and I want that array elements in excel sheet as follows
name arr1
manoj 724409
suharsha 298427
koti 702984
sudhakar 283789
chaitanya 769235
I tried only for single array(arr1) that also not worked
here is the code that I tried
str=`echo -e 'name\t arr1\t arr2\n'`
i=0
len=${#arr1[#]}
while [ $i -lt $len ]
do
str="${str}${arr1[$i]}"
str=`echo -e $str'\t'`
if [[ ${arr1[$i]} =~ [0-9]{1} ]]; then
str=`echo -e $str'\n'`
fi
i=`expr $i + 1`
done
awk '{print $str}' > output.xls
I modified my question. I don't need to store both array's information. Just one is enough.
The reason your code does not produce a viable result is because the .xls format is not a plaintext format. So let's back up:
Are you attempting to write an excel sheet, or an excel compatible format?
If you can use an excel compatible format, excel (and libre-office) are capable of reading .csv files.
If you need the .xls/xlsx format, you will need some sort of library / application to do the writing.
Additionally are you using a windows machine, or a *nix machine?
If on a *nix machine (as I am assuming, since you are asking about shellscript), then your only choice will be some sort of library or a .csv file.
If on windows, then you could use powershell to spawn an Excel COM object. This will effectively let you control an active excel application in the background and create a sheet with it.
There are additional options of writing excel files if you are willing to leave the shell, but that's not in the scope of your question.
I've been using imagemagick for a few days. My objective is that I want to merge several screenshots into a video and I want to make a nice transition between frames. So I've found the -morph option, for example
convert original/*.jpg -morph 10 new/img%03d.jpg
which will create a total of 12 images, from img001.jpg to img012.jpg.
Is there a way that the images in the output be like, for example, one every X number? for example img00X.jpg img00(x+x).jpg and so on?
If I understand well your requirements, you want to obtain:
img002.jpg img004.jpg img006.jpg ... img024.jpg
instead of:
img001.jpg img002.jpg img003.jpg ... img012.jpg
I'm not sure you can do it directly with imageMagick, however you can go to your new directory and type:
for i in `ls -r`;do mv $i `printf "img%03d.jpg" $(($(expr ${i:3:3} + 0)*2))`;done
(I assume that there are no other files in this directory)
actually it is possible yet without leading zeroes. for example morph every png with 5 transitions steps and increments output counter by 3
im *.png -morph 5 -set filename:a %[fx:t*3] "output-%[filename:a].jpg"
I was wondering if anybody knows a way to achieve this without breaking/mesing with the data itself?
I have a CSV file which is delimited by '|' which was created by retrieving data from Sharepoint using an SPQuery and exported using out-file (because export-csv is not an option since I would have to store the data in a variable and this would eat at the RAM of the server, querying remotely unfortuntely will also not work so i have to do this on the server itself). Nevertheless I have the Data i need but i want to perform some manipulations and move and autocalc certain data within an excel file and export the said excel file.
The problem I have right now is that I sort of need a header to the file. I have tried using the following code:
$header ="Name|GF_GZ|GF_Title|GF_UniqueId|GF_OldId|GFURL|GF_ITEMREDIRECTID"
$file = Import-Csv inputfilename.csv -Header $header | Export-Csv D:\outputfilename.csv
In powershell but the issue here is that when i perform the second Export-Csv it will delimit at anything that has a comma and thus remove it, i sort of need the data to remain intact.
I have tried playing with the -Delimit '|' setting both on the import and the export path but no matter what i do it seems to be cutting off the data. Is there a better way to simply add a row at the Top (a header) without messing with the already existing file structure?
I have found out that using a delimiter such as -delimiter '°' or any other special case character will remove my problem entirely, but i can never be sure if such a character is going to show up in the dataset and thus (as stated already) am looking for a more "elegant" solution.
Thanks
One option you have is to create the original CSV with the headers first. Then when you are exporting the SharePoint data, use the switch -Append in the Out-File command to append the SP data to the CSV.
I wouldn't even bother messing with it in csv format.
$header ="Name|GF_GZ|GF_Title|GF_UniqueId|GF_OldId|GFURL|GF_ITEMREDIRECTID"
$in_file = '.\inputfilename.csv'
$out_file = '.\outputfilename.csv'
$x = Get-Content $in_file
Set-Content $out_file -Value $header,$x
There's probably a more eloquent/refined two-liner for some of this, but this should get you what you need.
Implementing a GIT repository for a project we are including the DB structure by generating a dump on the post-commit hook on deployment.
What I would like to have is a simple versioning system for the file based on the timestamp of the last change to the tables structure.
After finding this post with the suggestion to check for the dates of the the *.frm files in the MySQL data dir I thought the solution would be to implement it based on that last date as part of the generated file. This is:
Find out the latest date-time of the files of the DB (i.e. /var/lib/mysql/databaseX/) via an ls command (of type ls -la *.frm)
compare that value (last changed file) with the one of a certain file (ie /project/dump_2012102620001.sql) where the numbers correspond to the last generated dump.
If files timestamp is after that date generate the mysqldump command, otherwise ignore so the dump does not get generated and
committed as a change to GIT
Unfortunately my Linux console/bash concepts are too far from being capable and have not found any similar script to use.
You can use [[ file1 -ot file2 ]] to test whether file1 is older than file2.
last=$(ls -tr /path/to/db/files/*.frm | tail -n1)
if [[ dump -ot $last ]] ; then
create_new_dump
fi
You can save yourself a lot of grief by just dumping the table structure every time with the appropriate mysqldump command as this is relatively lightweight since it won't include table contents. Strip out the variable timestamp information at the top and compare with the previous file. Store if different.