I have the following CSV file, the first line is the header:
admission_no;first_name;last_name;gender;date_of_birth;join_date;form_id;stream_id;school_photo
1003;"cavin";"cavin";"cavin";"male";1992-11-02;2007-01-25;1;1
1004;"joshua";"joshua";"joshua";"male";1992-11-03;2007-01-26;1;1
1005;"elijah";"elijah";"elijah";"male";1992-11-04;2007-01-27;1;1
1006;"lawrent";"lawrent";"lawrent";"male";1992-11-05;2007-01-28;1;1
1007;"steven";"steven";"steven";"male";1992-11-06;2007-01-29;1;2
1008;"javan";"javan";"javan";"male";1992-11-07;2007-01-30;1;2
1009;"miller";"miller";"miller";"male";1992-11-08;2007-01-31;1;2
1010;"javis";"javis";"javis";"male";1992-11-09;2007-02-01;1;2
1011;"fredrick";"fredrick";"fredrick";"male";1992-11-10;2007-02-02;1;3
1012;"fredrick";"fredrick";"fredrick";"male";1992-11-11;2007-02-03;1;3
1013;"nahashon";"nahashon";"nahashon";"male";1992-11-12;2007-02-04;1;3
1014;"nelson";"nelson";"nelson";"male";1992-11-13;2007-02-05;1;3
1015;"martin";"martin";"martin";"male";1992-11-14;2007-02-06;1;4
1016;"felix";"felix";"fwlix";"male";1992-11-15;2007-02-07;1;4
1017;"tobias";"tobias";"tobias";"male";1992-11-16;2007-02-08;1;4
1018;"dennis";"dennis";"dennis";"male";1992-11-17;2007-02-09;1;4
1019;"bildad";"bildad";"bildad";"male";1992-11-18;2007-02-10;1;5
1020;"syslvester";"sylvester";"sylvester";"male";1992-11-19;2007-02-11;1;5
And my database table columns are: admission_no, first_name, last_name, gender, date_of_birth, join_date, form_id, stream_id and school_photo.
Using the CakeDC Utils plugin to import the data, I get a flash message:
Successfully imported 0 records from file.csv
I have tried removing the header, changing the delimiter or even adding NULL for the school_photo column since it is nullable but nothing seems to work.
Can someone tell me what am doing wrong?
am generating the csv using Ubuntu libre Office
the import function:
function import() {
$modelClass = $this->modelClass;
if( $this->request->is('POST') ) {
$records_count = $this->$modelClass->find('count');
try {
$this->$modelClass->importCSV($this->request->data[$modelClass]['CsvFile']['tmp_name']);
}catch (Exception $e) {
$import_errors = $this->$modelClass->getImportErrors();
$this->set('import_errors', $import_errors);
$this->Session->setFlash(__('Error importing')." ".$this->request->data[$modelClass]['CsvFile']['name']. ",". __('column mismatch'));
$this->redirect(array('action' => 'import'));
}
$new_records_count = $this->$modelClass->find('count') - $records_count;
$this->Session->setFlash(__('Successfully imported')." ". $new_records_count . " ".'records from' . " ".$this->request->data[$modelClass]['CsvFile']['name']);
//$this->redirect(array('action' => 'index'));
}
$this->set('modelClass', $modelClass);
$this->render('../Common/import');
}//end import
the view file
<h3>Import <?php echo Inflector::pluralize($modelClass); ?> from CSV data file</h3>
<?php
echo $this->Form->create($modelClass, array('action' => 'import', 'type' => 'file'));
echo $this->Form->input('CsvFile', array('label' => '', 'type' => 'file'));
echo $this->Form->end('Submit');
?>
Make sure your line endings are something your server can read. I've been burned by this a million times... especially if using a mac to generate the CSV from Excel. save as "csv for windows".
Background: OS X tends to save files with \r (carriage return) line endings, while Windows uses \r\n (carriage return, line feed). To make it worse all other Unix-like systems (e.g. Linux) tend to use only \n.
Thus if you save a file on OS X and the open it on Windows, Windows thinks the file is only one big line.
You can verify this with a good editor (a programming editor which can display whitespace as symbols) or by looking at the actual hex Code of the file.
Related
I have a csv file (UTF-8 with BOM) like this
NAME,F1,F2,
test1,field1,field2
test2,field1,field2
test3,field1,field2
test4,field1,field2
test5,field1,field2
test6,field1,field2
I would like to discard the first three lines and create new csv (UTF-8 with BOM)
NAME,F1,F2,
test4,field1,field2
test5,field1,field2
test6,field1,field2
I get some idea from the page and code this in JSR223 PreProcessor
def originalCsvFile = new File('g:/Workspace/1.csv')
def newCsvFile = new File('g:/Workspace/2.csv')
originalCsvFile.readLines().take(5).each {line ->
newCsvFile.withWriter('UTF-8') { writer ->
writer.writeLine line
}
}
The above code does not work.
It is better to put the new csv path to the variable, I want to get the variable in jmeter CSV Data Set Config
Do you realize that:
take(5) function returns 5 first lines of the list
newCsvFile.withWriter function overwrites the file with the new data each time it's being called
So I believe you're looking for copying and pasting something like this:
originalCsvFile.readLines().eachWithIndex { line, index ->
if (index == 0 || index > 3) {
newCsvFile.withWriterAppend('UTF-8') { writer ->
writer.writeLine line
}
}
}
More information on Groovy scripting in JMeter: Apache Groovy - Why and How You Should Use It
Not as elegant, perhaps, but this is how I would do it:
List li = originalCsvFile.readLines()
newCsvFile.append(li[0] + "\n", 'UTF-8') //headers
li[4..-1].each { newCsvFile.append(it + "\n", 'UTF-8') }
I have an application that generates a txt file with thousands of lines. I have to delete some lines manually by going through the file (using vim). However, I might need to generate the same file again if a change in format is required. That will make me go through the file again to delete the same lines.
The solution to avoid deleting manually repeatedly is that vim somehow logs the line number when I delete a line. I can then use some script to remove those lines. Is it possible to get this behavior in vim?
Otherwise, is there any other editor to get this behavior? There are many lines I have to delete and it's not feasible for me to log each line number manually.
As suggested by phd and wxz, I was able to use git-diff of the file to extract the deleted lines by using node package gitdiff-parser for parsing the diff.
const gitDiffParser = require('gitdiff-parser')
const { exec } = require("child_process");
let p = new Promise( (res,rej) => {
exec("git diff -U0 file.txt", (error, stdout) => {
res(stdout)
});
});
p.then(s=>{
diff = gitDiffParser.parse(s);
diff[0].hunks.forEach(element => {
console.log(`start: ${element.oldStart}, end: ${element.oldStart + element.oldLines - 1}`)
});
})
Another solution or say hack was to append line number in each line of the file and extract the undeleted line numbers after removing the required lines.
I'm trying to find and replace some special chars in a file encoded in ISO-8859-1, then write the result to a new file encoded in UTF-8:
package inv
class MigrationScript {
static main(args) {
new MigrationScript().doStuff();
}
void doStuff() {
def dumpfile = "path to input file";
def newfileP = "path to output file"
def file = new File(dumpfile)
def newfile = new File(newfileP)
def x = [
"þ":"ş",
"ý":"ı",
"Þ":"Ş",
"ð":"ğ",
"Ý":"İ",
"Ð":"Ğ"
]
def r = file.newReader("ISO-8859-1")
def w = newfile.newWriter("UTF-8")
r.eachLine{
line ->
x.each {
key, value ->
if(line.find(key)) println "found a special char!"
line = line.replaceAll(key, value);
}
w << line + System.lineSeparator();
}
w.close()
}
}
My input file content is:
"þ": "ý": "Þ":" "ð":" "Ý":" "Ð":"
Problem is my code never finds the specified characters. The groovy script file itself is encoded in UTF-8. I'm guessing that may be the cause of the problem, but then I can't encode it in ISO-8859-1 because then I can't write "Ş" "Ğ" etc in it.
I took your code sample, run it with an input file encoded with charset ISO-8859-1 and it worked as expected. Can you double check if your input file is actually encoded with ISO-8859-1? Here is what I did:
I took file content from your question and saved it (using SublimeText) to a file /tmp/test.txt using Save -> Save with Encoding -> Western (ISO 8859-1)
I checked file encoding with following Linux command:
file -i /tmp/test.txt
/tmp/test.txt: text/plain; charset=iso-8859-1
I set up dumpfile variable with /tmp/test.txt file and newfile variable to /tmp/test_2.txt
I run your code and I saw in the console:
found a special char!
found a special char!
found a special char!
found a special char!
found a special char!
found a special char!
I checked encoding of the Groovy file in IntelliJ IDEA - it was UTF-8
I checked encoding of the output file:
file -i /tmp/test_2.txt
/tmp/test_2.txt: text/plain; charset=utf-8
I checked the content of the output file:
cat /tmp/test_2.txt
"ş": "ı": "Ş":" "ğ":" "İ":" "Ğ":"
I don't think it matters, but I have used the most recent Groovy 2.4.13
I'm guessing that your input file is not encoded properly. Do double check what is the encoding of the file - when I save the same content but with UTF-8 encoding, your program does not work as expected and I don't see any found a special char! entry in the console. When I display contents of ISO-8859-1 file I see something like that:
cat /tmp/test.txt
"�": "�": "�":" "�":" "�":" "�":"%
If I save the same content with UTF-8, I see the readable content of the file:
cat /tmp/test.txt
"þ": "ý": "Þ":" "ð":" "Ý":" "Ð":"%
Hope it helps in finding source of the problem.
My first question is:
Is this possible to do this, since now I have a perl script which reads Html file and extract data to display on another html file.
If the answer for the question above is Yes, my second question would be:
How to do this?
Sorry to ask frankly as this, but since I'm so new for perl, and I have to take this task, so I'm here for some useful advice or suggestion to guide me through this task. Appreciate your help in advance.
Here's a part of the code, since the whole chunk is quite long:
$date=localtime();
($TWDAY, $TMTH, $TD1D, $TSE, $TYY) = split(/\s+/, $date);
$TSE =~ s/\://g;
$STAMP=_."$TD1D$TMTH$TYY";
#ServerInfo=();
#--------------------------------------------------------------------------- -------------------------------
# Read Directory
#----------------------------------------------------------------------------------------------------------
$myDir=getcwd;
#----------------------------------------------------------------------------------------------------------
# INITIALIZE HTML FORMAT
#----------------------------------------------------------------------------------------------------------
&HTML_FORMAT;
#----------------------------------------------------------------------------------------------------------
# REPORT
#----------------------------------------------------------------------------------------------------------
if (! -d "$myDir/report") { mkdir("$myDir/report");};
$REPORTFILE="$myDir/report/checkpack".".htm";
open OUT,">$REPORTFILE" or die "\nCannot open out file $REPORTFILE\n\n";
print OUT "$Tag_Header";
#----------------------------------------------------------------------------------------------------------
sub numSort {
if ($b < $a) { return -1; }
elsif ($a == $b) { return 0;}
elsif ($b > $a) { return 1; }
}
#ArrayDir = sort numSort #DirArray;
#while (<#ArrayDir>) {
#OutputDir=grep { -f and -T } glob "$myDir/*.htm $myDir/*.html";
#}
#----------------------------------------------------------------------------------------------------------
#ReadLine3=();
$xyxycnt=0;
foreach $InputFile (#OutputDir) { #---- MAIN
$filename=(split /\//, $InputFile) [-1]; print "-"x80 ; print "\nFilename\t:$filename\n";
open IN, "<$InputFile" or die "Cannot open Input file $InputFile\n";
#MyData=();
$DataCnt=0;
#MyLine=();
$MyLineCnt=0;
while (<IN>) {
$LINE=$_;
chomp($LINE);
$LINE=~s/\<br\>/XYXY/ig;
$LINE=~s/\<\/td\>/ \nXYZXYZ\n/ig;
$LINE=~s/\<dirname\>/xxxdirnameyyy/ig;
$LINE=linetrim3($LINE);
$LINE=linetrim($LINE);
$LINE=~s/XYXY/\<br\>/ig;
$LINE=~s/xxxdirnameyyy/< dirname >/ig;
$LINE=~s/^\s+//ig;
print OUT2 "$LINE\n";
if (defined($LINE)) { $MyData[$DataCnt]="$LINE"; $DataCnt++ ; }
}
close IN;
foreach $ReadFile (#MyData) { #--- Mydata
$MyLineCnt++;
$MyLine[$MyLineCnt]="";
#### FILENAME
$ServerInfo[0]="$filename";
#### IP ADDRESS
if ($ReadFile =~ /Host\/Device Name\:/) {
#print "$ReadFile\n"
($Hostname)=(split /\:|\s+/, $ReadFile)[3]; print "$Hostname\n";
&myServerInfo("$Hostname","1");
}
if ($ReadFile =~ /IP Address\(es\)/) {#ListIP=(); $SwIP=1; $CntIP=0 ; };
#### OPERATING SYSTEM & VERSION
if ($ReadFile =~ /Operating System\:/) {
$SwIP=0;
$OS= (split /\:|\s+/, $ReadFile)[3]; &myServerInfo("$OS","3") ; print "$OS\n";
$OSVer= (split /\:|\s+/, $ReadFile)[-2]; &myServerInfo("$OSVer","4") ; print "$OSVer\n";
};
#### GET IP VALUE
if ($SwIP==1) {
$ReadFile=(split /\:/,$ReadFile) [2];
$ReadFile=~s/[a-z|A-Z]|\(|\)|\// /ig; print "$ReadFile\n";
if ($CntIP==0) {
#$ListIP[$CntIP]=(split /\s+/,$ReadFile) [1];
#ListIP="$ReadFile";
} elsif ($CntIP==1) { print "\n\t\t $ReadFile\n" ; $ListIP[$CntIP]="\n$ReadFile";
} else { print "\t\t $ReadFile\n" ; $ListIP[$CntIP]="\n$ReadFile"; };
$CntIP++;
}
I'm afraid if you don't understand what is going on in this program and you also don't understand how to approach a task like this at all, Stack Overflow might not be the right place to get help.
Let me try to show you the approach I would take with this. I'm assuming there is more code.
First, write down a list of everything you know:
What is the input format of the existing file
Where does the existing file come from now
What is the output format of the existing file
Where does the generated output file go afterwards
What does the new file look like
Where does the new file come from
Use perltidy to indent the inherited code so you can read it better. The default options should be enough.
Read the code, take notes about what pieces do what, add comments
Write a unit test for the desired output format. You can use Test::More. Another useful testing module here is Test::File.
Refactor the part that generated the output format to work with a certain data structure. Use your tests to make sure you don't break it.
Write code to parse the new file into the data structure from the point above. Now you can plug that in and get the expected output.
Refactor the part that takes the old input file from the existing file location to be a function, so you can later switch it for the new one.
Write code to get the new file from the new file location.
Document what you did so the next guy is not in the same situation. Remember that could be you in half a year.
Also add use strict and use warnings while you refactor to catch errors more easily. If stuff breaks because of that, make it work before you continue. Those pragmas tell you what's wrong. The most common one you will encounter is Global symbol "$foo" requires explicit package name. That means you need to put my in front of the first assignment, or declare the variable before.
If you have specific questions, ask them as a new question with a short example. Read how to ask to make sure you will get help on those.
Good luck!
After seing your comment I am thinking you want a different input and a different output. In that case, disregard this, throw away the old code and start from scratch. If you don't know enough Perl, get a book like Curtis Poe's Beginning Perl if you already know programming. If not, check out Learning Perl by Randal L. Schwartz.
I am working with ANSI 835 plain text files and am looking to capture all data in segments which start with “BPR” and end with “TRN” including those markers. A given file is a single line; within that line the segment can, but not always, repeats. I am running the process on multiple files at a time and ideally I would be able to record the file name in which the segment(s) occur.
Here is what I have so far, based on an answer to another question:
#!/bin/sed -nf
/BPR.*TRN/ {
s/.*\(BPR.*TRN\).*/\1/p
d
}
/from/ {
: next
N
/BPR/ {
s/^[^\n]*\(BPR.*TRN\)[^n]*/\1/p
d
}
$! b next
}
I run all files I have through this and write the results to a file which looks like this:
BPR*I*393.46*C*ACH*CCP*01*011900445*DA*0000009046*1066033492**01*071923909*DA*72
34692932*20150120~TRN
BPR*I*1611.07*C*ACH*CCP*01*031100209*DA*0000009108*1066033492**01*071923909*DA*7
234692932*20150122~TRN
BPR*I*1415.25*C*CHK************20150108~TRN
BPR*H*0*C*NON************20150113~TRN
BPR*I*127.13*C*CHK************20150114~TRN
BPR*I*22431.28*C*ACH*CCP*01*071000152*DA*99643*1361236610**01*071923909*DA*72346
92932*20150112~TRN
BPR*I*182.62*C*ACH*CCP*01*071000152*DA*99643*1361236610**01*071923909*DA*7234692
932*20150115~TRN
Ideally each line would be prepended with the file name like this:
IDI.Aetna.011415.64539531.rmt:BPR*I*393.46*C*ACH*CCP*01*011900445*DA*0000009046*1066033492**01*071923909*DA*72
34692932*20150120~TRN
IDI.BCBSIL.010915.6434438.rmt:BPR*I*1611.07*C*ACH*CCP*01*031100209*DA*0000009108*1066033492**01*071923909*DA*7
234692932*20150122~TRN
IDI.CIGNA.010215.64058847.rmt:BPR*I*1415.25*C*CHK************20150108~TRN
IDI.GLDRULE.011715.646719.rmt:BPR*H*0*C*NON************20150113~TRN
IDI.MCREIN.011915.6471442.rmt:BPR*I*127.13*C*CHK************20150114~TRN
IDI.UHC.011915.64714417.rmt:BPR*I*22431.28*C*ACH*CCP*01*071000152*DA*99643*1361236610**01*071923909*DA*72346
92932*20150112~TRN
IDI.UHC.011915.64714417.rmt:BPR*I*182.62*C*ACH*CCP*01*071000152*DA*99643*1361236610**01*071923909*DA*7234692
932*20150115~TRN
The last two lines would be an example of a file where the segment pattern repeats.
Again, prepending each line with the file name is ideal. What I really need is to be able to process a given single-line file which has the “BPR…TRN” segment repeating and write all segments in that file to my output file.
Try with awk:
awk '
/BPR/ { sub(".*BPR","BPR") }
/TRN/ { sub("TRN.*","TRN") }
/BPR/,/TRN/ { print FILENAME ":" $0 }
' *.rmt