Im trying to import data into tables from a file using BTEQ import.
im facing weird errors while doing this
Like:
if im using text file as input data file with ',' as delimiter as filed seperator im getting the error as below:
*** Failure 2673 The source parcel length does not match data that was defined.
or
if im using EXCEL file as input data file im getting the error as below:
* Growing Buffer to 53200
* Error: Import data size does not agree with byte length.
The cause may be:
1) IMPORT DATA vs. IMPORT REPORT
2) incorrect incoming data
3) import file has reached end-of-file.
*** Warning: Out of data.
please help me out by giving the syntax for BTEQ import using txt file as input data file and also the syntax if we use EXCEL file as the input data file
Also is there any specific format for the input data file for correct reading of data from it.
if so please give me the info about that.
Thanks in advance:)
EDIT
sorry for not posting the script in first.
Im new to teradata and yet to explore other tools.
I was asked to write the script for BTEQ import
.LOGON TDPD/XXXXXXX,XXXXXX
.import VARTEXT ',' FILE = D:\cc\PDATA.TXT
.QUIET ON
.REPEAT *
USING COL1 (VARCHAR(2)) ,COL2 (VARCHAR(1)) ,COL3 (VARCHAR(56))
INSERT INTO ( COL1 ,COL2 ,COL3) VALUES ( :COL1 ,:COL2 ,:COL3);
.QUIT
I executed the above script and it is successful using a txt(seperating the fileds with comma) file and giving the datatype as varchar.
sample input txt file:
1,b,helloworld1
2,b,helloworld2
3,D,helloworld1
12,b,helloworld1
I also tried to do the same using tab(\t) as the field seperator but it giving the same old error.
Q) Does this work only for comma seperated txt files?
Please could u tell me where can i find the BTEQ manual...
Thanks a lot
Can you post your BTEQ script? May I also ask why you are using BTEQ instead of FastLoad or MultiLoad?
The text file error is possibly due to the data types declared in the using clause. I believe they need to be declared as VARCHAR when reading delimited input (eg. declare as VARCHAR(10) for INTEGER fields).
As for Excel, I can't find anything in the BTEQ manual that says that BTEQ can handle .xls files.
For your tab delimited files, are you doing this (that's a tab character below)?
.import vartext ' '
Or this?
.import vartext '\t'
The former works, the latter doesn't.
The BTEQ manual that I have is on my work machine. One of the first Google results for "BTEQ manual" should yield one online.
Related
I have a .csv file that contains multiple columns with texts in it. These texts contain commas, which makes things messy when I try to read the file into Python.
When I tried:
import pandas as pd
directory = 'some directory'
dataset = pd.read_csv(directory)
I got the following error:
ParserError: Error tokenizing data. C error: Expected 3 fields in line 42, saw 5
After doing some research, I found the clevercsv package.
So, I ran:
import clevercsv as csv
dataset = csv.read_csv(directory)
Running this, I got the error:
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 4359705: character maps to <undefined>
To overcome this, I tried:
dataset = csv.read_csv(directory, encoding="utf8")
However, 10 hours later my computer was still working on reading it. So I expect that something went wrong there.
Furthermore, when I open the file in Excel, it does split cells well. Therefore, What I tried was to save the .csv file as a .xlsx and then save it as .csv in Python with an uncommon delimiter ('~'). However, when I save my .csv file as a .xlsx file, the size of the file gets smaller, which indicates that only a part of the file is saved and that is not what I want.
Lastly, I have tried the solutions given here and here. But neither seem to work for my problem.
Given that Excel reads in the file without problems, I do expect that it should be possible to read it into Python as well. Who can help me with this?
UPDATE:
When using dataset = pd.read_csv(directory, sep = ',', error_bad_lines=False)I manage to read in the .csv. But many lines are skipped. Is there a better way for this?
panda should be work https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html
Dou you tried somthing like dataset = pd.read_csv(directory, sep = ',', header = None)
Regards
I have a simple csv file that's pipe delimited that I can load into Databricks, then disuplay the df and it displays nicely. I then try with my main dataset which is formatted the same way and an export from SQL server. After it loads the output shows that it was loaded (lists the field names and the data type it inferred -- all string though which isn't a good sign)
df = spark.read.format("csv").options(header='true', quote='"', delimiter="|",ignoreLeadingWhiteSpace='true',inferSchema='true').load("/mnt/gl/mainfile.csv")
Then I do display (df) and I don't see a nice display. Instead it shows the following:
Job 34 View
(Stages: 1/1)
Job 35 View
(Stages: 1/1)
Job 36 View
(Stages: 1/1)
Obviously the csv is at fault here but I've no idea how to go about solving this - I've already been careful with how I export it from SQL server so not sure what I'd do differently there.
Ok I solved it. If you get a similar issue it might mean your csv is not formatted properly. Open up your cv using a text editor like Ron's Editor then visually inspect the data. On my dataset for some reason the final field which is a $ amount, had a " in front of it , but not at the end of it.
e.g. "12344.67
Not sure why SQL Server would do that (I was using Import/Export Wizard) but I got rid of the " delimiter from my exported csv and it now works fine
My client uses SAS 9.3 running on an AIX (IBM Unix) server. The client interface is SAS Enterprise Guide 5.1.
I ran into this really puzzling problem: when using PROC IMPORT in combination with dbms=xlsx, it seems impossible to filter rows based on the value of a character variable (at least, when we look for an exact match).
With an .xls file, the following import works perfectly well; the expected subset of rows is written to myTable:
proc import out = myTable(where=(myString EQ "ABC"))
datafile ="myfile.xls"
dbms = xls replace;
run;
However, using the same data but this time in an .xlsx file, an empty dataset is created (having the right number of variables and adequate column types).
proc import out = myTable(where=(myString EQ "ABC"))
datafile ="myfile.xlsx"
dbms = xlsx replace;
run;
Moreover, if we exclude the where from the PROC IMPORT, the data is seemingly imported correctly. However, filtering is still not possible. For instance, this will create an empty dataset:
data myFilteredTable;
set myTable;
where myString EQ "ABC";
run;
The following will work, but is obviously not satisfactory:
data myFilteredTable;
set myTable;
where myString LIKE "ABC%";
run;
Also note that:
Using compress or other string functions does not help
Filtering using numerical columns works fine for both xls and xlsx files.
My preferred method to read spreadsheets is to use excel libnames, but this is technically not possible at this time.
I wonder if this is a known issue, I couldn't find anything about it so far. Any help appreciated.
It sounds like your strings have extra values on the end not being picked up by compress. Try using the countc function on MyString to see if any extra characters exist on the end. You can then figure out what characters to remove with compress once they're determined.
I am totally new to Stata and am wondering how to import .xlsx data in Stata. Let's say the data is in the subdirectory Data and has name "a b c.xlsx". So, from working directory, the data is in /Data
I am trying to do
import excel using "\Data\a b c.xlsx", sheet("a")
but it's not working
it's not working
is anything but a useful error report. For future questions, please report the exact error given by Stata.
Let's say the file is in the directory /home/roberto then
clear
set more off
import excel using "/home/roberto/a b c.xlsx"
list
should work.
If you are already in /home/roberto (which you can verify using display c(pwd)), then
import excel using "a b c.xlsx"
should work.
Using backslashes to refer to directories is not encouraged. See Stata tip 65: Beware the backstabbing backslash, by Nick Cox.
See also help cd.
I am trying to use the the Utilities > Member Import Utility to create an XML file that I can then use to import member data.
I have about seventy members to import. I was able to work through the mapping with what appeared to be a good match, but when I click the button, I get the following error:
Line does not match structure
I am using a .csv file to bring the data and I have selected comma as the deliminator. I can map the fields but when I click Create XML I get the Parse error.
Any suggestions on how to resolve this?
Thanks.
I found the answer. I appears to automatically understand that it is relative. When I simply put the name of the file in it went in with error.
So the correct path is: customer.txt
However, because the username is a number and not alpha numeric it cannot be imported.