I am trying to copy a dbf database into a sqlite database
I have the following code
db=dataset.connect('sqlite:///:memory:')
table =db['table1']
for record in DBF(settings['xbasefile']):
db['table1'].insert(record)
this loads the record but fails to insert with a datatype mismatch on the ID column because the row coming in has a format like
ID:text
field1:Text
this function
table=db['table1']
seems to assume an int id for the table. Any way to get this to do an insert with the text id that is in the table?
Ended up using the dbf2sqlite utility which automatically creates the table with correct columns from the dbf
Related
I am new to ADF. I have a pipeline which deletes all rows of any of the attributes are null. Schema : { Name, Value, Key}
I tried using a data flow with Alter Table and set both source and sink to be the same table but it always appends to the table instead of overwriting it which creates duplicate rows and the rows I want to delete still remain. Is there a way to overwrite the table.
Assuming that your table is SQL table, I have tried to overwrite the source table after deleting the specific null values. It successfully deleted the records but got the duplicate records even after exploring various methods.
So, as an alternate you can try the below methods to achieve your requirement:
By Creating new table and deleting old table:
This is my sample source table names mytable.
Alter transformation
Give new table in the sink and in settings->post SQL scripts. give the drop command to delete the source dataset. Now your sink table is your required table. drop table [dbo].[mytable]
Result table(named newtable) and old table.
Source table deleted.
Deleting null values from source table using script activity
Use script activity to delete the null values from source table.
Source table after execution.
I used ODBC to create a sheet in Excel and add a row to it.
Literally the commands were just:
create table 'update5' ('age' NUMBER);
insert into 'update5'.'age' values (1);
This works and I can see the rows in the sheet and via DBVisualiser and my ODBC query results.
Later, I wrote more SQL to add another row like so:
insert into 'update5' ('age') values (2);
but I get the error:
[Microsoft][ODBC Excel Driver] Cannot expand named range.
I do not know why named ranges are being used, is there a way I can set ODBC to not use them?
Without knowing more about what your doing, what you're working with, and your end-goal I can't give a definite answer - however, if you're saying this works fine as-is:
create table 'update5' ('age' NUMBER);
insert into 'update5'.'age' values (1);
...then it stands to reason that this:
insert into update5 values (2);
...will not work because your missing:
quotation marks (which may or may not be optional in your environment), and,
the field name to which you want to export.
In the first insert into statement you have:
'update5' <-- the destination table
.'age' <-- the destination field
values (1); <-- the value to insert
...so if you're just trying to add a record with the number 2 to the same field, use the same code:
insert into 'update5'.'age' values (2);
More Information:
w3schools : SQL INSERT INTO Statement
TutorialsPoint : SQL INSERT Query (Tutorial)
Found it.
When you create an table in Excel via ODBC you create a named range of the same name within that table.
When you try to insert like this:
insert into 'update5'.'age' values (2);
It is interpreted as you trying to add to the the named range called update5 which is inside the table update5.
You need to use the syntax:
insert into [update5$].'age' values (2);
to add values to the table called update5.
I have the below situation. I receive data from a datasource in CSV format every two weeks. I upload that to a postgres dB. I need to ensure the following
data in postgres should not be deleted
any updates in CSV need to be carried over without adding new rows
any new data marked by uinque ID Needs to be added
In other words the diff between the data set needs to be appended to postgres
In today's implementation I am using node-postgres to stream the data to postgres
I dont know how to implement the updates
Any ideas ? Ideally if there is a way to create a temp table stream the new data and do a diff between the old and temp table will be good.
If the CSV has a unique ID already, and you're using PostgreSQL 9.5+, then you can use INSERT ... ON CONFLICT DO UPDATE .... Otherwise you could create a plpgsql stored procedure with parameters (either individual values or a single ROW parameter), that does
UPDATE table SET
value = param_value
...
WHERE ID = param_id;
IF NOT found THEN
INSERT INTO table (ID, value, ...)
VALUES (param_id, param_value, ...);
END IF;
And execute that function for each row on the CSV. You can first import the CSV into a temporary table and do
SELECT the_above_function(f.id, f.value, ...)
FROM csv_temp_table f;
I'm trying to update one table from one database to another database in visual studio 2012 using schema compare. The table exists in both databases, the first one has more columns than the one in the second db. The schema compare shows me that this table is different displaying me the new columns on the left size of the window.
When I try to generate the script for updating the table on the database in the right side, is creating a temp table but is not listing the whole columns, so it gives me an error in one column that is NOT NULL.
Any idea of this behavior? shouldn't be straightforward for this simple change?
I have a csv file that I want to load the data in my database table. In my control file I am trying to load the data but I get a constraint error that one of my columns that I am not selecting cannot be null value. So instead of an insert can I do an update in my control file?
This is my error Traffic_Profile_Name cannot be null, but I don't need this column so I rather do an update based on ID only.
Record 1: Rejected - Error on table TRAFFIC_PROFILE_TEMP.
ORA-01400: cannot insert NULL into ("SERVICE_USER"."TRAFFIC_PROFILE_TEMP"."TRAFFIC_PROFILE_NAME")
I only have the ID in my first column in the csv, I want to update the table based on ID is 124 updated the other 5 columns Y,Y,Y,Y,STANDARD Here is my csv file below:
124,Y,Y,Y,Y,STANDARD
125,Y,Y,Y,Y,BENIGN
126,Y,N,N,N,BENIGN
140,Y,Y,N,N,FRAME
141,Y,Y,N,N,FRAME
My control file:
LOAD DATA
INFILE '/home/ye831c/migration/log/conv2015_10_LogicalComponent_CosProfile.csv'
BADFILE '/home/ye831c/migration/bad/conv2015_10_LogicalComponent_CosProfile.bad'
DISCARDFILE '/home/ye831c/migration/bad/conv2015_10_LogicalComponent_CosProfile.dsc'
APPEND
INTO TABLE TRAFFIC_PROFILE_temp
FIELDS TERMINATED BY ","
TRAILING NULLCOLS
(TRAFFIC_PROFILE_ID, PE_INGRESS_FLAG, PE_EGRESS_FLAG, CE_INGRESS_FLAG, CE_EGRESS_FLAG, COS_PROFILE_TYPE)