I want to use a string I've created in a for loop to determine the dtypes of a Dataframe to send through sqlalchemy.
dtype_str = "'var1' : sqlalchemy.types.VARCHAR(8), 'var2' : sqlalchemy.types.VARCHAR(4)"
df.to_sql('table',connect,dtype={dtype_str})
When I run that I get an error saying "ValueError: The type of var1 is not a SQLAlchemy type.
If I just copy and paste it into without the quotes it works. I'm trying to build something that can work on different datasets so I don't want to have to copy/paste that string each time.
Related
I'm trying to use PySpark's split() method on a column that has data formatted like:
[6b87587f-54d4-11eb-95a7-8cdcd41d1310, 603, landing-content, landing-content-provider]
my intent is to extract the 4th element after the last comma.
I'm using a syntax like:
mydf.select("primary_component").withColumn("primary_component_01",f.split(mydf.primary_component, "\,").getItem(0)).limit(10).show(truncate=False)
But I'm consistently getting this error:
"cannot resolve 'split(mydf.primary_component, ',')' due to data
type mismatch: argument 1 requires string type, however,
'mydf.primary_component' is of
structuuid:string,id:int,project:string,component:string
type.;;\n'Project [primary_component#17,
split(split(primary_component#17, ,)[1], \,)...
I've also tried escaping the "," using \, \\ or not escaping it at all and this doesn't make any difference. Also, removing the ".getItem(0)" produces no difference.
What am I doing wrong? Feeling a dumbass but I don't know how to fix this...
Thank you for any suggestions
You are getting the error:
"cannot resolve 'split(mydf.`primary_component`, ',')' due to data
type mismatch: argument 1 requires string type, however,
'mydf.`primary_component`' is of
struct<uuid:string,id:int,project:string,component:string>
because your column primary_component is using a struct type when split expects string columns.
Since primary_component is already a struct and you are interested in the value after your last comma you may try the following using dot notation
mydf.withColumn("primary_component_01","primary_component.component")
In the error message, spark has shared the schema for your struct as
struct<uuid:string,id:int,project:string,component:string>
i.e.
column
data type
uuid
string
id
int
project
string
component
string
For future debugging purposes, you may use mydf.printSchema() to show the schema of the spark dataframe in use.
enter image description here
select from list by index ${locator_var} ${inp_msge_type}
--getting error as expected string, int found
select from list by index ${locator_var} 7
-----not getting any error
${inp_msge_type}----contains 7 from DB query the result is stored in this variable, to avoid hard coding we need to do this
Is there any way to write
Do not add links to screenshots of code, or error messages, and format the code pieces accordingly - use the ` (tick) symbol to surround them.
The rant now behind us, your issue is that the keyword Select From List By Index expects the type of the index argument to be a string.
When you called it
Select From List By Index ${locator_var} 7
, that "7" is actually a string (though it looks like a number), because this is what the framework defaults to on any typed text. And so it works.
When you get the value from the DB, it is of the type that the DB stores it with; and probably the table schema says it is int. So now you pass an int to the keyword - and it fails.
The fix is simple - just cast (convert) the variable to a string type:
${inp_msge_type}= Convert To String ${inp_msge_type}
, and now you can call the keyword as you did before.
QuotedPremium column is a string feature so I need to convert it to numeric value in order to use algorithm.
So, for that I am using Edit Metadata module, where I specify data type to be converted is Floating Point.
After I run it - I got an error:
Could not convert type System.String to type System.Double, inner exception message: Input string was not in a correct format.
What am I missing here?
As mentioned in comments, you must change column where numbers are handled as text to numeric type data and it shouldn't have any null values. Now answering the question of how to substitute NULL's in data using ML studio and converting to numeric type.
Substitute NULL's in data
Use Execute R Script module for that, and add this code in it.
dataset1 <- maml.mapInputPort(1); # class: data.frame
dataset1[dataset1 == "NULL"] = 0; # Wherever cell's value is "NULL", replace it with 0
maml.mapOutputPort("dataset1"); # return the modified data.frame
Image for same:
Convert to numeric data
As you have added in your answer, this can be done using the Edit Metadata module.
I'm trying to do a Subsonic Query with an IN statement containing multiple strings. If I manually hard-code the strings, it works fine. Example:
Dim qry As New [Select]("mySelectColumn")
qry.From(table.Schema)
qry.Where(table.Columns.mycolumn).In("string1", "string2", "string3")
Hoever, I need to be able to pull the IN statement strings from a single variable in VB, which would make the last line look like:
qry.Where(table.Columns.mycolumn).In(combinedString)
But whenever I try and concat the strings together into a single VB string, I get no results. I can't even tell exactly what SQL it's trying to pass. Using buildsqlstatement() just shows the IN statement with :mycolumn0In1, :mycolumn0In2, :mycolumn0In3...I can't tell what it's actually trying to do.
I've tried these variations for the VB variable to no avail:
mystring = """mystring1"", ""mystring2"""
mystring = "'mystring1', 'mystring2'"
Any ideas how I can concat multiple strings into one IN clause with VB and Subsonic?
A coworker pointed me to http://blog.levo.us/index.php/2008/06/25/subsonic-select-where-in-solution/ and I eventually found a solution. My problem was that I was not explicitly declaring the VB variable as an arraylist. I was simply Dim'ing it and assigning it the results of a function which were returned as an ArrayList.
what didn't work - the getstrings() function returns an ArrayList:
Dim myarraylist = getstrings()
...
qry.Where(table.Columns.mycolumn).In(myarraylist)
What did work:
Dim myarraylist as ArrayList = getstrings()
...
qry.Where(table.Columns.mycolumn).In(myarraylist)
I am trying to save data into my database using a vb form. I am using a datetimepicker to insert the date into my database. Here's an example
saveCommand.Parameters.AddWithValue("A_BOOKDATE", abookdatePicker1.Text).DbType = DbType.DateTime
In my database i set its attribute to time ,now the thing is i used the same line of code on another database and it worked but this one is giving this error
'Failed to convert parameter value from string to datetime'
How do i convert this string to datetime.
Many Thanks
Use DateTimePicker.Value instead of DateTimePicker.Text:
saveCommand.Parameters.AddWithValue("A_BOOKDATE", abookdatePicker1.Value) _
.DbType = DbType.DateTime
(If you ever actually have to convert a string to a DateTime, use DateTime.TryParseExact or DateTime.ParseExact, but in this case it's better not to use the string representation at all.)
You could also try to use
Convert.ToDateTime Method (String) (this one will return a datatime object or throw and exception)
DateTime.TryParse Method (String, DateTime) (this one returns the datetime as an output parameter, returns null on failure)