I've been trying to set in SELECT parameter - slicestart time to gather just lately changed records:
WHERE pv.CreatedAt >= \\'{0:yyyyMMdd-HH}\\'', Time.AddHours(SliceStart, 0))"
and I got error:
Database operation failed. Error message from database execution :
ErrorCode=FailedDbOperation,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A
database operation failed with the following error: 'Conversion failed
when converting date and/or time from character
string.',Source=,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A
database operation failed with the following error: 'Conversion failed
when converting date and/or time from character
string.',Source=Microsoft.DataTransfer.ClientLibrary,''Type=System.Data.SqlClient.SqlException,Message=Conversion
failed when converting date and/or time from character
string.,Source=.Net SqlClient Data Provider,SqlErrorNumber=241,'.
I see you are using the Time.AddHours() function but passing 0 as a parameter which would add zero hours to the SliceStart value. If you did not intend to increment the SliceStart time try the following:
"sqlReaderQuery": "$$Text.Format('select ... where pv.CreatedAt >= \'{0:yyyy-MM-dd}\', SliceStart)"
Cheers,
Luis
In addition to what #Luis answer.
You have to make sure that the source date format must match the format of date being passed.
For eg:- If the source is Oracle then write query as below:-
WHERE TO_DATE(pv.CreatedAt,'YYYY-MM-DD') >= \\'{0:yyyy-MM-dd}\\'',SliceStart))"
Related
I am trying to add a minute delay to the "delay until" action by getting the utcnow timestamp, adding a minute and converting into the time format that the "delay until" expects (https://learn.microsoft.com/en-us/azure/connectors/connectors-native-delay#add-the-delay-until-action)
None of the following works and throw the error below, what am i missing here, and is there a way where we can test these function independently without logic apps ?
1) #addMinutes(utcNow('o'),1,'YYYY-MM-DDThh:mm:ssZ')
2) #addMinutes(utcNow('YYYY-MM-DDThh:mm:ssZ'),1,'YYYY-MM-DDThh:mm:ssZ')
3) #addMinutes(utcNow(),1,'YYYY-MM-DDThh:mm:ssZ')
Unable to process template language expressions in action 'Delay_until' inputs at line '1' and column '2265': 'The string was not recognized as a valid DateTime. There is an unknown word starting at index 0.'.
Please try to use this expression:
addMinutes(utcNow(),1)
I did a test, it seems to be no problem:
I'm trying to set the last modified filter in a azure data factory dataset dynamically.
I'm using the following expression:
#formatDateTime(adddays(utcnow(),-2),'yyyy-mm-ddThh:mm:ss.fffZ')
I'm getting the following error:
Activity Copy1 failed: Failure happened on 'Source' side. ErrorCode=UserErrorInvalidValueInPayload,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Failed to convert the value in 'modifiedDatetimeStart' property to 'System.Nullable`1[[System.DateTime, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]' type. Please make sure the payload structure and value are correct.,Source=Microsoft.DataTransfer.DataContracts,''Type=System.Reflection.TargetInvocationException,Message=Exception has been thrown by the target of an invocation.,Source=mscorlib,''Type=System.FormatException,Message=The DateTime represented by the string is not supported in calendar System.Globalization.GregorianCalendar.,Source=mscorlib,'
I'm also not able to preview the data with this filter. I guess something is wrong here. Any ideas?
From the error message I understand that the string represenation of the date is not supported by the calander.
The DateTime represented by the string is not supported in calendar
Why do you need to format the string for the comparison?
Actually the following commands are tested & working after publish & trigger:
#utcnow()
#adddays(utcnow(),-2)
It's the preview functionality in the front end that is not able to deal with the expressions. This will hopefully be solved by Microsoft.
Perhaps, as a workaround, you could use this expression to get a rid of the extra characters in your datetime expression:
#substring(formatDateTime(adddays(utcnow(),-2), 'o'), 0, 23)
I tested this with utcnow() and it should return the datetime in the desired format:
"value": "2019-04-12T10:11:51.108Z"
turns out you can solve the above prepending a conversion to string to your line above, so from
#formatDateTime(adddays(utcnow(),-2),'yyyy-mm-ddThh:mm:ss.fffZ')
change it to
#string(formatDateTime(adddays(utcnow(),-2),'yyyy-mm-ddThh:mm:ss.fffZ'))
it works on my end
Encountered the same problem in data flow:
currentUTC()
did not work for pulling the last modified file in blob storage but
currentTimestamp()
did
I'm using Azure Data Factory(V2) to schedule a copy pipeline activity - the requirement is that every day the job should run and select everything from a table, from the last 5 days. I have scheduled the copy and tried the following syntax in the source dataset:
select * from [dbo].[aTable] where [aDate] >= '#{formatDateTime(adddays(pipeline().parameters.windowStart, 'yyyy-MM-dd HH:mm' ),-5)}'
But this doesn't work, I'm getting an error stating that adddays is expecting an int for it's second parameter but is receiving a string.
Can anybody advise on the proper way to nest this??
Thanks
I cant test this right now, so I'll risk a possible answer just by looking at your query. I think it should be like this:
select * from [dbo].[aTable] where [aDate] >= '#{formatDateTime(adddays(pipeline().parameters.windowStart, -5), 'yyyy-MM-dd HH:mm')}'
Hope this helps!
I am trying to use REST call to HP QC ALM12. It works so far but hit a problem when trying to query based on a Date field. Wondering what's the correct syntax to specify a Date query. For example, for the field 'creation-time', how do I filter result within a period ? I have tried these but got HTTP 500 error:
{creation-time[>= '2015-01-01' AND <= '2015-06-01']}
{creation-time[>= '2015-01-01T00:00:00.000Z']
thanks for any help.
figured out the correct syntax, it is:
field_name[>=yyyy-MM-dd AND <=yyyy-MM-dd]
not quoted.
OK, so my latest project requires loading an Excel 2007 spreadsheet into a SQL Server table. I'm working in SSIS 2008R2. Based on some stuff I found on the internet, I opened the Excel source in Advanced editor and changed the datatype of the long column to DT_NTEXT, so that it wouldn't truncate it. Then I made the database column VARCHAR(MAX). This runs correctly in debug mode on my laptop.
Then I deployed it to the development server and attempted to load the same test file. It failed with the following error messages:
Error: Code: 0xC0208265
Source: Main Data Flow Task Get Main Data [1]
Description: Failed to retrieve long data for column "DESCR".
End Error
Error: Code: 0xC020901C
Source: Main Data Flow Task Get Main Data [1]
Description: There was an error with output column "DESCR" (72) on output "Excel Source Output" (9). The column status returned was: "DBSTATUS_UNAVAILABLE".
End Error
Error: Code: 0xC0209029
Source: Main Data Flow Task Get Main Data [1]
Description: SSIS Error Code DTS_E_INDUCEDTRANSFORMFAILUREONERROR. The "output column "DESCR" (72)" failed because error code 0xC0209071 occurred, and the error row disposition on "output column "DESCR" (72)" specifies failure on error. An error occurred on the specified object of the specified component. There may be error messages posted before this with more information about the failure.
End Error
Searching for information about the error, I found about a million sites offering the same three suggested solutions:
Add 'IMEX=1' to the extended properties of the connection string.
It was already there.
Change the TypeGuessRows key in the registry.
This was set to zero on the server, which I understand to mean that it should look at the entire file. Nevertheless, I changed it to 8 to match my laptop. The same error occurred when I ran it again. Then I changed it to 1,763, which is more than the number of rows in the spreadsheet. It still gave the same error. So, I put it back to zero. (There's a 1,900-character value in the first row of my test file, so it shouldn't really matter how many it checks, in this case.)
Change the datatype to DT_WSTR(4000) in the source.
The column is supposed to have up to 10,000 characters, so I'm not sure this would be a good idea even if it worked. However, I tried it anyway. This time it gave me a truncation error. I changed the truncation error disposition to "ignore failure" and it loaded the data, but truncated the value to 255 characters. I have verified that the length is 4000 and doesn't get changed when I save the file, but it's still truncating at 255 characters.
I have no idea what else to look at. Any help would be appreciated.
UPDATE 1/29: The package, without any changes, works correctly when running on the pre-production server. It still fails when running on the development server. Both servers have the same version of SSIS (including minor version numbers) as well as the same versions of Windows, Access and Excel. I do not know how to explain this, nor do I know how to tell if it would work in production.
I created a new package with similar non-functional requirements (Excel 2007 file, SSIS 2008, SQL Server 2008 R2, VARCHAR(MAX) target column) and it worked just fine after deployment into the database server. My package:
Metadata at the Excel Source component's output (checked using Advanced Editor): DT_NTEXT
Derived Column component between source and destination to cast to non-unicode from unicode using (DT_TEXT,1252)
Metadata at the OLE DB Destination component's input (checked using Advanced Editor): DT_TEXT
Target Column data type: VARCHAR(MAX)
I do not explicitly use the extended property IMEX in the connection
Executed by right-clicking on the package at the database server, and loaded a file with a few thousand characters per record into the table without truncation. Hope this helps
I have faced this issue while importing an excel file with a field containing more than 255 characters. I solved the issue using Python.
Simply, import the excel in a pandas data frame and then calculate the length of each of those string values per row.
Then, sort the dataframe in descending order. This will enable SSIS to allocate maximum space for that field as it scans the first 3 rows to allocate storage:
df = pd.read_excel(f,sheet_name=0,skiprows = 1)
df = df.drop(df.columns[[0]], axis = 1)
df['length'] = df['Item Description'].str.len()
df.sort_values('length', ascending=False, inplace=True)
writer = ExcelWriter('Clean/Cleaned_'+f[5:])
df.to_excel(writer,sheet_name='Billing',index=False)
writer.save()