I have tables that are getting continuously ingested. I need to find the source of those tables from where those tables are getting ingested in Kusto. I don't know how to identify this. Please help me with this.
Thanks,
Manoranjini
.show commands
| where CommandType == "DataIngestPull"
If the data URL was not masked, you should be able to see something like this:
Text
.ingest-from-storage into mytable<| #'https://mycluster.blob.core.windows.net/mypath/myfile.csv?''******'
Related
I have an ADF pipeline which is iterating over a set of files, performing various operations and I have an Azure CosmosDB (SQL API) instance where I would like to insert the name of file and a timestamp, mainly to keep track on which files have been already processed and which not, but in the future I might want to add some other bits of data related to each file.
What I have is my CosmosDB
And currently I am trying to utilice the Copy Data Activity for the insert part.
One problem that I have is that this particular activity expects source while at this point I have only the filename. In theory it was an option to use the Blob Storage from where I read the file at the beginning, but since the Blob Storage is set to store binary files I got the following error if I try to use it as source
Because of that I created a dummy CosmosDB Linked service, but I have several issues with this approach:
Generally the idea for dummy source is not very appealing to me
I haven't find a lot of information on the topic but it seems that if I want to use something in the Sink I need to SELECT from the source
Even though I have selected a value for the id the item is not saved with the selected value from the Source query, but as you can see from the first screenshot I got a GUID and only the name is as I want it.
So my questions are two. I just learn ADF but this approach doesn't look like the proper way to insert item into CosmosDB from activity, so a better/more common approach would be appreciated. If there is not better proposal, how can I at least apply my own value for the id column? If I create the item in the CosmosDB GUI and save it from there, as you can see I am able to use the filename as id which for now seems like a good idea to me, but I wasn't able to add custom value (string or int) when I was trying through the activity, so how can I achieve this?
This is how my Sink looks like
I am able to read hive external table using spark-shell but, when I try to read data from hive managed table it only shows column names.
Please find queries here:
Could you please try using database name as well along with table name?
sql(select * from db_name.test_managed)
If still result is same, request you to please share output of describe formatted for both the tables.
I have a query like;
SELECT
*
INTO [documentdb]
FROM
[iothub]
TIMESTAMP BY eventenqueuedutctime
I need to use * because data is dynamic and dont have specific schema. Problem is Iothub system information data is written to documentdb in this query. Is there any way to exclude Iothub system information data?
Thanks.
This is not possible currently but this will be possible in Job Compatibility Level 1.2 in near future. For now, one workaround is that you could create a post create trigger in Cosmos DB to remove this property from the document.
To answer your question, Azure stream analytics service doesn't have an in-built support for excluding columns from dynamic data (iothub information). But, we can achieve this by using UDF. Here is more info on UDF.
UDF can help us in deleting the column from input data and returning us the updated json.
There are two steps basically to achieve this:
Create a JavaScript UDF.
Go to functions from left hand side navigation (below inputs).
Click on Add --> JavaScript UDF.
Give a function alias = removeiothubinfo
keep output type - any.
copy paste following code into function definition.
function main(input) {
delete input['IoTHub'];
return input;
}
Click on Save
Update query
Go to query mode and copy paste the following query :
WITH NewInput AS
(
SELECT
udf.removeiothubinfo(iothub) AS UpdatedJson
FROM
[iothub]
)
SELECT
UpdatedJson.*
INTO
[documentdb]
FROM
NewInput
Click on Save
I suggest you to test your query before running the job by uploading a sample file containing similar structure for json.
Edited
Also, even in job compatibility level 1.2 there has been no additional functionality to achieve this. Check this out for more info.
As #chetangm said in his answer, no such filtering mechanism is supported in ASA so far. Yes, you could use create trigger in Cosmos db, however it need to be triggered in sdk code or REST API. It won't be triggered automatically.
I provide you with another workaround that using Azure Function Cosmos DB Triggered. It could be executed when data is added to or changed in Azure Cosmos DB. You just need to remove the fields you don't want in the function code.
I need to get the Product description manually from database, so please suggest which table it contains.
My Finding
Productslp table contains p_dscription column but that is clob datatype and I am unable to get the data from that.
I would suggest, you raise a support ticket with SAP Product support, since it may be a matter of correct Oracle JDBC driver.
We use HANA DB and had the same issue, they provided me with an updated driver and that resolved the issue for me.
Alternatively you can check the description in backoffice/hmc if that is an acceptable solution.
You can get using the SQL tab in hac with this query.
Select ps.p_description,ps.p_summary from products as p join productslp as ps ON p.pk=ps.itempk where pk=8796158722049
See the image below for result
HI,
I have a SP which returning more than 100 fields with 1000+ row. I need to to save all in temp table and and rum my customize query to get the appropriate data.
I did many search but i am unable to find the right solutions for my project. I will appreciate if anyone can share his idea.
create table #SP_Result
(i need to create field dynamically according to the SP return result )
exec Ministry..civil_record
"2010-08-07","Autogen",20,NULL,NULL,NULL,NULL,NULL,NULL,NULL
I need dump the result from SP to #SP_Result.
Why don't you run the query itself, in your "customised query", rather than try to capture the result set of the stored proc ? That's the normal method.
All those Nulls look like a bastard of a de-normalised "table", where many rows will not apply to the task. It is much, much faster to deal with the database in a normalised, set-oriented manner.