Is it possible to convert CSV to JSON using built-in/managed/3rd party template, without using Azure Function via Azure Logic App?
Below is using Azure Function, which is generated automaticately. However, I cannot find the link like what it mentions. Ideally, no Azure function is required.
http://blogs.recneps.org/post/Processing-a-flat-file-with-Azure-Logic-Apps
https://social.msdn.microsoft.com/Forums/en-US/e0ea1adc-1979-44df-a4d1-52290338bc78/transform-csv-in-logic-app?forum=azurelogicapps
Below, No CSV to JSON available.
https://learn.microsoft.com/en-us/azure/logic-apps/logic-apps-enterprise-integration-liquid-transform
I will admit this is not my proudest work but it seems to work fairly well. I was able to turn a CSV file from my onedrive into JSON objects.
//Updated with less variables, splits, set and replaces actions
Input
Output(second object, first object and last needs to be purged)
How? A lot of steps that could possibly be removed and merged but. Using splits and replace actions I could single out each line and down the line create a JSON object. I was going first for an array but eventually, it was not that hard to make it into a JSON object. Not entirely sure how it works with null values.
This is probably not the best way to handle this, drawbacks here is that it is a lot of actions, the first object is the headers and that needs to be removed, there will also be a very last object that is just null(which is fine).
Entire schema
Concurrency set to 1 here
Related
I am using ruamel.yaml library to load and process YAML file.
The YAML file can get updated after I have called
yaml.load(yaml_file_path)
So, I need to call load() on the same YAML file multiple times.
Is there a way/optimization parameter to pass to loader to load only the new entries in the YAML file?
There is no such facility currently built into ruamel.yaml.
If a file consists of multiple YAML documents, you can optimize the loading, by splitting the file on the document marker (---). This is fairly trivial and then you can load a single document from start to finish.
If you only want to reload parts of a document things get more difficult. If there are anchors and aliases involved, there is no easy way to do this as you may need a (non-updated) anchor definition in an updated part that needs an alias. If there are no such aliases, and you know the structure of your file, and have a way to determine what got updated, you can do partial loads and update your data structure. You would need to do some parsing of the YAML document, but if you only use a subset of YAML possibilities, this is often possible.
E.g. if you know that you only have simple scalar keys at the root level mapping of a YAML document, you can parse the document and extract non-indented strings that are followed by the value indicator. Any such string that is not in your "old" data structure is a new key and its value should be parsed (i.e. the YAML document content until the next non-indented string).
The above is far less trivial to do for any added data that is not added at the root level (whether mapping or sequence).
Since there is no indication within the YAML specification of the complexity of a YAML docment (i.e. whether it includes anchors, alias, tags etc), any of this is less easy to built in ruamel.yaml itself.
Without specific information on the format of your YAML document, and what can get updated, specific implementation details cannot be given. I assume however that you will not update and write out the loaded data, if that is so, make sure to use
yaml = YAML(typ='safe')
when possible as this will get you much faster loading times than the default round-trip loader provides.
In my karate tests i need to write response id's to txt files (or any other file format such as JSON), was wondering if it has any capability to do this, I haven't seen otherwise in the documentation. In the case of no, is there a simple JavaScript function to do so?
Try the karate.write(value, filename) API but we don't encourage it. Also the file will be written only to the current "build" directory which will be target for Maven projects / stand-alone JAR.
value can be any data-type, and Karate will write the bytes (or plain-text) out. There is no built-in support for any other format.
Here is an example.
EDIT: for others coming across this answer in the future the right thing to do is:
don't write files in the first place, you never need to do this, and this question is typically asked by inexperienced folks who for some reason think that the only way to "save" a response before validation is to write it to a file. No, please don't waste your time - and please just match against the response. You can save it (or parts of it) to variables while you make other HTTP requests. And do not write your tests so that scenarios (or features) depend on other scenarios, this is a very bad practice. Also note that by default, Karate will dump all HTTP requests and responses in the log file (typically in target/karate.log) and also in the HTML report.
see if karate.write() works for you as per this answer
write a custom Java (or JS function that uses the JVM) to do what you want using Java interop
Also note that you can use karate.toCsv() to convert JSON into CSV if needed.
My justification for writing to a file is a different one. I am using karate explicitly to implement a mock. I want to expose an endpoint wherein the upstream system will send some basic data through json payload using POST/PUT method and karate will construct the subsequent payload file and stores it the specific folder, and this newly created payload file will be exposed through another GET call.
We are consuming apis returning json in our projects. The json from those api can contain rather large structures which need to be mapped into other large structures (usually json but could be xml or csv rarely).
We used to use dataweave (from Mulesoft) to do that, and if you're not familiar with dataweave, it's pretty good at that sort of mapping. It let's you define a sample input, and while editing the dataweave it shows you a preview of the result in a separate pane.
For some apis we switched to using nodejs (because it offers better control and debugging than Mule, long story). But I'd really like the same mapping experience as dataweave.
So I guess the question is: can I use vscode to define an input file in a directory, a transformation file in javascript and have the resulting mapped output display in a pane which is updated live?
Is there some plugin offering that? Couldn't find it.
My understanding is the following:
You have a mule workflow which needs to read a file(you edited the file in vscode) and execute a server side javascript (nodjs) to transform the file and after the result is obtained, the mapped result will be pushed into a web page ? right ?
All happen under a given mule workflow right? and you are wondering there is any mule connector to do this process ? right ?
I am trying to create a unique id and attach it to a LabVIEW file so that later I can read back that id and know I am looking at exactly the same file. I must be able to generate the id and read it back programmatically. The information must be hidden from the user accessing the file and persist across operating systems and storage mediums.
I was thinking of just adding it into the file 'comment' section, but it appears the Windows no longer supports file comments.
Any ideas?
I would suggest using the Set Tag method on the VI (with persistence) and saving multiple copies of the VI using the Save Instrument method, where each copy would then have its own unique value of the tag. Note that the method is defined as a scripting method, so you would probably need to enable scripting in the LV options to access it.
I don't think there's any API for enumerating tags, so there would be no way to find the value unless you know what the tag is called. Obviously, it's encoded somewhere in the VI file itself, and presumably it would be possible to extract a list of tags if you know the structure of the file, but that's not knowledge I would expect most students to have.
I'm not sure if you can call the set tag method from Python using an external connection to VI server, but certainly the LV code for this should be pretty simple - basically a for loop with ID generation, a couple of method calls and saving a CSV file so you can keep track of which ID belongs to which student. If you use existing IDs, you don't even need to generate and save the IDs.
In my karate tests i need to write response id's to txt files (or any other file format such as JSON), was wondering if it has any capability to do this, I haven't seen otherwise in the documentation. In the case of no, is there a simple JavaScript function to do so?
Try the karate.write(value, filename) API but we don't encourage it. Also the file will be written only to the current "build" directory which will be target for Maven projects / stand-alone JAR.
value can be any data-type, and Karate will write the bytes (or plain-text) out. There is no built-in support for any other format.
Here is an example.
EDIT: for others coming across this answer in the future the right thing to do is:
don't write files in the first place, you never need to do this, and this question is typically asked by inexperienced folks who for some reason think that the only way to "save" a response before validation is to write it to a file. No, please don't waste your time - and please just match against the response. You can save it (or parts of it) to variables while you make other HTTP requests. And do not write your tests so that scenarios (or features) depend on other scenarios, this is a very bad practice. Also note that by default, Karate will dump all HTTP requests and responses in the log file (typically in target/karate.log) and also in the HTML report.
see if karate.write() works for you as per this answer
write a custom Java (or JS function that uses the JVM) to do what you want using Java interop
Also note that you can use karate.toCsv() to convert JSON into CSV if needed.
My justification for writing to a file is a different one. I am using karate explicitly to implement a mock. I want to expose an endpoint wherein the upstream system will send some basic data through json payload using POST/PUT method and karate will construct the subsequent payload file and stores it the specific folder, and this newly created payload file will be exposed through another GET call.