I'm working on converting a library from full .NetFramework to .NetCore
I'm looking for a replacement for Microsoft.Azure.Management.HDInsight.Job, which hasn't been updated in over a year and is not compatible with .NetCore. I was hoping that the functionality would be rolled up into the much-more-recently-updated and netcore-compatible Microsoft.Azure.Management.HDInsight, but that doesn't appear to be the case.
I'm down to use the REST API, but I haven't been able to find the same functionality there. Any guidance would be appreciated.
You could try to install Microsoft.Azure.Management.HDInsight.Job with Package Manager to install some prerelease versions, so that its dependencies would not be conflict with your asp.net core.
I test them, no matter it is preview, it also have the functionality what you want.
Write in Package Manager Console such as:
Install-Package Microsoft.Azure.Management.HDInsight.Job -Version 1.0.7-preview
You could only install the version <= 1.0.7-preview. If not, you may could not install it.
For more detail, you could refer to this article.
I found the REST API I was looking for. It is the WebHCat API, not an Azure API.
MapReduce Job creation: https://cwiki.apache.org/confluence/display/Hive/WebHCat+Reference+MapReduceJar
Pig Job creation:
https://cwiki.apache.org/confluence/display/Hive/WebHCat+Reference+Pig
Hive Job creation:
https://cwiki.apache.org/confluence/display/Hive/WebHCat+Reference+Hive
Sqoop Job creation: https://learn.microsoft.com/en-us/azure/hdinsight/hadoop/apache-hadoop-use-sqoop-curl and https://sqoop.apache.org/docs/1.99.3/RESTAPI.html
Hopefully they will release 3.0.0 soon
https://github.com/Azure/azure-sdk-for-net/issues/9219
Related
I'm looking for information on writing DataFlow jobs in NodeJS. The tutorials are all referring to Java or Python.
Any ideas if it's possible?
Until now(2.13.0), Beam allows writing pipeline by Python, Java and Go.
Beam community is working on cross language pipeline feature that allows call PTransforms written in another language, but NodeJS is not on the roadmap.
Check[1] for more information on it.
[1] https://beam.apache.org/roadmap/portability
There is support for TypeScript now.
As of February 2023. it's still experimental but main functionality is covered, from Apache Beam documentation:
The Typescript SDK supports Node v16+ and is still experimental.
You can install it via npm:
npm install apache-beam
More instructions and documentation can be found in the official sites at npm and github.
Link npm: https://www.npmjs.com/package/apache-beam
Link github (example): https://github.com/apache/beam-starter-typescript
We are working on a Node and Angular 5 project in which we are using Nexus as a repository of our application. This application is transpiled, packaged and uploaded to the Nexus repository. Then we have created a pipe in Jenkins that downloads the latest version of the application from Nexus and executes it. But now we are having a problem. There is no call to the Nexus API to return the path to the latest version of a node package (generated by npm publish).
The call we are making is the following:
http://HOST-NEXUS/service/rest/v1/search?repository=my-repository&name=my-node-project
But it has a problem: this call only returns the first 50 versions. If you already have more than 50 versions of the application in the repository, it does not return the last ones.
So I was wondering would you know any way to extend this limit of 50 versions? Or better yet, do you know how to get the latest version directly? Something like this (this example doesn't work in Node, but it does work in Maven):
http://HOST-NEXUS/service/rest/v1/search?repository=my-repository&name=my-node-project&version=LATEST
Thanks and best regards.
Support for "latest" download via REST was added to NXRM3 in version 3.16.0. See the JIRA ticket for more information.
An example of provided on the release blog but probably will depend on your repository, format and what you're trying to fetch. There is also a Swagger UI provided for REST in NXRM3 that can be used to form your own statements. It can be accessed in the administration section under System -> API. Sonatype's REST documentation may also help.
Api document: {nexus_url}/service/rest/swagger.json
GET {nexus_url}/service/rest/v1/search?name=project-name&sort=version
I have written some durable functions using version 1.6.2 . The new 1.7.0 is now out. I want to upgrade.
Will just doing
func extensions install -p Microsoft.Azure.WebJobs.Extensions.DurableTask -v 1.7.0 do the job? or do I need to do something else?
We don't need to do anything else except upgrading the extension. 1.7.0 release doesn't change existing APIs but adds new features and fixes bugs, so our code depending on 1.6.2 extension will not break.
As for the new features, some of them are related to DurableOrchestrationClient which is not implemented in current JS SDK, so we can't benefit from them unless SDK is upgraded. We can try others like call built-in HTTP API for starting new orchestration, e.g locally call http://localhost:7071/runtime/webhooks/durabletask/orchestrators/{functionName} to strat new orchestrator instead of using Http trigger as a starter.
About Durable Functions commands in Azure Functions Core Tools, the new release of Cli has not been pushed to npm so we can't update it as well.
I am currently using the HDInsight Hadoop Emulator, which comes with Pig Version .12. Our problem involves parsing xml files and I'd like to use the XPath command from PiggyBank, but it is only available with Pig version .13.
a. Can I Upgrade Pig in the emulator? How would I go about doing that?
b. Is the version of Pig really critical, or could I just get the latest version of the PiggyBank.jar file and use that?
currently there is no way to update component versions for HDInsight emulator (or at least that's very hard to do).
I have never used PiggyBank, but from the introduction page (https://cwiki.apache.org/confluence/display/PIG/PiggyBank) it seems that it is a collection of UDFs which should work with Pig 0.12. So i guess using the jar directly (and of course registering it in pig) should work.
Also, we are looking for an updated story for HDInsight emulator - so feel free to reach us at hdivstool at microsoft dot com if you have any thoughts, comments, requirements.
Xiaoyong Zhu from HDInsight team
I'm using the official nuget package of Windows Azure Storage Client Library to retrieve items of my Azure tables.
Recently I updated the package from version 2.0.2.0 to 2.0.5.0 and my app stopped working because the results returned by my storage query are different with the new version.
I'm looking for the library changelog in order to understand how to fix the issue.
Do you know where can I find it ?
The link provided on the nuget page seems to be outdated (it's a changelog between 1.x and 2.x, not between 2.0.2 and 2.0.5 !). Also, the Windows Azure Storage team's blog is not updated.
Please refer to the changelog.txt that is always updated with the respective source code changes.
The changelog is always up to date at https://github.com/Azure/azure-storage-net/blob/master/changelog.txt