Aerospark build (and running with PySpark) - apache-spark

I am having trouble building the Aerospark connector # https://github.com/aerospike/aerospark
I have created fresh Ubuntu box, installed JDK, SBT, Maven, Scala 2.10, and then followed the steps at the github page above. During the build i get this error and I am not sure which is the most direct way to build this properly....
[error] 1 error was encountered during merge
java.lang.RuntimeException: deduplicate: different file contents found in
the following: /opt/astools/aerospark/lib/aerospike-helper-java-
1.0.6.jar:META-INF/maven/com.aerospike/aerospike-client/pom.xml
/root/.m2/repository/com/aerospike/aerospike-helper-java/1.0.6/aerospike-
helper-java-1.0.6.jar:META-INF/maven/com.aerospike/aerospike-client/pom.xml
/root/.ivy2/cache/com.aerospike/aerospike-client/jars/aerospike-client
3.3.1.jar:META-INF/maven/com.aerospike/aerospike-client/pom.xml
Are there any updated instructions for building this - the git link in the instructions seems outdated incidentally.
As an aside, has anyone tried this with PySpark?

There was an issue with the build file which affected ubuntu. Can you pull the latest changes and go through the build process again?

Related

Set Glue Code to External Libraries in Cucumber

We have multiple testing repos, and some of the scenarios depend on the steps already created in a separate repo, so I'm trying to build the JAR file and include it in the external libraries of the other repo. Then I define my gluecode in the IntelliJ runner with two separate lines:
com.edge.automation
C:\Users\MY_NAME\.m2\repository\com\reissue-automation\2.0.3-SNAPSHOT\reissue-automation-2.0.3-SNAPSHOT-tests.jar!\com.reissue.automation.stepdefinitions
IntelliJ is able to recognize the Gherkin sentence, but when I run it, it is throwing this exception:
eissueautomationstepdefinitions'
at io.cucumber.core.options.CucumberPropertiesParser.parseAll(CucumberPropertiesParser.java:156)
at io.cucumber.core.options.CucumberPropertiesParser.parse(CucumberPropertiesParser.java:88)
at io.cucumber.core.cli.Main.run(Main.java:48)
at io.cucumber.core.cli.Main.main(Main.java:33)
Does anybody know what this error means or if it's possible to include glue code from external libraries?
I ended up running the following command to copy my dependencies into the target folder which added it to the classpath.
mvn install dependency:copy-dependencies -DskipTests
Then it picked up the glue no problem.
com.edge.automation
com.reissue.automation.stepdefinitions
If anyone has a better solution feel free to post.
May you add the detailed steps on how you accomplish it?
I tried running
mvn install dependency:copy-dependencies -DskipTests
And then running my maven command for running the test cases and not luck.
If you can add your folder structure and how you put it in the glue code of cucumber runner will be appreciated.
Also cucumber version might help.
Thanks

Building maciejhirsz/logos on Rust nightly-2020-06-13

I have a project using Rust nightly with logos 0.11.4 as a dependency. When I run cargo build, I get
error: failed to download `logos v0.11.4`
Caused by:
unable to get packages from source
Caused by:
failed to parse manifest at `C:\Users\jonat\.cargo\registry\src\github.com-1ecc6299db9ec823\logos-0.11.4\Cargo.toml`
Caused by:
readme file with name '../README.md' was not found
It turns out that logos has a nested package called logos-derive, and the Cargo.toml for this project has the line readme = "../README.md". In other words, it's pointing to the README from the parent directory. However, the Cargo source registry places the logos-0-11.4 and logos-derive-0.11.5 directories at the same level, rather than nesting them, which explains why the file is not found. I can actually get it to build by creating an empty .cargo/registry/src/github.com-1ecc6299db9ec823/README.md. But that's obviously not a satisfactory long-term fix.
Note that I was able to build just fine two days ago (2020-06-11), but despite there not being any changes to logos or logos-derive in that time, I'm now unable to build - even after checking out my project from that time and downgrading my toolchain to nightly-2020-06-11. I'm not sure how that's possible or what in my build process has changed.
Am I doing something wrong, is logos-derive doing something wrong, or has Cargo changed something recently? Anyone have a proper solution?
The issue seems to have mysteriously disappeared again in nightly-2020-06-13. I don't know what the cause was, but I guess this is resolved.

error MSB4086: A numeric comparison was attempted on "$(MSBuildVersion)" that evaluates to "" instead of a number

Trying to deploy the project with Azure Kudu (reading from a BitBucket repo) I get the following error in Deployment Center within Azure Portal...
This error happened with all the latest versions of Microsoft.Net.Compilers NuGet package. The only version that worked was 3.0.0 from 5 months ago.
Anyone knows what's the cause of this? Is Azure Kudu not yet updated?
Command: "D:\home\site\deployments\tools\deploy.cmd"
Handling .NET Web Application deployment.
MSBuild auto-detection: using msbuild version '14.0.23107.0 built by: D14REL' from 'D:\Program Files (x86)\MSBuild\14.0\Bin'.
Restoring NuGet package Microsoft.Net.Compilers.3.1.1.
GET https://api.nuget.org/v3-flatcontainer/microsoft.net.compilers/3.1.1/microsoft.net.compilers.3.1.1.nupkg
OK https://api.nuget.org/v3-flatcontainer/microsoft.net.compilers/3.1.1/microsoft.net.compilers.3.1.1.nupkg 937ms
Installing Microsoft.Net.Compilers 3.1.1.
Adding package 'Microsoft.Net.Compilers.3.1.1' to folder 'D:\home\site\repository\packages'
Added package 'Microsoft.Net.Compilers.3.1.1' to folder 'D:\home\site\repository\packages'
NuGet Config files used:
D:\local\UserProfile\AppData\Roaming\NuGet\NuGet.Config
Feeds used:
D:\home\.nuget\
https://api.nuget.org/v3/index.json
Installed:
1 package(s) to packages.config projects
D:\home\site\repository\packages\Microsoft.Net.Compilers.3.1.1\tools\Microsoft.Managed.Core.targets(63,60): error MSB4086: A numeric comparison was attempted on "$(MSBuildVersion)" that evaluates to "" instead of a number, in condition "$(MSBuildVersion) >= 16.1.0". [D:\home\site\repository\MyCompany\MyCompany.csproj]
Failed exitCode=1, command="D:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe" "D:\home\site\repository\MyCompany\MyCompany.csproj" /nologo /verbosity:m /t:Build /t:pipelinePreDeployCopyAllFilesToOneFolder /p:_PackageTempDir="D:\local\Temp\8d727f56aa60788";AutoParameterizationWebConfigConnectionStrings=false;Configuration=Release;UseSharedCompilation=false /p:SolutionDir="D:\home\site\repository\.\\"
An error has occurred during web site deployment.
\r\nD:\Program Files (x86)\SiteExtensions\Kudu\84.10807.4030\bin\Scripts\starter.cmd "D:\home\site\deployments\tools\deploy.cmd"
### EDIT ###
I also opened an issue # Kudu GitHub repo.
I’ve had a similar issue with a kudu deployment and typescript versions. The basic issue seem to be that kudu is on its way out in favour of Azure DevOps so some dependencies of msbuild are becoming out of date. These are three options that might help you resolve the issue:
Switch to Azure DevOps - you’ll have access to various build agents and much more control over what they are capable of, but generally they’ll just be more up to date
Amend your build script - this is the script that kudu uses and you can customise it and then put it in source control, see this kudu doc for more info. Looks like you’re targeting msbuild 14 but I’m pretty sure 15 should be available and you can point your build script at this and/or make other tweaks
Process some of your build offline, ie before committing - this was my temporary solution with typescript, I was able to transpile to JavaScript and commit the js meaning I could disable typescript processing for kudu
It may also help to dig around in the targets file referenced in your log and trace back where the problem starts through the various msbuild conditions in that file. You can see all this by exploring via a command prompt in kudu.

Buildozer failing with pip dependencies, 'no valid dependency graph'

I'm attempting to compile a kivy app for android using the python3crystax recipe. When building only with recipes listed in python-for-android, it compiles fine, but whenever I attempt to add a pip dependency, it fails with this error:
[ERROR]: Didn't find any valid dependency graphs.
[ERROR]: This means that some of your requirements pull in conflicting dependencies.
[ERROR]: Exiting.
When using the python2 recipe, however, it compiles fine. Does python3crystax support installing dependencies over pip?
It sounds like your dependency does have a recipe, but that the recipe isn't marked as python3crystax compatible. Give more information about what you're attempting if you want a more specific answer.
I was able to workaround this issue by changing the python-for-android branch that buildozer uses from stable to master in my buildozer.spec file. This gave me access to the new python3 recipe and that seems to be working fine.

TeamCity WindowsAzure.targets causing errors in deployment

We are currently trying to setup an Angular 2 project that also contains a Cloud Service project in its solution. We have set up numerous Angular 1 projects containing Cloud Service projects and have had no errors. When trying to run the MSBuild step for the Angular 2 project, we get the following error on the TeamCity build server:
C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v14.0\Windows Azure Tools\2.7\Microsoft.WindowsAzure.targets(2787, 5): error MSB3021: Unable to copy file "C:\TeamCityBuildAgent\temp\buildTmp\.NETFramework,Version=v4.6.1.AssemblyAttributes.cs" to "C:\SourceControlFolder\Apps\MyApp\MyApp.CloudDeployment\obj\Debug\Website\C:\TeamCityBuildAgent\temp\buildTmp\.NETFramework,Version=v4.6.1.AssemblyAttributes.cs". The given path's format is not supported.
The main problem seems to be the build attempting to throw the full C:\TeamCityBuildAgent.. path into the obj\Debug\Website folder.
My current MSBuild parameters are:
/p:Configuration="Dev"
/p:platform="Any CPU"
/p:OctoPackEnforceAddingFiles=true
/p:OctoPackProjectName=MyApp_Dev
I've tried the following solutions:
Making sure the latest Visual Studio Update is installed (for MSBuild)
Running the install tool to repair Azure Tools 2.7 on the build server.
Changing the debug configuration in the build.
Ensuring TypeScript 1.7.6 is installed on the build server (even though it's an Azure.targets issue) [https://github.com/Microsoft/TypeScript/issues/6215]
Updated the project to Azure Tools 2.9. The same error remains except it's failing on the 2.9 folder instead now.
Curious if anyone else has experienced this error and knows a fix. I'll keep this post updated as I try other solutions.
Update
It appears that what is causing this issue is this line within the .csproj file:
<FilesToIncludeForPublish>AllFilesInProjectFolder</FilesToIncludeForPublish>
This is specified for our build configurations to copy all files generated by the ng build to the output directory of the publish. What I don't understand at the moment is how this works on all of the devs' local machines but does not work on our build server. Will keep this posted as I find more info or any kind of workaround.
This ended up being the fix:
Instead of targeting the .sln in MSBuild, we specifically targeted the .csproj file. This fixed the build errors and allowed us to keep the FilesToIncludeForPublish tag inside.

Resources