Constraints:
My application requires SSE4.2 instruction set.
I am using packer.io to provision my Windows Azure VM (OpenLogic 6.5 OS.)
Windows Azure returns an AMD-processor-backed-VM about 15% of the time. The rest of the time - they are Intel-processor-based. AMD processors do not support SSE4.2, but they do support SSE4a. So, my application is terminated with SIGILL on AMD processors.
Questions:
Can I request specific architecture (Intel CPU) when Packer
provisions a VM? I know that instance types >= A8 come only with Intel processors, but they are more expensive, and I would not want to use them for development.
If Packer cannot do it, what are the other options
(Powershell, ect...) that would give me this functionality?
Thank you.
Answering my own question. Azure does not provide a way to request processor type. The only way to ensure Intel processor is to not use A-series machines (as confirmed by a MSFT representative.) Thus, no tool can do it.
Related
We have Service Fabric application, that creates VMSS when we create the clusters. While creation we have to select the VM Size. We have 3 different regions where we have deployed our application.
Although the VM size selected is same for all 3 regions. The processor assigned is different. That's not a problem if the processors are similar in their performance. But it's not.
https://learn.microsoft.com/en-us/azure/virtual-machines/dv2-dsv2-series#dv2-series
Above link states:
Dv2-series sizes run on Intel® Xeon® Platinum 8272CL (Cascade Lake), Intel® Xeon® 8171M 2.1GHz (Skylake) or the the Intel® Xeon® E5-2673 v4 2.3 GHz (Broadwell) or the Intel® Xeon® E5-2673 v3 2.4 GHz (Haswell) processors with Intel Turbo Boost Technology 2.0.
With same code, one region is performing well, but other regions usually have CPU maxed out. On talking to Microsoft support, they said processors are assigned randomly and they cannot change it.
Only option suggested by support, that we try change the cluster and node by stopping and starting all VMSS instance at the same time in the Azure portal manually.
If we look at the performance benchmark for 2 processor assigned to us:
https://www.cpubenchmark.net/compare/Intel-Xeon-E5-2673-v4-vs-%5BDual-CPU%5D-Intel-Xeon-Platinum-8171M/2888vs3220.2
Now the only option we are left with is try restarting VMSS for n number of times or upgrade to different size.
Anyone faced similar situation? If yes what was the resolution?
Or any information, what are the design consideration by which Microsoft assigns a particular processor to the VM?
I wouldn't read too much into the comparative specs between each processor because you're ultimately not getting the full processor - you're only buying the vCPUs. Each is are supposed to have similar performance from one to another. This would suggest that Microsoft may pack more VMs onto an 8171M host than onto an E5-2673 so the vCPUs across either machine are closer in performance equivalency within the SKU series.
Put simply, you have no idea how many VMs Microsoft is running off any given processor and it would only make sense that they run more off a higher performing host system.
If you want to have full operational performance of the processor, you'd have to buy a dedicated host. Note that the pricing sheet names precisely which processor SKU you get for your money unlike the vCPU mix and match happening in the D#_v2 SKU series.
I have a python script that runs programs in parallel using joblib and it works just fine (100% cpu consumption on local machine).
Lately, I've migrated the python script to data science virtual machine (DSVM) on azure but found that the multiprocessing logic does not work (cpu consumption remains low level 5% on average).
Does anyone know whether azure VM supports parallelization? if yes, what python package I should use? "joblib" is currently used.
if no, any alternatives you can suggest for parallelization on azure VM?
Thank you in advance and stay healthy!!!
Regards,
Kelvin
What is your local computer OS (Windows, Linux, distro, version)? was the test run in real HW or virtual? what virtualization technology (Hyper-V, vmware, other)?
The virtualization offered by Azure should match real HW so I would first look at the OS and dependencies in regards to parallelization, i.e. process or thread base, etc.
I'm not pro on Python however think there should be a way for you to identify what is blocking one process or thread to start another, kind of bottleneck. Knowing this should lead in the right direction to resolve the issue.
Cesar
I am trying to use the InfiniBand on an A8 machine on Azure. Actually, the ping-pong test works fine however, I can not run a simple RDMA based program. I can find the device by ibv_get_device_list(NULL) but I can not open it by the ibv_open_device(). Is that true that all RDMA based applications must use Intel MPI?
Thank you,
As of today, Azure RDMA instances only allow Intel MPI to utilize RDMA capabilities. This is why any general purpose RDMA app will not work.
The good news is that this is going to change soon, as Azure intend to introduce support for general purpose RDMA in the very near future (2018), so hang tight!
Thanks
For the record, Azure finally opened IBVERBS API for certain instance types.
I was checking project Embedded ECG data acquisition system from instructables and there is mension a TODO:
Combining the OS and bare-bone firmware
UNDER CONSTRUCTION
** Since the bootloader only loads one firmware to the Core,
I need to modify the ELF file, to have Linux and bare-bone Core at the same time **
It seems to me as interresting approach how to make full featured Linux and critical realtime OS on one board (for example Raspberry PI). It is really possible? I have heard, that Linux can be setup to not use some cores. But I suppose that Linux use virtual memory and bare-bone firmware does usually not. Can the memory be shared between these OS. What about interruptions? Can these two OS handle interruptions separately? Can boot loader load these two systems to both core at once? I can imagine that one thread in boot loader will skip to address of bare-bone OS. It is correct approach?
Yes, it is possible, even if the full setup is not straightforward.
A couple of examples:
Xilinx released a white paper explaining how to run Linux + FreeRTOS on a dual-core Zynq ARM
Evidence explained how to run Linux + Erika Enterprise RTOS on a dual-core Freescale imx6 ARM
Those examples are based on system partitioning by hard-coding the assignment of the different cores to different OSs.
If your system is capable of hardware-assisted virtualization, you can use an hypervisor for making (and enforcing) such partitioning. You can for example use Siemen's Jailhouse, KVM or Xen.
Kind of. This is what people already do to some extent with network stack / driver. For example IsoStack idea works in a similar way. There's a project which actually implements this on linux by dedicating cores to network cards, but my google-fu is failing me.
I am thinking about running some graphics intensive programs on Windows Azure virtual machine, but not sure what kind of hardware do they have. Does all the VM have the same GPU? What is your experience of it?
The GPUs in Azure Virtual Machine are likely to be very basic and will most probably not have anywhere near the processing power you will need for carrying out intensive graphics manipulation. To my knowledge MS don't publish the details of the graphics hardware behind their Virtual Machines (If they actually use them at all?).
There's a question here on running WPF in an Azure cloud service which may be helpful.
Can Azure run WPF?
The N series Azure VMs support beefy GPUs. The NC series VM sports Tesla K80, with DDA (discreet device assignment) it supposed to provide close to bare metal performance. NV series VMs offer Tesla M60 with nVidia GRID.
More:
https://www.hpcwire.com/2015/09/29/microsoft-puts-gpu-boosters-on-azure-cloud/
https://blogs.technet.microsoft.com/hybridcloudbp/2016/12/13/n-series-azure-vms-with-gpu/
It's fascinating that there are FPGAs in Azure machines too (although not publicly accessible):
https://www.microsoft.com/en-us/research/project/project-catapult/