Version: Python 3.7 (Spyder) -- OS: Windows10 -- System: Core i5 (6th gen) + 16gb RAM
I wrote a program which handles a lot of data. The following structure is used to accomplish this:
Program Description
A GUI interface is used as a main function (class). In here an interface pops up, asks the user for input, uses this input the make all kinds of calculation, which are specified in different functions.
The first function is an import function, where in a specified (by user) folder is searched for all .wav files and where the data of these are imported. All imported items are appended (numpy.append) to one big array.
The big array (for 20 files about 2.000.000.000 datapoints) is used to calculated characteristics of the sound file. The reason it has so many datapoints is because the samplerate of the .wav file is set on 78125 samples/s, which I need for accurate calculations.
After the calculations, 2 plots are generated in a specified folder and 2 csv's are also stored in that folder with the requested data.
Problem Statement
Running the main function (program) in the spyder environment, works totally fine. It take about 10 minutes to go through all the data and generates the output.
Compiling the function to an .exe using PyInstaller, works fine, no errors, everything dependency imported. However when running the program a MemoryError pops up almost immediatly (see image below).
Image: error message from command line when executing the exe file
Tried solutions
Running the python script via CLI, gives the same error
Running the .exe program with only 2 files to import, works all file, but incredibly slow (much slower than executed via spyder)
Questions
Why has spyder enough memory to process all data with no problems, but when executing the .py via command line or when executing the .exe file, there is always a memory error?
Why does the .exe or the .py via CL run slower than in the spyder IDE?
Goal
This program should be able to process noise data on every laptop in the company (also 8gb ram sometimes). So I want to find a way to let the program allocate all available RAM on the used machine.
Thanks in advance, for any help!
Meanwhile I have found the answer to my question thanks to Axe319:
The Spyder IDE was running on a 64bit version of python, making the program run smoothly and without any issues. Nevertheless my python interpreter was still a 32bit version of python.
Steps taken to solve the problem:
uninstall python 32bit version
install python 64bit version
install all used packages again using pip install -packages-
install PyInstaller again using pip install pyinstaller
Compile the program to .exe using PyInstaller
After this all seems to be working!
Related
How do I solve this?
TypeError: As of 3.10, the *loop* parameter was removed from Lock() since it is no longer necessary
I'm trying to use Binance socket manager, and I'm getting this error.
Should just be a case of upgrading your websockets version from 9.1 to 10.x
pip install --upgrade websockets
I've had the same issue. My bot ran fine on MacOS, but it popped up when I installed Fedora on the Apple instead. Never resolved it before moving on to other OS's, but I don't know if it would have happened on Ubuntu or Zorin, because a PIP problem stopped me long before then.
As for my primary, an MSI gaming laptop running Windows 11, I never had the issue on the command line python, IDLE, PyCharm, Visual Studio, nor Visual Studio Code, UNTIL this morning when my laptop overheated and shutdown. When I booted up again, the system no longer recognized the modules I had been using (pandas, pytz, python-binance) and they had to be installed again (from an elevated command line, which seemed odd). Then when running the program from VS, there comes the error again. Command prompt returns the same error, however, IDLE runs the program without issue. I'm not knowledgeable enough to say how to directly fix the bug, or even why it's happening, but it seems that there may be methods of skirting it. The error reads 'As of 3.10...' so if you cannot find an application that can run it, you might try rolling it back to 3.9. Sorry I can't be of any real aid, here. Hope you find your answers. I'll keep looking, too.
I've come up with several solutions.
I created my own ticker:
play = client.get_symbol_ticker(symbol='BTCUSDT)
def start_ticker():
global play
while True:
play = client.get_symbol_ticker(symbol='BTCUSDT')
print(play['Price'])
time.sleep(1)
bsm = ThreadedWebsocketManager()
bsm.start()
start_ticker()
Now, this is just a sort of preliminary example. I've tied it into my actual trading loop and removed the print function, but store and process the data second by second. I run multiple tokens simultaneously and set the sleep at the end of the entire loop, after the condition evaluations have been processed. You can tweak the rest time after testing the duration of your loop, but overall it's hasn't ever shown to be critical for it to be off by fragments of a second. One caveat is that it only delivers the flat price, but you can check the documentation for additional queries you can pull from: Python Binance 0.2.0 Websockets Documentation
Install Python 3.9:
This is easiest on Windows, as no system processes rely on it. If you install it parallel to your current version, you'll have to take extra steps to address it rather than the later version, such as with PATH edits or virtual environments. An easy tool for this is Anaconda, which can create the virtual environment with little fuss.
I run my trader on a PC running Fedora, which has proven to be more reliable with server connections (unfortunately, Windows 11 can't keep proper time without a looping PowerShell script that manually resyncs, and I get Windows semaphore errors even with the time issue fixed). However, Fedora relies on up-to-date Python for some system functions, so you have to install the pre-3.10 version beside it an create a symbolic link and a virtual environment to run it.
Modify the python-binance module to use a different Loop function, which I believe can be done with PyCharm or Anacondas, but from what I read it's not the best of ideas and I don't see a need for it at the moment. Also, I would probably just break it.
TypeError: As of 3.10, the *loop* parameter was removed from Queue() since it is no longer necessary
I was getting this error when i try to use proxybroker package.
I just downgrade python version to 3.6.8 and now error is gone.
Maybe your error to occurred by python version.
maybe helps
Make sure u installed asyncio and imported that. I had the same problem.
a small part of my code is dedicated to unzipping a zip file and looking for certain things in text files. The problem I am facing right now is that python processes the text files in a certain order in my system but it seems to use another order when the code is run on my professor's system. That order of processing is critical for what the code is trying to do. It processes the files in the correct order in my system but doesn't on his, consequently resulting in wildly different outputs. My OS is open SUSE 15.0,his is open SUSE 15.1 he runs python 3.6.9, I run 3.6.5.
I tried downloading everything from github and running the code on my end but still cannot reproduce the issue he is having.
For what its worth, here is the list of modules I am using.
os,errno,sys,logging,shutil,argparse,pandas, zipfile
What are some possible reasons for this kind of discrepancy?
I'm trying to run a program (http://dar.linux.free.fr/) that I built myself using Cygwin for Windows/32. I'm now trying to run it using a more recent (not by much, just a few months) version of Cygwin. Instead of running, it spits out:
2 [main] dar (xxxx) shared_info::initialize: size of shared memory region changed from 37742 to 38776
I presume this is due to attempting to run using incompatible dynamic libraries. A curious thing is that if I run it from a regular (ie non-Cygwin) then it runs fine. It also runs if I type
cmd /S dar
although in the latter case I get some warnings about there being no terminal and hence no user interaction.
OK so maybe I should just rebuild it using the current version of Cygwin. But this is a slow and painful process, and not one I'd like to have to go through every time I upgrade Cygwin. And the fact that the code runs fine outside the Cygwin environment gives me hope that I can make it work.
I tried changing the PATH in the Cygwin environment to force it to pick up all the Cygwin dynamic libraries that were bundled with the dar executable (ie the same ones that are used when running outside Cygwin). But I had no luck there.
Can anyone suggest a simple solution or work-around?
Many thanks!
I'm midway through a VHDL class and have been able to play relatively nice with the ISE and Digilent toolchain in Linux... until trying to reflash a PicoBlaze program. For details, I am currently running and targeting,
Fedora 21 64-bit (3.19.3-200.fc21.x86_64)
Nexys2 development board from Digilent (with a Spartan3)
Xilinx ISE 14.7
Adept 2.16.1 Runtime
Adept 2.2.1 Utilities
I've been able to run ISE and program the Nexys2 bit files with iMPACT just fine so far in Linux, but this current project is to write an assembly program for the PicoBlaze soft core processor, compile and update the memory of the running vector without having to resynthesize any VHDL.
Using the steps from Kris Chaplin's post, I can compile a PSM to HEX and then convert that HEX file to an SVF in dosbox. From here I can use Digilent's Adept tool in Windows to program a top_level.bit file which has the PicoBlaze already synthesized, I could also do this in ISE's iMPACT in Linux. After the design is running, I can use Adept to program the SVF file into the running memory of the design and everything is peachy. However, trying to load the SVF into iMPACT in Linux throws an exception,
EXCEPTION:iMPACT:SVFYacc.c:208:1.10 - Data mismatch.
The only issue I've found online with that error shows that there should be an '#' symbol that needs to be removed, but I haven't seen any '#'s anywhere in the SVF.
I also tried to convert the SVF to XSVF. iMPACT doesn't throw an error loading the XSVF, but programming/executing the XSVF freezes the design instead of running the new program.
Adept doesn't have a comparable GUI in Linux that I've seen, just a cmd line tool 'djtgcfg'. Just like iMPACT, I've been able to program the toplevel.bit file fine with
$ djtgcfg prog -d Nexys2 -i 0 -f ../../toplevel.bit
but attempting to program the svf file with the same call doesn't seem to affect anything. It says it should take a few minutes and immediately reports "Programming succeeded" but I don't see any change on the device.
I'd really like to keep my environment all in Linux if I can, I don't have quite enough room on my laptop to juggle between two VMs.
Is it possible to use use iMPACT to write an SVF file to the Nexus2? Or can/should I be using the Adept utility differently?
Has anyone gotten this to work? Thanks a ton!
There are many better ways to reconfigure the PicoBlaze InstructionROM without resynthesizing:
use Xilinx's data2mem tool
This toll is shipped with ISE and can patch BlockRAM contents in bit-files
=> requires FPGA reprogramming
use PicoBlaze's embedded JTAGLoader6
Enable the embedded JTAGLoader6 design in the template file. Use JTAG_Loader_RH_64 binary or JTAG_Loader_Win7_64.exe to upload a hex-file via JTAG into the PicoBlaze ROM.
=> reconfigure ROM at runtime, no FPGA reprogramming needed
The manual from Ken Chapman offers several pages on how to use JTAG_Loader. Additionally, have a look into the PicoBlaze discussions at forums.xilinx.com. There are some discussions regarding bugs and issues around JTAG_Loader and how to solve them.
Also have a look into opbasm from Kevin Thibedeau as an alternative and improved PicoBlaze assembler. It is also shipped with an ROM patch tool.
I know it's a little bit late for the original poster, but I suspect I am taking the same class and I believe I have found a solution to upload picoblaze code on linux.
Download the KCPSM3 zip file from Xilinx IP Download, extract the contents and move the executables from the JTAG_loader folder to your working directory.
In dosbox run hex2svfsetup.exe for the nexys2 board select menu options 4 - 0 - 1 - 8
Use the assembler to create the .hex file
In dosbox run hex2svf.exe to create the svf file
Then run svf2xsvf.exe -d -i < input.svf > -o < output.xsvf >
The contrary to the JTAG_Loader_quick_guide.pdf in the initial zip file use impact and open the xsvf file and program using the xsvf file.
I am working on a MATLAB project which we want to export as .exe. The resulting file must then be able to run on both 32 and 64-bits Windows 7 PCs.
After a little research we realized this problem was easier to approach by developing on a 32-bit version of MATLAB building then a 32-bits .exe file.
Till this point, all our development was being carried in the 64-bits version of MATLAB. With it we had been able to successfully generate and run 64-bits .exe versions.
Now that we switched to MATLAB 32-bits, however, and we generate the .exe, something goes wrong and the following error is shown:
Undefined function ‘fmincon’ for input arguments of type ‘function handle’.
This is the line of code in which fmincon first appears:
Options = optimoptions('fmincon', 'DiffMinChange', 10);
A few remarks:
The same scripts which worked on MATLAB 64-bits also work on MATLAB
32-bits. Within the MATLAB environment, everything runs smoothly.
The scripts (with the same exact code) can still be made executable on MATLAB 64-bits without any problem.
In both cases, we properly installed the runtime required for the MATLAB executable to be run on the PC.
We have tried to run the 32-bits .exe in both 64-bits and 32-bits machines with the same result.
Is it possible that the 32-bits version of MATLAB's deployed executable has problems dealing with functions from the Optimization Toolbox (as fmincon is)?
What else could be the cause of this problem? Does anyone have an idea how to fix it?
The problem was only solved thanks to MATLAB's support. This is related to a bug in version R2014a, explained and patched in this Mathworks link.