Is there a specific format of storing environment variables in .bash_profile? - python-3.x

Following is the python code I am using to send messages to slack. The following code throws an error when I try to get the api_key from environment, but it works perfectly fine when I replace webhook with the actual API key.
import requests
import json
import os
data = {
"text" : "hi there"
}
webhook = os.environ.get("SLACK_API_KEY")
requests.post(webhook, json.dumps(data))
The SLACK_API_KEY is an environment variable which I have stored in .bash_profile folder of my system. The API key has the following format:
https://hooks.slack.com/services/alpha_numeric/alpha_numeric/alpha_numeric
This is how I have stored the API key in my .bash_profile folder:
export SLACK_API_KEY="https://hooks.slack.com/services/alpha_numeric/alpha_numeric/alpha_numeric"
This is the error when I try to export the api_key from environment.
Traceback (most recent call last):
File "/Users/nikhilsawal/OneDrive/investment_portfolio/helper_functions.py", line 10, in <module>
requests.post(webhook, json.dumps(data))
File "/Users/nikhilsawal/OneDrive/investment_portfolio/track_proj/lib/python3.8/site-packages/requests/api.py", line 119, in post
return request('post', url, data=data, json=json, **kwargs)
File "/Users/nikhilsawal/OneDrive/investment_portfolio/track_proj/lib/python3.8/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/Users/nikhilsawal/OneDrive/investment_portfolio/track_proj/lib/python3.8/site-packages/requests/sessions.py", line 516, in request
prep = self.prepare_request(req)
File "/Users/nikhilsawal/OneDrive/investment_portfolio/track_proj/lib/python3.8/site-packages/requests/sessions.py", line 449, in prepare_request
p.prepare(
File "/Users/nikhilsawal/OneDrive/investment_portfolio/track_proj/lib/python3.8/site-packages/requests/models.py", line 314, in prepare
self.prepare_url(url, params)
File "/Users/nikhilsawal/OneDrive/investment_portfolio/track_proj/lib/python3.8/site-packages/requests/models.py", line 388, in prepare_url
raise MissingSchema(error)
requests.exceptions.MissingSchema: Invalid URL 'None': No schema supplied. Perhaps you meant http://None?
[Finished in 0.149s]

Your env var is it stored correctly, to check your environment variable you could do :echo $SLACK_API_KEY in your shell or you could do env and grep after it.
Pay attention that ~/.bash_profile if for user-specific settings and actions (login shells only) alternatively there is .bashrc which is executed for interactive non-login shells.
I suggest to put your variable in ~/.profile and then don't forget to source ~/.profile or wherever you decide to place you env variable .
If they do not exists you can create them, I recommend this post which explains the differences.
Also you can set the var it directly from the script:
os.environ["SLACK_API_KEY"] = "value"
Did a small test with ~/.bash_profile for your case:

Related

Why does the execution of a Python script fail on running it with a batch file with the Windows Task Scheduler which uses vir env?

I have a Python script on my Windows machine which has to be run in a virtual environment in order to satisfy package dependencies.
I have created a batch file to use it with Task Scheduler which looks like the following:
call activate vir_env
python "C:\Users\xxx\Documents\Anaconda\envs\vir_env\Scripts\script.py"
call conda deactivate
pause
set /p id="Press enter when finished"
This batch file executes successfully when I run it by manually double clicking on it. But it is failing after a period of execution when I schedule it for execution by the Task Scheduler. (I am saving this batch file on my desktop and pointing the scheduler to read it from there.)
I also see the following line on cmd while running it manually.
DevTools listening on ws://127.0.0.1:61347/devtools/browser/d86ec8f2-7af2-4a2b-89f4-6c6f7025cc02
But I get the following notification if I schedule and run it with Task Scheduler:
DevTools listening on ws://127.0.0.1:55329/devtools/browser/e8cd5010-6b41-4d35-a465-78a75e87a547
That can be seen in the error output as posted below.
DevTools listening on ws://127.0.0.1:55329/devtools/browser/e8cd5010-6b41-4d35-a465-78a75e87a547
Traceback (most recent call last):
File "C:\Users\xxx\Documents\Anaconda\envs\vir_env\Scripts\script.py", line 432, in <module>
gv.save(deps, buffer, fmt='png')
File "C:\Users\xxx\Documents\Anaconda\envs\vir_env\lib\site-packages\holoviews\util\__init_.py", line 820, in save
return renderer_obj.save(obj, filename, fmt=fmt, resources=resources,
File "C:\Users\xxx\Documents\Anaconda\envs\vir_env\lib\site-packages\holoviews\plotting\renderer.py", line 627, in save
rendered = self_or_cls(plot, fmt)
File "C:\Users\xxx\Documents\Anaconda\envs\vir_env\lib\site-packages\holoviews\plotting\renderer.py", line 201 in __call__
data = self._figure_data(plot, fmt, **kwargs)
File "C:\Users\xxx\Documents\Anaconda\envs\vir_env\lib\site-packages\holoviews\plotting\bokeh\renderer.py", line 131, in _figure_data
img = get_screenshot_as_png(plot.state, driver=state.webdriver)
File "C:\Users\xxx\Documents\Anaconda\envs\vir_env\lib\site-packages\bokeh\io\export.py", line 223, in get_screenshot_as_png
web_driver.maximize_window()
File "C:\Users\xxx\Documents\Anaconda\envs\vir_env\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 737, in maximize_window
self.execute(command, params)
File "C:\Users\xxx\Documents\Anaconda\envs\vir_env\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\xxx\Documents\Anaconda\envs\vir_env\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.comnon.exceptions.NoSuchWindowException: Message: Browsing context has been discarded
How can this be possible?
The dev tool link seems to be different in the two cases.
Could this be the potential cause of the issue?
I had the same problem but found a solution without using task scheduler
Create a file and name it eg Example.bat
Edit the file and paste this code
set Path_To_File=C:\Path\To\File
set SunDay_Time=11:34
set MonDay_Time=13:20
set TuesDay_Time=07:30
set WednesDay_Time=02:57
set ThursDay_Time=11:42
set Friday_Time=23:59
set Saturday_Time=23:59
#echo off
:loop
timeout 10
for /f %%i in ('powershell ^(get-date^).DayOfWeek') do set day=%%i
SET "HH=%TIME:~0,2%
SET "MM=%TIME:~3,2%
if %day%==Sunday set Time_Run=%SunDay_Time%
if %day%==Monday set Time_Run=%MonDay_Time%
if %day%==Tuesday set Time_Run=%TuesDay_Time%
if %day%==Wednesday set Time_Run=%WednesDay_Time%
if %day%==Thursday set Time_Run=%ThursDay_Time%
if %day%==Friday set Time_Run=%Friday_Time%
if %day%==Saturday set Time_Run=%Saturday_Time%
if "%HH%:%MM%" EQU "%Time_Run%" start "%Path_To_File%"
goto loop
Edit this section
Modify the path name of the program you want to run at the specified time, and modify the time you want to run the program through this section, and then save the file
set Path_To_File=C:\Path\To\File
set SunDay_Time=11:34
set MonDay_Time=13:20
set TuesDay_Time=07:30
set WednesDay_Time=02:57
set ThursDay_Time=11:42
set Friday_Time=23:59
set Saturday_Time=23:59
Press Windows and R and type shell:startup
Create a file and name it example.vbs
Edit the file and paste this code
Set WshShell = CreateObject("WScript.Shell")
WshShell.Run chr(34) & "C:\Path\To\File.bat" & Chr(34), 0
Set WshShell = Nothing
Replace C:\Path\To\File.bat with the path of the batch file and save the file

python script in cron not reading a CSV unless it creates the CSV itself

I have the following script. It works when I run it in command line, and it works when I run it in cron.
The variable 'apath' is the absolute path of the file.
cat=['a','a','a','a','a','b','b','b','b','b']
val=[1,2,3,4,5,6,7,8,9,10]
columns=['cat','val']
data=[cat,val]
dict={key:value for key,value in zip(columns,data)}
statedata_raw=pd.DataFrame(data=dict)
statedata_raw.to_csv(apath+'state_data.csv',index=False)
statedata_raw2=pd.read_csv(apath+'state_data.csv')
statedata_raw2.to_csv(apath+'state_data2.csv',index=False)
But when I try to run the first part manually, creating the first csv, and then run the second part through cron, the second read_csv statement fails. I checked the permissions on the state_data.csv file and they are fine. It's set to -rwxr-xr-x
To be specific: I first run this script manually through command line. It executes and creates state_data.csv. Then I check the permissions of state_csv, and they are -rwxr-xr-x
cat=['a','a','a','a','a','b','b','b','b','b']
val=[1,2,3,4,5,6,7,8,9,10]
columns=['cat','val']
data=[cat,val]
dict={key:value for key,value in zip(columns,data)}
statedata_raw=pd.DataFrame(data=dict)
statedata_raw.to_csv(apath+'state_data.csv',index=False)
and then this script via cron, which fails, and gives the error message below
statedata_raw2=pd.read_csv(apath+'state_data.csv')
statedata_raw2.to_csv(apath+'state_data2.csv',index=False)
This is the error that I get from the system
Traceback (most recent call last):
File "/users/michaelmader/wdtest.py", line 39, in <module>
statedata_raw2=pd.read_csv(apath+'state_data.csv')
File "/opt/miniconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 676, in parser_f
return _read(filepath_or_buffer, kwds)
File "/opt/miniconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 448, in _read
parser = TextFileReader(fp_or_buf, **kwds)
File "/opt/miniconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 880, in __init__
self._make_engine(self.engine)
File "/opt/miniconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 1114, in _make_engine
self._engine = CParserWrapper(self.f, **self.options)
File "/opt/miniconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 1891, in __init__
self._reader = parsers.TextReader(src, **kwds)
File "pandas/_libs/parsers.pyx", line 374, in pandas._libs.parsers.TextReader.__cinit__
File "pandas/_libs/parsers.pyx", line 678, in pandas._libs.parsers.TextReader._setup_parser_source
OSError: Initializing from file failed
To summarize
Run complete script through Terminal: state_data2.csv is created: pass
Run complete script through cron: state_data2.csv is created: pass
Run first part through Terminal, second part through cron: fail
I am on MacOS and I already gave crontab full disk access in system preferences.
I figured out the problem. The issue was the permissions that were granted to cron in MacOS. I thought I had solved it by giving \usr\bin\crontab full disk access, but I actually needed to give full disk access to usr\sbin\cron
The steps for doing this can be found here: https://blog.bejarano.io/fixing-cron-jobs-in-mojave/.
Once I made that change everything worked fine.

Indexing audio to get timestamps for each word using python

I have an audio file and I want to get the timestamps for each word. I want to know during which time period was each the words spoken.
(For example if an audio file says "I am a doctor" I want to know during which instant "I" was said, "am" was said and so on)
I want to do this using python.
I have tried the following code.
from SimpleAudioIndexer import SimpleAudioIndexer as sai
indexer = sai(mode="ibm", src_dir="D:/Codes/Python/audio recognition",
username_ibm="", password_ibm="")
indexer.index_audio(basename = "target.wav")
indexer.save_indexed_audio("{}/indexed_audio".format(indexer.src_dir))
indexer.load_indexed_audio("{}/indexed_audio.txt".format(indexer.src_dir))
print(indexer.get_timestamps())
However I am running into following error.
Traceback (most recent call last):
File "D:\Codes\Python\audio recognition\rec.py", line 5, in <module>
indexer.index_audio(basename = "target.wav")
File "C:\Users\Awais\AppData\Roaming\Python\Python37\site-
packages\SimpleAudioIndexer\__init__.py", line 1108, in index_audio
self._index_audio_ibm(*args, **kwargs)
File "C:\Users\Awais\AppData\Roaming\Python\Python37\site-
packages\SimpleAudioIndexer\__init__.py", line 928, in _index_audio_ibm
replace_already_indexed=replace_already_indexed)
File "C:\Users\Awais\AppData\Roaming\Python\Python37\site-
packages\SimpleAudioIndexer\__init__.py", line 730, in _prepare_audio
self._filtering_step(basename)
File "C:\Users\Awais\AppData\Roaming\Python\Python37\site-
packages\SimpleAudioIndexer\__init__.py", line 638, in _filtering_step
universal_newlines=True).communicate()
File "C:\Program Files (x86)\Microsoft Visual
Studio\Shared\Python37_64\lib\subprocess.py", line 800, in __init__
restore_signals, start_new_session)
File "C:\Program Files (x86)\Microsoft Visual
Studio\Shared\Python37_64\lib\subprocess.py", line 1207, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
From what I can tell from your code/errors is that you're running Windows, but you're using forward slashes "/" in your src_dir string. This won't work as Windows uses the backslash "\" when navigating folders. Such as:
C:\Windows
while many Unix based systems use the forward slash like this:
/home/Awais
In your code you should try to change the "/" to "\" and see if that makes a difference.
If it is your intention on using this program on both Windows and Unix systems, you should use python3's "pathlib" to ensure your directory paths work on either OS. Here is a link that you can use to learn more about handling paths in python3:
https://medium.com/#ageitgey/python-3-quick-tip-the-easy-way-to-deal-with-file-paths-on-windows-mac-and-linux-11a072b58d5f

AWS CLI - All commands return Unknown output type: [None]

All of my aws-cli commands returned
Unknown output type: [None]
I checked my configuration
$ aws configure
appeared normal but i was unable to edit my 'Default output format'
I ran my aws-cli command with --debug and saw
MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/awscli/clidriver.py", line 208, in main
return command_table[parsed_args.command](remaining, parsed_args)
File "/Library/Python/2.7/site-packages/awscli/clidriver.py", line 345, in __call__
return command_table[parsed_args.operation](remaining, parsed_globals)
File "/Library/Python/2.7/site-packages/awscli/clidriver.py", line 517, in __call__
call_parameters, parsed_globals)
File "/Library/Python/2.7/site-packages/awscli/clidriver.py", line 638, in invoke
self._display_response(operation_name, response, parsed_globals)
File "/Library/Python/2.7/site-packages/awscli/clidriver.py", line 657, in _display_response
formatter = get_formatter(output, parsed_globals)
File "/Library/Python/2.7/site-packages/awscli/formatter.py", line 272, in get_formatter
raise ValueError("Unknown output type: %s" % format_type)
ValueError: Unknown output type: [None]
$aws configure
press Enters
see the "Default output format [None]:"
input one of "json, text or table "(all in lower case)
after that rerun your command.
Run the command "aws configure", then check if the word "JSON" in "Default output format [JSON]:" is in upper case or lower case? If it's in upper case then running any aws command shows "Unknown ouput type : JSON".
Or alternately open the file C:\Users\<user>\.aws\config file and check the entry "output = json". If the word json is in upper case then running any aws command shows "Unknown ouput type : JSON".
Solution:
Replace the upper case JSON with lower case json.
My ~/.aws/config was somehow in a bad state, there were multiple declarations for the same setting under a single role header. Editing the file manually fixed my issue.
The info under Configuration Settings and Precedence https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html led me to the right place.
Figure out your ARN and your role and set up everything (region and clustername) as in the below command in the respective place.
aws eks update-kubeconfig --name eks-cluster-name --region aws-region --role-arn arn:aws:iam::XXXXXXXXXXXX:role/testrole
Also, read on the documentation here below.
https://aws.amazon.com/premiumsupport/knowledge-center/eks-api-server-unauthorized-error/

Python3: configparser KeyError when run as Cronjob

i have a simple python3 Script that works when I'm running it from console:
import configparser
file = 'config_test.ini'
config = configparser.ConfigParser()
config.read(file)
for key in config['test_section']: print(key)
the called ini-file looks like this:
[test_section]
testkey1 = 5
testkey2 = 42878180
testkey3 = WR50MS10:1100012307
testkey4 = WR50MS04:1100012010
testkex5 = 192.168.200.168
and the script runs fine and returns the five keys of the ini-file.
No I configured it as a cronjob every minute (running on rasbian on Raspberry Pi) via:
* * * * * python3 /home/pi/s0/testconfig.py >> /tmp/cron_debug_log.log 2>&1
and the log looks like this:
Traceback (most recent call last):
File "/home/pi/s0/testconfig.py", line 7, in <module>
for key in config['test_section']: print(key)
File "/usr/lib/python3.2/configparser.py", line 941, in __getitem__
raise KeyError(key)
KeyError: 'test_section'
Does anyone have an idea what I did wrong
Kind regards
When running from cron, your script runs in a different working directory. This means the filename it is trying to open is relative to a different directory.
You should use the absolute path to the config file in your script.
A common way of doing it in python, assuming the config file resides in the same directory as the script you're running, is using the __file__ builtin:
config_file = os.path.join(os.path.dirname(__file__), 'config_test.ini')

Resources