I'm building a telegram bot using telepot(python) and a weird thing happened:
since the bot is supposed to keep running, I've set a
while 1:
time.sleep(.3)
at the end of my bot, right after I define how to handle the MessageLoop.
The bot has some print() to assert that the bot is setting up (or better, that it's setting up the message handler) and that it is reading the chat, waiting for any input.
The problem is that if I run the code without the
while 1:
time.sleep(.3)
it prints the messages and stops the execution (not having a loop to wait for new input), but if I add the while 1: (...) the code stops before being able to print anything.
Here's the code:
"""Launch the bot."""
import json
import telepot
from telepot.loop import MessageLoop
import time
from base import splitter
from base import handler
messageHandler = handler.Handler()
with open('triggers.json') as f:
triggers = json.load(f)
with open('config.json') as f:
config = json.load(f)
botKey = config['BOT_KEY']
# define the bot and the botname
bot = telepot.Bot(botKey)
botName = bot.getMe()['username']
# split commands in arrays ordered by priority
configSplitter = splitter.Splitter()
triggers = configSplitter.splitByPriority(triggers)
# define the handler
messageHandler.setBotname(botName)
messageHandler.setMessages(triggers)
# handle the messageLoop
print("setting up the bot...")
MessageLoop(bot, messageHandler.handle).run_as_thread()
print("Multibot is listening!")
# loop to catch all the messages
while 1:
time.sleep(.3)
Python version: 3.6 32-bit
The solution was to add sys.stdout.flush() after the various print()to restore the functionality of print(), while in order to make the code work again i had to change the telepot function
MessageLoop(bot, messageHandler.handle).run_as_thread()
which wasn't working properly to:
bot.message_loop(messageHandler.handle)
Related
Greetings!! I am trying using below watchdog module to monitor a shared path that works fine but I am not getting the idea to
generate outlook email alert, after the time span of 20 minutes if no
modification or update happening to the specified path. Below is the
code:
import os
import sys
import time
import logging
from watchdog.observers import Observer
from watchdog.events import LoggingEventHandler
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO,
format='%(asctime)s - %(message)s',
datefmt='%Y-%m-%d %H:%M:%S')
os.system('//fleet.ad/data/Data4/VMSSHARE/its/DOCS')
print("found")
# Defining your own path
path = "//bleet.ad/data/Data4/VMSSHARE/its/DOCS"
print("found")
# Initilaize logging event handler
event_handler = LoggingEventHandler()
# Initialize Observer
observer = Observer()
observer.schedule(event_handler, path, recursive=True)
# Start the observer
observer.start()
try:
while True:
# set the thread sleep time
time.sleep(1)
except KeyboardInterrupt:
observer.stop()
observer.join()
However, tried this piece of code to receive outlook email alerts, but not sure how to make it work with above script:
import os
import smtplib
import requests
EMAIL_ADDRESS = os.environ.get('USER_ID')
EMAIL_PASSWORD = os.environ.get('USER_PASSWORD')
r = requests.get("https://fleet.my.salesforce.com", timeout=5)
#if r.status_code!= 200:
with smtplib.SMTP('smtp.office365.com', 587) as smtp:
smtp.ehlo()
smtp.starttls()
smtp.ehlo()
smtp.login(EMAIL_ADDRESS, EMAIL_PASSWORD)
subject = 'ALARM: MAPILab is stuck from copying Public folders to destination'
body = 'Make sure server is restarted and it is backed up'
msg = f'Subject:{subject}\n\n{body}'
smtp.sendmail(EMAIL_ADDRESS, 'ryadav#elementcorp.com', msg)
Challenge for me is :
r = requests.get("https://fleet.my.salesforce.com", timeout=5)
Instead of website monitor , how should I ask to look for the code output?
You have to attach methods to your event handler, like this:
my_event_handler.on_created = on_created
my_event_handler.on_deleted = on_deleted
my_event_handler.on_modified = on_modified
my_event_handler.on_moved = on_moved
where the methods look like:
def on_created(event):
print(f"{event.src_path} has been created!")
as described in this blog post: http://thepythoncorner.com/dev/how-to-create-a-watchdog-in-python-to-look-for-filesystem-changes/
Have your handlers update a last_changed timestamp variable whenever there is a change. (Since all your handlers do the same thing regardless of what the change was, you could get away with just defining one handler method, maybe call it on_any_change and attach it to all 4 handler methods. Oh look, the watchdog docs show that there is an on_any_event method, so you could just hook into that.)
Then in your while True loop, if the current time - last_changed is greater than your threshold, send the email.
I wanted to run Telethon events in multi thread, multithreading.
For the following code, what I expected is multi thread, cocurrent independent events. await asyncio.sleep( 3 ) will sleep 3 seconds. So If you send any text on telegram to the account, it will wait 3 seconds and send me back "hi.". I expected that if I send a text for several times at once, let's say 5 times, it should wait for approximately 3 seconds and send me "hi" back at once, but not like wait for 3 seconds, send "hi.", wait for 3 seconds, send "hi." .. for 5 times and takes approximately 15 seconds. This is not what I expected for.
So, How can I modify the code to run the code as expected? Or it's impossible in telethon?
I read google, stackoverflow, github issue, but couldn't find a solution by myself. So I want your help here. Thank you.
import time
from pathlib import Path
import re
import random
import asyncio
from telethon import TelegramClient, events
from telethon.sessions import StringSession
from telethon import utils
import logging
logging.basicConfig(level=logging.ERROR)
api_id = ""
api_hash = ''
phone = ''
string = Path('./string').read_text()
message = "hi."
if __name__ == '__main__':
client = TelegramClient(StringSession(string), api_id, api_hash, sequential_updates=True)
#client.on(events.NewMessage(incoming=True))
async def handle_new_message(event):
print('message:', event.message.message)
# // not working here, the sleep blocks next events but not only current event.
print('wait for 3 seconds...')
await asyncio.sleep( 3 )
await event.respond(message)
print('Auto-replying...')
client.start(phone)
client.run_until_disconnected()
# client.loop.run_forever()
# string = StringSession.save(client.session)
# Path('./string').write_text(string)
You are using sequential_updates so until the first update is not completed it will not move to other update. I guess you need to turn off sequential updates.
I have snort logging DDOS alerts to file; I use Syslog-ng to parse the logs and output in json format into redis (wanted to set it up as a buffer, I use 'setex' command with expiry of 70 secs).
The whole thing seems not to be working well; any ideas to make it easier is welcome.
I wrote a simple python script to listen to redis KA events and count the number of snort alerts per second. I tried creating two other threads; one to retrieve the json-formatted alerts from snort and the second to count the alerts. The third is supposed to plot a graph using matplotlib.pyplot
#import time
from redis import StrictRedis as sr
import os
import json
import matplotlib.pyplot as plt
import threading as th
import time
redis = sr(host='localhost', port = 6379, decode_responses = True)
#file = open('/home/lucidvis/vis_app_py/log.json','w+')
# This function is still being worked on
def do_plot():
print('do_plot loop running')
while accumulated_data:
x_values = [int(x['time_count']) for x in accumulated_data]
y_values = [y['date'] for y in accumulated_data]
plt.title('Attacks Alerts per time period')
plt.xlabel('Time', fontsize=14)
plt.ylabel('Snort Alerts/sec')
plt.tick_params(axis='both', labelsize=14)
plt.plot(y_values,x_values, linewidth=5)
plt.show()
time.sleep(0.01)
def accumulator():
# first of, check the current json data and see if its 'sec' value is same
#that is the last in the accumulated data list
#if it is the same, increase time_count by one else pop that value
pointer_data = {}
print('accumulator loop running')
while True:
# pointer data is the current sec of json data used for comparison
#new_data is the latest json formatted alert received
# received_from_redis is a list declared in the main function
if received_from_redis:
new_data = received_from_redis.pop(0)
if not pointer_data:
pointer_data = new_data.copy()
print(">>", type(pointer_data), " >> ", pointer_data)
if pointer_data and pointer_data['sec']==new_data["sec"]
pointer_data['time_count'] +=1
elif pointer_data:
accumulated_data.append(pointer_data)
pointer_data = new_data.copy()
pointer_data.setdefault('time_count',1)
else:
time.sleep(0.01)
# main function creates the redis object and receives messages based on events
#this function calls two other functions and creates threads so they appear to run concurrently
def main():
p = redis.pubsub()
#
p.psubscribe('__keyspace#0__*')
print('Starting message loop')
while True:
try:
time.sleep(2)
message = p.get_message()
# Obtain the key from the redis emmitted event if the event is a set event
if message and message['data']=='set':
# the format emmited by redis is in a dict form
# the key is the value to the key 'channel'
# The key is in '__keyspace#0__*' form
# obtain the last field of the list returned by split function
key = message['channel'].split('__:')[-1]
data_redis = json.loads(redis.get(str(key)))
received_from_redis.append(data_redis)
except Exception e:
print(e)
continue
if __name__ == "__main__":
accumulated_data = []
received_from_redis = []
# main function creates the redis object and receives messages based on events
#this function calls two other functions and creates threads so they appear to run concurrently
thread_accumulator = th.Thread(target = accumulator, name ='accumulator')
do_plot_thread = th.Thread(target = do_plot, name ='do_plot')
while True:
thread_accumulator.start()
do_plot_thread.start()
main()
thread_accumulator.join()
do_plot_thread.join()
I currently do get errors per se ; I just cant tell if the threads are created or are working well. I need ideas to make things work better.
sample of the alert formated in json and obtained from redis below
{"victim_port":"","victim":"192.168.204.130","protocol":"ICMP","msg":"Ping_Flood_Attack_Detected","key":"1000","date":"06/01-09:26:13","attacker_port":"","attacker":"192.168.30.129","sec":"13"}
I'm not sure I understand exactly your scenario, but if you want to count events that are essentially log messages, you can probably do that within syslog-ng. Either as a Python destination (since you are already working in python), or maybe even without additional programming using the grouping-by parser.
I created a bot for my Discord server, that goes to the Reddit API for a given subreddit, and posts the Top 10 results for the Day in the Discord chat, based on the subreddit(s) that you input. It disregards self posts, and really only posts pictures and GIFs. The Discord message command would look something like this: =get funny awww news programming, posting the results for each subreddit as it gets them from the Reddit API (PRAW). THIS WORKS WITH NO PROBLEM. I know that the bot's ability to hit the API and post to discord works.
I added another command =getshuffled which puts all of the results from the subreddits in a large list, and then shuffles them before posting. This works really well with a request of up to ~50 subreddits.
This is what I need help with:
Because it can be such a large list of results, 1000+ results from 100+ subreddits, the bot is crashing on really big requests. Based on what help I got from my question yesterday, I understand what is going wrong. The bot is starting, it is talking to my Discord server, and when I pass it a long request, it stops talking to the server for too long while the Reddit API call is done, and it the Discord connection fails.
So, what I think I need to do, is have a subprocess for the code that goes to the Reddit API and pulls the results, (which I think will let the discord connection stay running), and then pass those results BACK to the bot when it is finished....
Or... this is something that Asyncio can handle on its own...
I'm having a hard time with the subprocess call, as I knew I would.
Basically, I either need help with this subprocess trickery, or need to know if I'm being an idiot and Asyncio can handle all of this for me. I think this is just one of those "I don't know what I don't know" instances.
So to recap: The bot worked fine with smaller amounts of subreddits being shuffled. It goes through the args sent (which are subreddits), grabbing info for each post, and then shuffling before posting the links to discord. The problem is when it is a larger set of subreddits of ~ 50+. In order to get it to work with the larger amount, I need to have the Reddit call NOT block the main discord connection, and that's why I'm trying to make a subprocess.
Python version is 3.6 and Discord.py version is 0.16.12
This bot is hosted and running on PythonAnywhere
Code:
from redditBot_auth import reddit
import discord
import asyncio
from discord.ext.commands import Bot
#from discord.ext import commands
import platform
import subprocess
import ast
client = Bot(description="Pulls posts from Reddit", command_prefix="=", pm_help = False)
#client.event
async def on_ready():
return await client.change_presence(game=discord.Game(name='Getting The Dank Memes'))
def is_number(s):
try:
int(s)
return True
except:
pass
def show_title(s):
try:
if s == 'TITLES':
return True
except:
pass
async def main_loop(*args, shuffled=False):
print(type(args))
q=10
#This takes a integer value argument from the input string.
#It sets the number variable,
#Then deletes the number from the arguments list.
title = False
for item in args:
if is_number(item):
q = item
q = int(q)
if q > 15:
q=15
args = [x for x in args if not is_number(x)]
if show_title(item):
title = True
args = [x for x in args if not show_title(x)]
number_of_posts = q * len(args)
results=[]
TESTING = False #If this is turned to True, the subreddit of each post will be posted. Will use defined list of results
if shuffled == False: #If they don't want it shuffled
for item in args:
#get subreddit results
#post links into Discord as it gets them
#The code for this works
else: #if they do want it shuffled
output = subprocess.run(["python3.6", "get_reddit.py", "*args"])
results = ast.literal_eval(output.decode("ascii"))
# ^^ this is me trying to get the results back from the other process.
. This is my get_reddit.py file:
#THIS CODE WORKS, JUST NEED TO CALL THE FUNCTION AND RETURN RESULTS
#TO THE MAIN_LOOP FUNCTION
from redditBot_auth import reddit
import random
def is_number(s):
try:
int(s)
return True
except:
pass
def show_title(s):
try:
if s == 'TITLES':
return True
except:
pass
async def get_results(*args, shuffled=False):
q=10
#This takes a integer value argument from the input string.
#It sets the number variable,
#Then deletes the number from the arguments list.
title = False
for item in args:
if is_number(item):
q = item
q = int(q)
if q > 15:
q=15
args = [x for x in args if not is_number(x)]
if show_title(item):
title = True
args = [x for x in args if not show_title(x)]
results=[]
TESTING = False #If this is turned to True, the subreddit of each post will be posted. Will use defined list of results.
NoGrabResults = False
#This pulls the data and creates a list of links for the bot to post
if NoGrabResults == False:
for item in args:
try:
#get the posts
#put them in results list
except Exception as e:
#handle error
pass
try:
#print('____SHUFFLED___')
random.shuffle(results)
random.shuffle(results)
random.shuffle(results)
except:
#error stuff
print(results)
#I should be able to read that print statement for the results,
#and then use that in the main bot function to post the results.
.
#client.command()
async def get(*args, brief="say '=get' followed by a list of subreddits", description="To get the 10 Top posts from a subreddit, say '=get' followed by a list of subreddits:\n'=get funny news pubg'\n would get the top 10 posts for today for each subreddit and post to the chat."):
#sr = '+'.join(args)
await main_loop(*args)
#THIS POSTS THE POSTS RANDOMLY
#client.command()
async def getshuffled(*args, brief="say '=getshuffled' followed by a list of subreddits", description="Does the same thing as =get, but grabs ALL of the posts and shuffles them, before posting."):
await main_loop(*args, shuffled=True)
client.run('my ID')
UPDATE: Following advice, I had the command passed through a ThreadPoolExecutor as shown:
async def main(*args, shuffled):
if shuffled==True:
with concurrent.futures.ThreadPoolExecutor() as pool:
results = await asyncio.AbstractEventLoop().run_in_executor(
executor=pool, func=await main_loop(*args, shuffled=True))
print('custom thread pool', results)
but this still results in errors when the script tries to talk to Discord:
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<Client._run_event() running at /home/GageBrk/.local/lib/python3.6/site-packages/discord/client.py:307> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7f28acd8db28>()]>>
Event loop is closed
Destination must be Channel, PrivateChannel, User, or Object. Received NoneType
Destination must be Channel, PrivateChannel, User, or Object. Received NoneType
Destination must be Channel, PrivateChannel, User, or Object. Received NoneType
...
It is sending the results correctly, but discord is still losing connection.
praw relies on the requests library, which is synchronous meaning that the code is blocking. This can cause your bot to freeze if the blocking code takes too long to execute.
To get around this, a separate thread can be created that handles the blocking code. Below is an example of this. Note how blocking_function will use time.sleep to block for 10 minutes (600 seconds). This should be more than enough to freeze and eventually crash the bot. However, since the function is in it's own thread using run_in_executor, the bot continues to operate as normal.
New versions
import time
import asyncio
from discord.ext import commands
from concurrent.futures import ThreadPoolExecutor
def blocking_function():
print('entering blocking function')
time.sleep(600)
print('sleep has been completed')
return 'Pong'
client = commands.Bot(command_prefix='!')
#client.event
async def on_ready():
print('client ready')
#client.command()
async def ping(ctx):
loop = asyncio.get_event_loop()
block_return = await loop.run_in_executor(ThreadPoolExecutor(), blocking_function)
await ctx.send(block_return)
client.run('token')
Older async version
import time
import asyncio
from discord.ext import commands
from concurrent.futures import ThreadPoolExecutor
def blocking_function():
print('entering blocking function')
time.sleep(600)
print('sleep has been completed')
return 'Pong'
client = commands.Bot(command_prefix='!')
#client.event
async def on_ready():
print('client ready')
#client.command()
async def ping():
loop = asyncio.get_event_loop()
block_return = await loop.run_in_executor(ThreadPoolExecutor(), blocking_function)
await client.say(block_return)
client.run('token')
I have a serial device (Arduino) regularly outputting log data, which shold be written in a Log file. Also the device takes spontaneous commands over serial. I send the commands to a Raspberry over Telegram, which are handled and sent to the arduino by Telepot, which runs in a separate thread.
How can I make sure that the two processes get along with each other?
I am a complete Beginner in Multithreading.
Here is a shortened version of my Code:
import time
import datetime
import telepot
import os
import serial
from time import sleep
ser = None
bot = None
def log(data):
with open('logfile', 'w') as f:
file.write("Timestamp" + data)
#The handle Function is called by the telepot thread,
#whenever a message is received from Telegram
def handle(msg):
chat_id = msg['chat']['id']
command = msg['text']
print( 'Command Received: %s' % command)
if command = '/start':
bot.sendMessage(chat_id, 'welcome')
elif command == 'close_door':
#This serial write could possibly happen while a
#ser.readline() is executed, which would crash my program.
ser.write("Close Door")
elif command == 'LOG':
#Here i should make sure that nothing
#is waiting from the Arduino
#so that the next two Serial lines are the Arduinos
#respoonce to the "LOG" command.
#and that hanlde is the only
#function talking to the Serial port now.
ser.write("LOG")
response = ser.readline()
response += "\0000000A" + ser.readline()
#The Arduinos response is now saved as one string
#and sent to the User.
bot.sendMessage(chat_id, response)
print("Command Processed.")
bot = telepot.Bot('BOT TOKEN')
bot.message_loop(handle)
ser = serial.Serial("Arduino Serial Port", 9600)
print( 'I am listening ...')
while True:
#anything to make it not run at full speed (Recommendations welcome)
#The log updates are only once an hour.
sleep(10)
#here i need to make sure it does not collide with the other thread.
while ser.in_waiting > 0:
data = ser.readline()
log(data)
This code is not my actual code, but it should represent exactly what I'm trying to do.
My last resort would be to put the serial code in the threads loop function, But this would require me to change the libary which would be ugly.
I looked up some stuff about Queues in Asincio, and locking functions. However i don't really understand how to apply that. Also I don't use the async telepot.
After reading more on locking and threads, I found an answer with help of the links provided in this Question: Locking a method in Python?
It was often recommended to use Queues, however I don't know how.
My solution (code may have errors, but the principle works)
import time
import random
import datetime
import telepot
import os
import serial
from time import sleep
#we need to import the Lock from threading
from threading import Lock
ser = None
bot = None
def log(data):
with open('logfile', 'w') as f:
file.write("Timestamp" + data)
#create a lock:
ser_lock = Lock()
#The handle Function is called by the telepot thread,
#whenever a message is received from Telegram
def handle(msg):
#let the handle function use the same lock:
global ser_lock
chat_id = msg['chat']['id']
command = msg['text']
print( 'Command Received: %s' % command)
if command == '/start':
bot.sendMessage(chat_id, 'welcome')
elif command == 'close_door':
#This serial write could possibly happen while a
#ser.readline() is executed, which would crash my program.
with ser_lock:
ser.write("Close Door")
elif command == 'LOG':
#Here i should make sure that nothing
#is waiting from the Arduino
#so that the next two Serial lines are the Arduinos
#respoonce to the "LOG" command.
#and that hanlde is the only
#function talking to the Serial port now.
#the lock will only be open when no other thread is using the port.
#This thread will wait untill it's open.
with ser_lock:
while ser.in_waiting > 0:
data = ser.readline()
log(data)
#Should there be any old data, just write it to a file
#now i can safely execute serial writes and reads.
ser.write("LOG")
response = ser.readline()
response += "\0000000A" + ser.readline()
#The Arduinos response is now saved as one string
#and sent to the User.
bot.sendMessage(chat_id, response)
print("Command Processed.")
bot = telepot.Bot('BOT TOKEN')
bot.message_loop(handle)
ser = serial.Serial("Arduino Serial Port", 9600)
print( 'I am listening ...')
while True:
#anything to make it not run at full speed (Recommendations welcome)
#The log updates are only once a
sleep(10)
#here i need to make sure it does not collide with the other thread.
with ser_lock:
while ser.in_waiting > 0:
data = ser.readline()
log(data)