I am trying to run an XMLRPC server and an XMLRPC client on Mininet hosts, using the script below.
from mininet.topo import Topo
from mininet.net import Mininet
from mininet.node import OVSController
class MyTopo(Topo):
def __init__(self):
# Initialize topology
Topo.__init__(self)
# Add hosts
server1 = self.addHost('server1')
server2 = self.addHost('server2')
# Add switch
s1 = self.addSwitch('s1')
# Add links
self.addLink(server1, s1)
self.addLink(server2, s1)
if __name__ == '__main__':
net = Mininet(topo=MyTopo(), controller=OVSController)
net.start()
print(net.hosts[0].cmd('python3 xmlrpc_server.py'))
print(net.hosts[1].cmd('python3 xmlrpc_client.py'))
The file xmlrpc_server.py is:
from xmlrpc.server import SimpleXMLRPCServer
import threading
def is_even(n):
return n%2 == 0
server = SimpleXMLRPCServer(("0.0.0.0", 8000), logRequests=True, allow_none = True)
server.register_function(is_even, "is_even")
print("Listening on port 8000...")
server_thread = threading.Thread(target=server.serve_forever)
server_thread.start()
The file xmlrpc_client.py is:
import xmlrpc.client
proxy = xmlrpc.client.ServerProxy("http://10.0.0.1:8000/")
print("3 is even: %s" % str(proxy.is_even(3)))
print("100 is even: %s" % str(proxy.is_even(100)))
The problem is that although I have used a thread, when I run the xmlrpc_server.py script on server1, the execution pauses at line server_thread.start() waiting for the script execution to be completed before moving on and thus never goes on to the next line, which means that the XMLRPC client script never runs. How do I overcome this problem?
P.S.: xmlrpc_server.py and xmlrpc_client.py can be executed through the server terminals (by using the commands xterm server1 and xterm server2 on Mininet CLI and then using the commands python3 xmlrpc_server.py and python3 xmlrpc_client.py on the xterm terminals that open), but I need to start the server and client through a python script so as to perform some further calculations after the communication between the two servers.
Replace print(net.hosts[0].cmd('python3 xmlrpc_server.py')) with print(net.hosts[0].sendCmd('python3 xmlrpc_server.py')). Connection is sometimes refused, but that issue can be resolved with exception handling on the client script.
Related
I've tried every method in the title to run it in the background but here are the issues i got trying to use them:
pythonw and pyw: server doesn't work, going to localhost:8000 error with ERR_EMPTY_RESPONSE.
& arg and START/B: doesn't start script in background and instead output server log
so now I'm short on ideas on how to run this script in the background.
Using pythonw, it should help to explicitly redirect stdout and stderr to a file – maybe this behavior is somehow related to the problem described here (although this seems to be specific to Python 2.7). Not capturing the output by redirecting it to os.devnull seems to work as well.
The following script produces a minimum working server example with pythonw for me (using Python 3.7.9):
import http.server
import os
import sys
if __name__ == "__main__":
sys.stdout = sys.stderr = open(os.devnull, "w")
httpd = http.server.HTTPServer(("localhost", 8000), http.server.SimpleHTTPRequestHandler)
httpd.serve_forever()
I am able to execute a command inside the server but I want to get that output in my local pc. I don't want want to use ssh keys. I want to use just the host's username and password as well as vm's(server's) username and password. I couldn't find a direct method to get the shell output of server in my pc, but this method seems half done like generate and save the output in server and then get the file from server. I am finding it difficult to get file(sample.txt) from server to local host(pc).
[EDIT: I am able to do this using via via method(Store the output to server and then get it back into local pc, is there a direct method ?) ]
from pyVim import connect
from config import *
from pyVmomi import vim, vmodl
import ssl
service_instance = connect.SmartConnect(host="yyyyyyy", port=some_number,user="xxx" , pwd=pwd,sslContext=ssl._create_unverified_context())
content = service_instance.RetrieveContent()
vm = searcher.FindByIp(ip="zzzzzz", vmSearch=True)
creds = vim.vm.guest.NamePasswordAuthentication(username='root', password=vmpwd)
pm = service_instance.content.guestOperationsManager.processManager
#checks python version and stores in sample.txt in server
ps = vim.vm.guest.ProcessManager.ProgramSpec(programPath='/usr/bin/python', arguments='--version &> sample.txt')
res = pm.StartProgramInGuest(vm, creds, ps)
print(res) #Prints pid
This does the job but I would appreciate if someone knows how to directly get the output of shell command from server to my local pc. This code makes a file with the output of cmd inside the server and it gets copied into my local pc
from pyVim import connect
from config import *
from pyVmomi import vim, vmodl
import ssl
import os
import requests
service_instance = connect.SmartConnect(host="xxxx", port=aaa,user="yyy" , pwd=pwd,sslContext=ssl._create_unverified_context())
content = service_instance.RetrieveContent()
# # Find a VM
vm = searcher.FindByIp(ip="aaaa", vmSearch=True)
creds = vim.vm.guest.NamePasswordAuthentication(username='root', password=vmpwd)
pm = service_instance.content.guestOperationsManager.processManager
#executes and saves sample.txt into server
ps = vim.vm.guest.ProcessManager.ProgramSpec(programPath='/usr/bin/python', arguments='--version &> sample.txt')
res = pm.StartProgramInGuest(vm, creds, ps)
dest="/Users/username/Desktop/vcenter/sample.txt" #My local pc
src="/root/sample.txt" #Server's directory
fti = content.guestOperationsManager.fileManager.InitiateFileTransferFromGuest(vm, creds, src)
resp=requests.get(fti.url, verify=False)
#Writes into file
with open(dest, 'wb') as f:
f.write(resp.content)
I have the following problem.
I'd like to run tests on the local flask server before deploying to production. I use pytest for that. My conftest.py looks like that for the moment:
import pytest
from toolbox import Toolbox
import subprocess
def pytest_addoption(parser):
"""Add option to pass --testenv=local to pytest cli command"""
parser.addoption(
"--testenv", action="store", default="exodemo", help="my option: type1 or type2"
)
#pytest.fixture(scope="module")
def testenv(request):
return request.config.getoption("--testenv")
#pytest.fixture(scope="module")
def testurl(testenv):
if testenv == 'local':
return 'http://localhost:5000/'
else:
return 'https://api.domain.com/'
This allows me to test the production api by typing the command pytest and to test a local flask server by typing pytest --testenv=local
This code WORKS flawlessly.
My problem is that I have to manually instantiate the local flask server from the terminal each time I want to test locally like this:
source ../pypyenv/bin/activate
python ../app.py
Now I wanted to add a fixture that initiates a terminal in the background at the beginning of the tests and closes the server down after having finished testing. After a lot of research and testing, I still cannot get it to work. This is the line I added to the conftest.py:
#pytest.fixture(scope="module", autouse=True)
def spinup(testenv):
if testenv == 'local':
cmd = ['../pypyenv/bin/python', '../app.py']
p = subprocess.Popen(cmd, shell=True)
yield
p.terminate()
else:
pass
The errors I get are from the requests package that says that there is no connection/ refused.
E requests.exceptions.ConnectionError:
HTTPConnectionPool(host='localhost', port=5000): Max retries exceeded
with url: /login (Caused by
NewConnectionError(': Failed to establish a new connection:
[Errno 111] Connection refused',))
/usr/lib/python3/dist-packages/requests/adapters.py:437:
ConnectionError
This means for me that the flask server under app.py is not online. Any suggestions? I am open to more elegant alternatives
For local testing the Flask test_client is a more elegant solution. See the docs on Testing. You can create a fixture for the test_client and create test requests with that:
#pytest.fixture
def app():
app = create_app()
yield app
# teardown here
#pytest.fixture
def client(app):
return app.test_client()
And use it like this:
def test_can_login(client):
response = client.post('/login', data={username='username', password='password'})
assert response.status_code == 200
If the only problem are the manual steps, maybe consider a bash script that does your manual setup for you and after that executes pytest.
I am using the following for this purpose so that testing configuration is also preserved in the test server
#pytest.fixture(scope="session")
def app():
db_fd, db_path = tempfile.mkstemp()
app = create_app({
'TESTING': True,
'DATABASE': db_path
})
yield app
os.close(db_fd)
os.unlink(db_path)
from flask import request
def shutdown_server():
func = request.environ.get('werkzeug.server.shutdown')
if func is None:
raise RuntimeError('Not running with the Werkzeug Server')
func()
#pytest.fixture
def server(app):
#app.route('/shutdown',methods=('POST',))
def shutdown():
shutdown_server()
return 'Shutting down server ...'
import threading
t = threading.Thread(target=app.run)
yield t.start()
import requests
requests.post('http://localhost:5000/shutdown')
References
https://flask.palletsprojects.com/en/1.1.x/tutorial/tests/
How do I terminate a flask app that's running as a service?
How to stop flask application without using ctrl-c
With a bash script (thanks #ekuusela) I now finally succeeded in what I wanted.
I added a fixture that calls the bashscript spinserver.sh in a new terminal window. This works in ubuntu, the command is different in different environments (see Execute terminal command from python in new terminal window? for other environments).
#pytest.fixture(scope="session", autouse=True)
def client(testenv):
if testenv != 'local':
pass
else:
p = subprocess.Popen(['gnome-terminal', '-x', './spinserver.sh'])
time.sleep(3)
yield
Here the very simple bashscript
#!/bin/bash
cd ..
source pypyenv/bin/activate
python app.py
The sleep command is necessary because the server takes some time to
initialize.
Don't forget to make your bash script executable (chmod
u+x spinserver.sh)
I tried to do a teardown after yield, but p.kill does not really
close the window. This is acceptable for me as it does not matter
if I have to manually close a terminal window & I can even see
flask debugging if necessary
I've searched a lot for this and have not yet found a definitive solution. The closest thing I've found is this:
import shutil
from os.path import join
import os
import time
import sys
minecraft_dir = ('server diectory')
world_dir = ('server world driectory')
def server_command(cmd):
os.system('screen -S -X stuff "{}\015"'.format(cmd))
on = "1"
while True:
command=input()
command=command.lower()
if on == "1":
if command==("start"):
os.chdir(minecraft_dir)
os.system('"C:\Program Files\Java\jre1.8.0_111\bin\java.exe" -Xms4G -Xmx4G -jar craftbukkit-1.10.2.jar nogui java')
print("Server started.")
on = "0"
else:
server_command(command)
When I launch this program and type 'start' the CMD flashes up and closes instantly. Instead I want the CMD to stay open with the minecraft sever running from it. I'm not sure why this happens or what the problem is, any help would be greatly appreciated.
p.s. I have edited this to my needs (such as removing a backup script that was unnecessary) but it didn't work before. The original link is: https://github.com/tschuy/minecraft-server-control
os.system will simply run the command then return to your python script with no way to further communicate with it.
On the other hand using subprocess.Popen gives you access to the process while it runs, including writing to it's .stdin which is how you send data to the server:
def server_command(cmd):
process.stdin.write(cmd+"\n") #just write the command to the input stream
process = None
executable = '"C:\Program Files\Java\jre1.8.0_111\bin\java.exe" -Xms4G -Xmx4G -jar craftbukkit-1.10.2.jar nogui java'
while True:
command=input()
command=command.lower()
if process is not None:
if command==("start"):
os.chdir(minecraft_dir)
process = subprocess.Popen(executable, stdin=subprocess.PIPE)
print("Server started.")
else:
server_command(command)
you can also pass stdout=subprocess.PIPE so you can also read it's output and stderr=subprocess.PIPE to read from it's error stream (if any)
As well instead of process.stdin.write(cmd+"\n") you could also use the file optional parameter of the print function, so this:
print(cmd, file=process.stdin)
Will write the data to process.stdin formatted in the same way that print normally does, like ending with newline for you unless passing end= to override it etc.
Both of the above answers do not work in the environment I tried them in.
I think the best way is to use RCON, not sending keys to a window.
RCON is the protocol used by games to run commands.
Many python libraries support Minecraft RCON, and the default server.properties file has an option for RCON.
We will use the python module: MCRON.
Install it. It works for windows, mac, linux.
Type:
pip install mcrcon
Lets configure your server to allow RCON.
In server.properties, find the line 'enable-rcon' and make it look like this:
enable-rcon=true
Restart and stop your server.
Find the line 'rcon.password' and set it to any password you will remember.
You can leave the port default at 25575.
Now, open your terminal and type:
mcron localhost
Or your server ip.
You will be prompted to enter the password you set.
Then you can run commands and will get the result.
But we are doing this with python, not the PYPI MCRON scripts - so do this.
from mcrcon import MCRcon as r
with r('localhost', 'insertyourpasswordhere') as mcr:
resp = mcr.command('/list')
print(resp) #there are 0/20 players online: - This will be different for you.
I am spinning up a Waitress web server in a background thread to run functional tests. How it is possible to clean up and quit waitress at the end of test run in clean(ish) manner? The public waitress API gives only one way entry point which expects KeyboardInterrupt as the quit signal.
Currently I just run the server in a daemon thread and all new web servers are pending clean up until the test runner quits.
My test web server code:
"""py.test fixtures for spinning up a WSGI server for functional test run."""
import threading
import time
from pyramid.router import Router
from waitress import serve
from urllib.parse import urlparse
import pytest
from backports import typing
#: The URL where WSGI server is run from where Selenium browser loads the pages
HOST_BASE = "http://localhost:8521"
class ServerThread(threading.Thread):
"""Run WSGI server on a background thread.
This thread starts a web server for a given WSGI application. Then the Selenium WebDriver can connect to this web server, like to any web server, for running functional tests.
"""
def __init__(self, app:Router, hostbase:str=HOST_BASE):
threading.Thread.__init__(self)
self.app = app
self.srv = None
self.daemon = True
self.hostbase = hostbase
def run(self):
"""Start WSGI server on a background to listen to incoming."""
parts = urlparse(self.hostbase)
domain, port = parts.netloc.split(":")
try:
# TODO: replace this with create_server call, so we can quit this later
serve(self.app, host='127.0.0.1', port=int(port))
except Exception as e:
# We are a background thread so we have problems to interrupt tests in the case of error. Try spit out something to the console.
import traceback
traceback.print_exc()
def quit(self):
"""Stop test webserver."""
# waitress has no quit
# if self.srv:
# self.srv.shutdown()
Webtest provides a WSGI server named StopableWSGIServer that gets started in a separate thread, and then can be shutdown() when you are done running your tests.
Check out: https://docs.pylonsproject.org/projects/webtest/en/latest/api.html#module-webtest.http
According to the documentation it was specifically built for use with casperjs or selenium.