I'm using the yii framework and trying to get its unit tests running while connected over ssh on a CentOS server. When I run phpunit, it tries to launch Firefox, which fails with the error "no display specifiied"
General theory
Error: no display specified
To understand that error message you first have to understand how the X Windowing System works - that is the name of the framework used by Linux (and other types of Unix) systems used to display graphical user interfaces.
X consists of two parts - there is a client and a server. Client is the program that wants to draw the interface - in your case that would be Firefox. Server is a program that makes drawing possible. There are X servers available for all the major operating systems. Linuxes and OSX usually ship with one, on Windows you will have to find and install one - Cygwin/X is one option, but there are others.
So why is this client/server architecture even necessary?
Most of the time its not even needed. If you happen to run Linux locally, then you will not even notice that there is any kind of client/server communication happening somewhere - but there is.
Where X shines though is that this architecture means that network capabilities are built right into it. You can run a client (Firefox) on one machine and display the GUI on a completely different machine. You could run 10 different clients on 10 different machines and have them all display output on a single machine thanks to X. Think VNC or Remote Desktop - X is somewhat similar, but you could say that its on steroids compared to those. And X has had this ability for a really long time.
Whenever your start up a X client (a program that wants to display graphical user interface) it looks for an X server. One possibility for the client to find one is an environment variable named DISPLAY. I'm on OSX and this is what I see.
[~]> echo $DISPLAY
/tmp/launch-ihNtDq/org.x:0
This point to my local X server. It could point to any server on my local network. When the client finds this environment variable it will connect to it and the user interface pops up.
If client cannot find this environment variable - you will get the familiar
Error: no display specified
Back to Yii
Looks like Yii has Selenium tests bundled. PHPUnit needs to start up Selenium RC to control a Firefox instance to run those tests. Selenium RC (or maybe Firefox itself) fails to find DISPLAY environment variable. And dies with the above error.
How do you solve this issue?
There are 3 options
1) install Yii, PHPUnit and all their dependencies locally. Selenium runs just fine on Windows. It wont use the X protocol on Windows, so none of that business with X clients and X servers. And you can then run Yii testsuite locally.
2) install an X server on your Windows box. Then enable 'X Forwarding' in your ssh client settings (or use the -X command line parameter for ssh). When you do that, then there will be a DISPLAY variable set when you are logged in to that CentOS server. You can verify it by typing the echo command above. Then the X client on CentOS can talk (show GUI) to the X server on your Windows machine - all the X traffic is tunneled over the ssh connection. This however means that you need java (which selenium RC is built in) and Firefox on that CentOS server. You may or may not have them there.
3) use a virtual framebuffer - for example Xvfb - an X server that performs all drawing operations in memory, not showing any output anywhere.
What good is that? Selenium has commands for taking screenshots at any point during the test run and saving them to files. For example a typical Selenium test would check whether an element exist on the page - and to make a screenshot when it does not. Screenshot would then be saved in a file, which you can view later to determine what the reason for the failure was. Making screenshots works just fine with a virtual framebuffer.
Final clarification
Note that Selenium tests are merely one type of tests that PHPUnit can run. Selenium in not required to write PHPUnit tests, its an optional add-on. But Yii testsuite apparently relies upon it.
Last, but not least
Integration tests (which Selenium tests are) are not usually ran on production systems, because there is a chance that test data is left behind in your production database. Also, getting good test results means isolating from external factors as much as possible - the contents of your production database will be constantly changing and this may affect your tests.
Normally all tests will be executed somewhere else (your development machine, dedicated QA server, whatever you have), before the new code is deployed to production servers. After all the point of tests is to verify that the system works after the changes. There is not much value in running them on production systems - the code does not change after it has been deployed.
Of course - its up to you - if you see value in doing those tests on production system, go right ahead.
Its simpler than you think. Run Selenium locally on your desktop, make sure phpunit is setup on the remote server. Then start a reverse SSH tunnel in your SSH connection. This varies depending on your SSH client. In PuTTY, there is a setting for SSH tunnels and you can reverse the direction by selecting the remote option. Check out this page for details. With OpenSSH from the command line, its done like this:
ssh -R 4444:localhost:4444 user#remoteserver
This will listen on the remote server on port 4444 and forward it to your selenium server running on localhost on your desktop port 4444.
Once you've done that, you'll need to change the TEST_BASE_URL setting in yourproject/protected/tests/WebTestCase.php to go to the remote server's URL for your yii project.
The simplest way to run a gui test on from another agent machine on a Windows client is to use "psexec" (http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx).
Related
I have several scripts on my local machine. These scripts run install and configuration commands to setup my Elasticsearch nodes. I have 15 nodes coming and we definitely do not want to do that by hand.
For now, let's call them Script_A, Script_B, Script_C and Script_D.
Script_A will be the one to initiate the procces, it currently contains:
#!/bin/bash
read -p "Enter the hostname of the remote machine: " hostname
echo "Now connecting to $hostname!"
ssh root#$hostname
This works fine obviously and I can get into any server I need to. My confusion is running the other scripts remotely. I have read few other articles/SO questions but I'm just not understanding the methodology.
I will have a directory on my machine as follows:
Elasticsearch_Installation
|
|=> Scripts
|
|=> Script_A, Script_B, etc..
Can I run the Script_A, which remotes into the server, then come back to my local and run Script_B and so on within the remote server without moving the files over?
Please let me know if any of this needs to be clarified, I'm fairly new to the Linux environment in general.. much less running remote installs from scripts over the network.
Yes you can. Use ssh in non interactive mode, it will be like launching a command in your local environment.
ssh root#$hostname /remote/path/to/script
Nothing will be changed in your local system, you will be at the same point where you launched the ssh command.
NB: this command will ask you a password, if you want a really non interactive flow, set up host a passwordless login, like explained here
How to ssh to localhost without password?
You have a larger problem than just setting up many nodes: you have to be concerned with ongoing maintenance and administration of all those nodes, too. This is the space in which configuration management systems such as Puppet, Ansible, and others operate. But these have a learning curve to overcome, and they require some infrastructure of their own. You would probably benefit from one of them in the medium-to-long term, but if your new nodes are coming next week(ish) then you probably want a solution that you can use immediately to get up and going.
Certainly you can ssh into the server to run commands there, including non-interactively.
My confusion is running the other scripts remotely.
Of course, if you want to run your own scripts on the remote machine then they have to be present there, first. But this is not a major problem, for if you have ssh then you also have scp, the secure copy program. It can copy files to the remote machine, something like this:
#!/bin/bash
read -p "Enter the hostname of the remote machine: " hostname
echo "Now connecting to $hostname!"
scp Script_[ABCD] root#${hostname}:./
ssh root#hostname ./Script_A
I also manage Elasticsearch clusters with multiple nodes. A hack that works for me is using Terminator Terminal Emulator and split it into multiple windows/panes, one for each ES node. Then you can broadcast the commands you type in one window into all the windows.
This way, you run commands & view their results almost interactively across all nodes parallely. You could also save this layout of windows in Terminator, and then you can get this view quickly using a shortcut.
PS, this approach will only work of you have only small number of nodes & that too for small tasks only. The only thing that will scale with the number of nodes & the number of times and variety of tasks you need to perform will probably be a config management solution like Puppet or Salt.
Fabric is another interesting project that may be relevant to your use case.
I have a virtual machine on a cloud host that stays running in non-desktop mode. It has the Cinnamon desktop environment installed, but I don't typically leave the desktop environment open.
I have a cron job I'd like to run every hour which requires opening an app which requires a desktop environment (headful Chrome).
I've been able to schedule the cron job to use the desktop environment by adding the DISPLAY envvar in the job definition:
1 * * * * DISPLAY=:20 /path/to/script/to/execute
However, this only works when I use remote desktop software to open Cinnamon from my laptop.
I'd like to be able to leave the web server running, without having a remote desktop connection permanently open on my end, and run cron jobs that depend on a desktop environment.
Do y'all have a suggestion for running a cron job as I want to? Maybe a way to leave a desktop environment open without using a remote desktop? Or a way to open a desktop environment within a cron job?
If you don't have a display attached, you'll need to find a headless system that can run with a virtual DISPLAY. Headless Chrome is probably the easiest for you to swap in, and has convenient python bindings [related question].
You might also want to look into running selenium, depending on the task you need to run.
Finally, you might be interested in running a VNC server; depending on the config of your server, you can configure it to stay alive while your client disconnects. usually this is useful for temporary network hiccups, but set to a high enough timeout value, you may get the effects of a persistent, virtual, non-transient display.
Fairly new to cloud computing, so bear with me if question is obvious or silly. With tons of information available on internet, I was able to successfully create an ec2 linux instance and installed R and Rstudio on it. Ran my scripts on it which went really well but took too long (16 hrs) and very expensive as well since I require instances with high memory and vCPUs .
In my programs, I am essentially running the same scripts for different datasets.
My question is, is there any way I can run multiple similar instances of ec2 (with exactly same software installed and my scripts). So, this way, I will be able to run my scripts on every dataset on a separate instance simultaneously in less amount of time.
So what I have tried so far. I have created an AMI image of my existing instance and launched it. But I couldn't SSH it because of its weird username and ip address, something like "root#10.0.0.1". I can see both instances are running (original and the AMI image instance), I can SSH into original but not into the other one. I am able to login to the RStudio for original instance on port 8787.
Another question is how to launch this AMI imaged instance using SSH (Putty) in parallel with original instance. What problem will it cause if I use both of them in the browser (RStudio in this case) simultaneously?
Please help me with this!Thanks!
Problem: For a school project, I was running several machine learning algorithms on pretty large size data which happened to requre 30-35GB of memory and my PC couldn't handle it. I was using R/RStudio. So, I resorted to AWS for my memory limitation problem.
What I did initially: I created an ec2-instance, installed R/RStudio. Everything worked out perfectly and I was able to run my programs on RStudio through browser. I actually, ran my scripts on a very small dataset on this AWS instance to see how things are going. To much of my surprise it took very long for the whole script to run even with this small dataset. Soon enough, I realized that all these algorithms in my programs could be run independently for the same set of features with a little tweak in the scripts.
So, I decided to play with AWS little bit. I recreated the programs such that everything stayed the same except the learning algorithms in each script. In other words, I wanted to simultaneously run a copy of these programs with different algorithms so that I could get everything running simultaneously and produce the results in a smaller amount of time.
Now, my goal was to run multiple copies of this instance (original instance). And I should be able to run RStudio on my browser for each of these instances e.g. 5 ec2 instances will have 5 RStudio running concurrently on browser's different tabs. With that, I would be able to run all RStudio for each instance on my browser.
Then, I created an image (AMI) of this instance and then I created multiple instances from the AMI but I was missing out few points while creating those new instances from AMI which caused the problem I asked in the question above.
I initially suspected that it has something to do with port 8787 and I might not be able to run multiple RStudio for each ec2 instance in the browser. However, that was not the problem at all.
There are few very important things to take care of while you create the new instances from an AMI.
Mistake: While CREATING new instances from this AMI, I was NOT selecting two important things correctly i.e. VPC and Security Group.
Correct method is:
VPC -- On the "Configuration Instance Details" page:
a. Click the "Network" dropdown and select the VPC which was created for the original instance. (Original instance is the one which is used to create the AMI (image))
b. Click the "Auto-assign Public IP" dropdown and select Enable
Security Group -- On the "Configure Security Group" page:
a. for "Assign a security group" option, tick the "Select an existing security group" options
b. If there are more than one security groups in the list then select the one which was created for the original instance (OR create a new Security Group and make sure that it has the same kind of inbound and outbound port accesses)
Once I set this up, as Marc B mentioned in the comments, each instance gets its own IP address, and a local subnet address is assigned as well
IP address of instance looks like : ec2-33-444-22-111.us-west-1.compute.amazonaws.com
subnet looks like: 127.0.0.35
Now, after learning this, I recreated 5 instances from my AMI. So, now I had 5 instances with RStudio on each of them. All of them were running perfectly fine because I was able to SSH into each of them.
Now I thought I should be able to work with these instances in different tabs of the browser and run my scripts in them. But I wasn't able to login to all the RStudio instances in my browser tabs. Only one of them was working fine and the others were just not working in the browser. However, I was able to SSH into all of them from PuTTY. I could have ran my scripts from Linux (SSH) as well but I wanted to run them using RStudio.
After spending a good number of hours on this, I figured out the problem that the RStudio server needs to be started manually for each ec2 instance in the linux except the very first instance.
For one of the ec2-instance (besides the one which was working fine on browser), I did the following to start the RStudio server manually as below:
SSH using PuTTY
Become root: sudo su
Go to this path where RStudio was installed on my Linux instance: cd /usr/lib/rstudio-server/bin
start RServer with this command : rstudio-server start
Now go back to the browser, open another tab and use your ec2-instance address and port number (http://ec2-33-444-22-111.us-west-1.compute.amazonaws.com:8787). And now you should get the login page of RStudio for this instance as well.
Now, with a similar process, I had to manually run RStudio-servers for all other instance in order to be able to access them through the browser. Then I thought, if there is a way to start the RStudio server when Linux starts up every time. Then came up with a solution. To do this, I made a change in one of the configuration files of Linux as follows:
Become root: sudo su
go to this path: cd /etc/rc.d
vi the file rc.local and add the following command:
/usr/lib/rstudio-server/bin/rstudio-server start
save the changes you made.
close the SSH connection
Then, I went back to the AWS console, stopped this instance and created an AMI (image) of it. Now the above changes will be effective for each instance that I create from this AMI i.e. now RStudio server will be started as soon as the instance boots and will be accessible through the browser.
Now I can use multiple RStudio instances using different tabs of my browser. Make sure you are using the correct instance address in the browser. Port number stays same for all i.e. 8787
I have an iMac and when develop in Ruby on Rails I run everything on it: MySQL server, Redis service, ElasticSearch service, guard, and, of course, the proper rails server. By doing so, my computer runs pretty slow.
So I just bought a CPU and install linux in it, along with MySQL, Redis & ElasticSearch. Now I connect to that services from my iMac and it runs way faster.
However, Rspec/Guard still takes ages to load/run.
So, how do I make the linux server to take the hit and actually run this programs while I keep editing the code in my mac?
I know it may be a crude way and i haven't tested it
Place your project in Dropbox folder on your machine and have it open on the second computer
Run it on the quick machine. You can use following command to access it through ip address on the computers in same network
rails server -b 192.168.1.12 -p 8000
When saving code it should sync from one machine to another, may have slight delay for syncing
or set up Vagrant.
Thats the only ways I can see.
We have a set of Mac machines (mostly PPC) that are used for running Java applications for experiments. The applications consist of folders with a bunch of jar files, some documentation, and some shell scripts.
I'd like to be able to push out new version of our experiments to a directory on one Linux server, and then instruct the Macs to update their versions, or retrieve an entire new experiment if they don't yet have it.
../deployment/
../deployment/experiment1/
../deployment/experiment2/
and so on
I'd like to come up with a way to automate the update process. The Macs are not always on, and they have their IP addresses assigned by DHCP, so the server (which has a domain name) can't contact them directly. I imagine that I would need some sort of daemon running full-time on the Macs, pinging the server every minute or so, to find out whether some "experiments have been updated" announcement has been set.
Can anyone think of an efficient way to manage this? Solutions can involve either existing Mac applications, or shell scripts that I can write.
You might have some success with a simple Subversion setup; if you have the dev tools on your farm of Macs, then they'll already have Subversion installed.
Your script is as simple as running svn up on the deployment directory as often as you want and checking your changes in to the Subversion server from your machine. You can do this without any special setup on the server.
If you don't care about history and a version control system seems too "heavy", the traditional Unix tool for this is called rsync, and there's lots of information on its website.
Perhaps you're looking for a solution that doesn't involve any polling; in that case, maybe you could have a process that runs on each Mac and registers a local network Bonjour service; DNS-SD libraries are probably available for your language of choice, and it's a pretty simple matter to get a list of active machines in this case. I wrote this script in Ruby to find local machines running SSH:
#!/usr/bin/env ruby
require 'rubygems'
require 'dnssd'
handle = DNSSD.browse('_ssh._tcp') do |reply|
puts "#{reply.name}.#{reply.domain}"
end
sleep 1
handle.stop
You can use AppleScript remotely if you turn on Remote Events on the client machines. As an example, you can control programs like iTunes remotely.
I'd suggest that you put an update script on your remote machines (AppleScript or otherwise) and then use remote AppleScript to trigger running your update script as needed.
If you update often then Jim Puls idea is a great one. If you'd rather have direct control over when the machines start looking for an update then remote AppleScript is the simplest solution I can think of.