This is an issue we come up against again and again, how to get live website assets uploaded by a client into the local development environment. The options are:
Download all the things (can be a lengthy process and has to be repeated often)
Write some insane script that automates #1
Some uber clever thing which maps a local folder to a remote URL
I would really love to know how to achieve #3, have some kind of alias/folder in my local environment which is ignored by Git but means that when testing changes locally I will see client uploaded assets where they should be rather than broken images (and/or other things).
I do wonder if this might be possible using Panic Transmit and the 'Transmit disk' feature.
Update
Ok thus far I have managed to get a remote server folder mapped to my local machine as a drive (of sorts) using the 'Web Disk' option in cPanel and then the 'Connect to server' option in OS X.
However although I can then browse the folder contents in a safe read only fashion on my local machine when I alias that drive to a folder in /sites Apache just prompts me to download the alias file rather that follow it as a symlink might... :/
KISS: I'd go with #2.
I usally put a small script like update_assets.sh in the project's folder which uses rsync to download the files:
rsync --recursive --stats --progress -aze user#site.net:~/path/to/remote/files local/path
I wouldn't call that insane :) I prefer to have all the files locally so that I can work with them when I'm offline or on a slow connection.
rsync is quite fast and maybe you also want to check out the --delete flag to delete local files when they were removed from remote.
Related
I hope that I will not waste everyone's time, nor embarrass myself, but please hear/read my problem. I am new to this, so please bear with me.
Someone at work wrote a crude code in Node.js and I can see the .html files by having localhost: 8080 as the URL in the browser, while having the VisualStudio starting the npm with npm start command. Am I explaining this clear enough?
The webpages are displayed and all, but now comes the hurdle.
How can i have those pages served from a a Linux server?
If by analogy, I put some.html page inside the /var/www/ in a Apache server, pointing to the server's IP/somepage.html i can visualise it, what needs to be set up on a similar Node.js server?
Where do I have to put those files, inside what directory and what configuration is needed?
I thought to create a small LXC container and have those files and services saved as a template, but first I need to set this up correctly. Can Apache serve those files, do I have to make another configuration first?
I have those files served from a Windows machine from local host, and put the same files in a /node ,/opt/ www directory in a Linux machine, but no dice.
We have a scenario in which single server is running, which is getting data from the network span.
Every developer should work on their machine locally but the data to work on is only available in the server. how can I get the data to be replicated into each developers machine so that once they have completed development on their local machine, developers can push it to a GIT in the server.
PS: The network span data is constantly written to the server (data is in size of 100s of GB's).
What we have tried so far:
So we created a GIT server in the server we were getting the data on. But once a developer log in using his username then he creates a new branch in a directory. This works fine until another developer logs into the server with his username and switches to another branch in the same directory which will cause all the developers branch to the new one. which is not what we were expecting.
Probabily this question should go to https://serverfault.com/, but, anyway...
The git advantage is to have local and remotre repositories, so, in the server, you should have "only" the remote repositories, and they should be cloned in localmachines.
to work with that paradigm, or with the one you are asking for, you need a umask of 007 (depending on your distribution edit /etc/login.defs and change there)
You should have diferent groups for the diferent kind of shared projects, and a user to "own all the repositories", for example, git-adm ).
With all the prerequisites, you create with that user the base folder for all the repositories:
sudo -i
mkdir /srv/git
chown git-adm:gitgrp /srv/git
chmod g+s /srv/git
exit
The last line in the "sticky bite", wich allows to mantain the group (and avoid the problems you previously stated), so, in order to cerate a repository should be something like:
sudo su - git-adm
mkdir /srv/git/<group>/<repoName>.git
cd /srv/git/<group>/<repoName>.git
git init --bare
exit
And thats all: if the folder /srv/git/<group>/ we owned for a diferent group, then it'll keep the group.
I have a big problem and I can't understand this topic. I have a server with a website. I created a repository there with git init. Than I made a git add * to add all files from my server to the repository. Than I made a commit to commit all files to the repository.
Than I cloned it with git clone ssh://username#mysite.com/wordpress/.git to my local client.
All worked fine and I got a copy from my project. No I changed something on my local version and made a commit with a push. I looked in FileZilla but the content in the file don't changed. In the other direction when I changed something on the sever and pulled it to the local copy I saw the changes. Do you know why the changes which I made on the local copy are not visible on my sever?
Thank you for your help!
You need to push changes to a central repository that both your local machine and server can pull from (or add them as remotes for each other). A service such as GitHub works nicely for this. Here are instructions for a full workflow that works well for this. Updated instructions can be found in this gist. This workflow uses hooks to do the heavy lifting so that updates to your server are automated.
Using Git to Manage a Live Web Site
Overview
As a freelancer, I build a lot of web sites. That's a lot of code changes to track. Thankfully, a Git-enabled workflow with proper branching makes short work of project tracking. I can easily see development features in branches as well as a snapshot of the sites' production code. A nice addition to that workflow is that ability to use Git to push updates to any of the various sites I work on while committing changes.
You'll need to have Git installed on your development machines as well as on the server or servers where you wish to host your website. This process can even be adapted to work with multiple servers such as mirrors behind a load balancer.
Setting up Passwordless SSH Access
The process for updating a live web server relies on the use of post hooks within the Git environment. Since this is fully automated, there is no opportunity to enter login credentials while establishing the SSH connection to the remote server. To work around this, we are going to set up passwordless SSH access. To begin, you will need to SSH into your server.
ssh user#hostname
Next, you'll need to make sure you have a ~/.ssh in your user's home directory. If not, go ahead and create one now.
mkdir ~/.ssh
On Mac and Linux, you can harness the power of terminal to do both in one go.
if [ ! -d ~/.ssh ]; then mkdir ~/.ssh; fi
Next you'll need to generate a public SSH key if you don't already have one. List the files in your ~/.ssh directory to check.
ls -al ~/.ssh
The file you're looking for is usually named similarly to id_rsa.pub. If you're not sure, you can generate a new one. The command below will create an SSH key using the provided email as a label.
ssh-keygen -t rsa -b 4096 -C "your_email#example.com"
You'll probably want to keep all of the default settings. This will should create a file named id_rsa in the ~/.ssh directory created earlier.
When prompted, be sure to provide a secure SSH passphrase.
If you had to create an SSH key, you'll need to configure the ssh-agent program to use it.
ssh-add ~/.ssh/id_rsa
If you know what you are doing, you can use an existing SSH key in your ~/.ssh directory by providing the private key file to ssh-agent.
If you're still not sure what's going on, you should two files in your ~/.ssh directory that correspond to the private and public key files. Typically, the public key will be a file by the same name with a .pub extension added. An example would be a private key file named id_rsa and a public key file named id_rsa.pub.
Once you have generated an SSH key on your local machine, it's time to put the matching shared key file on the server.
ssh user#hostname 'cat >> ~/.ssh/authorized_keys' < ~/.ssh/id_rsa.pub
This will add your public key to the authorized keys on the remote server. This process can be repeated from each development machine to add as many authorized keys as necessary to the server. You'll know you did it correctly when you close your connection and reconnect without being prompted for a password.
Configuring the Remote Server Repository
The machine you intend to use as a live production server needs to have a Git repository that can write to an appropriate web-accessible directory. The Git metadata (the .git directory) does not need to be in a web-accessible location. Instead, it can be anywhere that is user-writeable by your SSH user.
Setting up a Bare Repository
In order to push files to your web server, you'll need to have a copy of your repository on your web server. You'll want to start by creating a bare repository to house your web site. The repository should be set up somewhere outside of your web root. We'll instruct Git where to put the actual files later. Once you decide on location for your repository, the following commands will create the bare repository.
mkdir mywebsite.git
cd mywebsite.git
git init --bare
A bare repository contains all of the Git metadata without any HEAD. Essentially, this means that your repository has a .git directory, but does not have any working files checked out. The next step is to create a Git hook that will check out those files any time you instruct it to.
If you wish to run git commands from the detached work tree, you'll need to set the environmental variable GIT_DIR to the path of mywebsite.git before running any commands.
Add a Post-Receive Hook
Create a file named post-receive in the hooks directory of your repository with the following contents.
#!/bin/sh
GIT_WORK_TREE=/path/to/webroot/of/mywebsite git checkout -f
Once you create your hook, go ahead and mark it as executable.
chmod +x hooks/post-receive
GIT_WORK_TREE allows you to instruct Git where the working directory should be for a repository. This allows you to keep the repository outside of the web root with a detached work tree in a web accessible location. Make sure the path you specify exists, Git will not create it for you.
Configuring the Local Development Machine
The local development machine will house the web site repository. Relevant files will be copied to the live server whenever you choose to push those changes. This means you should keep a working copy of the repository on your development machine. You could also employ the use of any centralized repository including cloud-based ones such as GitHub or BitBucket. Your workflow is entirely up to you. Since all changes are pushed from the local repository, this process is not affected by how you choose to handle your project.
Setting up the Working Repository
On your development machine, you should have a working Git repository. If not, you can create on in an existing project directory with the following commands.
git init
git add -A
git commit -m "Initial Commit"
Add a Remote Repository Pointing to the Web Server
Once you have a working repository, you'll need to add a remote pointing to the one you set up on your server.
git remote add live ssh://server1.example.com/home/user/mywebsite.git
Make sure the hostname and path you provide point to the server and repository you set up previously. Finally, it's time to push your current website to the live server for the first time.
git push live +master:refs/head/main
This command instructs Git to push the current main branch to the live remote. (There's no need to send any other branches.) In the future, the server will only check out from the main branch so you won't need to specify that explicitly every time.
Build Something Beautiful
Everything is ready to go. It's time to let the creative juices flow! Your workflow doesn't need to change at all. Whenever you are ready, pushing changes to the live web server is as simple as running the following command.
git push live
Setting receive.denycurrentbranch to "ignore" on the server eliminates a warning issued by recent versions of Git when you push an update to a checked-out branch on the server.
Additional Tips
Here are a few more tips and tricks that you may find useful when employing this style of workflow.
Pushing Changes to Multiple Servers
You may find the need to push to multiple servers. Perhaps you have multiple testing servers or your live site is mirrored across multiple servers behind a load balancer. In any case, pushing to multiple servers is as easy as adding more urls to the [remote "live"] section in .git/config.
[remote "live"]
url = ssh://server1.example.com/home/user/mywebsite.git
url = ssh://server2.example.com/home/user/mywebsite.git
Now issuing the command git push live will update all of the urls you've added at one time. Simple!
Ignoring Local Changes to Tracked Files
From time to time you'll find there are files you want to track in your repository but don't wish to have changed every time you update your website. A good example would be configuration files in your web site that have settings specific to the server the site is on. Pushing updates to your site would ordinarily overwrite these files with whatever version of the file lives on your development machine. Preventing this is easy. SSH into the remote server and navigate into the Git repository. Enter the following command, listing each file you wish to ignore.
git update-index --assume-unchanged <file...>
This instructs Git to ignore any changes to the specified files with any future checkouts. You can reverse this effect on one or more files any time you deem necessary.
git update-index --no-assume-unchanged <file...>
If you want to see a list of ignored files, that's easy too.
git ls-files -v | grep ^[a-z]
References
Deploy Your Website Changes Using Git
A simple Git deployment strategy for static sites
Using Git to manage a website
Ignoring Local Changes to Tracked Files in Git
pushing the code merely updates the remote repository's references.
It doesn't change the checked out working copy.
Consider that you could add a colleague's repository as a remote. If you pushed and the behaviour was that it would auto-checkout that new code, that would affect what they're working on.
It sounds like what you really want is a continuous integration tool, be it something full featured or merely an rsync triggered from a git hook.
you should only ever push to a bare repository (unless you know exactly what you are doing; and even then, you should only ever push to a bare repository).
you shouldn't clone a working copy's .git/ directory.
I've just set up my first cron-jon to run a stock script every night.
Running it manually works fine.
It's stored in /admin/stock_update.php
The command i'm running is /usr/bin/php -q /admin/stock_update.php
But I'm getting emails saying no input file is specified?
Any ideas?
Cheers
Network services almost never expose actual paths on the server's hard disk drive and even if they could it isn't a behaviour you can rely on. So the fact that your file is located at /admin/stock_update.php in the FTP server doesn't say much about actual location on disk, which is what local command-line utilities expect.
In PHP, you can find path on disk of current file with the __FILE__ magic constant. You can create a test script:
<?php
var_dump(__FILE__);
... upload it to the same FTP location and execute through the web server. If that's not an option because files in your FTP account in not visible from the web you can run the file from cron and check the email.
Do you have CloudLinux kernel installed on that server and CageFS filestyem? If yes try running this:
cagefsctl -w cpaneluser; cagefsctl -m cpaneluser
Then try running the cron again
I see several db.* files in my /home directory, and it seems they come from perforce. For example, some files are db.archmap, db.bodtext, db.change, db.changex
Are these files useful? Can I delete them? They are making my /home directory messy
You have started a server using your home directory as the Perforce server's P4ROOT folder. Those files are files that are generated from starting the server and cannot be deleted unless you want to hose your server installation. It's not clear to me how you've started the server instance, so I'll try and cover multiple bases with my answer.
If you want to start up the server under your own account, you should set the P4ROOT environment variable and point it to where you want the server to store its files. Alternatively, when you start the server, you can specify the root folder on the command line using the -r option:
p4d -r /home/mark/p4server
which would put the server's files into the directory called 'p4server' off of my home directory.
Typically it is best to run the perforce server using a user that is dedicated to running perforce. I use a user called 'perforce'. I set P4ROOT (and other variables) in that users environment. If you cannot use a separate user, it might be easier to use the -r command line option that I mentioned above.
Those files are only server files, not client files. So it is safe to delete them, but if you start the server back up it will recreate them. So you might want to uninstall the server.
Unless you are running a beta version, they have p4sandbox coming soon(maybe in the beta, I forget) which MAY create those files. I don't have a beta version, so I can't verify what new files the client may or may not create.
You can check the documentation here to see what these files do/are for.