I have a file in my root directory catalog.json. I use a bitbucket pipeline to perform this step:
- aws s3 cp catalog.json s3://testunzipping/ --recursive
However, I get an error that:
Skipping file /opt/atlassian/pipelines/agent/build/catalog.json/. File does not exist.
Why is it checking for the catalog.json file in this path? Why is not extracting the file from the root? How can I modify the command accordingly?
The problem is that you are trying to use --recursive for a single file. The command is trying to evaluate it as a directory because of the --recursive parameter. And it's ending up giving File does not exist error. I reproduced it by trying to copy a single file with --recursive parameter, I got the same error, but after removing it, it worked.
You can use this ;
aws s3 cp catalog.json s3://testunzipping/
Also for the answer of your other question;
Bitbucket Pipelines runners are using /opt/atlassian/pipelines/agent/build directory. This is actually pipelines' root directory. Runners are pulling code from the repo there and processing it according to your pipeline structure. It's a built-in situation.
Related
I have two gitlab repos. Repo A (contains application source code), Repo B(contains gitlab-ci.yml file, a shell script and settings.xml file).
My idea is to separate the gitlab-ci.yml pipeline, shell script and settings.xml file from the application source code repos for security reasons.
For that, to call pipeline from repo B, config changes I did in repo A is as follows.
In Settings-->CI/CD-->General Pipelines->CI/CD config file
I have mentioned: .gitlab-ci.yml#gitlabgroup/subgroup/repoB
As I need shell script and settings file also. in the pipeline file(.gitlab-ci.yml) I and using curl command to download the shell script and settings.xml file.
When pipeline file is triggered(push event to repo A), I am getting below errors:
ERROR: Job Failed: command terminated with exit code 1
This error comes at the time of executing shell script and uses of settings.xml for mvn command in the pipeline are encountered.
. ./shell_script.sh cmd_arg1 and mvn help:evaluate -Dexpression="project.name -q -DforceSdtout -s settings.xml"
When I place .gitlab-ci.yml, shell script and settings.xml to repo A it executes without any error.
whenever I m trying to call the pipeline from another repo, I am getting the error.
I have checked, shell script and settings.xml files are downloaded properly.
unable to identify the actual cause of the error.
kindly help in resolving this error.
Thank You.
I completed MyProject1 and have uploaded it in git in fine way with commits after adding each new features. And now I'm starting MyProject2 and was trying to add the URL for the remote repository. But then I found out that I had mistakenly added the URL in Documents instead of MyProject1 folder because of which MyProject2 folder is also being tracked in MyProject1. And I'm not being able to add URL to MyProject2 but instead facing merge issues.
Is there any way to detach the track from my Document folder without deleting any of my files from Documents.
Structure is this way(I'm using Linux):
Documents
(And inside Documents there is:)
MyProject1
MyProject2
And other folders which are also being tracked.
We can manually do it with below steps
create a file .gitignore in base directory.
If MyProject1 and MyProject2 are already part of git tracking, Please run commands
git rm -r --cached MyProject1/
git rm -r --cached MyProject2/
Open the file in text editor and add below lines in file
MyProject1/
MyProject2/
Best way:
Create a .gitignore file following #Vinayagam R
Ignore file locally
Those methods won't affect other contributors working on the same remote repository:
Use update-index:
If you want to stop tracking a file at certian point.
git update-index --assume-unchanged yourDirectoryName
--assume-unchanged is the flag which means the files should not change locally. In other words, it is used when ignore files that you do not need to change locally (or should not change).
To revert it use update-index --no-assume-unchanged yourDirectoryName
Using .exclude
In your working directory edit .git/info/exclude
I have the following line inside of a .sh file:
aws s3 cp s3://bucket/folder/ /home/ec2-user/ --recursive
When I run this line in the console, it runs fine and completes as expected. When I run this line inside of a .sh file, the line returns the following error.
Unknown options: --recursive
Here is my full .sh script.
echo "Test"
aws s3 cp s3://bucket/folder/ /home/ec2-user/ --recursive
echo "Test"
python /home/ec2-user/Run.py
If I manually add the Run.py file (not having it copied over from S3 as desired) and run the script I get the following output.
Test
Unknown options: --recursive
Test
Hello World
If I remove the files which are expected to be transferred by S3 out of the Linux environment and rely on the AWS S3 command I get the following output:
Test
Unknown options: --recursive
Test
python: can't open file '/home/ec2-user/Run.py': [Errno 2] No such file or directory
Note that there are multiple files that can be transferred inside of the specified S3 location. All of these files are transferred as expected when running the AWS command from the console.
I initially thought this was a line ending error, as I am copying the .sh file over from Windows. I have made sure that my line endings are working and we see this by the rest of the script running as expected. As a result, this issue seems isolated to the actual AWS command. If I remove the --recursive from the call, it will transfer over a single file successfully.
Any ideas on why the --recursive option would be working in the console but not in the .sh file?
P.s.
$ aws --version
aws-cli/2.1.15 Python/3.7.3 Linux/4.14.209-160.339.amzn2.x86_64 exe/x86_64.amzn.2 prompt/off
I think it is about the position of the option or you have somewhere the AWS CLI path which is pointing to older version because recursive option was added later. For me, it works both ways inside the shell script as well as the console.
aws s3 cp --recursive s3://bucket/folder/ /home/ec2-user/
$ aws --version
aws-cli/2.1.6 Python/3.7.4 Darwin/20.2.0 exe/x86_64 prompt/off
Try printing the version inside the shell script.
cp --recursive method lists source path and copies (overwrites) all to the destination path.
Plus instead of doing recursive use sync.sync recursively copies new and updated files from the source directory to the destination. Only creates folders in the destination if they contain one or more files.
aws s3 sync 3://bucket/folder/ /home/ec2-user/
sync method first lists both source and destination paths and copies only differences (name, size etc.).
Whats the proper way to form relative file path/ or file path for the runsettings parameter
<SettingsFile>xxxxxAutomation.testsettings</SettingsFile>
In my local environment this works but when i push it to build release agents it fails since
its looking for the file in a different directory which is my release folder
but instead its looking for this file in what appears to be in a folder directory.
This is the directory its looking at E:\Agent_work\r10\a\xxxxxxxxx.testsettings,
here is my current way i have it set
<SettingsFile>xxxxxAutomation.testsettings</SettingsFile>
here is the error message
Error: The test settings file E:\Agent_work\r10\a\xxxxxxxxxx.testsettings, specified in the MSTestAdapter settings, is not available. Either access to the file is denied or the file does not exist. Ensure that the test settings file is available and try again.
instead of
E:\Agent_work\r10\a_XXXXX CI Build\drop\XXXXX_Automation_Test\bin\Release\xxxxxxxxx.testsettings
There is no documentation that states how to formulate the file path on msdn and didn't see
anything online.
I suggest you can add a Copy Files task to copy the missing file from .../bin/release folder(Use the browse option) to default working directory:
I suggest you resolve this issue via modifying your pipeline instead of settings file, so that you can also work well locally with the unchanged runsettings file.
In addition: Clean Target Folder input should be disabled(unchecked) cause our target folder is the default working folder. Deleting all files in that folder would cause unexpected issues.
Recently I migrated from opensuse to centos and after that GIT has started to ignore my custom commit-msg hook. It simply doesn't execute it. (I checked it by add small piece of code to "add_ChangeId" function )
Hook generates Change-Id hash for every commit
GIT version: 1.8.1.2
File is located in following location: .git/hooks/
For debugging purposes I even have set 0777 permissions to the whole .git directory
Here is the full text of commit-msg file - http://pastebin.com/zmYNi0ED
timoras you are gold. Then I tried to execute script using sh .git/hooks/scriptname it worked, but when tried to call it using .git/hooks/scriptname and shell returned that I haven't permissions to execute it.
After that I looked at fstab, and found out that have forgot to add exec flag to the partition where this file was located.
Now everything works.
One more time thanks timoras!