GitLab submodule complains about AWS CloudFormation templates located in submodule - gitlab

Question

I am facing a problem with GitLab include and I'm wondering whether it's possible to do what I intent to.
I have 2 GitLab repositories:
my-infrastructure
my-prod-deployment
In my-infrastructure I have the YML file that defines a job with a command referencing a local file. For example in my-infrastructure I have the following:
A gitlab-infrastructure.yml template
image: amazon/aws-cli
variables:
FOO: ReplaceMe
stages:
- one
agent_ui:
stage: microservices
script:
- aws cloudformation deploy --stack-name sample --template-file templates/aws-template.yml
and also I have a templates/aws-template.yml that has some cloud formation code.
Notice that the GitLab template needs access to a local file that exists in the same project my-infrastructure
Now in the other project my-prod-deployment I have a .gitlab-ci.yml with
include:
- project: mycompany/my-infrastructure
ref: main
file: gitlab-infrastructure.yml
variables:
FOO: Bar
When I run this CI/CD pipeline I can see the FOO variable being properly overriden and I can see that the included job's script is executed. The problem is that I get a
$ aws cloudformation deploy --stack-name sample --template-file templates/aws-template.yml
Invalid template path templates/aws-template.yml
This is probably because the local relative path is in my-infrastructure, but not in my-prod-deployment that file is not locally available in this project and therefore it can't be found.
Is there any solution to this?
Maybe a way to include not only gitlab but also other files or similar?
Or maybe some kind of shortcut or link to a different repo folder?
Or maybe a way to temporary copy a remote folder to the local CI/CD pipeline execution?
Notice that I cannot use an absolute or URL path for that script parameter since that specific tool (AWS CLI) does not allow it. Otherwise I wouldn't face this relative path issue
UPDATE 1: I have tried a workaround with git submodules separating the gitlab template in a different project and adding my-infrastructure as a submodule
cd my-prod-deployment
git submodule add git#gitlab.com:mycompany/my-infrastructure.git
so that my .gitlab-ci.yml looks like this
include:
- project: mycompany/my-gitlab-templates
ref: main
file: gitlab-infrastructure.yml
variables:
CLOUDFORMATION_SUBMODULE: my-infrastructure
FOO: Bar
and my repo has a local folder my-infrastructure, but I am shocked to find that it still complains about the AWS CloudFormation template path, so I've added AWS Cloud Formation tag to the question and edited it.
This is the error
$ aws cloudformation deploy --stack-name sample --template-file $CLOUDFORMATION_SUBMODULE/templates/aws-template.yml
Invalid template path my-infrastructure/templates/aws-template.yml
Cleaning up project directory and file based variables 00:00
ERROR: Job failed: exit code 1
There is a my-infrastructure/templates/aws-template.yml path under my repo. It's part of the submodule. So I don't understand why this workaround does not work.
Any help would be appreciated.

Answer 1

I fixed the issue with the git submodule approach.
I had to add 2 changes as per https://docs.gitlab.com/ee/ci/git_submodules.html
Add the submodule with relative path
git submodule add ../my-infrastructure.git
So that .gitmodules displays the relative url within the same server
[submodule "my-infrastructure"]
path = my-infrastructure
url = ../my-infrastructure.git
and add this variable to .gitlab-ci.yml
variables:
# submodule behavior
GIT_SUBMODULE_STRATEGY: recursive