I'm trying to upload a directory with all the files and subdirectories to GCS. Can someone point me how to achieve this using NodeJS SDK ?
Sample directory structure:
project/
├── src
| ├── route.js
| └── hooks.js
├── pages
| ├── index.js
| └── login.js
└── components
└── Nav
├── nav.js
└── Nav.module.css
I try to get static file to using 'python manage.py collectstatic', but it's not working.
I run my django project in Ubuntu 20.04.4, and using Nginx as a webserver, and Guicorn for WSGI server.
Here is my error log,
You have requested to collect static files at the destination
location as specified in your settings.
This will overwrite existing files!
Are you sure you want to do this?
Type 'yes' to continue, or 'no' to cancel: yes
Traceback (most recent call last):
File "manage.py", line 22, in <module>
main()
File "manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/home/devadmin/venvs/bio_platform/lib/python3.8/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
utility.execute()
File "/home/devadmin/venvs/bio_platform/lib/python3.8/site-packages/django/core/management/__init__.py", line 440, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/home/devadmin/venvs/bio_platform/lib/python3.8/site-packages/django/core/management/base.py", line 414, in run_from_argv
self.execute(*args, **cmd_options)
File "/home/devadmin/venvs/bio_platform/lib/python3.8/site-packages/django/core/management/base.py", line 460, in execute
output = self.handle(*args, **options)
File "/home/devadmin/venvs/bio_platform/lib/python3.8/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 209, in handle
collected = self.collect()
File "/home/devadmin/venvs/bio_platform/lib/python3.8/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 135, in collect
handler(path, prefixed_path, storage)
File "/home/devadmin/venvs/bio_platform/lib/python3.8/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 368, in copy_file
if not self.delete_file(path, prefixed_path, source_storage):
File "/home/devadmin/venvs/bio_platform/lib/python3.8/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 278, in delete_file
if self.storage.exists(prefixed_path):
File "/home/devadmin/venvs/bio_platform/lib/python3.8/site-packages/django/core/files/storage.py", line 362, in exists
return os.path.lexists(self.path(name))
File "/home/devadmin/venvs/bio_platform/lib/python3.8/site-packages/django/contrib/staticfiles/storage.py", line 39, in path
raise ImproperlyConfigured(
django.core.exceptions.ImproperlyConfigured: You're using the staticfiles app without having set the STATIC_ROOT setting to a filesystem path.
I found the many solution from google, and i guessing that it cause because i didn't set a STATIC_ROOT properly.
Here is my project dir structure and code below;
└── bio_platform
├── common
│ ├── __init__.py
│ ├── __pycache__
│ │ ├── __init__.cpython-310.pyc
│ │ ├── __init__.cpython-38.pyc
│ │ ├── __init__.cpython-39.pyc
│ │ ├── admin.cpython-310.pyc
│ │ ├── admin.cpython-38.pyc
│ │ ├── admin.cpython-39.pyc
│ │ ├── apps.cpython-310.pyc
│ │ ├── apps.cpython-38.pyc
│ │ ├── apps.cpython-39.pyc
│ │ ├── forms.cpython-310.pyc
│ │ ├── forms.cpython-38.pyc
│ │ ├── forms.cpython-39.pyc
│ │ ├── models.cpython-310.pyc
│ │ ├── models.cpython-38.pyc
│ │ ├── models.cpython-39.pyc
│ │ ├── urls.cpython-310.pyc
│ │ ├── urls.cpython-38.pyc
│ │ ├── urls.cpython-39.pyc
│ │ ├── views.cpython-310.pyc
│ │ ├── views.cpython-38.pyc
│ │ └── views.cpython-39.pyc
│ ├── admin.py
│ ├── apps.py
│ ├── forms.py
│ ├── migrations
│ │ ├── __init__.py
│ │ └── __pycache__
│ ├── models.py
│ ├── tests.py
│ ├── urls.py
│ └── views.py
├── config
│ ├── __init__.py
│ ├── __pycache__
│ │ ├── __init__.cpython-310.pyc
│ │ ├── __init__.cpython-38.pyc
│ │ ├── __init__.cpython-39.pyc
│ │ ├── settings.cpython-310.pyc
│ │ ├── settings.cpython-38.pyc
│ │ ├── settings.cpython-39.pyc
│ │ ├── urls.cpython-310.pyc
│ │ ├── urls.cpython-38.pyc
│ │ ├── urls.cpython-39.pyc
│ │ ├── wsgi.cpython-310.pyc
│ │ ├── wsgi.cpython-38.pyc
│ │ └── wsgi.cpython-39.pyc
│ ├── asgi.py
│ ├── settings
│ │ ├── __pycache__
│ │ ├── base.py
│ │ ├── local.py
│ │ └── prod.py
│ ├── urls.py
│ └── wsgi.py
├── db.sqlite3
├── manage.py
├── node_modules
│ └── bootstrap
│ ├── LICENSE
│ ├── README.md
│ ├── dist
│ ├── js
│ ├── package.json
│ └── scss
├── package-lock.json
├── pybo
│ ├── __init__.py
│ ├── __pycache__
│ │ ├── __init__.cpython-310.pyc
│ │ ├── __init__.cpython-38.pyc
│ │ ├── __init__.cpython-39.pyc
│ │ ├── admin.cpython-310.pyc
│ │ ├── admin.cpython-38.pyc
│ │ ├── admin.cpython-39.pyc
│ │ ├── apps.cpython-310.pyc
│ │ ├── apps.cpython-38.pyc
│ │ ├── apps.cpython-39.pyc
│ │ ├── forms.cpython-310.pyc
│ │ ├── forms.cpython-38.pyc
│ │ ├── forms.cpython-39.pyc
│ │ ├── models.cpython-310.pyc
│ │ ├── models.cpython-38.pyc
│ │ ├── models.cpython-39.pyc
│ │ ├── urls.cpython-310.pyc
│ │ ├── urls.cpython-38.pyc
│ │ ├── urls.cpython-39.pyc
│ │ └── views.cpython-310.pyc
│ ├── admin.py
│ ├── apps.py
│ ├── forms.py
│ ├── migrations
│ │ ├── 0001_initial.py
│ │ ├── 0002_question_author.py
│ │ ├── 0003_answer_author.py
│ │ ├── 0004_answer_modify_date_question_modify_date.py
│ │ ├── 0005_comment.py
│ │ ├── 0006_answer_voter_question_voter_alter_answer_author_and_more.py
│ │ ├── 0007_auto_20220411_1325.py
│ │ ├── __init__.py
│ │ └── __pycache__
│ ├── models.py
│ ├── templatetags
│ │ ├── __pycache__
│ │ └── pybo_filter.py
│ ├── tests.py
│ ├── urls.py
│ └── views
│ ├── __pycache__
│ ├── answer_views.py
│ ├── base_views.py
│ ├── comment_views.py
│ ├── question_views.py
│ └── vote_views.py
├── static
│ ├── bootstrap.min.css
│ ├── bootstrap.min.js
│ ├── jquery-3.6.0.min.js
│ └── style.css
├── templates
│ ├── base.html
│ ├── common
│ │ ├── login.html
│ │ └── signup.html
│ ├── form_errors.html
│ ├── navbar.html
│ └── pybo
│ ├── answer_form.html
│ ├── comment_form.html
│ ├── question_detail.html
│ ├── question_form.html
│ └── question_list.html
└── winehq.key
As you could notice, i divide my setting files like this,
It because i'd like to operate Server development environment and Local development environment separately.
├── settings
│ │ ├── __pycache__
│ │ ├── base.py
│ │ ├── local.py
│ │ └── prod.py
And here is my full code of setting files
(I blanked the private parts which is like IP address).
base.py
"""
Django settings for config project.
Generated by 'django-admin startproject' using Django 4.0.3.
For more information on this file, see
https://docs.djangoproject.com/en/4.0/topics/settings/
For the full list of settings and their values, see
https://docs.djangoproject.com/en/4.0/ref/settings/
"""
from pathlib import Path
# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent.parent
# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/4.0/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'blank' #private part
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True
ALLOWED_HOSTS = ['blank'] #private part
# Application definition
INSTALLED_APPS = [
'common.apps.CommonConfig',
'pybo.apps.PyboConfig',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
]
MIDDLEWARE = [
'django.middleware.security.SecurityMiddleware',
'django.contrib.sessions.middleware.SessionMiddleware',
'django.middleware.common.CommonMiddleware',
'django.middleware.csrf.CsrfViewMiddleware',
'django.contrib.auth.middleware.AuthenticationMiddleware',
'django.contrib.messages.middleware.MessageMiddleware',
'django.middleware.clickjacking.XFrameOptionsMiddleware',
]
ROOT_URLCONF = 'config.urls'
TEMPLATES = [
{
'BACKEND': 'django.template.backends.django.DjangoTemplates',
'DIRS': [BASE_DIR / 'templates'],
'APP_DIRS': True,
'OPTIONS': {
'context_processors': [
'django.template.context_processors.debug',
'django.template.context_processors.request',
'django.contrib.auth.context_processors.auth',
'django.contrib.messages.context_processors.messages',
],
},
},
]
WSGI_APPLICATION = 'config.wsgi.application'
# Database
# https://docs.djangoproject.com/en/4.0/ref/settings/#databases
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': BASE_DIR / 'db.sqlite3',
}
}
# Password validation
# https://docs.djangoproject.com/en/4.0/ref/settings/#auth-password-validators
AUTH_PASSWORD_VALIDATORS = [
{
'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
},
{
'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
},
]
# Internationalization
# https://docs.djangoproject.com/en/4.0/topics/i18n/
LANGUAGE_CODE = 'ko-kr'
TIME_ZONE = 'Asia/Seoul'
USE_I18N = True
USE_TZ = True
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/4.0/howto/static-files/
STATIC_URL = 'static/'
#STATIC_ROOT = os.path.join(BASE_DIR,'static')
STATICFILES_DIRS = [
BASE_DIR / 'static',
]
# Default primary key field type
# https://docs.djangoproject.com/en/4.0/ref/settings/#default-auto-field
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
LOGIN_REDIRECT_URL = '/'
LOGOUT_REDIRECT_URL = '/'
local.py
from .base import *
ALLOWED_HOSTS = []
prod.py
from .base import *
import os
ALLOWED_HOSTS = ['blank'] #private part
STATIC_URL = 'static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'static')
#STATIC_ROOT = BASE_DIR / 'static/'
STATICFILES_DIRS = []
# concept addon
Also, here is my bio_platform.service for nignx server.
bio_platform.service
server {
listen 80;
server_name blank; #private part
location = /favicon.ico { access_log off; log_not_found off; }
location /static {
alias /home/devadmin/projects/bio_platform/static;
}
location / {
include proxy_params;
proxy_pass http://unix:/tmp/gunicorn.sock;
}
}
I did some of solution that given in google and stackoverflow also which is like:
what i try
set a STATIC_ROOT and STATIC_URL on prod.py
set a STATIC_ROOT and STATIC_URL on base.py
But as you know, it didn't work and django.core.exceptions.ImproperlyConfigured: You're using the staticfiles app without having set the STATIC_ROOT setting to a filesystem path. error is still remain.
For know, i'm not sure where sould i set STATIC_ROOT or STATIC_URL, STATICFILES_DIR.
Please helpe me guys....
I did found what i wrong in my project.
The problem was caused by path setting which is bring prod.py settings automaticly from venvs folder.
I made a bio_platform.sh file under the path
~/venvs/
before
#!/bin/bash
cd ~/projects/bio_paltform
export DJANGO_SETTINGS_MODULE=config.settings.prod
. ~/venvs/bio_paltform/bin/activate
As you guys could see that i typed wrong that bio_paltform.
So i change the bio_paltform to bio_platform.
after
#!/bin/bash
cd ~/projects/bio_platform
export DJANGO_SETTINGS_MODULE=config.settings.prod
. ~/venvs/bio_platform/bin/activate
After i modify this file, python managy.py collectstatic order is work.
$ python manage.py collectstatic
You have requested to collect static files at the destination
location as specified in your settings:
/home/devadmin/projects/bio_platform/static
This will overwrite existing files!
Are you sure you want to do this?
Type 'yes' to continue, or 'no' to cancel: yes
128 static files copied to '/home/devadmin/projects/bio_platform/static'.
How to ignore the hidden files while recursively traversing the directories.
My file structure is of following type:
7_jan
├── 7_jan_25_cropped
│ ├── 1.tiff
|
│
│
├── 7_jan_50_cropped
│ ├── 1.tiff
│ ├── 10.tiff
│ ├── 11.tiff
│ ├── 12.tiff
│ ├── 13.tiff
│ ├── 14.tiff
│
└── 7_jan_75_cropped
├── 1.tiff
├── 10.tiff
├── 11.tiff
├── 12.tiff
I am recursively storing each file path so that I can later operate upon them but meanwhile the .DS_Store file is also getting stored which I don't want to store. How to remove that?
folders = []
files = []
rec_folders = []
for entry in os.scandir('/Users/swastik/csre/dataset'):
if entry.is_dir():
folders.append(entry.path)
for recentry in os.scandir(entry.path):
if not recentry.path.startswith('.'):
rec_folders.append(recentry.path)
elif entry.is_file():
files.append(entry.path)
print('Folders:')
print(folders)
print('Further files:')
print(rec_folders)
Output-
Folders:
['/Users/swastik/csre/dataset/7_jan']
Further folders:
['/Users/swastik/csre/dataset/7_jan/7_jan_75_cropped',
'/Users/swastik/csre/dataset/7_jan/.DS_Store',
'/Users/swastik/csre/dataset/7_jan/7_jan_50_cropped',
'/Users/swastik/csre/dataset/7_jan/7_jan_25_cropped']
Here , it is also storing the .DS_Store file, which I don't want.
You can just replace if not recentry.path.startswith('.'): with if not recentry.name.startswith('.'):, so that it will ignore your .DS_Store file.
I m currently trying to configure some directory environment to manage different clients
Puppet Master version : "puppet-server-3.8.1-1" (centos 6)
Here is my tree from the puppet master /etc/puppet :
├── organisation
│ ├── environment.conf
│ ├── manifests
│ │ ├── accounts.pp
│ │ ├── lab_accounts.pp
│ │ ├── lab_nodes.pp
│ │ └── nodes.pp
│ └── modules
│ ├── account
│ │ ├── files
│ │ ├── lib
│ │ ├── spec
│ │ │ └── classes
│ │ └── templates
│ └── dns
│ ├── manifests
│ │ └── init.pp
│ └── templates
│ ├── resolv.conf.erb
│ └── resolv.conf.fqdn.erb
├── production
│ ├── environment.conf
│ ├── manifests
│ │ ├── accounts.pp
│ │ ├── lab_accounts.pp
│ │ └── lab_nodes.pp
│ └── modules
│ ├── account
│ │ ├── CHANGELOG
│ │ ├── files
│ │ ├── lib
│ │ ├── LICENSE
│ │ ├── manifests
│ │ │ ├── init.pp
│ │ │ └── site.pp
│ │ ├── metadata.json
│ │ ├── Modulefile
│ │ ├── Rakefile
│ │ ├── README.mkd
│ │ ├── spec
│ │ │ ├── classes
│ │ │ ├── defines
│ │ │ │ └── account_spec.rb
│ │ │ └── spec_helper.rb
│ │ └── templates
│ ├── dns
│ │ ├── manifests
│ │ │ └── init.pp
│ │ └── templates
│ │ ├── resolv.conf.erb
│ │ └── resolv.conf.fqdn.erb
│ └── sshkeys
│ └── manifests
│ └── init.pp
└── README.md
Now the configuration files :
/etc/puppet.conf
[main]
logdir = /var/log/puppet
rundir = /var/run/puppet
ssldir = $vardir/ssl
dns_alt_names = centos66a.local.lab,centos66a,puppet,puppetmaster
[master]
environmentpath = $confdir/environments
basemodulepath = $confdir/modules:/opt/puppet/share/puppet/modules
[agent]
classfile = $vardir/classes.txt
localconfig = $vardir/localconfig
server = puppet
Here is the environment I called "organisation" :
/etc/puppet/environments/organisation/environment.conf
modulepath = /etc/puppet/environments/organisation/modules
environment_timeout = 5s
Now I declare my nodes in "nodes.pp" :
/etc/puppet/environments/organisation/manifests/nodes.pp
node 'centos66a.local.lab' {
include dns
}
node 'gcacnt02.local.lab' {
include dns
}
Here is the output when I try to sync my node to the master :
gcacnt02:~ # hostname
gcacnt02.local.lab
gcacnt02:~ # puppet agent -t
Info: Creating a new SSL key for gcacnt02.local.lab
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for gcacnt02.local.lab
Info: Certificate Request fingerprint (SHA256): 49:73:11:78:99:6F:50:BD:6B:2F:5D:B9:92:7C:6F:A9:63:52:92:53:DB:B8:A1:AE:86:21:AF:36:BE:B0:94:DB
Info: Caching certificate for gcacnt02.local.lab
Info: Caching certificate for gcacnt02.local.lab
Info: Retrieving pluginfacts
Info: Retrieving plugin
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find default node or by name with 'gcacnt02.local.lab, gcacnt02.local, gcacnt02' on node gcacnt02.local.lab
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
if I move /etc/puppet/environments/organisation/manifests/nodes.pp to /etc/puppet/environments/production/manifests/nodes.pp it works just fine.
When I print "manifest" from "organisation" and "production" I get a correct output asweel :
[root#centos66a environments]# puppet config print manifest --section master --environment production
/etc/puppet/environments/production/manifests
[root#centos66a environments]# puppet config print manifest --section master --environment organisation
/etc/puppet/environments/organisation/manifests
I m probably missing something here but can't put my finger on it...
Thank you
Problem resolved.
Configuration on the master is OK.
Since Puppet scan for directory in the environment directory set by the "environmentpath" variables, I thought that the master would automaticly reply to nodes in setup in each environment. This is false.
Default environment is : Production.
If you set any other environment you have to configure each puppet agent node to query to a specific environment
In my case, my node is gcacnt02.local.lab. So to fix the issue I had to add the following variable in /etc/puppet/puppet.conf
[main]
logdir = /var/log/puppet
rundir = /var/run/puppet
ssldir = $vardir/ssl
[agent]
classfile = $vardir/classes.txt
localconfig = $vardir/localconfig
environment = lan
My CouchApp has the following folder structur, where files inside the app folder are compiled into the _attachments folder:
my_couchapp
├── _attachments/
│ ├── app.js
│ ├── app-tests.js
│ └── index.html
├── app/
│ └── app.js
├── Assetfile
└── views/
I want to exclude the file Assetfile, _attachments/app-tests.js and the folder app.
My current .couchappignore looks like this:
[
"app",
"Assetfile",
"_attachments/app-tests.js"
]
But this doesn't seem to work. All files beginning with app inside the _attachments folder are not pushed.
How do I define folders and specific files to be excluded when the CouchApp is pushed via couchapp push?
After a little more experimentation I found a way: the app folder can be excluded by specifying app$, so the final .couchappignore now looks like this:
[
"app$",
"Assetfile",
"app-tests.js"
]
In case you arrived here looking for a way to ignore subfolders, you are just like me. Here's my problem:
my-couchapp/
├── node_modules/
│ ├── react.js
│ ├── url/
│ ├── browserify/
│ └── coffee-script/
├── app/
│ └── app.js
└── views/
I wanted to include node_modules/react.js and node_modules/url/ (and all subfolders), but didn't want to include node_modules/browserify/ and node_modules/coffeescript.
I was trying
[
"node_modules/browserify$",
"node_modules/coffee-script$"
]
but it wasn't working.
[
"node_modules\/browserify",
"node_modules\/coffee-script"
]
also didn't work.
The only thing that worked was
[
"browserify",
"coffee-script"
]
I don't know why.