Replace multiple lines with multiple lines after finding a context - search

I have a file which contains below lines.
xyz_write =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = hostname.com)(PORT = 80))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SID = database)
)
)
I want to find xyz_write and replace the lines below it till empty line is found with below lines.
xyz_write =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = hostname2.com)(PORT = 80))
(CONNECT_DATA =
(FAILOVER = TRUE)
(SERVER = DEDICATED)
(SID = database)
(SERVICE = servicename)
)
)
I tried using sed command by storing the first set of lines in a variable and replacing it will second set of lines. But it did not help. How can we achieve this ?
I tried below method:
I assigned first set of lines to a variable as below.
VARIABLE1="xyz_write =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = hostname.com)(PORT = 80))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SID = database)
)
)"
and
VARIABLE2="xyz_write =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = hostname2.com)(PORT = 80))
(CONNECT_DATA =
(FAILOVER = TRUE)
(SERVER = DEDICATED)
(SID = database)
(SERVICE = servicename)
)
)".
Then i tried running the command
"cat source_file.txt | sed "s/$VARIABLE1/$VARIABLE2/g" > destination_file.txt".
This will not work due to line breaks. There should be some way to achieve this.

The file filename.txt contains the lines as below
$cat filename.txt
<blank line>
abc_write =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = hostname2.com)(PORT = 80))
(CONNECT_DATA =
(FAILOVER = TRUE)
(SERVER = DEDICATED)
(SID = database)
(SERVICE = servicename)
)
)
<blank line>
xyz_write =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = hostname2.com)(PORT = 80))
(CONNECT_DATA =
(FAILOVER = TRUE)
(SERVER = DEDICATED)
(SID = database)
(SERVICE = servicename)
)
)
<blank line>
I wanted just one entry for the context. For "xyz_write" I needed single entry in the file. While adding a new entry to the file, check if it already exists. If already a entry is present for "xyz_write", warn the user and update the entry with new value. Hence I wanted to replace. Somehow I achieved this by removing the existing entry and appending the new entry at the end of file.
Command to Remove the existing entry.
\#This value is a command line input.
SERVICE_NAME="xyz_write"
\#This will remove the existing entry. The command will delete the lines till empty line after the context. -i option will write to the same input file.
sed -i "/$SERVICE_NAME/,/^$/d" filename.txt
Append the new value at the end of file.
NEW_VALUE="bcd_write
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = hostname2.com)(PORT = 80))
(CONNECT_DATA =
(FAILOVER = TRUE)
(SERVER = DEDICATED)
(SID = database)
(SERVICE = servicename)
)
)"
echo $NEW_VALUE >>file_name.txt
Thus my purpose is accomplished. Its not actually replacing the multiple lines. Its achieved by removing existing lines and appending the new lines. Hope this helps somebody.
Thanks,
-Adithya

Related

How to find _ transferSize in har file exported using browsermob-proxy in python

I am trying to export .har file using firefox-selenium-browsermob-proxy-python. Using the below code.
bmp_loc = "/Users/project/browsermob-proxy-2.1.4/bin/browsermob-proxy"
server = Server(bmp_loc)
server.start()
proxy = server.create_proxy(params={'trustAllServers': 'true'})
selenium_proxy = proxy.selenium_proxy()
caps = webdriver.DesiredCapabilities.FIREFOX
caps['marionette'] = False
proxy_settings = {
"proxyType": "MANUAL",
"httpProxy": selenium_proxy.httpProxy,
"sslProxy": selenium_proxy.sslProxy,
}
caps['proxy'] = proxy_settings
driver = webdriver.Firefox(desired_capabilities=caps)
proxy.new_har("generated_har",options={'captureHeaders': True})
driver.get("someurl")
browser_logs = proxy.har
I am interested to get _transferSize in the .har file to perform some analysis but unable to get that, instead I am getting that as 'comment':
"redirectURL": "", "headersSize": 1023, "bodySize": 38, "comment": ""
whereas manually downloading the .har file using firefox I am getting _transferSize
Version used:
browsermob_proxy==2.1.4
selenium==4.0.0
Can anybody please help me to resolve this?
You can get _transferSize by adding headersSize and bodySize from the har file itself.
urls = ["https://google.com"]
for ur in urls:
server = proxy.start_server()
client = proxy.start_client()
client.new_har("demo.com")
# print(client.proxy)
options = webdriver.ChromeOptions()
options.add_argument("--disk-cache-size=0")
options = {
'enable_har': True
}
driver = webdriver.Chrome(seleniumwire_options=options)
driver.request_interceptor = proxy.interceptor
driver.get(ur)
time.sleep(40)
row_list = []
json_dictionary = json.loads(driver.har)
repeat_url_list = []
repeat_urls = defaultdict(lambda:[])
resp_size = 0
count_url = 0
url_time = 0
status_list = []
status_url = defaultdict(lambda:[])
a_list = []
with open("network_log2.har", "w", encoding="utf-8") as f:
# f.write(json.dumps(driver.har))
for i in json_dictionary['log']['entries']:
f.write(str(i))
f.write("\n")
url = i['request']['url']
a_list.append(url)
timing = i['time']
if timing>2000:
timing = round(timing/2000,1)
url_time += 1
status = i['response']['status']
if status in status_list:
status_url[status] = status_url[status] + 1
else:
status_url[status] = 1
status_list.append(status)
size = i['response']['headersSize'] + i['response']['bodySize']
if size//1000 > 500:
resp_size += 1
if url in repeat_url_list:
repeat_urls[url] = 1
else:
repeat_url_list.append(url)
rurl_count = len(repeat_urls)

NETCDF4 file doesn't grow beyond 2GB

I have a NETCDF4 file which doesn't grow beyond 2GB.
I am using the following sample data - I am converting over 200 txt files to netcdf4 file
STATIONS_ID;MESS_DATUM; QN;FF_10;DD_10;eor
3660;201912150000; 3; 4.6; 170;eor
3660;201912150010; 3; 4.2; 180;eor
3660;201912150020; 3; 4.3; 190;eor
3660;201912150030; 3; 5.2; 190;eor
3660;201912150040; 3; 5.1; 190;eor
3660;201912150050; 3; 4.8; 190;eor
The code looks like:
files = [f for f in os.listdir('.') if os.path.isfile(f)]
count = 0
for f in files:
filecp = open(f, "r", encoding="ISO-8859-1")
# NC file setup
mydata = netCDF4.Dataset('v5.nc', 'w', format='NETCDF4')
mydata.description = 'Measurement Data'
mydata.createDimension('STATION_ID',None)
mydata.createDimension('MESS_DATUM',None)
mydata.createDimension('QN',None)
mydata.createDimension('FF_10',None)
mydata.createDimension('DD_10',None)
STATION_ID = mydata.createVariable('STATION_ID',np.short,('STATION_ID'))
MESS_DATUM = mydata.createVariable('MESS_DATUM',np.long,('MESS_DATUM'))
QN = mydata.createVariable('QN',np.byte,('QN'))
FF_10 = mydata.createVariable('FF_10',np.float64,('FF_10'))
DD_10 = mydata.createVariable('DD_10',np.short,('DD_10'))
STATION_ID.units = ''
MESS_DATUM.units = 'Central European Time yyyymmddhhmi'
QN.units = ''
FF_10.units = 'meters per second'
DD_10.units = "degree"
txtdata = pd.read_csv(filecp, delimiter=';').values
#txtdata = np.genfromtxt(filecp, dtype=None, delimiter=';', names=True, encoding=None)
if len(txtdata) > 0:
df = pd.DataFrame(txtdata)
sh = txtdata.shape
print("txtdata shape is ", sh)
mydata['STATION_ID'][:] = df[0]
mydata['MESS_DATUM'][:] = df[1]
mydata['QN'][:] = df[2]
mydata['FF_10'][:] = df[3]
mydata['DD_10'][:] = df[4]
mydata.close()
filecp.close()
count +=1
Your problem is that you create the same file in the loop. So your file size is limited to the biggest initial data file.
Open the file once, and add each new data to the end of netcdf data arrays.
If you get 124 values in the first file, you put:
mydata['STATION_ID'][0:124] = df[0]
and you get 224 from the second file, you put
mydata['STATION_ID'][124:124+224] = df[0]
So, in case data files are downloaded from https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/10_minutes/wind/recent/ to <text file path>
import netCDF4
import codecs
import pandas as pd
import os
import numpy as np
mydata = netCDF4.Dataset('v5.nc', 'w', format='NETCDF4')
mydata.description = 'Wind Measurement Data'
mydata.createDimension('STATION_ID',None)
mydata.createDimension('MESS_DATUM',None)
mydata.createDimension('QN',None)
mydata.createDimension('FF_10',None)
mydata.createDimension('DD_10',None)
STATION_ID = mydata.createVariable('STATION_ID',np.short,('STATION_ID'))
MESS_DATUM = mydata.createVariable('MESS_DATUM',np.long,('MESS_DATUM'))
QN = mydata.createVariable('QN',np.byte,('QN'))
FF_10 = mydata.createVariable('FF_10',np.float64,('FF_10'))
DD_10 = mydata.createVariable('DD_10',np.short,('DD_10'))
STATION_ID.units = ''
MESS_DATUM.units = 'Central European Time yyyymmddhhmi'
QN.units = ''
FF_10.units = 'meters per second'
DD_10.units = "degree"
fpath = <text file path>
files = [f for f in os.listdir(fpath)]
count = 0
mydata_startindex=0
for f in files:
filecp = open(fpath+f, "r", encoding="ISO-8859-1")
txtdata = pd.read_csv(filecp, delimiter=';')
chunksize = len(txtdata)
if len(txtdata) > 0:
mydata['STATION_ID'][mydata_startindex:mydata_startindex+chunksize] = txtdata['STATIONS_ID']
mydata['MESS_DATUM'][mydata_startindex:mydata_startindex+chunksize] = txtdata['MESS_DATUM']
mydata['QN'][mydata_startindex:mydata_startindex+chunksize] = txtdata[' QN']
mydata['FF_10'][mydata_startindex:mydata_startindex+chunksize] = txtdata['FF_10']
mydata['DD_10'][mydata_startindex:mydata_startindex+chunksize] = txtdata['DD_10']
mydata_startindex += chunksize

Python3 urllib.error.URLError: <urlopen error [Errno 110] Connection timed out> on server

I have script which have to build an new .xml file from .xml file on some URL, when I run script on my PC, everything is okay. But after I try to run on my server, it throws urllib.error.URLError: <urlopen error [Errno 110] Connection timed out>
I don't know why and I didn't find any good source where this problem was described, so I want this script running on my server automatically as CRON, but I can't.. Does not works..
Code:
#!/usr/bin/env python3
import urllib.request, urllib.parse, urllib.error
import xml.etree.ElementTree as ET
url = 'http://eshop.cobi.cz/modules/xmlfeeds/xml_files/feed_5.xml'
uh = urllib.request.urlopen(url, timeout=30)
tree = ET.parse(uh)
root = tree.getroot()
data = ET.Element('SHOP')
for r in root.findall('SHOPITEM'):
name = r.find('PRODUCT').text
try:
description = r.find('DESCRIPTION').text
except AttributeError:
description = ""
continue
price = r.find('WHOLESALEPRICE').text
new_price = (float(price) * 0.8)
final_price = round(new_price, 2)
final_price = str(final_price)
ean = r.find('EAN').text
sku = r.find('PRODUCTNO').text
for i in r.findall('IMAGES'):
img = i.find('IMGURL').text
status = r.find('ACTIVE').text
if status == '1':
status = '1'
else:
status = '0'
continue
element2 = ET.SubElement(data, 'SHOPITEM')
s_elem2_1 = ET.SubElement(element2, 'PRODUCTNAME')
s_elem2_2 = ET.SubElement(element2, 'PRODUCT')
s_elem2_3 = ET.SubElement(element2, 'DESCRIPTION')
s_elem2_4 = ET.SubElement(element2, 'PRICE_VAT')
s_elem2_5 = ET.SubElement(element2, 'EAN')
s_elem2_6 = ET.SubElement(element2, 'PRODUCTNO')
s_elem2_7 = ET.SubElement(element2, 'IMGURL')
s_elem2_8 = ET.SubElement(element2, 'DELIVERY_DATE')
s_elem2_9 = ET.SubElement(element2, 'CATEGORYTEXT')
s_elem2_10 = ET.SubElement(element2, 'MANUFACTURER')
s_elem2_11 = ET.SubElement(element2, 'ACTIVE')
s_elem2_1.text = "Cobi " + name
s_elem2_2.text = "Cobi " + name
s_elem2_3.text = description
s_elem2_4.text = final_price
s_elem2_5.text = ean
s_elem2_6.text = sku
s_elem2_7.text = img
s_elem2_8.text = "7"
s_elem2_9.text = "Heureka.cz | Dětské zboží | Hračky | Stavebnice | Stavebnice Cobi"
s_elem2_10.text = "Cobi"
s_elem2_11.text = status
xml_content = ET.tostring(data)
with open('cobifeed.xml', 'wb') as f:
f.write(xml_content)
f.close()
try ping shop.cobi.cz from cmd/bash and wget http://eshop.cobi.cz/modules/xmlfeeds/xml_files/feed_5.xml
Probably You dont have network access to server eshop.cobi.cz

Airflow Dag Run Stalls Mid-Run

I have setup airflow v1.10.6 on ubuntu, using python 3.6.8. and have written a simple practice dag that runs a set of 30 parallel tasks. I'm triggering the dag through the UI as this will be run irregularly and I'm just learning. I'm now configured to LocalExecutor, parallelism=4 in airflow.cfg (full cfg file is below). The dag run starts and can be viewed in the UI, completes exactly 4 task instances, then the dag run appears to stall in the UI, all other downstream tasks hold either scheduled or no_status forever. If I reset everything, bump up parallelism=10 in airflow.cfg, and re-trigger the run, 10 task instances run and the dag run stalls again in the same way. When the run appears to stall on the UI, I see no errors in the scheduler logs, or webserver logs and the cpus sure looks like the machine is running tasks. I get the exact same behavior running this dag on CeleryExecutor and running airflow worker as well. Also seems to occur regardless of whether I have schedule_interval=None or timedelta(days=1).
Not to state the obvious but, as I understand it, "parallelism" means as the first 4 tasks instances finish, then others should be able to open up. Any ideas as to why I am getting this behavior?
Perhaps a sidenote, perhaps not: I have also installed rabbitmq-server for use with Celery and am running the database on an AWS RDS instance in the same vpc.
DAG is below:
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.operators.python_operator import PythonOperator
from datetime import datetime, timedelta
import time
default_args = {
'owner': 'airflow',
'depends_on_past': True,
'start_date': datetime(2019, 6, 1),
'email': ['airflow#example.com','evanburgess#gmail.com'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=1),
}
dag = DAG('example', default_args=default_args, schedule_interval=timedelta(days=1))
def do_a_process(flname, i, j):
time.sleep(5)
mkarr = PythonOperator(
task_id='make_random_array',
default_args=default_args,
python_callable=do_a_process,
op_args=('filepath',10000,10000),
dag=dag)
for i in range(30):
smo = PythonOperator(
task_id='gaussian_%i_sigma' % i,
default_args=default_args,
python_callable=do_a_process,
op_args=('fileapth', i, i),
dag=dag
)
stat = PythonOperator(
task_id='stats_%i_sigma' % i,
default_args=default_args,
python_callable=do_a_process,
op_args=('fileapth', i, i),
dag=dag
)
mkarr >> smo
smo >> stat
And my complete config file (comments removed)is below as well:
[core]
dags_folder = /home/ubuntu/airflow/dags
base_log_folder = /home/ubuntu/airflow/logs
remote_logging = False
remote_log_conn_id =
remote_base_log_folder =
encrypt_s3_logs = False
logging_level = INFO
fab_logging_level = WARN
logging_config_class =
colored_console_log = True
colored_log_format = [%%(blue)s%%(asctime)s%%(reset)s] {%%(blue)s%%(filename)s:%%(reset)s%%(lineno)d} %%(log_color)s%%(levelname)s%%(re
set)s - %%(log_color)s%%(message)s%%(reset)s
colored_formatter_class = airflow.utils.log.colored_log.CustomTTYColoredFormatter
log_format = [%%(asctime)s] {%%(filename)s:%%(lineno)d} %%(levelname)s - %%(message)s
simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s
log_filename_template = {{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log
log_processor_filename_template = {{ filename }}.log
dag_processor_manager_log_location = /home/ubuntu/airflow/logs/dag_processor_manager/dag_processor_manager.log
hostname_callable = socket:getfqdn
default_timezone = utc
executor = LocalExecutor
sql_alchemy_conn = postgresql+psycopg2://airflow:mypassword#my_url_rds.amazonaws.com:5432/airflow
sql_engine_encoding = utf-8
sql_alchemy_pool_enabled = True
sql_alchemy_pool_size = 5
sql_alchemy_max_overflow = 10
sql_alchemy_pool_recycle = 1800
sql_alchemy_pool_pre_ping = True
sql_alchemy_schema =
parallelism = 4
dag_concurrency = 4
dags_are_paused_at_creation = True
max_active_runs_per_dag = 1
load_examples = False
plugins_folder = /home/ubuntu/airflow/plugins
fernet_key = bighexadecimal
donot_pickle = False
dagbag_import_timeout = 30
dag_file_processor_timeout = 50
task_runner = StandardTaskRunner
default_impersonation =
security =
secure_mode = False
unit_test_mode = False
task_log_reader = task
enable_xcom_pickling = True
killed_task_cleanup_time = 60
dag_run_conf_overrides_params = False
worker_precheck = False
dag_discovery_safe_mode = True
default_task_retries = 0
[cli]
api_client = airflow.api.client.local_client
endpoint_url = http://localhost:8080
[api]
auth_backend = airflow.api.auth.backend.default
[lineage]
backend =
[atlas]
sasl_enabled = False
host =
port = 21000
username =
password =
[operators]
default_owner = airflow
default_cpus = 1
default_ram = 512
default_disk = 512
default_gpus = 0
[hive]
default_hive_mapred_queue =
[webserver]
base_url = http://localhost:8080
web_server_host = 0.0.0.0
web_server_port = 8080
web_server_ssl_cert =
web_server_ssl_key =
web_server_master_timeout = 120
web_server_worker_timeout = 120
worker_refresh_batch_size = 1
worker_refresh_interval = 30
secret_key = temporary_key
workers = 4
worker_class = sync
access_logfile = -
error_logfile = -
expose_config = False
authenticate = False
filter_by_owner = False
owner_mode = user
dag_default_view = tree
dag_orientation = LR
demo_mode = False
log_fetch_timeout_sec = 5
hide_paused_dags_by_default = False
page_size = 100
rbac = False
navbar_color = #007A87
default_dag_run_display_number = 25
enable_proxy_fix = False
cookie_secure = False
cookie_samesite =
default_wrap = False
[email]
email_backend = airflow.utils.email.send_email_smtp
[smtp]
smtp_host = localhost
smtp_starttls = True
smtp_ssl = False
smtp_port = 25
smtp_mail_from = airflow#example.com
[sentry]
sentry_dsn =
[celery]
celery_app_name = airflow.executors.celery_executor
worker_concurrency = 16
worker_log_server_port = 8793
broker_url = pyamqp://test:test#localhost:5672/
result_backend = db+postgresql://airflow:snowSNOWsnowSNOW#processing-1.cdpofxkpfulr.us-west-2.rds.amazonaws.com:5432/airflow
flower_host = 0.0.0.0
flower_url_prefix =
flower_port = 5555
flower_basic_auth =
default_queue = default
sync_parallelism = 0
celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG
ssl_active = False
ssl_key =
ssl_cert =
ssl_cacert =
pool = prefork
[celery_broker_transport_options]
[dask]
cluster_address = 127.0.0.1:8786
tls_ca =
tls_cert =
tls_key =
[scheduler]
job_heartbeat_sec = 5
scheduler_heartbeat_sec = 5
run_duration = -1
num_runs = -1
processor_poll_interval = 1
min_file_process_interval = 0
dag_dir_list_interval = 300
print_stats_interval = 30
scheduler_health_check_threshold = 30
child_process_log_directory = /home/ubuntu/airflow/logs/scheduler
scheduler_zombie_task_threshold = 300
catchup_by_default = True
max_tis_per_query = 512
statsd_on = False
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
statsd_allow_list =
max_threads = 2
authenticate = False
use_job_schedule = True
[ldap]
uri =
user_filter = objectClass=*
user_name_attr = uid
group_member_attr = memberOf
superuser_filter =
data_profiler_filter =
bind_user = cn=Manager,dc=example,dc=com
bind_password = insecure
basedn = dc=example,dc=com
cacert = /etc/ca/ldap_ca.crt
search_scope = LEVEL
ignore_malformed_schema = False
[mesos]
master = localhost:5050
framework_name = Airflow
task_cpu = 1
task_memory = 256
checkpoint = False
authenticate = False
[kerberos]
ccache = /tmp/airflow_krb5_ccache
principal = airflow
reinit_frequency = 3600
kinit_path = kinit
keytab = airflow.keytab
[github_enterprise]
api_rev = v3
[admin]
hide_sensitive_variable_fields = True
[elasticsearch]
host =
log_id_template = {dag_id}-{task_id}-{execution_date}-{try_number}
end_of_log_mark = end_of_log
frontend =
write_stdout = False
json_format = False
json_fields = asctime, filename, lineno, levelname, message
[elasticsearch_configs]
use_ssl = False
verify_certs = True
[kubernetes]
worker_container_repository =
worker_container_tag =
worker_container_image_pull_policy = IfNotPresent
delete_worker_pods = True
worker_pods_creation_batch_size = 1
namespace = default
airflow_configmap =
dags_in_image = False
dags_volume_subpath =
dags_volume_claim =
logs_volume_subpath =
logs_volume_claim =
dags_volume_host =
logs_volume_host =
env_from_configmap_ref =
env_from_secret_ref =
git_repo =
git_branch =
git_subpath =
git_user =
git_password =
git_sync_root = /git
git_sync_dest = repo
git_dags_folder_mount_point =
git_ssh_key_secret_name =
git_ssh_known_hosts_configmap_name =
git_sync_credentials_secret =
git_sync_container_repository = k8s.gcr.io/git-sync
git_sync_container_tag = v3.1.1
git_sync_init_container_name = git-sync-clone
git_sync_run_as_user = 65533
worker_service_account_name =
image_pull_secrets =
gcp_service_account_keys =
in_cluster = True
affinity =
tolerations =
kube_client_request_args = {"_request_timeout" : [60,60] }
run_as_user =
fs_group =
[kubernetes_node_selectors]
[kubernetes_annotations]
[kubernetes_environment_variables]
[kubernetes_secrets]
[kubernetes_labels]
Here are my requirements:
alembic==1.3.0
amqp==2.5.2
apache-airflow==1.10.6
apispec==3.1.0
argcomplete==1.10.0
asn1crypto==0.24.0
atomicwrites==1.3.0
attrs==19.3.0
Automat==0.6.0
Babel==2.7.0
bcrypt==3.1.7
billiard==3.6.1.0
blinker==1.4
boto3==1.7.84
botocore==1.10.84
cached-property==1.5.1
cachetools==3.1.1
celery==4.3.0
certifi==2019.9.11
cffi==1.13.2
chardet==3.0.4
Click==7.0
cloud-init==19.2
colorama==0.4.1
colorlog==4.0.2
command-not-found==0.3
configobj==5.0.6
configparser==3.5.3
constantly==15.1.0
croniter==0.3.30
cryptography==2.8
defusedxml==0.6.0
dill==0.3.1.1
distro-info===0.18ubuntu0.18.04.1
dnspython==1.16.0
docutils==0.15.2
ec2-hibinit-agent==1.0.0
eventlet==0.25.1
Flask==1.1.1
Flask-Admin==1.5.3
Flask-AppBuilder==1.13.1
Flask-Babel==0.12.2
Flask-Bcrypt==0.7.1
Flask-Caching==1.3.3
Flask-JWT-Extended==3.24.1
Flask-Login==0.4.1
Flask-OpenID==1.2.5
Flask-SQLAlchemy==2.4.1
flask-swagger==0.2.13
Flask-WTF==0.14.2
flower==0.9.3
funcsigs==1.0.0
future==0.16.0
gevent==1.4.0
google-api-core==1.14.3
google-api-python-client==1.7.11
google-auth==1.7.0
google-auth-httplib2==0.0.3
google-auth-oauthlib==0.4.1
google-cloud-bigquery==1.21.0
google-cloud-bigtable==0.33.0
google-cloud-container==0.3.0
google-cloud-core==1.0.3
google-cloud-dlp==0.12.1
google-cloud-language==1.3.0
google-cloud-spanner==1.9.0
google-cloud-speech==1.2.0
google-cloud-storage==1.22.0
google-cloud-texttospeech==0.5.0
google-cloud-translate==2.0.0
google-cloud-videointelligence==1.11.0
google-cloud-vision==0.39.0
google-resumable-media==0.4.1
googleapis-common-protos==1.6.0
graphviz==0.13.2
greenlet==0.4.15
grpc-google-iam-v1==0.11.4
grpcio==1.25.0
grpcio-gcp==0.2.2
gunicorn==19.9.0
hibagent==1.0.1
httplib2==0.14.0
hyperlink==17.3.1
idna==2.8
importlib-metadata==0.23
incremental==16.10.1
iso8601==0.1.12
itsdangerous==1.1.0
Jinja2==2.10.3
jmespath==0.9.4
json-merge-patch==0.2
jsonpatch==1.16
jsonpointer==1.10
jsonschema==3.1.1
keyring==10.6.0
keyrings.alt==3.0
kombu==4.6.6
language-selector==0.1
lazy-object-proxy==1.4.3
librabbitmq==2.0.0
lockfile==0.12.2
Mako==1.1.0
Markdown==2.6.11
MarkupSafe==1.1.1
marshmallow==2.19.5
marshmallow-enum==1.5.1
marshmallow-sqlalchemy==0.18.0
monotonic==1.5
more-itertools==7.2.0
netifaces==0.10.4
numpy==1.17.4
oauthlib==3.1.0
ordereddict==1.1
packaging==19.2
PAM==0.4.2
pandas==0.25.3
pandas-gbq==0.11.0
paramiko==2.6.0
pendulum==1.4.4
pika==0.13.0
pluggy==0.13.0
prison==0.1.0
protobuf==3.10.0
psutil==5.6.5
psycopg2==2.7.7
py==1.8.0
PyAMQP==0.0.8.5
pyasn1==0.4.7
pyasn1-modules==0.2.7
pycparser==2.19
pycrypto==2.6.1
pydata-google-auth==0.1.3
Pygments==2.4.2
pygobject==3.26.1
PyJWT==1.7.1
PyNaCl==1.3.0
pyOpenSSL==19.0.0
pyparsing==2.4.5
pyrsistent==0.15.5
pyserial==3.4
pysftp==0.2.9
pytest==5.2.2
python-apt==1.6.4
python-daemon==2.1.2
python-dateutil==2.8.1
python-debian==0.1.32
python-editor==1.0.4
python3-openid==3.1.0
pytz==2019.3
pytzdata==2019.3
pyxdg==0.25
PyYAML==5.1.2
requests==2.22.0
requests-oauthlib==1.3.0
requests-unixsocket==0.1.5
rsa==4.0
s3transfer==0.1.13
scipy==1.3.2
SecretStorage==2.3.1
service-identity==16.0.0
setproctitle==1.1.10
six==1.13.0
SQLAlchemy==1.3.10
ssh-import-id==5.7
sshtunnel==0.1.5
systemd-python==234
tabulate==0.8.5
tenacity==4.12.0
termcolor==1.1.0
text-unidecode==1.2
thrift==0.11.0
tornado==5.1.1
Twisted==17.9.0
tzlocal==1.5.1
ufw==0.36
unattended-upgrades==0.1
unicodecsv==0.14.1
uritemplate==3.0.0
urllib3==1.25.6
vine==1.3.0
wcwidth==0.1.7
Werkzeug==0.16.0
WTForms==2.2.1
zipp==0.6.0
zope.deprecation==4.4.0
zope.interface==4.3.2
All I can say is if you run into this issue, work again on your install. I used pip3 to install all but rabbitmq-server, for that I installed using apt on ubuntu. Anaconda gave me issues.

How to have bolded headers, but non-bolded cells - filled with worksheet.append_table of pygsheets==2.0.1

The python3 (pygsheets 2.0.1) script below will bold all the cells starting at A2.
Is there an easy way (i.e., in one command) to ask for all these cells not to be bolded?
Code:
import boto3, botocore
import datetime
import json
import pygsheets
currentDT = str(datetime.datetime.now())
def create_spreadsheet(outh_file, spreadsheet_name = "jSonar AWS usage"):
client = pygsheets.authorize(outh_file=outh_file, outh_nonlocal=True)
spread_sheet = client.create(spreadsheet_name)
return spread_sheet
def get_regions():
region = "us-west-1"
regions = dict()
ec2 = boto3.client("ec2", region_name=region)
ec2_responses = ec2.describe_regions()
ssm_client = boto3.client('ssm', region_name=region)
for resp in ec2_responses['Regions']:
region_id = resp['RegionName']
tmp = '/aws/service/global-infrastructure/regions/%s/longName' % region_id
ssm_response = ssm_client.get_parameter(Name = tmp)
region_name = ssm_response['Parameter']['Value']
regions[region_id] = region_name
return(regions)
def rds_worksheet_creation(spread_sheet, regions, spreadsheet_index):
worksheet = spread_sheet.add_worksheet("RDS", rows=100, cols=26, src_tuple=None, src_worksheet=None, index=spreadsheet_index)
worksheet.cell('A1').set_text_format('bold', True).value = 'DBInstanceIdentifier'
worksheet.cell('B1').set_text_format('bold', True).value = 'MasterUsername'
worksheet.cell('C1').set_text_format('bold', True).value = 'Region'
worksheet.cell('D1').set_text_format('bold', False).value = 'Sent Query to (Name)'
worksheet.cell('E1').set_text_format('bold', False).value = 'Sent Query to (email)'
worksheet.cell('F1').set_text_format('bold', False).value = 'WorksheetCreated: %s' % currentDT
cells_data = list()
for region, region_h in sorted(regions.items()):
client = boto3.client('rds', region_name=region)
clnt = boto3.client('ssm', region_name=region)
db_instances = client.describe_db_instances()
for instance in db_instances['DBInstances']:
MasterUsername = instance['MasterUsername']
DBInstanceIdentifier = instance['DBInstanceIdentifier']
cells_data.append([DBInstanceIdentifier, MasterUsername, region_h])
worksheet.append_table(cells_data, start='A2')
if __name__ == "__main__":
spread_sheet = create_spreadsheet(spreadsheet_name = "jSonar AWS usage",
outh_file = '/home/qa/.aws/client_secret.json')
regions = get_regions()
rds_worksheet_creation(spread_sheet, regions, 0)
spread_sheet.share("me#corp.com")
Output:
If i understand correctly you want to un-bold multiple cells in single command.
To set format to a range of cells create a Datarange and use apply_format.
model_cell = Cell('A1')
model_cell.set_text_format('bold', False)
Datarange('A1','A10', worksheet=wks).apply_format(model_cell)
docs

Resources