Ansible Molecule fails to create the docker instance - python-3.x

I am trying to get Molecule working with Docker on a Centos7 host.
The 'molecule create' command fails to create docker instance, and gives error: 'Unsupported parameters for (community.docker.docker_container) module: ....'
Checking with 'docker ps -a' confirms instance was not created.
I can confirm starting containers manually with 'docker run' works
fine.
It's only the molecule failing to start the container
I am stuck with this, and your help is highly appreciated
Here is the complete error message:
TASK [Create molecule instance(s)] *********************************************
changed: [localhost] => (item=centos7)
TASK [Wait for instance(s) creation to complete] *******************************
FAILED - RETRYING: Wait for instance(s) creation to complete (300 retries left).
failed: [localhost] (item={'started': 1, 'finished': 0, 'ansible_job_id': '353704502720.15356', 'results_file': '/home/guest/.ansible_async/353704502720.15356', 'changed': True, 'failed': False, 'item': {'image': 'test-runner', 'name': 'centos7', 'pre_build_image': True}, 'ansible_loop_var': 'item'}) => {"ansible_job_id": "353704502720.15356", "ansible_loop_var": "item", "attempts": 2, "changed": false, "finished": 1, "item": {"ansible_job_id": "353704502720.15356", "ansible_loop_var": "item", "changed": true, "failed": false, "finished": 0, "item": {"image": "test-runner", "name": "centos7", "pre_build_image": true}, "results_file": "/home/guest/.ansible_async/353704502720.15356", "started": 1}, "msg": "Unsupported parameters for (community.docker.docker_container) module: command_handling Supported parameters include: api_version, auto_remove, blkio_weight, ca_cert, cap_drop, capabilities, cgroup_parent, cleanup, client_cert, client_key, command, comparisons, container_default_behavior, cpu_period, cpu_quota, cpu_shares, cpus, cpuset_cpus, cpuset_mems, debug, default_host_ip, detach, device_read_bps, device_read_iops, device_requests, device_write_bps, device_write_iops, devices, dns_opts, dns_search_domains, dns_servers, docker_host, domainname, entrypoint, env, env_file, etc_hosts, exposed_ports, force_kill, groups, healthcheck, hostname, ignore_image, image, init, interactive, ipc_mode, keep_volumes, kernel_memory, kill_signal, labels, links, log_driver, log_options, mac_address, memory, memory_reservation, memory_swap, memory_swappiness, mounts, name, network_mode, networks, networks_cli_compatible, oom_killer, oom_score_adj, output_logs, paused, pid_mode, pids_limit, privileged, published_ports, pull, purge_networks, read_only, recreate, removal_wait_timeout, restart, restart_policy, restart_retries, runtime, security_opts, shm_size, ssl_version, state, stop_signal, stop_timeout, sysctls, timeout, tls, tls_hostname, tmpfs, tty, ulimits, user, userns_mode, uts, validate_certs, volume_driver, volumes, volumes_from, working_dir", "stderr": "/tmp/ansible_community.docker.docker_container_payload_1djctes1/ansible_community.docker.docker_container_payload.zip/ansible_collections/community/docker/plugins/modules/docker_container.py:1193: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.\n", "stderr_lines": ["/tmp/ansible_community.docker.docker_container_payload_1djctes1/ansible_community.docker.docker_container_payload.zip/ansible_collections/community/docker/plugins/modules/docker_container.py:1193: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead."]}
$ pip list | egrep "mole|docker|ansible"
ansible 2.10.7
ansible-base 2.10.17
ansible-compat 2.2.0
ansible-core 2.12.0
docker 6.0.0
molecule 4.0.1
molecule-docker 2.0.0
$ docker --version
Docker version 20.10.14, build a224086
$ ansible --version
ansible 2.10.17
config file = /etc/ansible/ansible.cfg
configured module search path = ['/home/guest/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /home/guest/mole3/mole3/lib/python3.8/site-packages/ansible
executable location = /home/guest/mole3/mole3/bin/ansible
python version = 3.8.11 (default, Sep 1 2021, 12:33:46) [GCC 9.3.1 20200408 (Red Hat 9.3.1-2)]
$ molecule --version
molecule 4.0.1 using python 3.8
ansible:2.10.17
delegated:4.0.1 from molecule
docker:2.0.0 from molecule_docker requiring collections: community.docker>=3.0.0-a2
$ cat molecule/default/molecule.yml
---
dependency:
name: galaxy
enabled: False
driver:
name: docker
platforms:
- name: centos7
image: test-runner
pre_build_image: true
provisioner:
name: ansible
verifier:
name: ansible

Related

Ansible dnf module returns success even when dnf fails

This is the initial state of my system:
I have myrpm-2.0 installed.
This rpm creates just a file.
/usr/share/myfile exists as a file.
/root# rpm -q myrpm
myrpm-2.0-0.x86_64
/root# rpm -ql myrpm-2.0-0.x86_64
/usr/share
/usr/share/myfile
/root# ls -lrt /usr/share | grep myfile
-rw-r--r-- 1 root root 11 May 25 17:32 myfile
/root#
Now I am simulating an error condition. I manually removed the file and created a directory in it's place.
/root# ls -lrt /usr/share | grep myfile
drwxr-xr-x 2 root root 4096 May 25 18:33 myfile
/root#
I prepared a higher version 3.0 of the same rpm, which copies the same file.
/root# rpm -ql /root/update/myrpm-3.0-0.x86_64.rpm
/usr/share
/usr/share/myfile
/root#
4-Test-1: Trying rpm upgrade using ansible's builtin dnf module.
Following is my playbook:
/root# cat test.yaml
---
- hosts: localhost
tasks:
- name: update rpm
dnf:
name: "myrpm"
state: latest
/root#
Executing this playbook gives return code as 0.
/root# ansible-playbook -vvv test.yaml
...
changed: [localhost] => {
"changed": true,
"invocation": {
"module_args": {
"allow_downgrade": false,
"autoremove": false,
"bugfix": false,
"conf_file": null,
"disable_excludes": null,
"disable_gpg_check": false,
"disable_plugin": [],
"disablerepo": [],
"download_dir": null,
"download_only": false,
"enable_plugin": [],
"enablerepo": [],
"exclude": [],
"install_repoquery": true,
"install_weak_deps": true,
"installroot": "/",
"list": null,
"lock_timeout": 30,
"name": [
"myrpm"
],
"releasever": null,
"security": false,
"skip_broken": false,
"state": "latest",
"update_cache": false,
"update_only": false,
"validate_certs": true
}
},
"msg": "",
"rc": 0,
"results": [
"Installed: myrpm-3.0-0.x86_64",
"Removed: myrpm-2.0-0.x86_64"
]
}
META: ran handlers
META: ran handlers
PLAY RECAP *******************************************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
/root# echo $?
0
/root#
rc is 0 as per the verbose log.
However, I can see that dnf command has failed.
/root# dnf history | head -n 3
ID | Command line | Date and time | Action(s) | Altered
-------------------------------------------------------------------------------
27 | | 2022-05-25 18:39 | Upgrade | 1 EE
/root# dnf history info 27 | tail -6
Packages Altered:
** Upgrade myrpm-3.0-0.x86_64 #My_Update
Upgraded myrpm-2.0-0.x86_64 ##System
Scriptlet output:
1 error: unpacking of archive failed on file /usr/share/myfile: cpio: File from package already exists as a directory in system
2 error: myrpm-3.0-0.x86_64: install failed
/root#
4-Test-2: I made my system in exact same state as 3, and used direct dnf command instead of playbook.
/root# dnf update myrpm
...
Preparing : 1/1
Upgrading : myrpm-3.0-0.x86_64 1/2
Error unpacking rpm package myrpm-3.0-0.x86_64
error: unpacking of archive failed on file /usr/share/myfile: cpio: File from package already exists as a directory in system
Cleanup : myrpm-2.0-0.x86_64 2/2
error: myrpm-3.0-0.x86_64: install failed
Verifying : myrpm-3.0-0.x86_64 1/2
Verifying : myrpm-2.0-0.x86_64 2/2
Failed:
myrpm-3.0-0.x86_64
Error: Transaction failed
/root# echo $?
1
/root#
Any clue on why the playbook did not show task as failed?
Got to know that this is a bug fixed in higher versions of ansible.
Refer https://github.com/ansible/ansible/issues/77917

Validation Error in running Hyperledger Caliper V0.4.2 with Fabric 2.1

When I have been running the Caliper V0.4.2 for measuring the benchmarks of Fabric 2.1.0 1.4.7, I got the below error
ValidationError: child "version" fails because ["version" must be one of [1.0]]. child "clients" fails because ["clients" is required]. child "channels" fails because ["channels" must be an object]. child "organizations" fails because ["organizations" must be an object]. child "orderers" fails because ["orderers" is required]. child "peers" fails because ["peers" is required]
at Object.exports.process (/home/gow/fabric-benchmarks/caliper-benchmarks/node_modules/#hapi/joi/lib/errors.js:202:19)
at internals.Object._validateWithOptions (/home/gow/fabric-benchmarks/caliper-benchmarks/node_modules/#hapi/joi/lib/types/any/index.js:763:31)
at module.exports.internals.Any.root.validate (/home/gow/fabric-benchmarks/caliper-benchmarks/node_modules/#hapi/joi/lib/index.js:145:23)
at Function._validateTopLevel (/home/gow/fabric-benchmarks/caliper-benchmarks/node_modules/#hyperledger/caliper-fabric/lib/configValidator.js:208:26)
at Function.validateNetwork (/home/gow/fabric-benchmarks/caliper-benchmarks/node_modules/#hyperledger/caliper-fabric/lib/configValidator.js:58:25)
at new FabricConnector (/home/gow/fabric-benchmarks/caliper-benchmarks/node_modules/#hyperledger/caliper-fabric/lib/fabric-connector.js:107:25)
at CaliperEngine.connectorFactory [as adapterFactory] (/home/gow/fabric-benchmarks/caliper-benchmarks/node_modules/#hyperledger/caliper-fabric/lib/connectorFactory.js:26:23)
at CaliperEngine.run (/home/gow/fabric-benchmarks/caliper-benchmarks/node_modules/#hyperledger/caliper-core/lib/manager/caliper-engine.js:93:36)
at Function.handler (/home/gow/fabric-benchmarks/caliper-benchmarks/node_modules/#hyperledger/caliper-cli/lib/launch/lib/launchManager.js:62:43)
at Object.module.exports.handler (/home/gow/fabric-benchmarks/caliper-benchmarks/node_modules/#hyperledger/caliper-cli/lib/launch/launchManagerCommand.js:46:44)
isJoi: true,
name: 'ValidationError',
details:
[ { message: '"version" must be one of [1.0]',
path: [Array],
type: 'any.allowOnly',
context: [Object] },
{ message: '"clients" is required',
path: [Array],
type: 'any.required',
context: [Object] },
{ message: '"channels" must be an object',
path: [Array],
type: 'object.base',
context: [Object] },
{ message: '"organizations" must be an object',
path: [Array],
type: 'object.base',
context: [Object] },
{ message: '"orderers" is required',
path: [Array],
type: 'any.required',
context: [Object] },
{ message: '"peers" is required',
path: [Array],
type: 'any.required',
context: [Object] } ],
_object:
{ name: 'Caliper Benchmarks',
version: '2.1.0',
caliper: { blockchain: 'fabric' },
channels: [ [Object] ],
organizations: [ [Object] ] },
annotate: [Function] }
Command that I run are,
npm init -y
npm install --only=prod #hyperledger/caliper-cli#0.4.0
npx caliper bind --caliper-bind-sut fabric:2.1
./network.sh deployCC -ccn simple -ccp ../../caliper-benchmarks/src/fabric/scenario/simple/node -ccl javascript
npx caliper launch manager --caliper-workspace ./ --caliper-networkconfig networks/fabric/test-network.yaml --caliper-benchconfig benchmarks/scenario/simple/config.yaml --caliper-flow-only-test --caliper-fabric-gateway-enabled
Hyperledger network creation, channel creation and Chaincode deployment all were successful.
How to solve this error?
Looks like you are using the main branch of caliper-benchmarks. I would suggest you https://github.com/hyperledger/caliper-benchmarks/blob/main/networks/fabric/README.md
From what you have posted, you should use the latest available version of caliper which is 0.4.2, not 0.4.0. To do so the command you should issue is
npm install --only=prod #hyperledger/caliper-cli
You should bind to the latest 2.2 sdk (don't bind to 2.1 as fabric doesn't support 2.1). Note this is selecting the version of the sdk used to interact with a fabric network, not the version of the fabric network you are using. 2.2 will work with a fabric 2.2 and later 2.x versions
npx caliper bind --caliper-bind-sut fabric:2.2
Finally it looks like you tried to change the version number in the network configuration file. Don't change the version (it must be 2.0.0) as it refers to the version of the network configuration file and not the version of fabric you want to test
As you look to be using test-network, depending on which branch of fabric-samples you are using defines the version of fabric. The main branch tests the latest version of fabric which should have no problems.

Stormcrawler not retrieving all text content from web page

I'm attempting to use Stormcrawler to crawl a set of pages on our website, and while it is able to retrieve and index some of the page's text, it's not capturing a large amount of other text on the page.
I've installed Zookeeper, Apache Storm, and Stormcrawler using the Ansible playbooks provided here (thank you a million for those!) on a server running Ubuntu 18.04, along with Elasticsearch and Kibana. For the most part, I'm using the configuration defaults, but have made the following changes:
For the Elastic index mappings, I've enabled _source: true, and turned on indexing and storing for all properties (content, host, title, url)
In the crawler-conf.yaml configuration, I've commented out all textextractor.include.pattern and textextractor.exclude.tags settings, to enforce capturing the whole page
After re-creating fresh ES indices, running mvn clean package, and then starting the crawler topology, stormcrawler begins doing its thing and content starts appearing in Elasticsearch. However, for many pages, the content that's retrieved and indexed is only a subset of all the text on the page, and usually excludes the main page text we are interested in.
For example, the text in the following XML path is not returned/indexed:
<html> <body> <div#maincontentcontainer.container> <div#docs-container> <div> <div.row> <div.col-lg-9.col-md-8.col-sm-12.content-item> <div> <div> <p> (text)
While the text in this path is returned:
<html> <body> <div> <div.container> <div.row> <p> (text)
Are there any additional configuration changes that need to be made beyond commenting out all specific tag include and exclude patterns? From my understanding of the documentation, the default settings for those options are to enforce the whole page to be indexed.
I would greatly appreciate any help. Thank you for the excellent software.
Below are my configuration files:
crawler-conf.yaml
config:
topology.workers: 3
topology.message.timeout.secs: 1000
topology.max.spout.pending: 100
topology.debug: false
fetcher.threads.number: 100
# override the JVM parameters for the workers
topology.worker.childopts: "-Xmx2g -Djava.net.preferIPv4Stack=true"
# mandatory when using Flux
topology.kryo.register:
- com.digitalpebble.stormcrawler.Metadata
# metadata to transfer to the outlinks
# metadata.transfer:
# - customMetadataName
# lists the metadata to persist to storage
metadata.persist:
- _redirTo
- error.cause
- error.source
- isSitemap
- isFeed
http.agent.name: "My crawler"
http.agent.version: "1.0"
http.agent.description: ""
http.agent.url: ""
http.agent.email: ""
# The maximum number of bytes for returned HTTP response bodies.
http.content.limit: -1
# FetcherBolt queue dump => comment out to activate
# fetcherbolt.queue.debug.filepath: "/tmp/fetcher-dump-{port}"
parsefilters.config.file: "parsefilters.json"
urlfilters.config.file: "urlfilters.json"
# revisit a page daily (value in minutes)
fetchInterval.default: 1440
# revisit a page with a fetch error after 2 hours (value in minutes)
fetchInterval.fetch.error: 120
# never revisit a page with an error (or set a value in minutes)
fetchInterval.error: -1
# text extraction for JSoupParserBolt
# textextractor.include.pattern:
# - DIV[id="maincontent"]
# - DIV[itemprop="articleBody"]
# - ARTICLE
# textextractor.exclude.tags:
# - STYLE
# - SCRIPT
# configuration for the classes extending AbstractIndexerBolt
# indexer.md.filter: "someKey=aValue"
indexer.url.fieldname: "url"
indexer.text.fieldname: "content"
indexer.canonical.name: "canonical"
indexer.md.mapping:
- parse.title=title
- parse.keywords=keywords
- parse.description=description
- domain=domain
# Metrics consumers:
topology.metrics.consumer.register:
- class: "org.apache.storm.metric.LoggingMetricsConsumer"
parallelism.hint: 1
http.protocol.implementation: "com.digitalpebble.stormcrawler.protocol.selenium.RemoteDriverProtocol"
https.protocol.implementation: "com.digitalpebble.stormcrawler.protocol.selenium.RemoteDriverProtocol"
selenium.addresses: "http://localhost:9515"
es-conf.yaml
config:
# ES indexer bolt
es.indexer.addresses: "localhost"
es.indexer.index.name: "content"
# es.indexer.pipeline: "_PIPELINE_"
es.indexer.create: false
es.indexer.bulkActions: 100
es.indexer.flushInterval: "2s"
es.indexer.concurrentRequests: 1
# ES metricsConsumer
es.metrics.addresses: "http://localhost:9200"
es.metrics.index.name: "metrics"
# ES spout and persistence bolt
es.status.addresses: "http://localhost:9200"
es.status.index.name: "status"
es.status.routing: true
es.status.routing.fieldname: "key"
es.status.bulkActions: 500
es.status.flushInterval: "5s"
es.status.concurrentRequests: 1
# spout config #
# positive or negative filters parsable by the Lucene Query Parser
# es.status.filterQuery:
# - "-(key:stormcrawler.net)"
# - "-(key:digitalpebble.com)"
# time in secs for which the URLs will be considered for fetching after a ack of fail
spout.ttl.purgatory: 30
# Min time (in msecs) to allow between 2 successive queries to ES
spout.min.delay.queries: 2000
# Delay since previous query date (in secs) after which the nextFetchDate value will be reset to the current time
spout.reset.fetchdate.after: 120
es.status.max.buckets: 50
es.status.max.urls.per.bucket: 2
# field to group the URLs into buckets
es.status.bucket.field: "key"
# fields to sort the URLs within a bucket
es.status.bucket.sort.field:
- "nextFetchDate"
- "url"
# field to sort the buckets
es.status.global.sort.field: "nextFetchDate"
# CollapsingSpout : limits the deep paging by resetting the start offset for the ES query
es.status.max.start.offset: 500
# AggregationSpout : sampling improves the performance on large crawls
es.status.sample: false
# max allowed duration of a query in sec
es.status.query.timeout: -1
# AggregationSpout (expert): adds this value in mins to the latest date returned in the results and
# use it as nextFetchDate
es.status.recentDate.increase: -1
es.status.recentDate.min.gap: -1
topology.metrics.consumer.register:
- class: "com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer"
parallelism.hint: 1
#whitelist:
# - "fetcher_counter"
# - "fetcher_average.bytes_fetched"
#blacklist:
# - "__receive.*"
es-crawler.flux
name: "crawler"
includes:
- resource: true
file: "/crawler-default.yaml"
override: false
- resource: false
file: "crawler-conf.yaml"
override: true
- resource: false
file: "es-conf.yaml"
override: true
spouts:
- id: "spout"
className: "com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout"
parallelism: 10
- id: "filespout"
className: "com.digitalpebble.stormcrawler.spout.FileSpout"
parallelism: 1
constructorArgs:
- "."
- "seeds.txt"
- true
bolts:
- id: "filter"
className: "com.digitalpebble.stormcrawler.bolt.URLFilterBolt"
parallelism: 3
- id: "partitioner"
className: "com.digitalpebble.stormcrawler.bolt.URLPartitionerBolt"
parallelism: 3
- id: "fetcher"
className: "com.digitalpebble.stormcrawler.bolt.FetcherBolt"
parallelism: 3
- id: "sitemap"
className: "com.digitalpebble.stormcrawler.bolt.SiteMapParserBolt"
parallelism: 3
- id: "parse"
className: "com.digitalpebble.stormcrawler.bolt.JSoupParserBolt"
parallelism: 12
- id: "index"
className: "com.digitalpebble.stormcrawler.elasticsearch.bolt.IndexerBolt"
parallelism: 3
- id: "status"
className: "com.digitalpebble.stormcrawler.elasticsearch.persistence.StatusUpdaterBolt"
parallelism: 3
- id: "status_metrics"
className: "com.digitalpebble.stormcrawler.elasticsearch.metrics.StatusMetricsBolt"
parallelism: 3
streams:
- from: "spout"
to: "partitioner"
grouping:
type: SHUFFLE
- from: "spout"
to: "status_metrics"
grouping:
type: SHUFFLE
- from: "partitioner"
to: "fetcher"
grouping:
type: FIELDS
args: ["key"]
- from: "fetcher"
to: "sitemap"
grouping:
type: LOCAL_OR_SHUFFLE
- from: "sitemap"
to: "parse"
grouping:
type: LOCAL_OR_SHUFFLE
- from: "parse"
to: "index"
grouping:
type: LOCAL_OR_SHUFFLE
- from: "fetcher"
to: "status"
grouping:
type: FIELDS
args: ["url"]
streamId: "status"
- from: "sitemap"
to: "status"
grouping:
type: FIELDS
args: ["url"]
streamId: "status"
- from: "parse"
to: "status"
grouping:
type: FIELDS
args: ["url"]
streamId: "status"
- from: "index"
to: "status"
grouping:
type: FIELDS
args: ["url"]
streamId: "status"
- from: "filespout"
to: "filter"
grouping:
type: FIELDS
args: ["url"]
streamId: "status"
- from: "filter"
to: "status"
grouping:
streamId: "status"
type: CUSTOM
customClass:
className: "com.digitalpebble.stormcrawler.util.URLStreamGrouping"
constructorArgs:
- "byDomain"
parsefilters.json
{
"com.digitalpebble.stormcrawler.parse.ParseFilters": [
{
"class": "com.digitalpebble.stormcrawler.parse.filter.XPathFilter",
"name": "XPathFilter",
"params": {
"canonical": "//*[#rel=\"canonical\"]/#href",
"parse.description": [
"//*[#name=\"description\"]/#content",
"//*[#name=\"Description\"]/#content"
],
"parse.title": [
"//TITLE",
"//META[#name=\"title\"]/#content"
],
"parse.keywords": "//META[#name=\"keywords\"]/#content"
}
},
{
"class": "com.digitalpebble.stormcrawler.parse.filter.LinkParseFilter",
"name": "LinkParseFilter",
"params": {
"pattern": "//FRAME/#src"
}
},
{
"class": "com.digitalpebble.stormcrawler.parse.filter.DomainParseFilter",
"name": "DomainParseFilter",
"params": {
"key": "domain",
"byHost": false
}
},
{
"class": "com.digitalpebble.stormcrawler.parse.filter.CommaSeparatedToMultivaluedMetadata",
"name": "CommaSeparatedToMultivaluedMetadata",
"params": {
"keys": ["parse.keywords"]
}
}
]
}
Attempting to use Chromedriver
I installed the latest versions of Chromedriver and Google Chrome for Ubuntu.
First I start chromedriver in headless mode at localhost:9515 as the stormcrawler user (via a separate python shell, as shown below), and then I restart the stormcrawler topology (also as stormcrawler user) but end up with a stack of errors related to Chrome. The odd thing however is that I can confirm chromedriver is running OK within the Python shell directly, and I can confirm that both the driver and browser are actively running via ps -ef). This same stack of errors also occurs when I attempt to simply start chromedriver from the command line (i.e., chromedriver --headless &).
Starting chromedriver in headless mode (in python3 shell)
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--no-sandbox')
options.add_argument('--headless')
options.add_argument('--window-size=1200x600')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--disable-setuid-sandbox')
options.add_argument('--disable-extensions')
options.add_argument('--disable-infobars')
options.add_argument('--remote-debugging-port=9222')
options.add_argument('--user-data-dir=/home/stormcrawler/cache/google/chrome')
options.add_argument('--disable-gpu')
options.add_argument('--profile-directory=Default')
options.binary_location = '/usr/bin/google-chrome'
driver = webdriver.Chrome(chrome_options=options, port=9515, executable_path=r'/usr/bin/chromedriver')
Stack trace from starting stormcrawler topology
Run command: storm jar target/stormcrawler-1.0-SNAPSHOT.jar org.apache.storm.flux.Flux --local es-crawler.flux --sleep 60000
9486 [Thread-26-fetcher-executor[3 3]] ERROR o.a.s.util - Async loop died!
java.lang.RuntimeException: org.openqa.selenium.WebDriverException: unknown error: Chrome failed to start: exited abnormally.
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Build info: version: '4.0.0-alpha-6', revision: '5f43a29cfc'
System info: host: 'stormcrawler-dev', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.15.0-33-generic', java.version: '1.8.0_282'
Driver info: driver.version: RemoteWebDriver
remote stacktrace: #0 0x55d590b21e89 <unknown>
at com.digitalpebble.stormcrawler.protocol.selenium.RemoteDriverProtocol.configure(RemoteDriverProtocol.java:101) ~[stormcrawler-1.0-SNAPSHOT.jar:?]
at com.digitalpebble.stormcrawler.protocol.ProtocolFactory.<init>(ProtocolFactory.java:69) ~[stormcrawler-1.0-SNAPSHOT.jar:?]
at com.digitalpebble.stormcrawler.bolt.FetcherBolt.prepare(FetcherBolt.java:818) ~[stormcrawler-1.0-SNAPSHOT.jar:?]
at org.apache.storm.daemon.executor$fn__10180$fn__10193.invoke(executor.clj:803) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: org.openqa.selenium.WebDriverException: unknown error: Chrome failed to start: exited abnormally.
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
...
Confirming that chromedriver and chrome are both running and reachable
~/stormcrawler$ ps -ef | grep -i 'driver'
stormcr+ 18862 18857 0 14:28 pts/0 00:00:00 /usr/bin/chromedriver --port=9515
stormcr+ 18868 18862 0 14:28 pts/0 00:00:00 /usr/bin/google-chrome --disable-background-networking --disable-client-side-phishing-detection --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-gpu --disable-hang-monitor --disable-infobars --disable-popup-blocking --disable-prompt-on-repost --disable-setuid-sandbox --disable-sync --enable-automation --enable-blink-features=ShadowDOMV0 --enable-logging --headless --log-level=0 --no-first-run --no-sandbox --no-service-autorun --password-store=basic --profile-directory=Default --remote-debugging-port=9222 --test-type=webdriver --use-mock-keychain --user-data-dir=/home/stormcrawler/cache/google/chrome --window-size=1200x600
stormcr+ 18899 18877 0 14:28 pts/0 00:00:00 /opt/google/chrome/chrome --type=renderer --no-sandbox --disable-dev-shm-usage --enable-automation --enable-logging --log-level=0 --remote-debugging-port=9222 --test-type=webdriver --allow-pre-commit-input --ozone-platform=headless --field-trial-handle=17069524199442920904,10206176048672570859,131072 --disable-gpu-compositing --enable-blink-features=ShadowDOMV0 --lang=en-US --headless --enable-crash-reporter --lang=en-US --num-raster-threads=1 --renderer-client-id=4 --shared-files=v8_context_snapshot_data:100
~/stormcrawler$ sudo netstat -lp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 localhost:9222 0.0.0.0:* LISTEN 18026/google-chrome
tcp 0 0 localhost:9515 0.0.0.0:* LISTEN 18020/chromedriver
IIRC you need to set some additional config to work with ChomeDriver.
Alternatively (haven't tried yet) https://hub.docker.com/r/browserless/chrome would be a nice way of handling Chrome in a Docker container.

Ansible K8s Install Fails - No module named Kubernetes, Failed to import the required Python Library

I am running my playbook with the following command:
ansible-playbook kubernetes_cluster.yml -b --ask-pass -vvv
My error message is:
Traceback (most recent call last):
File "/tmp/ansible_k8s_payload_2BRCdT/ansible_k8s_payload.zip/ansible/module_utils/k8s/common.py", line 33, in <module>
import kubernetes
ImportError: No module named kubernetes
fatal: [kube1.idm.nac-issa.org]: FAILED! => {
"changed": false,
"error": "No module named kubernetes",
"invocation": {
"module_args": {
"api_key": null,
"api_version": "v1",
"append_hash": false,
"apply": false,
"ca_cert": null,
"client_cert": null,
"client_key": null,
"context": null,
"force": false,
"host": null,
"kind": "Namespace",
"kubeconfig": "/root/.kube/config",
"merge_type": null,
"name": "",
"namespace": null,
"password": null,
"proxy": null,
"resource_definition": null,
"src": null,
"state": "present",
"username": null,
"validate": null,
"validate_certs": null,
"wait": false,
"wait_condition": null,
"wait_sleep": 5,
"wait_timeout": 120
}
},
"msg": "Failed to import the required Python library (openshift) on kube1.idm.nac-issa.org's Python /usr/bin/python. Please read module documentation and install in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"
My environment:
ansible 2.9.16
config file = /home/ansuser/ansible/ansible.cfg
configured module search path = ['/home/ansuser/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/lib/python3.6/site-packages/ansible
executable location = /usr/bin/ansible
python version = 3.6.8 (default, Apr 16 2020, 01:36:27) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
I followed the directions from a similar question and performed the pip installs on the target host (not the management host)-
Ansible K8s module: Failed to import the required Python library (openshift) on Python /usr/bin/python3
and Cannot execute k8s module
but that hasn't made any difference
pip3 install openshift pyyaml kubernetes and also `sudo pip3 install --upgrade --user openshift`
I'm stuck figuring out what to try next. Any ideas are appreciated!

Sauce Labs OnDemand: multiCapabilities build information is not accepted

We are running mocha nodejs tests with protractor on Sauce Labs and the attribute 'build' is being ignored. This is my protractor.conf.js
multiCapabilities: [
{
name: 'chrome-tests-windows',
browserName: 'chrome',
version: '52',
platform: 'Windows 7',
shardTestFiles: true,
maxInstances: 25,
build: 'solisko-1',
tags: ['tag1', 'tag2']
}]
When I look at the metadata in Sauce Labs Dashboard I see this:
Build null
Tags ["tag1","tag2"]
Browser googlechrome
Browser version 52.0.2743.82
OS Windows 2008
Any idea why the build name is ignored?
I guess, for the build to be populated on Sauce platform. You need to send the below property in Protractor config file. Check here
/** * Use sauceBuild if you want to group test capabilities by a
build ID */
sauceBuild?: string;
I dont see a build option in capabilities object as per the documentation here
From different source I found out that this is the correct attribute:
multiCapabilities: [
{
Build: 'some-build-name',
BuildName: 'some-build-name',
name: 'chrome-tests-mac',
browserName: 'chrome',
version: '52',
platform: 'OS X 10.12',
shardTestFiles: true,
maxInstances: 25
}
]

Resources