We are having an issue with the DB2 connection is successful yet hangs -- no errors nor does code continue to the next line of execution.....broke this down to just running the CLI to perform a basic validate and it shows the same symptom.
dsadmin#usc1:/opt/ibm/clidriver/bin$ db2cli writecfg add -database DBNAME -host 10.0.0.444 -port 9470 -parameter "SecurityTransportMode=SSL"
dsadmin#usc1:/opt/ibm/clidriver/bin$ db2cli validate -dsn MAX_DEV -connect -user UID -passwd PWD
===============================================================================
Client information for the current copy:
===============================================================================
Client Package Type : IBM Data Server Driver For ODBC and CLI
Client Version (level/bit): DB2 v10.5.0.5 (special_33523/64-bit)
Client Platform : Linux/X8664
Install/Instance Path : /opt/ibm/clidriver
DB2DSDRIVER_CFG_PATH value: <not-set>
db2dsdriver.cfg Path : /opt/ibm/clidriver/cfg/db2dsdriver.cfg
DB2CLIINIPATH value : <not-set>
db2cli.ini Path : /opt/ibm/clidriver/cfg/db2cli.ini
db2diag.log Path : /opt/ibm/clidriver/db2dump/db2diag.log
===============================================================================
db2dsdriver.cfg schema validation for the entire file:
===============================================================================
Success: The schema validation completed successfully without any errors.
===============================================================================
The output of the validate command ends with the "Success" message however does not return to command line....hangs....
===========================DB2 CLI TRACE==================================
dsadmin#usc1:/opt/ibm/clidriver/bin$ vi 'clitrace;'
[ db2cli.ini Location: /opt/ibm/clidriver/cfg/db2cli.ini ]
[ db2dsdriver.cfg Location: /opt/ibm/clidriver/cfg/db2dsdriver.cfg ]
[ CLI Driver Type: IBM Data Server Driver For ODBC and CLI ]
SQLAllocEnv( phEnv=<NULL pointer> )
---> Time elapsed - 0 seconds
SQLAllocEnv( )
<--- SQL_ERROR Time elapsed - +1.100000E-005 seconds
SQLAllocEnv( phEnv=&00007ffd94147934 )
---> Time elapsed - +3.130975E+001 seconds
SQLAllocEnv( phEnv=0:1 )
<--- SQL_SUCCESS Time elapsed - +1.780000E-004 seconds
SQLFreeEnv( hEnv=0:1 )
---> Time elapsed - +2.500000E-005 seconds
SQLFreeEnv( )
<--- SQL_SUCCESS Time elapsed - +2.100000E-005 seconds
================END OF FILE==================================
Related
I am currently working on Azure Device Update using layers meta-azure-device-update and meta-swupdate. I want to run a post-install script. I have followed sources mentioned below:
1.sw-description.rst
https://git.rigado.com/vesta/swupdate/-/blob/acf50e361a8752db48e69ffe3c20a167c402d35f/doc/source/sw-description.rst#board-specific-settings
2.adu-swupdate.sh
https://github.com/Azure/iot-hub-device-update/blob/main/src/adu-shell/scripts/adu-swupdate.sh
The image was built successfully and I was able to locate adu-swupdate.sh in .swu file which I provided Azure Device Update. The install failed giving below mentioned error:
Sep 21 07:21:30 rpi AducIotAgent[281]: -> 07:21:29 PUBLISH | IS_DUP: false | RETAIN: 0 | QOS: DELIVER_AT_MOST_ONCE | TOPIC_NAME: $iothub/twin2021-09-21T07:21:30.2396Z [E] Install failed, extendedResultCode = 1 [Install]
Sep 21 07:21:30 rpi AducIotAgent[281]: 2021-09-21T07:21:30.2398Z [E] Install failed. error 0, 1 - Expecting service to send Cancel action [ADUC_Workflow_WorkCompletionCallback]
The sw-update log is as given below:
Swupdate v2021.04.0
Licensed under GPLv2. See source distribution for detailed copyright notices.
[INFO ] : SWUPDATE running : [main] : Running on raspberrypi4 Revision 1.0
[INFO ] : SWUPDATE running : [print_registered_handlers] : Registered handlers:
[INFO ] : SWUPDATE running : [print_registered_handlers] : dummy
[INFO ] : SWUPDATE running : [print_registered_handlers] : archive
[INFO ] : SWUPDATE running : [print_registered_handlers] : tar
[INFO ] : SWUPDATE running : [print_registered_handlers] : uboot
[INFO ] : SWUPDATE running : [print_registered_handlers] : bootloader
[INFO ] : SWUPDATE running : [print_registered_handlers] : raw
[INFO ] : SWUPDATE running : [print_registered_handlers] : rawfile
[INFO ] : SWUPDATE running : [print_registered_handlers] : rawcopy
[INFO ] : SWUPDATE running : [main] : software set: stable mode: copy2
[TRACE] : SWUPDATE running : [listener_create] : creating socket at /tmp/swupdateprog
[TRACE] : SWUPDATE running : [network_initializer] : Main loop daemon
[TRACE] : SWUPDATE running : [listener_create] : creating socket at /tmp/sockinstctrl
[TRACE] : SWUPDATE running : [network_thread] : Incoming network request: processing...
[INFO ] : SWUPDATE started : Software Update started !
[TRACE] : SWUPDATE running : [network_initializer] : Software update started
[WARN ] : SWUPDATE running : [scan_mtd_devices] : MTD is not present on the target
[TRACE] : SWUPDATE running : [extract_file_to_tmp] : Found file
[TRACE] : SWUPDATE running : [extract_file_to_tmp] : filename sw-description
[TRACE] : SWUPDATE running : [extract_file_to_tmp] : size 1144
[TRACE] : SWUPDATE running : [extract_file_to_tmp] : Found file
[TRACE] : SWUPDATE running : [extract_file_to_tmp] : filename sw-description.sig
[TRACE] : SWUPDATE running : [extract_file_to_tmp] : size 256
[TRACE] : SWUPDATE running : [swupdate_verify_file] : Verify signed image: Read 1144 bytes
[TRACE] : SWUPDATE running : [swupdate_verify_file] : Verified OK
[TRACE] : SWUPDATE running : [get_common_fields] : Version 0.1.0.1
[TRACE] : SWUPDATE running : [parse_hw_compatibility] : Accepted Hw Revision : 1.0
[TRACE] : SWUPDATE running : [_parse_images] : Found compressed Image: core-image-base-raspberrypi4.ext4.gz in device : /dev/mmcblk0p3 for handler raw
[TRACE] : SWUPDATE running : [_parse_scripts] : Found Script: adu-swupdate.sh
[ERROR] : SWUPDATE failed [0] ERROR : feature 'postinstall' required for 'adu-swupdate.sh' in sw-description is absent!
[ERROR] : SWUPDATE failed [0] ERROR : Compatible SW not found
[ERROR] : SWUPDATE failed [1] Image invalid or corrupted. Not installing ...
[TRACE] : SWUPDATE running : [network_initializer] : Main thread sleep again !
[INFO ] : No SWUPDATE running : Waiting for requests...
[INFO ] : SWUPDATE running : [endupdate] : Swupdate *failed* !```
So, After hours of exploration and reading each page of SW-UPDATE Wiki, I figured out that there are handlers for each function which we must enable before using it.
you can read more about them here. https://sbabic.github.io/swupdate/handlers.html
These handlers are available in the meta-swupdate/recipes-support/swupdate/defconfig
#
# Automatically generated file; DO NOT EDIT.
# Swupdate Configuration
#
CONFIG_HAVE_DOT_CONFIG=y
#
# Swupdate Settings
#
#
# General Configuration
#
# CONFIG_CURL is not set
# CONFIG_CURL_SSL is not set
# CONFIG_SYSTEMD is not set
CONFIG_DEFAULT_CONFIG_FILE="/etc/swupdate.cfg"
CONFIG_SCRIPTS=y
CONFIG_HW_COMPATIBILITY=y
CONFIG_HW_COMPATIBILITY_FILE="/etc/hwrevision"
CONFIG_SW_VERSIONS_FILE="/etc/sw-versions"
#
# Socket Paths
#
CONFIG_SOCKET_CTRL_PATH=""
CONFIG_SOCKET_PROGRESS_PATH=""
CONFIG_SOCKET_REMOTE_HANDLER_DIRECTORY="/tmp/"
CONFIG_MTD=y
CONFIG_LUA=y
CONFIG_LUAPKG="lua"
# CONFIG_FEATURE_SYSLOG is not set
#
# Build Options
#
CONFIG_CROSS_COMPILE=""
CONFIG_SYSROOT=""
CONFIG_EXTRA_LDLIBS=""
#
# Debugging Options
#
# CONFIG_DEBUG is not set
# CONFIG_WERROR is not set
# CONFIG_NOCLEANUP is not set
# CONFIG_BOOTLOADER_EBG is not set
CONFIG_UBOOT=y
# CONFIG_BOOTLOADER_NONE is not set
# CONFIG_BOOTLOADER_GRUB is not set
CONFIG_UBOOT_FWENV="/etc/fw_env.config"
CONFIG_UBOOT_DEFAULTENV="/etc/u-boot-initial-env"
# CONFIG_SSL_IMPL_NONE is not set
CONFIG_SSL_IMPL_OPENSSL=y
# CONFIG_SSL_IMPL_MBEDTLS is not set
# CONFIG_DOWNLOAD is not set
# CONFIG_HASH_VERIFY is not set
# CONFIG_SIGNED_IMAGES is not set
# CONFIG_ENCRYPTED_IMAGES is not set
# CONFIG_SURICATTA is not set
CONFIG_WEBSERVER=y
CONFIG_MONGOOSE=y
CONFIG_MONGOOSEIPV6=y
CONFIG_MONGOOSESSL=y
CONFIG_GUNZIP=y
# CONFIG_ZSTD is not set
#
# Parser Features
#
CONFIG_LIBCONFIG=y
CONFIG_PARSERROOT=""
# CONFIG_JSON is not set
# CONFIG_LUAEXTERNAL is not set
# CONFIG_SETSWDESCRIPTION is not set
#
# Image Handlers
#
# CONFIG_UBIVOL is not set
CONFIG_CFI=y
# CONFIG_CFIHAMMING1 is not set
# CONFIG_DISKPART is not set
CONFIG_RAW=y
# CONFIG_RDIFFHANDLER is not set
CONFIG_LUASCRIPTHANDLER=y
CONFIG_SHELLSCRIPTHANDLER=y
# CONFIG_HANDLER_IN_LUA is not set
# CONFIG_ARCHIVE is not set
# CONFIG_REMOTE_HANDLER is not set
# CONFIG_SWUFORWARDER_HANDLER is not set
# CONFIG_BOOTLOADERHANDLER is not set
# CONFIG_SSBLSWITCH is not set
# CONFIG_UCFWHANDLER is not set
So to enable post and preinstall script feature you should edit this defconfig file at
CONFIG_SHELLSCRIPTHANDLER=y
This will enable the post and preinstall script feature for your swupdate.
I'm attempting to use Stormcrawler to crawl a set of pages on our website, and while it is able to retrieve and index some of the page's text, it's not capturing a large amount of other text on the page.
I've installed Zookeeper, Apache Storm, and Stormcrawler using the Ansible playbooks provided here (thank you a million for those!) on a server running Ubuntu 18.04, along with Elasticsearch and Kibana. For the most part, I'm using the configuration defaults, but have made the following changes:
For the Elastic index mappings, I've enabled _source: true, and turned on indexing and storing for all properties (content, host, title, url)
In the crawler-conf.yaml configuration, I've commented out all textextractor.include.pattern and textextractor.exclude.tags settings, to enforce capturing the whole page
After re-creating fresh ES indices, running mvn clean package, and then starting the crawler topology, stormcrawler begins doing its thing and content starts appearing in Elasticsearch. However, for many pages, the content that's retrieved and indexed is only a subset of all the text on the page, and usually excludes the main page text we are interested in.
For example, the text in the following XML path is not returned/indexed:
<html> <body> <div#maincontentcontainer.container> <div#docs-container> <div> <div.row> <div.col-lg-9.col-md-8.col-sm-12.content-item> <div> <div> <p> (text)
While the text in this path is returned:
<html> <body> <div> <div.container> <div.row> <p> (text)
Are there any additional configuration changes that need to be made beyond commenting out all specific tag include and exclude patterns? From my understanding of the documentation, the default settings for those options are to enforce the whole page to be indexed.
I would greatly appreciate any help. Thank you for the excellent software.
Below are my configuration files:
crawler-conf.yaml
config:
topology.workers: 3
topology.message.timeout.secs: 1000
topology.max.spout.pending: 100
topology.debug: false
fetcher.threads.number: 100
# override the JVM parameters for the workers
topology.worker.childopts: "-Xmx2g -Djava.net.preferIPv4Stack=true"
# mandatory when using Flux
topology.kryo.register:
- com.digitalpebble.stormcrawler.Metadata
# metadata to transfer to the outlinks
# metadata.transfer:
# - customMetadataName
# lists the metadata to persist to storage
metadata.persist:
- _redirTo
- error.cause
- error.source
- isSitemap
- isFeed
http.agent.name: "My crawler"
http.agent.version: "1.0"
http.agent.description: ""
http.agent.url: ""
http.agent.email: ""
# The maximum number of bytes for returned HTTP response bodies.
http.content.limit: -1
# FetcherBolt queue dump => comment out to activate
# fetcherbolt.queue.debug.filepath: "/tmp/fetcher-dump-{port}"
parsefilters.config.file: "parsefilters.json"
urlfilters.config.file: "urlfilters.json"
# revisit a page daily (value in minutes)
fetchInterval.default: 1440
# revisit a page with a fetch error after 2 hours (value in minutes)
fetchInterval.fetch.error: 120
# never revisit a page with an error (or set a value in minutes)
fetchInterval.error: -1
# text extraction for JSoupParserBolt
# textextractor.include.pattern:
# - DIV[id="maincontent"]
# - DIV[itemprop="articleBody"]
# - ARTICLE
# textextractor.exclude.tags:
# - STYLE
# - SCRIPT
# configuration for the classes extending AbstractIndexerBolt
# indexer.md.filter: "someKey=aValue"
indexer.url.fieldname: "url"
indexer.text.fieldname: "content"
indexer.canonical.name: "canonical"
indexer.md.mapping:
- parse.title=title
- parse.keywords=keywords
- parse.description=description
- domain=domain
# Metrics consumers:
topology.metrics.consumer.register:
- class: "org.apache.storm.metric.LoggingMetricsConsumer"
parallelism.hint: 1
http.protocol.implementation: "com.digitalpebble.stormcrawler.protocol.selenium.RemoteDriverProtocol"
https.protocol.implementation: "com.digitalpebble.stormcrawler.protocol.selenium.RemoteDriverProtocol"
selenium.addresses: "http://localhost:9515"
es-conf.yaml
config:
# ES indexer bolt
es.indexer.addresses: "localhost"
es.indexer.index.name: "content"
# es.indexer.pipeline: "_PIPELINE_"
es.indexer.create: false
es.indexer.bulkActions: 100
es.indexer.flushInterval: "2s"
es.indexer.concurrentRequests: 1
# ES metricsConsumer
es.metrics.addresses: "http://localhost:9200"
es.metrics.index.name: "metrics"
# ES spout and persistence bolt
es.status.addresses: "http://localhost:9200"
es.status.index.name: "status"
es.status.routing: true
es.status.routing.fieldname: "key"
es.status.bulkActions: 500
es.status.flushInterval: "5s"
es.status.concurrentRequests: 1
# spout config #
# positive or negative filters parsable by the Lucene Query Parser
# es.status.filterQuery:
# - "-(key:stormcrawler.net)"
# - "-(key:digitalpebble.com)"
# time in secs for which the URLs will be considered for fetching after a ack of fail
spout.ttl.purgatory: 30
# Min time (in msecs) to allow between 2 successive queries to ES
spout.min.delay.queries: 2000
# Delay since previous query date (in secs) after which the nextFetchDate value will be reset to the current time
spout.reset.fetchdate.after: 120
es.status.max.buckets: 50
es.status.max.urls.per.bucket: 2
# field to group the URLs into buckets
es.status.bucket.field: "key"
# fields to sort the URLs within a bucket
es.status.bucket.sort.field:
- "nextFetchDate"
- "url"
# field to sort the buckets
es.status.global.sort.field: "nextFetchDate"
# CollapsingSpout : limits the deep paging by resetting the start offset for the ES query
es.status.max.start.offset: 500
# AggregationSpout : sampling improves the performance on large crawls
es.status.sample: false
# max allowed duration of a query in sec
es.status.query.timeout: -1
# AggregationSpout (expert): adds this value in mins to the latest date returned in the results and
# use it as nextFetchDate
es.status.recentDate.increase: -1
es.status.recentDate.min.gap: -1
topology.metrics.consumer.register:
- class: "com.digitalpebble.stormcrawler.elasticsearch.metrics.MetricsConsumer"
parallelism.hint: 1
#whitelist:
# - "fetcher_counter"
# - "fetcher_average.bytes_fetched"
#blacklist:
# - "__receive.*"
es-crawler.flux
name: "crawler"
includes:
- resource: true
file: "/crawler-default.yaml"
override: false
- resource: false
file: "crawler-conf.yaml"
override: true
- resource: false
file: "es-conf.yaml"
override: true
spouts:
- id: "spout"
className: "com.digitalpebble.stormcrawler.elasticsearch.persistence.AggregationSpout"
parallelism: 10
- id: "filespout"
className: "com.digitalpebble.stormcrawler.spout.FileSpout"
parallelism: 1
constructorArgs:
- "."
- "seeds.txt"
- true
bolts:
- id: "filter"
className: "com.digitalpebble.stormcrawler.bolt.URLFilterBolt"
parallelism: 3
- id: "partitioner"
className: "com.digitalpebble.stormcrawler.bolt.URLPartitionerBolt"
parallelism: 3
- id: "fetcher"
className: "com.digitalpebble.stormcrawler.bolt.FetcherBolt"
parallelism: 3
- id: "sitemap"
className: "com.digitalpebble.stormcrawler.bolt.SiteMapParserBolt"
parallelism: 3
- id: "parse"
className: "com.digitalpebble.stormcrawler.bolt.JSoupParserBolt"
parallelism: 12
- id: "index"
className: "com.digitalpebble.stormcrawler.elasticsearch.bolt.IndexerBolt"
parallelism: 3
- id: "status"
className: "com.digitalpebble.stormcrawler.elasticsearch.persistence.StatusUpdaterBolt"
parallelism: 3
- id: "status_metrics"
className: "com.digitalpebble.stormcrawler.elasticsearch.metrics.StatusMetricsBolt"
parallelism: 3
streams:
- from: "spout"
to: "partitioner"
grouping:
type: SHUFFLE
- from: "spout"
to: "status_metrics"
grouping:
type: SHUFFLE
- from: "partitioner"
to: "fetcher"
grouping:
type: FIELDS
args: ["key"]
- from: "fetcher"
to: "sitemap"
grouping:
type: LOCAL_OR_SHUFFLE
- from: "sitemap"
to: "parse"
grouping:
type: LOCAL_OR_SHUFFLE
- from: "parse"
to: "index"
grouping:
type: LOCAL_OR_SHUFFLE
- from: "fetcher"
to: "status"
grouping:
type: FIELDS
args: ["url"]
streamId: "status"
- from: "sitemap"
to: "status"
grouping:
type: FIELDS
args: ["url"]
streamId: "status"
- from: "parse"
to: "status"
grouping:
type: FIELDS
args: ["url"]
streamId: "status"
- from: "index"
to: "status"
grouping:
type: FIELDS
args: ["url"]
streamId: "status"
- from: "filespout"
to: "filter"
grouping:
type: FIELDS
args: ["url"]
streamId: "status"
- from: "filter"
to: "status"
grouping:
streamId: "status"
type: CUSTOM
customClass:
className: "com.digitalpebble.stormcrawler.util.URLStreamGrouping"
constructorArgs:
- "byDomain"
parsefilters.json
{
"com.digitalpebble.stormcrawler.parse.ParseFilters": [
{
"class": "com.digitalpebble.stormcrawler.parse.filter.XPathFilter",
"name": "XPathFilter",
"params": {
"canonical": "//*[#rel=\"canonical\"]/#href",
"parse.description": [
"//*[#name=\"description\"]/#content",
"//*[#name=\"Description\"]/#content"
],
"parse.title": [
"//TITLE",
"//META[#name=\"title\"]/#content"
],
"parse.keywords": "//META[#name=\"keywords\"]/#content"
}
},
{
"class": "com.digitalpebble.stormcrawler.parse.filter.LinkParseFilter",
"name": "LinkParseFilter",
"params": {
"pattern": "//FRAME/#src"
}
},
{
"class": "com.digitalpebble.stormcrawler.parse.filter.DomainParseFilter",
"name": "DomainParseFilter",
"params": {
"key": "domain",
"byHost": false
}
},
{
"class": "com.digitalpebble.stormcrawler.parse.filter.CommaSeparatedToMultivaluedMetadata",
"name": "CommaSeparatedToMultivaluedMetadata",
"params": {
"keys": ["parse.keywords"]
}
}
]
}
Attempting to use Chromedriver
I installed the latest versions of Chromedriver and Google Chrome for Ubuntu.
First I start chromedriver in headless mode at localhost:9515 as the stormcrawler user (via a separate python shell, as shown below), and then I restart the stormcrawler topology (also as stormcrawler user) but end up with a stack of errors related to Chrome. The odd thing however is that I can confirm chromedriver is running OK within the Python shell directly, and I can confirm that both the driver and browser are actively running via ps -ef). This same stack of errors also occurs when I attempt to simply start chromedriver from the command line (i.e., chromedriver --headless &).
Starting chromedriver in headless mode (in python3 shell)
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--no-sandbox')
options.add_argument('--headless')
options.add_argument('--window-size=1200x600')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--disable-setuid-sandbox')
options.add_argument('--disable-extensions')
options.add_argument('--disable-infobars')
options.add_argument('--remote-debugging-port=9222')
options.add_argument('--user-data-dir=/home/stormcrawler/cache/google/chrome')
options.add_argument('--disable-gpu')
options.add_argument('--profile-directory=Default')
options.binary_location = '/usr/bin/google-chrome'
driver = webdriver.Chrome(chrome_options=options, port=9515, executable_path=r'/usr/bin/chromedriver')
Stack trace from starting stormcrawler topology
Run command: storm jar target/stormcrawler-1.0-SNAPSHOT.jar org.apache.storm.flux.Flux --local es-crawler.flux --sleep 60000
9486 [Thread-26-fetcher-executor[3 3]] ERROR o.a.s.util - Async loop died!
java.lang.RuntimeException: org.openqa.selenium.WebDriverException: unknown error: Chrome failed to start: exited abnormally.
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
Build info: version: '4.0.0-alpha-6', revision: '5f43a29cfc'
System info: host: 'stormcrawler-dev', ip: '127.0.0.1', os.name: 'Linux', os.arch: 'amd64', os.version: '4.15.0-33-generic', java.version: '1.8.0_282'
Driver info: driver.version: RemoteWebDriver
remote stacktrace: #0 0x55d590b21e89 <unknown>
at com.digitalpebble.stormcrawler.protocol.selenium.RemoteDriverProtocol.configure(RemoteDriverProtocol.java:101) ~[stormcrawler-1.0-SNAPSHOT.jar:?]
at com.digitalpebble.stormcrawler.protocol.ProtocolFactory.<init>(ProtocolFactory.java:69) ~[stormcrawler-1.0-SNAPSHOT.jar:?]
at com.digitalpebble.stormcrawler.bolt.FetcherBolt.prepare(FetcherBolt.java:818) ~[stormcrawler-1.0-SNAPSHOT.jar:?]
at org.apache.storm.daemon.executor$fn__10180$fn__10193.invoke(executor.clj:803) ~[storm-core-1.2.3.jar:1.2.3]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.2.3.jar:1.2.3]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_282]
Caused by: org.openqa.selenium.WebDriverException: unknown error: Chrome failed to start: exited abnormally.
(unknown error: DevToolsActivePort file doesn't exist)
(The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
...
Confirming that chromedriver and chrome are both running and reachable
~/stormcrawler$ ps -ef | grep -i 'driver'
stormcr+ 18862 18857 0 14:28 pts/0 00:00:00 /usr/bin/chromedriver --port=9515
stormcr+ 18868 18862 0 14:28 pts/0 00:00:00 /usr/bin/google-chrome --disable-background-networking --disable-client-side-phishing-detection --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-gpu --disable-hang-monitor --disable-infobars --disable-popup-blocking --disable-prompt-on-repost --disable-setuid-sandbox --disable-sync --enable-automation --enable-blink-features=ShadowDOMV0 --enable-logging --headless --log-level=0 --no-first-run --no-sandbox --no-service-autorun --password-store=basic --profile-directory=Default --remote-debugging-port=9222 --test-type=webdriver --use-mock-keychain --user-data-dir=/home/stormcrawler/cache/google/chrome --window-size=1200x600
stormcr+ 18899 18877 0 14:28 pts/0 00:00:00 /opt/google/chrome/chrome --type=renderer --no-sandbox --disable-dev-shm-usage --enable-automation --enable-logging --log-level=0 --remote-debugging-port=9222 --test-type=webdriver --allow-pre-commit-input --ozone-platform=headless --field-trial-handle=17069524199442920904,10206176048672570859,131072 --disable-gpu-compositing --enable-blink-features=ShadowDOMV0 --lang=en-US --headless --enable-crash-reporter --lang=en-US --num-raster-threads=1 --renderer-client-id=4 --shared-files=v8_context_snapshot_data:100
~/stormcrawler$ sudo netstat -lp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 localhost:9222 0.0.0.0:* LISTEN 18026/google-chrome
tcp 0 0 localhost:9515 0.0.0.0:* LISTEN 18020/chromedriver
IIRC you need to set some additional config to work with ChomeDriver.
Alternatively (haven't tried yet) https://hub.docker.com/r/browserless/chrome would be a nice way of handling Chrome in a Docker container.
I'm working in osx with SQL Server using a docker image to be able to use it, running:
docker run -d --name sqlserver -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=myStrongPass' -e 'MSSQL_PID=Developer' -p 1433:1433 microsoft/mssql-server-linux:2017-latest
I can connect successfully in Azure Data Studio GUI with the following configuration
But the connection does not works in my nodejs code using mssql module.
const poolConnection = new sql.ConnectionPool({
database: 'myDbTest',
server: 'localhost',
port: 1433,
password: '*******',
user: 'sa',
connectionTimeout: 5000,
options: {
encrypt: false,
},
});
const [error, connection] = await to(poolConnection.connect());
The error always is the same:
ConnectionError: Login failed for user 'sa'
Is my first time working with SQL Server and is confusing for me the fact that I can connect correctly in the Azure Studio GUI but I can't do it in code.
I'm trying create new login users with CREATE LOGIN and give them privileges based on other post here in stackoverflow but nothing seems to work.
UPDATE:
I realize that i can connect correctly if i put master in database key.
Example:
const poolConnection = new sql.ConnectionPool({
database: 'master', <- Update here
server: 'localhost',
port: 1433,
password: '*******',
user: 'sa',
connectionTimeout: 5000,
options: {
encrypt: false,
},
});
1) Db that i can connect
2) Db that i want to connect but i can't.
Container error
2020-03-18 03:59:14.11 Logon Login failed for user 'sa'. Reason: Failed to open the explicitly specified database 'DoctorHoyCRM'. [CLIENT: 172.17.0.1]
I suspect a lot of people miss the sa password complexity requirement:
The password should follow the SQL Server default password policy, otherwise the container can not setup SQL server and will stop working. By default, the password must be at least 8 characters long and contain characters from three of the following four sets: Uppercase letters, Lowercase letters, Base 10 digits, and Symbols. You can examine the error log by executing the docker logs command.
An example based on: Quickstart: Run SQL Server container images with Docker
docker pull mcr.microsoft.com/mssql/server:2017-latest
docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=myStr0ngP4ssw0rd" -e "MSSQL_PID=Developer" -p 1433:1433 --name sqlserver -d mcr.microsoft.com/mssql/server:2017-latest
docker start sqlserver
Checking that the docker image is running (it should not say "Exited" under STATUS)...
docker ps -a
# CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
# af9f01eacab2 mcr.microsoft.com/mssql/server:2017-latest "/opt/mssql/bin/nonr…" 45 seconds ago Up 34 seconds 0.0.0.0:1433->1433/tcp sqlserver
Testing from within the docker container that SQL Server is installed and running...
docker exec -it sqlserver /opt/mssql-tools/bin/sqlcmd \
-S localhost -U "sa" -P "myStr0ngP4ssw0rd" \
-Q "select ##VERSION"
# --------------------------------------------------------------------
# Microsoft SQL Server 2017 (RTM-CU19) (KB4535007) - 14.0.3281.6 (X64)
# Jan 23 2020 21:00:04
# Copyright (C) 2017 Microsoft Corporation
# Developer Edition (64-bit) on Linux (Ubuntu 16.04.6 LTS)
Finally, testing from NodeJS...
const sql = require('mssql');
const config = {
user: 'sa',
password: 'myStr0ngP4ssw0rd',
server: 'localhost',
database: 'msdb',
};
sql.on('error', err => {
console.error('err: ', err);
});
sql.connect(config).then(pool => {
return pool.request()
.query('select ##VERSION')
}).then(result => {
console.dir(result)
}).catch(err => {
console.error('err: ', err);
});
$ node test.js
tedious deprecated The default value for `config.options.enableArithAbort` will change from `false` to `true` in the next major version of `tedious`. Set the value to `true` or `false` explicitly to silence this message. node_modules/mssql/lib/tedious/connection-pool.js:61:23
{
recordsets: [ [ [Object] ] ],
recordset: [
{
'': 'Microsoft SQL Server 2017 (RTM-CU19) (KB4535007) - 14.0.3281.6 (X64) \n' +
'\tJan 23 2020 21:00:04 \n' +
'\tCopyright (C) 2017 Microsoft Corporation\n' +
'\tDeveloper Edition (64-bit) on Linux (Ubuntu 16.04.6 LTS)'
}
],
output: {},
rowsAffected: [ 1 ]
}
Hope this helps.
System: ubuntu 14.04
mongodb 3.0.3 tar ball is downloaded from mongodb download center
connected to mongodb without auth, then from mongo shell, created a user for 'test' db. following is the command.
db.createUser({user: "user1",
pwd: "test123",
roles: [ { role: "readWrite", db: "test" }
]})
Verified that user details in admin db. Following is the command & result:
> db.system.users.findOne({user:'user1'})
{
"_id" : "testdb.user1",
"user" : "user1",
"db" : "testdb",
"credentials" : {
"SCRAM-SHA-1" : {
"iterationCount" : 10000,
"salt" : "kNfOd1vs+QT+ueH7SI6Vzw==",
"storedKey" : "JCesIKSW1pb74ddo2Y19rEO1GVY=",
"serverKey" : "d87Sb1htoD5K8zecAy73JPZyHdc="
}
},
"roles" : [
{
"role" : "readWrite",
"db" : "test"
}
]
}
Now exit from the mongo shell, killed the mongod.
Started the mongodb with auth, following is the command.
$ ./mongod --auth
Connected to mongo shell as usual, see the below:
$ ./mongo
MongoDB shell version: 3.0.3
connecting to: test
> show collections
2016-05-11T22:33:46.302+0530 E QUERY Error: listCollections failed: {
"ok" : 0,
"errmsg" : "not authorized on test to execute command { listCollections: 1.0 }",
"code" : 13
}
at Error (<anonymous>)
at DB._getCollectionInfosCommand (src/mongo/shell/db.js:646:15)
at DB.getCollectionInfos (src/mongo/shell/db.js:658:20)
at DB.getCollectionNames (src/mongo/shell/db.js:669:17)
at shellHelper.show (src/mongo/shell/utils.js:625:12)
at shellHelper (src/mongo/shell/utils.js:524:36)
at (shellhelp2):1:1 at src/mongo/shell/db.js:646
> db.auth({user:'user1', pwd:'test123'})
1
> use test
switched to db test
> db.collone.insert({name:'firstcollection'})
WriteResult({ "nInserted" : 1 })
> show collections
collone
system.indexes
> db.collone.find()
{ "_id" : ObjectId("5733669fb7d44cd444ebf028"), "name" : "firstcollection" }
> exit
bye
When i tried to do the authentication while starting the mongo shell, getting authentication failed error. See below:
$ ./mongo test -u 'user1' -p 'test123' --authenticationDatabase 'admin'
MongoDB shell version: 3.0.3
connecting to: test
2016-05-11T22:37:21.559+0530 E QUERY Error: 18 Authentication failed.
at DB._authOrThrow (src/mongo/shell/db.js:1266:32)
at (auth):6:8
at (auth):7:2 at src/mongo/shell/db.js:1266
exception: login failed
All this is just a POC that i'm trying to do.
Once it's success, my target is to connect from mongoose client(from Node.js app) to mongod.
The following command from a stackoverflow post can help me to set up connection from mongoose to mongod with auth.
I'm new using expect and is puzzling me big time. It works perfectly with one pattern but when the second case comes up it just ignores the exit completely. First, this is my code.
#!/usr/bin/expect
#Usage migration_test.xpct <ssh_password> <vmname> <no_migraciones>
set timest [ timestamp -format %Y-%m-%d_%H-%M ]
set vmname [lindex $argv 1]
log_file migtest_${vmname}_${timest}.log ;
set password [lindex $argv 0]
set num [lindex $argv 2]
set failureMsg "Status: Failure\n\r"
set timeout 60
spawn ssh admin#localhost -p 10000
expect "yes/no" {
send "yes\r"
expect "*?assword" { send "$password\r" }
} "*?assword" { send "$password\r" }
for {set i 0} {$i < $num} {incr i 1} {
expect "OVM> " {
send "show Vm name=$vmname\r"
expect {
$failureMsg { }
-re "Status = Running\n\r" {
exp_continue
}
-re "Server = .*? \\\[(.*?)(1|2)?\\\]\n\r" {
set destserver $expect_out(2,string);
if { $destserver == 1 } {
send_user "\n\nMIGRATION [ expr $i+1 ] of $num\n\n"
send "migrate Vm name=$vmname destServer=serv_prod02\r"
expect {
-re "JobId: (.*?)\n\r" {
set jobid $expect_out(1,string);
send "show Job id=$jobid\r";
expect {
-re "Command:(.*?)\n\r" { send_user "\n\nWaiting 30secs before next migration\n\n";
sleep 30; }
}
}
-re "Status: Failure\n\r" { send_user "\n\nExiting\n"; exit 1 }
}
} else {
send_user "\n\nMIGRATION [expr $i+1] of $num\n\n"
send "migrate Vm name=$vmname destServer=serv_prod01\r"
expect {
-re "JobId: (.*?)\n\r" {
set jobid $expect_out(1,string);
send "show Job id=$jobid\r";
expect {
-re "Command:(.*?)\n\r" { send_user "\n\nWaiting 30secs before next migration\n\n";
sleep 30; }
}
}
-re "Status: Failure\n\r" { send_user "\n\nExiting\n"; exit 1 }
}
}
}
}
}
}
send "exit\r"
expect eof
The problem comes when it reaches the "migrate vm" section. That's a job I'm sending to a CLI (oracle ovm cli to be precise) and the job can either fail or success. I want to print the job details when it success but finish the entire execution if the job fails (since it already shows the reason and I don't have to expand the job details).
Here is how the output of a successful job looks:
MIGRATION 5 of 12
migrate Vm name=slestest_temp_share_vm destServer=serv_prod01
Command: migrate Vm name=slestest_temp_share_vm
destServer=serv_prod01
Status: Success
Time: 2016-04-13 10:45:24,174
JobId: 12345678978
OVM> show Job id=12345678978
Command: show Job id=12345678978
Status: Success Time: 2016-04-13 10:45:24,188
Data:
Run State = Success
Summary State = Success
Done = Yes
Summary Done = Yes
Job Group = No
Username = admin
Creation Time = Apr 13, 2016 10:44:45 am
Start Time = Apr 13, 201 10:44:45 am
End Time = Apr 13, 2016 10:45:23 am
Duration = 37s
Id = 12345678978 [Migrate Vm: slestest_temp_share_vm to Server: serv_prod01]
Name = Migrate Vm: slestest_temp_share_vm to Server:serv_prod01
Description = Migrate Vm: slestest_temp_share_vm to
Server: serv_prod01 Locked = false
OVM>
Waiting 30secs before next migration
And here is how a failured job looks like:
MIGRATION 4 of 12
migrate Vm name=slestest_temp_share_vm destServer=serv_prod01
Command: migrate Vm name=slestest_temp_share_vm destServer=serv_prod01
Status: Failure
Time: 2016-04-13 11:31:08,819
JobId: 1460564963372
Error Msg: Job failed on Core: OVMAPI_5001E Job: 1460564963372/Migrate Vm: slestest_temp_share_vm to Server: serv_prod01/Migrate Vm: slestest_temp_share_vm serv_prod01, failed. Job Failure Event: 1460565064570/Server Async Command Failed/OVMEVT_00C014D_001 Async command failed serv_prod02. Object: slestest_temp_share_vm, PID: 1724,
Server error: Command: ['xm', 'migrate', '--live', '0004fb00000600009f354416bab38df6', '8.8.8.1'] failed (1): stderr: Error: ti
stdout: Usage: xm migrate
Migrate a domain to another machine.
Options:
-h, --help Print this help.
-l, --live Use live migration.
-p=portnum, --port=portnum
Use specified port for migration.
-n=nodenum, --node=nodenum
Use specified NUMA node on target.
-s, --ssl Use ssl connection for migration.
-c, --change_home_server
Change home server for managed domains.
, on server: serv_prod02, associated with object: 0004fb00000600009f354416bab38df6 [Wed Apr 13 11:31:04 2016]
Why does the Status: Failure is ignored? Also, when that happens it seems it jumps an iteration of the loop, if it was in the 5th it then shows "Migration 7 of 12" for example.
Thanks everyone
I can suggest two things, one you can rewrite code to avoid duplicacy. Second, I think you are matching for both \n\r at the end of pattern. Try with \n alone or use \n?\r? which will match zero, one, or both line endings.
-re "Server = .*? \\\[(.*?)(1|2)?\\\]\n" {
set destserver $expect_out(2,string);
send_user "\n\nMIGRATION [ expr $i+1 ] of $num\n\n"
if { $destserver == 1 } {
send "migrate Vm name=$vmname destServer=serv_prod02\r"
} else {
send "migrate Vm name=$vmname destServer=serv_prod01\r"
}
expect {
-re "JobId: (.*?)\n" {
set jobid $expect_out(1,string);
send "show Job id=$jobid\r";
expect {
-re "Command:(.*?)$" {
send_user "\n\nWaiting 30secs before next migration\n\n";
sleep 30;
}
}
}
-re "Status: Failure\n" { send_user "\n\nExiting\n"; exit 1 }
}
}
Well, after some tests I found the problem. It seems I didn't understand how the timeout worked in expect. Every time a failured migration was performed it exceeded the timeout.
This wasn't evident for me because, although the timeout was exceeded, the script still kept waiting for the answer and printed it anyways, just none of the patterns I was expecting to get were being checked.
The solution was either use the "timeout" command or set it higher. I did the later and everything is running fine now.