I have one problems, but i can't understand how to describe you for solve my problems. I m tying to understand you.
I have create a custom module that create a form and has some fields like id,url,dom pattern,time,schedule.
** After submit 2 item by using this form, it store all data to my database table.
pattern_store_table
id | url | pattern | time schedule
--------------------------------------------------------------------
1 | http://example.com/page1 | #div_1234 h2 | 1460864020 | 60
2 | http://example.com/page2 | #div_4567 h2 | 1465311839 | 30
Then I have created some function for collect data from other websites using third party dom parsing script, based on there websites url and Html Dom pattern. Then, i store this data as my drupal node.
And i create a list page based on "pattern_store_table" table data items. And create a function for call my scrap functionality.
So, when i hit "//example.com/scrap/id/1" into my browser, it scrap title from "http://example.com/page1" and store as my drupal node, and when i hit "//example.com/scrap/id/2" into my browser, it scrap title from "http://example.com/page2" and store as my drupal node.
My Problem is:
I don't want to hit always "//example.com/scrap/id/1" into browser. I want to create a function that call "//example.com/scrap/id/1" automatically every "schedule" fields timewise like "60 (for id 1)", "30 (for id 2)" then collect and store data into my database.
I listen about CRON, but i don't know how can i use it with my custom module.
Anybody please help me to automate this system step by step?
Sorry for my bad english. :(
Related
I am trying to retrieve data from http://www.professorpaddle.com/rivers/riverlist.asp which automatically defaults to Washington state as the state id. However, I want to pull data from the table for Oregon. Can this be done as a property? So far I've tried writing a .iqy file with the following and it still doesn't work:
WEB
1
http://www.professorpaddle.com/rivers/riverlist.asp?hstateid=["oregon"]
Selection=EntirePage
Formatting=All
PreFormattedTextToColumns=True
ConsecutiveDelimitersAsOne=True
SingleBlockTextImport=False
I am new to VBA but open to using it as well.
You've got two issues:
The page you linked uses a POST not a GET request so you have to write the parameters separately, not as part of the URL.
I'm not sure why you're supplying oregon as the hstateid. When examining the request in Chrome's dev tools I see 37 as the value of this parameter.
And the brackets around the value indicate that you want you to specify variable values for the parameter but I think you're trying to just have a static value here.
So your query should start with something like:
WEB
1
http://www.professorpaddle.com/rivers/riverlist.asp
hstateid=37
Is it possible to make real-time monitoring report by using Power BI?
I have an experience only with elasticsearch+kibana and I want to make the same dashboard by using Power BI:
There is a data source - RequestDateTime, Sendor, IsSuccess, RequestType, Request, Response. For example:
2016-10-20 12:00:12 | Test 1 | True | SetUserInfo | xml here... | xml here...
2016-10-20 12:00:18 | Test 2 | False | GetUserInfo | xml here... |
This data can be downloaded from Azure SQL database, by using simple sql-query.
I want to make a simple column chart. X-axis should be a timeline, accoring to the RequestDateTime field. Y-axis should be a count of records, that is correspond to filters.
Report should have a filters by sendor, isSuccess, RequestType and RequestDateTime range (for example - last 6 hours).
The report should be able to be refreshed in a real-time mode, according to the new events in the database.
The pipeline I usually do in my work using app.powerbi for real time visualization is :
Create a dataset: In your workspace, select the "create" option, there you can import or connect to data from Files and Databases. You can choose Azure SQL database and follow instructions.
Create a report: In your dataset list, select the icon to create a report (here you define the filters that you want) and save it.
Create a dashboard: Pin the charts to a dashboard. I have found that only dashboard refresh itself when data is added or deleted in the database, thus, you can see the real time in the dashboard.
You may find this video useful.
Could any please suggest me some way we could use cucumber to repeat a test (e.g Login/Logout) multiple time, but changing only username/password for each time?
I could do for a single pair of username/password, but I'm looking for a way to re-use the test for multiple data set (The data set may be an array which store username/password, and the expected result which is true or false).
Thanks.
Scenario Outline allows you to run your Features based on the provided Examples in a tabular format, so to test login multiple time with different values you instruct to your step definition to run for every provided examples, In this we need to mention table column field in angle brackets in feature file.
Scenario Outline: Creating a new account
Given I am not authenticated
When I go to Login page
And I fill in "user_email" with "<email>"
And I fill in "user_password" with "<password>"
And I press "Sign up"
Then I should see "logged in as <email>"
Examples:
| email | password |
| testing#xyz.com | secretpass |
| testing2#xyz.com | fr33z3 |
For more references, Cucumber tutorials
I have started writing the following feature within an app designed to manage a cleaning business:
Feature: Creating a new cleaner
In order to allow Franchisees to allocate cleaners to jobs they need to be uploaded to the system
Background:
Given I am currently logged in to my account
And I have navigated to the "Cleaners" page
And I want to add a new cleaner to the database
Scenario: Add a new cleaner to the system
Given I have brought up the "Add Cleaner" form
Then I will need to complete the fields within the following form:
| first_name |
| last_name |
| email |
| date_of_birth |
| postcode |
| mobile |
| other_phone |
| address_1 |
| address_2 |
| work_radius |
| **days_available** |
| notes |
When I have entered valid data
Then I can save to the database
And I will have added a new cleaner to the system
In addition to welcoming comments on the way I have written the scenarios etc, my main problem is that I can't work out how to simulate selecting from a pre-populated field:
Populating the days_available should allow the franchisee to choose which days of the week, and which hours within those days, that a cleaner will be available for work. This obviously makes it possible to return queries which only show available cleaners for any given day/time of day.
Really hope someone can explain how this is done?
Just a quick comment on the structure of your feature file ... the 'Then' step in your feature should be asserting that something has or has not been done successfully.
Given I have logged into the site
When I add a new Cleaner to the site
Then I should see that the Cleaner has been added successfully
I would recommend using language that can be easily understood. Your scenario doesn't need to be instructions on how to use the site. Excessive navigational steps can make you lose track of the purpose of the scenario.
To answer your question regarding days_available accurately, would require some knowledge of how the site is structured and how the days_available are entered. Are you choosing from select lists, filling in form fields, etc? Also, since you are testing, you could consider setting the data from within your step (ie. hash, array) instead of passing all of the info in via a table.
Just some food for thought. Cheers.
Based on your updated post, I would suggest the following:
The step And I want to add a new cleaner to the database doesn't seem like an actionable step and could be removed. Same for the step When I have entered valid data. If you handle filling out the form in the previous step, you have already entered valid data.
If you need to multiple available days, I would consider making it its own step
And(/^the cleaner is available from (.*?) to (.*?) on (.*?)$/) do |start_time, end_time, day|
#fill in start time
#fill in end time
#select day
end
Background:
Given I am currently logged in to my account
And I have navigated to the "Cleaners" page
Scenario:
And I bring up the "Add Cleaners" form
And I complete the form with
| first name | Bob |
| last name | Smith |
...
And the cleaner is available from 0600 to 1800 on W
When I submit the Add Cleaners form
Then I should see the new cleaner has been successfully added
I'm developing a web scraper in Node JS and I'm trying to figure out the best approach to combine data from one list, with data on another list.
For example:
Step 1: Scrape data from website A
In this step, I scrape some data using cheerio/request and store this in a list, which is then displayed on the screen in a jQuery data table. The data table has a checkbox next to each scraped row of data and the value I have assigned to each checkbox is a "URL".
Step 2: Scrape again based on URLs chosen in checkboxes
In step 2, the app will scrape another website based on the URLs selected in step 1 and will retrieve some data values from these URLs that are scraped.
My dilemma
I wish to use some data values that were scraped in step 1 along with some data values scraped in step 2. However currently in my app, the data from Step 1 has been lost because it's not being saved anywhere.
Since this is a sort of dynamic search whereby a user will search for data, scrape it and then not neccessarily want to see it again, I think saving data into a database would be overkill? So I'm wondering if I should save the list data from step 1 into a session variable and then link them up together again using the URL (in the checkbox) as the key?
Thanks for your help!
Anthony
if you dont want to do saving for these data then pass it as another inputs of your form , try
< input type=hidden value=json.stringify(item)/>
I dont think using database for storage of the scrapped conetent would be an overkill.
The ideal points to note in this process would be.
Use a document store like mongoDB to dump your json data directly. I proposed mongoDB because you get more resources to refer.
Initially open up a db connection in node and your scraping deamon can reuse it for each time when it parses the http result using cheerio and dumps it to db.
Here Once you get the output from http request to your Target URL, the cheerio parse steps should take more time than dumping the data to a db.
I have followed a similar approach for scrapping movie data from imdb and you can find the corresponding code in this repo, if you are interested. I have also used cheerio and its a good selection in my opinion.