My requirement is to get the "created date" / "modified date" of a dataset on a web page into a .xls or .csv file.
The URL is: https://data.medicare.gov/Hospital-Compare/Agency-For-Healthcare-Research-And-Quality-Nationa/sdhm-um6i/about
From this web page, I would like to import created date which you can see on the right side using script task in SSIS.
I have tried using excel data -> import from web . It doesn't work!
It would be of great help if someone has any suggestions on this.
Thanks
There is nothing native to SSIS for parsing web pages like this. That said, I hope you like .NET coding.
I've done something similar with the Html Agility Pack Work through their documentation and examples to understand the basics of how to parse HTML with their library.
Once you have that settled, it's a simple matter of determining the path to the element(s) you desire. I have found the best approach to be open the page with Chrome, right click on the thing of interest and choose "Inspect Element". Expand fields until you get to the thing you're after and then right click in the inspector window and select Copy XPath
That results in the following xpath selector
//*[#id="aboutSection"]/div/div/div/div[1]/p[1]/span/span
I'd then use it in code like
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(this.html);
string createDate = string.Empty;
createDate = doc.DocumentNode.SelectSingleNode("//*[#id="aboutSection"]/div/div/div/div[1]/p[1]/span/span").InnerText.Trim();
You'll need to fiddle with the exact mechanics of the SelectSingleNode call but that's the general concept. You might also notice that you have an attribute data-rawdatetime with a value there. I'm guessing that's epoch date but if you can translate that to the presented value of "Oct 20, 2011", you might be better off as that numeric value won't be dependent upon regional settings.
Related
I would like to create a spreadsheet that I can refresh and pull in each weeks English premier League fixtures, each week I would like to refresh this and see the weeks future fixtures. I have tried to use the import function from Data/From Web and selected the box with the table of fixtures however no data gets pulled into the spreadsheet.
The website I am using is - "http://data.7m.com.cn/matches_data/92/en/index.shtml"
I am open to understand a better way of doing this import and also if there is a better website to use I am also happy to change. I have chosen this one as it seems to have the most simplified listing of the fixtures.
I have also tried this website - https://www.premierleague.com/fixtures
When the import completes it actually skips all the fixtures and returns all the other information.
Should i be looking to some of the HTML elements within the script of the web page to extract the data?
For example on the following site - https://www.premierleague.com/fixtures I am looking for a file received by the website that updates the fixtures each week (after some direction from Google) I hit the F12 command and look within the "Network" tab however I cant understand how the website, this or the others quoted create the weekly fixtures.
Any suggestions on how to pull this into Excel or another tool would be fantastic.
Welcome to [so]! it sounds like you haven't done as much research as you could have. Your first link, in the top corner has links to "Free Feed" which take you to customizable widgets and from there is a link to a customizable live template.The first page also has a link to "Data" , I'm not sure what that consists of or whether it will help (since I'm not much of a sports fan on my continent, and even less on yours!
As for importing into Excel, I didn't have an issue with the table I could see, but once again I'm not clear on what data you're trying to get and what you want to do with it.
On the ribbon's Data tab click From Web.
Enter the first URL from your question and hit Enter
When the Navigator window loads, click "Table 1" and then click Load.
Below is what Excel then automatically loaded as a table:
If instead of clicking Load, you were to click Edit then you are brought into the Power Query Editor, where you can customizable tons of stuff. The one I was interested in was Use First Row on Headers. After choosing that, and clicking Close & Load, and 30 seconds of formatting later I had:
With Power Query you can choose, remove, split, or combine columns from this or other tables. It's fairly advanced but you should be able to find a good Power Query tutorial online, to see examples of what you can do, to learn about other ways you can customize the import and/or analysis of the data.
Edit:
More Information:
Here are the instructions for all versions:
Office Support : Connect to a web page (Power Query)
I have a Windows Application, which has a control type: Client and
class name =WindowsForms10.WIndows.8.app.0.3ce0bb8_r11_ad1
It is like a sub title bar. It has a text which displays the number of records in the table below. I want to extract that number of records. How ever when I spy using UIMap-CodedUI Test Builder, it highlight the entire title bar and does not only highlight the number of records. Is there any way I can handle this.
I want to extract 495 from the title bar. How can I do that?
Thank you all for your reply.
I could figure out an answer.
1) 495 is not shown as a UI control.
2) I found a work around , I can calculate the rows in the table below using UISyncDataGridTable.Rows.Count
and it serves my purpose.
You could use AutoHotkey. This or this function should be able to complete the task you want. The installer also comes with windows spy, which might help you pinpoint the control class you want to extract the string from.
Please excuse if there is a better way to do this, I'm not too familiar with SharePoint web parts, I'm only just now getting into creating them.
What I want to do is have a web part that I can drop into a page, give it a parameter, and then the web part does it's API calls with said parameter, does some XSL transformation, and spits out the HTML that I want to display on the page. In my mind it seems like a simple little web part to write, I just want to make sure this is the best way to go about it and not be duplicating something that could already be existing.
I've gone through this creating a basic web part walkthrough, and I figure I can use the web part property to get the parameter from the user. How would I get it to output HTML though? I changed the "Hello, world!" text to output some HTML and it seemed to have gotten stripped out?
Thanks a ton, and sorry if anything seems noob-ish!
From what I see, there are two main parts to what you want to do.
Get your parameter into your query
Format the results with some HTML
Reuse it later
There are a lot of ways to do this, but I'm going to outline what I believe to be the fastest and most simple.
Getting your parameter into your query
In SharePoint 2010, SharePoint designer has some great options for this.
So step 1 is to open the webpage in SharePoint Designer.
Click the Insert tab and click the Data View dropdown and select the list you want to get data from.
Now click on the Options tab and click on Parameters
The dialog shows you some of the options for pulling info into your query.
A query string parameter is easy to use. And the server variables are also really handy.
For our example, we'll pick query string and set the default value to 1 (which will be the ID of an item we want to be retrieved). Since we've set this as the default, even if the query string is blank we'll still get our default in our parameter.
Then click on Filter.
We set our Field Criteria to use the parameter we just made.
Now you can save the page in SharePoint Designer and test your results. Pretty fast.
Format the results with some HTML
In SharePoint, XSLT is the best way to put your own HTML around some data you've retrieved from SharePoint.
Also, for getting a parameter into your query, we can add that with a parameter binding:
If you're new to XSL, here's the best xsl tool:
http://spexp-blog-files.s3.amazonaws.com/blog/files/spe-magic-data-view-builder.xsl
To use the magic dataview builder, save this file to documetn library in your farm.
If you’d like to centralize your XSL template for greater reuse (highly recommended), you can put the individual XSL files into a central location for storage. In a WSS or SharePoint Foundation environment, I’d recommend using a Document Library in the root site of your Site Collection. In a MOSS or SharePoint Server 2010 environment, I’d recommend placing the XSL files in the /Style Library/XSL Style Sheets location, as this is where SharePoint stores its XSL by default.
Then paste the url to the file into the XSL Link to replace main.xsl and change Default to FALSE. This property is located near the end of the <XmlDefinition> tag as you view the webpart in Designer.
... <XslLink Default="TRUE">main.xsl</XslLink><Toolbar Type="Standard"/></View></XmlDefinition>
From there, you can save the page and the XSL wizard from magic data view builder will walk you through creating your own xsl to output your html.
Reusing Your Webpart
To reuse it, click save "To Site Gallery" while your cursor is on the wepart in SharePoint Designer. This will save the webpart into the gallery under the "Custom" group. From there, you an insert your webpart into other pages just like the out of the box webparts.
I think we need more information about what you are trying to get the web part to do. If you use a SharePoint Data View web part, you can point it at a data source, write some XSL to transform the data to render the way you want it will output the HTML to the page. For example, let's say I have a SharePoint list of training classes. I could create a data view web part that takes a query string or other input parameter as its filter, and filter the list of classes to show me just the one that matches the ID of my query string or input param. Then I could format my XSL to output details about the class rendered in whatever fancy HTML I want.
Does that help?
"I changed the "Hello, world!" text to output some HTML and it seemed
to have gotten stripped out?"
Please be aware that in your sample, it uses a LiteralControl() which by default HTML Escapes any text. So if you need HTML outputed via the LiteralControl.Text element, you will need to set the Mode to "Passthrough".
protected override void CreateChildControls()
{
base.CreateChildControls();
LiteralControl message = new LiteralControl();
// !! add this !!
message.Mode = LiteralMode.PassThrough;
message.Text = DisplayMessage; // <-- containing HTML
Controls.Add(message);
}
I have written a Dialog that uses an Advanced Find where the user types in part of a search term, i.e., similar to:
Dialog’s Query Step: Demystifying the Advanced Mode Capabilities.
In my case I created one to find all the Contacts in a post code area starting with "xyz..."
It works fine and I get the contacts in a picklist. How can I get this in a grid of results like you can for a regular advanced find? As I understand it the Query CRM data operation only gives you radio buttons or a list as output types. But is there some ingenious way of grabbing the results perhaps via a child workflow and writing them to a grid?
I'm sure you're looking for something a little more out of the box, but one approach would be to use an HTML web resource. You can mimic the dialog look and feel by using the CRM 2011 Style Guide included in the SDK (in the sdk\resources\styleguide folder).
I think this cant be done out of the box. You can of course create a webresource and then embed the advanced find view using fetchxml query.
Please take a look at the following url. Hope it helps.
http://crmentropy.blogspot.com/2009/11/embedding-advanced-find-views-in-entity.html
I want to add text to body element but I don't know how. Which method will work on the body tag?
Sorry for my english and thanks for replies.
In Watir, you can manipulate a web page (DOM) using JS, just like that:
browser.execute_script("document.getElementById('pageContent').appendChild(document.createTextNode('Great Success!'));")
I assume that the point of the question is:
All users are not just interacting by just clicking buttons and links on the web app, some of them are doing nasty things like altering http requests to make your system do something that it is not supposed to do... or to just have some fun.
To mimic this behavior, you could write a ui-test that alters forms on the web page, so that for example, one could type in anything into any field instead of a limited dropdown.
To do that, ui test has to:
manipulate DOM to set form inputs free of limitations (replace select's with input's, etc.)
ui test has to know, which values to use, in many cases it's pointless to enter random values. Your webapp has to provide some good "unwanted" options.
Why would you want to modify the webpage in Watir? It's for automated testing, not DOM manipulation.
If you want to add something to the DOM element in javascript, you can do it like that:
var txt = document.createTextNode(" This text was added to the DIV.");
document.getElementById('myDiv').appendChild(txt);
Or use some DOM manipulation library, like jQuery.
If you have not worked your way though the watir tutorial, I would suggest you do so. It deals with things like filling in text fields etc.
Learn to use the developer tools for your browser, Firebug for Firefox, or the built in tools for IE and CHrome. They will let you look at things as you interact with the site.
If the element is not a normal HTML input field of some sort, then you are dealing with a custom control. Many exist and they are varied and there is no one set solution for dealing with them. Without knowing which control you are using, and being able ourselves to interact with a sample of it, or at least see the HTML, it is very very difficult to advise you, we basically have to just guess (which is often a waste of everyone's time)
Odds are if you have a place you can enter text, then it is some form of input control, it might not start out that way, you may need to click on some other element, to make the input area appear, but without a sample of HTML all we can do is guess.
If this is a commercial control, see if you can find a demo site that shows the control in action. Try googling things like class names for the elements and often you get lucky