Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 months ago.
Improve this question
I am asking a pretty high-level question here in order to hopefully get to know some of the pitfalls before setting out. I am planning an application that will visit specific web sites to collect, process and format tabular data. It must then somehow take certain web browser actions (follow a link, post a form, click a button etc) in response to the data that has been collected, giving feedback if something breaks in the process. A central requirement is that it must be easily adaptable to different pages, i.e. the data and menu options on the web pages are largely the same, but formatted differently. The format of the page can change without notice, so error detection and handling must be good.
I was thinking of going with C# and simply using the WebBrowser class in .NET, seeing as it at least has good facilities for manipulating the DOM and running JavaScript without any additional configuration. However, I am reasonably language agnostic. The major thing I am worried about is that it WebBrowser doesn't seem to be as tightly developed for actually performing actions (mouse clicks etc). I am wondering if this is going to bite me in the ass. Also, it is a plus if the program behaves indistinguishly from a human user when seen from the server side.
Has anyone here worked with these kinds of tasks? I have to emphasize that I am not doing testing of web applications here; this is more a robot. Are there any libraries/frameworks out there that are better suited than the .NET standard library with regards to flexibility and ease of use? Are there any major pitfalls to look out for?
I suggest you look at mechanize in combination with beautifulsoup it's perl or python but it's exactly what you need.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
I'm wondering, if there is Axure RP alternative for a better price. I used other apps, but ended with Axure RP because of two fundamental advantages:
it's not only wireframe tool, but it lets me prototype whole web/app with functional demonstrations
it generates/exports prototype and customers can easily try personaly (it's HTML site and i upload it to my server and clients just get email with link to follow)
So, is there anything else like Axure RP meeting my two criteria with better price? All apps i've seen mainly fail in second condition.
Yes - atomic.io lets you do both of these things.
You can go from low fidelity, right up to fully functioning prototypes that use things like logic, data, and variables.
You can also easily share an URL with anyone. (see: https://atomic.io/learn/sharing)
And there is a free plan. :-)
atomic.io
You can use moqups.com , it's a web app.
it lets you to make prototypes with linked pages and more .
also it lets you to share your Design through a URL on its site.
There are a ton of prototyping tools.
https://www.cooper.com/prototyping-tools
It always depends on what your main goal to achieve is.
Justinmind (https://www.justinmind.com) is very close to Axure.
If you do not mind sharing your prototypes you can look into a bunch of different tools that are webbased (i.e. https://www.figma.com)
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I am working on a scraping project for a company. I used Python selenium, mechanize , BeautifulSoup4 etc. libraries and had been successful on putting data into MySQL database and generating reports they wanted.
But I am curious : why there is no standardization on structure of websites. Every site has a different name\id for username\password fields. I looked at Facebook and Google Login pages, even they have different naming for username\password fields. also, other elements are also named arbitrarily and placed anywhere.
One obvious reason I can see is that bots will eat up lot of bandwidth and websites are basically targeted to human users. Second reason may be because websites want to show advertisements.There may be other reasons too.
Would it not be better if websites don't have to provide API's and there would be a single framework of bot\scraper login. For example, Every website can have a scraper friendly version which is structured and named according to a standard specification which is universally agreed on. And also have a page, which shows help like feature for the scraper. To access this version of website, bot\scraper has to register itself.
This will open up a entirely different kind of internet to programmers. For example, someone can write a scraper that can monitor vulnerability and exploits listing websites, and automatically close the security holes on the users system. (For this those websites have to create a version which have such kind of data which can be directly applied. Like patches and where they should be applied)
And all this could be easily done by a average programmer. And on the dark side , one can write a Malware which can update itself with new attacking strategies.
I know it is possible to use Facebook or Google login using Open Authentication on other websites. But that is only a small thing in scraping.
My question boils down to, Why there is no such effort there out in the community? and If there is one, kindly refer me to it.
I searched over Stack overflow but could not find a similar. And I am not sure that this kind of question is proper for Stack overflow. If not, please refer me to the correct Stack exchange forum.
I will edit the question, if something there is not according to community criteria. But it's a genuine question.
EDIT: I got the answer thanks to #b.j.g . There is such an effort by W3C called Semantic Web.(Anyway I am sure Google will hijack whole internet one day and make it possible,within my lifetime)
EDIT: I think what you are looking for is The Semantic Web
You are assuming people want their data to be scraped. In actuality, the data people scrape is usually proprietary to the publisher, and when it is scraped... they lose exclusivity on the data.
I had trouble scraping yoga schedules in the past, and I concluded that the developers were conciously making it difficult to scrape so third parties couldn't easily use their data.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 4 years ago.
Improve this question
I have to write a bunch of small web services. They must be defined by a WSDL and work via SOAP-RPC, in order to work with an existing workflow engine and service registry framework. I can, however, serve them on a service stack/platform of my choice.
I'm presently writing them in Java, and it's not too bad. But I'm thinking my life might be easier if I was able to write these services in Haskell. Searching on Google, it looks like, once upon a time, someone else had the same idea and started a project called "HAIFA". However, it looks like HAIFA hasn't been maintained for some years, and I couldn't find any other frameworks supporting serving up services written in Haskell as SOAP web services.
Does anyone know of any other frameworks that will allow me to easily write SOAP-based web services using Haskell?
If not, has anyone done this manually (i.e., use XML libraries from hackage to process the incoming soap-rpc requests, and create soap-rpc compliant replies)? Was it difficult to do? Any gotchas? Was it worth the effort?
Since HAIFA is dead now there are no equivalent frameworks for SOAP web services in Haskell now. So I would advise you to use some bunch of frameworks.
May be Yesod + shoap will be suitable.
I think such a tendency in domain of SOAP WS frameworks in Haskell because of smooth transition to REST/JSON technologies.
Also may be these two articles will be useful for you
http://www.cin.ufpe.br/~haskell/hwsproxygen/files/HWSProxyGen.pdf
and
http://www.jofcis.com/publishedpapers/2010_6_9_2859_2867.pdf
If you just need to send vanilla XML/SOAP messages then you could probably just open connections and read/write Aeson. But if the SOAP services need WS-Security, WS-Addressing, etc., support, you would be better off using an existing framework. For example: https://metro.java.net/
Disclaimer: I have been involved in Metro.
It would be great to have a SOAP framework in Haskell though.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 5 years ago.
Improve this question
I'm looking to set up a small site for a friend that has some widgets they want to sell online. I don't think I will have much time for maintenance once it goes live (for that matter, I don't expect I'll have much time for initial setup and configuration), and I am looking for something that is dead-simple for a non-technical user to maintain (financial/payment info, add/remove/change products).
The second most important part would be good integration with a payment provider. I'm not too fussy what language it's in if it meets my other criteria (if I don't know the language I will learn enough to get the site running).
Also important is that I'd prefer to stick to open-source products, mostly because I don't think this project will have much of a budget for high-end commercial products (at least not until it makes some sales).
The last time I did this sort of stuff we were building custom sites from scratch for clients with very specific needs. I do not have recent experience with the current generation of blogging tools (Wordpress, Joomla, etc...) and I don't really know which off-the-shelf combo of platforms and plugins are best to get something up and running in as little time as possible.
Hosting your own online store is a full-time occupation, no different from running your own brick-and-mortar store. Anything that accepts online payments will be targeted by criminals for online fraud.
If your business is selling widgets and not running online stores, I strongly, strongly suggest using a hosted service with its own web integration and payment handling. I know people who have used both Weebly and Etsy and who are happy with them.
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 5 years ago.
Improve this question
Does anybody know what large companies are currently using agile iconix process??
The only ones I know are the one I could find on the ICONIX Software Engineering corporate website:
Case studies: see how ESRI Professional Services, Virginia Department of Motor Vehicles, and Large Synoptic Survey Telescope are succeeding with ICONIX Process
I may be wrong but to me, the ICONIX methodology isn't really widely used and it
looks more like a way to sell their Enterprise Architect product.
And personally, I never had big successes with too much UML centric approaches (à la MDA).
I like the process and used it well in several projects. I just want to give some of my thoughts on it:
Iconix is based on domain driven design. Domain comes first. This is fine, however we need to be aware of a boundary conditions. To put is simply, domain driven design works for the relatively complex projects. There may not be a domain model as design pattern at all since it may not be the best choice for every system.
Iconix assumes sophisticated deisgn. Not every project needs it and not every project has developers capable of absorbing it. There are tons of data-centric or purely data manipulation applications out there.
No community, stale web site. I don't know of anybody who uses the process.