I want to know if there are any outsource solutions for a crawler that will parse only the links and pages form a given website, and will output:
1.The link tree
2.The pages (where necessary)
thanks!
You dont need any particular framework to achieve this task. What languages do you know? If you know Java you can use HttpClient or HttpUnit libs to help you with crawling tasks.
If you are python user, there is great framework called Scrapy (http://scrapy.org/). You should check it out.
Related
What is another way to crawl the web besides following hyperlinks?
Most major sites use Sitemaps. This gives your crawler a fast way of discovering URLs and can be used with or instead of following outlinks.
The crawler commons project provides a Sitemap parser in Java.
I need to detect programatically if a website has an e-commerce platform/system
I don't need to know which one, I just need to know if the website has one.
(I have a big list of websites so I probably need to scrape them)
any suggestions on how I could do this without using external websites (like rescan.io/builtwith/etc) would be greatly appreciated!
thank you!
You can use a package called Puppeteer which is used to do web-scraping in node.js.
I don't know what platforms you are trying to look for, but I guess you could try something like giving the list of websites you want to check to a node.js process and ask Puppeteer to scrape them all. Then you look at the content you get back and for example look for Shopify's CDN in the tags or check the tags for keywords.
You will definitely need to check each different platform like Magento or Shopify for unique source code that clearly sets apart the framework you are looking at from other tools.
I want to create a website which consists of videos which will be uploaded by the admin and many useful information which all will be uploaded by an admin.
I have got the domain name, remote LAMP server.
I have gone through many tutorials and I tired creating using Drupal 8. Website was fine but while creating a new existing module the whole thing got crashed and I don't want to goback and recover it since it is a big mess.
So I want to try from first keeping it very simple.
Kindly guide me to achieve this. I will put all my effort to learn it.
Any help any documents which will help me to create will be greatly accepted.
I am beginner in C++, html. Can you guys let me know how can I achieve in creating a website.
You can find your answer here on your own:
So there are various languages that are being used in the web that perform various purposed and hence you can choose what functionality do you need in your website and according to that you can write the code in that language.
Lets take a quick look at some of the most widely used languages of the web:
1: HTML
This is a Markup Language which can be used basically just for writing the content and displaying on your webpage. You can create too many pages and link them to form a website.
2: CSS
This language helps you design your webpage and thus makes your webpage look way too better. A site only written in html in not preferrable.
3: JavaScript
This language is a scripting language that helps you do various cool stuff like handling input events (like click, hover,etc), change the content of your webpage dynamically, bring popups , etc.
4: PHP
This scripting language is also being widely used as this lets you work with the forms and submit to a database. In fact, if you learn this language you can write the logic behind your own facebook and you can give it a face using HTML, CSS.
Once you are done learning all these languages, you can now learn some of the cool libraries like:
Bootstrap (for css, js)
Jquery (Javascript)
Has anyone has done Eventbrite integration with an Expression Engine site? We'd like to set up events with Eventbrite and have them handle all ticket management. But we'd like to be able to display the events within the Expression Engine site and then enable users to click on the link to be redirected to Eventbrite. I've viewed the API and it looks like we can create custom EE pages with the API.
More importantly I'd like to let users search for events from our main site.
Has anyone done this type of work and have any hints or resources?
Thanks.
Todd Perkins got started on a module for this some time ago, but there hasn't been any action on it since then. Could be a good starting point for you though.
https://github.com/toddperkins/eventbrite
Eventbrite has a great PHP-based API client library that should be able cover all of your API interaction needs.
These PHP examples might be useful as well:
https://github.com/ryanjarvinen/eventbrite.php/tree/master/examples
http://eventbrite.github.com/#examples
Please let #EventbriteAPI know if you make any major progress on this project. I'm sure they would love to add an Expression Engine integration to their open source projects list and application showcase!
Hi
I am creating a website using HTML and CSS only and I want to add the following feature:
Any person visiting the site should be able to post a message and that message should appear below with his name and time of posting.
How do I go about creating it?
Please help.
Thanks In advance
You won't be able to do this using HTML and CSS only (at least not in any sane way).
You will need a database of some sort (e.g. MySql, Microsoft Access etc) and you will need some sort of server-side scripting language such as PHP, .NET, Ruby etc.
I suggest picking a language and then finding some tutorials. Most tutorials will take you through using a database.
This is completely impossible using HTML and CSS only. You will have to use a server-side scripting language.