I need to detect programatically if a website has an e-commerce platform/system
I don't need to know which one, I just need to know if the website has one.
(I have a big list of websites so I probably need to scrape them)
any suggestions on how I could do this without using external websites (like rescan.io/builtwith/etc) would be greatly appreciated!
thank you!
You can use a package called Puppeteer which is used to do web-scraping in node.js.
I don't know what platforms you are trying to look for, but I guess you could try something like giving the list of websites you want to check to a node.js process and ask Puppeteer to scrape them all. Then you look at the content you get back and for example look for Shopify's CDN in the tags or check the tags for keywords.
You will definitely need to check each different platform like Magento or Shopify for unique source code that clearly sets apart the framework you are looking at from other tools.
Related
Hi I am running a delivery shop and using delivery.com for my shop.
I also want to have my own website which user can order service or choose delivery time on the website like delivery.com does.
I wonder if is it better to develop from scratch or can I use wordpress and customize it? or are there plugins or widgets I can use for my website?
Please give me some advice what will be the best way to build my website.
Thank you.
Well technically anything is pretty much possible in Wordpress but it won't happen out of the box, unless you already have a theme in mind which serves this purpose. I won't be surprised if there is a theme out there which you can use to do this.
Personally I would build this from scratch as you would be able to build exactly what you want and you can build it with scalability in mind as well.
Wordpress tends to fall off when the site starts getting A LOT of traffic.
Check out Ruby on Rails, you would be surprised how at how simple it can be to build something like this.
The server will go down (if) when your site will visit 1000+ persons, because one of the problem of universal script that can be used (in opinion of creators) every where is that it make 100 sql queryes to display one simple page.
I'd like to code a website where you can find search results from many websites.
So my question is, if this scenario is possible and if yes, if you guys have any suggestions how I would be able to do this.
Here my workflow:
I search for something on my website. For example: "asdf"
My code then executes the search from the other website. for example:
https://www.google.ch/#q=asdf&safe=images
Then there will be shown some results, of course. But how can I directly take the results and show them on my website, without opening the other website?
I have to say, that the websites I'm looking for, haven't got any API for that.
I probably wouldn't recommend to scrap a web page directly in the client.
I'm not even sure if you can achieve it easily without getting some Cross Domain Policy problems anyway.
A solution like APIfy might help you doing what you want:
http://apify.heroku.com/resources
Or you can still make your own server site API "layer" for this particular website?
Keeping in mind that scrapping a web page is always a fragile process where the format can chance at any moment.
I want to develop an application that will visualize the recommendations of Google instant. It is for a course project and for now, I don't know much about web programming tools. What I wonder is that is it possible to retrieve that data from another web page. If you think it is possible and it is possible with which platform, could you please guide me to the correct direction?
Without more information on what you're actually trying to do, it's difficult to give a proper answer. From what I can understand, you just want a list of the auto-completed items from a Google search, to manipulate however you like?
In which case, using the highest-rated answer from here, you can use http://suggestqueries.google.com/complete/search?client=firefox&q=YOURQUERY to give you a JSON object which you can then manipulate to get the auto-complete results. The client= part is needed, but I haven't looked at various options you can put in there.
Personally, I've never used JSON before, so can't give you any help on how to go about parsing it, but you can find more information about it on the JSON website, and w3 website.
Will need to act like javascript or run a javascript engine OR a browser add on and communication with that add on.
What happens as you type is a javascript function is called. So you need to call this function in your own or mimic what it does. I guess it calls a web service/ web page form programamtically (ajax) with what you have typed. The server responds with the suggestions. Not very difficult as long as Google does not deny you if it realizes your not a browser. i think they like only 100 free API calls but you can google google about that.
Http Components in java will help calling the serice, with cookeis etc. You should use the dev tools on firefox to see what happens under the hood when you type in the google search bar and see the code.
I have a simple HTML site with 100+ pages or so. I want to add a search bar at the top so the user can search the site. I know about Google Custom Search, but it shows ads unless you pay at least $100. Obviously I'd like ad-less search on my site for free if at all possible!
I've also heard about Lucene/Solr, but they do not actually crawl the site. For that I would apparently need Nutch.
Anyway, the site I have runs on a Microsoft IIS6 server, but I have basically no knowledge as to how Solr, Nutch, etc. gets "installed" on the server.
Also: I'd like to point out that I do have a local copy of the site. Perhaps I can do one big initial nutch "crawl" locally that will create an .xml for Solr?? That would help me get "up and running", but probably wouldn't be a good long-term solution.
..so should I just use Google Custom Search? or is there a not-extremely-painful-to-implement alternative? The brain hurts folks.
You did not mention how many search requests you want to handle but if you use the json-rest-api of google's custom search you have 100 searchqueries a day for free and you can display them without any ads on your page.
An simple example request can be found here.
Here is an easy way that works pretty well, although you may be looking for something more than this.
http://sitecomber.com/getsitecomber/
You can create code to paste into your site in about 2 minutes. It doesn't get easier than that. Search is powered by Google, but results are isolated to your website.
EDIT: This no longer works.
Hi
I am creating a website using HTML and CSS only and I want to add the following feature:
Any person visiting the site should be able to post a message and that message should appear below with his name and time of posting.
How do I go about creating it?
Please help.
Thanks In advance
You won't be able to do this using HTML and CSS only (at least not in any sane way).
You will need a database of some sort (e.g. MySql, Microsoft Access etc) and you will need some sort of server-side scripting language such as PHP, .NET, Ruby etc.
I suggest picking a language and then finding some tutorials. Most tutorials will take you through using a database.
This is completely impossible using HTML and CSS only. You will have to use a server-side scripting language.