Legacy code, is preparing a lot of data, and sometimes it is not used in rendered TWIG template.
Is there a way to know, what wasn't use during rendering process?
Will be nice to see, or force to render all other data, that was not touched.
PS: my idea is just get rid of that not used things, and will be nice to find and see them, with not digging to millions lines of code
PS2: twig is used as separate composer package, and is not connected neither Symphony or Drupal
From a performance/maintenance point of view, is it better to write my custom modules with netsuite all as one big JS, or multiple segmented script files.
If you compare it with a server side javascript language, say - Node.js the most popular, every module is written into separate file.
I generally take the approach of Object oriented javascript and put each class in a separate file which helps to organise the code.
One of the approach you can take is in development keep separate files and finally merge all files using js minifier tool like Google closure compiler when you deploy your code for production usage which can give you best of both worlds, if you are really bothered about every nano/mini seconds of performance.
If you see SuiteScript 2.0 architecture, it encourages module architecture which is easier to manage as load only those modules that you need, and it is easier to maintain multiple code files i.e. one per module considering future enhancements, bug fixes and code reuse.
Performance can never be judge by the line count of your module. We generally maintain modules for maintaining the readability and simplicity of the code. It is a good practice to put all generic functionalities in to an Utility script and use it as a library across all the modules. Again it depends on your code logic and programming style. So if you want to create multiple segments of your js file for more readability I dont think its a bad idea.
I'm using snippets to break up long templates into more manageable chunks, is this bad practice and should I only use snippets for reusable code?
Thanks
Lee
It depends of what you are doing in your templates.
It is indeed generally a good practice to reuse part of templates in smaller chunks, but it's recommended to avoid the usage of embeds because of underlying performance issues.
Snippet are cool as soon as you don't have to pass variables to your sub-templates.
Maybe a good approach for you if your templates are not already totally coded is to use the 'template partial' technique. See the following tutorials here under.
Template Partials using Stash
Homegrown plugin to create template “partials” for ExpressionEngine
You should look at using an add-on, I use either Low Variables or Republic Variables (free), where you can store snippets and parse them early or late. They can also be saved as files which is handy.
You can find them on http://devot-ee.com
I'm looking to scrape public data off of many different local government websites. This data is not provided in any standard format (XML, RSS, etc.) and must be scraped from the HTML. I need to scrape this data and store it in a database for future reference. Ideally the scraping routine would run on a recurring basis and only store the new records in the database. There should be a way for me to detect the new records from the old easily on each of these websites.
My big question is: What's the best method to accomplish this? I've heard some use YQL. I also know that some programming languages make parsing HTML data easier as well. I'm a developer with knowledge in a few different languages and want to make sure I choose the proper language and method to develop this so it's easy to maintain. As the websites change in the future the scraping routines/code/logic will need to be updated so it's important that this will be fairly easy.
Any suggestions?
I would use Perl with modules WWW::Mechanize (web automation) and HTML::TokeParser (HTML parsing).
Otherwise, I would use Python with the Mechanize module (web automation) and the BeautifulSoup module (HTML parsing).
I agree with David about perl and python. Ruby also has mechanize and is excellent for scraping. The only one I would stay away from is php due to it's lack of scraping libraries and clumsy regex functions. As far as YQL goes, it's good for some things but for scraping it really just adds an extra layer of things that can go wrong (in my opinion).
Well, I would use my own scraping library or the corresponding command line tool.
It can use templates which can scrape most web pages without any actual programming, normalize similar data from different sites to a canonical format and validate that none of the pages has changed its layout...
The command line tool doesn't support databases through, there you would need to program something...
(on the other hand Webharvest says it supports databases, but it has no templates)
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 11 years ago.
So I assume I'm not the only one. I'm wondering if there are others out there who have compiled a personal code library. Something that you take from job to job that has examples of best practices, things you are proud of, or just common methods you see yourself using over and over.
I just recently started my C# library. It already has quite a few small items. Common Regex validations, interfaces for exception handling, some type conversion overloads, enum wrappers, sql injection detection methods, and some common user controls with AJAX toolkit examples.
I'm wondering what kind of things do you have in yours?
I use my own wiki where I post code snippets and commentaries.
I find that more useful than having my own library. And since they are essentially notes and not full programs there isn't a problem with who owns the code (you or your employer ).
PS: I don't hide the fact that I have that from my employer. In fact most of them were positive and even asked for a copy.
Because I primarily do web development, I've abstracted out some common features that I end up doing frequently on sites for clients.
Ajax Emailer. Nearly every site I work on has some type of contact form. I wrote a utility that allows me to drop some HTML on a page, having JavaScript field validation, and a PHP library that requires me to change a few parameters to work with each client's mail server. The only thing I have to write is CSS each time I include it on to a page.
Stylesheet skeleton generator. I wrote a small JavaScript utility that walks the DOM for whatever page it has been included on and then stubs out a valid CSS skeleton so that I can immediately start writing styles without having to do the repetitive task for every site I work on.
JavaScript Query String Parser. Occasionally I need to parse the query string but it doesn't warrant any major modifications to the server (such as installing PHP), so I wrote a generic JavaScript utility that I can easily configure for each site.
I've got other odds and end utilities, as well, but they are kind of hacked together for personal use. I'd be embarrassed to let anyone see the source.
Update
Several people have asked for my stylesheet skeleton generator in the comments so I'm providing a link to the project here. It's more or less based on the way that I structure my XHTML and format my CSS, but hopefully you'll find it useful.
I have found that using Snipplr makes this incredibly convenient. You can tag items, save favorites, search by keyword, etc. I mostly use it for Vim-related snippets (common commands, vimrc file, etc.), but it can be used for anything. Check it out.
I have my personal C++ cross platform library here: http://code.google.com/p/kgui/
It's open source LGPL, I use it in my hobby / volunteer projects. I started it about 3 years ago and have been slowly adding functionality to it.
Back in the days of C programming on MacOS 7, i did write a fairly extensive OO library (yes, OOP in very old C) mostly to handle dialog windows. I abandoned it for PowerPlant (a nice C++ from Metrowerks) during the switch from 68k to PPC processors.
A little after that, i began writing web apps, first in PHP, recently in Django. On this aspect, my reusable code is limited to some tricks and code style.
But for all non-web (or with only small web componets), i've been using Lua. It's so fast to write and rewrite code, that there's very little incentive in reusing code. I mean, what's the point of copying a 10 line function and then adapt it? it's faster to rewrite it just for this project.
That's not so wasteful as it sounds. Lua code is so succint that my apps can be very complex, but seldom have more than a couple thousands lines.
At the same time, several Lua projects imply interfacing to C libraries. It's very easy to write bindings to existing libraries, so i just do that as a subproject. And these modules are what i do reuse! once and again... with very little (if any) changes from one project to the other.
In short: non-web projects are usually one-off Lua code, and some heavily reused binding modules.
I use Source Code Library from http://www.highdots.com/products/source-code-library/ since I can manage different textfiles, notes, screenshots and different programming languages.
I have several utility MATLAB functions that I have taken with me as I move from job to job, particularly ones that enforce W3C standards on the plots I make to ensure that text and background colors have a good luminosity ratio. I also have a function that uses ActiveX to insert a MATLAB figure into PowerPoint.
I keep my personal code libraries on CPAN. I'm not even sure how I'd do this in other languages anymore. It's just too integrated in the way that I think about programming now.
For my PHP work I started with a small file of simple things: a mail function that checks inputs for header attacks, and email validator, an input srubber, that type of thing. Over time it has grown into a application framework for quickly developing one off applications that can be templated by our graphic designer.
I have a library that i use quite extensively. I started fresh with c# and kinda threw all of the legacy stuff out the window. I find them very handy and i rewrite/refactor them often (some of them). Some of the stuff i have is:
Auxiliary (things like IsRunningLocal, InternetDetection)
Standard Classes or Structs for: Address, CreditCard, Person
I have .dll's for both win and web stuff, some very logical like a .dll for shopping cart stuff
I wrote a quick and simple library in Java which I can add code snippets to. I plan to extend it to a full framework for development at some point but only when time allows. I have all sorts in there from simple functions to full blown pages and features. Its so helpful to have when developing because as a web designer, all I need to do is change the CSS of the page.