Throwing this out there but I know vitaly is such a hawk that he'll probably give me the answer or at least a solid hint ;)
I am basically uploading a batch of records and checking if the Company has already been added to a Company table. If not, I'll add the Company and then add all the records linking with the new CompanyID.
This isn't a pg-promise specific issue for sure but some of the dialogue here https://github.com/vitaly-t/pg-promise/blob/master/examples/select-insert.md show it's a true design concern and people are trying to come up with an elegant solution ... at least back in October.
I'm still new to asynchronous stuff but my gut says while I could insert a delay or possible chain promises together and process them in parallel (https://daveceddia.com/waiting-for-promises-in-a-loop/ or Resolve promises one after another (i.e. in sequence)?) but what vitaly mentions related to single query alternatives may be the real way to do this and not lock down the event loop
I'll hack on this today but I also want to do this in an elegant way since it's a common pattern
Thanks to vitaly in advance ;)
PS you rock vitaly
Related
I am new to SuiteScript and want to make our code more efficient. Looking at our code it seems there are many script of the same type for the same record type. For example, 3 clientscripts on a sales order. Is it bad practice to roll all of these scripts into the same script?
I also want to centralise the code. The last 3 years I've written in C# and any reusable code was placed in a relevant class. I want to rewrite the code we have in the same manner. For example, any method that is to do with a sales order is placed in a module called SalesOrderServices. This module can then be added to any of the scripts and the methods are all available if needed. My concern is this would make things less efficient due to loading all the modules in the Services module, even if they are not really needed. So as a second part to this question, is this a good idea or will it make our code be less efficient?
There is quite a lot to consider in a question like this, and there won't be one correct answer, but I'll chime in with my perspective.
It is not necessarily bad practice to combine the similar scripts into one, but it also is not bad practice to keep them separate. That's really a decision that only you and your team can decide on what is most efficient for you to maintain.
I do think you are right to want to break out any reusable functionality into separate modules, but I would be careful putting "everything related to a Sales Order" into one module. My personal preference is to design and group code based on features and business processes rather than around record types. If you try to modularize based on record type, what happens when you have an approval process that touches both Purchase Orders and Vendor Bills? Where will that live? I prefer small, focused modules rather than large monolithic ones, but that is just my preference. That doesn't work best for everyone and every team.
Have you proven that loading additional modules or Script records is a performance bottleneck for your system? I would be very surprised if that were the case, and so I would caution against premature optimization of these sorts of things. There are many facets of NetSuite that operate on the order of seconds and are out of your control, so saving a few micro- or milliseconds here and there isn't going to do anything appreciable for your users.
Disclaimer: I know that it is bad to not use an 'ORDER BY' in SQL when sorted data is required.
I am currently supporting a Pro*C program which is having a wierd-problem.
One of the possible causes of the wierd-problem may be that the original developers (from a long time ago) have not used ORDER BY in their SQL even though the program logic depends on it!
The program has been working fine all these years and started showing problems only recently.
We are trying to pin the wierd-problem to the ORDER BY mistake (there are other cause candidates like a recent port from Solaris to Linux which took place).
What shadowy things on the database end should we look at that may have changed the old sort order? Things like data files etc?
Anybody have any experience with Pro*C on Solaris magically sorting the result-set?
Thanks!
Since you know that the program cares about the order in which results are returned and you know that the query that is submitted is missing an ORDER BY clause, is there a reason that you don't just fix the problem rather than looking to try to figure out whether the actual order of results may have changed? If you fix the known ORDER BY problem and the "weird problem" you have disappears, that would provide some pretty good evidence that the "weird problem" is, in fact, caused by the missing ORDER BY.
Unfortunately, there are lots of things that might have caused the order of results to change many of which may be impossible to track down. The most obvious cause would be a change in the execution plan. That, in turn, may have been caused either because statistics changed or because statistics didn't change enough or because of a patch or because of an initialization parameter change or because of a client configuration change among other things. If you are licensed to use the AWR (Automatic Workload Repository), you might be able to find evidence that the plan has changed by looking to see if there are multiple PLAN_HASH_VALUE values for the SQL_ID in DBA_HIST_SQLSTAT over different days. If there are, you'd still have to try to figure out whether the different plans actually caused the results to be returned in a different order. Beyond a query plan change, though, there are dozens of other possible causes. The physical order of data on disk may have changed because someone reorganized the table or because someone moved data files around on the disk or because the SAN automatically rebalanced something by moving data around. Some data may have been cached (or may not have been cached) in general in the past that is now cached. An Oracle patch may have been applied.
I suggest that change your physical table with view and make your required order in that view.
example
TABLE_NOT_SORTED --> rename to --> PHYS_TABLE_NOT_SORTED
CREATE VIEW TABLE_NOT_SORTED
AS
SELECT * FROM PHYS_TABLE_NOT_SORTED
ORDER BY DESIRED_COLUMNS
For response to comment:
According to this question and Ask Tom's Answer, it seems that since Oracle does not guarantee a default sorting if you do not use "ORDER BY", they are free to change it. They are absolutely right of course. If you need sorting, use Order By.
Other than that we can not say anything about your code or default ordering.
So I am verifying new operational management systems, and one of these OS's sends pick lists to a scale-able number of handheld devices. It sends these using messages, and their pick lists may contain overlapping jobs. So in my virtual world, I need to make sure that two simulated humans don't pick the same job - whenever someone picks a job, all the job lists get refreshed, so that the picked job doesn't appear on anyone else's handheld anymore, but for me the message is still in the queue being handled, so I have to make sure to discard that option.
Basically I have this giant list with a mutex, and the more "people" hitting it faster, the slower I can handle messages, to the point where I'm no longer at real-time, which is bad, because I can't actually validate the system because I can't keep up with the messages. (two guys on the same isle will recognize that one is going to pick one object and the next guy should pick the 2nd item, but I need to check every single job i'm about to pick and see if it has been claimed by someone else already)
I've considered localized binning of the lists, but it actually doesn't solve the problem in the stupid case that breaks it anyway, tons of people working on the same row. Now granted this would probably be confusing for the real people as well, as in real life they need to do the same resolution, but I'm curious what the currently accepted "best" solution to this problem is.
PS - I already am implementing this in c++ and it's fast, fast enough that in any practical test I don't "need" this question answered, it's more because I'm curious that I'm asking.
Thanks in advance!
I see a problem in the design "giant list with (one) mutex". You simply can't provide the whole list in synchronized fashion, if the list size and/or access rate is unlimited. Basic math works against you. So what i would do is a mutexed flag on each job. You can't prevent a job from being displayed on someone's screen, but you can assure that he gets a graceful "no more available" error and THEN the updated list. If you ever wanted to reserve a seat on highly popular gig, you may have witnessed the solution.
I'm writing a phonebook search, that will query multiple remote sources but I'm wondering how it's best to approach this task.
The easiest way to do this is to take the query, start a thread per remote source query (limiting max results to say 10), waiting for the results from all threads and aggregating the list into a total of 10 entries and returning them.
BUT...which of the remote source is more important if all sources return at least 10 results, so then I would have to do a search on the search results. While this would yield accurate information it seems inefficient and unlikely to scale up well.
Is there a solution commercial or open source that I could use and extend, or is there a clever algorithm I can use that I've missed?
Thanks
John, I believe what you want is federated search. I suggest you check out Solr as a framework for this. I agree with Nick that you will have to evaluate the relative quality of the different sources yourself, and build a merge function. Solr has some infrastructure for this, as this email thread shows.
To be honest I haven't seen a ready solution, but this is why we programmers exist: to create a solution if one is not readily availble :-)
The way I would do it is similar to what you describe: using threads - if this is a web application then ajax is your friend for speed and usability, for a desktop app gui representation is not even an issue.
It sounds like you can't determine or guess upfront which source is the best in terms of reliability, speed & number of results. So you need to setup you program so that it determines best results on the fly. Let's say you have 10 data sources, and therfore 10 threads. When you fire up your threads - wait for the first one to return with results > 0. This is going to be you "master" result. As other threads return you can compare them to your "master" result and add new results. There is really no way to avoid this if you want to provide unique results. You can start displaying results as soon as you have your first thread. You don't have to update your screen right away with all the new results as they come in but if takes some time user may become agitated. You can just have some sort of indicator that shows that more results are available, if you have more than 10 for instance.
If you only have a few sources, like 10, and you limit the number of results per source you are waiting for, to like 10, it really shouldn't take that much time to sort through them in any programming language. Also make sure you can recover if your remote sources are not available. If let's say, you are waiting for all 10 sources to come back to display data - you may be in for a long wait, if one of the sources is down.
The other approach is to f00l user. Sort of like airfare search sites do - where they make you want a few seconds while they collect and sort results. I really like Kayak.com's implementation - as it make me feel like it's doing something unlike some other sites.
Hope that helps.
On a particular project we're working with a total of 10 team members.
After about a year working on the project (and using Mantis as a bug-/feature-tracker eversince), the bugtracker gets more and more difficult to use, as no standard has been setup that explains how to create new tasks, how to comment tasks etc. This leads to multiple entries for the same bugs, inability to easily find bugs when searching for them etc.
How do you organize your bugtracker? Do you use a lot of (sub)categories for different portions of your application (GUI, Backend etc), do you use tags in the title of tasks (i.e. "[GUI][OptionPage] The error")?
Is anyone in your team allowed to introduce new tasks or is this step channeled through a single "Mantis-master" (who would then know whether a new report is a duplicate or an entirely new entry)?
Always link a version control system commit to an issue and back so that you know which commits were made do solve which issue and why a certain commit was done.
What we did is to introduce a role for approve entries to the bug tracker. This role can be shared by different people. The process is either to approve, to approve with a small edit, or to reject the entry with the request for further editing or clarification.
It is better for the general understanding if the role is not given to people working in the (core) team.
In a "large" mantis system on the open web, I've seen the rules go something like
New: Anyone can enter a bug.
Acknowledged: A select few people can upgrade it to this level. These people have seen every new bug for a while, and thus they'll know if it's a duplicate. Or they can pass it back to the reporter for clarification until they understand it well enough to do this job.
Confirmed: Set by decision makers who basically say "We will be doing this".
I don't actually remember where it was, and more importantly I don't know how well it worked.