I have a Netezza DB with one table of 4 billion records. I am working on a migration project to move the data to SQL Server.
I created a simple SSIS ETL but that is running for a very long time and stops due to buffer memory issue.
What is the efficient quicker way(s) of transferring such huge amount of data?
You can try to separate the source data into batches, for example 1,000,000 rows(depends on your memory) a batch to merge into Netezza table.
DECLARE #BatchSize INT = 10000
WHILE 1 = 1
BEGIN
INSERT INTO [dbo].[Destination] --WITH (TABLOCK) -- Uncomment for 2008
(
FirstName
,LastName
,EmailAddress
,PhoneNumber
)
SELECT TOP(#BatchSize)
s.FirstName
,s.LastName
,s.EmailAddress
,s.PhoneNumber
FROM [dbo].[SOURCE] s
WHERE NOT EXISTS (
SELECT 1
FROM dbo.Destination
WHERE PersonID = s.PersonID
)
IF ##ROWCOUNT < #BatchSize BREAK
END
Related
I have a list of values in multiple columns in the spreadsheet from an excel workbook. How do I extract those values and store them into a variable? The spreadsheet looks as follows:
A B C
1 Col1 Col2 Col3
2 11 22 33
3 44 55 66
4 77 88 99
Here's the code I am using
$file = "C:\demo.xlsx"
$sheetName = "Sheet1"
$objExcel = New-Object -ComObject Excel.Application
$workbook = $objExcel.Workbooks.Open($file)
$sheet = $workbook.Worksheets.Item($sheetName)
$objExcel.Visible=$false
$rowCount = ($sheet.UsedRange.Rows).count
for ($i=2; $i -le $rowCount-1; $i++)
{
numberList += $sheet.Cells.Item($i,1).text -join "','"
numberList += $sheet.Cells.Item($i,2).text -join "','"
numberList += $sheet.Cells.Item($i,3).text -join "','"
}
$numberList
$objExcel.quit()
It's not concatenating values into single quotes & commas.
Am I using -join at the wrong place?
-join takes a collection of strings on the left-hand side, but in this expression:
$sheet.Cells.Item($i,1).text -join "','"
the left-hand side operand ($sheet.Cells.Item($i,1).text) is always going to be just a single string.
Wait with the quoting and concatenation until after you've collected all the cell values:
# enumerate all the cell values, store in $numberList
$numberList = for ($i=2; $i -le $rowCount-1; $i++)
{
$sheet.Cells.Item($i,1).Text
$sheet.Cells.Item($i,2).Text
$sheet.Cells.Item($i,3).Text
}
# Now we quote them ...
$numberList = $numberList |ForEach-Object { "'${_}'" }
# ... and finally join them into one big string
$numberList -join ','
Here is a brief history of my installers:
Over the years I had INSTALL-A and INSTALL-B.
Both of those installers included the same embedded INSTALL-C.
When either application INSTALL-A or INSTALL-B is uninstalled it performs this action:
procedure CurUninstallStepChanged(CurUninstallStep: TUninstallStep);
var
strCalendarInterfacesUninstallPath: String;
iResultCode: Integer;
begin
if CurUninstallStep = usUninstall then
begin
if (RegQueryStringValue(HKEY_LOCAL_MACHINE,
'SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall\{xxxx}_is1',
'QuietUninstallString', strCalendarInterfacesUninstallPath )) then
begin
if SuppressibleMsgBox(CustomMessage('UninstallCalendarInterfaces'), mbConfirmation, MB_YESNO, idNo) = idYes then
begin
if (not Exec('>', strCalendarInterfacesUninstallPath, '', SW_SHOW, ewWaitUntilTerminated, iResultCode)) then
SuppressibleMsgBox(SysErrorMessage(iResultCode), mbError, MB_OK, idOk);
end;
end;
end;
end;
In my recent releases of INSTALL-B I have changed the way it works and it no longer installs INSTALL-C. But I still have the uninstall code above in the installer.
The user might still have INSTALL-A on their computer. Of course, INSTALL-C can be uninstalled from the control panel.
So, is it now considered OK to just remove the uninstall code check from INSTALL-B?
Thanks.
PS. I apologise if this is a bit off topic for StackOverflow. But I can't work out the right place to ask this time of question. I have tried CodeReview before but I don't think that fits this question either.
I have the following basic xpage. I wan to add the fields on the reg1 and over1 and show it on total1.
I think I have to update the over1 field to refelect the new value or something like that. It always shows 0.
Any ideas?
<?xml version="1.0" encoding="UTF-8"?>
<xp:view xmlns:xp="http://www.ibm.com/xsp/core"
xmlns:xe="http://www.ibm.com/xsp/coreex">
<xp:this.data>
<xp:dominoDocument var="document1" formName="TestHrs"></xp:dominoDocument>
</xp:this.data>
<xp:table style="width:383.0px">
<xp:tr>
<xp:td>Regular</xp:td>
<xp:td>Overtime</xp:td>
<xp:td>Total</xp:td>
</xp:tr>
<xp:tr>
<xp:td>
<xe:djNumberTextBox id="reg1"
value="#{document1.rgHr1}">
</xe:djNumberTextBox>
</xp:td>
<xp:td>
<xe:djNumberTextBox id="over1"
value="#{document1.ovHr1}">
<xe:this.onChange><![CDATA[var reg1 = getComponent("reg1").getValue()
var over1 = getComponent("over1").getValue();
getComponent("total1").setValue(reg1 + over1);]]></xe:this.onChange>
</xe:djNumberTextBox>
</xp:td>
<xp:td>
<xe:djNumberTextBox id="total1"
value="#{document1.toHr1}" readOnly="true">
</xe:djNumberTextBox></xp:td>
</xp:tr>
<xp:tr>
<xp:td>
<xe:djNumberTextBox id="reg2"
value="#{document1.rgHr2}">
</xe:djNumberTextBox>
</xp:td>
<xp:td>
<xe:djNumberTextBox id="over2"
value="#{document1.ovHr2}">
<xe:this.onChange><![CDATA[#{javascript:var total1 = getComponent("reg2").getValue() + getComponent("over2").getValue();
getComponent("total2").setValue(total2);}]]></xe:this.onChange>
</xe:djNumberTextBox>
</xp:td>
<xp:td>
<xe:djNumberTextBox id="total2"
value="#{document1.toHr2}" readOnly="true">
</xe:djNumberTextBox></xp:td>
</xp:tr>
</xp:table>
</xp:view>
I would recommend not to use ssjs but csjs for this.
In the same event but with client Javascript, sum the values and put the total in the total field.
Keep in mind that as a readonly field, maybe you should check the property "Show disabled control for read-only".
Maybe you will need to do the sum in the querySave event of the document because as a read-only field the value is not updated. Not sure about this.
I find making the change in SSJS works best on an onBlur or onKeyUp, and making sure I'm getting a number (isNaN). CCJS may be better, as you can see from this link http://www.bleedyellow.com/blogs/xpages/entry/how_to_create_column_totals_in_xpages_view_and_repeat_controls?lang=en
I'm putting the post in below for a better answer. And it's not my post, but one I kept for reference.
Cheers,
Brian
In a Notes view it is fairly easy to add a column total, but how is this done in an XPages view control, or a repeat control ? It took me a trip to Lotusphere 2011 to find out.
Example:
We need the totals to:
1. stay right-aligned under the column that is being totalled, even when the table is being dynamically resized.
2. to show the total of the values from the documents that are being displayed on the page
1. For this example we need to add a panel to the tag of the view or repeat control:
<xp:viewPanel id="viewPanel1">
<xp:this.facets>
<xp:panel xp:key="footer" id="totals1">
<xp:tr>
<xp:td colspan="7"></xp:td>
<xp:td>
<xp:text escape="true" id="computedField1">
</xp:td>
<xp:td colspan="2"></xp:td>
<xp:td>
<xp:text escape="true" id="computedField2">
</xp:td>
<xp:td>
<xp:text escape="true" id="computedField3">
</xp:td>
</xp:tr>
</xp:panel>
</xp:this.facets>
The trick is in the xp:key property of the panel. Setting this to "footer", as is done for a pager control that you wish to have appear in the footer, allows you to add the extra row. Into this last row of the table that forms the view or repeat control, we place the computed fields. (Note that this example assumes that there are 7 columns before the column titled 'Value')
2. Lets assume that we provide a combo box on the XPage to allow the user to filter the documents that will display in the view or repeat control.
With Firebug, we can see that XPages creates a dynamic id and name for each component. For the combo box it is view:_id1:_id2:_id94:comboBox1.
But notice that all components on a custom control are allocated the same prefix of 'view:_id1:_id2:_id94:' to the name we gave the component:
So we can add the following client-side Javascript code to the 'On Change' event of our combo box:
var fcast = 0;
var i = 0;
var id_prefix = "#{id:comboBox1}".split("comboBox1")[0];
while (dojo.byId(id_prefix+"repeat1:"+i+":inputText3")) {
var it3= dojo.byId(id_prefix+"repeat1:"+i+":inputText3").innerHTML;
it3 = it3.replace("$","").replace(/,/gi,"");
fcast = fcast + parseFloat(it3);
i++;
}
var tot = formatCurrency(fcast);
dojo.byId(id_prefix+"computedField1").innerHTML = tot;
The third line uses "#{id:comboBox1}" to return the full id given to the comboBox1 field at run time. Please note that this code does not work if placed in a script library.
The while loop starts from the 0 row (first) and loops for every row on the view/repeat control where the inputText3 field exists.
Firebug shows that the values are rendered as span tags, so we need to use .innerHTML to get and set values.
The $ and commas need to be removed using the replace method, so the totals can be accumulated.
Finally, the formatCurrency function rounds to two decimal places and reformats the total with the $ and the commas. (This is a custom function, not shown here.)
Similar code can be created for computedField2 & computedField3.
So, if your customers need a report that shows column totals, or even need a report with filters that show totals, you can make use of these techniques.
Shout out to Paul Hannan and Marie Kehoe for their being the catalyst for this solution.
Im new to verilog and having troubling translating behavioral level code to register transfer level if any one can help me translate this and explain the difference it would be greatly appreciated. My code seems to be working but i cant seem to find any help as to write it in register transfer level.
module combinational_mult(product,multiplier,multiplicand);
input [31:0] multiplier,
input [63:0] multiplicand;
output product;
reg [63:0] product;
reg c;
reg [63:0] m;
integer i;
always #( multiplier or multiplicand )
begin
//initialize
product[63:32] = 32'd0;
product[31:0] = multiplier;
m = multiplicand;
c = 1'd0;
//add,shift algorithm for unsigned multiplication.
//following the notes.
for(i=0; i<32; i=i+1)
begin
if(product[0]) {c,product[63:32]} = product[31:16] + m ;
product[63:0] = {c,product[63:1]};
c = 0;
end
endmodule
module testbench;
reg [31:0] multiplier;
reg [63:0] multiplicand;
initial begin
$dumpfile("dump.vcd");
$dumpvars;
#10ns;
multiplier = 16'b1101_1001_1101_1001;
multiplicand = 16'b0110_1010_1101_1000;
#50ns;
multiplier = 16'b0;
multiplicand = 16'b0;
$finish;
end
combinational_mult dut ( product, multiplier, multiplicand);
endmodule
As your design is purely combinational, there isnt really a RTL (register transfer level) description that makes much sense or is that meaningful.
When describing digital sequential logic at an RTL level from Verilog; you need to have an idea of the registers (data holding) elements in your system and what affects the value each of these registers takes on. In your case, as there are not registers, you dont really have any RTL there, just a multiplier which is a valid unit in RTL. You can go deeper and describe your multipler in terms of how the various adders in your system are hooked up, but thats a bit lower than what Id call RTL.
A quick google search yields a text book chapter on RTL: http://cseweb.ucsd.edu/classes/sp13/cse140-a/lectures/rtl.pdf
Does anyone know what the %-sign in the netstat output below means? I can't find anything in the man page nor on the web.
tcp6 0 0 192.168.8.16%31621:5555 192.168.8.65%3162:47107 TIME_WAIT
tcp6 0 0 192.168.8.16%31621:5555 192.168.8.65%3162:47083 TIME_WAIT
tcp6 0 0 192.168.8.16%31621:5555 192.168.8.65%3162:47256 TIME_WAIT
Any input is greatly appreciated!
It's a IPv6 zone index.