I am not sure if I am going about this the correct way but I have a c# method which loads an excel sheet into a 2 dimentional object array. In this array item 1,1 - 1,16 contain headers, then 2-1 - 2-16 contain data that match up with those headers as do x-1 - x-16 from there on in. I would like to turn this array into a data table so ultimately I can have it in a format I will then import into an access or SQL server db depending on a clients needs. I have tried using the following code to no avail, but I have a feeling I am way off. Any help on this would be very much appreciated.
private void ProcessObjects(object[,] valueArray)
{
DataTable holdingTable = new DataTable();
DataRow holdingRow;
holdingTable.BeginLoadData();
foreach(int row in valueArray)
{
holdingRow = holdingTable.LoadDataRow(valueArray[row], true);
}
}
Any chance you're using a repository pattern (like subsonic or EF) or using LinqToSql?
You could do this (LinqToSql for simplicity):
List<SomeType> myList = valueArray.ToList().Skip([your header rows]).ConvertAll(f => Property1 = f[0] [the rest of your convert statement])
DataContext dc = new DataContext();
dc.SomeType.InsertAllOnSubmit(myList);
dc.SubmitChanges();
Related
I'm trying to dynamically reference Excel sheets or tables within the .dat for a Mixed Integer Problem in Vehicle Routing that I'm trying to solve in CPLEX (OPL).
The setup is a: .mod = model, .dat = data and a MS Excel spreadsheet
I have a 2 dimensional array with customer demand data = Excel range (for coding convenience I did not format the excel data as a table yet)
The decision variable in .mod looks like this:
dvar boolean x[vertices][vertices][scenarios]
in .dat:
vertices from SheetRead (data, "Table!vertices");
and
scenarios from SheetRead (data, "dont know how to yet"); this might not be needed
without the scenario Index everything is fine.
But as the demand for the customers changes in this model I'd like to include this via changing the data base reference.
Now what I'd like to do is one of 2 things:
Either:
Change the spreadsheet in Excel so that depending on the scenario I get something like that in .dat:
scenario = 1:
vertices from SheetRead (data, "table-scenario-1!vertices");
scenario = 2:
vertices from SheetRead (data, "table-scenario-2!vertices");
so changing the spreadsheet for new base data,
or:
Change the range within the same spreadsheet:
scenario = 1:
vertices from SheetRead (data, "table!vertices-1");
scenario = 2:
vertices from SheetRead (data, "table!vertices-2");
either way would be fine.
Knowing how 3D Tables in Excel are created using multiple spreadsheets with 2D Tables grouped, the more natural approach seems to be, to have vertices always reference the same range in every Excel spreadsheet while depending on the scenario the spreadsheet/page is switched, but I just don't know how to.
Thanks for the advice.
Unfortunately, the arguments to SheetConnection must be a string literal or an Id (see the OPL grammar in the user manual here). And similarly for SheetRead. This means, you cannot have dynamic sources for a sheet connection.
As we discussed in the comments, one option is to add an additional index to all data: the scenario. Then always read the data for all scenarios and in the .mod file select what you want to actually use.
at https://www.ibm.com/developerworks/community/forums/html/topic?id=5af4d332-2a97-4250-bc06-76595eef1ab0&ps=25 I shared an example where you can set a dynamic name for the Excel file. The same way you could have a dynamic range, the trick is to use flow control.
sub.mod
float maxOfx = 2;
string fileName=...;
dvar float x;
maximize x;
subject to {
x<=maxOfx;
}
execute
{
writeln("filename= ",fileName);
}
and then the main model is
main {
var source = new IloOplModelSource("sub.mod");
var cplex = new IloCplex();
var def = new IloOplModelDefinition(source);
var opl = new IloOplModel(def,cplex);
for(var k=11;k<=20;k++)
{
var opl = new IloOplModel(def,cplex);
var data2= new IloOplDataElements();
data2.fileName="file"+k;
opl.addDataSource(data2);
opl.generate();
if (cplex.solve()) {
writeln("OBJ = " + cplex.getObjValue());
} else {
writeln("No solution");
}
opl.postProcess();
opl.end();
}
}
I have a map1 which holds the information as
[40256942,6] [60246792,5]
Now that I want to prepare a map2 that holds information such as
itemNo, 40256942
qty, 6
itemNo, 60246792
qty, 5
to prepare final information as json
“partialArticlesInfo”: [{itemNo:”40256942”, availQty:”6”}, {itemNo:”60246792”, availQty:”5”}]
I am trying to iterate map1 to retrieve values and set that against the key. But I am getting only one entry which is last one. Is there any way , I get the new map with entries such as mentioned above
Map<String, String> partialArticlesInfo = new HashMap<String,String>();
Map<String, String> partialArticlesTempMap = null;
for (Map.Entry<String,String> entry : partialStockArticlesQtyMap.entrySet())
{
partialArticlesTempMap = new HashMap<String,String>();
partialArticlesTempMap.put("itemNo",entry.getKey());
partialArticlesTempMap.put("availQty",entry.getValue());
partialArticlesInfo.putAll(partialArticlesTempMap);
}
In Java (I'm assuming you're using Java, in the future it would be helpful to specify that) and every other language I know of, a map holds mappings between keys and values. Only one mapping is allowed per key. In your "map2", the keys are "itemNo" and "availQty". So what is happening is that your for loop sets the values for the first entry, and then is overwriting them with the data from the second entry, which is why that is the only one you see. Look at Java - Map and Map - Java 8 for more info.
I don't understand why you are trying to put the data into a map, you could just put it straight into JSON with something like this:
JSONArray partialArticlesInfo = new JSONArray();
for (Map.Entry<String,String> entry : partialStockArticlesQtyMap.entrySet()) {
JSONObject stockEntry = new JSONObject();
stockEntry.put("itemNo", entry.getKey());
stockEntry.put("availQty", entry.getValue());
partialArticlesInfo.put(stockEntry);
}
JSONObject root = new JSONObject();
root.put("partialArticlesInfo",partialArticlesInfo);
This will take "map1" (partialStockArticlesQtyMap in your code) and create a JSON object exactly like your example - no need to have map2 as an intermediate step. It loops over each entry in map1, creates a JSON object representing it and adds it to a JSON array, which is finally added to a root JSON object as "partialArticlesInfo".
The exact code may be slightly different depending on which JSON library you are using - check the docs for the specifics.
I agree with Brendan. Another solution would be otherwise to store in the Set or List objects like the following.
class Item {
Long itemNo;
int quantity;
public int hashCode() {
Long.hashCode(itemNo) + Integer.hashCode(quantity);
}
public int equals(Object other) {
other instanceOf Item && other.itemNo == this.itemNo && other.quantity = this.quantity;
}
}
}
then you can use the JsonArray method described by him to get the Json string in output
This means that adding new variables to the object won't require any more effort to generate the Json
I apologize if my question is not adequately described. I am a .Net / C# / SSIS newbie. Also apologize if already answered, I've tried searching here and Google for a few hours without luck.
Background: I need to pull data rows from a SharePoint 365 list and unpivot certain columns into a format ready for import to a SQL Server table. I realize that SSIS has an Odata Source and built-in Unpivot component, and I've successfully used those for proof of concept.
However, I believe I need a custom script component because the nummber of columns to unpivot from the source SharePoint list is variable. Each month or so, a new column will be added (it relates to a financial forecasting "tool" in SharePoint, and the latest forecasted month changes). My understanding is that source columns must be defined in SSIS at design time, so if my source columns are changing, the only way I can think to address this without manually changing the SSIS data flow each month is to programatically combine the Odata source and unpivot functions into a custom script component.
I understand, or can figure out, the unpivot logic. The part I am struggling with is how to actually connect to and expose a given list and it's data rows / columns as lists that I can loop through and perform my mapping to the output columns.
My "starting point" for requested guidance is this:
1) Created and successfully connected to the SharePoint site in question using standard SSIS Odata Connection Manager.
2) Created standard "Script Component", type = source, on the visual designer.
3) From script component properties, associated the Odata connection manager with the name of "myConnection".
4) Need help -> within the script component, open a connection to a specific list, read it's contents, and perform unpivot logic.
For illustration purposes, assume the source is a SharePoint list with two "fixed" string columns titled Study and Site, and a variable number of columns with names matching month-end dates (e.g. 9/30/2016, 10/31/2016, etc.) that contain integer values. I want to map the study and site source columns to destination columns of the same name and unpivot the month columns where column name is mapped to ProjectionMonth and the integer value is mapped to ProjectionValue.
Here's the basic algorithm I have in mind (I realize this isn't compilable - that's where I need your help!):
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
using System.Data.SqlClient;
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
IDTSConnectionManager100 connMgr;
SqlConnection sqlConn; // from MSDN tutorial, but I don't know how to adapt to Odata/SharePoint 365 list
SqlDataReader sqlReader; // not sure how to adapt
public override void AcquireConnections(object Transaction)
{
connMgr = this.Connections.myConnection;
sqlConn = (SqlConnection)connMgr.AcquireConnection(null); // This is from MSDN tutorial, but I don't know how to adapt to Odata
}
public override void PreExecute()
{
//Not sure how to adapt to odata to read specific SharePoint list
SqlCommand cmd = new SqlCommand("SELECT * FROM <some sharepoint list>", sqlConn);
sqlReader = cmd.ExecuteReader();
}
public override void PostExecute()
{
sqlReader.Close(); // Not sure how to adapt.
}
public override void CreateNewOutputRows()
{
string myStudy;
string mySite;
string myProjectionMonth;
string myProjectionValue;
// This is a rough representation of the logic needed.
// I realize that the actual code to access column values / names depends on the class(es) I need to use, but not sure what those classes are / how to access
foreach (myListRow in sqlConn.rows)
{
myStudy = myListRow.Columns["Study"].value;
mySite = myListRow.Columns["Site"].value;
foreach (myColumn in myListRow.Columns)
if (DateTime.TryParse(myColumn.Name, out dateValue))
{
myProjectionMonth = myColumn.Name;
myProjectionValue = myColumn.Value;
Output0Buffer.AddRow();
Output0Buffer.Study = myStudy;
Output0Buffer.Site = mySite;
Output0Buffer.ProjectionMonth = myProjectionMonth;
Output0Buffer.ProjectionValue = myProjectionValue;
}
}
}
}
Edit: as an example, assume the source SharePoint list has the following:
Study Site 9/30/2016 10/31/2016
123 ABC 5 10
I want to the script component to connect to the list, read it's contents, and return the following unpivoted data set for eventual load into SQL Server:
Study Site ProjectionMonth ProjectionValue
123 ABC 9/30/2016 5
123 ABC 10/31/2016 10
So this is probably not an ideal way of doing it, and it doesn't leverage the standard SSIS Odata Connection Manager that I wanted... but it does technically get the job done and it's good enough for me, for now.
Would be interested on any suggested feedback / improvements / etc., if you have any.
#region Namespaces
using System;
using Microsoft.SharePoint.Client;
using System.Security;
using System.Collections.Generic;
#endregion
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
public override void CreateNewOutputRows()
{
// Connect to SharePoint
ClientContext context = new ClientContext("https://<redacted>.sharepoint.com/Development");
SecureString passWord = new SecureString();
foreach (char c in Variables.sharepointPassword.ToCharArray()) passWord.AppendChar(c);
context.Credentials = new SharePointOnlineCredentials("<redacted>#<redacted>.onmicrosoft.com", passWord);
// Define the target list
List EnrollmentList = context.Web.Lists.GetByTitle("Enrollment Projections");
// Find all fields in the target list
FieldCollection myFields = EnrollmentList.Fields;
context.Load(myFields);
context.ExecuteQuery();
// Load all rows from the target list
CamlQuery query = CamlQuery.CreateAllItemsQuery(100);
ListItemCollection items = EnrollmentList.GetItems(query);
context.Load(items);
context.ExecuteQuery();
//Store valid date fields
List<Field> myDateFields = new List<Field>();
foreach (Field tempField in myFields)
{
DateTime tempDate;
if (DateTime.TryParse(tempField.Title, out tempDate))
{
myDateFields.Add(tempField);
}
}
string myStudy;
string mySite;
string myMonth;
string myValue;
foreach (ListItem listItem in items)
{
myStudy = listItem["Study"].ToString();
mySite = listItem["Site"].ToString();
foreach (Field tempField in myDateFields)
{
myMonth = tempField.Title;
myValue = listItem[tempField.InternalName.ToString()].ToString();
Output0Buffer.AddRow();
Output0Buffer.Study = myStudy;
Output0Buffer.Site = mySite;
Output0Buffer.ProjectedMonth = myMonth;
Output0Buffer.ProjectedValue = Convert.ToInt32(myValue);
}
}
}
}
I've been trying to create tables and make them to leave some space between its bottom border and whatever comes after the table (usually text).
As far as I have crawl through ooxml specification I understand that I need to add to the table this chain of elements tblPr (table properties) -> tblpPr (table position properties), and set the attribute bottomFromText to the specific amount space I want between the table and the next element, also the vertAnchor attribute (right now I'm configuring this with the "text" value) and finally the tblpY attribute.
A q&d snippet of what I'm doing is this (java and apache poi):
XWPFTable table = document.createTable();
CTTblPr _cttblpr = table.getCTTbl().addNewTblPr();
_cttblpr.addNewTblpPr().setBottomFromText(BigInteger.valueOf(284));
_cttblpr.getTblpPr().setVertAnchor(STVAnchor.TEXT);
_cttblpr.getTblpPr().setTblpY(BigInteger.valueOf(1));
My main reference has been this. Also I have been creating (with LibreOffice writer and Microsoft Office 2007) simple documents with just a table and the space I want and extracting the files inside it (word/document.xml specifically) to see in place this. All my efforts to achieve this have been unsuccessful by now.
Do you know what is wrong here? I strongly believe I have missconcepts...
Thank you in advance.
You're right, you need w:bottomFromText, for example:
<w:tbl>
<w:tblPr>
<w:tblpPr w:leftFromText="187" w:rightFromText="187" w:bottomFromText="4320" w:vertAnchor="text" w:tblpY="1"/>
<w:tblOverlap w:val="never"/>
</w:tblPr>
Based on the above, your code looks plausible.
For comparison, if you were doing it with docx4j, you'd create that in one of 2 ways.
The first way is to explicitly use the JAXB object factory:
org.docx4j.wml.ObjectFactory wmlObjectFactory = new org.docx4j.wml.ObjectFactory();
Tbl tbl = wmlObjectFactory.createTbl();
JAXBElement<org.docx4j.wml.Tbl> tblWrapped = wmlObjectFactory.createBodyTbl(tbl);
// Create object for tblPr
TblPr tblpr = wmlObjectFactory.createTblPr();
tbl.setTblPr(tblpr);
// Create object for tblpPr
CTTblPPr tblppr = wmlObjectFactory.createCTTblPPr();
tblpr.setTblpPr(tblppr);
tblppr.setLeftFromText( BigInteger.valueOf( 187) );
tblppr.setRightFromText( BigInteger.valueOf( 187) );
tblppr.setBottomFromText( BigInteger.valueOf( 4320) );
tblppr.setVertAnchor(org.docx4j.wml.STVAnchor.TEXT);
tblppr.setTblpY( BigInteger.valueOf( 1) );
// Create object for tblOverlap
CTTblOverlap tbloverlap = wmlObjectFactory.createCTTblOverlap();
tblpr.setTblOverlap(tbloverlap);
tbloverlap.setVal(org.docx4j.wml.STTblOverlap.NEVER);
The second is to unmarshall a string:
String openXML = "<w:tbl xmlns:w=\"http://schemas.openxmlformats.org/wordprocessingml/2006/main\">
+ "<w:tblPr>
+ "<w:tblpPr w:bottomFromText=\"4320\" w:leftFromText=\"187\" w:rightFromText=\"187\" w:tblpY=\"1\" w:vertAnchor=\"text\"/>"
+ "<w:tblOverlap w:val=\"never\"/>"
+"</w:tblPr>"
etc
+"</w:tbl>";
Tbl tbl = (Tbl)XmlUtils.unmarshalString(openXML);
I have data in a database, and my code is accessing it using LINQ to Entities.
I am writing some software where I need to be able to create a dynamic script. Clients may write the scripts, but it is more likely that they will just modify them. The script will specify stuff like this,
Dataset data = GetDataset("table_name", "field = '1'");
if (data.Read())
{
string field = data["field"];
while (cway.Read())
{
// do some other stuff
}
}
So that script above is going to read data from the database table called 'table_name' in the database into a list of some kind based on the filter I have specified 'field='1''. It is going to be reading particular fields and performing normal comparisons and calculations.
The most important thing is that this has to be dynamic. I can specify any table in our database, any filter and I then must be able to access any field.
I am using a script engine that means the script I am writing has to be written in C#. Datasets are outdated and I would rather keep away from them.
Just to re-iterate I am not really wanting to keep with the above format, and I can define any method I want to behind the scenes for my C# script to call. The above could end up like this for instance,
var data = GetData("table_name", "field = '1'");
while (data.ReadNext())
{
var value = data.DynamicField;
}
Can I use reflection for instance, but perhaps that would be too slow? Any ideas?
If you want to read dynamically a DataReader context, it's a pretty easy step:
ArrayList al = new ArrayList();
SqlDataReader dataReader = myCommand.ExecuteReader();
if (dataReader.HasRows)
{
while (dataReader.Read())
{
string[] fields = new string[datareader.FieldCount];
for (int i =0; i < datareader.FieldCount; ++i)
{
fields[i] = dataReader[i].ToString() ;
}
al.Add(fields);
}
}
This will return an array list composed by a dynamic object based on the number of field the reader has.