I have data in a database, and my code is accessing it using LINQ to Entities.
I am writing some software where I need to be able to create a dynamic script. Clients may write the scripts, but it is more likely that they will just modify them. The script will specify stuff like this,
Dataset data = GetDataset("table_name", "field = '1'");
if (data.Read())
{
string field = data["field"];
while (cway.Read())
{
// do some other stuff
}
}
So that script above is going to read data from the database table called 'table_name' in the database into a list of some kind based on the filter I have specified 'field='1''. It is going to be reading particular fields and performing normal comparisons and calculations.
The most important thing is that this has to be dynamic. I can specify any table in our database, any filter and I then must be able to access any field.
I am using a script engine that means the script I am writing has to be written in C#. Datasets are outdated and I would rather keep away from them.
Just to re-iterate I am not really wanting to keep with the above format, and I can define any method I want to behind the scenes for my C# script to call. The above could end up like this for instance,
var data = GetData("table_name", "field = '1'");
while (data.ReadNext())
{
var value = data.DynamicField;
}
Can I use reflection for instance, but perhaps that would be too slow? Any ideas?
If you want to read dynamically a DataReader context, it's a pretty easy step:
ArrayList al = new ArrayList();
SqlDataReader dataReader = myCommand.ExecuteReader();
if (dataReader.HasRows)
{
while (dataReader.Read())
{
string[] fields = new string[datareader.FieldCount];
for (int i =0; i < datareader.FieldCount; ++i)
{
fields[i] = dataReader[i].ToString() ;
}
al.Add(fields);
}
}
This will return an array list composed by a dynamic object based on the number of field the reader has.
Related
Suppose I have two tables USER_GROUP and USER_GROUP_DATASOURCE. I have a classic relation where one userGroup can have multiple dataSources and one DataSource simply is a String.
Due to some reasons, I have a custom RecordMapper creating a Java UserGroup POJO. (Mainly compatibility with the other code in the codebase, always being explicit on whats happening). This mapper sometimes creates simply POJOs containing data only from the USER_GROUP table, sometimes also the left joined dataSources.
Currently, I am trying to write the Multiset query along with the custom record mapper. My query thus far looks like this:
List<UserGroup> = ctx
.select(
asterisk(),
multiset(select(USER_GROUP_DATASOURCE.DATASOURCE_ID)
.from(USER_GROUP_DATASOURCE)
.where(USER_GROUP.ID.eq(USER_GROUP_DATASOURCE.USER_GROUP_ID))
).as("datasources").convertFrom(r -> r.map(Record1::value1))
)
.from(USER_GROUP)
.where(condition)
.fetch(new UserGroupMapper()))
Now my question is: How to create the UserGroupMapper? I am stuck right here:
public class UserGroupMapper implements RecordMapper<Record, UserGroup> {
#Override
public UserGroup map(Record rec) {
UserGroup grp = new UserGroup(rec.getValue(USER_GROUP.ID),
rec.getValue(USER_GROUP.NAME),
rec.getValue(USER_GROUP.DESCRIPTION)
javaParseTags(USER_GROUP.TAGS)
);
// Convention: if we have an additional field "datasources", we assume it to be a list of dataSources to be filled in
if (rec.indexOf("datasources") >= 0) {
// How to make `rec.getValue` return my List<String>????
List<String> dataSources = ?????
grp.dataSources.addAll(dataSources);
}
}
My guess is to have something like List<String> dataSources = rec.getValue(..) where I pass in a Field<List<String>> but I have no clue how I could create such Field<List<String>> with something like DSL.field().
How to get a type safe reference to your field from your RecordMapper
There are mostly two ways to do this:
Keep a reference to your multiset() field definition somewhere, and reuse that. Keep in mind that every jOOQ query is a dynamic SQL query, so you can use this feature of jOOQ to assign arbitrary query fragments to local variables (or return them from methods), in order to improve code reuse
You can just raw type cast the value, and not care about type safety. It's always an option, evne if not the cleanest one.
How to improve your query
Unless you're re-using that RecordMapper several times for different types of queries, why not do use Java's type inference instead? The main reason why you're not getting type information in your output is because of your asterisk() usage. But what if you did this instead:
List<UserGroup> = ctx
.select(
USER_GROUP, // Instead of asterisk()
multiset(
select(USER_GROUP_DATASOURCE.DATASOURCE_ID)
.from(USER_GROUP_DATASOURCE)
.where(USER_GROUP.ID.eq(USER_GROUP_DATASOURCE.USER_GROUP_ID))
).as("datasources").convertFrom(r -> r.map(Record1::value1))
)
.from(USER_GROUP)
.where(condition)
.fetch(r -> {
UserGroupRecord ug = r.value1();
List<String> list = r.value2(); // Type information available now
// ...
})
There are other ways than the above, which is using jOOQ 3.17+'s support for Table as SelectField. E.g. in jOOQ 3.16+, you can use row(USER_GROUP.fields()).
The important part is that you avoid the asterisk() expression, which removes type safety. You could even convert the USER_GROUP to your UserGroup type using USER_GROUP.convertFrom(r -> ...) when you project it:
List<UserGroup> = ctx
.select(
USER_GROUP.convertFrom(r -> ...),
// ...
I am trying to refactor some code. All along the code base, I have code that stores user-related informations like this :
this.users[user][field] = data ;
Note that there could be an arbitrary number of subfields (including none), like this:
this.users[user][field][subfield1][subfield2] = data ;
Theses informations are retrieved like this :
var result = this.users[user][field] ;
In production, the actual data will be stored into Redis.
To prepare for this, I would like to refactor those access into two functions, say function storeUserData(user, fields, data) and function retrieveUserData(user, field).
I can do it trivially if there is only one field. But how can I write those two functions to allow for an arbitrary number of subfields (ideally none as well) ?
Edit : the long-term goal is to blur the difference between redis keys and node.js arrays.
That way I could for instance access a node subfield like this : 'users.user.id' and also have it in redis like this :users.user.*.id. Does that seem feasible ?
You can pass the fields argument as an Array. Then in your read function do something like
function retrieveUserData(user, fields) {
// imagine fields is ['field', 'subfield1']
var fieldVariable = this.users[user]
for (f = 0; f < fields.length; ++f) {
fieldVariable = fieldVariable[fields[f]];
}
// in this line fieldVariable will point to this.users[user]['field']['subfield1']
return fieldVariable;
}
Hope it helps!
I need to grab a large amount of data from one set of tables and SQLBulkInsert into another set...unfortunately the source tables are ALL varchar(max) and I would like the destination to be the correct type. Some tables are in the millions of rows...and (for far too pointless policital reasons to go into) we can't use SSIS.
On top of that, some "bool" values are stored as "Y/N", some "0/1", some "T/F" some "true/false" and finally some "on/off".
Is there a way to overload IDataReader to perform type conversion? Would need to be on a per-column basis I guess?
An alternative (and might be the best solution) is to put a mapper in place (perhaps AutoMapper or custom) and use EF to load from one object and map into the other? This would provoide a lot of control but also require a lot of boilerplate code for every property :(
In the end I wrote a base wrapper class to hold the SQLDataReader, and implementing the IDataReader methods just to call the SQLDataReader method.
Then inherit from the base class and override GetValue on a per-case basis, looking for the column names that need translating:
public override object GetValue(int i)
{
var landingColumn = GetName(i);
string landingValue = base.GetValue(i).ToString();
object stagingValue = null;
switch (landingColumn)
{
case "D4DTE": stagingValue = landingValue.FromStringDate(); break;
case "D4BRAR": stagingValue = landingValue.ToDecimal(); break;
default:
stagingValue = landingValue;
break;
}
return stagingValue;
}
Works well, is extensible, and very fast thanks to SQLBulkUpload. OK, so there's a small maintenance overhead, but since the source columns will very rarely change, this doesn't really affect anything.
Hi I have an existing database with a table with 30 fields, I want to split the table into many models so I could retrieve/save fields that I need and not every time retrieve/save the whole object from the db. using c#.
I think I should be using Code-First. Could someone provide an example or a tutorial link?
thanks,
You don't need to split table to be able to load a subset of field or persist subset of fields. Both operations are available with the whole table mapped to single entity as well.
For selection you simply have to use projection:
var data = from x in context.HugeEntities
select new { x.Id, x.Name };
You can use either anonymous type in projection or any non-mapped class.
For updates you can simply use:
var data = new HugeEntity { Id = existingId, Name = newName };
context.HugeEntities.Attach(data);
var dataEntry = context.Entry(data);
dataEntry.Property(d => d.Name).IsModified = true; // Only this property will be updated
context.SaveChanges();
Or:
var data = new HugeEntity { Id = existingId };
context.HugeEntities.Attach(data);
data.Name = newName;
context.SaveChanges(); // Now EF detected change of Name property and updated it
Mapping multiple entities to single table must follows very strict rules and it is possible only with table splitting were all entities must be related with one-to-one relation (and there are some problems with more than two entities per split table in code first) or with table-per-hierarchy inheritance. I don't think that you want to use any of them for this case.
I am making some changes to a site using ExpressionEngine, there is a concept I can't seem to quite understand and am probably coding workarounds that have pre-provided methods.
Specifically with channels and members, the original creator has added several custom fields. I am from a traditional database background where each column in a table has a specific meaningful name. I am also used to extending proprietary data by adding a related table joined with a unique key, again, field names in the related table are meaningful.
However, in EE, when you add custom fields, it creates them in the table as field_id_x and puts an entry into another table telling you what these are.
Now this is all nice from a UI point of view for giving power to an administrator but this is a total headache when writing any code.
Ok, there's template tags but I tend not to use them and they are no good in a database query anyway.
Is there a simple way to do a query on say the members table and then address m_field_1 as what its really called - in my case "addresslonglat".
There are dozens of these fields in the table I am working on and at the moment I am addressing them with fixed names like "m_field_id_73" which means nothing.
Anybody know of an easy way to bring the data and its field names together easily?
Ideally i'd like to do the following:
$result = $this->EE->db->query("select * from exp_member_data where member_id = 123")->row();
echo $result->addresslonglat;
Rather than
echo $result->m_field_id_73;
This should work for you:
<?php
$fields = $data = array();
$member_fields = $this->EE->db->query("SELECT m_field_id, m_field_name FROM exp_member_fields");
foreach($member_fields->result_array() as $row)
{
$fields['m_field_id_'.$row['m_field_id']] = $row['m_field_name'];
}
$member_data = $this->EE->db->query("SELECT * FROM exp_member_data WHERE member_id = 1")->row();
foreach($member_data as $k => $v)
{
if($k != 'member_id')
{
$data[$fields[$k]] = $v;
}
}
print_r($data);
?>
Then just use your new $data array.