I can observe that fetching all records from a small table (100 records) can take 1600 miliseconds, even using a ":memory:" SQLite database.
This happens only when the model inherits from AuditBase; otherwise the performance is fast (around 8 miliseconds).
My test code is:
public class Record : AuditBase
{
[PrimaryKey]
[AutoIncrement]
public int Id { get; set; }
public string Name { get; set; }
}
Table is created and pre-populated with 100 records:
using var db = appHost.Resolve<IDbConnectionFactory>().Open();
if (db.CreateTableIfNotExists<Record>())
{
for (int i = 0; i < 100; i++)
{
db.Insert(new Record()
{
Name = $"Name {i}" ,
CreatedBy = "TEST",
ModifiedBy = "TEST",
CreatedDate = DateTime.Now,
ModifiedDate = DateTime.Now
});
}
}
Selecting the data from the service:
public class MyServices : Service
{
public IAutoQueryDb AutoQuery { get; set; } = null!;
public object Any(Hello request)
{
var sw = new Stopwatch();
sw.Start();
var records = Db.Select<Record>();
var loadTime = sw.ElapsedMilliseconds;
Console.WriteLine($"Took {loadTime}ms to load {records.Count} records");
return new HelloResponse { Result = $"Hello, {request.Name}!" };
}
}
The console output will read: Took 1584ms to load 100 records.
Why does it take so long?
I'm not able to repro this perf issue, I've created 2 programs, 1 which inherits AuditBase as done in this example at:
https://gist.cafe/fe8d04617ee1b061f24b0d8d56948119
Which shows it completes in 23ms.
Whilst another example which replaces DateTime properties with long's to store Unix Timestamps to avoid DateTime conversions during Select at:
https://gist.cafe/a7c55a56842899a1ce34f11456027a9b
Which completes in roughly the same time at 21ms.
Related
I need some help to map an anonymous object using Automapper. The goal is combine Product and Unity in a ProductDto (in which unity is a product's property).
Autommaper CreateMissingTypeMaps configuration is set to true.
My Classes:
public class Product
{
public int Id { get; set; }
}
public class Unity
{
public int Id { get; set; }
}
public class ProductDto
{
public int Id { get; set; }
public UnityDto Unity{ get; set; }
}
public class UnityDto
{
public int Id { get; set; }
}
Test Code
Product p = new Product() { Id = 1 };
Unity u = new Unity() { Id = 999 };
var a = new { Product = p, Unity = u };
var t1 = Mapper.Map<ProductDto>(a.Product);
var t2 = Mapper.Map<UnityDto>(a.Unity);
var t3 = Mapper.Map<ProductDto>(a);
Console.WriteLine(string.Format("ProductId: {0}", t1.Id)); // Print 1
Console.WriteLine(string.Format("UnityId: {0}", t2.Id)); // Print 999
Console.WriteLine(string.Format("Anonymous ProductId: {0}", t3.Id)); // Print 0 <<< ERROR: It should be 1 >>>
Console.WriteLine(string.Format("Anonymous UnityId: {0}", t3.Unity.Id)); // Print 999
There are two maps added to the profile:
CreateMap<Product, ProductDto>();
CreateMap<Unity, UnityDto>();
The problem is how Automapper map anonymous objects. I haven't time to check out Automapper source code but I got the desired behaviour with minor changes on my anonymous object:
var a = new { Id = p.Id, Unity = u };
By doing this, I might even delete previous mappings because now it is using only CreateMissingTypeMaps.
Note: As matter of fact I'm not sure if it is really an issue or I it was just my unreal expectations.
Okay so I am using Entity Framework 6.1 and attempting code first with seeding. I have a drop always intializer that ALWAYS WORKS. However I want to use database migration which I have set up and it works. UNTIL I normalize out a table and then try to seed it I get a primary key error.
Basically in my context when I uncomment out the changes in the 'TODO' section and the schema changes I get a primary key violation when attempting population of the newly normalized out table. It will work for the initializer that does the drop always, but I want my migration data table and not to drop the database everytime I make changes in case I want to rollback ever. I have tried changing the attribute of the 'PersonId' to Identity and to None and back to Identity. So the caveat is if it is set to 'Identity' it will work but the values will keep incrementing to higher values each time 1,2,3,4 then 5,6,7,8;etc. If I set it to none it works the first time and then when it is split in the mapping and normalized it blows up. I have tried custom dbcc commands and it does not like that either, as even with setting dbcc to reseed with the two new tables it does not like it. It is as if it has no idea about seeding the new table when being done explicitly.
Does anyone know how to do a seeding process that the model can handle if you normalize out the mapping of an object to multiple tables? I am trying a bunch of different patterns and getting nowhere fast.
So POCO Object
public class Person
{
[DatabaseGenerated(DatabaseGeneratedOption.None)]
public int PersonId { get; set; }
[Column(TypeName = "varchar")]
[Required]
[MaxLength(32)]
public string FirstName { get; set; }
[Column(TypeName = "varchar")]
[Required]
[MaxLength(32)]
public string LastName { get; set; }
[Column(TypeName = "varchar")]
public string OverlyLongDescriptionField { get; set; }
}
Context for code First:
public class EasyContext : DbContext
{
public EasyContext() : base("name=EasyEntity")
{
//Database.SetInitializer<EasyContext>(new EasyInitializer());
Database.SetInitializer(new MigrateDatabaseToLatestVersion<EasyContext, Migrations.Configuration>("EasyEntity"));
}
public DbSet<ProductOrder> ProductOrder { get; set; }
public DbSet<Person> Person { get; set; }
public DbSet<Product> Product { get; set; }
public DbSet<Audit> Backup { get; set; }
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
modelBuilder.HasDefaultSchema("dbo");
modelBuilder.Conventions.Remove<PluralizingTableNameConvention>();
modelBuilder.Conventions.Remove<ManyToManyCascadeDeleteConvention>();
//TODO Let's normalize out a long descriptive field
//modelBuilder.Entity<Person>()
//.Map(m =>
//{
// m.Properties(p => new { p.FirstName, p.LastName });
// m.ToTable("Person");
//})
//.Map(m =>
//{
// m.Properties(p => new { p.OverlyLongDescriptionField });
// m.ToTable("PersonDescription");
//});
}
}
Initializer for DropCreateAlways:
public class EasyInitializer : DropCreateDatabaseAlways<EasyContext>
{
protected override void Seed(EasyContext context)
{
SeedingValues.SeedingForDatabaseDrop(context);
base.Seed(context);
}
}
Configuration for migrations:
internal sealed class Configuration : DbMigrationsConfiguration<EasyEntity.EasyContext>
{
public Configuration()
{
AutomaticMigrationsEnabled = true;
AutomaticMigrationDataLossAllowed = true;
ContextKey = "EasyEntity.EasyContext";
}
protected override void Seed(EasyContext context)
{
SeedingValues.SeedingWithoutDatabaseDrop(context);
base.Seed(context);
}
}
Base Seeding class:
internal static class SeedingValues
{
public static void SeedingForDatabaseDrop(EasyContext context)
{
BaseSeed(context);
}
public static void SeedingWithoutDatabaseDrop(EasyContext context)
{
context.Person.ClearRange();
BaseSeed(context);
}
private static void BaseSeed(EasyContext context)
{
IList<Person> persons = new List<Person>
{
new Person { PersonId = 1, FirstName = "Brett", LastName = "Guy", OverlyLongDescriptionField = "OMG Look I have a bunch of text denormalizing a table by putting a bunch of stuff only side related to the primary table." },
new Person { PersonId = 2, FirstName = "Neil", LastName = "Person"},
new Person { PersonId = 3, FirstName = "Ryan", LastName = "Other"},
new Person { PersonId = 4, FirstName = "Aaron", LastName = "Dude"},
};
foreach (var person in persons)
context.Person.AddOrUpdate(person);
}
}
ClearingHelper
public static void ClearRange<T>(this DbSet<T> dbSet) where T : class
{
using (var context = new EasyContext())
{
dbSet.RemoveRange(dbSet);
}
}
Okay so the issue is with a newly created table being not populated and the old table being populated. So if I have following off of my example a POCO class 'Person' and have plurilization removed. Without any explicit mapping and just a DbSet will create a table Person. If I then do my mapping to split tables.
modelBuilder.Entity<Person>()
.Map(m =>
{
m.Properties(p => new { p.PersonId, p.FirstName, p.LastName });
m.ToTable("Person");
})
.Map(m =>
{
m.Properties(p => new { p.PersonId, p.OverlyLongDescriptionField });
m.ToTable("PersonDescription");
});
I get a Primary Key violation with my seeding process. This is due to if I look at the database newly updated it still is retaining the old table and created a new one. However it does not know how to remove the data with this method:
public static void ClearRange<T>(this DbSet<T> dbSet) where T : class
{
using (var context = new EasyContext())
{
dbSet.RemoveRange(dbSet);
}
}
Becase the set is partial. So I am thinking: "Well if my data for seeding is contained at this point and I need to alter it moving forward I can in theory just clean up the tables directly with SQL commands." This is not the approach I wanted per say but it does work.
So I add more data to my clearing helper:
public static void ResetIdentity(string tableName)
{
using (var context = new EasyContext())
{
context.Database.ExecuteSqlCommand($"DBCC CHECKIDENT('{tableName}', RESEED, 0)");
}
}
public static void DeleteTable(string tableName)
{
using (var context = new EasyContext())
{
context.Database.ExecuteSqlCommand($"DELETE {tableName}");
}
}
public static void DeleteTableAndResetIdentity(string tableName)
{
using (var context = new EasyContext())
{
context.Database.ExecuteSqlCommand($"DELETE {tableName}");
context.Database.ExecuteSqlCommand($"DBCC CHECKIDENT('{tableName}', RESEED, 0)");
}
}
Then I add this to my Seeding routine's cleanup portion:
ClearingHelper.DeleteTable("dbo.PersonDescription");
ClearingHelper.DeleteTableAndResetIdentity("dbo.Person");
This is unfortunate in two ways to do it this way in that:
It is slower doing a delete as it goes row by row.
If I migrate backwards I will have to change this.
But it works! I can now have changes to the schema with normalizing out POCO's and still run a seeding routine.
I have created the following example to test Foreign Keys and up to this point, it works well. What I would like to be able to do, is use this framework that I built to set the property of the relationship and have it Save the child object when the Parent is saved and automatically set the PrimaryKey and Foreign Key.
The DataManager class exposes the Connection
public class DataManager
{
DataManager()
{
OrmLiteConfig.DialectProvider = SqliteDialect.Provider;
ConnectionString = SqliteFileDb;
updateTables();
}
private void updateTables()
{
using (var dbConn = OpenDbConnection())
{
dbConn.DropAndCreateTable<Person>();
dbConn.DropAndCreateTable<PhoneNumber>();
}
}
public static string SqliteFileDb = "~/App_Data/db.sqlite".MapAbsolutePath();
private static DataManager manager;
public static DataManager Manager {
get
{
if (manager == null)
manager = new DataManager();
return manager;
}
}
public IDbConnection InMemoryDbConnection { get; set; }
public IDbConnection OpenDbConnection(string connString = null)
{
connString = ConnectionString;
return connString.OpenDbConnection();
}
protected virtual string ConnectionString { get; set; }
protected virtual string GetFileConnectionString()
{
var connectionString = SqliteFileDb;
return connectionString;
}
}
These are my POCO's with the BaseClass used to Achieve my results:
public class Person : LiteBase
{
[AutoIncrement]
[PrimaryKey]
public int Id { get; set; }
public string Name { get; set; }
public string Surname { get; set; }
private List<PhoneNumber> numbers;
public List<PhoneNumber> PhoneNumbers {
get
{
if (numbers == null)
numbers = GetList<PhoneNumber>(p => p.Person == Id);
return numbers;
}
}
}
public class PhoneNumber
{
public string Number { get; set; }
public string Description { get; set; }
[AutoIncrement]
[PrimaryKey]
public int Id { get; set; }
[References(typeof (Person))]
public int Person { get; set; }
public void AddPerson(Person person)
{
Person = person.Id;
}
}
public class LiteBase:INotifyPropertyChanged
{
public List<T> GetList<T>(Expression< Func<T,bool>> thefunction) where T : new()
{
var objects = new List<T>();
using (var conn = Data.DataManager.Manager.OpenDbConnection())
{
objects = conn.Where<T>(thefunction);
}
return objects;
}
public T GetItem<T>(Expression<Func<T, bool>> thefunction) where T : new()
{
T obj = new T();
using (var conn = Data.DataManager.Manager.OpenDbConnection())
{
obj = conn.Where<T>(thefunction).FirstOrDefault<T>();
}
return obj;
}
public event PropertyChangedEventHandler PropertyChanged;
[NotifyPropertyChangedInvocator]
protected virtual void OnPropertyChanged(string propertyName)
{
PropertyChangedEventHandler handler = PropertyChanged;
if (handler != null)
handler(this, new PropertyChangedEventArgs(propertyName));
}
}
Simple Class to Create Person and PhoneNumber objects
public class PersonManager
{
public void CreatePerson(string name, string surname, string number)
{
using (var conn = DataManager.Manager.OpenDbConnection())
{
var pnum = new PhoneNumber { Number = number };
var person = new Person
{
Name=name,
Surname = surname,
};
conn.Save<Person>(person);
var id = conn.GetLastInsertId();
person.Id = (int)id;
pnum.AddPerson(person);
conn.Save<PhoneNumber>(pnum);
}
}
public List<Person> GetPeople()
{
List<Person> people;
using (var conn = DataManager.Manager.OpenDbConnection())
{
people = conn.Select<Person>();
}
return people;
}
public List<PhoneNumber> GetNumbers()
{
List<PhoneNumber> numbers;
using (var conn = DataManager.Manager.OpenDbConnection())
{
numbers = conn.Select<PhoneNumber>();
}
return numbers;
}
}
And here is the usage:
var manager = new PersonManager();
manager.CreatePerson("John", "Doe", "12345679");
manager.CreatePerson("Jack", "Smith", "12345679");
manager.CreatePerson("Peter", "Jones", "12345679");
manager.CreatePerson("Dan", "Hardy", "12345679");
var people = manager.GetPeople();
var numbers = manager.GetNumbers();
for (int i = 0; i < people.Count; i++)
{
Console.WriteLine("{0} {1} {2}",
people[i].Name,people[i].Surname,people[i].Id);
}
for (int n = 0; n < numbers.Count; n++)
{
Console.WriteLine("PN: {0} {1}",
numbers[n].Number,numbers[n].Person);
}
for (int p = 0; p < people.Count; p++)
{
var person = people[p];
Console.WriteLine("{0}: {1} {2} {3}",
person.Id,person.Name,person.Surname,person.GetItem<PhoneNumber>(x=>x.Person==person.Id).Number);
}
The output is as I expected :
John Doe 1
Jack Smith 2
Peter Jones 3
Dan Hardy 4
PN: 12345679 1
PN: 12345679 2
PN: 12345679 3
PN: 12345679 4
1: John Doe 12345679
2: Jack Smith 12345679
3: Peter Jones 12345679
4: Dan Hardy 12345679
What I really would like to be able to do is the following:
var john = new Person
{
Name = "John",
Surname = "Smith",
PhoneNumber = new PhoneNumber { Number = "123456789" }
};
conn.Save<Person>(john);
var number = john.PhoneNumber.Number
Is this at all possible?
By default OrmLite v3 blobs all complex types properties in a string field and you need to explicitly set all references.
In the next major v4 release (ETA late Nov 2013), OrmLite adds some support for external references with the [Reference] attribute, which lets you tell OrmLite these properties should be stored in an external table and not blobbed, e.g:
public class Customer
{
[AutoIncrement]
public int Id { get; set; }
public string Name { get; set; }
[Reference]
public CustomerAddress PrimaryAddress { get; set; }
[Reference]
public List<Order> Orders { get; set; }
}
This will allow you to call db.SaveReferences() to save the reference properties, e.g:
var customer = new Customer
{
Name = "Customer 1",
PrimaryAddress = new CustomerAddress {
AddressLine1 = "1 Humpty Street",
City = "Humpty Doo",
State = "Northern Territory",
Country = "Australia"
},
Orders = new[] {
new Order { LineItem = "Line 1", Qty = 1, Cost = 1.99m },
new Order { LineItem = "Line 2", Qty = 2, Cost = 2.99m },
}.ToList(),
};
Assert.That(customer.Id, Is.EqualTo(0)); //Id is not saved yet
//Inserts customer, populates auto-incrementing customer.Id
//Specify `references:true` to populate the ForeignKey ids and
//save the related rows as well, e.g:
db.Save(customer, references:true);
Assert.That(customer.Id, Is.GreaterThan(0));
Assert.That(customer.PrimaryAddress.CustomerId, Is.EqualTo(customer.Id));
Assert.That(customer.Orders.All(x => x.CustomerId == customer.Id));
Saving References manually
For more fine-grained control you can also choose which references you want to save, e.g:
db.Save(customer); //Doesn't save related rows
//1:1 PrimaryAddress Reference not saved yet
Assert.That(customer.PrimaryAddress.CustomerId, Is.EqualTo(0));
//1:1 PrimaryAddress Reference saved and ForeignKey id populated
db.SaveReferences(customer, customer.PrimaryAddress);
//1:Many Orders References saved and ForeignKey ids populated
db.SaveReferences(customer, customer.Orders);
Loading all related rows with the entity
You can then load the master row and all its references with db.LoadSingleById, e.g:
var dbCustomer = db.LoadSingleById<Customer>(customer.Id);
dbCustomer.PrintDump();
Assert.That(dbCustomer.PrimaryAddress, Is.Not.Null);
Assert.That(dbCustomer.Orders.Count, Is.EqualTo(2));
Am Using EntityFramework codefirst approach.my coding is
class Blog
{
[Key]
public int BlobId { get; set; }
public string Name { get; set; }
public virtual List<Post> Posts { get; set; }
}
class Post
{
[Key]
public int PostId { get; set; }
public string Title { get; set; }
public string Content { get; set; }
public int BlobId { get; set; }
public virtual Blog Blob { get; set; }
}
class BlogContext:DbContext
{
public BlogContext() : base("constr") { }
public DbSet<Blog> Blogs { get; set; }
public DbSet<Post> Posts { get; set; }
}
class Program
{
static void Main(string[] args)
{
using (var db = new BlogContext())
{
Console.WriteLine("Enter a name for a new blob:");
var name = Console.ReadLine();
var b = new Blog { Name = name };
db.Blogs.Add(b);
db.SaveChanges();
Till this step i created two tables(Blogs and Posts)in my SQlserver.The BlobId is primary key in Blogs table.and foreign key in Posts table.and Blogid in blog table is auto incremented.postid in posts table is also auto incremented
class Program
{
static void Main(string[] args)
{
using (var db = new BlogContext())
{
Console.WriteLine("Enter a name for a new blob:");
var name = Console.ReadLine();
var b = new Blog { Name = name };
db.Blogs.Add(b);
db.SaveChanges();
Here i added name in the blogtable
var id1 = from val in db.Blogs
where val.Name == name
select val.BlobId;
Now by using Name am obtaining the blogid of blogs table
Console.WriteLine("Enter Title:");
var title = Console.ReadLine();
Console.WriteLine("Enter Content");
var content = Console.ReadLine();
var c = new Post { Title = title, Content = content, BlobId = id1};
db.Posts.Add(c);
db.SaveChanges();
here am reading the data for title,content.Then adding the title,content and blogid(which i obtained from another table) into Posts table
I getting error at BlobId = id1
Am getting Cannot implicitly convert type 'System.Linq.IQueryable' to 'int' this error
}
Console.ReadLine();
}
}
Can you help me to solve this.If you did not understand what i explained please reply me
The following query is a sequence of elements, not a scalar value, even though you believe that there is only one result, it is still a collection with one element when the results of the query are iterated over:
var id1 = from val in db.Blogs
where val.Name == name
select val.BlobId;
Change this to:
int id1 = (from val in db.Blogs
where val.Name == name
select val.BlobId).First();
This query will execute immediately and return the first element in the sequence. It will throw an exception if there is no match, so you may want to use FirstOrDefault and assign to a nullable int instead.
Update: Here's a gist that more fully demonstrates the issue https://gist.github.com/pauldambra/5051550
Ah, more update... If I make the Id property on the Mailing class a string then it all works. Should I just give up on integer ids?
I have 2 models
public class Mailing
{
public int Id { get; set; }
public string Sender { get; set; }
public string Subject { get; set; }
public DateTime Created { get; set; }
}
public class Recipient
{
public Recipient()
{
Status = RecipientStatus.Pending;
}
public RecipientStatus Status { get; set; }
public int MailingId { get; set; }
}
On my home page I want to grab the last 10 mailings. With a count of their recipients (eventually with a count of different status recipients but...)
I have made the following index
public class MailingWithRecipientCount : AbstractMultiMapIndexCreationTask<MailingWithRecipientCount.Result>
{
public class Result
{
public int MailingId { get; set; }
public string MailingSubject { get; set; }
public string MailingSender { get; set; }
public int RecipientCount { get; set; }
}
public MailingWithRecipientCount()
{
AddMap<Mailing>(mailings => from mailing in mailings
select new
{
MailingId = mailing.Id,
MailingSender = mailing.Sender,
MailingSubject = mailing.Subject,
RecipientCount = 0
});
AddMap<Recipient>(recipients => from recipient in recipients
select new
{
recipient.MailingId,
MailingSender = (string) null,
MailingSubject = (string)null,
RecipientCount = 1
});
Reduce = results => from result in results
group result by result.MailingId
into g
select new
{
MailingId = g.Key,
MailingSender = g.Select(m => m.MailingSender)
.FirstOrDefault(m => m != null),
MailingSubject = g.Select(m => m.MailingSubject)
.FirstOrDefault(m => m != null),
RecipientCount = g.Sum(r => r.RecipientCount)
};
}
}
I query using
public ActionResult Index()
{
return View(RavenSession
.Query<RavenIndexes.MailingWithRecipientCount.Result, RavenIndexes.MailingWithRecipientCount>()
.OrderByDescending(m => m.MailingId)
.Take(10)
.ToList());
}
And I get:
System.FormatException: System.FormatException : Input string was not
in a correct format. at System.Number.StringToNumber(String str,
NumberStyles options, NumberBuffer& number, NumberFormatInfo info,
Boolean parseDecimal)
Any help appreciated
Yes, integer ids are a pain. This is mainly because Raven always stores a full string document key, and you have to think about when you are using the key or your own id and translate appropriately. When reducing, you also need to align the int and string data types.
The minimum to get your test to pass is:
// in the "mailings" map
MailingId = mailing.Id.ToString().Split('/')[1],
// in the reduce
MailingId = g.Key.ToString(),
However - you could make your index a whole lot smaller and perform better by taking the sender and subject strings out of it. You can just put them in with a transform.
Here is a simplified complete index that does the same thing.
public class MailingWithRecipientCount : AbstractIndexCreationTask<Recipient, MailingWithRecipientCount.Result>
{
public class Result
{
public int MailingId { get; set; }
public string MailingSubject { get; set; }
public string MailingSender { get; set; }
public int RecipientCount { get; set; }
}
public MailingWithRecipientCount()
{
Map = recipients => from recipient in recipients
select new
{
recipient.MailingId,
RecipientCount = 1
};
Reduce = results => from result in results
group result by result.MailingId
into g
select new
{
MailingId = g.Key,
RecipientCount = g.Sum(r => r.RecipientCount)
};
TransformResults = (database, results) =>
from result in results
let mailing = database.Load<Mailing>("mailings/" + result.MailingId)
select new
{
result.MailingId,
MailingSubject = mailing.Subject,
MailingSender = mailing.Sender,
result.RecipientCount
};
}
}
As an aside, did you know about the RavenDB.Tests.Helpers package? It provides a simple base class RavenTestBase that you can inherit from that does most all of the legwork for you.
using (var store = NewDocumentStore())
{
// now you have an initialized, in-memory, embedded document store.
}
Also - you probably shouldn't scan the assembly for indexes in a unit test. You might introduce indexes that weren't part of what you were testing. The better route is to create the index indvidually, like this:
documentStore.ExecuteIndex(new MailingWithRecipientCount());