How do we usually deal with versioning of an aggregate root?
I was thinking along this line (I'm in a survey-design domain).
One way to have versioning is to have an explicit method to create a new version, based on the existing one. For example, Study (an aggregate root).
So initially we have an aggregate root, whose root-entity is Study with (business) key "ABC", version "1".
By invoking the method "newVersion()" on the Study, a copy of that Study and all the other entities that belong to the same aggregate root will be created.
So basically, versioning is done through creation a separate instance (of aggregate root). The ID is composite (business key + version).
How do we know if it's a branch? or is it just one version up? (1.1? or 2). I guess, this simple rule would work: if there's no further version associated, then it's "one version up" (2); if there's already another version, than it's a branch (1.1).
Another concern: noise.
But that means, we cannot work on / modify existing version. We'd have to create a newVersion everytime we want to make modifications to our object. Everytime??? Hmmm.... Doesn't sound right.
Or... we can make rule like this, based on a flag (active / not-active, or published / un-published). If the flag is "not-active", we can modify the AR directly, without creating a new version. If the flag is active we have to either: (a) set it to "not-active" first, and modify.... or (b) create a newVersion and work on the version (initially set to "not-active").
Any thoughts / experience you want to share on this matter?
I think you will find things a bit confusing in researching this question, because there are two very different concepts at play:
Versioning as a concurrency control mechanism to support optimistic concurrency
Versioning as an explicit domain concept
Versioning to support Optimistic Concurrency
Optimistic concurrency is when two simultaneous transactions are allowed to start, but if they both try and modify the same data item, only the first one is permitted to proceed. See Concurrency Control for an overview of different locking strategies.
In summary, you leave versioning up to the persistence technology, because the purpose of the version is to detect simultaneous writes to the persistence layer.
When using this pattern, it's common to not even keep copies of old versions, however it's certainly possible to do so as an audit trail/change log.
Versioning as an explicit domain concept
Based on your question, and the need to support potential branching strategies, it sounds like versioning is an explicit domain concept in your domain - i.e. the concept of a "Version" is something that your domain experts talk about, and working with versions is an important part of the ubiquitous language.
However, you raise a few different concepts which indicate that the domain needs further exploration:
Version branching
User-defined version naming/tagging (but still connected to a 'chain' of versions)
Explicit version changes (user requested) vs implicit version changes (automatic on every change)
If I understand your intent correctly, with explicit versioning, the current 'active'/'live'/'tip' version is mutable and can be modified without tracking the change, until the user 'commits' it - it becomes immutable, and a new 'live' version that is mutable is created.
Some other concepts that may come up if you explore this version:
Branch merging (once you have split two branches, what happens if you want to bring them back together?)
Rolling back - if you have an old version, do you support 'undoing' one or more changes?
Given the above, you may also find some insights from the way that version control systems work both centralised (e.g. subversion) and distributed (e.g. git and mercurial), as they present an active working model of version tracking with a mixture of mutable and immutable elements.
The open questions here suggest to me that you need to explore this in more detail with your domain experts. With DDD sometimes it's easy to get lost in what you can do, but I strongly encourage you to try and understand what you need to do.
How do your users/domain experts think about the world? What kind of operations do they want to be able to do? What is the purpose of these operations towards their initial goal? Your aim is to distill the answers to these questions into a model that effectively encapsulates the processes they work with.
Edit to Consider Modelling
Based on your comment - my first response would be to challenge the interpretation of the word 'version' when thinking about the modified questionnaire. In fact, I'd be tempted to challenge the modelling of the template/survey relationship. Consider a possible set of entities:
Template
Defines the set of questions in the questionnaire
Supports operations:
StartSurvey
Various operations to modify the questions and options in the template etc.
Survey
Rather than referencing a 'live' template, the survey would own it's own questionnaire
When you call Template.StartSurvey it returns a Survey that is prefilled with the list of questions from the template
A survey also supports modifying the questions - but this doesn't change the template it was created from
Unlike a template, a survey also maintains a list of recorded answers, and offers operations to set the answers
It probably also includes a lifecycle state wherein in some states answering questions is permitted, but once 'submitted' you can't modify the answers (just guessing on this one).
In this world, the survey is 'stamped out' from the template, but then lives an independent life. You can modify the questionnaire in the survey all you like, and it won't effect the template.
The trade-off here is that if you do modify the template, none of the surveys that have already been created from it would get updated - but it sounds like that might be safer for you anyway?
You could also support operations to convert a survey back into a template so that if you like the look of a modified survey, you could 'templatize' it so it could be used for future surveys.
Related
Recently I faced with the following invariants in my domain Model:
An Offer treated as Expired if ExpiryAt (DateTimeOffset) < DateTimeOffset.Now.
A Director of the Company cannot be younger than 18 years old
When Document is downloaded we should set DownloadedAt field with DateTimeOffset.Now
In Application Layer to keep purity and for better testing we usually isolate System.DateTime with IDateTime interface which allow to mock Now in UnitTests.
But all these 3 scenarios belong to Domain Layer and not to Application Layer. We should not Inject external interfaces into DomainModel to keep it pure. But from other side it might be bad to use DateTime.Now or DateTimeOffset.Now directly in DomainLayer since this adds a dependency to system clock and make it harder to test sometimes since DateTime.Now will never return the same result.
So the question is - how do you deal with this dilemma?
Options I see:
Provide now as parameter to Domain Entity methods. This is viable option and simplify testing though makes code more verbose and sometimes even stupid.
Just use DateTime.Now in domain layer. I already mentioned cons of this approach.
Anything else you might suggest from your experience?
From the different options accessing the static DateTime.Now() functionality is obviously the most disadvantageous. It both does not allow for testing and also hides the domain models dependency to some non-deterministic infrastructure inside the implementation details.
The option to inject some interface to a service that can be viewed is a little better because it makes the dependency explicit and also allows for unit testing by stubbing the non-deterministic output to return some deterministic value of your choice.
But still, at runtime your domain model needs to access some infrastructure dependency. This might be a reasonable compromise in some cases, but if possible I would try to avoid that to keep the domain model pure.
If you look at the current date time in your case from a different angle it becomes more obvious that it is actually nothing else like a normal input parameter. You could see it as something like a reference date time instead of the current date time.
Referring to your first example - checking if an offer has expired - from the domain model's point-of-view it needs to check if the offer has expired at some given point in time. This given point in time just happens to be current date time in one of the use cases where the domain logic is exercised.
So bottom line, I recommend to inject the value of the (current) date time rather than an interface to some functionality in such cases. It makes explicit what data is really need in addition to the data the domain encapsulates on its own and requires for performing the business logic.
Also, it makes more explicit what the client code (e.g. the use case or application service) wants to tell or ask the domain model. For instance, check if the offer has expired as of now or if needed, tell me if the offer was already expired at a given point in time or even if it will be expired at an important point in time.
As further reading I recommend this great article from Vladimir Khorikov where he elaborates more on that topic.
I want to sanity check myself on a view projection, in regards to if an intermediary concept can purely exist in the read model while providing a bridge between commands.
Let me use a contrived example to explain.
We place an order which raises an OrderPlaced event. The workflow then involves generating a picking slip, which is used to prepare a shipment.
A picking slip can be generated from an order (or group of orders) without any additional information being supplied from any external source or user. Is it acceptable then that the picking slip can be represented purely as a read model?
So:
PlaceOrderCommand -> OrderPlacedEvent
OrderPlacedEvent -> PickingSlipView
The warehouse manager can then view a picking slip, select the lines they would like to ship, and then perform a PrepareShipment command. A ShipmentPrepared event will then update the original order, and remove the relevant lines from the PickingSlipView.
I know it's a toy example, but I have a conceptually similar use case where a colleague believes the PickingSlip should be a domain entity/aggregate in its own right, as it's conceptually different to order. So you have PlaceOrder, GeneratePickingSlip, and PrepareShipment commands.
The GeneratePickingSlip command however simply takes an order number (identifier), transforms the order data into a picking slip entity, and persists the entity. You can't modify or remove a picking slip or perform any action on it, apart from using it to prepare a shipment.
This feels like introducing unnecessary overhead on the write model, for what is ultimately just a transformation of existing information to enable another command.
So (and without delving deeply into the problem space of warehouses and shipping)...
Is what I'm proposing a legitimate use case for a read model?
Acting as an intermediary between two commands, via transformation of some data into a different view. Or, as my colleague proposes, should every concept be represented in the write model in all cases?
I feel my approach is simpler, and avoiding unneeded complexity, but I'm new to CQRS and so perhaps missing something.
Edit - Alternative Example
Providing another example to explore:
We have a book of record for categories, where each record is information about products and their location. The book of record is populated by an external system, and contains SKU numbers, mapped to available locations:
Book of Record (Electronics)
SKU# Location1 Location2 Location3 ... Location 10
XXXX Introduce Remove Introduce ... N/A
YYYY N/A Introduce Introduce ... Remove
Each book of record is an entity, and each line is a value object.
The book of record is used to generate different Tasks (which are grouped in a TaskPlan to be assigned to a person). The plan may only cover a subset of locations.
There are different types of Tasks: One TaskPlan is for the individual who is on a location to add or remove stock from shelves. Call this an AllocateStock task. Another type of Task exists for a regional supervisor managing multiple locations, to check that shelving is properly following store guidelines, say CheckDisplay task. For allocating stock, we are interested in both introduced and removed SKUs. For checking the displays, we're only interested in newly Introduced SKUs, etc.
We are exploring two options:
Option 1
The person creating the tasks has a View (read model) that allows them to select Book of Records. Say they select Electronics and Fashion. They then select one or more locations. They could then submit a command like:
GenerateCheckDisplayTasks(TaskPlanId, List<BookOfRecordId>, List<Locations>)
The commands would then orchestrate going through the records, filtering out locations we don't need, processing only the 'Introduced' items, and creating the corresponding CheckDisplayTasks for each SKU in the TaskPlan.
Option 2
The other option is to shift the filtering to the read model before generating the tasks.
When a book of record is added a view model for each type of task is maintained. The data might be transposed, and would only include relevant info. ie. the CheckDisplayScopeView might project the book of record to:
Category SKU Location
Electronics (BookOfRecordId) XXXX Location1
Electronics (BookOfRecordId) XXXX Location3
Electronics (BookOfRecordId) YYYY Location2
Electronics (BookOfRecordId) YYYY Location3
Fashion (BookOfRecordId) ... ... etc
When generating tasks, the view enables the user to select the category and locations they want to generate the tasks for. Perhaps they select the Electronics category and Location 1 and 3.
The command is now:
GenerateCheckDisplayTasks(TaskPlanId, List<BookOfRecordId, SKU, Location>)
Where the command now no longer is responsible for the logic needed to filter out the locations, the Removed and N/A items, etc.
So the command for the first option just submits the ID of the entity that is being converted to tasks, along with the filter options, and does all the work internally, likely utilizing domain services.
The second option offloads the filtering aspect to the view model, and now the command submits values that will generate the tasks.
Note: In terms of the guidance that Aggregates shouldn't appear out of thin air, the Task Plan aggregate will create the Tasks.
I'm trying to determine if option 2 is pushing too much responsibility onto the read model, or whether this filtering behavior is more applicable there.
Sorry, I attempted to use the PickingSlip example as I thought it would be a more recognizable problem space, but realize now that there are connotations that go along with the concept that may have muddied the waters.
The answer to your question, in my opinion, very much depends on how you design your domain, not how you implement CQRS. The way you present it, it seems that all these operations and aggregates are in the same Bounded Context but at first glance, I would think that there are 3 (naming is difficult!):
Order Management or Sales, where orders are placed
Warehouse Operations, where goods are packaged to be shipped
Shipments, where packages are put in trucks and leave
When an Order is Placed in Order Management, Warehouse reacts and starts the Packaging workflow. At this point, Warehouse should have all the data required to perform its logic, without needing the Order anymore.
The warehouse manager can then view a picking slip, select the lines they would like to ship, and then perform a PrepareShipment command.
To me, this clearly indicates the need for an aggregate that will ensure the invariants are respected. You cannot select items not present in the picking slip, you cannot select more items than the quantities specified, you cannot select items that have already been packaged in a previous package and so on.
A ShipmentPrepared event will then update the original order, and remove the relevant lines from the PickingSlipView.
I don't understand why you would modify the original order. Also, removing lines from a view is not a safe operation per se. You want to guarantee that concurrency doesn't cause a single item to be placed in multiple packages, for example. You guarantee that using an aggregate that contains all the items, generates the packaging instructions, and marks the items of each package safely and transactionally.
Acting as an intermediary between two commands
Aggregates execute the commands, they are not in between.
Viewing it from another angle, an indication that you need that aggregate is that the PrepareShippingCommand needs to create an aggregate (Shipping), and according to Udi Dahan, you should not create aggregate roots (out of thin air). Instead, other aggregate roots create them. So, it seems fair to say that there needs to be some aggregate, which ensures that the policies to create shippings are applied.
As a final note, domain design is difficult and you need to know the domain very well, so it is very likely that my proposed solution is not correct, but I hope the considerations I made on each step are helpful to you to come up with the right solution.
UPDATE after question update
I read a couple of times the updated question and updated several times my answer, but ended up every time with answers very specific to your example again and I'm most likely missing a lot of details to actually be helpful (I'd be happy to discuss it on another channel though). Therefore, I want to go back to the first sentence of your question to add an important comment that I missed:
an intermediary concept can purely exist in the read model, while providing a bridge between commands.
In my opinion, read models are disposable. They are not a single source of truth. They are a representation of the data to easily fulfil the current query needs. When these query needs change, old read models are deleted and new ones are created based on the data from the write models.
So, only based on this, I would recommend to not prepare a read model to facilitate your commands operations.
I think that your solution is here:
When a book of record is added a view model for each type of task is maintained. The data might be transposed, and would only include relevant info.
If I understand it correctly, what you should do here is not create view model, but create an Aggregate (or multiple). Then this aggregate can receive the commands, apply the business rules and mutate the state. So, instead of having a domain service reading data from "clever" read models and putting it all together, you have an aggregate which encapsulates the data it needs and the business logic.
I hope it makes sense. It's a broad topic and we could talk about it for hours probably.
I am new to Event Sourcing and I am considering using it for an industrial application to track events happening in a production facility.
Since the book of record is the production facility itself and not the system, and also because not everything is automated, workers will need to report at a given point in time (the recorded time) what they did at another point in time (the effective time). Therefore, I will be using events such as: TankFilledRecorded, TankOutputConnectedToPipeInputRecorded, ContainerMovedToFacilityAreaRecorded, etc. where these events refer to entities such as a tank, a pipe, or a facility area for example. These events will have both a recorded time and an effective time. Note that there is no submission or approval process for a record to be considered legit.
Domain-driven design (DDD) encourages to design events that are representative of what happens in the domain (like the ones above).
However, in my domain, I don’t care so much about how a tank, a pipe or a facility area came to existence. I just need to know that something exists from a particular point in time, and I also need to know if it is not there after a particular point in time. The main objective of the software is to track liquids and powders flowing in a circuit made of these pipes, tanks and other components. It is not an asset management system and should not become one.
Therefore, what would be the correct DDD way to design an event that represents the fact that there is a tank, a pipe or an area in the production facility?
It is a subtle question but language is important, particularly in DDD.
Here is what I came up with:
1 EntityExistenceAcknowledgmentRecorded
TankExistenceAcknowledgmentRecorded
PipeExistenceAcknowledgmentRecorded
FacilityAreaExistenceAcknowledgmentRecorded
TankDisappearanceAcknowledgmentRecorded
PipeDisappearanceAcknowledgmentRecorded
FacilityAreaDisappearanceAcknowledgmentRecorded
It seems awful to use this in the ubiquitous language. I don’t see myself talking in these terms or providing a UI with such vocabulary. But it does represent exactly what happens though.
2 EntityRegistered
TankRegistered
PipeRegistered
FacilityAreaRegistered
TankUnregistered
PipeUnregistered
FacilityAreaUnregistered
It seems much simpler and it also seems to be meaningful except for one thing. “Registered” conveys the existence of the representation of an entity in the system with immediate effect, without the possibility of saying now that the entity existed 2 days ago. Think about a UserRegistered event in a website that would indicate that the user “existed” from 10 days ago. What would that even mean?
Events are facts and you cannot change the past. However, I do need a way for my users to invalidate a record in which they made a mistake such as a typo. They can record now that they acknowledged the existence of a facility area a week ago and might realize later than there was something wrong, such as a typo in the name of the entity. They would invalidate the record and create a new one. But, invalidate something that has been “registered” does not sound right.
3 Keep looking
Try to dig more in the domain (event storming) and find the real events that brought the entities into existence even if these events are of no use in the problem that needs to be solved.
TankBuiltRecorded
PipeBuiltRecorded, PipeDeliveredRecorded
FacilityArea<something_meaningful>Recorded
TankDestroyedRecorded, TankDecommissionedRecorded
PipeDecommissionedRecorded
FacilityArea<something_meaningful>Recorded
A caution
TankFilled
TankFilledReported
TankFilledReportSubmitted
TankFilledReportSubmissionReceived
Think carefully about whether the increased precision is motivated by business value.
Therefore, what would be the correct DDD way to design an event that represents the fact that there is a tank, a pipe or an area in the production facility?
What is the business doing today? Is there already a process in place for tracking the lifetime of the hardware in the plant (a maintenance log, perhaps?) There's likely to be vocabulary in that place that gives you ideas as to what spellings would make sense in the code.
Events are facts and you cannot change the past.
That's true - but you can back date events. The effective date of the information is often distinct from the reported date of information.
I do need a way for my users to invalidate a record in which they made a mistake such as a typo.
Yes - error correction is an important part of the process that you are modeling.
You should probably review Greg Young's talk Answering a Question, which was based on this thread. It's a discussion of capturing and modeling of temporality.
Here's the good news: you are running into the right problem. Because you are capturing information about an external system, there are going to be opportunities for errors and conflicts, and you need to (a) figure out the protocols for addressing them, and then (b) model that process correctly. That might include exception reports generated by the system when it observes conflicting information, or compensating events, or even automated conflict resolution (for the easy cases -- see also Stop Over Engineering).
I would like to implement CQRS and ES using Axon framework
I've got a pretty complex HTML form which represents recruitment process with six steps.
ES would be helpful to generate historical statistics for selected dates and track changes in form.
Admin can always perform several operations:
assign person responsible for each step
provide notes for each step
accept or reject candidate on every step
turn on/off SMS or email notifications
assign tags
Form update (difference only) is sent from UI application to backend.
Assuming I want to make changes only for servers side application, question is what should be a Command and what should be an Event, I consider three options:
Form patch is a Command which generates Form Update Event
Drawback of this solution is that each event handler needs to check if changes in form refers to this handler ex. if email about rejection should be sent
Form patch is a Command which generates several Events ex:. Interviewer Assigned, Notifications Turned Off, Rejected on technical interview
Drawback of this solution is that some events could be generated and other will not because of breaking constraints ex: Notifications Turned Off will succeed but Interviewer Assigned will fail due to assigning unauthorized user. Maybe I should check all constraints before commands generation ?
Form patch is converted to several Commands ex: Assign Interviewer, Turn Off Notifications and each command generates event ex: Interviewer Assigned, Notifications Turned Off
Drawback of this solution is that some commands can fail ex: Assign Interviewer can fail due to assigning unauthorized user. This will end up with inconsistent state because some events would be stored in repository, some will not. Maybe I should check all constraints before commands generation ?
The question I would call your attention to: are you creating an authority for the information you store, or are you just tracking information from the outside world?
Udi Dahan wrote Race Conditions Don't Exist; raising this interesting point
A microsecond difference in timing shouldn’t make a difference to core business behaviors.
If you have an unauthorized user in your system, is it really critical to the business that they be authorized before they are assigned responsibility for a particular step? Can the system really tell that the "fault" is that the responsibility was assigned to the wrong user, rather than that the user is wrongly not authorized?
Greg Young talks about exception reports in warehouse systems, noting that the responsibility of the model in that case is not to prevent data changes, but to report when a data change has produced an inconsistent state.
What's the cost to the business if you update the data anyway?
If the semantics of the message is that a Decision Has Been Made, or that Something In The Real World Has Changed, then your model shouldn't be trying to block that information from being recorded.
FormUpdated isn't a particularly satisfactory event, for the reason you mention; you have to do a bunch of extra work to cast it in domain specific terms. Given a choice, you'd prefer to do that once. It's reasonable to think in terms of translating events from domain agnostic forms to domain specific forms as you go along.
HttpRequestReceived ->
FormSubmitted ->
InterviewerAssigned
where the intermediate representations are short lived.
I can see one big drawback of the first option. One of the biggest advantage of CQRS/ES with Axon is scalability. We can add new features without worring about regression bugs. Adding new feature is the result of defining new commands, event and handlers for both of them. None of them should not iterfere with ones existing in our system.
FormUpdate as a command require adding extra logic in one of the handler. Adding new attribute to patch and in consequence to command will cause changes in current logic. Scalability is no longer advantage in that case.
VoiceOfUnreason is giving a very good explanation what you should think about when starting with such a system, so definitely take a look at his answer.
The only thing I'd like to add, is that I'd suggest you take the third option.
With the examples you gave, the more generic commands/events don't tell that much about what's happening in your domain. The more granular events far better explain what exactly has happened, as the event message its name already points it out.
Pulling Axon Framework in to the loop, I can also add a couple of pointers.
From a command message perspective, it's safe to just take a route and not over think it to much. The framework quite easily allows you to adjust the command structure later on. In Axon Framework trainings it is typically suggested to let a command message take the form of a specific action you're performing. So 'assigning a person to a step would typically be a AssignPersonToStepCommand, as that is the exact action you'd like the system to perform.
From events it's typically a bit nastier to decide later on that you want fine grained or generic events. This follows from doing Event Sourcing. Since the events are your source of truth, you'll thus be required to deal with all forms of events you've got in your system.
Due to this I'd argue that the weight of your decision should lie with how fine grained your events become. To loop back to your question: in the example you give, I'd say option 3 would fit best.
over time our information strategy has gone all over the place and we are looking to have a clearer policy and a more explicit way for everyone to be in sync on information sharing. Some things to note is that the org is 300+ people and is in multiple countries across the world. Also, we have people that are comfortable in Sharepoint, people that are comfortable in confluence, etc so there is definately a "change" factor here
Here are our current issues and what we are thinking about doing about them. I would love to hear feedback, suggestions, etc.
The content we have today:
Technical design info / architecture docs
Meeting minutes, action items, etc
Project plans and roadmaps
organization business mgmt info - travel, budget info, headcount info, etc
Project pages with business analysis, requirements, etc
Here are some of our main issues:
Where should data go - Confluence WIKI versus Sharepoint versus intranet site - we use confluence WIKI for #1, #2, #3, #5 but we also use sharepoint for #1, #3, #4, #5. We are trying to figure out if we should mandate each number to a specific place to make things consistent. We are using Sharepoint more a directory structure of documents, and we are using confluence for more adhoc changable content.
Stale Data - this is maybe a cultural thing with the org but at certain points in time data just becomes stale and is no longer relevant. What is the best way to ensure old data doesn't create a lot of noise and to ensure that the latest correct data is up to date. Should there be people in the org responsible for this or should it be an implicit "everyones job". This is more of an issue when people leave, join, etc . .
More active usage - whats is the best way to get people off of email and trying to stop and think "could this be useful for others . . let me put it in a centralized place instead of in email chains" . .
also, any other stories of good ways to improve an org's communication and information management
A fundamental root cause of information clutter is "no ownership".
People are assigned to projects. The projects end (or are cancelled), the people move on and the documents remain behind to gather "dust" and become information clutter.
This is hard to prevent. The wiki vs. sharepoint doesn't address the clutter, it just shifts the technology base that's used to accumulate clutter.
Let's look at the clutter
Technical design info / architecture docs. Old ones don't matter. There's current and there's irrelevant. Wiki.
Last year's obsolete design information is -- well -- obsolete.
Meeting minutes, action items, etc. Action items become part of someone's backlog in a development sprint, or, they're probably never going to get done. Backlogs are wiki items. Everything else is history that might be interesting but usually isn't. If it didn't create a sprint backlog items, update an architecture, or solve a development problem, the meeting was probably a waste of time.
Project plans and roadmaps. The sprint backlog matters -- this is what a "plan and roadmap" aspires to be. If you have to supplement your plans with roadmaps, you probably ought to give up on the planning and just use Scrum and just keep the backlog current.
The original plan is someone's guess at project inception time, and not really very interesting to the current project team.
Organization business mgmt info - travel, budget info, headcount info, etc. This is a weird mixture of highly structured stuff (budget, organization) and unstructured stuff ("travel"?)
How much history do you need? None? Wiki at best. Financial or HR System is where it belongs. But, in big organizations, the accounting systems can be difficult and cumbersome to use, so we create secondary sources of information like a SharePoint page with out-of-date budget numbers because the real budget numbers are buried inside Oracle Financials.
Project pages with business analysis, requirements, etc. This is your backlog. Your project roadmap and your requirements and your analysis ought to be a single document. In the wiki.
History rarely matters. Someone's concept at project inception time of what the requirements are doesn't matter very much any more. What the requirements evolved to in their final form matters far more than any history. This is wiki material.
How old is 'too old'?
I've worked with customers that have 30-year old software. The software -- obviously -- is relevant because it's in production.
The documentation, however, is all junk. The software has been maintained. It's full of change control records. The "original" specifications would have to be meticulously rewritten with each change control folded in. Since the change control documents can be remarkably pervasive, the only way to see where the changes were applied is to read the source and -- from that -- reverse engineer the current-state specification.
If we can only understand a 30-year old app by reverse engineering the source, then, chuck the 30-year old pile of paper. It's useless.
As soon as maintenance is done, the "original" specification has been devalued.
How to clean it up?
If you create the wiki page or sharepoint site, you own it forever.
When you leave, your replacement owns it forever.
Each manager is 100% responsible for every piece of information their staff creates. They have to delete things. The weak solution is to "archive" stuff. Which is just a polite way of saying "delete" without the "D-word".
Cleanup must be every manager's ongoing responsibility. If they can't remember what it is, or why they own it, they should be required (or "encouraged") to delete it. Everything unaccessed in the last two years should be archived without question. Everything 10 years old is just irrelevant history.
It's painful, and it doesn't appear to be value-creating work. After all, we work in IT. Our job is to "write" software, not delete it. No one will do it unless compelled on threat of firing.
The cost of storage is relatively low. The cost of cleanup appears higher.
How to stop the email chain?
Refuse to participate. Create a "Break the Chain" campaign focused on replacing email chains with wiki updates (or sharepoint updates).
Be sure your wiki provides links and is faster to edit than an email.
You can't force people to give up a really, really convenient solution (Email). You have to make the wiki more valuable and almost as convenient as email.
Ramp up the value on the wiki. Deprecate email chains. Refuse to respond to email chains. Refuse to accept "to do" action items through email.
You can use Confluence Wiki for storing documents as attachements and have the Wiki's paths work as the file paths in Sharepoint.
Re: stale data: have ownership of the data (both person and team) and ensure that deliverables for the owners include maintenance of ALL the data.
As far as "Off email", this is hard to do as you can't force people to do this short of actively monitoring all email... but you can try some deliverables with metrics regarding content added to the Wiki. That way people would be more likely to want to re-use the work already done on the email to paste into Wiki to meet the "quota" instead of composing fresh stuff.
Our company and/or team used all 3 of these approaches with some degree of success in the past
Is there a reason not to have the wiki hold the files?
Also, perhaps limiting the mail server to not allowing attachments on internal emails is too draconian, but asking folks to put everything in the wiki that needs to be emailed more than once is pretty darn useful.
Efficient information management is indeed a very hard problem. We found that "the simpler the better" principle can make miracles to solve it.
Where should data go - we are big believers of the wiki approach. In fact, we use Confluence for sharing possibly every type of information, except really large binary files. For those, we use Dropbox. Its simplicity is an absolutely killer feature. (Tip: you can integrate them with the Dropbox in Confluence plugin.)
Finding stale data - in our definition, stale data is something that is not updated or viewed for a specific period of time. The Archiving Plugin of Confluence can quickly and automatically find these, then report them to the authors and administrators, who may potentially update them (or remove them, see next item). There is, of course, information that never expires, but the plugin is able to skip them after you mark the corresponding pages.
Removing stale data - we are fairly aggressive on this. If the data is not (highly) relevant anymore, clean it up now! We can safely follow this practice, because we never actually delete data. We just move outdated data to hidden archive spaces using, again, the Archiving Plugin. If we changed our mind later, it is very easy to find it in the the archive, view it or even to recover it.
More active usage - our rule: if the information is required to be persistent, don't email it. Put it to a wiki page instead. The hard thing for some people is to find the best location for the information (which space? where in the page hierarchy?). Badly organized spaces with vague scope are another big efficiency divider, unfortunately. Large companies may consider introducing a wiki gardener to cure this.