Entity Framework V2.0

Microsoft has started the work on Entity Framework V2.00 now. Apparently, this version will have a great deal more transparency during the process, making sure it ends up as something the users want.

Good stuff.

NHibernate, beware?

Update:

A petition against the current quality of Entity Framework has begun. Read it to get a brief overview of the issues, and add your signature if you agree.

PropertyNamesGenerator (source)

I recently wrote the post Replace strings for property names with type-safe version

This is the source code for the project it describes. I'm posting it because of a request, so hopefully you'll have use for it.

PropertyNamesGenerator

• Generates type-safe versions of property/field names.


• Generates C# or VB files


• The only parser implemented is for parsing NHibernate mapping-files

The NHibernate parser handles regular classes as well as subclasses.
Another parser which will probably get there is for regular domain classes.

Warning: This is not production-grade software, but a simple project I made for myself to handle a current need! This should not be used as a guideline for good design, nor expected to work flawlessly. No such thing have been attempted.

You can download it here.

The zip includes the source, the current binary, and a sample bat-file to use it.

How to use it:
To run the generator, start it with something like:

• "C:\path\PropertyNamesGenerator.exe" /lang:cs /files:"../../source/Domain/*.hbm.xml" /out:"Output"

An explanation of the various params can be found by running it with a -? switch. It prints:
Run like: PropertyNamesGenerator.exe /files:"path/*.hbm.xml" /out:"path/PropertyNames/" /lang:cs
Parameters:
/Files [Mandatory] - Must have both a directory and a search string.
/Out [Mandatory] - Must specify an output directory.
/lang [Optional] - Output-language. C# or VB. (C# is default)
-V - Verbose output.
Note: All directories can be relative.

It currently only supports searching for files in the directories explicitly specified, i.e. not subdirectories. This is a concious choice.

To integrate it with your build, typically use a bat-file in the same manner as the Generate.bat example file included in the zip.

To integrate it with your build, you can add something like the line below to your Domain Pre-build event: (Project/Build Events)
call "$(SolutionDir)\..\Tools\PropertyNamesGenerator\Generate.bat"
Note that you will need a full path to the PropertyNamesGenerator.exe-file in the bat-file to use it like this.

It will generate files on the form [ClassName].[lang] in the output directory specified. Using a PropertyNames folder to put all of these in is recommended.

You will of course have to include these file in your solution to integrate them. Do this every time you have added and parsed a new hbm-file.

Persistance-solutions and characteristics

The other day I invited my colleagues to a discussion about persistance solutions. The goal was to cover the various options for persistance in a software project, the pros and cons of each approach, and to increase everyone’s knowledge about it.

The approach planned was to identify the possible solutions, identify important characteristics to help compare each solution, and to go through each possibility.

The different types we identified were:

• Code generation


• Object relational mapper (custom made)


• Object relational mapper (commercial/open source)


• Row data gateway (Which would be active record with domain logic)


• Table data gateway (Dataset)


• Stored procedures/views


• Manual SQL


• Object database

And the characteristics in no particular order:

• Development time


• Flexibility


• Performance


• Stability


• Complexity


• Refactoring


• Expertise


• Effect/limitation/intrusion on system


• Rapid Application Development (RAD)-support


• Attachment to data type


• Amount of developers


• Company demands/guidelines


• What you are creating

I think it is best to start off by defining the characteristics; what they are and why they are important.

• Development time
  
  
  • The time it takes to get your persistance solution ready to use and understood. This can either be the time it takes to configure and use an existing solution or the time it takes to develop your own.


• Flexibility
  
  
  • The flexibility of the system in terms of handling the various storage challenges in a project. This also includes handling a system which inevitably grows and changes.


• Performance
  
  
  • Obviously performance is important, right? This includes how the system performs in general, how it scales, how it can help handle special cases. A word of caution though: Even though performance is essential, don’t optimize prematurely. In any system of reasonable size, you will have special cases that will become bottlenecks. However, with a profiler and the possibility of fine-tuning those cases, you should be quite alright.


• Stability
  
  
  • You need to know that the system won’t break down under normal usage or high peaks. Testing is essential for this.


• Complexity
  
  
  • How complex is it to use and understand? How much help do you get with error messages? What do you when you need to debug? Is it too complex for the average developer on the project, making people take (eventually unmanageable) shortcuts?


• Refactoring
  
  
  • What sort of help do you get when the need for refactoring appears (often)? How hard is it to do changes?


• Expertise
  
  
  • You always need to take into account the expertise and skill level of the people on the project. This is more important that selecting “the best” solution.


• Effect/limitation/intrusion on system
  
  
  • What does it take to integrate the persistance solution with your overall system? Will your domain model be POCO? Do you need to inherit from a base class? Implement an interface?


• Rapid Application Development (RAD)-support
  
  
  • What kind of GUI-support does the solution have? If you need something quick and dirty, what do you do? The winner here should be pretty clear (If not, you’ll soon find out)


• Attachment to database data type
  
  
  • How close is it tied to database data types? Can it for instance handle the types of several databases?


• Amount of developers
  
  
  • How many developers the project has. This could influence the complexity of the solution, or the time available to get it up and running.


• Company demands/guidelines/policy
  
  
  • Company policies/limitations are a part of normal life for a consultant, and often there is little you can do about it. The company might have policies against using anything open source, demand that all database access should go through stored procedures, etc.
  
  
  • What can you do about this? Unless you have a small project or are able to influence the architecture group, chances are you’ll have to manage. I guess you can pray that the architecture group doesn’t consist of a bunch of non-coding architects that selects technologies out of marketing slides.
  
  
  • Another point I need to raise here: If your company still live in the “don’t use open source”-world, chances are that most decision-makers are a bit out of sync with what is happening in the .NET world today. Agreed, just a few years ago things were pretty thin in terms of open source projects on .NET, but today things have most definitely turned to the better.


• What you are creating
  
  
  • The most important point! Make sure whatever you choose will work for your project. Unneeded complexity is expensive! But remember that going from one solution to the other is often hard, so beware the danger of a project growing on a bad solution; you’ll have a tough time getting out of it

That should cover the basics of what is important to think about before choosing a persistance solution. Let’s get to the actual solutions!

Code generation

This involves generating the data access and potentially partly the domain model from your database schema. You specify rules for how you want things generated. There are a couple of potential problems with this approach.

• If the software has a binary solution specifying the mapping, you’ll have no way to map two differing changes, which in effect means you can only have exclusive checkout. In a multi-person project this is highly unproductive, and don’t forget that you’ll be unable to use branches, as you can’t merge them later.


• The other is that once you want to add a field or property to a domain object, you’ll have to update the database structure and generate the files before you can use it. This does take some time. I’m not saying that most other approaches are must faster; the annoying thing is that you can’t simply add properties etc. during testing for instance.

I have to admit that I have limited knowledge of this type of persistance-solution, but I believe this approach could be successful. There is no reason why the general mapping solution between the database and domain objects should be any poorer than with a general OR-Mapper. It depends what extra functionality is available and how everything is implemented (works). By this I mean the SQL generated, how the session is handled, lazy loading, how it affects the domain model and system, if it supports queries or SQL/stored procedures for special cases, type safe support for anything and everything, maintainable/understandable script files, and a few more I probably forget at the moment.

Object relational mapper (custom made)

An OR-Mapper is a piece of software that handles the problem of transforming your domain objects data into its equivalent form in the database. Your domain model is made of objects and pointers; the database model is made of rows, columns, keys and relationships. These are very different ways of handling data.

By creating your own OR-Mapper you will have to handle the problem of mapping from the domain model to the database for starter. This is just a small part of what you need to handle though, and to make it clear => There are slim chances that creating your own OR-Mapper will be beneficial for your project. Doing it will be time-consuming and error-prone, you will reinvent the wheel without need (See Object relational mapper (commercial/open source)), you will tie up your best resources for a substantial amount of time, you will have to implement a lot of added functionality to make it functional. Oh, and I doubt any project will wait before the mapper is finished before starting the rest of the development - which means some other form of temporary solution must be created.. (An object database might be the best choice in such a scenario.)

However, as a colleague commented, it is probably a dream for most developers to do it. Why? First, it’s complex, so you’ll learn heaps by doing it. Second, you don’t have to relate to the business side, so you really just define most of the tasks yourself. Perfect or what?

Object relational mapper (commercial/open source)

This is the same as above, except you use an already existing solution from a vendor or open source project. I recommend this approach.

Why create your own OR-Mapper when great solutions already exist? If you choose a premade mapper you will (potentially) get:

• Shorter development time – You will need time to configure, use and understand the solution. This is time-consuming as well, but far easier than creating your own. Getting started should also be fairly easy, even though the tougher concepts need more time.


• There’s a chance that present/new developers have used the solution before


• A solution tested in many projects, which means less bugs and more working features.


• New versions at no development cost.

You need to live with a few risks though:

• Harder to customize and debug. You’ll have to use time on strange error messages which can make little sense.


• The company might stop development and support, or the developer base of an open source project could die out. If you create your own OR-Mapper, the main developer(s) could quit as well.

In general, getting something of the shelf is far cheaper than building it yourself, as long as your requirements are met. I believe that this solution most of the time far surpasses any positive effects of building it yourself.

Picking a mapper at random just by browsing some marketing slides are not the way to go. Make sure you thoroughly read specs and user feedback, or the best thing if possible: talk to people that has experience with what you are considering, if it is a popular solution they shouldn’t be too hard to find.

There are a range of solutions in this field, but personally I’ve had the pleasure of working with perhaps the most known of them: NHibernate. NHibernate is an open source OR-mapper, which is a port of the well known OR-Mapper Hibernate from the Java world. NHibernate is a great piece of software, it has been used on numerous projects, and quite a lot of information is available through blogs and forum.

I’m not going to list all the reasons to use NHibernate, a quick search on the web should give you that, but a few things:

• NHibernate lets you have (almost) POCO objects (You need to mark all persistable fields/properties as virtual. NHibernate subclasses your objects with the virtual proxy pattern, to give you lazy loading etc.)


• NHibernate has mapping files to map between your domain object and database. These enable easy modeling of inheritance, collections, etc.


• It is mostly type safe (with NHibernate Query Generator at least), except when you need to write advanced queries. (You could use something like my PropertyNamesGenerator though)


• You can use Hibernate Query Language to create special queries in the cases where performance isn’t good enough.


• Automatic lazy loading

A few bad things as well:

• There is some overhead involved, and performance has been noted as an issue on several occasions. I think this is more of a design issue with your average developer though. If you try to retrieve gazillions of data from lots of tables, you can’t expect it to be lightning fast. Proper table and index design, as well as good use of lazy loading should get you well under way. For the special cases where you do get a performance issue – use a profiler to see what the issue is, use query analyser to look at the SQL, use HQL or use manual SQL behind a well designed layer to access the data, use DTO’s to limit the data loaded and sent…. There are plenty of possibilities


• NHibernate has some strange error messages. Before you get to know it enough, you’re bound to use a few hours trying to figure out error messages which doesn’t make much sense. I base this on experience with the 1.2 version, not the new 2.0 release.

If you don’t want to go down the open source way, you’ve probably been (or will be) introduced to Microsofts new OR-Mapper Entity Framework. It’s Microsoft, so it’s bound to be good, right? (…)
Note: I have only read about Entity Framework, and thus my experience (both good and bad) are of questionable quality, so make sure you do your own research before making any conclusions.
The pros:

• Integrates nicely with LINQ (But Ayende has a project going to bring this to NHibernate as well


• Microsoft helps bring the concept of an OR-Mapper to public knowledge, which really is good


• With Microsoft’s size, there’s a good chance that the project will continue. This depends upon how many uses it of course; it might get shut down like Microsoft’s music service did.


• Developers versed exclusively in the Microsoft world have something besides datasets and manual ADO.NET to use.

The cons:

• They’ve built a completely new product - which means you should think they should be able to leverage the experience of already existing OR-mappers. You’ll need a good amount of resources to do this though. For some reason it doesn’t like Microsoft has quite lived up to this.


• Having read quite a few blogs about Entity Framework and spoken to Microsoft employees about it, I must say I’m initially skeptical. At least for enterprise development. According to one Microsoft employee, Entity Framework is only believed to have a third of its user group in the enterprise software world; the rest is your simple application developers. The needs are quite different.


• One of the aspects that alarms me quite a bit relating to enterprise development and Entity Framework (or really any modern type of development – read: with source control), is that until recently the mapping files were unmergable. (The XML created were put in a “random” order. A small change could lead to big changes in a document) I can’t for the life of it understand that Entity Framework could be designed as anything but a play version if that wasn’t an important design point from the beginning. Ayende had a post about a meeting he had with Microsoft about this point, and based on his reactions it seems likely that the team defended this decision. It seems Microsoft has improved this feature after the range of reactions on it, as mentioned here


• Explicit lazy loading. You have to explicitly say that you want to load a lazy-loadable collection. I think this sounds mostly annoying, as you need to fill your code with logic of testing if a collection has been loaded, and then explicitly loading it, compared to the NHibernate way of automatic lazy loading by simply using it. There is a good thing about it - the fact that you won’t get unexpected database calls from the GUI-layer because you forgot to load everything you needed. This could lead to a performance hit and other problems. I don’t believe that merits this solution.

Even though I’m skeptical about the current quality of Entity Framework, and would recommend using NHibernate instead, I’m positive to Microsoft’s general move into this realm. With their funds, future versions have the potential of becoming really useful, with hopefully seamless integration with the rest of the framework.

For now I’m mostly scared that I’ll be put on a project where its use will be mandatory. Unfortunately, we’re still in a world were non-Microsoft software is looked at with skepticism from many of the decision makers in companies.

Row data gateway (Which is Active Record if you have domain logic)

This is the same as having a gateway which gives you objects per row in the database. If you add domain logic to these objects you have what is called Active Record. I’m going to concentrate on the Active Record approach, as I can’t see a good reason why you’d want to have a simple row data gateway in .NET. Active record is a domain object which handles persisting itself.

Active Record has the advantage of being simple and quick to implement. It is not hard to understand, and is a good way to make a quick prototype while retaining a domain model. It breaks down once your database gets complicated, and once you don’t have a one to one mapping between an Active Record object and a database table.

You can use Castle Active Record to do this. It is built on top of NHibernate. In terms of refactoring away from Active Record if the complexity increases, apparently there is a way to automatically go from Active Record to a full OR-Mapper NHibernate solution automatically. I haven’t tested this though.
(You’ll probably end up with a NHibernate solution through Castle then as well. This is not a bad thing! Castle integrates very well with NHIbernate through its NHibernateFacility, and easily allows for instance a Dependency Injection approach as well. I’ve written about it in a previous post)

Table data gateway (Dataset)

In the .NET world, table data gateway is the same as the dataset-approach.

The major benefit of using datasets is the unmatched framework support for it, where creating a datagridview and datasource, using databinding, adding a navigator to handle paging, etc., is extremely simple and powerful. For quick demos, or Rapid Application Development (RAD), nothing can match it.

There are two main problems with this approach

• It really doesn’t scale. Once you start adding business logic, you’ll have problems with the lack of object orientation, with the lack of type safety, with duplication of logic, and lots more.


• Expectations, if you use it with prototypes. If you give the business side a quickly running demo with this approach, you’ll get into problems when you try to explain to them how long it will really take to build.

If you know you are creating a very isolated, not to be extended, solution, by all means use the dataset approach – nothing can match it in speed or simplicity. If there’s a chance you need to add more to it later, opt for another alternative. You’ll have a hard time refactoring it later on.

Stored procedures/views

Putting everything in stored procedures/views is another approach that has been used. I’m not going to bother saying much about it, as it’s not really a viable alternative. You’ll be better of using this approach than manually concatenating information into SQL queries though (Like avoiding SQL Injection attacks)

It is a possible approach if you have special cases where you just can’t make the performance demands without using stored procedures.

Manual SQL

Don’t bother. But if you have to, at least limit its use to a database layer. And make sure you remove hazardous characters so you avoid SQL Injection.

Object database

An alternative quite unlike the rest. Whereas the previous sections concentrated on ways of working with a relational database, you also have the option of using an object database. If you choose this approach you can just pass your objects to and from the database.
Again, I haven’t tested this, but I have very experienced colleagues who have little but positive things to say about this approach.
The pros of this approach includes

• No need to map between a relational design and a domain model


• Don’t need to update a schema several places


• According to some benchmarks, they can be superior for certain kind of tasks. It has been said that they are very efficient at specific queries, while they are slower at more general queries.


• Most of the object databases also support some sort of query language when the need arises


• Some even fully support SQL, but I have no idea how this works in practice.

The cons

• Practical knowledge of these are still fairly limited


• Hard to access from other parts of the company network, for reporting purposes for instance

One approach some of my colleagues took was to go through the whole development period with an object database before converting to a NHibernate solution before going live. The conversion into a relational database was only done because of company demands. This was still a success, but if you do this you have the danger of not quite knowing how long it will take to set up the final solution or exactly how the data will perform.

I look forward to testing this in a real world project, you should too.

Conlusive thoughts

In summary I’d say that you should

• Use the table data gateway/dataset approach if you have a short and sweet application


• Use Active Record if you need to get results fairly quickly, and have a close relationship between your table and domain structure


• Find out whether a commercial/open source OR-Mapper or code generation tool or an object database suits your needs best - if you need a somewhat complex application


• Build your own OR-mapper if you’re forced to, it’s not good for the project, but you’re lucky :)

Replace strings for property names with type-safe version

At times you have the misfortune of having to write property/field (I’ll just call it properties from now on) names as strings, for instance in advanced NHibernate queries. (In general, NHibernate Query Generator avoids this for you in most cases, but there are still some times it cannot be used. Say inheritance for instance). If you’re a bit slow on progressing in mocking tools, this might be a problem as well (It shouldn’t still be though…)

The big problem with non-type safe strings is refactoring, which should happen all the time on your projects. Property names will change, and when these are used as non-typesafe strings in your system it has a few unfortunate consequences:

• You could break something without knowing it. Worst case is you won’t find it until production


• The knowledge above could restrain you from doing refactorings.


• You will have to spend time doing text-searches in the system to find out if and where it is used.

So for a personal project I’m currently playing with, I figured I wanted to do something about it. So I created a small generator project which generates static classes where you have access to the domain objects property names in a type safe manner. What this does for me is:

• If I do changes to a property of a domain object, and that has been used as a (previously) non-type safe identifier, my compiler will complain.


• I can do refactorings all the time without worry.

For now it looks for instance like this in use:



This will return “Text”, which is the name of a property in the Resource class or a base class.

I’ll be the first to admit that this is not ideal. Property names as strings are not something you want to deal with, but at times you have no choice, and this feels like the better of the two options.

For those interested in how I chose to solve it:
The generator takes in an input path, a search string and an output path. This is put in a bat-file which is called from the domain projects build. Currently I’ve only implemented a parser for NHibernate mapping files (HBM), which for each class creates a file with each property listed for that class. (I can see situations where you would want to use properties not just used in persistance, but that’s all I need for now). Inheritance is handled by duplicating the properties of the base class in the subclasses.
In short, the file created for the example above looks like:

Regular Expressions in Visual Studio – the top-down approach

Basically every approach to regular expressions I’ve seen uses the technique of learning all the symbols first, then using that to create various regular expressions. Let’s call that the bottom-up approach. I figured I could add some value by doing it the other way around – by showing specific strategies where regular expressions make sense, showing the regular expressions needed, and then explaining how it works. In other words, the top-down approach.

Once you’ve read this post you should be able to use regular expressions to do pattern matching and extraction in Visual Studio. I’ve kept the number of regular expression meta-characters to a minimum, (hopefully) making it easily understandable as well. You’ll be far from an expert on regular expressions after reading this, but hopefully I can either help get you interested in regular expressions (You should!) or learn some tips about using it with Visual Studio.

I’ve tried to divide the contents into logical sections, so just skip whatever doesn’t sound useful.

On Visual Studio

The built in regular expression support in Visual Studio (2005) is fairly strange – it doesn’t quite follow conventional regular expression syntax, it doesn’t follow the .NET Framework syntax, and irregular behavior has been found. It’s not better that it seems extremely slow once you give it fairly large files either.

Besides its drawbacks, it has become the place I use Regular Expressions the most. And trust me, I like using Regular Expressions. So why use Visual Studio? The probable reason is that I use it on a day to day basis, and since it’s not uncommon that I use it for searching, I guess it just became natural to start using it for other tasks as well.

Oh, and by the way: There’s lots of regular expression software out there you should give a try. Free, good software, with somewhat more standards-based meta-characters, probably faster, likely to have more functionality related to regular expressions, and various other improvements.

Enabling Regular Expressions

• Regular expressions are used from the “Find and Replace” dialog


• ”Match case” also applies to your regular expressions


• The "Find in files" "Find and Replace" dialog (ctrl-shift-f) enables you to search only certain file types


• The "Find in files" "Find and Replace" dialog shows all results in the Find window, instead of just finding the next, as with the "Quick Find" dialog.


• The most used regular expression meta-characters are available in the button next to find once you enable regular expressions.

I’ll add the actual dialog Visual Studio displays here just for reference. It describes each meta-character fairly well.

Regular Expression Strategies

General

Ever after learning regular expressions I seem to always find good places to use them. Here’s the list of strategies(or common situations) I’ve covered in this post:

1. Extract content from lines with a common pattern


2. Remove empty lines (optionally with whitespace)


3. Remove lines not following the pattern you are looking for


4. Add characters around each line


5. Retrieve the contents of all lines containing a pattern

Regex strategy #1 - Extract content from lines with a common pattern

Since this is the first example, I’ll walk through this one extra slowly

Description: You have a number of lines of contents that have some sort of similar text you want to extract.

Example: We have two lines of text, and want to extract the value in the target part of the xml.

<xml target="test" />
<xml info="" target="test2"/ >

Steps:

1. Identify a unique string in the lines straight before the content you want to extract
   
   
   • target="


2. Identify a unique string or character on the other side of the content


• "


3. The major part of the job is done; just create the regular expression to use the two first points :)

The regular expression we begin with to match the parts we identified above (Don’t test it yet, I’m skipping some important details still):

• target=" [^"]*"

Splitting this up we get three parts: target=", and " are just plain text matching. [^"]* is different though. If we take a look at the picture above, we can recognize two meta-characters:

• [^] – Any one character not in the set


• * - Zero or more

So [^"]* means match a character which is not a ", zero or more times. All regular expressions are greedy by default, so it will try to match as many characters as possible.

To try to show this clearly I will differentiate the different parts of the regex and what it will match by using font styles.
The regex:

• target=" [^"]*"

Will match:

<xml target="test" />
<xml info="" target=”test2"/ >

We’re not quite there yet. First of all we need to escape all non-letter and non-number characters with a \ to make sure they are interpreted as plain text characters. I didn’t do that above as it would have made it harder to understand at first. We get:

• target\=\"[^\"]*\"

Harder to read, but necessary. You’ll get used to it.

We’re now able to match the text we want in both lines. However, we’re still unable to retrieve the information from the fields. To do that we first need to match the entire line, so that we can remove the parts we don’t want. To do that we add .* to the beginning and end of the expression:

• .*target\=\"[^\”]*\".*

The . (dot) is a special character which is interpreted as "Any single character" (Except line break). The regular expression will then read:

• .* - Match as many characters as possible (Up until the (last) target\=\" part of the text


• target\=\" - Match the plain text target=",


• [^\"]* - Match as many characters as possible until we reach a "-character


• " - Match the "-character.


• .* - Match as many characters as possible. (Since this is the last one, it means to the end of the line)

With the above expression we match the entire line. The final part we need is a way of extracting the information we want. We’ll do that by adding { and }. This is basically a grouping construct, and you can have any number of them in your expression.

• .*target\=\"{[^\"]*}\".*

We now have the entire regular expression we need! Let’s test it. Copy the example xml text into a text document and open it in Visual Studio. Open a "Find and Replace" dialog, and add the regular expression. Then add \1 in the "Replace with: " part. \1 means the part you have between your first { and }. In other words:

• Find: .*target\=\"{[^\"]*}\".*


• Replace: \1

The result is:

test
test2

Success! :)

Final comments:
Interested in how the regular expression engine actually does the matching? Let’s see what it matches for each part of the expression, until we reach the final match.

• .* - Match as many characters as possible. This will actually match the entire line. Since the .-character doesn’t match a line break it stops at the end of the line (in other programs with other options, it is possible to make it match line breaks as well.) Remember, it is greedy, so it wants to match as much as possible.


• target\=\" - Match the plain text: target=",. Now, to be able to fulfill this requirement, it has to "let go" of some of the matched characters. So it let’s go of one and one character until it finds that it can match the string. That’s why, if there had been several target="-parts in the line, it would have matched the last one.


• [^\"]* - Match as many characters as possible until we reach a "-character. Match one and one character, until it reaches a "-character.


• " - Match the "-character.


• .* - Match as many characters as possible.

In fact, different regular expression engines do the matching in a different way, but this is all you need to know to understand how it works.

Regex strategy #2 - Remove empty lines (optionally with whitespace)

Description: You have a document with several empty lines you want to remove.

Example: : We want to get rid of the empty line between the two words.

SomeText


SomeOtherText

Steps:

1. All you really need is the following:
   
   
   
   • Find: ^:b*$\n
   
   
   • Replace:

The meta-characters used here means:

• ^ - Beginning of line


• $ - End of line


• :b – Space or tab


• \n – Line break

When splitting the regular expression into parts it can be read like this:

• ^ - The match must start from the beginning of the line


• :b* - It will be followed by zero or more (as many as possible) whitespace characters.


• $ - The match must end at the end of the line. In other words, the line must contain either no characters or only whitespace characters to give a match.


• \n – Finally we match the line break for the line as well.

This regular expression will match empty lines, including the line break. By replacing them with nothing you get the effect of removing the lines.

Final comments:
Make sure you have a line break after your last line, or it won’t match the regular expression, as the regular expression requires a line break. Optionally you could have appended a * to the \n, making the expression *:b*$\n*. This would have removed the need for the final line break.

Using Excel to sort the data is a good alternative to remove empty lines. That won’t work if you don’t want the data sorted of course.


 

Regex strategy #3 - Remove lines not following the pattern you are looking for

Description: With the apparent weakness of regular expression matching in Visual Studio, in that you need to replace the entire line to retain the information you want (instead of getting the matches in a separate window), it is not uncommon to get into the situation where you have lines that you need to remove to be able to focus on the lines containing the information you want.

Example: We got some more XML, this time with comments we want to get rid of:

<!—xml comment -->
<element>
<!—another comment -->
<subelement />

Steps:
The general strategy here is to do a (preferably) two-step action to

• Mark the lines you don’t want with a special identifier


• Remove all lines with the special identifier

The actual steps:

1. Identify a unique string or recurring pattern in the lines you want to remove
   
   • <!


2. Write the regular expression to identify and then add the identifier to the line
   
   • Find: ^.*\<\!
   
   
   • Replace: #\0


3. Remove the lines containing the special identifier
   
   
   
   • Find: ^\#.*$\n
   
   
   • Replace:

With point 2, you want to single out all the lines you don’t need (Preferably in one operation, but not always possible). If you have a unique identifier across the entire document, then where you match in the string and add the identifier is unimportant. Often, placing it in the beginning or end of the line is a good starting point.

The first find operation (^.*\<\!) should be easy to read now, as there are no new characters. But for the sake of it: Match start of line, match as many characters as possible until we match the last set of <! in the line.

The replace operation is slightly different. # is the special unique identifier we have used here. It has no special meaning – you could have used an x, or three x-es for that matter, as long as it is unique for the entire line or position throughout the document. Whereas \1 meant the first specified match we found (marked by { and } on each side), \0 holds a copy of the entire line that had a match in it. In effect this means that we simply add a # to the beginning of each line matched.

With point 3, we want to remove all lines with the identifier. So we find each line with the special identifier in the beginning, match the rest of the line with .*$\n, just as before, and then replace it with an empty string.

Final comments:
Another possibility is matching the lines you want to keep, tagging them with the special identifier, and then removing the lines not containing the identifier. Of course you will have to remove the special identifier in the end, so there’s one more replace involved. Not a big thing though.

A third possibility is to use strategy #5 – Retrieve the contents of all lines containing a pattern. This is most useful when you need to match across several documents.

 

Regex strategy #4 – Add characters around each line

Description: At times I seem to end up in the situation where I have lines of information that I need to surround with information or characters.

Example: We have a number of lines of information which need to be used in an SQL in-query as strings, and thus need the necessary surrounding characters. The SQL query is: SELECT * FROM something WHERE name in (…). For those unfamiliar with SQL syntax, we want to add a ‘ to the left of the expression and ‘, on the right side. The information:

Test1
Test2
Test3

Steps:
As long as none of the lines involved already has the necessary characters and no characters that will invalidate the statement, all we need to do is:

• Find: ^.*$


• Replace: ‘\0’,

The find simply matches everything on each line, from start to end character.
The replace puts our plain text characters on each side of the expression. The result is:

‘Test’,
‘Test2’,
‘Test3’,

You’ll need to remove the last ,-character yourself, and paste it into the SQL query, making it:

SELECT * FROM something WHERE name in (‘Test’,
‘Test2’,
‘Test3’)

Final comments:
Another common usage is creating SQL inserts. The logic is just the same; just add some other information around.

Regex strategy #5 - Retrieve the contents of all lines containing a pattern

Description: You want to get the contents of all lines matching a certain pattern, either because the lines make sense for themselves, or because you want to do further work on the lines. This can of course be for one or multiple documents.

This is an alternative to using strategy #3 (Removing lines not following the pattern you are looking for). Strategy #3 is probably faster when you work on a single document.

Example: Reusing a previous example contents. This time we want to retrieve all of our XML-comments:

<!--xml comment -->
<element>
<!--another comment -->
<subelement />

Steps:

1. The regular expression needed to find the correct lines aren’t really the important part here, as what we wan’t to show is how we go from there. We don’t even need to use a regular expression in this case, just search for <!--. The important part here is to do this search in the “Find in Files”-dialog (ctrl-shift-f). The results listed should be:
   
   
   • C:\aPath\file.txt(1): <!--xml comment -->
   
   
   • C:\aPath\file.txt(3): <!--another comment -->


2. Now copy the results from the “Find Results”-window and paste them into a document.


3. We need to remove the path information. Use the following:
   
   
   • Find: ^[^\:]*.[^\:]*.
     
     
     • ^ - The match must start from the beginning of the line
     
     
     • [^\:]* - Match zero or more (as many as possible) non-:-characters.
     
     
     • . - Match the :-character
     
     
     • [^\:]*- Match zero or more (as many as possible) non-:-characters.
     
     
     • . - Match the :-character
   
   
   • Replace:


4. Now we remain with only the lines we’re interested in.

Where do you go from here?

If you really want to learn about regular expressions, you’ll need to get Mastering Regular Expressions (Jeffrey Friedl). The book is THE book on regular expressions.

If you for some reason don’t want to get the book, a series of 10 videocasts from Zain Naboulsi (Is this thing on) should be your second choice. Besides being well made, it focuses only on .NET, which Mastering Regular Expressions only does partly.

IoC Containers generally, Castle Windsor specifically

First I’ll have a go at reinventing the wheel by trying to give my own definition of an IoC Container

An Inversion of Control (IoC) Container is a non-invasive configurable intelligent factory-component

Splitting this definition in parts, we get

• Factory, because it is responsible for creating objects for you.


• Intelligent, because it understands what dependencies you have, and create them for you recursively.


• Configurable, because you can configure the usage through code or configuration files.


• Non-invasive, because the objects used doesn’t need to know about the container.

An IoC Container really makes the Dependency Injection pattern a lot better. Why? I’ll look more into that throughout this post.

My previous post is a foundation for this one, as it explains the Dependency Injection pattern, and gives it a place in the world. I’ll use no time on that here. This post exists to build on that content, and show what more benefits you can achieve by using the dependency injection pattern with an IoC Container. This isn’t meant to be a comprehensive introduction to IoC Containers or Castle Windsor, but should give you a good idea of what it can do for you. Once you’ve read it, just continue on with what I’ve added in “References and further reading”.

I’ll use a simple example to help explain how it works. We have a client, which uses a service, which again depends on IStorageObject, which have two specific implementations. This is not meant as a real world example, but bear with me. The dependencies and sample implementation are shown below:












My previous post shows how we can handle this manually, and the pros and cons with the different solutions. We would like to get an instance of client, but we need to handle all the dependencies it has as well. If we have to write this manually, it will turn out as something like this:



This doesn’t look too bad, and the dependency injection approach is often better than a lot of alternatives. But think of it. As soon as this starts expanding, I can bet you that you’ll have a mouthful.

Now to do the same with an IoC Container, you can typically write:



For the container to create Client, it needs to create an instance of Client, but also an instance of both IService and IStorageObject. The container will understand that it needs to create both, and does so depending on how you have configured it. More on that later.

What if we add another dependency to the service implementation? The manual code would have to change to something like this (Each and every place service is created in the code. Resharper could help you quite a bit, but it’s far from a solution to the real issue.):



Already guessed how it would look with the container approach?



You are right, just the same :)


This has a number of advantages:

• Easy to swap the implementation of an interface - just change the configuration.


• If you find that Client suddenly needs another dependency, you don’t have to rewrite any of the Client creation code.


• Removes explicit dependency-handling throughout the system


• Single point to find out how objects are instantiated and connected


• Easy to begin using Client in other places. (Less boilerplate code to write)


• Additional benefits of tunneling dependencies through one source: Decorators, Interceptors, Proxies, as well as handling object lifestyle. Have a look here for a very good example.


• See my previous post for more on the general benefits of dependency injection.

I am going to have a look at an IoC Container called Castle Windsor now. There are many other alternatives as well. I can’t speak too much about the others as I only have real life experience with this. If we talk only about the general dependency injection problem/solutions, the different projects have a lot of similarities, with some differences in support for config vs code, need for attributes etc.

Configuration and use - with XML

You can use a separate configuration file, or add the castle definitions to your app/web.config.



The configuration above states:

• The string identifier of our only component is “storageobject”


• The service (or interface) the container will create (if any - you don't need to specify one), is IStorageObject, from TheNamespace namespace


• The specific implementation of this interface is DatabaseObject.

In practice, this means that if the container is ever asked for an implementation of “storageobject” or the IStorageObject interface, directly or through resolving dependencies, it will return an instance of DatabaseObject..



This code simply shows how the container would be started with this approach.

Configuration and use - with code

You can define everything in code:




The pros and cons of using XML versus code is that you can do changes without recompiling with the XML approach, while it is type safe when you specify it in code. Use whatever is best for your situation.

Lifestyle

Advanced handling of the lifestyle of objects has never been so easy. By adding a keyword, you can specify how your object lifestyle will be:



The previous configuration updated with specification of lifestyle:

Facilities

Adds integration with other tools

• NHibernate


• ActiveRecord


• WCF integration


• Remoting


• Logging


• Startable


• Custom


• +++

Other

Castle has a number of other possibilities as well, like specifying what to do when objects are created and destroyed (Commision and Decomission).

Note

The last couple of points feels kind of pointless without giving them much extra attention, but I'll leave that to other people who have described it much better. A good starting point would be this or this.

References and further reading

Inversion of Control and Dependency Injection with Castle Windsor Container - Part I (Simone Busoli)

Inversion of Control and Dependency Inject: Working with Windsor Container (Ayende)

A not so interesting post, with much more interesting comments.

Castle Windsor tutorials (BitterCoder's Wiki)

Inversion of Control Containers and the Dependency Injection pattern (Martin Fowler)

Castle Project

Conclusive thoughts

Although a short introduction, I hope I have captured enough information to get you interested. I’ve been using Castle for quite some time on my last project, and I am very happy with the results. I advise you to give it a try!

_

Objects, dependencies and patterns to manage it

The purpose of this post is to pass on knowledge about an important aspect of software development, with the hopes that someone else can have use for it. It is a presentation I held at work changed into written form.

The topic is managing dependencies between objects; how it affects the systems we build, and a few thoughts on different ways to handle it. We are going to look at three patterns: Factory, Service Locator and Dependency Injection, which in various ways help us control the dependencies. The purpose here is not to explore all the details of the various patterns, but rather to capture how the patterns handle these situations in general.

As a way of comparing the patterns, I’ve put together a few well known principles that has important things to say on the quality of software.

Simplicity

• Complexity is so important to handle. Complex code is hard to read, change and manage. The challenge is to write code that people understand, not just code that compiles.

Separation of concerns (SoC) / High cohesion

• Software that separates its different parts well is easier to understand and manage. The more a unit of code,which from now on should be understood as either a module, class or method, handles only a separate piece of the puzzle, the better. The Single Responsibility Principle fits well here: One unit of code should do only one thing, and do it very well.

Loose coupling

• How much your unit of code depends on other parts of the system. Less coupling means code that is easier to understand, maintain and reuse in different contexts.

Ease of change

• Highly related to all of the points above. We want code that is easy to change. The reason for this is of course that there are few environments were things change more often, be it because of business or technological needs, or really just realizing the nature of most software projects.

Encapsulation

• Design choices that are not important to know of outside of your unit of code, should not be visible there either. It only increases complexity, makes things harder to change, and removes focus from what is really important.

Testability

• Highly testable code has a number of benefits for the quality of your system, and is an important principle to live up to for the vast majority of projects.

I have created an example we will use throughout the post. It consists of two elements, a client and a service. The simplicity makes it impossible not to understand, as well as hopefully easier to relate to a number of situations, though making it harder to show real world problems and their solutions directly.



The service could be anything from an in-process calculator to an expensive web service or database service. The context has a lot to say for the conclusions, and it is vital that you relate the examples that will follow to the many similar situations you see on a daily basis. Depending on whether we would expect change here, we might or might not include an interface.

This is an example of the implementation of Client (in C#):



Potential problems with this code

• It is hard to change, because it is connected to a concrete implementation of the service in the constructor.

• You need to read the Client code to find that it has a dependency to Service.

• Testability would be hurt if this was a resource-expensive or unreliable service.

Potential advantages of this code

• It might do exactly what it needs to do - If it seems natural that Client should be responsible for creating Service, if there is no benefit of changing the implementation of Service in different situations, if testing isn’t hurt by not accessing it, then by all means keep it as simple as you can. YAGNI (You ain’t gonna need it) and KISS (Keep it simple, stupid) are important principles to follow.


• As a side note concerning testing: It is possible to test this with TypeMock.

What if we increase the complexity of this example? Let's add an IStorageObject, which is a common interface for two ways of storing data, DatabaseObject and CacheObject. As the diagram below shows, Service now has a dependency towards DatabaseObject (And to IStorageObject, though not shown explicitly).



And the implementation:





As simple as this example is, we already have a potential for tight coupling. Even though we are coding to interfaces, this is not that easy to change. Reusing Client or Service in a different context is very hard. Replacing the implementation with a mock or stub during testing is in most cases impossible. And even though this example hides it, both Service and Client is now responsible for both creating and using the dependency, responsibilities we often want to separate (e.g. because of complex initialization). In real life, the complexities and couplings are often increased tenfold, and only then can you see the true value of these principles.

Pattern #1 - Factory

The factory pattern is one of the GoF patterns, a creational pattern, responsible only for creating objects. I have created a dead simple implementation of the factory pattern, where you have a ServiceFactory class with a static method to give you the concrete instance you want.



The dependencies can be modeled like this:




Advantages

• Client only has a direct dependency on IService now, not the concrete initialization.


• The code is more testable, if we have a factory where we can switch the concrete instance with a mock object.


• The usage of Service will be clearer and easier to start using as all creational logic has been separated.


• Having a single place to get instances of a service means you have more general possibilities, like the transparent handling of instances (always new or singleton). I will look at others throughout this and the next post.


• We have separated the responsibility of creating and using the service. If creating a service means several manual steps to configure it, then this is even more valuable.


• The Factory could return a different implementation based on input, a common reason to use a factory.

Problems

• In our situation, we still have a compile time dependency on the specific IService implementation. The dependency has only moved a step out.


• Reuse of Client in different contexts is hard, because of the still tight coupling to Service.


• If we look back at the example with the deeper dependencies. Should we make another factory to handle this creation? Has this really loosened the coupling that much, and do we want to use all our time writing boilerplate factory-code?


• We still have to read the implementation of Client to find that it has a dependency towards Service.

That being said, let’s look at how it lives up to the different principles listed in the beginning


Simplicity

• The factory pattern is easy to use, and is one of the most known patterns. Thus, using it will help give clarity to what you try to achieve, even to developers less versed in the "patterns world". Though it is not correct in every situation, and too many factories is not necessarily a good thing either.

Separation of concerns (SoC) / High cohesion

• The creation and use of a dependency is separated.

Loose coupling

• The example still has a compile time dependency on the concrete implementation of IService. In many cases the factory pattern can help with the coupling, for instance by sending in some form of input.

Ease of change

• Reusing Client in a different context is hard because of the still tight coupling.

Testability

• If the implementation of the factory allows you to switch implementations, then yes, it is more testable. You need to initialize the factory with every test however, and even switch back again to the previous implementation to make sure that you don’t break other tests.

Pattern #2 – Service Locator

An (potentially global) object that gives access to services the application needs.


The example changes to:





Instead of the factory class we now have the service locator that creates and/or returns the service. The biggest change here is that you now have an assembler class which typically configures the Service Locator with the correct instances.

Two points concerning the implementation

• In the example above I have used a string to get the correct type of service. This might as well have been solved with sending in a type or using generics if you want type safety and support for refactoring, at the cost of some flexibility though.


• The example uses a Service Locator interface to separate the LocateService method from the methods Assembler uses to register services. The implementation is still a singleton, and could be changed to not use GetInstance first.

Advantages

• Encapsulates the logic to find services.


• A separate interface to deal with.


• A central point of control. Easier to do similar things with the services you get, like handling instances and caching.


• Can add a mock implementation during testing.

Problems

• Hard to handle deeper dependencies well.


• Advanced initialization of objects, which the factory does so well, is hard.


• Possibly code to reset the service locator to make sure that one test does not break others.

And onto the principles:


Simplicity

• Easy to use. A single method to call.

Separation of concerns (SoC) / High cohesion

• The creation and use of a dependency is separated.

Loose coupling

• If you have an implementation that uses an XML file to configure up the specific instances, then there is no compile time dependency.

Ease of change

• As there is less coupling, it is easier to change as well. Reusing Client in a different context is still often troublesome though.

Testability

• This pattern does make it even easier to test than Factory, but you still have to write extra code to create and change the service locator.

Pattern #3 – Inversion of Control / Dependency Injection

A fancy word for a really simple principle: it simply means injecting dependencies.







As you can see in the images above, Client now receives the specific implementation of IService from the outside.

Before going further, I think it is necessary to write a couple of lines about the confusion concerning the two concepts Inversion of Control and Dependency Injection. Inversion of Control is a common concept in computing. It relates to a situation where you instead of taking the active approach and asking for something, the framework is the active part that calls you. (i.e. events, the startup method of an application, etc) Dependency Injection is a specific form of Inversion of Control, where instead of asking for a specific dependency (As was done with Factory and Service Locator), the dependency is handed to you.

There are three types of Dependency Injection: Constructor, Setter and Interface injection.

Constructor injection
- what is shown above, injecting the dependencies in the constructor.

Advantages

• Any user of Client will see that it has a dependency on IService.


• Unlike the other injection approaches, Client is completely initialized once created, as you can’t create it without giving it an instance of IService.


• You can set IService as ReadOnly, since it is set in the constructor.

Problems

• If you have optional dependencies, especially if they are resource-intensive to create, then it is not a good idea to initialize them in the constructor.


• If you have many optional dependencies, and want to give a separate constructor for each valid case, then you soon end up with a lot of constructors.

Setter injection
- Injecting the dependency through a property.



Advantages

• Good way to handle optional dependencies, especially if they are resource-intensive.


• Support for Dependency Injection in legacy code without having to change the constructor, for instance to improve testing, or to add new functionality.

Problems

• Harder to know what you need to do to use the object, which can lead to surprises.


• Somewhat harder to track down exceptions.

Interface injection
– a special form of setter injection: Specifying injector methods in an interface.



As this is almost identifcal to Setter injection, there isn’t much else to say about it. The only difference is that it can help separate out what are really dependencies that must be filled. However, this seems to be the least used form of dependency injection.


Advantages of using Dependency Injection in general

• Easier to test: You can simply inject a mock.


• Potential for less coupling, as it doesn’t ask for a specific implementation.


• Easy to reuse Client in different contexts, because of the point above.


• Not necessarily compile time binding. And even if so, different implementations of Client can easily be bound to different implementation of IService.

Problems

• You’re not supposed to inject everything. One problem is that you can break encapsulation, another that you should take care not to create interfaces for all objects in your system. Coding to interfaces is a good thing, but making interfaces where you don’t expect change or testing isn’t a good enough reason, will make it hard to see where you truly expect change in the system.


• It can be cumbersome to do Dependency Injection manually. I look at a solution for that in my next post.

So how does Dependency Injection live up to the initial principles?


Simplicity

• Simple to write and understand. But writing manually can be time-consuming.

Separation of concerns (SoC) / High cohesion

• The creation and use of a dependency is separated. Also, Client only uses the dependency now. Compared to the other patterns, Client doesn’t even need to ask for an implementation.

Loose coupling

• The code has a potential for even less coupling, as it only uses it's dependencies.

Ease of change

• Easy to change in terms of using Client in different contexts, but manual dependency injection code isn’t any easier to change.

Encapsulation

• Don’t break encapsulation where it’s not natural.

Testability

• By far the easiest way of testing objects where you need to add a mock implementation.

References and further reading

Inversion of Control Containers and the Dependency Injection pattern [Martin Fowler]

A few of the posts on a longer discussion of Dependency Injection: (You can find more from each)
Dependency Injection
Dependency Injection: More than a testing seam
Defending Dependency Injection
Dependency Injection - Keep your privates to yourself

Dependency Injection - MSDN Magazine
Guice video. Some good general points as well, if you're bored with reading.
Applying Domain-Driven Design and Patterns (Book) - On Dependency Injection and Service Locator

GoF (Book) - On Factory
Head First Design Patterns (Book) - On Factory

Conclusive thoughts

At the outset of creating this presentation, I was planning to only write about the Dependency Injection pattern. However, I felt that putting it in a bigger context, and trying to present the different patterns in a more uncommon way would be more helpful. This is by far perfect, but if it can help anyone better see the usage of these patterns and part of their place in the world, then it was worth rewriting this for the web. Not the least, if anyone can further educate me as well, what could be better?

This was part 1 of the presentation. Part 2 handles IoC Containers, and how you can do Dependecy Injection without having to do it manually.