VS Add-in: Extending Sealed Classes (without Extension methods)

If you are ever in the situation when you have a .NET Framework class with most of the functionality you want, but not really doing everything to the extent you need, you have a few options:

• You can inherit from the class and extend as you see fit


• You can create a utility class to perform the extra functionality


• You can use extension methods to add functionality to the class


• You can create a new wrapper class which holds the class as an internal variable and create wrapper-methods for whatever methods you need from the class you are extending, and add any extra functionality you need.

Inheriting from the class is likely the best approach. Unfortunately, about 40% of the .NET 2.0 Framework classes are sealed, which of course means you can't inherit from them.

The second approach is becoming somewhat obsolete, as it can lead to a sort of functional programming which takes you away from proper object-oriented programming. It makes things just a bit harder to find and understand.

Number three, extension methods, is the new cool kid on the block. Add new methods just like magic to any class. The possibilities are enormous! The problem is that you can't do anything with the internals of the class, you can only add new isolated methods.

The last approach can be the way to go if extension methods won't do. This way you'll have complete control over what you want to be public and you can extend and improve on existing and new functionality. There are two problems with this approach. Number one is clarity and number two is identity. If you create an extension of the DataSet for instance, it never is a dataset. You can, in a way, get around the issue by inheriting from one of it's base classes or interfaces though (Using that as a common base).

Be sure only to use it when no other approach is better. The clarity/identity issues can be confusing for someone seeing the class at first. But I do think the wrapper/composition approach have some merit, especially considering the extreme amount of sealed classes that exist. I made it because I was annoyed to find that there was no support for doing something like that besides typing it in manually. And if you have a lot of methods to wrap, that means a whole lot of time typing nonsense methods, not to forget the comments you should add if you want it understandable. No way I'm doing that unless I'm really keen on wasting a lot of time.

That's why I've created a code-generation add-on for the last approach. It's a Visual Studio Add-in to help extend sealed classes.



I’ll begin with showing some results, and then I’ll get to how and why it works/makes sense/you should use it. In this example, I'd like to extend the StringBuilder class. To do that I need to write something like:



Once I press SPACE or TAB while at the end of this line, the add-in starts its work on using this StringBuilderExtender class to extend System.Text.StringBuilder. The result is this:




More examples here and here.


What happened is this:

• Added an internal StringBuilder instance


• Recreated all public constructors, methods, properties and fields, and made sure all of them use the internal StringBuilder instance.


• Added any available comment from the StringBuilder class.


• Changed returntypes from StringBuilder to StringBuilderExtender where necessary.


• Added the Serializable-attribute, since StringBuilder is serializable.


• Listed the interfaces StringBuilder implements in a comment next to the class, in case you'd like to implement the same.


• Added the System.Data namespace to the using statements if it did not exist.

Why would you want to do this?

• You really want to extend one of the sealed classes.


• You need new functionality in a sealed class, and you either only have .NET 2.0, or extension methods just won’t do it.


• You reuse existing functionality, making a quick browse of the code enough to understand the majority of what the extended class does. Compare that to reimplementing a larger part of the functionality of a sealed class.

Why would you not want to do this?

• You are somewhat pretending to extend a class you cannot. This can get you into problems with equality and comparison. The commented interfaces do make it simple to extend any common interfaces though.


• Potential performance hit. You do add some overhead, and internal performance tweaks through for instance Win32 code could have has less effect.

More on the internals

In terms of what you can visually do

• You need to specify the keywords for the add-in on the class line. The input must always be in the form specified above:
  
  
  • public class StringBuilderExtender SealedClassExtender(System.Text.StringBuilder)
  
  
  • or in other words
  
  
  • [visibility] class ClassName SealedClassExtenderKeyword(TypeFullName)


• You can use both SealedClassExtender and the short form scx/SCX as keywords.


• The TypeFullName is, obviously, the full name of the type.


• You can also extend a class by specifying the path to the assembly. The form is then
  
  
  • public class StringBuilderExtender scx(System.Text.StringBuilder, C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorlib.dll)
  
  
  • or
  
  
  • [visibility] class ClassName SealedClassExtenderKeyword(TypeFullName,FullAssemblyPath)

Apart from the list above it does a few more things in the background

• Adds a default constructor if one isn’t specified in the base class.


• With any method, property or field that returns the base type (StringBuilder), it returns the extended type instead (StringBuilderExtender). A new private constructor with the base type is added if necessary to fulfill this.


• Static classes does not get an internal instance but just point directly at the class.


• Lots more fun things to fix every special case.

If you do not specify a path it will try to resolve where to load it from by itself. It will currently do this by first trying to load the type from memory, then specifically checking all assemblies in the current application domain, and then trying to resolve the name of the assembly from the types full name. It will first try to look in the current runtime directory, then in the executing assembly path.

The comments are loaded by using the XML-document corresponding to most framework dlls, as comments are not part of the information retrievable through reflection.

Be aware though, the add-on will rewrite the entire document, so only add using-, class- and potentially namespace-declarations, anything else will be overwritten.

It doesn’t handle everything though. Specifically it doesn’t do:

• Generics


• Events


• Identity-handling (Trying to guess how you want the identity-part handled will probably just end up with a bad guess, you’ll have to do this however you want it. Methods for equals, GetHashCode etc. are thus not created.)


• Limited attribute support.


• It also currently loads all assemblies into the current app domain while running. This is just because of a time-constraint on my part, and I’ll fix that.

If you have gotten this far, send me a note if you see a use for it, just don't get it, or have any comments.

Usage

To use this, you first need to install it like any other add-in. Copy the files in the zip to the correct Addins folder, for instance \My Documents\Visual Studio XXXX\Addins. (If the Addins folder does not exist, just create it yourself) Once this is done, restart visual studio, and you can enable the add-in in the "Tools->Add-in Manager"-dialog.

Downloads

Download the 2005 version from here.
Download the 2008 version from here.

Entity Framework V2.0

Microsoft has started the work on Entity Framework V2.00 now. Apparently, this version will have a great deal more transparency during the process, making sure it ends up as something the users want.

Good stuff.

NHibernate, beware?

Update:

A petition against the current quality of Entity Framework has begun. Read it to get a brief overview of the issues, and add your signature if you agree.

PropertyNamesGenerator (source)

I recently wrote the post Replace strings for property names with type-safe version

This is the source code for the project it describes. I'm posting it because of a request, so hopefully you'll have use for it.

PropertyNamesGenerator

• Generates type-safe versions of property/field names.


• Generates C# or VB files


• The only parser implemented is for parsing NHibernate mapping-files

The NHibernate parser handles regular classes as well as subclasses.
Another parser which will probably get there is for regular domain classes.

Warning: This is not production-grade software, but a simple project I made for myself to handle a current need! This should not be used as a guideline for good design, nor expected to work flawlessly. No such thing have been attempted.

You can download it here.

The zip includes the source, the current binary, and a sample bat-file to use it.

How to use it:
To run the generator, start it with something like:

• "C:\path\PropertyNamesGenerator.exe" /lang:cs /files:"../../source/Domain/*.hbm.xml" /out:"Output"

An explanation of the various params can be found by running it with a -? switch. It prints:
Run like: PropertyNamesGenerator.exe /files:"path/*.hbm.xml" /out:"path/PropertyNames/" /lang:cs
Parameters:
/Files [Mandatory] - Must have both a directory and a search string.
/Out [Mandatory] - Must specify an output directory.
/lang [Optional] - Output-language. C# or VB. (C# is default)
-V - Verbose output.
Note: All directories can be relative.

It currently only supports searching for files in the directories explicitly specified, i.e. not subdirectories. This is a concious choice.

To integrate it with your build, typically use a bat-file in the same manner as the Generate.bat example file included in the zip.

To integrate it with your build, you can add something like the line below to your Domain Pre-build event: (Project/Build Events)
call "$(SolutionDir)\..\Tools\PropertyNamesGenerator\Generate.bat"
Note that you will need a full path to the PropertyNamesGenerator.exe-file in the bat-file to use it like this.

It will generate files on the form [ClassName].[lang] in the output directory specified. Using a PropertyNames folder to put all of these in is recommended.

You will of course have to include these file in your solution to integrate them. Do this every time you have added and parsed a new hbm-file.

Persistance-solutions and characteristics

The other day I invited my colleagues to a discussion about persistance solutions. The goal was to cover the various options for persistance in a software project, the pros and cons of each approach, and to increase everyone’s knowledge about it.

The approach planned was to identify the possible solutions, identify important characteristics to help compare each solution, and to go through each possibility.

The different types we identified were:

• Code generation


• Object relational mapper (custom made)


• Object relational mapper (commercial/open source)


• Row data gateway (Which would be active record with domain logic)


• Table data gateway (Dataset)


• Stored procedures/views


• Manual SQL


• Object database

And the characteristics in no particular order:

• Development time


• Flexibility


• Performance


• Stability


• Complexity


• Refactoring


• Expertise


• Effect/limitation/intrusion on system


• Rapid Application Development (RAD)-support


• Attachment to data type


• Amount of developers


• Company demands/guidelines


• What you are creating

I think it is best to start off by defining the characteristics; what they are and why they are important.

• Development time
  
  
  • The time it takes to get your persistance solution ready to use and understood. This can either be the time it takes to configure and use an existing solution or the time it takes to develop your own.


• Flexibility
  
  
  • The flexibility of the system in terms of handling the various storage challenges in a project. This also includes handling a system which inevitably grows and changes.


• Performance
  
  
  • Obviously performance is important, right? This includes how the system performs in general, how it scales, how it can help handle special cases. A word of caution though: Even though performance is essential, don’t optimize prematurely. In any system of reasonable size, you will have special cases that will become bottlenecks. However, with a profiler and the possibility of fine-tuning those cases, you should be quite alright.


• Stability
  
  
  • You need to know that the system won’t break down under normal usage or high peaks. Testing is essential for this.


• Complexity
  
  
  • How complex is it to use and understand? How much help do you get with error messages? What do you when you need to debug? Is it too complex for the average developer on the project, making people take (eventually unmanageable) shortcuts?


• Refactoring
  
  
  • What sort of help do you get when the need for refactoring appears (often)? How hard is it to do changes?


• Expertise
  
  
  • You always need to take into account the expertise and skill level of the people on the project. This is more important that selecting “the best” solution.


• Effect/limitation/intrusion on system
  
  
  • What does it take to integrate the persistance solution with your overall system? Will your domain model be POCO? Do you need to inherit from a base class? Implement an interface?


• Rapid Application Development (RAD)-support
  
  
  • What kind of GUI-support does the solution have? If you need something quick and dirty, what do you do? The winner here should be pretty clear (If not, you’ll soon find out)


• Attachment to database data type
  
  
  • How close is it tied to database data types? Can it for instance handle the types of several databases?


• Amount of developers
  
  
  • How many developers the project has. This could influence the complexity of the solution, or the time available to get it up and running.


• Company demands/guidelines/policy
  
  
  • Company policies/limitations are a part of normal life for a consultant, and often there is little you can do about it. The company might have policies against using anything open source, demand that all database access should go through stored procedures, etc.
  
  
  • What can you do about this? Unless you have a small project or are able to influence the architecture group, chances are you’ll have to manage. I guess you can pray that the architecture group doesn’t consist of a bunch of non-coding architects that selects technologies out of marketing slides.
  
  
  • Another point I need to raise here: If your company still live in the “don’t use open source”-world, chances are that most decision-makers are a bit out of sync with what is happening in the .NET world today. Agreed, just a few years ago things were pretty thin in terms of open source projects on .NET, but today things have most definitely turned to the better.


• What you are creating
  
  
  • The most important point! Make sure whatever you choose will work for your project. Unneeded complexity is expensive! But remember that going from one solution to the other is often hard, so beware the danger of a project growing on a bad solution; you’ll have a tough time getting out of it

That should cover the basics of what is important to think about before choosing a persistance solution. Let’s get to the actual solutions!

Code generation

This involves generating the data access and potentially partly the domain model from your database schema. You specify rules for how you want things generated. There are a couple of potential problems with this approach.

• If the software has a binary solution specifying the mapping, you’ll have no way to map two differing changes, which in effect means you can only have exclusive checkout. In a multi-person project this is highly unproductive, and don’t forget that you’ll be unable to use branches, as you can’t merge them later.


• The other is that once you want to add a field or property to a domain object, you’ll have to update the database structure and generate the files before you can use it. This does take some time. I’m not saying that most other approaches are must faster; the annoying thing is that you can’t simply add properties etc. during testing for instance.

I have to admit that I have limited knowledge of this type of persistance-solution, but I believe this approach could be successful. There is no reason why the general mapping solution between the database and domain objects should be any poorer than with a general OR-Mapper. It depends what extra functionality is available and how everything is implemented (works). By this I mean the SQL generated, how the session is handled, lazy loading, how it affects the domain model and system, if it supports queries or SQL/stored procedures for special cases, type safe support for anything and everything, maintainable/understandable script files, and a few more I probably forget at the moment.

Object relational mapper (custom made)

An OR-Mapper is a piece of software that handles the problem of transforming your domain objects data into its equivalent form in the database. Your domain model is made of objects and pointers; the database model is made of rows, columns, keys and relationships. These are very different ways of handling data.

By creating your own OR-Mapper you will have to handle the problem of mapping from the domain model to the database for starter. This is just a small part of what you need to handle though, and to make it clear => There are slim chances that creating your own OR-Mapper will be beneficial for your project. Doing it will be time-consuming and error-prone, you will reinvent the wheel without need (See Object relational mapper (commercial/open source)), you will tie up your best resources for a substantial amount of time, you will have to implement a lot of added functionality to make it functional. Oh, and I doubt any project will wait before the mapper is finished before starting the rest of the development - which means some other form of temporary solution must be created.. (An object database might be the best choice in such a scenario.)

However, as a colleague commented, it is probably a dream for most developers to do it. Why? First, it’s complex, so you’ll learn heaps by doing it. Second, you don’t have to relate to the business side, so you really just define most of the tasks yourself. Perfect or what?

Object relational mapper (commercial/open source)

This is the same as above, except you use an already existing solution from a vendor or open source project. I recommend this approach.

Why create your own OR-Mapper when great solutions already exist? If you choose a premade mapper you will (potentially) get:

• Shorter development time – You will need time to configure, use and understand the solution. This is time-consuming as well, but far easier than creating your own. Getting started should also be fairly easy, even though the tougher concepts need more time.


• There’s a chance that present/new developers have used the solution before


• A solution tested in many projects, which means less bugs and more working features.


• New versions at no development cost.

You need to live with a few risks though:

• Harder to customize and debug. You’ll have to use time on strange error messages which can make little sense.


• The company might stop development and support, or the developer base of an open source project could die out. If you create your own OR-Mapper, the main developer(s) could quit as well.

In general, getting something of the shelf is far cheaper than building it yourself, as long as your requirements are met. I believe that this solution most of the time far surpasses any positive effects of building it yourself.

Picking a mapper at random just by browsing some marketing slides are not the way to go. Make sure you thoroughly read specs and user feedback, or the best thing if possible: talk to people that has experience with what you are considering, if it is a popular solution they shouldn’t be too hard to find.

There are a range of solutions in this field, but personally I’ve had the pleasure of working with perhaps the most known of them: NHibernate. NHibernate is an open source OR-mapper, which is a port of the well known OR-Mapper Hibernate from the Java world. NHibernate is a great piece of software, it has been used on numerous projects, and quite a lot of information is available through blogs and forum.

I’m not going to list all the reasons to use NHibernate, a quick search on the web should give you that, but a few things:

• NHibernate lets you have (almost) POCO objects (You need to mark all persistable fields/properties as virtual. NHibernate subclasses your objects with the virtual proxy pattern, to give you lazy loading etc.)


• NHibernate has mapping files to map between your domain object and database. These enable easy modeling of inheritance, collections, etc.


• It is mostly type safe (with NHibernate Query Generator at least), except when you need to write advanced queries. (You could use something like my PropertyNamesGenerator though)


• You can use Hibernate Query Language to create special queries in the cases where performance isn’t good enough.


• Automatic lazy loading

A few bad things as well:

• There is some overhead involved, and performance has been noted as an issue on several occasions. I think this is more of a design issue with your average developer though. If you try to retrieve gazillions of data from lots of tables, you can’t expect it to be lightning fast. Proper table and index design, as well as good use of lazy loading should get you well under way. For the special cases where you do get a performance issue – use a profiler to see what the issue is, use query analyser to look at the SQL, use HQL or use manual SQL behind a well designed layer to access the data, use DTO’s to limit the data loaded and sent…. There are plenty of possibilities


• NHibernate has some strange error messages. Before you get to know it enough, you’re bound to use a few hours trying to figure out error messages which doesn’t make much sense. I base this on experience with the 1.2 version, not the new 2.0 release.

If you don’t want to go down the open source way, you’ve probably been (or will be) introduced to Microsofts new OR-Mapper Entity Framework. It’s Microsoft, so it’s bound to be good, right? (…)
Note: I have only read about Entity Framework, and thus my experience (both good and bad) are of questionable quality, so make sure you do your own research before making any conclusions.
The pros:

• Integrates nicely with LINQ (But Ayende has a project going to bring this to NHibernate as well


• Microsoft helps bring the concept of an OR-Mapper to public knowledge, which really is good


• With Microsoft’s size, there’s a good chance that the project will continue. This depends upon how many uses it of course; it might get shut down like Microsoft’s music service did.


• Developers versed exclusively in the Microsoft world have something besides datasets and manual ADO.NET to use.

The cons:

• They’ve built a completely new product - which means you should think they should be able to leverage the experience of already existing OR-mappers. You’ll need a good amount of resources to do this though. For some reason it doesn’t like Microsoft has quite lived up to this.


• Having read quite a few blogs about Entity Framework and spoken to Microsoft employees about it, I must say I’m initially skeptical. At least for enterprise development. According to one Microsoft employee, Entity Framework is only believed to have a third of its user group in the enterprise software world; the rest is your simple application developers. The needs are quite different.


• One of the aspects that alarms me quite a bit relating to enterprise development and Entity Framework (or really any modern type of development – read: with source control), is that until recently the mapping files were unmergable. (The XML created were put in a “random” order. A small change could lead to big changes in a document) I can’t for the life of it understand that Entity Framework could be designed as anything but a play version if that wasn’t an important design point from the beginning. Ayende had a post about a meeting he had with Microsoft about this point, and based on his reactions it seems likely that the team defended this decision. It seems Microsoft has improved this feature after the range of reactions on it, as mentioned here


• Explicit lazy loading. You have to explicitly say that you want to load a lazy-loadable collection. I think this sounds mostly annoying, as you need to fill your code with logic of testing if a collection has been loaded, and then explicitly loading it, compared to the NHibernate way of automatic lazy loading by simply using it. There is a good thing about it - the fact that you won’t get unexpected database calls from the GUI-layer because you forgot to load everything you needed. This could lead to a performance hit and other problems. I don’t believe that merits this solution.

Even though I’m skeptical about the current quality of Entity Framework, and would recommend using NHibernate instead, I’m positive to Microsoft’s general move into this realm. With their funds, future versions have the potential of becoming really useful, with hopefully seamless integration with the rest of the framework.

For now I’m mostly scared that I’ll be put on a project where its use will be mandatory. Unfortunately, we’re still in a world were non-Microsoft software is looked at with skepticism from many of the decision makers in companies.

Row data gateway (Which is Active Record if you have domain logic)

This is the same as having a gateway which gives you objects per row in the database. If you add domain logic to these objects you have what is called Active Record. I’m going to concentrate on the Active Record approach, as I can’t see a good reason why you’d want to have a simple row data gateway in .NET. Active record is a domain object which handles persisting itself.

Active Record has the advantage of being simple and quick to implement. It is not hard to understand, and is a good way to make a quick prototype while retaining a domain model. It breaks down once your database gets complicated, and once you don’t have a one to one mapping between an Active Record object and a database table.

You can use Castle Active Record to do this. It is built on top of NHibernate. In terms of refactoring away from Active Record if the complexity increases, apparently there is a way to automatically go from Active Record to a full OR-Mapper NHibernate solution automatically. I haven’t tested this though.
(You’ll probably end up with a NHibernate solution through Castle then as well. This is not a bad thing! Castle integrates very well with NHIbernate through its NHibernateFacility, and easily allows for instance a Dependency Injection approach as well. I’ve written about it in a previous post)

Table data gateway (Dataset)

In the .NET world, table data gateway is the same as the dataset-approach.

The major benefit of using datasets is the unmatched framework support for it, where creating a datagridview and datasource, using databinding, adding a navigator to handle paging, etc., is extremely simple and powerful. For quick demos, or Rapid Application Development (RAD), nothing can match it.

There are two main problems with this approach

• It really doesn’t scale. Once you start adding business logic, you’ll have problems with the lack of object orientation, with the lack of type safety, with duplication of logic, and lots more.


• Expectations, if you use it with prototypes. If you give the business side a quickly running demo with this approach, you’ll get into problems when you try to explain to them how long it will really take to build.

If you know you are creating a very isolated, not to be extended, solution, by all means use the dataset approach – nothing can match it in speed or simplicity. If there’s a chance you need to add more to it later, opt for another alternative. You’ll have a hard time refactoring it later on.

Stored procedures/views

Putting everything in stored procedures/views is another approach that has been used. I’m not going to bother saying much about it, as it’s not really a viable alternative. You’ll be better of using this approach than manually concatenating information into SQL queries though (Like avoiding SQL Injection attacks)

It is a possible approach if you have special cases where you just can’t make the performance demands without using stored procedures.

Manual SQL

Don’t bother. But if you have to, at least limit its use to a database layer. And make sure you remove hazardous characters so you avoid SQL Injection.

Object database

An alternative quite unlike the rest. Whereas the previous sections concentrated on ways of working with a relational database, you also have the option of using an object database. If you choose this approach you can just pass your objects to and from the database.
Again, I haven’t tested this, but I have very experienced colleagues who have little but positive things to say about this approach.
The pros of this approach includes

• No need to map between a relational design and a domain model


• Don’t need to update a schema several places


• According to some benchmarks, they can be superior for certain kind of tasks. It has been said that they are very efficient at specific queries, while they are slower at more general queries.


• Most of the object databases also support some sort of query language when the need arises


• Some even fully support SQL, but I have no idea how this works in practice.

The cons

• Practical knowledge of these are still fairly limited


• Hard to access from other parts of the company network, for reporting purposes for instance

One approach some of my colleagues took was to go through the whole development period with an object database before converting to a NHibernate solution before going live. The conversion into a relational database was only done because of company demands. This was still a success, but if you do this you have the danger of not quite knowing how long it will take to set up the final solution or exactly how the data will perform.

I look forward to testing this in a real world project, you should too.

Conlusive thoughts

In summary I’d say that you should

• Use the table data gateway/dataset approach if you have a short and sweet application


• Use Active Record if you need to get results fairly quickly, and have a close relationship between your table and domain structure


• Find out whether a commercial/open source OR-Mapper or code generation tool or an object database suits your needs best - if you need a somewhat complex application


• Build your own OR-mapper if you’re forced to, it’s not good for the project, but you’re lucky :)

Replace strings for property names with type-safe version

At times you have the misfortune of having to write property/field (I’ll just call it properties from now on) names as strings, for instance in advanced NHibernate queries. (In general, NHibernate Query Generator avoids this for you in most cases, but there are still some times it cannot be used. Say inheritance for instance). If you’re a bit slow on progressing in mocking tools, this might be a problem as well (It shouldn’t still be though…)

The big problem with non-type safe strings is refactoring, which should happen all the time on your projects. Property names will change, and when these are used as non-typesafe strings in your system it has a few unfortunate consequences:

• You could break something without knowing it. Worst case is you won’t find it until production


• The knowledge above could restrain you from doing refactorings.


• You will have to spend time doing text-searches in the system to find out if and where it is used.

So for a personal project I’m currently playing with, I figured I wanted to do something about it. So I created a small generator project which generates static classes where you have access to the domain objects property names in a type safe manner. What this does for me is:

• If I do changes to a property of a domain object, and that has been used as a (previously) non-type safe identifier, my compiler will complain.


• I can do refactorings all the time without worry.

For now it looks for instance like this in use:



This will return “Text”, which is the name of a property in the Resource class or a base class.

I’ll be the first to admit that this is not ideal. Property names as strings are not something you want to deal with, but at times you have no choice, and this feels like the better of the two options.

For those interested in how I chose to solve it:
The generator takes in an input path, a search string and an output path. This is put in a bat-file which is called from the domain projects build. Currently I’ve only implemented a parser for NHibernate mapping files (HBM), which for each class creates a file with each property listed for that class. (I can see situations where you would want to use properties not just used in persistance, but that’s all I need for now). Inheritance is handled by duplicating the properties of the base class in the subclasses.
In short, the file created for the example above looks like:

Regular Expressions in Visual Studio – the top-down approach

Basically every approach to regular expressions I’ve seen uses the technique of learning all the symbols first, then using that to create various regular expressions. Let’s call that the bottom-up approach. I figured I could add some value by doing it the other way around – by showing specific strategies where regular expressions make sense, showing the regular expressions needed, and then explaining how it works. In other words, the top-down approach.

Once you’ve read this post you should be able to use regular expressions to do pattern matching and extraction in Visual Studio. I’ve kept the number of regular expression meta-characters to a minimum, (hopefully) making it easily understandable as well. You’ll be far from an expert on regular expressions after reading this, but hopefully I can either help get you interested in regular expressions (You should!) or learn some tips about using it with Visual Studio.

I’ve tried to divide the contents into logical sections, so just skip whatever doesn’t sound useful.

On Visual Studio

The built in regular expression support in Visual Studio (2005) is fairly strange – it doesn’t quite follow conventional regular expression syntax, it doesn’t follow the .NET Framework syntax, and irregular behavior has been found. It’s not better that it seems extremely slow once you give it fairly large files either.

Besides its drawbacks, it has become the place I use Regular Expressions the most. And trust me, I like using Regular Expressions. So why use Visual Studio? The probable reason is that I use it on a day to day basis, and since it’s not uncommon that I use it for searching, I guess it just became natural to start using it for other tasks as well.

Oh, and by the way: There’s lots of regular expression software out there you should give a try. Free, good software, with somewhat more standards-based meta-characters, probably faster, likely to have more functionality related to regular expressions, and various other improvements.

Enabling Regular Expressions

• Regular expressions are used from the “Find and Replace” dialog


• ”Match case” also applies to your regular expressions


• The "Find in files" "Find and Replace" dialog (ctrl-shift-f) enables you to search only certain file types


• The "Find in files" "Find and Replace" dialog shows all results in the Find window, instead of just finding the next, as with the "Quick Find" dialog.


• The most used regular expression meta-characters are available in the button next to find once you enable regular expressions.

I’ll add the actual dialog Visual Studio displays here just for reference. It describes each meta-character fairly well.

Regular Expression Strategies

General

Ever after learning regular expressions I seem to always find good places to use them. Here’s the list of strategies(or common situations) I’ve covered in this post:

1. Extract content from lines with a common pattern


2. Remove empty lines (optionally with whitespace)


3. Remove lines not following the pattern you are looking for


4. Add characters around each line


5. Retrieve the contents of all lines containing a pattern

Regex strategy #1 - Extract content from lines with a common pattern

Since this is the first example, I’ll walk through this one extra slowly

Description: You have a number of lines of contents that have some sort of similar text you want to extract.

Example: We have two lines of text, and want to extract the value in the target part of the xml.

<xml target="test" />
<xml info="" target="test2"/ >

Steps:

1. Identify a unique string in the lines straight before the content you want to extract
   
   
   • target="


2. Identify a unique string or character on the other side of the content


• "


3. The major part of the job is done; just create the regular expression to use the two first points :)

The regular expression we begin with to match the parts we identified above (Don’t test it yet, I’m skipping some important details still):

• target=" [^"]*"

Splitting this up we get three parts: target=", and " are just plain text matching. [^"]* is different though. If we take a look at the picture above, we can recognize two meta-characters:

• [^] – Any one character not in the set


• * - Zero or more

So [^"]* means match a character which is not a ", zero or more times. All regular expressions are greedy by default, so it will try to match as many characters as possible.

To try to show this clearly I will differentiate the different parts of the regex and what it will match by using font styles.
The regex:

• target=" [^"]*"

Will match:

<xml target="test" />
<xml info="" target=”test2"/ >

We’re not quite there yet. First of all we need to escape all non-letter and non-number characters with a \ to make sure they are interpreted as plain text characters. I didn’t do that above as it would have made it harder to understand at first. We get:

• target\=\"[^\"]*\"

Harder to read, but necessary. You’ll get used to it.

We’re now able to match the text we want in both lines. However, we’re still unable to retrieve the information from the fields. To do that we first need to match the entire line, so that we can remove the parts we don’t want. To do that we add .* to the beginning and end of the expression:

• .*target\=\"[^\”]*\".*

The . (dot) is a special character which is interpreted as "Any single character" (Except line break). The regular expression will then read:

• .* - Match as many characters as possible (Up until the (last) target\=\" part of the text


• target\=\" - Match the plain text target=",


• [^\"]* - Match as many characters as possible until we reach a "-character


• " - Match the "-character.


• .* - Match as many characters as possible. (Since this is the last one, it means to the end of the line)

With the above expression we match the entire line. The final part we need is a way of extracting the information we want. We’ll do that by adding { and }. This is basically a grouping construct, and you can have any number of them in your expression.

• .*target\=\"{[^\"]*}\".*

We now have the entire regular expression we need! Let’s test it. Copy the example xml text into a text document and open it in Visual Studio. Open a "Find and Replace" dialog, and add the regular expression. Then add \1 in the "Replace with: " part. \1 means the part you have between your first { and }. In other words:

• Find: .*target\=\"{[^\"]*}\".*


• Replace: \1

The result is:

test
test2

Success! :)

Final comments:
Interested in how the regular expression engine actually does the matching? Let’s see what it matches for each part of the expression, until we reach the final match.

• .* - Match as many characters as possible. This will actually match the entire line. Since the .-character doesn’t match a line break it stops at the end of the line (in other programs with other options, it is possible to make it match line breaks as well.) Remember, it is greedy, so it wants to match as much as possible.


• target\=\" - Match the plain text: target=",. Now, to be able to fulfill this requirement, it has to "let go" of some of the matched characters. So it let’s go of one and one character until it finds that it can match the string. That’s why, if there had been several target="-parts in the line, it would have matched the last one.


• [^\"]* - Match as many characters as possible until we reach a "-character. Match one and one character, until it reaches a "-character.


• " - Match the "-character.


• .* - Match as many characters as possible.

In fact, different regular expression engines do the matching in a different way, but this is all you need to know to understand how it works.

Regex strategy #2 - Remove empty lines (optionally with whitespace)

Description: You have a document with several empty lines you want to remove.

Example: : We want to get rid of the empty line between the two words.

SomeText


SomeOtherText

Steps:

1. All you really need is the following:
   
   
   
   • Find: ^:b*$\n
   
   
   • Replace:

The meta-characters used here means:

• ^ - Beginning of line


• $ - End of line


• :b – Space or tab


• \n – Line break

When splitting the regular expression into parts it can be read like this:

• ^ - The match must start from the beginning of the line


• :b* - It will be followed by zero or more (as many as possible) whitespace characters.


• $ - The match must end at the end of the line. In other words, the line must contain either no characters or only whitespace characters to give a match.


• \n – Finally we match the line break for the line as well.

This regular expression will match empty lines, including the line break. By replacing them with nothing you get the effect of removing the lines.

Final comments:
Make sure you have a line break after your last line, or it won’t match the regular expression, as the regular expression requires a line break. Optionally you could have appended a * to the \n, making the expression *:b*$\n*. This would have removed the need for the final line break.

Using Excel to sort the data is a good alternative to remove empty lines. That won’t work if you don’t want the data sorted of course.


 

Regex strategy #3 - Remove lines not following the pattern you are looking for

Description: With the apparent weakness of regular expression matching in Visual Studio, in that you need to replace the entire line to retain the information you want (instead of getting the matches in a separate window), it is not uncommon to get into the situation where you have lines that you need to remove to be able to focus on the lines containing the information you want.

Example: We got some more XML, this time with comments we want to get rid of:

<!—xml comment -->
<element>
<!—another comment -->
<subelement />

Steps:
The general strategy here is to do a (preferably) two-step action to

• Mark the lines you don’t want with a special identifier


• Remove all lines with the special identifier

The actual steps:

1. Identify a unique string or recurring pattern in the lines you want to remove
   
   • <!


2. Write the regular expression to identify and then add the identifier to the line
   
   • Find: ^.*\<\!
   
   
   • Replace: #\0


3. Remove the lines containing the special identifier
   
   
   
   • Find: ^\#.*$\n
   
   
   • Replace:

With point 2, you want to single out all the lines you don’t need (Preferably in one operation, but not always possible). If you have a unique identifier across the entire document, then where you match in the string and add the identifier is unimportant. Often, placing it in the beginning or end of the line is a good starting point.

The first find operation (^.*\<\!) should be easy to read now, as there are no new characters. But for the sake of it: Match start of line, match as many characters as possible until we match the last set of <! in the line.

The replace operation is slightly different. # is the special unique identifier we have used here. It has no special meaning – you could have used an x, or three x-es for that matter, as long as it is unique for the entire line or position throughout the document. Whereas \1 meant the first specified match we found (marked by { and } on each side), \0 holds a copy of the entire line that had a match in it. In effect this means that we simply add a # to the beginning of each line matched.

With point 3, we want to remove all lines with the identifier. So we find each line with the special identifier in the beginning, match the rest of the line with .*$\n, just as before, and then replace it with an empty string.

Final comments:
Another possibility is matching the lines you want to keep, tagging them with the special identifier, and then removing the lines not containing the identifier. Of course you will have to remove the special identifier in the end, so there’s one more replace involved. Not a big thing though.

A third possibility is to use strategy #5 – Retrieve the contents of all lines containing a pattern. This is most useful when you need to match across several documents.

 

Regex strategy #4 – Add characters around each line

Description: At times I seem to end up in the situation where I have lines of information that I need to surround with information or characters.

Example: We have a number of lines of information which need to be used in an SQL in-query as strings, and thus need the necessary surrounding characters. The SQL query is: SELECT * FROM something WHERE name in (…). For those unfamiliar with SQL syntax, we want to add a ‘ to the left of the expression and ‘, on the right side. The information:

Test1
Test2
Test3

Steps:
As long as none of the lines involved already has the necessary characters and no characters that will invalidate the statement, all we need to do is:

• Find: ^.*$


• Replace: ‘\0’,

The find simply matches everything on each line, from start to end character.
The replace puts our plain text characters on each side of the expression. The result is:

‘Test’,
‘Test2’,
‘Test3’,

You’ll need to remove the last ,-character yourself, and paste it into the SQL query, making it:

SELECT * FROM something WHERE name in (‘Test’,
‘Test2’,
‘Test3’)

Final comments:
Another common usage is creating SQL inserts. The logic is just the same; just add some other information around.

Regex strategy #5 - Retrieve the contents of all lines containing a pattern

Description: You want to get the contents of all lines matching a certain pattern, either because the lines make sense for themselves, or because you want to do further work on the lines. This can of course be for one or multiple documents.

This is an alternative to using strategy #3 (Removing lines not following the pattern you are looking for). Strategy #3 is probably faster when you work on a single document.

Example: Reusing a previous example contents. This time we want to retrieve all of our XML-comments:

<!--xml comment -->
<element>
<!--another comment -->
<subelement />

Steps:

1. The regular expression needed to find the correct lines aren’t really the important part here, as what we wan’t to show is how we go from there. We don’t even need to use a regular expression in this case, just search for <!--. The important part here is to do this search in the “Find in Files”-dialog (ctrl-shift-f). The results listed should be:
   
   
   • C:\aPath\file.txt(1): <!--xml comment -->
   
   
   • C:\aPath\file.txt(3): <!--another comment -->


2. Now copy the results from the “Find Results”-window and paste them into a document.


3. We need to remove the path information. Use the following:
   
   
   • Find: ^[^\:]*.[^\:]*.
     
     
     • ^ - The match must start from the beginning of the line
     
     
     • [^\:]* - Match zero or more (as many as possible) non-:-characters.
     
     
     • . - Match the :-character
     
     
     • [^\:]*- Match zero or more (as many as possible) non-:-characters.
     
     
     • . - Match the :-character
   
   
   • Replace:


4. Now we remain with only the lines we’re interested in.

Where do you go from here?

If you really want to learn about regular expressions, you’ll need to get Mastering Regular Expressions (Jeffrey Friedl). The book is THE book on regular expressions.

If you for some reason don’t want to get the book, a series of 10 videocasts from Zain Naboulsi (Is this thing on) should be your second choice. Besides being well made, it focuses only on .NET, which Mastering Regular Expressions only does partly.

IoC Containers generally, Castle Windsor specifically

First I’ll have a go at reinventing the wheel by trying to give my own definition of an IoC Container

An Inversion of Control (IoC) Container is a non-invasive configurable intelligent factory-component

Splitting this definition in parts, we get

• Factory, because it is responsible for creating objects for you.


• Intelligent, because it understands what dependencies you have, and create them for you recursively.


• Configurable, because you can configure the usage through code or configuration files.


• Non-invasive, because the objects used doesn’t need to know about the container.

An IoC Container really makes the Dependency Injection pattern a lot better. Why? I’ll look more into that throughout this post.

My previous post is a foundation for this one, as it explains the Dependency Injection pattern, and gives it a place in the world. I’ll use no time on that here. This post exists to build on that content, and show what more benefits you can achieve by using the dependency injection pattern with an IoC Container. This isn’t meant to be a comprehensive introduction to IoC Containers or Castle Windsor, but should give you a good idea of what it can do for you. Once you’ve read it, just continue on with what I’ve added in “References and further reading”.

I’ll use a simple example to help explain how it works. We have a client, which uses a service, which again depends on IStorageObject, which have two specific implementations. This is not meant as a real world example, but bear with me. The dependencies and sample implementation are shown below:












My previous post shows how we can handle this manually, and the pros and cons with the different solutions. We would like to get an instance of client, but we need to handle all the dependencies it has as well. If we have to write this manually, it will turn out as something like this:



This doesn’t look too bad, and the dependency injection approach is often better than a lot of alternatives. But think of it. As soon as this starts expanding, I can bet you that you’ll have a mouthful.

Now to do the same with an IoC Container, you can typically write:



For the container to create Client, it needs to create an instance of Client, but also an instance of both IService and IStorageObject. The container will understand that it needs to create both, and does so depending on how you have configured it. More on that later.

What if we add another dependency to the service implementation? The manual code would have to change to something like this (Each and every place service is created in the code. Resharper could help you quite a bit, but it’s far from a solution to the real issue.):



Already guessed how it would look with the container approach?



You are right, just the same :)


This has a number of advantages:

• Easy to swap the implementation of an interface - just change the configuration.


• If you find that Client suddenly needs another dependency, you don’t have to rewrite any of the Client creation code.


• Removes explicit dependency-handling throughout the system


• Single point to find out how objects are instantiated and connected


• Easy to begin using Client in other places. (Less boilerplate code to write)


• Additional benefits of tunneling dependencies through one source: Decorators, Interceptors, Proxies, as well as handling object lifestyle. Have a look here for a very good example.


• See my previous post for more on the general benefits of dependency injection.

I am going to have a look at an IoC Container called Castle Windsor now. There are many other alternatives as well. I can’t speak too much about the others as I only have real life experience with this. If we talk only about the general dependency injection problem/solutions, the different projects have a lot of similarities, with some differences in support for config vs code, need for attributes etc.

Configuration and use - with XML

You can use a separate configuration file, or add the castle definitions to your app/web.config.



The configuration above states:

• The string identifier of our only component is “storageobject”


• The service (or interface) the container will create (if any - you don't need to specify one), is IStorageObject, from TheNamespace namespace


• The specific implementation of this interface is DatabaseObject.

In practice, this means that if the container is ever asked for an implementation of “storageobject” or the IStorageObject interface, directly or through resolving dependencies, it will return an instance of DatabaseObject..



This code simply shows how the container would be started with this approach.

Configuration and use - with code

You can define everything in code:




The pros and cons of using XML versus code is that you can do changes without recompiling with the XML approach, while it is type safe when you specify it in code. Use whatever is best for your situation.

Lifestyle

Advanced handling of the lifestyle of objects has never been so easy. By adding a keyword, you can specify how your object lifestyle will be:



The previous configuration updated with specification of lifestyle:

Facilities

Adds integration with other tools

• NHibernate


• ActiveRecord


• WCF integration


• Remoting


• Logging


• Startable


• Custom


• +++

Other

Castle has a number of other possibilities as well, like specifying what to do when objects are created and destroyed (Commision and Decomission).

Note

The last couple of points feels kind of pointless without giving them much extra attention, but I'll leave that to other people who have described it much better. A good starting point would be this or this.

References and further reading

Inversion of Control and Dependency Injection with Castle Windsor Container - Part I (Simone Busoli)

Inversion of Control and Dependency Inject: Working with Windsor Container (Ayende)

A not so interesting post, with much more interesting comments.

Castle Windsor tutorials (BitterCoder's Wiki)

Inversion of Control Containers and the Dependency Injection pattern (Martin Fowler)

Castle Project

Conclusive thoughts

Although a short introduction, I hope I have captured enough information to get you interested. I’ve been using Castle for quite some time on my last project, and I am very happy with the results. I advise you to give it a try!

_