Information is king

Ønsker du å jobbe med meg?

2011-04-03T23:23:00.007+02:00

Har du krevende utfordringer, og jobber i et firma eller skal sette sammen et prosjekt, med behov for en engasjert, positiv og konkurranseinnstilt tech lead/utvikler/arkitekt? Som forventer og stiller krav, som kontinuerlig jobber mot forbedring, og som vil utvikle seg innen både teknologi og ledelse? Og som ikke minst alltid har et sterkt fokus på målet?

Kort oppsummert har jeg

Prosjekterfaring fra alle tekniske hovedområder av et IT-system. Dette er oppnådd gjennom alt fra korte utviklings- og rådgivningsoppdrag, til to og et halvt år på et større forsikringssystem.

Organisasjonserfaring gjennom halvannet år som fagsjef i Objectware/Itera Consulting med arbeidsoppgaver rundt fagledelse, fagplanlegging, arrangering av fagdager og -kvelder, intervjuer, tilbudsarbeid og kundemøter.

Ekstern erfaring som tidligere leder og nåværende styremedlem av NNUG Oslo, samt foredragsholder hos NNUG, ved XP- og Smidig-konferansene, og på årets kommende Norwegian Developer Conference (NDC).

Etter over fire meget lærerike år i Objectware/Itera Consulting har jeg nå tatt en beslutning om å bytte arbeidsgiver. Tiden i Itera er noe jeg ser svært positivt tilbake på. Jeg har fått jobbe med utrolig dyktige kolleger, fått rikelig med utfordringer internt og i prosjekter, og i det hele tatt lært veldig mye.

For meg er det svært viktig å fortsette utviklingen. Jeg tror det gjøres best gjennom krevende utfordringer, og har du det så håper jeg du tar kontakt. Ønsker du å se mer detaljer om mine prosjekterfaringer, så er det tilgjengelig i min CV (pdf). I tillegg forsøker følgende bloggpost å gi et innblikk i hva jeg mener er viktig i IT-prosjekter, vel og merke fra et høyt perspektiv: link.

Rent praktisk er jeg i oppsigelsestid frem til 31 mai. Jeg satser derfor på at noe av det første jeg gjør for ny arbeidsgiver blir å gå på scenen på NDC i starten av juni. Ønsker du å ta nærmere kontakt er jeg tilgjengelig på mail: rune...@gmail.com, telefon: se CV, LinkedIn og Twitter.

Quality in software – a summary

2011-03-20T19:06:00.002+01:00

Who should read this: Software developers or other participants in IT projects who want a high level overview of what is needed to make projects succeed, with a technical bias.

All work in a project must be done to increase the likelihood of success. The process imposed, the roles involved, the people chosen and the technologies used should all work towards this goal.

For the record, I apologize for simplifying, not covering and simply forgetting important areas. I am trying to give you the pure and simple truth, however:

The pure and simple truth is rarely pure and never simple.

- Oscar Wilde

Process

IT projects are complex. The Standish Group’s Chaos Report has attempted to cover this over many years (2009 report in pdf). To handle the complexities of software, the industry appears to move towards streamlining project processes in increasingly agile ways. A common denominator for this work is the well known manifesto for Agile Software Development devised in early 2001:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on
the right, we value the items on the left more.

As anyone involved in real projects know, you need to adapt to changes and challenges, deliver and receive feedback on a variety of levels, and continuously work to reduce waste. You’ll never know everything in advance, and unexpected changes will occur.

The process within IT consists of both the main methodology used and the more technical practices followed.

Overall process

The industry today is steadily working towards being agile, solving problems through iterations and improving continuously. From a developers perspective, this is usually something you will have to accept. Even if it’s not your responsibility, it doesn’t mean you couldn’t or shouldn’t influence the process. Being part of a project team means you have a responsibility for the project’s success. Communicate your concerns. Help inform your coworkers and management.

Within agile there are many different processes; Lean, Scrum, Kanban, and many more. Which is the right one? The choice depends on experience, your organization and the challenge ahead. But never forget: mix and match what works for you - being agile certainly apply to both the product created and the process used. (And be wary of certified [process]-masters. Are you sure it means what you think it does?)

Whenever you care about the results of what you are doing, skilled people are a must. But even the brightest people fail because of misunderstandings or general problems with communication. One very import point to focus on is to improve the feedback loop. Have the customer and business experts work close with you throughout the process, and increase the frequency of releases tenfold (At least to test environments). Training on and delivering new releases reduces the risks involved, increases the feedback from users, and enables you to deliver the right functionality at the right time.

Before moving on, I thought it proper to address a common misconception with agile - that agile equals little or no planning. I have to insist that this is a false claim, and the reality is quite the opposite. If you want to learn more, this is a good article explaining the importance of it, and at what levels planning should occur.

Technical process

The technical process is pretty much anything that directly relates to what is produced; estimation, tasks, quality assurance, and releasing new versions.

In project and sprint planning, estimation is usually considered an important practice. Estimation is useful to consider what is involved in producing a feature, visualizing the cost of implementing it, and ensuring that everyone has a similar understanding of the scope. How estimation, planning and tasks are handled varies, and you should experiment with what method works for you. Do ensure that you work to build transparency and trust, as described in this quote from Gabrielle Benefield, Director of Agile Product Development in Yahoo!:

Keeping everything transparent, and letting the business know of any changes as they come up, means that the business can adapt quickly to make the best decisions. At my last company I saw us go from a state of permanent chaos, where we had an extremely ambitious roadmap but couldn’t deliver products, to a predictable state where we could genuinely sign up for projects that we could deliver. The business said they might not always like the answers (they always want things tomorrow, after all), but at least they believed our answers and were not frustrated from feeling that they were being consistently lied to.

The quote, and significantly more, can be found in Agile Estimating and Planning by Mike Cohn.

There will often be doubt when planning something; which technology is right, what main flow of logic is appropriate, and so on. Don’t be afraid to use prototyping to verify assumptions. Creating an end to end working example and/or ensuring that the toughest problems can be solved can save you a lot of time in the future.

I mentioned feedback earlier as a key principle. Within the technical process, this should be supported through a few good practices. Code reviews, pair programming and static code analysis are tools useful to get quick feedback on quality. Automated tests are another, with unit, integration and acceptance tests (and sometimes a few more) as common categories. Continuous integration is another practice that helps increase feedback and quality. By automating compilation and running of all tests for each check-in, you add another step for ensuring quality, increasing feedback and helping visualize the current state of the code.

The practices mentioned above are often introduced more from the technical personnel in the project. But to support the overall process need of more frequent releases to test and production, a number of practices are key: automating deployments, including moving code, configuring machine and web servers, handling environment specific configuration settings and setting up the database. It won’t come free of charge, but can bring about great benefits.

A lot more can be said about improving the release and deployment process. I recommend reading the book Continuous Delivery to get a better overview.

Technology

Beyond the general practices, what is important in technical development?

At both higher and lower levels, having a good overview of alternatives is necessary to select the right technology or solution (Never forget the hammer nail analogy). It is impossible to know the details of everything, but learning of qualities plus pros and cons of alternatives enables you to better select the correct one. Subsequent tests might find your initial knowledge about the technology was wrong, but that’s why you use prototyping anyway.

When it comes to code, I am a strong supporter of simplicity and readability. If what you’re creating can’t be understood without hassle, how is anyone going to be able to maintain it? Refactor as your understanding improves or changes occur, or be aware that you will end up with code that in near or distant future will take time to understand, even more energy to change and will undoubtedly create bugs. And automate testing of anything moderately complex. Pretty please?

Beyond mastering the technologies chosen, what more is there? Never forget that your system will go into production one day. Security, logging, monitoring, error handling, integrations, multithreading, scalability, hardware and firewall issues are areas you need to focus on. Now is the time to go ahead and read Release It!.

So much more can be said about technology and technical development, but it is hard to generalize. Find the right people and all will be good with the world.

People and roles

A software project can consist of a variety of people. You can have software developers, UI specialists, database administrators, operations, business experts, testers, project leader, project owner (and a variety of other project something roles).

No matter what roles exist, make sure the communication is frequent between all related parties. If you can co-locate it will help reduce misunderstandings.

Management 3.0 has a lot to say about leading people in IT projects. As the author says

I think that people are the most important parts of an organization and that managers must do all they can to keep people active, creative and motivated.

Converting WCF WSDL to single file (FlatWSDL) - correctly!

2010-08-01T23:28:00.009+02:00

Note! This post is slightly different than most FlatWSDL posts, and fixes the bug with removing the <XS-import>’s.

The team behind WCF chose to split all generated WSDL-definitions into multiple files, one for each namespace, schema, etc. That’s a valid approach, and most modern tools support that out of the box, it’s done correctly after the WSDL standard.

However, not all tools support it. Various blog posts discuss different tools, and in my case it was a mainframe integration that caused the problem. I needed to get WCF to output it as one file. The good news is that’s really simple. You can use either FlatWSDL by Thinktecture or FlatWSDL in WCFExtras (They’re almost identical).

The only problem is that both solutions emit a WSDL file without appropriate <XS:Import>’s for the schemas. Many tools are able to resolve the references without the import, but not all are as forgiving. This post describes a code change to WCFExtras that will fix it, but I’ll add mine here as well.

What you need to do is to download one of the FlatWSDL approaches (This post describes it, as well as a host of others), then change the code in ExportEndpoint in FlatWsdl.cs to what I’ve included below, and you’re good to go.

public void ExportEndpoint(WsdlExporter exporter, WsdlEndpointConversionContext context)
{
    XmlSchemaSet generatedXmlSchemas = exporter.GeneratedXmlSchemas;

    foreach (ServiceDescription generatedWsdl in exporter.GeneratedWsdlDocuments)
    {
    var referencedXmlSchemas = FindAllReferencedXmlSchemasRecursively(generatedWsdl, generatedXmlSchemas);
        ClearWsdlOfExistingSchemas(generatedWsdl);
        AddAllReferencedSchemas(generatedWsdl, referencedXmlSchemas);
    }

    RemoveSchemaLocationFromXmlSchemaImports(exporter, generatedXmlSchemas);
}

private static IEnumerable<XmlSchema> FindAllReferencedXmlSchemasRecursively(ServiceDescription wsdl, XmlSchemaSet generatedXmlSchemas)
{
    var referencedXmlSchemas = new List<XmlSchema>();
    foreach (XmlSchema schema in wsdl.Types.Schemas)
    {
        AddReferencedXmlSchemasRecursively(schema, generatedXmlSchemas, referencedXmlSchemas);
    }
    return referencedXmlSchemas;
}

/// <summary>
/// Recursively extract all the list of imported
/// schemas
/// </summary>
/// <param name="schema">Schema to examine</param>
/// <param name="generatedXmlSchemas">SchemaSet with all referenced schemas</param>
/// <param name="referencedXmlSchemas">List of all referenced schemas</param>
private static void AddReferencedXmlSchemasRecursively(
   XmlSchema schema,
   XmlSchemaSet generatedXmlSchemas,
   List<XmlSchema> referencedXmlSchemas
   )
{
    foreach (XmlSchemaImport import in schema.Includes)
    {
        ICollection realSchemas = generatedXmlSchemas.Schemas(import.Namespace);
        foreach (XmlSchema ixsd in realSchemas)
        {
            if (!referencedXmlSchemas.Contains(ixsd))
            {
                referencedXmlSchemas.Add(ixsd);
                AddReferencedXmlSchemasRecursively(ixsd, generatedXmlSchemas, referencedXmlSchemas);
            }
        }
    }
}

private static void ClearWsdlOfExistingSchemas(ServiceDescription wsdl)
{
    wsdl.Types.Schemas.Clear();
}

private static void AddAllReferencedSchemas(ServiceDescription wsdl, IEnumerable<XmlSchema> referencedXmlSchemas)
{
    foreach (XmlSchema schema in referencedXmlSchemas)
    {
        wsdl.Types.Schemas.Add(schema);
    }
}

private static void RemoveSchemaLocationFromXmlSchemaImports(WsdlExporter exporter, XmlSchemaSet schemaSet)
{
    var mySchemaSet = new XmlSchemaSet();
    mySchemaSet.Add(schemaSet);
    foreach (XmlSchema schema in mySchemaSet.Schemas())
    {
        exporter.GeneratedXmlSchemas.Remove(schema);
    }
}

Generating sample data for objects using AutoFixture, extending it with collections support, and using it as Silverlight sample data generator

2010-07-28T18:00:00.005+02:00

This blog post consists of three parts

- On generating sample data in general

- Extending AutoFixture with automatic collections generation support

- Adding a sample data generator in Silverlight for Visual Studio/Blend screen previews

[Note: Windows Live Writer did a horrible job in transferring the post to the blog. I've fixed quite a bit on the formatting, but it's still in poor quality. Apologies for that. Let me know if you want the code samples without the formatting]

Generating sample data in general

Creating objects with sample data is a challenge developers often need to handle. The most common use case is for testing purposes, where you need an object with various values set. Some might be important, but many just need to be set to anything (Also called an anonymous variable).

I’ve seen and used various solutions for this. Often you see it either done manually, by an object mother, with test data builders, or by loading existing objects from a database. Many people start out writing a simple reflection tool to handle it, but stops a few hours down the road once the actual complexity involved becomes apparent.

I had a new use case for it this time, and wanted to avoid many of the problems with the solutions above. After searching for and trying various reflection based tools, I found AutoFixture, and was impressed from the get-go. It pretty much does exactly what you’d expect, and has a clean interface as well. Some examples:

- Creating an anonymous string

     var anonymousText = fixture.CreateAnonymous<string>();

Result:

- anonymousText: aa48c714-6c6a-4ac4-9c27-3658c9e78d5f

- Creating an anonymous object

public class Person
{        public string Name { get; set; }        public int Age { get; set; }    }
..
var personWithAnonymousData = new Fixture().CreateAnonymous<Person>();

Result:

- Name: Nameb7c2a9fc-a5b0-4836-83d9-be1b057e0ff1

- Age: 1

Take a look a the AutoFixture cheat sheat for a number of good examples. There’s many things you can do to customize the behavior and output.

That covers the introduction to AutoFixture. If you need a sample data creator, for test data or other purposes, I advice you to check it out. Over to the next point.

Extending AutoFixture with automatic collections generation support

There was one feature I was missing in AutoFixture that I really wanted. If an object has a collection (, list, or any other sort of .NET collection) of something, that collection will only be initialized to an empty collection automatically. If I do a change to the Person object and rerun, the PhoneNumbers list below will be an empty list of strings.

public  class Person    {        public string Name { get; set; }        public int Age { get; set; }        public List<string> PhoneNumbers { get; set; }    }

AutoFixture have functionality to handle this. If you run the line below first..

fixture.Register(() => fixture.CreateMany<string>().ToList());

..then the output of PhoneNumbers will be something similar to:

[0]: 635bc212-8d16-4029-8423-1f45016fe020

[1]: fcbed54b-94f6-4eea-b773-e7ec5e75b6dd

[2]: 814c120a-7e9c-487a-b99e-9e23d9d511b0

However, I would like this to be handled automatically. Doing it the AutoFixture-way you’ll need to know the internal structure of a class to define that it should handle a certain collection. Another problem is collection interfaces. If PhoneNumbers was defined as IList<string>, I would be forced to define it since it causes a runtime exception of:

“AutoFixture was unable to create an instance from System.Collections.Generic.IList`1[System.String], most likely because it has no public constructor.”

So I set out to see if I could extend AutoFixture to handle collections automatically. Seemed like a fun enough way of getting to know AutoFixture better as well as some good reflection fun. The requirements I set was to handle generic collections and interfaces to collections automatically.

First how to use it (Only the second line is new):

var fixture = new Fixture();    fixture.Customizations.Add(new CollectionsGenerator(fixture));    var personWithAnonymousData = fixture.CreateAnonymous<Person>();

The new CollectionsGenerator:

/// <summary>       2: /// Support for handling automatic generation of anonymous data for collections       3: /// </summary>       4: internal class CollectionsGenerator : ISpecimenBuilder       5: {       6:     private readonly Fixture _parentFixture;       7:         8:     /// <param name="parentFixture">Requires parentFixture to ensure existing options are used.</param>       9:     public CollectionsGenerator(Fixture parentFixture)      10:     {      11:         _parentFixture = parentFixture;      12:     }      13:        14:     public object Create(object request, ISpecimenContext context)      15:     {      16:         if (!ReflectionHelper.IsGenericDotNetCollectionOrInterface(request))      17:             return new NoSpecimen(request);      18:        19:         Type collectionType;      20:         if (ReflectionHelper.ObjectIsADotNetCollectionInterface(request))      21:             collectionType = ReflectionHelper.GetConcreteCollectionTypeToMatchInterface(request);      22:         else          23:             collectionType = ReflectionHelper.GetUnderlyingSystemType(request);  24:                        25:         var returnCollection = (IList)Activator.CreateInstance(collectionType);
 
        AddAnonymousValuesToCollection(returnCollection, _parentFixture);
        
        return returnCollection;
    }      31:  
    private static void AddAnonymousValuesToCollection(IList collection, Fixture parentFixture)      33:     {
        Type genericType = collection.GetType().GetGenericArguments()[0];     35:         var createAnonymousMethod = typeof(SpecimenFactory).GetMethod("CreateAnonymous", new Type[] { typeof(ISpecimenBuilderComposer) }).MakeGenericMethod(new[] { genericType });     36:         for (int i = 0; i < parentFixture.RepeatCount; i++)     37:         {     38:             collection.Add(createAnonymousMethod.Invoke(null, new ISpecimenBuilderComposer[] { parentFixture }));     39:         }     40:     }     41: }

And an accompanying ReflectionHelper:

   1: public static class ReflectionHelper      2: {      3:     private const string UnderlyingSystemTypeString = "UnderlyingSystemType";      4:        5:     public static bool CanRetrieveUnderlyingSystemTypeFromObject(object input)      6:     {      7:         return (input != null)      8:                && (input.GetType().GetProperty(UnderlyingSystemTypeString) != null)      9:                && (input.GetType().GetProperty(UnderlyingSystemTypeString).GetValue(input, null) != null);     10:     }     11:       12:     public static bool InputIsAssignableFrom(object request, Type ofType)     13:     {     14:         return (request != null)     15:                && (request.GetType().GetProperty(UnderlyingSystemTypeString) != null)     16:                && (request.GetType().GetProperty(UnderlyingSystemTypeString).GetValue(request, null) != null)     17:                && (ofType.IsAssignableFrom((Type)request.GetType().GetProperty(UnderlyingSystemTypeString).GetValue(request, null)));     18:     }     19:       20:     public static Type GetUnderlyingSystemType(object input)     21:     {     22:         return (Type)input.GetType().GetProperty(UnderlyingSystemTypeString).GetValue(input, null);     23:     }     24:       25:     public static bool ObjectHasGenericTypeSpecified(object input)     26:     {     27:         return GetUnderlyingSystemType(input).IsGenericType && GetUnderlyingSystemType(input).GetGenericArguments().Length > 0;     28:     }     29:       30:     public static bool IsGenericDotNetCollectionOrInterface(object request)     31:     {     32:         if (!CanRetrieveUnderlyingSystemTypeFromObject(request))     33:             return false;     34:       35:         return    (ObjectIsADotNetCollection(request) || ObjectIsADotNetCollectionInterface(request))      36:                && (ObjectHasGenericTypeSpecified(request));     37:     }     38:       39:     public static bool ObjectIsADotNetCollection(object request)     40:     {     41:         return InputIsAssignableFrom(request, typeof(IList));     42:     }     43:       44:     public static bool ObjectIsADotNetCollectionInterface(object request)     45:     {     46:         var objectTypeName = GetUnderlyingSystemType(request).ToString();     47:       48:         var dotNetCollectionTypes = new List<string> //.NET Collections     49:                     {     50:                     "System.Collections.Generic.IList",     51:                     "System.Collections.Generic.IEnumerable",     52:                     "System.Collections.Generic.IEnumerator",     53:                     "System.Collections.Generic.ICollection",     54:                     "System.Collections.Generic.ISet",     55:                     "System.Collections.IList",     56:                     "System.Collections.IEnumerable",     57:                     "System.Collections.IEnumerator",  58:                     "System.Collections.ICollection",      59:                     };     60:              61:         return dotNetCollectionTypes.Any(objectTypeName.Contains);     62:     }     63:       64:     public static Type GetConcreteListTypeToMatchInterface(object request)     65:     {     66:         Type genericType = GetUnderlyingSystemType(request).GetGenericArguments()[0];     67:       68:         string genericListTypeName = "System.Collections.Generic.List`1"     69:                                      + "[[" + genericType.AssemblyQualifiedName + "]]"     70:                                      + ","     71:                                      + Type.GetType("System.Collections.IList").Assembly.FullName;     72:         return Type.GetType(genericListTypeName);     73:     }     74: }

Note that this isn’t release quality. There’s a few issues, like with recursion if a complex type contains an instance of itself, and the way .NET collections are identified.

Adding a sample data generator in Silverlight for Visual Studio/Blend screen previews

The actual use case I was trying to solve was creating automatic sample data for XAML pages in Silverlight. This would enable me to better see how pages looked in Visual Studio and Blend, and a way of testing data bindings as well.

Note that AutoFixture doesn’t support Silverlight just yet, but there is a fork available that supports it (If you want to use a newer version, you only need to do a couple of changes to trunk to make it work).

I wanted a more natural interface for the sample data, so I created a SampleDataGenerator class:

   1: /// <summary>      2: /// Creates sample data for object or collection of objects recursively      3: /// </summary>      4: public static class SampleDataGenerator      5: {      6:     public static T GenerateObjectWithData<T>()      7:     {      8:         return CreateFixtureWithDefaultSetup<T>().CreateAnonymous<T>();      9:     }     10:       11:     public static T GenerateObjectWithData<T>(int collectionItemAmount)     12:     {     13:         var fixture = CreateFixtureWithDefaultSetup<T>();     14:         fixture.RepeatCount = collectionItemAmount; //Object count generated per list     15:         return fixture.CreateAnonymous<T>();     16:     }     17:       18:     public static IEnumerable<T> CreateMany<T>()     19:     {     20:         return CreateFixtureWithDefaultSetup<T>().CreateMany<T>();     21:     }     22:     public static IEnumerable<T> CreateMany<T>(T seed)     23:     {     24:         return CreateFixtureWithDefaultSetup<T>().CreateMany(seed);     25:     }     26:       27:     public static IEnumerable<T> CreateMany<T>(int count)     28:     {     29:         return CreateFixtureWithDefaultSetup<T>().CreateMany<T>(count);     30:     }     31:       32:     public static IEnumerable<T> CreateMany<T>(T seed, int count)     33:     {     34:         return CreateFixtureWithDefaultSetup<T>().CreateMany<T>(seed, count);     35:     }     36:       37:     private static Fixture CreateFixtureWithDefaultSetup<T>()     38:     {     39:         var fixture = new Fixture();     40:         fixture.Customizations.Add(new StringGenerator(() => ""));     41:         fixture.Customizations.Add(new CollectionsGenerator(fixture));     42:         return fixture;     43:     }     44: }

At the bottom of the code above I have included use of the CollectionsGenerator class. I also changed the default behavior of StringGenerator, so it only outputs the property name instead of property name + guid.

You can then use it in a sample object that inherits the ViewModel:

                   1: public class EditPersonSampleData : EditPersonViewModel
{
    public EditPersonSampleData()
    {
        Person = SampleDataGenerator.GenerateObjectWithData<Person>();
    }
}

And then in the EditPersonView.xaml, include:

        1: xmlns:vm="clr-namespace:SomeNamespace.EditPerson" 
 
...
 
<Grid x:Name="LayoutRoot" Background="White" d:DataContext="{d:DesignInstance Type=vm:EditPersonSampleData, IsDesignTimeCreatable=True}">
 

That’s all you need to get sample data visible in a XAML viewer. Good way of checking that the interface looks OK and that the binding is set up correctly.

Vote here to get similar support in the actual AutoFixture product.

Changes in software – Different developers over time for a long-running project

2010-01-23T18:32:00.001+01:00

To do changes to software you need to understand it. You need to understand the exact piece of code you are changing, you need an overview of the system to be aware of any dependent parts, and you need to understand the technologies in use. In the real world you can also end up with a few more difficulties. You need to understand why things work the way they do. Will the changes you are imposing break existing logic? Does any clear documentation of how things work exist, or that you'll need to update? Can you trust the documentation that does exist?

Once you have a "long-running project" you get the added difficulty of handling information stored only in a developers or business specialists head. Your best option is to have the same developers working on the project for its lifetime. That can be challenging in terms of getting the right people in on it. When that's not possible you need to make sure you don't end up in a situation where the knowledge is lost.

The state of the actual code is perhaps the most important. Documentation will always be hard to keep up to date, and the best description of what you have is your code. It's essential that it is well structured and readable to make changes possible. Do refactorings when it helps clarify code, or when your understanding of the problem or domain changes. Write small and well named automatic tests that each verify a single piece of behavior and explains why and in what context. For this context, the tests will help clarify how a piece of code works and is intended to be used (Do you know what AAA is? If not, it's about time you do).

Consider increasing the truck factor of your code - meaning, how many people must be hit by a truck before a piece of information is lost. This has a lot to do with how you handle code ownership. Does individuals only make changes in their piece of the code, are there module leaders that supervise changes to parts of the system, or do you have a form of collective code ownership (XP's view on collective ownership: http://www.extremeprogramming.org/rules/collective.html)? Like every part of software development, there is no answer that is correct in every situation, but strong individual ownership is most likely to cause problems in my view. The truck factor is extremely low, a sort of "me versus they" mentality can lead to things falling between two chairs. Few developers knowing a piece of code means fewer can do changes to it if problems arise, and different parts of a system can end up being developed quite differently.

Documentation is another area. Do you document at all? Only high level requirements? Sequence or collaboration diagrams? Is the documentation kept up to date? No business decisions hidden in mail discussions or in developer or business users heads? In meeting minutes? This is such a tough area. I haven't quite made up my mind about this yet - but I do know one thing - keeping documentation truly up to date is extremely challenging. You can certainly try to document everything, but is it worth it in terms of ROI?

When a developer do choose to leave a project, make sure you work to capture as much knowledge as possible. Hopefully most of the knowledge of the developer is already shared with the team, but try to capture the rest as much as possible. It will cost a lot more to try and understand something later on.

Changes in software - One of the biggest challenges of software development

2010-01-17T21:05:00.017+01:00

Changes to software has been a problem since the dawn of computers, and it still is a challenge with every piece of software developed. Perhaps the biggest part of creating software is changing or expanding existing code. So handling changes is something we have to deal with all the time. It's the maintainability aspect of software development. The problem is that there are so many challenges involved, so many things that can go wrong. But do you know what the worst problem is? The fact that a lot of developers have very little focus on the issue in general, including the reasons behind it.

I tried to come up with a few of the top of my head, and listed almost thirty easily (Even without going into nitty gritty coding issues). No wonder it's hard to handle. There's no silver bullet to fix many of these issues, but as with everything else in life - you need to know about it to be able to do something with it.

Here's a number of things that affects and complicates changes in software:

Project

Many different developers over time
Legacy codebase
External system dependencies

Person

Developer with limited understanding of code in general
Developer with limited understanding of the projects code and architecture
Unmotivated developer
Changes done just before end of day/end of week/end of year
Different quality of developers
Misunderstanding the time it takes to write quality code
Not thoroughly understanding a problem

Code

The danger of quick fixes/hacks
Lack of understanding of how software rot
Unreadable code.
High coupling in the system. Making a change in code that is used many places has a higher risk.
Complex architecture (Hard to understand or wrong for the problem at hand)
Duplicate code

Business

Illogical business logic
Unavailable business people

Process

Time (Quality - if time permits)
Bug acceptance
Developer not feeling the pain of fixing production bugs
Task-switching developer
Picking up problems caused by changes
Lack of testing
Big bang releases / seldom releases
Individual code ownership
Refactoring / Not refactoring
Adding additional features at end of test/iteration/project

This post would get way too long if I were to cover all the reasons in detail, so I'm rather going to cover various individual ones in separate posts.

Did you miss something on the list?

Real programmers don't comment their code. If it was hard to write, it should be hard to understand

2010-01-14T23:06:00.001+01:00

Now that's just a plain lie. Real programmers do comment their code - though only certain parts of it.

First of all, an important rule: Your code should be so readable that you don't need comments to understand what it does.

Comments are a problem because they
- might not get get updated when the code changes.
- become an excuse to write unreadable code.

If you can't trust that the comment is correct, you'll have to go into a method to understand what it does. Once you do that you might have to step even further down into other methods, and soon you'll forget what you were doing initially.

Justifying unreadable code with comments is heresy. You should not need to write comments to explain what a piece of code does. What do you need to do if you have code that does need additional comments? Refactor! Often the text you would put in comments can be used as method names instead. Don't forget Joshua Kerievsky's tip in Refactoring to Patterns:

"What is the preferred size of small methods? I would say ten lines of code or fewer, with the majority of your methods using one to five lines of code. If you make the vast majority of a system's methods small, you can have a few methods that are larger, as long as they are simple to understand and don't contain duplication."

Don't be afraid of small methods with a readable signature. You have intellisense to help finish method names, and the performance hit of having many methods is negligible, so none of those arguments hold any value. Does it feel like your classes end up with too many methods? There's a good change they're doing too much. Remember to use the Single Responsibility Principle on both classes and methods - they should be responsible for doing only one thing, and should have one, and only one, reason to change.

When are comments appropriate?
- At major boundaries, for instance with libraries or at service contracts, giving context or similar.
- Clarification or expressing intent, if you fail to do it in code
- Warnings or particular notifications
- Regular expressions, if you have to use them
- TODO's, sparingly.
- When you have no other options.

There is always a need for comments in you code. The rule should be to not use them, but if you have a good reason - go right ahead.

I think Uncle Bob says it well in Clean Code:

"The proper use of comments is to compensate for our failure to express ourself in code. Note that I used the word failure. I meant it. Comments are always failures. We must have them because we cannot always figure out how to express ourselves without them, but their use is not a cause for celebration. [...] Every time you express yourself in code, you should pat yourself on the back. Every time you write a comment, you should grimace and feel the failure of your ability of expression."

Object Orientation revisited – Business layer patterns

2010-01-10T23:38:00.008+01:00

Object orientation is a fascinating topic. It is one of the core concepts for a major part of developers today, yet the understanding and the inclusion of it in our day to day work varies greatly. Even though many believe they do the same. 'Cause everyone is working object oriented with an object oriented language, right?

We had a day back in December at Objectware where we revisited object orientation, and its implications for our work. The day was structured with some presentations first, a good deal of group work, a reflection on the work, followed by a general discussion in the end. We had a continuous focus on discussions and experience sharing throughout. I started of with a session on general everyone-should-know information, more of a recap or introduction if you will, and then brought the focus to the use of object orientation in structuring our business logic, centralized or delegated design, and so on. A few colleagues of me followed on with topics concerning cohesion and coupling, information hiding and composition vs inheritance.

I thought I'd share some of the content of my presentation here. It's certainly not a new topic, it has been blogged and written about countless times, but still I meet a lot of variation on the knowledge in this area.

First of all - how is this valuable to you? Knowing how to structure your business logic both in various forms of applications, but also varying internally in an application, is paramount in ending up with an application that doesn't fight you continuously. In designing every application we want to:

Create an architecture that meets all technical and operational demands
AND that solves all quality attributes like performance, security and maintainability well

The architect should set standards concerning the business logic layer, but I've seen more than one architecture transform into something very different because of varying knowledge on this. It's always important to know the main differences so you can take informed decisions. This is one of those areas that most software developers should know - especially those that work on applications containing more than a trivial amount of business logic.

Introducing OO

Object orientation is about core principles like abstraction, composition, inheritance, encapsulation, polymorphism and decoupling. Several things can be said about this style of programming, like

”Division of responsibilities for an application or system into individual reusable and self-sufficient objects, each containing the data and the behavior relevant to the object.”

”An objectoriented design views a system as a series of cooperating objects, instead of a set of routines or procedural instructions.”

In general it can be said that Object Orientation should be considered when:

”You want to model your application based on real world objects and actions, or you already have suitable objects and classes that match the design and operational requirements. The object-oriented style is also suitable if you must encapsulate logic and data together in reusable components or you have complex business logic that requires abstraction and dynamic behavior.”

(All quotes from Microsoft Application Architecture Guide, 2nd Edition)

And you continuously work to

find the right abstraction of the real world for your problem
find fitting objects
find the right place to put code
rework the code at all times to make sure it is correct based on your current understanding
favor low coupling
limit duplication
make each object work on one thing only - ensuring high cohesion
work to have a test friendly design
and generally try to keep your code as to-the-point as possible, all the time ensuring that adding further behavior and extending the current functionality is as painless as possible (without over engineering of course :) )
and certainly much more than this..

How you structure your business layer certainly has a say in how well and how easily you can achieve these things.
You traditionally have four main ways of structuring this logic:

Procedurally oriented with Transaction script or Table module.
or
Object oriented with Active record or Domain model.

Let's take a closer look at each..

Transaction script

”Organizes business logic by procedures where each procedure handles a single request from the presentation.”

Martin Fowler, Patterns of Enterprise Application Architecture

Transaction script follows the current structure:

Note that this is an extremely simplified example. Using transaction script you certainly utilize all you know about class design and move logic into where it fits best, the main point I'm trying to make is that you have a central place to control the flow. The CreateOrder method controls what logic will happen from start to end.

If you use this all the way, the business layer will consist of a number of procedures that each implements one action from the user interface. Good design in this context is about minimizing duplication at the same time as meeting all demands. This has nothing to do with a database transaction, but refers to one monolithic logical operation. It is not uncommon to create one business component per database table.

The positive sides with it is that it's easy to understand and maps well to the flow of a user story/use case. But it breaks down on complicated logic, can be hard to test as it does many things, and can lead to duplicated code.
Being popular in general, it the .NET world it was quickly replaced in popularity by....

Table module

”A single instance that handles the business logic for all rows in a database table or view.”

Martin Fowler, Patterns of Enterprise Application Architecture

The Table module is built around a table of data. Its perspective is similar to that of Table script, except the focus is on a group of data. Operations are often designed as methods on an object that represents a table of rows. Because of that you always use a key or index to indicate the row in use. It is procedural, but has more of an object oriented focus than Transaction script.
Note that you don't need to encapsulate the DataSet in a custom class, it is common to just work directly with the DataSet as well.

Table Module quickly became a success in .NET because of the great tool support existing. Putting together a simple read-only application that shows data in grid form literally takes no time because of the focus Microsoft has had on the tooling aspect. Everything from GUI to database works smoothly for this. Generally you can say that RAD applications work great with this approach. And it's a fair compromise if you have little logic and don't need much abstraction over you data model - because there will be a very tight coupling between your data model and every part of the application.

A history lesson

The Table module pattern has many downsides though - even in .NET. If you have complex business logic, it won't cope with that very well. It's very data driven, and doesn't focus on the business side of things. You will end up with an application that is very tightly coupled to the database. It is poor at indicating relationships between objects, and polymorphism isn't exactly available. As long as you keep to the code generation and wizard support available, you'll be all right, but it can get complicated if you can't rely on that.

Microsoft had for many years an almost exclusive focus on this pattern - by their way of action basically saying that you could solve any problem with a dataset. Object bigots found various ways around it, often by creating mappers that converted datasets to objects, by cleverly encapsulating datasets or by creating custom data mappers using the data reader directly. The focus since Visual Basics heyday was on RAD applications, supporting novice developers, and not all that much more (Patterns & Practices shipped Enterprise Library 1.0 in 2005 though). The Microsoft community didn't seem to mature much more either, as there was an almost sole focus on software and tools that Microsoft shipped. From what I have come to understand this made the whole community much less used to object orientation than for instance the Java community became. And that is likely the main reason why the general Microsoft community is still so procedurally oriented still.

Luckily things have changed, both in terms of third party tools and to some extent Microsoft's focus. Tools like NHibernate and the Castle Project led the way with support for good object relational mapping and dependency injection tools, as well as MonoRail as an early MVC framework. Microsoft has supported enterprise applications through the work of Patterns and Practices on Enterprise Library. However, simplicity and enterprise library has never been used in the same sentence. In recent times though, Microsoft begun focusing on object relational mapping through LINQ To SQL and Entity Framework, an own MVC framework, ASP .NET MVC and their own IoC container, Unity, and the continuing support for composite UI applications with PRISM for WPF and Silverlight (Following the Composite UI Application Block for Winforms).

So the tide has to some extent turned in the Microsoft community, certainly the support for the two remaining patterns are continuously improving.

But less take a closer look at the object oriented patterns:

Active Record

”An object that wraps a row in a database table or view, encapsulates the database access, and adds domain logic on that data.”

Martin Fowler, Patterns of Enterprise Application Architecture

The example above uses Castle Active Record. As you can see, the object encapsulates one database row in principle, and the object contains both data and domain logic. It includes logic to directly manipulate the database, like Save or Delete, and also includes static methods that work on the entire collection, like Get, or GetOrdersCount.

The flow of logic is different in Active Record than in the two previous patterns in that the object contains the logic internally, making use of encapsulation and achieving higher cohesion. Instead of having the service dictate how the logic should flow, you ask the object to perform some domain logic, and although not shown here, it typically delegates responsibility to other objects.

Active Record is a useful framework for two reasons: simplicity and tooling support. It's quite easy to understand, and as long as you use frameworks like Castle Active Record or LINQ to SQL, it's also fairly easy to use. For simple object models it works great, and as long as it's OK to have the object model closely mimic the data model, Active Record is a good choice.

There's a few things you should consider before choosing Active Record though. Because of it's close connection to the data model, you have very limited support for designing your object model separate of the database. If you have a need for that, you should skip ahead to the next pattern. It's also a problem that it mixes responsibilities. The objects holds domain data and methods, but in addition you have attributes for mapping to the database, CRUD operations and static methods that work on the entire collection. And it sure isn't Persistence Ignorant (PI).

This is a popular pattern as well, and even though it seemed Microsoft tried to kill (http://bit.ly/8VkYnW) LINQ to SQL to avoid supporting it in addition to Entity Framework (more on this tool later), it's popularity seems to have saved it for now.
If Active Record doesn't quite do it for you, and your complexity is high enough, you should take a closer look at the next pattern:

Domain Model

”An object model of the domain that incorporates both behavior and data”

Martin Fowler, Patterns of Enterprise Application Architecture

The domain model pattern is the one that best supports object orientation. The point is to separate the domain logic into classes that are only concerned with modeling the domain and the corresponding rules well. The model classes should not know how to persist themselves (as in Active Record) and shouldn't be coupled to infrastructure logic. The classes should be POCO (Plan Old CLR Object), enabling a higher abstraction from the data model. This also means that the business logic of the domain model is simple to test, and you can easily get high coverage on this most important area without worrying about infrastructure and the like.

Domain model is the most complex to use, mostly because of the cost relating to mapping to the database (because of the impedance mismatch), and complexity regarding new ways of having to think about disconnected objects, lazy loading, less direct SQL to tweak and converning the next major point of this blogpost, not being familiar with delegated control. This complexity is biggest on first usage, and drops in subsequent projects.

Domain model shouldn't be used in all scenarios. Use it when you have a complex domain with complex business rules and when you want to be able to model the domain model free from database limitations. It is more complex to master than the other patterns, but in the right scenarios has great strengths.

To handle the persistence of your domain model you use an Object Relational Mapper (OR/M). This includes an API for CRUD operations, mapping between the database and the data and domain model and a query model and associated language. The modeling is usually done either via external XML mapping files or some kind of fluent interface, where you state how the model maps to the database. A lot can be said about Object Relational Mappers, but this is not the place. One thing though - various people have objections against using domain model and object relational mapping for the wrong reasons. Be skeptical about objections concerning security, performance and SQL injection. More on that another time.

Much has been said about domain models. In recent years, the most influential books on the topic have been Martin Fowler's Patterns of Enterprise Application Architecture and Eric Evans' Domain Driven Design.

Centralized or delegated control

If you take a look at sequence diagrams of logic designed as the two procedural patterns, transaction script and table module, compared to the object oriented ones, active record and domain model, you will see two quite different information flows. The procedural ones uses a centralized control style, whereas the object oriented ones use a delegated one.

The logic for transaction script or table module is controlled from one location. The script knows which steps the transaction needs to take to perform the task, and asks appropriate helper classes to solve each step. A sequence diagram will show you information going into helper classes, often with single parameters or a form of DTOs, and back again, then into new classes, and so on. You have a central point of control.

Active record and domain model works in a different way. Here the responsibility is typically delegated to one or more objects, which again delegates responsibility to other objects. A sequence diagram will show you a flow going into an object, then delegated into other objects, and so on, instead of going back and forth from the central location. In this way you have a delegated form of control.

I think the procedural, or centralized control style, is more common in .NET than in Java. The main reason is the support and focus that Microsoft has had.A research paper was published in IEEE Transactions on Software Engineering, called "Evaluating the Effect of a Delegated versus Centralized Control Style on the Maintainability of Object-Oriented Software", where about 150 senior, intermediate and junior developers, including a number of students, participated. The developers had to make various changes in both a delegated and a centralized design. The results:

"The results show that the most skilled developers, in particular, the senior consultants, require less time to maintain software with a delegated control style than with a centralized control style. However, more novice developers, in particular, the undergraduate students and junior consultants, have serious problems understanding a delegated control style, and perform far better with a centralized control style".

And then concluding:

"Thus, the maintainability of object-oriented software depends, to a large extent, on the skill of the developers who are going to maintain it. These results may have serious implications for object-oriented development in an industrial context: Having senior consultants design object-oriented systems may eventually pose difficulties unless they make an effort to keep the designs simple, as the cognitive complexity of 'expert' designs might be unmanageable for less skilled maintainers."

I think the conclusions with the delegated style of control also has a lot to do with familiarity. Since many .NET developers, including senior ones as well, have limited experience with this, imposing a delegated style can take some time getting used to.

Have I covered everything then?

Since I put the heading in here, I'm sure you already know the answer to the question. There is one pattern I haven't mentioned yet, or an anti-pattern anyway. And a common one, that is.

I've mentioned several times in this post how the .NET community often have had a procedural focus. I think that is the reason why this pattern seem to be so popular in .NET-land. Let's have a closer look at what I'm talking about.

Anemic Domain Model

The Anemic Domain Model looks like a domain model, has a rich structure of objects, but there's almost no behavior in the objects. The logic is typically controlled via a transaction script, with the model being simply data containers.

The examples are Hello World in complexity, but hopefully still captures the main difference between this and a regular domain model. As you can see, the logic has mostly been moved out of the domain model, and into a different class. It's quite common that these are referred to as xxHelper, xxManager, xxHandler, or other general terms.

This (anti-)pattern can be easier to understand for those struggling with delegated control, but this is a benefit with a cost. Other benefits is that you can brag about having a domain model to your friends (which you don't), and any other benefits you get with transaction script. The problem is that you get the same problems as with transaction script - a challenge with complex code, duplication of code and logic that is harder to unit test. In addition you get the complexity of using an OR/M on top of that.

Conclusion

None of these patterns is never right. That's part of the fun of software development. You need to take a close look at what you need to build and then take an informed guess. Take into account the positive and negative sides, and when you have made the choice - make real effort to diminish the effects of the negative sides.

The technical laws of December

2010-01-06T10:06:00.000+01:00

Throughout December I had the pleasure of collecting more or less clever statements about computers and software development in general. My twitter account had a steady flow of these up until Christmas.

In today’s society, we have to use so much of our energy filtering out knowledge, and it is interesting how cleverly written sentences can capture so much information. I’m going to elaborate on several of them in the time to come, but for now, here they are again.

Disclaimer: These aren't my quotes - they've been collected from a variety of places and people, and thus credit is due in many places. I apologize for not quoting correctly.

Dec1 tech law: Walking on water and developing software to specification are easy as long as both are frozen

Dec2 tech law: In theory, there is no difference between theory and practice, but in practice there is

Dec3 tech law: Real programmers don't comment their code. If it was hard to write, it should be hard to understand

Dec4 tech law: Computers are unreliable, but humans are even more unreliable. Any system which depends on human reliability is unreliable.

Dec5 tech law: There is never time to do it right, but always time to do it over

Dec6 tech law: Adding manpower to a late software project makes it later

Dec7 tech law: The degree of technical competence is inversely proportional to the level of management.

Dec8 tech law: The probability of bugs appearing is directly proportional to the number and importance of people watching

Dec9 tech law: If a program is useful, it will have to be changed. If a program is useless, it will have to be documented.

Dec10 tech law: Good enough isn't good enough, unless there is a deadline

Dec11 tech law: An expert is someone brought in at the last minute to share the blame

Dec12 tech law: The chances of a program doing what it's supposed to do is inversely proportional to the num lines of code used to write it

Dec13 tech law: profanity is the one language all computer users know

Dec14 tech law: All's well that ends.

Dec15 tech law: No matter how hard you work, the boss will only appear when you access the Internet.

Dec16 tech law: A meeting is an event at which the minutes are kept and the hours are lost.

Dec17 tech law: An expert is one who knows more and more about less and less until he knows absolutely everything about nothing.

Dec 18 tech law: it's not a bug, it's an undocumented feature

Dec19 tech law: A complex system that works is invariably found to have evolved from a simple system that works

Dec20 tech law: The documented interfaces between standard software modules will have undocumented quirks

Dec21 tech law: Bugs will appear in one part of a working program when another 'unrelated' part is modified

Dec22 tech law: When designing a program to handle all possible dumb errors, nature creates a dumber user

Dec23 tech law: Build a system that even a fool can use and only a fool will want to use it

Dec24 tech law: The cleverness of technical laws is inversely proportional to the number of laws

Walking on water and developing software to specification are easy as long as both are frozen

2010-01-03T19:11:00.001+01:00

Developing software is pretty complex, but it’s not exactly rocket science, right? You find out what you want created, you get a number of software developers that has studied how the technicalities work, and after a time you end up with a good, working piece of software. Correct?

It's a shame how few reading this can actually agree with it. Why is that? And can anything be done about it? If you have a few decent developers (they don't have to be that good), a small set of requirements, a business expert they can discuss those requirements with... You know what - you just got yourself the perfect recipe for a good working system. Fingers crossed!

Unfortunately the previous description doesn't map to many real systems. One of the biggest problems is that requirements change. This affects all parts of software development, but no other part can influence the result of this like the process used. Back in the days, and still in practice quite a few places today, they tried to scale the previously described way of solving software development no matter the size of the project. You just collect and analyze all the requirements, find a number of developers to implement the system, get one or more architects based on some measure of experience to help design the structure, plan and discuss the requirements, implement it, test it and release it. That works to a certain extent, to a certain size, for certain organizations, but it has many limitations. I'm not going to try to cover those extensively now, but it includes

You've got one shot at coming up with the features for the system. Now guess what happens -> You will try to come up with as much as humanly possible, important or not
You have little idea how the system is actually going to look or feel or just how it can be used -> You will try to make the requirements cover everything. This isn't really a bad thing, but can be quite a complex process.
Price the project beforehand, to move the risk of development to a software company -> Do you think those requirements change sessions will be fun?
Your understanding of what the system can bring you, what you really need it to do, and just how it can do that the best -> Will change!

The whole point of this post has become very well understood in the industry - requirements change, and you'll have to cope with that one way or another. If you want a useful system that is. A number of smart people put their heads together back in 2001 and came up with the "Manifesto for Agile Software Development", and a number of principles behind it.

What was the cure they came up with? Besides a few other good points, it was the formalization of an old principle - tackle complexity by dividing a problem into smaller, manageable pieces. (Note: The original paper on the Waterfall model did cover the importance of a certain amount of iterative behavior, but I guess it was lost to quite a few people on the way.) If you’re trying to tackle all problems at once – or in other words not developing software in an iterative manner, make sure you have very good reasons for it. Unless the project is small, with clearly defined knowledge and boundaries, you’re going to have a hard time producing the system you really need. Do you do it for control? There is no real form of control in software besides working software.

Is process the only thing you have to think about when handling change in software development? Not by a long shot, but it is an important part of it. I will cover more areas soon.

#GEEKBEER starting this Thursday (in Oslo)

2009-02-10T23:00:00.001+01:00

Tore Vestues, Sverre Hundeide, myself and more thought it was about time we got a regular social developer-night together. The purpose is simply to socialize and have a good time. No free beer, no surprises, just fellow developers/IT people.

We'll start up this Thursday (the 12) , 7pm, at Asylet (Grønland). Come along!

We hope to make it into a monthly thing, so don't despair if you can't make it this time.

Essential Project Practices: Communication and close customer collaboration

2009-01-21T23:59:00.001+01:00

I'm a firm believer in the need for close customer collaboration in projects, a high degree of communication and continual status updates to best manage basically anything. I think I would define it as one of the major "make or break" characteristics of a project.

I remember being asked during an interview with my current employer what I believed was the biggest reasons projects fail. Communication was the big thing that came to mind. Projects fail for a myriad of reasons, but not the least because of a lack of communication between a number of involved parties. I thought I'd try to get in to a few of those aspects here.

Working on communication at all times is important. The various roles in a project or organization has different responsibilities here.

Upper management and project champion
First of all, any project, and especially those related to organizational changes, needs upper management support. If the project needs to communicate, integrate or in other ways depend on other parts of the organization, it is hard to do it without the support from the top. The unofficial networks built by social interaction within an organization are one way it can work. But management support is extremely important, so you better find out how to get it - even if it is not your direct responsibility, pull some strings to find out if something can be done about it.

Another role of extreme importance is the project champion. A project champion is a person highly dedicated to see the project through from inception to success. Not completely about the topic, but this is a person that will help make sure that the needs of the project are taken care of throughout the organization. Communication is tool number one here.

Project manager -> developers
The project manager and the development group also need to communicate. I use project manager here to mean several possible positions, both project manager in the traditional meaning, and for instance product owners in Scrum tongue, or even in part Scrum masters. The two sides has different needs for communication. The project manager has the primary responsibility of delivering the project on time, within budget and according to specification. There's many ways of handling this.

Organizational requirements and processes is the primary one, but trust has no less importance.
Organizational requirements and processes is something you have to live with in some form. If you are able to build trust in what you deliver outside of the project scope as well, you'll most likely have more control of what you need to deliver. The major reason behind many of the demands and project artifacts found today is a lack of trust. The lack of trust is compensated for by putting in a myriad of artifacts, processes and control points. Even though that is often a false sense of security, I generally completely agree with the approach - if you don't have trust, the next best thing is to monitor the progress.

So how do you build trust? First and foremost you need to deliver (Hopefully on time, within budget, and according to specification :) ). Communication is tool number two. When problems arise they need to be communicated up the ladder as needed. If you have trust that problems once discovered will be taken through the right channels, you have a good basis for being able to solve any problem once they arise. To build trust, you need to communicate continuously. People are different, but little and often is often better than much and seldom. Manager Tools (brilliant podcast-series) has a podcast describing how humans relate to trust and relationships here.

Project -> Business expert and users
An essential step in delivering a product that fulfills needs is close collaboration with the business experts and users. It cannot be a single iteration waterfall-thing, it needs to be a continuous collaborative effort. You do something, then you verify what is done. You wonder about something, you contact the business expert or user. And how much easier is it to get both attention and clear answers if you have someone within the same room than over phone or email?

Developer -> Developer
Developers are certainly not excluded from the communication effort. Developers need to continuously communicate within the group to ensure everyone has a clear understanding of the problem at hand, to build project ownership, to share project knowledge, and to notify and get help for problems. And if you are a developer, what better way is there to increase your skill-set than to communicate with other developers? And try to be aware of groupthink, that is certainly not good to have to much of.

In conclusion
Writing about something like this is impossible to do well without references and ahead planning, so I'll easily admit that the above text is heavily flawed. However, the concept is very important, and that's why I wanted to share a few disconnected thoughts on it.

If you find the topic interesting it's not hard to find sources talking about it. It is slightly harder to find interesting sources.
The podcast mentioned above has interesting topics, but it is mostly related to managerial behaviours. You can access it here
If teams and communication is interesting, you'll find Peopleware's Productive Projects and Teams great.
If other aspects of communication seems interesting, I'll be happy to recommend books on the subject.

NNUG: Tonight's meeting

2009-01-20T23:29:00.004+01:00

Jimmy Nilsson and Tore Vestues was the speakers on todays NNUG-meeting.

Jimmy Nilsson is most known for having written "Applying Domain Driven Design and Patterns", one of the few good books on DDD, as well as having a practical focus on for instance TDD. Jimmy was nice enough to speak for us tonight after having held a LEAP course today. I haven't heard him live before, but if you can say anything about Jimmy, besides being very skilled, is that he is a really nice guy to listen to. Good stuff :) For those that missed it, his speech "En ny era för dataåtkomst?" (A new era of data access) took on the history and general approaches about data access, as well as more general info about TDD. Please remind me that we must have a topic with a more close look at the data access options soon.

Tore Vestues had a speech about about Code Quality. (In the interest of full disclosure: we are colleagues.) The speech was well worth the time, and the topic should be on the absolute top of your list of priorities.

Interesting Old Stuff: Management tutorial

2009-01-19T23:04:00.004+01:00

I browsed through an old mailbox I used long ago today, and stumbled upon an old weekly tutorial for a project management class I attended. That week's tutorial seems like it was about methodologies, continuous feedback and client demonstration. I know I would have a lot to say about that now :)

An interesting read out of a sort of sentimental fashion for me. I'll post it here just for fun

You are the IT project manager of an organisation developing innovative ‘leading-edge’ warehousing solutions. Although a number of individual software components have yet to be completed your clients are insisting on regular updates of your progress, backed up by demonstrations of the latest versions of the individual software components. Their insistence is being reinforced by the client’s threat to withhold staged payments that are required to be made as part of the contract terms.

How might this type of client behaviour affect the type of development methodology you employ?

With any project we believe it is important to supply the client with regular updates and demonstrations of the latest versions of the individual software components. When dealing with a very insistent client, as in this case, we believe that there is a need to use a methodology that incorporates user/client involvement throughout the project.

The development methodology type that is most suited for these requirements is one based on prototyping. By using such a methodology the project will focus on and deliver working prototypes, and show early efforts of integrating the different parts of the project. An example of a methodology that could be suitable for this type of project is Extreme Programming (XP). XP is a spiral type methodology with short iterations and close customer contact. It has a strong focus on integrating early and creating working prototypes. Since the customer will be updated on the progress of the project at short intervals and are able to change direction due to problems or new requirements they will be able to do real-time planning with the project.

Generally we believe that it is important to know and understand any development methodology, and not use any aspect of it without considering the consequences. Often you need to use parts of different methodologies, or change part of the one you are using, as it does not necessarily work perfect in your organisation.

Why do you believe the client might be so insistent on receiving regular ‘proof of progress’ updates?

There are many possible reasons for why a client would insist on receiving regular “proof of progress” updates. We believe that such an insistent client’s main reason for behaving like that is the risk of project failure. It is well known that many IT projects fail to complete at all, complete far over time or budget, or just isn’t used as planned in the organisation.

During a project there are many problems that occur. A client wants to see that everything is going as planned, that the project milestones are met, and that the contracted agreements are fulfilled.

If a project develops away from the project specifications and the project plan agreed upon with the client, then the client can demand corrections according to the contract or even bail out. The earlier this is discovered, the better it is for the client.

You discuss the issue with your own executive management and they make a suggestion that you merely ‘fake’ the demonstrations without making any genuine attempt to provide a genuine product. What are some of the implications here?

If you were to “fake” the demonstration of a product we believe that there can be both positive and negative implications to this.

Firstly, it might be positive for the project to do this if you are having a slow start to the project but you are still very confident that you will be able to finish the section in question on time.

Another reason might be if you are up to date on the coding, but you might still have problems showing visual proof of progress. By faking the demonstration you can avoid problems you might get by not producing visual results. In other words you will not be picking the low hanging fruits first just to have something to demonstrate.

On the other hand there are some negative implications too. You can risk giving the client false hopes about the probability of project success. If the client detects the faking of the demonstration you will have to take the consequences. This can range from the client loosing trust in you, a loss of reputation and a loss of the contract to possible lawsuits.

All of these negative implications are risks that can lead to economical consequences. Therefore it’s important that you consider the implications thoroughly before making a decision to fake such a presentation.

Interesting Blog Posts: NHibernate and Entity Framework Battle it Out in the Real World

2009-01-19T20:20:00.001+01:00

If an OR-mapper is what you need, here's a short and interesting real life example of how a showdown between EF and NH went.

Note: Look, I am saying OR-mapper here, so don't get me started with the "they are impossible to compare since EF is supposed to be a persistence-, object-, view-, query-, dataservices"-discussion. That is so 2008.

Interesting Software Thoughts: 8 Things We Hate About IT

2009-01-18T23:59:00.003+01:00

It's interesting to step a bit back from the daily technical crunch and see how the world of IT is viewed from the outside at times.

A management guru has a blog post titled the same as the header.

The points listed are the following, as well as a few thoughts I formed while reading them:

1. IT Limits Managers' Authority.
2. They're Missing Adult Supervision: IT needs close supervision. Short feedback loops through iterations that produce results are the way to get it. Milestones and a kill switch is another requirement.
3. They're Financial Extortionists: It's scary to think about all the money wasted on the appalling track record of IT in general. But there's undoubtedly a shared responsibility there, both on the IT and general business side.
4. Their Projects Never End: Setting deadlines, having milestones and defining Done is essential in any task.
5. The Help Desk is Helpless: Software has bugs. Especially since it is often a conscious decision to release something when it's "good enough". I don't envy the help desk personnel in IT in general.
6. They Let Outsourcers Run Amok: I strongly doubt IT is the main driver in outsourcing!
7. IT is Stocked with Out-of-Date Geeks: With the rate IT is changing, this is a most truthful and dangerous statement. It takes a lot of effort to stay up to date on the technologies and practices of IT. The number of years a person has been in IT doesn't necessarily add up to the skill level of the person.
8. IT Never Has Good News: Too seldom, that's for sure!

So what do we need to succeed with IT projects?

We need close customer collaboration.
We need projects with short feedback loops.
We need sane requirements engineering.
We need to know what Done means.
We need the business side to be active in the entire process to deliver what they need.
We need quality in the development force. Such a simple task :)
We need companies that dare to change. Running IT the old fashioned waterfall-way didn't work very well, did it? Time to try something new.
We need to be careful with new technologies. If it's new and you don't run a spike, then shame on you.
We need developers with a vested interest in the projects success. Not just their own. The project must have priority over individuals. Heroic individuals is not the best approach for a project.
We need to build trust. Now that's a big one!
We need a hell of a lot more than I have listed here, but these just came from the top of my mind.

One thing is for sure. The challenge is not only on our side of the table. But we should help educate the business people we work with every day on how we can more readily succeed.

Interesting .NET Bugs: Object Initializers and using-statements

2009-01-17T13:50:00.002+01:00

Ayende posted about this bug, but it's one hard to spot bug, so I felt it was valid to pass on.

If you use object intializers in the definition of a using statement it is normal to expect that if an exception occurs the dispose method will be called. After all that is the point of the pattern.

However, object initializers are compiled in an unexpected way here:

Beware.

(Both images stolen from Ayende. No harm intended.)

Interesting Software Resources: Domain Driven Design Quickly

2009-01-16T23:57:00.001+01:00

If you haven't had time to get into any literature about Domain Driven Design yet, InfoQ has created a free online short version of the Domain Driven Design bible which is well worth the time.

Order the book now, and read the resource until you get it. This is ESSENTIAL.

Why it is needed, an except:

Is it possible to create complex banking software without good domain knowledge? No way. Never. Who knows banking? The software architect? No. He just uses the bank to keep his money safe and available when he needs them. The software analyst? Not really. He knows to analyze a given topic, when he is given all the necessary ingredients. The developer? Forget it. Who then? The bankers, of course. The banking system is very well understood by the people inside, by their specialists. They know all the details, all the catches, all the possible issues, all the rules. This is where we should always start: the domain.

When we begin a software project, we should focus on the domain it is operating in. The entire purpose of the software is to enhance a specific domain. To be able to do that, the software has to fit harmoniously with the domain it has been created for. Otherwise it will introduce strain into the domain, provoking malfunction, damage, and even wreak chaos.

Essential Project Practices: Throwing more programmers at a project

2009-01-15T23:46:00.005+01:00

Fredrick Brooks Jr. made an interesting and in time very well known statement when he coined Brooks's Law in 1975:

"adding manpower to a late software project makes it later"

He himself said it was an "outrageous oversimplification", but it is an observation that has had value in all these years.

I think its primary value lie in the simplicity of the statement. It's a law that can be used to easily argue against common sense: that doubling the amount of developers should double the productivity. And even more, it's a statement which can easily catch the interest to find out why it was stated. One thing is that adding developers to a project won't double the productivity, quite another is to say that it won't increase productivity at all, and it sounds really absurd to say that I will make it later.

The Law is based on a few main points:

New programmers will have to learn about the project.
Communication overhead will increase

So what can you do to avoid fulfilling Brooks's Law? It's not like its gravity we're talking about here.
There's many factors involved:

How skilled are the new developers?
How well do they know the domain?
How many are they?
At what time do they come into the project. Early or late?
How well do they fit into the culture?
Are they team players that can accept the current process, or will they fight it?
How well can the work be segregated?
What practices are in place to ensure the quality of the work produced? Automated testing, Continuous Integration, close customer collaboration, code reviews and lot's more can help to ensure that things don't get out of hand.
...

Pretty obvious, but I doesn't hurt to have a law for it.

Essential Software Practices: Favor composition over inheritance

2009-01-14T23:41:00.002+01:00

This is an old and important principle that people seem to forget at times. So time to refresh your memory of this one as well.

In OO design you have two common ways of dealing with duplication and reuse: composition and inheritance.

Composition: Assembling objects. "Has a" relationships. Black box reuse (No internals visible)
Inheritance: Defined in terms of the parents implementation. "Is a" relationships. (Don't forget Liskov) White-box reuse (Access to parent internals).

As stated in the definition above, you use inheritance for "is a" relationships (that are substitutable).

The are several problems with using inheritance for anything else

You're not supposed to use inheritance for code reuse.
You can (sort of) break encapsulation since the sub-class is so tightly coupled to the parent's implementation.
Inheritance is set in stone at runtime, so no dynamic fun.
A subclass can easily make assumptions about the context the method it overrides is called.

Composition has other strengths than inheritance

Any object with the same type can be replaced at runtime. This is important for many things, for instance testing - as we want to test in isolation.
You can "just plug in behavior" with composition. A small object can help decide how a bigger object works.
It is the better approach when you want to reuse functionality.

It is also erroneous that composition does not use inheritance. It is typically done through implementing smaller interfaces rather than inheriting from a big class.

If you want to have a look at what Grady Booch defines as the "prototypical example of the flexibility of composition over inheritance", take a look at the strategy pattern.

To sum up a short text: This is really a no-brainer. Think for a couple of seconds before choosing an approach, and just use your gut feeling.

Essential Software Concepts: Static methods

2009-01-13T23:43:00.003+01:00

Static methods is a common functionality in many languages. Whereas instance methods work on instantiated objects, static methods work on types or instantiated objects.

The big question is - when is it appropriate to use static methods?

First the naive positives. These are the positives that often comes up, but which aren't really true.

Makes it more obvious which methods works on internal state. If it doesn't have state, it should be static.
Great for Factory Methods (See real negative).

And then the real pros and cons:

Real positive

Don't need to initialize an object. This can be good in the case of small and discrete methods. Some utility classes are good examples, for instance Math.
Can be good for simple and effective, fire and forget functions.

Real negative

Is hard to refactor away from.
Doesn't work with inheritance, polymorphism and interfaces.
Makes it hard to test (Some test-frameworks can). For those that can't it's impossible to mock the behavior, since there is no natural way of substituting the implementation at runtime.
Static Factory Methods makes implementers bound to the concrete implementation. Impossible to replace during test. Handling object graph wiring this way makes it even worse.
Simple static methods can evolve into larger (still, possibly, untestable) beasts, which are not only a hassle to refactor away from, but do they really give the advantage of risking it?

What you potentially can read from a static method

Unless global state exists, the static method can only work on the input parameters. It is often more natural to move the method to one of these. Remember to keep logic as close to where it belongs as possible.

Just a note. Untestable in the points above isn't necessarily true. I'm talking about the possibility of switching implementations with stubs/mocks to be able to isolate what we are testing. It depends what dependencies it has of course.

There is use for static methods. Just be sure that it outweighs the potential negative issues!

Windows 7 useful stuff (link)

2009-01-13T21:04:00.001+01:00

Tim Sneath, director of the Windows and Silverlight technical evangelism team, has an interesting post about 30 windows 7 "secrets". Certainly served as a good reminder to myself that I need to get the Windows 7 beta. The fact that they have spent much more time preparing 7 for power users is a great. I despise using things I can't customize.

note to self: download windows 7 beta shortly

Essential Software Principles: Don't Repeat Yourself (DRY)

2009-01-12T22:54:00.001+01:00

Don't Repeat Yourself, also called the Single Point of Truth, is about representing any and every piece of information in a single representation.

It is usually used only for code, but was meant to have a broader meaning. According to the authors of The Pragmatic Programmer it should include "database schemas, test plans, the build system, even documentation". They further refer to various techniques for achieving this, like code generation, automatic build systems and scripting languages.

It is an important principle to focus on. Whatever it is you do, if it's described in two places one is going to get out of sync from the other. Other issues come as well. Duplication will make it harder to do changes, it reduces clarity and is perfect for creating inconsistencies in logic.

The Pragmatic Programmer defines for reasons for duplication:

Imposed duplication: You feel you have no other choice
Inadvertent duplication: You don't realize you are duplicating
Impatient duplication: Duplication seems like the simplest option
Interdeveloper duplication: Multiple people duplicate

All are common cases for duplication. Being aware of each and working to avoid them are essential.

The prime example of imposed duplication is documentation vs code. Often this is something you cannot avoid. It can also bring value. For instance can different views of a system be terribly helpful in simplifying and understanding issues. But as we all know, these things quickly get out of sync. And the effort of keeping documentation up to date is seldom a funny effort.

Often when you have an example of Imposed Duplication, other ways exist to solve it. Code is often duplicated into documentation because it is not available externally in a concise form. Perhaps you could get away with the BDD form of doing it, having running pieces of business rules. That would in many ways remove duplication.

I don't have much to concrete to say about it concerning our prime artifact: the code. Except one thing. You don't duplicate unless you have a really, really good reason for it. Duplication leads to anger, anger leads to hate...hate leads to suffering.

Essential Testing Practices: Arrange, Act, Assert (AAA)

2009-01-11T21:09:00.001+01:00

Unit testing is an important practice for developing a system. There's many things to it, and many reasons behind it, things I'll probably get into at a later time. One of the aspects about HOW tests are written is the structure of the tests.

Since unit testing was introduced, there has been many ways of specifying unit tests. The usefulness of mocking frameworks has been more and more acknowledged, but the structure of tests have often suffered under immature frameworks (and languages!).

The structure of unit tests matter because it makes it
- easier to understand the usage of the code under test
- easier to change
- easier to fix
- easier to write. You should be able to do it in the natural way you think about it.

The Arrange, Act, Assert unit testing structure pattern is an old way of solving this. The concept is utterly simple to understand:

First you arrange the state of the test. The you act by executing the code that is tested. At last you assert by checking that what you expect to happen actually has.

Note: Both Typemock and RhinoMocks support the AAA syntax. That means you don't need to/shouldn't use the record/replay mode anymore.

Interesting Software Practices: Code in non-English

2009-01-10T13:20:00.000+01:00

Which language to write code and comments in can sometimes come up as a question in countries where English isn't the native language.

There's both good and bad things with doing this, and I thought I'd list a few thoughts about it.

The bad things about writing code and comments in your own language is:

People that don't speak the language can't easily work with the system. This involves both professionals moving to a foreign country and concerning outsourcing.
There will always be mix of languages involved. Any external libraries/frameworks/components will use a different language.
Any characters not present in the English language will have to be converted somehow. If there's no match, this can lead to various translations.

The good things about using your own language is:

You might have an easier time with your ubiquitous language. Speaking in a mixture of languages or trying to translate could lead to misunderstandings and loss of important distinctions.
You might use less time finding the correct words.

Did I forget anything obvious? What do you think?

Interesting Software Books: Dreaming in Code

2009-01-09T16:10:00.000+01:00

Dreaming in Code is a novel best described by the author himself:

"Dreaming in Code: Two Dozen Programmers, Three Years, 4,732 Bugs, and One Quest for Transcendent Software ... I spent three years following the work of the Chandler developers as they scaled programming peaks and slogged through software swamps. In Dreaming in Code I tell their stories."

This is not a book meant for the general developer, at least not so much to learn about software development. It is not supposed to be either. I'd recommend it for a developer as an interesting story only. But the main reason I'm mentioning it here is that I think it could have value for an outsider to learn many things about software development. And doing it through an interesting story. It does so partly by telling a story, and with that showing (an example) of a software process (granted quite different from many non-open source projects), and partly by getting into important information and characteristics about our industry.

Here's one of the less informative but quite funny jokes about our profession (I'm copying it since he said himself it could be found a myriad of places)

A Software Engineer, a Hardware Engineer, and a Departmental Manager were on their way to a meeting in Switzerland. They were driving down a steep mountain road when suddenly the brakes on their car failed. The car careened almost out of control down the road, bouncing off the crash barriers, until it miraculously ground to a halt scraping along the mountainside. The car's occupants, shaken but unhurt, now had a problem: They were stuck halfway down a mountain in a car with no brakes. what were they to do?
    "I know", said the Departmental Manager. "Let's have a meeting, propose a Vision, formulate a Mission Statement, define some Goals, and by a process of Continuous Improvement find a solution to the Critical Problems, and we can be on our way."
    "No, no", said the Hardware Engineer. "That will take far too long, and, besides, that method has never worked before. I've got my Swiss Army knife with me, and in no time at all I can strip down the car's braking system, isolate the fault, fix it, and we can be on our way."
    "Well", said the Software Engineer, "before we do anything, I think we should push the car back up the road and see if it happens again."

Essential Software Refactorings: Compose Method

2009-01-08T23:15:00.003+01:00

Joshua Kerievsky introduced a refactoring in his (great) book Refactoring to Patterns, which involves a practice I believe strongly in. The refactoring is called Compose Method. (Spoiler: It is really all about using short and descriptive methods.)

The technique is to transform complex logic into a number of self-explanatory methods. The whole point is to increase understandability, and by that also maintainability.

Here's a simple example taken from the industrial logic catalog page for this refactoring. You have a complex method. To increase clarity, you refactor out the individual parts into smaller methods, making the logic easier to understand (I don't fully agree with the coding standard though):

The main advantage with this approach is that core logic will be more readable and maintainable. A disadvantage is that classes will be very fragmented, which can be annoying. It can also affect debugging, since logic is so decentralized.

In terms of length of methods, Joshua defines composed methods as usually being about five lines long, rarely over ten.

Long methods have many undesirable properties, like being harder to understand, harder to refactor and containing duplicated logic.

Essential Software Concepts: Liskov Substitution Principle (LSP)

2009-01-07T22:30:00.001+01:00

The original definition of the principle is:

"Let q(x) be a property provable about objects x of type T. Then q(y) should be true for objects y of type S where S is a subtype of T."

This is all about substitution. Any place where a base class is used in the code, you should be able to use a subclass as well.

A few implications come from this:
- You shouldn't do runtime checks for which subtype a certain object is. That means it is not automatically extendable and must be modified each time a new subclass is added.
- You shouldn't do less in a derived class. That means for instance that you shouldn't override a certain method and just leave it empty. That's not always a violation of LSP, but it can be.
- You shouldn't hide the implemention of a base class method (by using new). If you have a method that takes in an object of the type of the base class and calls such a method, only the base class method will be called.
- Even if it is often said that IS A means inheritance is the right approach, that not really true. Substitutability should be taken into account.
- You can't simply look at the implementation of the model in isolation and decide if it is correct or not. You need to take into account the natural assumptions users will have about them. KISS still applies though.
- Not following this can bring out some hard to find bugs if someone else believes the implementation follows LSP.

Essential Software Concepts: The Open/Closed Principle (OCP)

2009-01-06T23:05:00.002+01:00

Another essential software concept is what is known as the Open/Closed Principle. It's again a simple concept that is important to follow. The definition is as follows:

"Software entities (classes, modules, functions, etc.) should be open for extension but closed for modification."

The whole point is that in places where you expect to extend the functionality you should work to make sure that you don't need to change existing code to solve it. The code should be designed in a way that enables you to extend the behavior by for instance adding new classes that automatically works with the current model. This is usually solved through abstractions and polymorphism, and a number of patterns can help solve this. A common mantra for code that is expected to change is nonetheless followed: code to abstractions, not implementations.

What the principle tries to achieve is to make sure that changes have as little as possible effect on the current implementation. If you need to modify existing code you need to identify the various parts needing change and you impose the risk of making breaking changes. Extending the behavior can often be a lot easier and can enable a more flexible solution.

Does this mean that every piece of code should be written using abstractions to allow extensions more easily? Certainly not! Abstractions can add unnecessary code, can make the parts that are supposed to change less obvious and can add to the complexity of the solution. Unless you know that something is expected to change, an old childhood rule is nice to follow: Trick me once, shame on you, trick me twice, shame on me. Don't do the extra work up front unless you know it is necessary. But once you need it, don't hesitate to put it in. Uncle Bob has a wise quote about this:

"Resisting premature abstraction is as important as abstraction itself".

Essential Software Concepts: The Single Responsibility Principle (SRP)

2009-01-05T21:46:00.003+01:00

The Single Responsibility Principle defines that a class should have only one responsibility, also defined as only one reason to change. (Defined by Uncle Bob in Agile Principles, Patterns, and Practices in C#. A book well worth reading.)

In other words, it should only be responsible for one thing. Do only one thing. Handle only one area. Potato, Potato.

The principle is really simple in concept, yet it can be difficult to follow at times. It is an important principle to follow because it will make your classes more robust to change. Coupled responsibilities can lead to code that breaks more easily, changes that are more expensive to implement, tests that are harder to make concrete, less readable code and more code to step through when debugging.

That doesn't mean you _always_ need to follow it. The cost of separating things can be higher than the cost of holding them together. If you expect no change, then stay with that. Possibly separate the concepts by creating separate interfaces. That way the implementation stays coupled, but at least the rest of the system is not coupled to the mess.

Essential Management Practices: Don't micro manage

2009-01-04T22:14:00.001+01:00

Do you like to micro manage, or are you being micro managed?

That has really got to stop. It is one of the most annoying practices I know about, and one which makes no sense in our industry.

Software development is a form of knowledge management. Developers (obviously) use their heads as their tools. If there is one thing knowledge management professionals despise, it is being told at a low level exactly what to do.

In essence this just tells someone that you don't trust their work or judgment. Not exactly a good way to get motivated employees that feels ownership for the project.

I'm not advocating that you shouldn't follow up work. Absolutely not. In fact, code reviews should be a part of any development organization. I think that is quite different though. Verifying and sharing a solution is different from closely managing how to solve an issue.

Essential Software Concepts: Side-Effect-Free Functions

2009-01-03T16:50:00.001+01:00

I'll start of with a section about naming and understanding of the concepts involved.

Side-Effect-Free Functions is a name that will mean different things to different people. The explanation in common English is "a function without unintentional consequences". If you use Side effects as explained in software science (wikipedia), a side-effect means any state change in the system. The explanation will then change to "a function without state changes". Eric Evans in his Domain Driven Design (DDD) book further narrows the side-effect concept to mean "any change in the state of the system that will affect future operations". So we have a situation where we have a common expression which is supposed to be understandable, but you need to be aware that that people can think of three different things when you talk about this.

Another concept we need to define is Functions. Functions is another concept with different meanings in different settings. In some languages, functions and methods means different things, which means you'll have two separate concepts. In .NET you only have methods, but in my experience functions and methods are used as synonyms in .NET realm. In DDD functions are explained as being operations that return results with no observable side effects. In DDD it is further explained that methods that result in modifications to observable state are called commands. I think that's a definition that is lost on most developers. I think it is more important to focus on the concept rather than that, so I won't pay more attention to that.

(Just a stupid observation: If Functions as defined in DDD means "operations that return results with no observable side effects", then coining an expression called "Side-Effect-Free Functions" does in fact not make any sense, since you can't have side effects in DDD Functions.)

There is a reason why we would want to separate operations that will change state and operations that only queries for information. Queries for information is in essence safe to use at any given moment, whereas state-changing operations, even if perceived safe, can be very hard to be certain about in a complex system. By separating them completely, you'll have an easier job handling that complexity.

I'm not so sure that this should be applied in every situation. But what I believe is an absolute demand is that you should be able to trust that a given method/function/operation does exactly what it says it does. Trust is the big thing here. If you have a method called GetBills(), and you in fact do a call to a GenerateBills() method within, we have a big trust issue! Let's not focus on the fact that these don't belong together, but you should at least have followed the concepts behind Intention-Revealing Interfaces, and named it as GenerateAndGetBills() or such.

I believe the biggest issue here is the same as I came into in Intention-Revealing Interfaces; If you can't be sure that the parts of the system you are working with are behaving the way you think they should, then you'll use all your focus making sure they do, and not on the problem at hand.

DDD also has a conclusion worth reading: "Place as much of the logic of the program as possible into functions, operations that return results with no observable side effect. Strictly segregate commands into very simple operations that do not return domain information."

Update

Have a look at the link provided in Morten's comment below for more info on commands and queries. Thanks :)

Essential Software Concepts: Intention-Revealing Interfaces

2009-01-02T17:17:00.001+01:00

Donald Knuth famously said "Software is hard". And that is an absolute truth. But that doesn't mean we shouldn't always strive to make things as simple as possible. One concept that is easy to understand that can help in that is Intention-Revealing Interfaces.

The interface comprises all the publicly visible parts. Together they should combine to an Intention-Revealing Interface. That means an interface where the intent and usage pattern is clear.

One of the largest costs of software development is understanding and changing code. That certainly involves the maintenance phase of the project, but living in the increasingly agile world we do, it most certainly is a continuous factor throughout the initial development of the system. If the code you need to work with is poorly described, you'll have to keep digging into code all the time, trying to get a full understanding of how everything works before you dare to do changes. Unfortunately, since our brain can only hold _so_ much information at a time, this severely limits our ability to to work efficiently. Instead of using our focus on the problem at hand, all is wasted on the surroundings. Like Eric Evans said, "If a developer must consider the implementation of a component in order to use it, the value of encapsulation is lost".

So work continuously to be as clear as possible in how you name your classes, methods, properties, etc. That means describing what they do, and not how they do it.

And a last thing. Don't be afraid of long method names. Use as long a name you need to describe what a particular part does. If you need to use many words to describe it, chances are you should start thinking about a refactoring.

Interesting General Knowledge - Pareto's principle

2009-01-01T20:35:00.002+01:00

I don't remember when I first heard of Pareto's principle, but I've always been fond of it. It was originally used to state that 80 percent of Italian income went to only 20 percent of the population, but has since been used as a rule of thumb for a myriad of other cases.

I think it fits very well in the software domain. The exact numbers are not the important fact, but rather as a guiding principle of what takes more time than the other.

It is not hard to think of a few examples that could follow the principle:

Creating demo-ware software will take 20% of the time, creating production grade software will take 80%.
Creating 80% of a system will take 20% of the time.
20% of a systems functionality will be used 80% of the time.
Resolving 80% of the defects will take 20% of the time.
Once you have done the 80% normal cases, fitting in the last 20% exceptional cases will take 80% percent of the time.
20% percent of a systems developers create 80% percent of the working functionality (Hopefully not :) )

I know these are crazy numbers. And I know this is certainly not true in a lot of cases. But still, you'll benefit if you keep Pareto's principle in mind.

Essential Software Concepts: Fail fast

2008-12-31T01:12:00.001+01:00

Software development is hard. Not only is development hard, the environment and hardware is not exactly fault proof. Let's not forget the users either, you can be sure they will do something you didn't think about. Or even the human factor in terms of deployment, database-updates and the like. So one of the few truths about software is that things will fail, the question is what you do about the failures.

You generally have two (three) options in dealing with errors: you can try to make the system recover, or you can fail at once (Or you can try to swallow the error, and cross your fingers that it will work out in the end).

Unless it is essential that the system shouldn't fail, I like to fail things fast. By that I mean, if something doesn't work properly then fail the operation as soon as possible. Failing doesn't mean crashing the application, you should provide an informative message to the user. Failing like this has, as usual, both advantages and disadvantages. The main advantage is related to tracking and fixing bugs. One of the major efforts in fixing bugs is understanding where and why it happened. If you fail early rather than late, you'll most likely have an easier task tracking the bug down. You don't have to worry about the system having worked in an inconsistent state and there's a larger chance that the stack trace actually shows where the problem is located.

The main disadvantage, or rather question to ask yourself is "is failing early fine in production?". If you can't say yes to this, then know that you'll have a much more complex bug-tracking time ahead of you. It's not an easy question though. Failing early can sometimes mean that users are unable to use important functionality because of a strict policy of failing early. Furthermore, a system that keeps failing is not good for building customer trust. Perhaps it is better to fix it in the background in some cases. Whatever you choose, it is certainly not a black or white question.

Essential Software Concepts: Broken windows

2008-12-30T00:17:00.001+01:00

The concept of broken windows is one of the concepts I remember best from the Pragmatic Programmer book. It is one of those simple concepts that you need to be aware of as a developer. But even though it is known to most developers (at least in principle), it is still violated all over the place - just to try to crank out a little more functionality. The technical debt you build up there you will pay for a long time.

I'll explain the concept, but if you don't know it, you really should go ahead and read the Pragmatic Programmer book. A quick and good read.

The concept can be explained like this: Say you have a house. As long as the house stands firm, there's a good chance it will keep that way. But as soon as a window is broken, and nothing is done about it, the downward spiral begins. Another window is broken, vandals run free, and eventually you get to the point where restoring the house it's former self is more costly than just replacing it.

This certainly holds true for software projects as well. If you keep the architecture sound and continuously work to hold the implementation maintainable; by separating concerns, avoiding duplication, and so forth and so on, and if you fix problems once you find them, you'll have a much larger chance of keeping things maintainable than if you don't.

Taking the time to implement something properly will benefit you in the long run (and I'm talking weeks and months here, not years). The additional time it will take to implement something properly cannot be compared to the cost of being bogged down in a poorly architected code. Doing changes (again, we're talking weeks and months here) will be more costly, and you don't need to pay that price.

Essential Database Practices: Parameterized queries

2008-12-29T01:17:00.002+01:00

When you access a relational database the old-fashioned way (not ORM's and LINQ), you generally have three options; you can call stored procedures, you can send queries manually as a concatenated text block or you can send queries manually as a parameterized query.

Option one and three are both valid ways of doing it, but if you ever create queries as concatenated text block WITHOUT parameters you need to stop. I'll explain why by explaining why parameterized queries are good for you.

You should use parameterized queries for three reasons:

SQL injection
Query plan caching
Query plan cache size

SQL injection

This is the most obvious. It is hopefully known to all, as much focus has been put on it in recent years. If not, have a look in wikipedia.

Query plan caching

Every time a query is executed in SQL Server, it goes through two main steps: parsing/compilation and execution. During the parsing/compilation step, SQL Server creates a plan for how it will run the query. Once this is completed, the query is put in the query plan cache and executed.

In the parsing/compilation step, before doing anything, SQL Server will look in the cache to see if the query plan has already been created. If it has, there is no need to create the plan again. Depending on the query and how many times the query is run, you can save a significant amount of time if the query is cached.

The cache check is done by a look up in a hash table where the form of entry is the entire SQL query, formatting included. The reason I listed query plan caching for parameterized queries, is that the contents of the parameters does not matter, thus running the same query with a number of different parameters will result in using the same plan each time. If you concatenate your queries manually, a new plan will be made for each call. Note that this is important to remember also for the queries you create in your stored procedures. Use sp_executesql instead of just using exec to execute your queries, then you can also add parameters to your dynamic SQL.

Query plan cache size

For each query plan created the cache increases in size. The queries go out of the cache when they are not reused, but this can take some time on a busy server. I don't think this usually has a big impact, but creating a large amount of query plans does fill up the cache size. That means they use up an unneccessary amount of your server's ram. Shame on you.

If you want to have a look at how much time the parsing/compilation step takes for a given query, run SET STATISTICS TIME ON before running the query. The actual cache can be accessed via the sys.dm_exec_cached_plans view.

New blog series: Essential this, Interesting that

2008-12-29T01:09:00.002+01:00

I recently got the idea of a new form of blog posts I think will be interesting. Basically it involves covering topics that I find to be either essential or interesting within a particular area, and labeling them visibly as such.

I hope you'll find value in it.

Happy holidays :)

Good stuff on Botnets, DDos and Scripting

2008-12-15T22:01:00.002+01:00

Rob Conery has a nice blogpost titled "The Perfect Storm Botnet".

A good, fairly short read - a bit like kids horror stories for computer geeks, except for the fact that these are true of course.

So just watch out for all those *-injection attacks kids, it's seldom pretty

Another "fagdag" done

2008-11-30T18:28:00.002+01:00

We once again had another "fagdag" yesterday, and I went away really pleased. The contents was good, but more than that, the discussions are really fruitful. I think we have some good things going there..

It was on a Saturday, so it didn't last into eternity, but here's what was covered if anyone cares:

Code Quality

Claims-based identity

Project demo: xxx Silverlight

BDD/TDD

Testing legacy code

Microsoft ORM Futures: Entity Framework V1/2, LINQ To SQL (lightning talk)

Windows Powershell (lightning talk)

Parallell computing in .NET 4.0/VS2010 (lightning talk)

Ninject (lightning talk)

XAML Power Toys (lightning talk)

Onion Architecture (lightning talk)

@Twitter

2008-11-29T22:03:00.001+01:00

Need to find out what all the buzz is about... You can find me at twitter.com/RuneSundling

SCRUM in 5 minutes

2008-11-13T22:25:00.004+01:00

Want to get the gist of SCRUM, but have only got 5 minutes?

Check out this pdf.

.NET 4.0 poster

2008-11-13T22:22:00.003+01:00

Want a fancy poster showing you the contents of .NET 4.0?

If you haven't seen it yet, you can get it here.

.NET and Parallelism

2008-10-29T23:42:00.009+01:00

I attended a really interesting session today about the current work being done with parallelism for the .NET framework 4.0.

And the future is bright! :)

You should see the session yourself, but I'll try to summarize what I got out of it anyway.

Threading and parallelism are important now, but will become ever more important in the future. Our processors have stopped getting noticeably faster, but they are continuing to become smaller and more efficient, which enables to have more and more of them. Currently most of us are running with two cores, and quite a few with four. Hardware producers are saying that 80-100 cores are withing reach within a not too distant future for regular machines. We absolutely need to design for this.

So we need to start taking threads ever more into account. But it is such a complicated field, with locking, contention etc.

The current implemenation of threads aren't helping either. Create a thread and it takes 1 mb of committed memory. Try to manage them, follow them with current debugger support in VS, etc etc - it's not easy.

First of all, the annoying Thread will be replaced by much more enjoyable Task in 4.0(The thread will live on, but the Task should take over in terms of being used.).

The much improved (compared to threads) Task API, their internal handling, and the new tools to support them and parallelism (in VS2010) looks really cool.

I can't tell you how tasks differs from threads yet, except that they are similar, but that tasks has a more advanced API accompanying them, and a different internal handling.

Consider the scenario of running a program on a two or four (or something else) core computer. What's the best number of threads you could possibly have then? The answer is of course two on the two-core, four on the four-core, and so on. (Don't design your app to run with two parallel execution paths if you have a two-core computer, you want these things to scale when you them on much more powerful servers.)

When you create a myriad of threads, you get the problem with the constant fight for resources and time. With tasks this is handled in a different way. Consider this simple scheme:

One global queue where all your tasks are created

One local queue for each of your processors

Once a task is created it is put into the global queue, and then moved into one of the processors local queue.

Each individual processor then processes each of the queued tasks in a LIFO fashion, as the last one added is the one most likely to have information cached. Once it is done with its tasks, it steals (stealing is good here) in a FIFO fashion from the other lists, attaining a few things:

None of your processes are standing lazily around

You will get into much less contention issues, since you grab tasks from different sides

Everything is executed in an orderly, efficient manner

So how do we use these tasks?

You create tasks much in the same fashion as you created threads. The difference is you have quite a few more options in how they are handled: if you want them to work together with the parent thread, if you want to crash the main thread if there is an unhandled exception on a child-thread, in creating generic versions to return a particular type of data (+++++++). Check it out yourself.

You also get more structured assistance, which looks really nice. Instead of running for instance a standard ForEach which runs through each item sequentially, you can use a new Parallel.ForEach which runs each task in parallel. This will work great if each step can be run in isolation and don't depend on each other, but even if they do, you can use the API to say stop, and then for each task run it check whether stop has been set, in practice stopping much more prematurely. With lists you can just add ToParallel() to create a paralell version of the handling of for instance a LINQ query.

I really should go on and on.

And I have forgotten to mention the new Visual Studio tools, haven't I? There are a couple of new debugger-windows which seem extremely helpful in visualising which tassk are currently running, which tasks have been created, what values the tasks have, how the stack trace of each task is, and what methods any and every tasks have hit. Hard to explain, but you'll like it!

This is just emerging, but this will make parallel life much easier in the future. I'll surely be looking much into this!

Windows Azure

2008-10-29T22:57:00.008+01:00

As you've bound to have heard by now, Microsoft recently announced Windows Azure at the PDC08, the new Cloud operating system.

I must say I'm really intreagued by this.

The possibilities are big. Think of the extreme costs you have for running and maintaining an application. For instance:

You need to be able to scale according to user interest

You need to be highly responsive - which means a server in Norway might work pretty bad when accessed from US or Asia

You need to have a server park in at the very least one place to just host the applications (if you can afford the downtime a electricity failure would cause, or possibly the loss of data and hardware a fire/earthquake could cause)

You need failover database solutions (If manually recreating a backup won't do)

Some data might not be allowed to be stored in some countries, you need to be sure that whatever solution you choose will handle this properly

Instead of these options you could get into some sort of external hosting solution. And this is in many regards what MS is offering with Azure, but I bet a fair bit of hosting-providers are literally shaking now after the new plans of MS have been introduced.

We are talking big, big scale investsments in this. The tight integration with the current MS software and process of doing things (Visual studio for instance) doesn't hurt in the process of making this popular. And it's not like they don't have quite a bit of internal expertise in this already; for instance through running the Live services.

One of the most important things are the continous expectations of ever more interactive and graphics-heavy software delivered at the blink of an eye over the internet. You can do this yourself, but I doubt the cost/value calculation will be on your size.

Another things is scaling. With Azure, or the cloud service offering, you can scale up or down according to user interest. An application with user peaks that vary over time can be tough financially, since you can end up with much more hardware set up than you usually need.

One big issue is privacy. If you host this in the cloud, you put you data in Microsofts hands. Are you cool with that?

As you have noticed, my detailed understand of Azure is lacking, to be honest I haven't bothered focusing too much on it, so much else which is intersting here at PDC! I do believe though that this can become such a big thing, and I'm expecting to have to consider it much more in the future - but for now I'll leave it to someone else.

Check out Azure here.

Microsoft completely miss the point of Agile Development?

2008-10-08T16:47:00.005+02:00

Slightly annoyed about a post in CIO about Visual Studio 2010. The contents of the article is not so interesting, but a user comment annoyed me.

"Microsoft, unfortunately, continues to show that they in fact completely miss the point of agile software development. Agility is about simplicity of design, of process, of feedback mechanisms. It is also about open, community-based tools, frameworks, and standards. MS keeps offering hilariously bloated, complex, monolithic, closed, and expensive IDE "solutions" that worsen every problem they attempt to solve. Visual Studio is now, at more than 43 million lines of code (and counting), so counter to agile development practices that I must question its architects' sanity or motives. Is all of this bureaucratic bloat forced upon the VS team by clueless marketing drones? That might explain the continuing madness."

This guy has just completely missed the point. A few things:

Agile = open, community-based tools, frameworks, and standards.
Why?? Agile is (a lot more than this, but also is) about using agile methodologies and practices to drive a project to success. Any software that can help in this endavour is great, but all I care about is using the best software. Whether it is Microsoft or Thoughtworks that delivers my CI system is unimportant to me, as long as I get to make the choice. I can use an open source IDE, CI, source control, build tool, test tool, etc. if I want to (and I often do), and that's all that matters.

MS keeps offering hilariously bloated, complex, monolithic, closed, and expensive IDE "solutions" that worsen every problem they attempt to solve.
Dude, Visual Studio is a great tool! Together with Resharper it really is a tool that rocks :)

That might explain the continuing madness
Keep up the madness! I, for one, can't wait to see what comes next.

Admittedly Microsoft wants as big a part of the pie as possible, and that can certainly lead to situations that is less than optimal, but just because everything isn't good about Microsoft certainly doesn't make everything wrong.

NNUG Oslo

2008-09-26T11:34:00.005+02:00

Just wanted to note that I've just been elected into the board of the Norwegian .NET User Group (NNUG) - Oslo.

I'm looking forward to the experience! Hopefully something good will come out of it for you too :)

Beware the enum-compilation!

2008-09-23T20:52:00.005+02:00

I was recently made aware of a real eye-opener when it comes to how enums are compiled by the c# compiler.

When you compile a c# project, each enum is internally replaced with the corresponding value of the enum. What that mean is that in the IL code the reference to the enum is replaced by the integer value in the enum.

This could give you a problem if you redefine an enum that is used across multiple dlls. Say you have an enum with two values, and you decide to switch their place in the enum definition. If you only compile the dll with the enum definition, the other dll will work the opposite of what is fairly natural to expect!

The enum tip of the day is thus:

Always specify the value of the enum in the defintion rather than using the default order.

Never replace a value with another, always add new values to a number higher than the current top

Always follow the points above. Would you like try to debug a patch creating that problem?

Definition of good quality solutions

2008-09-18T16:12:00.006+02:00

Recently I heard a new definition for what a good quality solution should be like:

robust and stable

easily maintainable

good performance

good user experience

If asked I would have listed this somewhat differently, but my point is rather:

This must be the most obvious thing. Could this possibly be necessary to say?
(Note: Was not directed at me :))

Could someone possibly try to create ..

nonrobust

unstable

hard to maintain

lousy performance

poor user experience

.. code deliberately?

I doubt it.

You won't always be able to do it - there's always compromises (maintainability+performance, time-contraints, etc), but still.

There's not really much point to this post. I just wanted to make a point. Or something.

Boo... Worth the time?

2008-09-17T22:44:00.005+02:00

One of my colleagues, Tore Vestues, has recently sparked an interest in Boo internally.

Boo is, for those who don't know it, just another .NET language. I like Ayendes definition:

Boo is an object oriented, statically typed language for the common language runtime with a Python inspired syntax and focused on language and compiler extensibility.

Today we had a voluntary Boo-night, which meant about 15 developers having a go at Boo for a while. Good stuff :)

First of all, I really don't know Boo (at least not yet), but there's one thing I totally dig about it - the way anything and everything can be put together at compile-time. You got code generation at compile-time, without all the runtime hazards of standard code-generation. It really just rocks compared to the everyday c#-code.

For instance you can define what a singleton is one place, then just add a [singleton]-attribute around your class to define for the compiler to compile this as a singleton. Or a better example Tore came up with: INotifyPropertyChanged. Anyone like writing that code over and over again? How sweet is it to get to write it once, and use it over and over again? I can think of lot's of examples were I keep rewriting annoying boring code, which isn't just boring to write, it provokes errors, and it makes the code much harder to read than a declarative style possibly could.

So what's the negative things then?

First of all, Boo is a separate .NET language. It will compile to IL-code, but you need to use it fully in it's own assembly. You can of course not mix and match. In other words, you won't get C#-code with Boo compile-time code-generation. Unfortunately :(

Number 1 once again. This is the biggest problem the way I see it. The language in itself seems really cool. But what's the chances for mainstream development? Will you implement part of your application in Boo, or everything? Can you justify the cost of learning up all developers on the project to the Boo way of doing things? (I don't know, so I'm really just asking questions) Is it mature enough or has proper tool-support? (not yet)

I guess most of the initial issues with Boo comes from the fact that it's just new (of sorts) - with the standard problems you have when something is new. There is not enough documention, tool-support, or followers.

But isn't many of the concepts I just mentioned possible in C# for instance? It is, to some extent. You have AOP-support through for instance PostSharp, which does it's magic after your code has compiled. Now I don't really know just how big a difference these two approaches make (Can someone please educate me?), but since it gets in after you compile, at least all your code needs to compile already - which removes the possibility for defining your own keywords for instance, like "print" or something.

In conclusion, Boo seems exciting, but I need more time to conclude properly. Is it worth the time? Might be. I know I definitely need to have a closer look.

Handling the object-database-problem (Presentation)

2008-08-31T14:15:00.016+02:00

A few days ago we once again had a "fagdag" - a full day dedicated to learning (and a splendid evening building team spirit afterwards of course).

I held a presentation about the different ways to handle the object-database issue, focusing mainly on the positives and negatives of the various approaches, thus, I believe, providing more value than giving any sort of tutorial on tools or approaches which can easily be googled.

The goal of the talk was to make sure that all my colleagues were up to speed on what is available in this area today, and what should be used when.

Here's a brief overview of what I talked about.

The issue is the difference between objects and relational databases. Databases focuses mainly on storage and (more or less) normalized relationships through keys, while objects focus mostly on the interaction between objects in a business domain. They typically have the concepts of inheritance and polymorphism close at hand, while databases only handles these if they must, and definitely not naturally. the approaches from the object or database side certainly don't go hand in hand, neither technically nor conceptually.

So which options do we have for handling the object-relational impedance mismatch, and what's the pros and cons of each approach?

I covered the following approaches:

Manual SQL

Stored procedures/views

Dataset

Active Record

OR-mapper (self-made)

OR-mapper (open source/commercial)

Code generation

As well as a few concrete implementations of active record and OR-mapper:

NHibernate

Linq To SQL

Entity Framework

Castle Active Record

One concept important to have in mind when comparing many of these options is Persistence Ignorance. This means the degree of which your domain model is freed from having any reference or concerns about how the entity should be persisted. A large degree of Persistence Ignorance means you can model your domain model without any restrictions or interruptive information. Note that this does not correlate with how easy the solution is to use in general.

Manual SQL

Often used with datasets or "manual" OR-mapping.

Pros
- Easy to use

Cons
- Can be hard to handle / loose overview
- You need to write lot's of code
- No indication to the DBA what accesses the database
- SQL-injection
- Query plan caching - if you run a query against SQL Server without using parameters properly, the entire query will be cached, whereas if you use proper parameters the query excluding the parameters will be cached - meaning you can reuse the cached query plan with any number of queries with different parameters.

Stored procedures / Views

Often used with datasets or "manual" OR-mapping.

Pros
- Easier for the DBA to know what accesses the database
- Well known
- Great for batches and reports
- Could be used well as a layer of abstraction
- Security if you can't handle it on the application layer or when several applications must access the database and you need a central handling of it. You should have a good reason for doing this.
- Tuning of queries

Cons
- Can be time-consuming to specify everything as stored procedures
- If business logic starts leaking into your stored procedures you're in problems
- Switch database type. Not usually an issue, but if you are planning on changing databases anytime soon, don't model everything in stored procedures.
- I believe it's more hostile to change than many other options.
- Error messages and debugging is less clear
- (Testing. This is being used by many as a reason against stored procedures. I don't agree with this, becuase stored procedures are really quite easy to unit test, as long as you keep them clear and concise and don't go superdynamic on them)

Dataset

Pros
- Rapid Application Development(RAD) support
- Very quick to get up
- Support throughout the .NET framework
- Microsoft product. Easy to get a customer into
- Quick to get going, can be sure that everyone involved in a project will know of it.
Cons
- Scales very poorly with complexity
- Often need to live with utility-classes for functionality
- The database schema often leaks down into the code, frequently even into the GUI-binding. Try changing anything there..
- Not exactly an object oriented approach
- Everything must be casted (stored as object)
- String-based access to table and column-names (Unless you're using typed datasets)
- Since they're so easy to use and can be filled anywhere, some solutions even fills them in the codebehind. Oh beware

Active Record

Row object (entity) with domain logic and knowledge of how to persist itself. 1-1 mapping towards a database table.

Pros
- Easy to get started
- Not duplicated mapping of property names as in OR-mapping with mapping files
- Mapping against database in object. Nice to have everything in one spot
- Centralised connection to the database in one layer
- Always works with entities. Automagic queries to the database

Cons
- Works well with a uncomplicated schema (1:1). As soon as your domain model won't map well with the database schema, Active Record won't do for you.
- Data centric. Active Record says your objects need to be the same as your database tables. In simple approaches this works well. But as I mentioned briefly in the beginning, what you care about when you model the database is often very different from what you want and need when you model the domain model. Thus an active record approach can produce a domain model which is unclear and bothersome to work with.
- Hard for the DBA to tune queries
- Not big on Persistence Ignorance (PI)
- Testing can be an issue because of the point above

OR-mapper

Automatic mapping between database and objects through mapping files or attributes

General
-------
Pros
- Domain centric approach. Can model the domain separately from the database (to some extent)
- Dynamic SQL-generation
- Support multiple databases (At the same time or simplify switching at a point in time)
- Good solutions have support for stored procedures and/or manual SQL in the cases where you need to do things on your own
- Fast development-time once up and running

Cons
- Complexity (will get back to it)
- Performance (will get back to it)

With mapping-files
------------------
Pros
- Separated object and mapping

Cons
- Must duplicate class name, property names etc.
- XML and pure text (Help with GUI-support, advanced plugins)

OR-mapper developed in-house
----------------------------
Pros
- Own functionality
- Debugging can be easier / possible
- Know that everything that affects a project is internal
- Cool for the one doing it

Cons
- Time-consuming
- Need to redevelop something already developed a myriad of times
- Noone will continue development and testing of your solution outside of the project/organisation
- Anyone new to the project won't know the solution

OR-mapper - commercial/open source
----------------------------------
Pros
- Relatively short startup time (compared to creating it yourself, not compared with most other approaches)
- Advanced functionality available
- Well tested in projects and scenarios
- Free bugfixing and new releases
- New project members might know the solution
- User group with knowledge of its use

Cons
- Does take time
- Can't debug on errors (Might be possible)
- (Generally) limited by the products limits
- Learning curve
- If it's not Microsoft, it might be an issue in your organisation (hope not)
- What if the product go out of production/stops further development?

Code generation

Generate data access and domain objects from database

I don't know this area too well myself, so I'll be short here

Pros
- Advanced generation scrips
- With so many solutions available, there must be something good?
Cons
- Data-centric
- Many of the same cons as with OR-mapper, in terms of complexity etc.

And then it's the concrete implementations:

NHibernate

- Most known and mature
- Port from Hibernate, well known and have existed for a long time
- Large crowd of followers
- Domain centric
- Functionality
-- Mapping files
-- Inheritance (all three approaches)
-- Stored procedure support
-- All types of relationships (1-1, 1-*, *-*)
-- Caching
-- Support multiple databases
-- LINQ to NHibernate
-Persistence Ignorance (PI). Only need to set everything virtual and have a default (any visibility) constructor

I believe it's the de-facto OR-mapper in .NET at the moment.

Entity Framework

MSDN: "Designed to provide strongly typed LINQ access for applications requring a more flexible object-relational mapping, across Microsoft SQL Server and third-party databases"

- Enterprise solution
- Several layers of mapping (Conceptual, Storage, Mapping)
- Is supposed to be more than an OR-mapper (Don't deliver on this in V1)
- Poor in terms of Persistence Ignorance
- Not very mature

But I'm positive to Microsofts entry into this arena. The fact that they don't believe datasets can solve every problem in the world anymore is a big step forward. Also, they have taken all the criticism to heart, and have started fixing many of the shortcomings for v2. They also have a blog describing the work and asking for feedback, as well as an expert group following the progress. That is good stuff. Still hope I don't have to work with it yet becuase of some fancy marketing slide though. But hopefully it does become a good contender for the OR-mapping crown, what could be better than better tools?

LINQ to SQL

MSDN: "Designed to provide strongly typed LINQ access for raplidly developed applications across the Microsoft SQL Server family of databases."
- Good for RAD-development
- Support only direct 1:1 mapping towards the database
- Only table-per-hierarchy inheritance
- Only attribute-mapping when you use the 08 designer
- Only SQL Server (Though this apparently wasn't becuase of any technical challenge.)

Castle Active Record

Pros
- Simple to get started
- Created on top of NHibernate, taking advantage of all it has to offer, as well as making it much simpler to get started
- Don't need to specify column name in the attribute if it is the same as the property name

Cons
- Standard problems with PI for Active Record
- Common to inherit from base class (although you don't have to)
- Data-centric
- 1:1

I created a couple of diagrams to help compare the four approaches in terms of Peristence Ignorance and OR-mapping in general:

When is refactoring unappropriate?

2008-08-22T14:31:00.007+02:00

In a recent meeting, our own self-proclaimed agile guru (not saying I disagree!), Odd Helge Gravalid, came up with a good suggestion on when refactoring can be really troublesome.

If you are working on a "legacy project", a system in production, lot's of ugly code, no structure - in other words refactoring heaven, refactoring can actually be dangerous. Say you do refactor lot's and lot's of the code, and at the same time, someone else need to do a fix and they bring in a bug in the code. If this is patched into the live system, and you try to find out what went wrong, you have a problem.

Why? Often you can simply start out by doing a code-comparison in these cases, to see what has been changed. Try doing that after a big refactoring (Refactoring here normally includes removing unused code, etc. Not really refactoring, but for lack of a more complete word....)

I'm not saying that you shouldn't refactor in these cases. Just be aware of the risks involved. And hopefully you do write a complete test suite for the code before you refactor it, making it a lot easier to track down any problems with both the refactoring and any new functionality.. (Remember: Typemock + legacy code testing = true)

MSTest ExpectedException doesn't support System.Exception

2008-08-21T13:36:00.002+02:00

This isn't an informative post, as I like to keep it, but MS just got me really annoyed.

The MSTests ExpectedException attribute, where you put in the exception you expect will be thrown, does not support System.Exception, only derived exceptions!

How crazy is that?

Obviously you should try to have as specific an exception as possible (And I follow that, must be why I haven't seen it before), but it's perfectly right to throw a System.Exception if a more detailed is not possible. Hey, it's even the one MS praises in every .NET 2.0-> certification... (ApplicationException earlier)

Refactoring of the persistance-solution in agile projects

2008-08-19T08:46:00.003+02:00

I invited my fellow colleagues to a discussion today about a topic I felt could make for some interesting learning: Refactoring of a persistance-solution in an agile project.

By that I mean, when/how/if do you refactor away the solution you have for communicating with the database. The general possibilities we reached was

You can drop the whole solution and create a new one after a few years. Evidently this happens a lot more often than you’d expect.

You hide the refactoring in a new big task, often accompanied by some cool buzzword

You try to isolate the old solution away, and start fresh with new functionality

A few more I’ll cover below

I was surprised that few had been in the situation where replacing the persistance-solution was deemed necessary. I don’t have the illusions that any non-developer (-strong) organization could see the real value in doing something like that, but I thought I’d get more examples of projects where that was the only solution. Say you run the msdn-friendly dataset-strong application which works great in the beginning, gets the GUI up quickly, and have a few initial iterations delivering great functionality and a GUI which rapidly fills with functions. Great stuff! Until you reach a level of complexity which just breaks the design completely down. What do you do then? A few possible solutions:

Explain to the customer that you need an iteration or two to redesign the application to accommodate the newfound complexity.

Quietly include the refactoring within the subsequent iterations tasks, far lowering the work delivered to the customer

Postpone everything, plan instead to get around to it once everything quiets down

Have separated things properly, making it easier to change from one solution to another

From the initial phase of the project, settling where we are going and the time it will take, using your experience to choose a better solution, perhaps NHibernate or Castle Active Record, and hopefully your domain-model will never got into this extreme a situation (with contant refactoring)

Hah, the object-relational impedance mismatch is obsolete; object databases will save the world!

First of all, there so no single answer to any of this, as with everything in our business – it depends. It depends very much on what kind of solution you are working on and what kind of environment you are working in.

Obviously number 1 isn’t going to go down well with the customer – “you’ve had such a good flow so far, just keep it going” is not a surprising answer. And that’s quite understandable, and really correct in many cases. You should have applied constant refactoring to your solution, enabling you to steer clear of that situation. Of course, if you make a msdn-friendly dataset-app in the first place, you probably know that it isn’t going to handle the most complex of tasks, so you’ll never get into this situation. I bet it does happen though, and has happened more than I can imagine (I hope not).

Number 2 is the one you should have been doing all along, except now it will far halt the current work. Perhaps that is okay. You could get to an 80% finished solution quickly, and ask if that is ok. If it is, then the customer will have gotten his money worth of application quickly. If not, you can be clear to the customer that to get this and this, it will cost exponentially more than it did before. Cause we all (should) know the Pareto principle, or 80-20 rule, 80 percent of the job takes 20 percent of the time (or cost), and vice versa. Again, the customer might not be too happy about that either.

Number 3: postpone doing anything until later. You’re already in a pretty dire situation, so postponing is not likely to do much to improve on that. If you don’t know what to do, then I guess just keep doing the same is probably the best – but if you get to that point, I don’t want to have anything to do with you anyway :)

Having separated things properly (4) will give you a good basis when you get into this situation. Hopefully it won’t be too costly to make the change to tougher concurrency-issues, more advanced business logic, etc.

As long as you didn’t go with a big design up front, but leveraged experience to choose a clever framework to start on, good for you. I currently fall very easily into the Domain Driven Design with NHibernate/Active record-camp (5), using a proper domain model and everything. Not nearly as fast as the dataset approach, but more applicable in the situations I’m usually in (The dataset-approach can be great though, don’t overdesign a simple application – there’s nothing ¬with more framework support from top to bottom in .NET than datasets!)

Slightly sarcastic with the object database-title here, I just have a feeling someone who falls for one without having much experience with the other side of the table (relational), could possibly proclaim it as the new(ish) silver bullet. Now I’d love to try using an object database, because it seems you do get around many of the issues you need to consider and constantly work on with the traditional relational database approach. However, being in the real world, you mostly have to live in an existing business environment, where relational databases are the big G. Converting the object database at the end of the development cycle is a possibility, but beware of pitfalls, some intelligent colleagues found quite a few (Perhaps they could write about it soon?).

You should always be aware of the total cost of ownership (TCO) of course, knowing that the initial development of an application is only a small part of the entire cost of it, maintenance being the biggest thief. But you know, in the real world, the one funding or driving the application development isn’t necessarily the one paying for the maintenance, so… Hopefully you’ll have someone who has a bit more professional integrity than that, but it’s no wonder shortcuts on that side are taken.

One of my colleagues, Sverre Hundeide, came up with a nice suggestion. The contents should be clear to all, but the conclusion has value. An OR-solution has a few possibilities in how it interferes with your system

Persistance ignorance - Nothing in your domain-model assembly references anything concerning the persistance solution used

POCO – For me, this is the same as above, but he means this has been defined as almost the same, except there can be some references in the assembly, extra metadata, etc.

IPOCO – The domain classes can inherit from a base class used for persistance

Code-generation – The domain and persistance code is generated from the database, often highly coupled.

Basically, the higher you are on the list, the easier it is to change to another solution. Simple, but an important thing to have in mind.

I guess it’s time for the conclusion now. Or it should have been anyway, except I’m not quite sure what I covered. Just Another blog post with a lack of well arranged contents. You’ll have to make do for now :)

SQL: Convert list of values to comma-separated list

2008-08-15T21:25:00.002+02:00

Ever had a list of values in SQL, but wanting them in a comma(or anything)-separated list?

Just do something like this:

SELECT c.co_companyName,
substring(List, 1, datalength(List)/2 - 1) AS 'Employees'
FROM co_Company c
CROSS APPLY (
SELECT e.em_employeeName + ',' AS [text()]
FROM em_Employee e
INNER JOIN co_Company c2
ON c2.co_Id = e.em_co_Id
ORDER BY 1
FOR XML PATH('')
) AS Dummy(List)

If you had a a company-table and an employee-table, you'd end up with a list of the companies in the table, with a column with a comma-separated list of all the employees.

By the way, apologies for how absurdly ugly this looks as plain text.

VS Add-in: Extending Sealed Classes (without Extension methods)

2008-07-23T02:40:00.025+02:00

If you are ever in the situation when you have a .NET Framework class with most of the functionality you want, but not really doing everything to the extent you need, you have a few options:

You can inherit from the class and extend as you see fit

You can create a utility class to perform the extra functionality

You can use extension methods to add functionality to the class

You can create a new wrapper class which holds the class as an internal variable and create wrapper-methods for whatever methods you need from the class you are extending, and add any extra functionality you need.

Inheriting from the class is likely the best approach. Unfortunately, about 40% of the .NET 2.0 Framework classes are sealed, which of course means you can't inherit from them.

The second approach is becoming somewhat obsolete, as it can lead to a sort of functional programming which takes you away from proper object-oriented programming. It makes things just a bit harder to find and understand.

Number three, extension methods, is the new cool kid on the block. Add new methods just like magic to any class. The possibilities are enormous! The problem is that you can't do anything with the internals of the class, you can only add new isolated methods.

The last approach can be the way to go if extension methods won't do. This way you'll have complete control over what you want to be public and you can extend and improve on existing and new functionality. There are two problems with this approach. Number one is clarity and number two is identity. If you create an extension of the DataSet for instance, it never is a dataset. You can, in a way, get around the issue by inheriting from one of it's base classes or interfaces though (Using that as a common base).

Be sure only to use it when no other approach is better. The clarity/identity issues can be confusing for someone seeing the class at first. But I do think the wrapper/composition approach have some merit, especially considering the extreme amount of sealed classes that exist. I made it because I was annoyed to find that there was no support for doing something like that besides typing it in manually. And if you have a lot of methods to wrap, that means a whole lot of time typing nonsense methods, not to forget the comments you should add if you want it understandable. No way I'm doing that unless I'm really keen on wasting a lot of time.

That's why I've created a code-generation add-on for the last approach. It's a Visual Studio Add-in to help extend sealed classes.

I’ll begin with showing some results, and then I’ll get to how and why it works/makes sense/you should use it. In this example, I'd like to extend the StringBuilder class. To do that I need to write something like:

Once I press SPACE or TAB while at the end of this line, the add-in starts its work on using this StringBuilderExtender class to extend System.Text.StringBuilder. The result is this:

More examples here and here.

What happened is this:

Added an internal StringBuilder instance

Recreated all public constructors, methods, properties and fields, and made sure all of them use the internal StringBuilder instance.

Added any available comment from the StringBuilder class.

Changed returntypes from StringBuilder to StringBuilderExtender where necessary.

Added the Serializable-attribute, since StringBuilder is serializable.

Listed the interfaces StringBuilder implements in a comment next to the class, in case you'd like to implement the same.

Added the System.Data namespace to the using statements if it did not exist.

Why would you want to do this?

You really want to extend one of the sealed classes.

You need new functionality in a sealed class, and you either only have .NET 2.0, or extension methods just won’t do it.

You reuse existing functionality, making a quick browse of the code enough to understand the majority of what the extended class does. Compare that to reimplementing a larger part of the functionality of a sealed class.

Why would you not want to do this?

You are somewhat pretending to extend a class you cannot. This can get you into problems with equality and comparison. The commented interfaces do make it simple to extend any common interfaces though.

Potential performance hit. You do add some overhead, and internal performance tweaks through for instance Win32 code could have has less effect.

More on the internals

In terms of what you can visually do

You need to specify the keywords for the add-in on the class line. The input must always be in the form specified above:
- public class StringBuilderExtender SealedClassExtender(System.Text.StringBuilder)
- or in other words
- [visibility] class ClassName SealedClassExtenderKeyword(TypeFullName)

You can use both SealedClassExtender and the short form scx/SCX as keywords.

The TypeFullName is, obviously, the full name of the type.

You can also extend a class by specifying the path to the assembly. The form is then
- public class StringBuilderExtender scx(System.Text.StringBuilder, C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorlib.dll)
- or
- [visibility] class ClassName SealedClassExtenderKeyword(TypeFullName,FullAssemblyPath)

Apart from the list above it does a few more things in the background

Adds a default constructor if one isn’t specified in the base class.

With any method, property or field that returns the base type (StringBuilder), it returns the extended type instead (StringBuilderExtender). A new private constructor with the base type is added if necessary to fulfill this.

Static classes does not get an internal instance but just point directly at the class.

Lots more fun things to fix every special case.

If you do not specify a path it will try to resolve where to load it from by itself. It will currently do this by first trying to load the type from memory, then specifically checking all assemblies in the current application domain, and then trying to resolve the name of the assembly from the types full name. It will first try to look in the current runtime directory, then in the executing assembly path.

The comments are loaded by using the XML-document corresponding to most framework dlls, as comments are not part of the information retrievable through reflection.

Be aware though, the add-on will rewrite the entire document, so only add using-, class- and potentially namespace-declarations, anything else will be overwritten.

It doesn’t handle everything though. Specifically it doesn’t do:

Generics

Events

Identity-handling (Trying to guess how you want the identity-part handled will probably just end up with a bad guess, you’ll have to do this however you want it. Methods for equals, GetHashCode etc. are thus not created.)

Limited attribute support.

It also currently loads all assemblies into the current app domain while running. This is just because of a time-constraint on my part, and I’ll fix that.

If you have gotten this far, send me a note if you see a use for it, just don't get it, or have any comments.

Usage

To use this, you first need to install it like any other add-in. Copy the files in the zip to the correct Addins folder, for instance \My Documents\Visual Studio XXXX\Addins. (If the Addins folder does not exist, just create it yourself) Once this is done, restart visual studio, and you can enable the add-in in the "Tools->Add-in Manager"-dialog.

Downloads

Download the 2005 version from here.
Download the 2008 version from here.

Entity Framework V2.0

2008-06-25T14:26:00.004+02:00

Microsoft has started the work on Entity Framework V2.00 now. Apparently, this version will have a great deal more transparency during the process, making sure it ends up as something the users want.

Good stuff.

NHibernate, beware?

Update:

A petition against the current quality of Entity Framework has begun. Read it to get a brief overview of the issues, and add your signature if you agree.

PropertyNamesGenerator (source)

2008-06-10T23:46:00.006+02:00

I recently wrote the post Replace strings for property names with type-safe version

This is the source code for the project it describes. I'm posting it because of a request, so hopefully you'll have use for it.

PropertyNamesGenerator

Generates type-safe versions of property/field names.

Generates C# or VB files

The only parser implemented is for parsing NHibernate mapping-files

The NHibernate parser handles regular classes as well as subclasses.
Another parser which will probably get there is for regular domain classes.

Warning: This is not production-grade software, but a simple project I made for myself to handle a current need! This should not be used as a guideline for good design, nor expected to work flawlessly. No such thing have been attempted.

You can download it here.

The zip includes the source, the current binary, and a sample bat-file to use it.

How to use it:
To run the generator, start it with something like:

"C:\path\PropertyNamesGenerator.exe" /lang:cs /files:"../../source/Domain/*.hbm.xml" /out:"Output"

An explanation of the various params can be found by running it with a -? switch. It prints:
Run like: PropertyNamesGenerator.exe /files:"path/*.hbm.xml" /out:"path/PropertyNames/" /lang:cs
Parameters:
/Files [Mandatory] - Must have both a directory and a search string.
/Out [Mandatory] - Must specify an output directory.
/lang [Optional] - Output-language. C# or VB. (C# is default)
-V - Verbose output.
Note: All directories can be relative.

It currently only supports searching for files in the directories explicitly specified, i.e. not subdirectories. This is a concious choice.

To integrate it with your build, typically use a bat-file in the same manner as the Generate.bat example file included in the zip.

To integrate it with your build, you can add something like the line below to your Domain Pre-build event: (Project/Build Events)
call "$(SolutionDir)\..\Tools\PropertyNamesGenerator\Generate.bat"
Note that you will need a full path to the PropertyNamesGenerator.exe-file in the bat-file to use it like this.

It will generate files on the form [ClassName].[lang] in the output directory specified. Using a PropertyNames folder to put all of these in is recommended.

You will of course have to include these file in your solution to integrate them. Do this every time you have added and parsed a new hbm-file.

Persistance-solutions and characteristics

2008-06-05T00:00:00.011+02:00

The other day I invited my colleagues to a discussion about persistance solutions. The goal was to cover the various options for persistance in a software project, the pros and cons of each approach, and to increase everyone’s knowledge about it.

The approach planned was to identify the possible solutions, identify important characteristics to help compare each solution, and to go through each possibility.

The different types we identified were:

Code generation

Object relational mapper (custom made)

Object relational mapper (commercial/open source)

Row data gateway (Which would be active record with domain logic)

Table data gateway (Dataset)

Stored procedures/views

Manual SQL

Object database

And the characteristics in no particular order:

Development time

Flexibility

Performance

Stability

Complexity

Refactoring

Expertise

Effect/limitation/intrusion on system

Rapid Application Development (RAD)-support

Attachment to data type

Amount of developers

Company demands/guidelines

What you are creating

I think it is best to start off by defining the characteristics; what they are and why they are important.

Development time
- The time it takes to get your persistance solution ready to use and understood. This can either be the time it takes to configure and use an existing solution or the time it takes to develop your own.

Flexibility
- The flexibility of the system in terms of handling the various storage challenges in a project. This also includes handling a system which inevitably grows and changes.

Performance
- Obviously performance is important, right? This includes how the system performs in general, how it scales, how it can help handle special cases. A word of caution though: Even though performance is essential, don’t optimize prematurely. In any system of reasonable size, you will have special cases that will become bottlenecks. However, with a profiler and the possibility of fine-tuning those cases, you should be quite alright.

Stability
- You need to know that the system won’t break down under normal usage or high peaks. Testing is essential for this.

Complexity
- How complex is it to use and understand? How much help do you get with error messages? What do you when you need to debug? Is it too complex for the average developer on the project, making people take (eventually unmanageable) shortcuts?

Refactoring
- What sort of help do you get when the need for refactoring appears (often)? How hard is it to do changes?

Expertise
- You always need to take into account the expertise and skill level of the people on the project. This is more important that selecting “the best” solution.

Effect/limitation/intrusion on system
- What does it take to integrate the persistance solution with your overall system? Will your domain model be POCO? Do you need to inherit from a base class? Implement an interface?

Rapid Application Development (RAD)-support
- What kind of GUI-support does the solution have? If you need something quick and dirty, what do you do? The winner here should be pretty clear (If not, you’ll soon find out)

Attachment to database data type
- How close is it tied to database data types? Can it for instance handle the types of several databases?

Amount of developers
- How many developers the project has. This could influence the complexity of the solution, or the time available to get it up and running.

Company demands/guidelines/policy
- Company policies/limitations are a part of normal life for a consultant, and often there is little you can do about it. The company might have policies against using anything open source, demand that all database access should go through stored procedures, etc.
- What can you do about this? Unless you have a small project or are able to influence the architecture group, chances are you’ll have to manage. I guess you can pray that the architecture group doesn’t consist of a bunch of non-coding architects that selects technologies out of marketing slides.
- Another point I need to raise here: If your company still live in the “don’t use open source”-world, chances are that most decision-makers are a bit out of sync with what is happening in the .NET world today. Agreed, just a few years ago things were pretty thin in terms of open source projects on .NET, but today things have most definitely turned to the better.

What you are creating
- The most important point! Make sure whatever you choose will work for your project. Unneeded complexity is expensive! But remember that going from one solution to the other is often hard, so beware the danger of a project growing on a bad solution; you’ll have a tough time getting out of it

That should cover the basics of what is important to think about before choosing a persistance solution. Let’s get to the actual solutions!

Code generation

This involves generating the data access and potentially partly the domain model from your database schema. You specify rules for how you want things generated. There are a couple of potential problems with this approach.

If the software has a binary solution specifying the mapping, you’ll have no way to map two differing changes, which in effect means you can only have exclusive checkout. In a multi-person project this is highly unproductive, and don’t forget that you’ll be unable to use branches, as you can’t merge them later.

The other is that once you want to add a field or property to a domain object, you’ll have to update the database structure and generate the files before you can use it. This does take some time. I’m not saying that most other approaches are must faster; the annoying thing is that you can’t simply add properties etc. during testing for instance.

I have to admit that I have limited knowledge of this type of persistance-solution, but I believe this approach could be successful. There is no reason why the general mapping solution between the database and domain objects should be any poorer than with a general OR-Mapper. It depends what extra functionality is available and how everything is implemented (works). By this I mean the SQL generated, how the session is handled, lazy loading, how it affects the domain model and system, if it supports queries or SQL/stored procedures for special cases, type safe support for anything and everything, maintainable/understandable script files, and a few more I probably forget at the moment.

Object relational mapper (custom made)

An OR-Mapper is a piece of software that handles the problem of transforming your domain objects data into its equivalent form in the database. Your domain model is made of objects and pointers; the database model is made of rows, columns, keys and relationships. These are very different ways of handling data.

By creating your own OR-Mapper you will have to handle the problem of mapping from the domain model to the database for starter. This is just a small part of what you need to handle though, and to make it clear => There are slim chances that creating your own OR-Mapper will be beneficial for your project. Doing it will be time-consuming and error-prone, you will reinvent the wheel without need (See Object relational mapper (commercial/open source)), you will tie up your best resources for a substantial amount of time, you will have to implement a lot of added functionality to make it functional. Oh, and I doubt any project will wait before the mapper is finished before starting the rest of the development - which means some other form of temporary solution must be created.. (An object database might be the best choice in such a scenario.)

However, as a colleague commented, it is probably a dream for most developers to do it. Why? First, it’s complex, so you’ll learn heaps by doing it. Second, you don’t have to relate to the business side, so you really just define most of the tasks yourself. Perfect or what?

Object relational mapper (commercial/open source)

This is the same as above, except you use an already existing solution from a vendor or open source project. I recommend this approach.

Why create your own OR-Mapper when great solutions already exist? If you choose a premade mapper you will (potentially) get:

Shorter development time – You will need time to configure, use and understand the solution. This is time-consuming as well, but far easier than creating your own. Getting started should also be fairly easy, even though the tougher concepts need more time.

There’s a chance that present/new developers have used the solution before

A solution tested in many projects, which means less bugs and more working features.

New versions at no development cost.

You need to live with a few risks though:

Harder to customize and debug. You’ll have to use time on strange error messages which can make little sense.

The company might stop development and support, or the developer base of an open source project could die out. If you create your own OR-Mapper, the main developer(s) could quit as well.

In general, getting something of the shelf is far cheaper than building it yourself, as long as your requirements are met. I believe that this solution most of the time far surpasses any positive effects of building it yourself.

Picking a mapper at random just by browsing some marketing slides are not the way to go. Make sure you thoroughly read specs and user feedback, or the best thing if possible: talk to people that has experience with what you are considering, if it is a popular solution they shouldn’t be too hard to find.

There are a range of solutions in this field, but personally I’ve had the pleasure of working with perhaps the most known of them: NHibernate. NHibernate is an open source OR-mapper, which is a port of the well known OR-Mapper Hibernate from the Java world. NHibernate is a great piece of software, it has been used on numerous projects, and quite a lot of information is available through blogs and forum.

I’m not going to list all the reasons to use NHibernate, a quick search on the web should give you that, but a few things:

NHibernate lets you have (almost) POCO objects (You need to mark all persistable fields/properties as virtual. NHibernate subclasses your objects with the virtual proxy pattern, to give you lazy loading etc.)

NHibernate has mapping files to map between your domain object and database. These enable easy modeling of inheritance, collections, etc.

It is mostly type safe (with NHibernate Query Generator at least), except when you need to write advanced queries. (You could use something like my PropertyNamesGenerator though)

You can use Hibernate Query Language to create special queries in the cases where performance isn’t good enough.

Automatic lazy loading

A few bad things as well:

There is some overhead involved, and performance has been noted as an issue on several occasions. I think this is more of a design issue with your average developer though. If you try to retrieve gazillions of data from lots of tables, you can’t expect it to be lightning fast. Proper table and index design, as well as good use of lazy loading should get you well under way. For the special cases where you do get a performance issue – use a profiler to see what the issue is, use query analyser to look at the SQL, use HQL or use manual SQL behind a well designed layer to access the data, use DTO’s to limit the data loaded and sent…. There are plenty of possibilities

NHibernate has some strange error messages. Before you get to know it enough, you’re bound to use a few hours trying to figure out error messages which doesn’t make much sense. I base this on experience with the 1.2 version, not the new 2.0 release.

If you don’t want to go down the open source way, you’ve probably been (or will be) introduced to Microsofts new OR-Mapper Entity Framework. It’s Microsoft, so it’s bound to be good, right? (…)
Note: I have only read about Entity Framework, and thus my experience (both good and bad) are of questionable quality, so make sure you do your own research before making any conclusions.
The pros:

Integrates nicely with LINQ (But Ayende has a project going to bring this to NHibernate as well

Microsoft helps bring the concept of an OR-Mapper to public knowledge, which really is good

With Microsoft’s size, there’s a good chance that the project will continue. This depends upon how many uses it of course; it might get shut down like Microsoft’s music service did.

Developers versed exclusively in the Microsoft world have something besides datasets and manual ADO.NET to use.

The cons:

They’ve built a completely new product - which means you should think they should be able to leverage the experience of already existing OR-mappers. You’ll need a good amount of resources to do this though. For some reason it doesn’t like Microsoft has quite lived up to this.

Having read quite a few blogs about Entity Framework and spoken to Microsoft employees about it, I must say I’m initially skeptical. At least for enterprise development. According to one Microsoft employee, Entity Framework is only believed to have a third of its user group in the enterprise software world; the rest is your simple application developers. The needs are quite different.

One of the aspects that alarms me quite a bit relating to enterprise development and Entity Framework (or really any modern type of development – read: with source control), is that until recently the mapping files were unmergable. (The XML created were put in a “random” order. A small change could lead to big changes in a document) I can’t for the life of it understand that Entity Framework could be designed as anything but a play version if that wasn’t an important design point from the beginning. Ayende had a post about a meeting he had with Microsoft about this point, and based on his reactions it seems likely that the team defended this decision. It seems Microsoft has improved this feature after the range of reactions on it, as mentioned here

Explicit lazy loading. You have to explicitly say that you want to load a lazy-loadable collection. I think this sounds mostly annoying, as you need to fill your code with logic of testing if a collection has been loaded, and then explicitly loading it, compared to the NHibernate way of automatic lazy loading by simply using it. There is a good thing about it - the fact that you won’t get unexpected database calls from the GUI-layer because you forgot to load everything you needed. This could lead to a performance hit and other problems. I don’t believe that merits this solution.

Even though I’m skeptical about the current quality of Entity Framework, and would recommend using NHibernate instead, I’m positive to Microsoft’s general move into this realm. With their funds, future versions have the potential of becoming really useful, with hopefully seamless integration with the rest of the framework.

For now I’m mostly scared that I’ll be put on a project where its use will be mandatory. Unfortunately, we’re still in a world were non-Microsoft software is looked at with skepticism from many of the decision makers in companies.

Row data gateway (Which is Active Record if you have domain logic)

This is the same as having a gateway which gives you objects per row in the database. If you add domain logic to these objects you have what is called Active Record. I’m going to concentrate on the Active Record approach, as I can’t see a good reason why you’d want to have a simple row data gateway in .NET. Active record is a domain object which handles persisting itself.

Active Record has the advantage of being simple and quick to implement. It is not hard to understand, and is a good way to make a quick prototype while retaining a domain model. It breaks down once your database gets complicated, and once you don’t have a one to one mapping between an Active Record object and a database table.

You can use Castle Active Record to do this. It is built on top of NHibernate. In terms of refactoring away from Active Record if the complexity increases, apparently there is a way to automatically go from Active Record to a full OR-Mapper NHibernate solution automatically. I haven’t tested this though.
(You’ll probably end up with a NHibernate solution through Castle then as well. This is not a bad thing! Castle integrates very well with NHIbernate through its NHibernateFacility, and easily allows for instance a Dependency Injection approach as well. I’ve written about it in a previous post)

Table data gateway (Dataset)

In the .NET world, table data gateway is the same as the dataset-approach.

The major benefit of using datasets is the unmatched framework support for it, where creating a datagridview and datasource, using databinding, adding a navigator to handle paging, etc., is extremely simple and powerful. For quick demos, or Rapid Application Development (RAD), nothing can match it.

There are two main problems with this approach

It really doesn’t scale. Once you start adding business logic, you’ll have problems with the lack of object orientation, with the lack of type safety, with duplication of logic, and lots more.

Expectations, if you use it with prototypes. If you give the business side a quickly running demo with this approach, you’ll get into problems when you try to explain to them how long it will really take to build.

If you know you are creating a very isolated, not to be extended, solution, by all means use the dataset approach – nothing can match it in speed or simplicity. If there’s a chance you need to add more to it later, opt for another alternative. You’ll have a hard time refactoring it later on.

Stored procedures/views

Putting everything in stored procedures/views is another approach that has been used. I’m not going to bother saying much about it, as it’s not really a viable alternative. You’ll be better of using this approach than manually concatenating information into SQL queries though (Like avoiding SQL Injection attacks)

It is a possible approach if you have special cases where you just can’t make the performance demands without using stored procedures.

Manual SQL

Don’t bother. But if you have to, at least limit its use to a database layer. And make sure you remove hazardous characters so you avoid SQL Injection.

Object database

An alternative quite unlike the rest. Whereas the previous sections concentrated on ways of working with a relational database, you also have the option of using an object database. If you choose this approach you can just pass your objects to and from the database.
Again, I haven’t tested this, but I have very experienced colleagues who have little but positive things to say about this approach.
The pros of this approach includes

No need to map between a relational design and a domain model

Don’t need to update a schema several places

According to some benchmarks, they can be superior for certain kind of tasks. It has been said that they are very efficient at specific queries, while they are slower at more general queries.

Most of the object databases also support some sort of query language when the need arises

Some even fully support SQL, but I have no idea how this works in practice.

The cons

Practical knowledge of these are still fairly limited

Hard to access from other parts of the company network, for reporting purposes for instance

One approach some of my colleagues took was to go through the whole development period with an object database before converting to a NHibernate solution before going live. The conversion into a relational database was only done because of company demands. This was still a success, but if you do this you have the danger of not quite knowing how long it will take to set up the final solution or exactly how the data will perform.

I look forward to testing this in a real world project, you should too.

Conlusive thoughts

In summary I’d say that you should

Use the table data gateway/dataset approach if you have a short and sweet application

Use Active Record if you need to get results fairly quickly, and have a close relationship between your table and domain structure

Find out whether a commercial/open source OR-Mapper or code generation tool or an object database suits your needs best - if you need a somewhat complex application

Build your own OR-mapper if you’re forced to, it’s not good for the project, but you’re lucky :)

Replace strings for property names with type-safe version

2008-06-04T23:19:00.008+02:00

At times you have the misfortune of having to write property/field (I’ll just call it properties from now on) names as strings, for instance in advanced NHibernate queries. (In general, NHibernate Query Generator avoids this for you in most cases, but there are still some times it cannot be used. Say inheritance for instance). If you’re a bit slow on progressing in mocking tools, this might be a problem as well (It shouldn’t still be though…)

The big problem with non-type safe strings is refactoring, which should happen all the time on your projects. Property names will change, and when these are used as non-typesafe strings in your system it has a few unfortunate consequences:

You could break something without knowing it. Worst case is you won’t find it until production

The knowledge above could restrain you from doing refactorings.

You will have to spend time doing text-searches in the system to find out if and where it is used.

So for a personal project I’m currently playing with, I figured I wanted to do something about it. So I created a small generator project which generates static classes where you have access to the domain objects property names in a type safe manner. What this does for me is:

If I do changes to a property of a domain object, and that has been used as a (previously) non-type safe identifier, my compiler will complain.

I can do refactorings all the time without worry.

For now it looks for instance like this in use:

This will return “Text”, which is the name of a property in the Resource class or a base class.

I’ll be the first to admit that this is not ideal. Property names as strings are not something you want to deal with, but at times you have no choice, and this feels like the better of the two options.

For those interested in how I chose to solve it:
The generator takes in an input path, a search string and an output path. This is put in a bat-file which is called from the domain projects build. Currently I’ve only implemented a parser for NHibernate mapping files (HBM), which for each class creates a file with each property listed for that class. (I can see situations where you would want to use properties not just used in persistance, but that’s all I need for now). Inheritance is handled by duplicating the properties of the base class in the subclasses.
In short, the file created for the example above looks like:

Regular Expressions in Visual Studio – the top-down approach

2008-04-09T22:20:00.010+02:00

Basically every approach to regular expressions I’ve seen uses the technique of learning all the symbols first, then using that to create various regular expressions. Let’s call that the bottom-up approach. I figured I could add some value by doing it the other way around – by showing specific strategies where regular expressions make sense, showing the regular expressions needed, and then explaining how it works. In other words, the top-down approach.

Once you’ve read this post you should be able to use regular expressions to do pattern matching and extraction in Visual Studio. I’ve kept the number of regular expression meta-characters to a minimum, (hopefully) making it easily understandable as well. You’ll be far from an expert on regular expressions after reading this, but hopefully I can either help get you interested in regular expressions (You should!) or learn some tips about using it with Visual Studio.

I’ve tried to divide the contents into logical sections, so just skip whatever doesn’t sound useful.

On Visual Studio

The built in regular expression support in Visual Studio (2005) is fairly strange – it doesn’t quite follow conventional regular expression syntax, it doesn’t follow the .NET Framework syntax, and irregular behavior has been found. It’s not better that it seems extremely slow once you give it fairly large files either.

Besides its drawbacks, it has become the place I use Regular Expressions the most. And trust me, I like using Regular Expressions. So why use Visual Studio? The probable reason is that I use it on a day to day basis, and since it’s not uncommon that I use it for searching, I guess it just became natural to start using it for other tasks as well.

Oh, and by the way: There’s lots of regular expression software out there you should give a try. Free, good software, with somewhat more standards-based meta-characters, probably faster, likely to have more functionality related to regular expressions, and various other improvements.

Enabling Regular Expressions

Regular expressions are used from the “Find and Replace” dialog

”Match case” also applies to your regular expressions

The "Find in files" "Find and Replace" dialog (ctrl-shift-f) enables you to search only certain file types

The "Find in files" "Find and Replace" dialog shows all results in the Find window, instead of just finding the next, as with the "Quick Find" dialog.

The most used regular expression meta-characters are available in the button next to find once you enable regular expressions.

I’ll add the actual dialog Visual Studio displays here just for reference. It describes each meta-character fairly well.

Regular Expression Strategies

General

Ever after learning regular expressions I seem to always find good places to use them. Here’s the list of strategies(or common situations) I’ve covered in this post:

Extract content from lines with a common pattern

Remove empty lines (optionally with whitespace)

Remove lines not following the pattern you are looking for

Add characters around each line

Retrieve the contents of all lines containing a pattern

Regex strategy #1 - Extract content from lines with a common pattern

Since this is the first example, I’ll walk through this one extra slowly

Description: You have a number of lines of contents that have some sort of similar text you want to extract.

Example: We have two lines of text, and want to extract the value in the target part of the xml.

<xml target="test" />
<xml info="" target="test2"/ >

Steps:

Identify a unique string in the lines straight before the content you want to extract
- target="

Identify a unique string or character on the other side of the content

The major part of the job is done; just create the regular expression to use the two first points :)

The regular expression we begin with to match the parts we identified above (Don’t test it yet, I’m skipping some important details still):

target=" [^"]*"

Splitting this up we get three parts: target=", and " are just plain text matching. [^"]* is different though. If we take a look at the picture above, we can recognize two meta-characters:

[^] – Any one character not in the set

* - Zero or more

So [^"]* means match a character which is not a ", zero or more times. All regular expressions are greedy by default, so it will try to match as many characters as possible.

To try to show this clearly I will differentiate the different parts of the regex and what it will match by using font styles.
The regex:

target=" [^"]*"

Will match:

<xml target="test" />
<xml info="" target=”test2"/ >

We’re not quite there yet. First of all we need to escape all non-letter and non-number characters with a \ to make sure they are interpreted as plain text characters. I didn’t do that above as it would have made it harder to understand at first. We get:

target\=\"[^\"]*\"

Harder to read, but necessary. You’ll get used to it.

We’re now able to match the text we want in both lines. However, we’re still unable to retrieve the information from the fields. To do that we first need to match the entire line, so that we can remove the parts we don’t want. To do that we add .* to the beginning and end of the expression:

.*target\=\"[^\”]*\".*

The . (dot) is a special character which is interpreted as "Any single character" (Except line break). The regular expression will then read:

.* - Match as many characters as possible (Up until the (last) target\=\" part of the text

target\=\" - Match the plain text target=",

[^\"]* - Match as many characters as possible until we reach a "-character

" - Match the "-character.

.* - Match as many characters as possible. (Since this is the last one, it means to the end of the line)

With the above expression we match the entire line. The final part we need is a way of extracting the information we want. We’ll do that by adding { and }. This is basically a grouping construct, and you can have any number of them in your expression.

.*target\=\"{[^\"]*}\".*

We now have the entire regular expression we need! Let’s test it. Copy the example xml text into a text document and open it in Visual Studio. Open a "Find and Replace" dialog, and add the regular expression. Then add \1 in the "Replace with: " part. \1 means the part you have between your first { and }. In other words:

Find: .*target\=\"{[^\"]*}\".*

Replace: \1

The result is:

test
test2

Success! :)

Final comments:
Interested in how the regular expression engine actually does the matching? Let’s see what it matches for each part of the expression, until we reach the final match.

.* - Match as many characters as possible. This will actually match the entire line. Since the .-character doesn’t match a line break it stops at the end of the line (in other programs with other options, it is possible to make it match line breaks as well.) Remember, it is greedy, so it wants to match as much as possible.

target\=\" - Match the plain text: target=",. Now, to be able to fulfill this requirement, it has to "let go" of some of the matched characters. So it let’s go of one and one character until it finds that it can match the string. That’s why, if there had been several target="-parts in the line, it would have matched the last one.

[^\"]* - Match as many characters as possible until we reach a "-character. Match one and one character, until it reaches a "-character.

" - Match the "-character.

.* - Match as many characters as possible.

In fact, different regular expression engines do the matching in a different way, but this is all you need to know to understand how it works.

Regex strategy #2 - Remove empty lines (optionally with whitespace)

Description: You have a document with several empty lines you want to remove.

Example: : We want to get rid of the empty line between the two words.

SomeText

SomeOtherText

Steps:

All you really need is the following:
- Find: ^:b*$\n
- Replace:

The meta-characters used here means:

^ - Beginning of line

$ - End of line

:b – Space or tab

\n – Line break

When splitting the regular expression into parts it can be read like this:

^ - The match must start from the beginning of the line

:b* - It will be followed by zero or more (as many as possible) whitespace characters.

$ - The match must end at the end of the line. In other words, the line must contain either no characters or only whitespace characters to give a match.

\n – Finally we match the line break for the line as well.

This regular expression will match empty lines, including the line break. By replacing them with nothing you get the effect of removing the lines.

Final comments:
Make sure you have a line break after your last line, or it won’t match the regular expression, as the regular expression requires a line break. Optionally you could have appended a * to the \n, making the expression *:b*$\n*. This would have removed the need for the final line break.

Using Excel to sort the data is a good alternative to remove empty lines. That won’t work if you don’t want the data sorted of course.

Regex strategy #3 - Remove lines not following the pattern you are looking for

Description: With the apparent weakness of regular expression matching in Visual Studio, in that you need to replace the entire line to retain the information you want (instead of getting the matches in a separate window), it is not uncommon to get into the situation where you have lines that you need to remove to be able to focus on the lines containing the information you want.

Example: We got some more XML, this time with comments we want to get rid of:

<!—xml comment -->
<element>
<!—another comment -->
<subelement />

Steps:
The general strategy here is to do a (preferably) two-step action to

Mark the lines you don’t want with a special identifier

Remove all lines with the special identifier

The actual steps:

Identify a unique string or recurring pattern in the lines you want to remove
- <!

Write the regular expression to identify and then add the identifier to the line
- Find: ^.*\<\!
- Replace: #\0

Remove the lines containing the special identifier
- Find: ^\#.*$\n
- Replace:

With point 2, you want to single out all the lines you don’t need (Preferably in one operation, but not always possible). If you have a unique identifier across the entire document, then where you match in the string and add the identifier is unimportant. Often, placing it in the beginning or end of the line is a good starting point.

The first find operation (^.*\<\!) should be easy to read now, as there are no new characters. But for the sake of it: Match start of line, match as many characters as possible until we match the last set of <! in the line.

The replace operation is slightly different. # is the special unique identifier we have used here. It has no special meaning – you could have used an x, or three x-es for that matter, as long as it is unique for the entire line or position throughout the document. Whereas \1 meant the first specified match we found (marked by { and } on each side), \0 holds a copy of the entire line that had a match in it. In effect this means that we simply add a # to the beginning of each line matched.

With point 3, we want to remove all lines with the identifier. So we find each line with the special identifier in the beginning, match the rest of the line with .*$\n, just as before, and then replace it with an empty string.

Final comments:
Another possibility is matching the lines you want to keep, tagging them with the special identifier, and then removing the lines not containing the identifier. Of course you will have to remove the special identifier in the end, so there’s one more replace involved. Not a big thing though.

A third possibility is to use strategy #5 – Retrieve the contents of all lines containing a pattern. This is most useful when you need to match across several documents.

Regex strategy #4 – Add characters around each line

Description: At times I seem to end up in the situation where I have lines of information that I need to surround with information or characters.

Example: We have a number of lines of information which need to be used in an SQL in-query as strings, and thus need the necessary surrounding characters. The SQL query is: SELECT * FROM something WHERE name in (…). For those unfamiliar with SQL syntax, we want to add a ‘ to the left of the expression and ‘, on the right side. The information:

Test1
Test2
Test3

Steps:
As long as none of the lines involved already has the necessary characters and no characters that will invalidate the statement, all we need to do is:

Find: ^.*$

Replace: ‘\0’,

The find simply matches everything on each line, from start to end character.
The replace puts our plain text characters on each side of the expression. The result is:

‘Test’,
‘Test2’,
‘Test3’,

You’ll need to remove the last ,-character yourself, and paste it into the SQL query, making it:

SELECT * FROM something WHERE name in (‘Test’,
‘Test2’,
‘Test3’)

Final comments:
Another common usage is creating SQL inserts. The logic is just the same; just add some other information around.

Regex strategy #5 - Retrieve the contents of all lines containing a pattern

Description: You want to get the contents of all lines matching a certain pattern, either because the lines make sense for themselves, or because you want to do further work on the lines. This can of course be for one or multiple documents.

This is an alternative to using strategy #3 (Removing lines not following the pattern you are looking for). Strategy #3 is probably faster when you work on a single document.

Example: Reusing a previous example contents. This time we want to retrieve all of our XML-comments:


<element>

<subelement />

Steps:

The regular expression needed to find the correct lines aren’t really the important part here, as what we wan’t to show is how we go from there. We don’t even need to use a regular expression in this case, just search for 
- C:\aPath\file.txt(3):

Now copy the results from the “Find Results”-window and paste them into a document.

We need to remove the path information. Use the following:
- Find: ^[^\:]*.[^\:]*.
  - ^ - The match must start from the beginning of the line
  - [^\:]* - Match zero or more (as many as possible) non-:-characters.
  - . - Match the :-character
  - [^\:]*- Match zero or more (as many as possible) non-:-characters.
  - . - Match the :-character
- Replace:

Now we remain with only the lines we’re interested in.

Where do you go from here?

If you really want to learn about regular expressions, you’ll need to get Mastering Regular Expressions (Jeffrey Friedl). The book is THE book on regular expressions.

If you for some reason don’t want to get the book, a series of 10 videocasts from Zain Naboulsi (Is this thing on) should be your second choice. Besides being well made, it focuses only on .NET, which Mastering Regular Expressions only does partly.

IoC Containers generally, Castle Windsor specifically

2008-02-22T19:44:00.034+01:00

First I’ll have a go at reinventing the wheel by trying to give my own definition of an IoC Container

An Inversion of Control (IoC) Container is a non-invasive configurable intelligent factory-component

Splitting this definition in parts, we get

Factory, because it is responsible for creating objects for you.

Intelligent, because it understands what dependencies you have, and create them for you recursively.

Configurable, because you can configure the usage through code or configuration files.

Non-invasive, because the objects used doesn’t need to know about the container.

An IoC Container really makes the Dependency Injection pattern a lot better. Why? I’ll look more into that throughout this post.

My previous post is a foundation for this one, as it explains the Dependency Injection pattern, and gives it a place in the world. I’ll use no time on that here. This post exists to build on that content, and show what more benefits you can achieve by using the dependency injection pattern with an IoC Container. This isn’t meant to be a comprehensive introduction to IoC Containers or Castle Windsor, but should give you a good idea of what it can do for you. Once you’ve read it, just continue on with what I’ve added in “References and further reading”.

I’ll use a simple example to help explain how it works. We have a client, which uses a service, which again depends on IStorageObject, which have two specific implementations. This is not meant as a real world example, but bear with me. The dependencies and sample implementation are shown below:

My previous post shows how we can handle this manually, and the pros and cons with the different solutions. We would like to get an instance of client, but we need to handle all the dependencies it has as well. If we have to write this manually, it will turn out as something like this:

This doesn’t look too bad, and the dependency injection approach is often better than a lot of alternatives. But think of it. As soon as this starts expanding, I can bet you that you’ll have a mouthful.

Now to do the same with an IoC Container, you can typically write:

For the container to create Client, it needs to create an instance of Client, but also an instance of both IService and IStorageObject. The container will understand that it needs to create both, and does so depending on how you have configured it. More on that later.

What if we add another dependency to the service implementation? The manual code would have to change to something like this (Each and every place service is created in the code. Resharper could help you quite a bit, but it’s far from a solution to the real issue.):

Already guessed how it would look with the container approach?

You are right, just the same :)

This has a number of advantages:

Easy to swap the implementation of an interface - just change the configuration.

If you find that Client suddenly needs another dependency, you don’t have to rewrite any of the Client creation code.

Removes explicit dependency-handling throughout the system

Single point to find out how objects are instantiated and connected

Easy to begin using Client in other places. (Less boilerplate code to write)

Additional benefits of tunneling dependencies through one source: Decorators, Interceptors, Proxies, as well as handling object lifestyle. Have a look here for a very good example.

See my previous post for more on the general benefits of dependency injection.

I am going to have a look at an IoC Container called Castle Windsor now. There are many other alternatives as well. I can’t speak too much about the others as I only have real life experience with this. If we talk only about the general dependency injection problem/solutions, the different projects have a lot of similarities, with some differences in support for config vs code, need for attributes etc.

Configuration and use - with XML

You can use a separate configuration file, or add the castle definitions to your app/web.config.

The configuration above states:

The string identifier of our only component is “storageobject”

The service (or interface) the container will create (if any - you don't need to specify one), is IStorageObject, from TheNamespace namespace

The specific implementation of this interface is DatabaseObject.

In practice, this means that if the container is ever asked for an implementation of “storageobject” or the IStorageObject interface, directly or through resolving dependencies, it will return an instance of DatabaseObject..

This code simply shows how the container would be started with this approach.

Configuration and use - with code

You can define everything in code:

The pros and cons of using XML versus code is that you can do changes without recompiling with the XML approach, while it is type safe when you specify it in code. Use whatever is best for your situation.

Lifestyle

Advanced handling of the lifestyle of objects has never been so easy. By adding a keyword, you can specify how your object lifestyle will be:

The previous configuration updated with specification of lifestyle:

Facilities

Adds integration with other tools

NHibernate

ActiveRecord

WCF integration

Remoting

Logging

Startable

Custom

Other

Castle has a number of other possibilities as well, like specifying what to do when objects are created and destroyed (Commision and Decomission).

Note

The last couple of points feels kind of pointless without giving them much extra attention, but I'll leave that to other people who have described it much better. A good starting point would be this or this.

References and further reading

Inversion of Control and Dependency Injection with Castle Windsor Container - Part I (Simone Busoli)

Inversion of Control and Dependency Inject: Working with Windsor Container (Ayende)

A not so interesting post, with much more interesting comments.

Castle Windsor tutorials (BitterCoder's Wiki)

Inversion of Control Containers and the Dependency Injection pattern (Martin Fowler)

Castle Project

Conclusive thoughts

Although a short introduction, I hope I have captured enough information to get you interested. I’ve been using Castle for quite some time on my last project, and I am very happy with the results. I advise you to give it a try!

_

Objects, dependencies and patterns to manage it

2008-02-19T20:44:00.033+01:00

The purpose of this post is to pass on knowledge about an important aspect of software development, with the hopes that someone else can have use for it. It is a presentation I held at work changed into written form.

The topic is managing dependencies between objects; how it affects the systems we build, and a few thoughts on different ways to handle it. We are going to look at three patterns: Factory, Service Locator and Dependency Injection, which in various ways help us control the dependencies. The purpose here is not to explore all the details of the various patterns, but rather to capture how the patterns handle these situations in general.

As a way of comparing the patterns, I’ve put together a few well known principles that has important things to say on the quality of software.

Simplicity

Complexity is so important to handle. Complex code is hard to read, change and manage. The challenge is to write code that people understand, not just code that compiles.

Separation of concerns (SoC) / High cohesion

Software that separates its different parts well is easier to understand and manage. The more a unit of code,which from now on should be understood as either a module, class or method, handles only a separate piece of the puzzle, the better. The Single Responsibility Principle fits well here: One unit of code should do only one thing, and do it very well.

Loose coupling

How much your unit of code depends on other parts of the system. Less coupling means code that is easier to understand, maintain and reuse in different contexts.

Ease of change

Highly related to all of the points above. We want code that is easy to change. The reason for this is of course that there are few environments were things change more often, be it because of business or technological needs, or really just realizing the nature of most software projects.

Encapsulation

Design choices that are not important to know of outside of your unit of code, should not be visible there either. It only increases complexity, makes things harder to change, and removes focus from what is really important.

Testability

Highly testable code has a number of benefits for the quality of your system, and is an important principle to live up to for the vast majority of projects.

I have created an example we will use throughout the post. It consists of two elements, a client and a service. The simplicity makes it impossible not to understand, as well as hopefully easier to relate to a number of situations, though making it harder to show real world problems and their solutions directly.

The service could be anything from an in-process calculator to an expensive web service or database service. The context has a lot to say for the conclusions, and it is vital that you relate the examples that will follow to the many similar situations you see on a daily basis. Depending on whether we would expect change here, we might or might not include an interface.

This is an example of the implementation of Client (in C#):

Potential problems with this code

It is hard to change, because it is connected to a concrete implementation of the service in the constructor.

You need to read the Client code to find that it has a dependency to Service.

Testability would be hurt if this was a resource-expensive or unreliable service.

Potential advantages of this code

It might do exactly what it needs to do - If it seems natural that Client should be responsible for creating Service, if there is no benefit of changing the implementation of Service in different situations, if testing isn’t hurt by not accessing it, then by all means keep it as simple as you can. YAGNI (You ain’t gonna need it) and KISS (Keep it simple, stupid) are important principles to follow.

As a side note concerning testing: It is possible to test this with TypeMock.

What if we increase the complexity of this example? Let's add an IStorageObject, which is a common interface for two ways of storing data, DatabaseObject and CacheObject. As the diagram below shows, Service now has a dependency towards DatabaseObject (And to IStorageObject, though not shown explicitly).

And the implementation:

As simple as this example is, we already have a potential for tight coupling. Even though we are coding to interfaces, this is not that easy to change. Reusing Client or Service in a different context is very hard. Replacing the implementation with a mock or stub during testing is in most cases impossible. And even though this example hides it, both Service and Client is now responsible for both creating and using the dependency, responsibilities we often want to separate (e.g. because of complex initialization). In real life, the complexities and couplings are often increased tenfold, and only then can you see the true value of these principles.

Pattern #1 - Factory

The factory pattern is one of the GoF patterns, a creational pattern, responsible only for creating objects. I have created a dead simple implementation of the factory pattern, where you have a ServiceFactory class with a static method to give you the concrete instance you want.

The dependencies can be modeled like this:

Advantages

Client only has a direct dependency on IService now, not the concrete initialization.

The code is more testable, if we have a factory where we can switch the concrete instance with a mock object.

The usage of Service will be clearer and easier to start using as all creational logic has been separated.

Having a single place to get instances of a service means you have more general possibilities, like the transparent handling of instances (always new or singleton). I will look at others throughout this and the next post.

We have separated the responsibility of creating and using the service. If creating a service means several manual steps to configure it, then this is even more valuable.

The Factory could return a different implementation based on input, a common reason to use a factory.

Problems

In our situation, we still have a compile time dependency on the specific IService implementation. The dependency has only moved a step out.

Reuse of Client in different contexts is hard, because of the still tight coupling to Service.

If we look back at the example with the deeper dependencies. Should we make another factory to handle this creation? Has this really loosened the coupling that much, and do we want to use all our time writing boilerplate factory-code?

We still have to read the implementation of Client to find that it has a dependency towards Service.

That being said, let’s look at how it lives up to the different principles listed in the beginning

Simplicity

The factory pattern is easy to use, and is one of the most known patterns. Thus, using it will help give clarity to what you try to achieve, even to developers less versed in the "patterns world". Though it is not correct in every situation, and too many factories is not necessarily a good thing either.

Separation of concerns (SoC) / High cohesion

The creation and use of a dependency is separated.

Loose coupling

The example still has a compile time dependency on the concrete implementation of IService. In many cases the factory pattern can help with the coupling, for instance by sending in some form of input.

Ease of change

Reusing Client in a different context is hard because of the still tight coupling.

Testability

If the implementation of the factory allows you to switch implementations, then yes, it is more testable. You need to initialize the factory with every test however, and even switch back again to the previous implementation to make sure that you don’t break other tests.

Pattern #2 – Service Locator

An (potentially global) object that gives access to services the application needs.

The example changes to:

Instead of the factory class we now have the service locator that creates and/or returns the service. The biggest change here is that you now have an assembler class which typically configures the Service Locator with the correct instances.

Two points concerning the implementation

In the example above I have used a string to get the correct type of service. This might as well have been solved with sending in a type or using generics if you want type safety and support for refactoring, at the cost of some flexibility though.

The example uses a Service Locator interface to separate the LocateService method from the methods Assembler uses to register services. The implementation is still a singleton, and could be changed to not use GetInstance first.

Advantages

Encapsulates the logic to find services.

A separate interface to deal with.

A central point of control. Easier to do similar things with the services you get, like handling instances and caching.

Can add a mock implementation during testing.

Problems

Hard to handle deeper dependencies well.

Advanced initialization of objects, which the factory does so well, is hard.

Possibly code to reset the service locator to make sure that one test does not break others.

And onto the principles:

Simplicity

Easy to use. A single method to call.

Separation of concerns (SoC) / High cohesion

The creation and use of a dependency is separated.

Loose coupling

If you have an implementation that uses an XML file to configure up the specific instances, then there is no compile time dependency.

Ease of change

As there is less coupling, it is easier to change as well. Reusing Client in a different context is still often troublesome though.

Testability

This pattern does make it even easier to test than Factory, but you still have to write extra code to create and change the service locator.

Pattern #3 – Inversion of Control / Dependency Injection

A fancy word for a really simple principle: it simply means injecting dependencies.

As you can see in the images above, Client now receives the specific implementation of IService from the outside.

Before going further, I think it is necessary to write a couple of lines about the confusion concerning the two concepts Inversion of Control and Dependency Injection. Inversion of Control is a common concept in computing. It relates to a situation where you instead of taking the active approach and asking for something, the framework is the active part that calls you. (i.e. events, the startup method of an application, etc) Dependency Injection is a specific form of Inversion of Control, where instead of asking for a specific dependency (As was done with Factory and Service Locator), the dependency is handed to you.

There are three types of Dependency Injection: Constructor, Setter and Interface injection.

Constructor injection
- what is shown above, injecting the dependencies in the constructor.

Advantages

Any user of Client will see that it has a dependency on IService.

Unlike the other injection approaches, Client is completely initialized once created, as you can’t create it without giving it an instance of IService.

You can set IService as ReadOnly, since it is set in the constructor.

Problems

If you have optional dependencies, especially if they are resource-intensive to create, then it is not a good idea to initialize them in the constructor.

If you have many optional dependencies, and want to give a separate constructor for each valid case, then you soon end up with a lot of constructors.

Setter injection
- Injecting the dependency through a property.

Advantages

Good way to handle optional dependencies, especially if they are resource-intensive.

Support for Dependency Injection in legacy code without having to change the constructor, for instance to improve testing, or to add new functionality.

Problems

Harder to know what you need to do to use the object, which can lead to surprises.

Somewhat harder to track down exceptions.

Interface injection
– a special form of setter injection: Specifying injector methods in an interface.

As this is almost identifcal to Setter injection, there isn’t much else to say about it. The only difference is that it can help separate out what are really dependencies that must be filled. However, this seems to be the least used form of dependency injection.

Advantages of using Dependency Injection in general

Easier to test: You can simply inject a mock.

Potential for less coupling, as it doesn’t ask for a specific implementation.

Easy to reuse Client in different contexts, because of the point above.

Not necessarily compile time binding. And even if so, different implementations of Client can easily be bound to different implementation of IService.

Problems

You’re not supposed to inject everything. One problem is that you can break encapsulation, another that you should take care not to create interfaces for all objects in your system. Coding to interfaces is a good thing, but making interfaces where you don’t expect change or testing isn’t a good enough reason, will make it hard to see where you truly expect change in the system.

It can be cumbersome to do Dependency Injection manually. I look at a solution for that in my next post.

So how does Dependency Injection live up to the initial principles?

Simplicity

Simple to write and understand. But writing manually can be time-consuming.

Separation of concerns (SoC) / High cohesion

The creation and use of a dependency is separated. Also, Client only uses the dependency now. Compared to the other patterns, Client doesn’t even need to ask for an implementation.

Loose coupling

The code has a potential for even less coupling, as it only uses it's dependencies.

Ease of change

Easy to change in terms of using Client in different contexts, but manual dependency injection code isn’t any easier to change.

Encapsulation

Don’t break encapsulation where it’s not natural.

Testability

By far the easiest way of testing objects where you need to add a mock implementation.

References and further reading

Inversion of Control Containers and the Dependency Injection pattern [Martin Fowler]

A few of the posts on a longer discussion of Dependency Injection: (You can find more from each)
Dependency Injection
Dependency Injection: More than a testing seam
Defending Dependency Injection
Dependency Injection - Keep your privates to yourself

Dependency Injection - MSDN Magazine
Guice video. Some good general points as well, if you're bored with reading.
Applying Domain-Driven Design and Patterns (Book) - On Dependency Injection and Service Locator

GoF (Book) - On Factory
Head First Design Patterns (Book) - On Factory

Conclusive thoughts

At the outset of creating this presentation, I was planning to only write about the Dependency Injection pattern. However, I felt that putting it in a bigger context, and trying to present the different patterns in a more uncommon way would be more helpful. This is by far perfect, but if it can help anyone better see the usage of these patterns and part of their place in the world, then it was worth rewriting this for the web. Not the least, if anyone can further educate me as well, what could be better?

This was part 1 of the presentation. Part 2 handles IoC Containers, and how you can do Dependecy Injection without having to do it manually.