eWorld.UI - Matt Hawley

Ramblings of Matt

Mercurial Conversion from Team Foundation Server

March 16, 2010 09:12 by matthaw
In addition to blogging, I'm also using Twitter. Follow me @matthawley

One of my many (almost) daily tasks when working on the CodePlex platform since releasing Mercurial as a supported version control system, is converting projects from Team Foundation Server (TFS) to Mercurial. I'm happy to say that of all the conversions I have done since mid-January, the success rate of migrating full source history is about 95%. To get to this success point, I have had to learn and refine several techniques utilizing a few different tools to get the job done. Before I jump into the meat of the post, there are several setup tasks that need to be done first.


Configuring Mercurial

Mercurial comes with a pre-packaged extension, convert, which supports nearly all major version control systems (Subversion, CVS, git, Darcs, Bazaar, Monotone, GNU Arch, and yes - even itself). Because this is an extension, you need to enable it in the Mercurial configuration. Open the Mercurial.ini file located at C:\Users\<UserName> and add the following lines

convert =

Once this is saved, you can test if this extension is working by typing the command hg help convert If things are configured correctly, it should display the help information regarding the convert extension.


Conversion Setup

Mercurial's convert extension allows you to have somewhat full control when performing the conversion. These options include mapping usernames, including/excluding certain paths, and renaming branches. These configuration options are to our advantage since TFS requires Active Directory and stores all commits in the format of DOMAIN\User (though this format is perfectly acceptable, it's not ideal). Utilizing the ability to map usernames, create a new text file named auths.txt and start adding your mappings:

CONTOSO\Mark = Mark

It is only necessary to add username mappings that appear in the TFS source history. Later, you will see how this username mapping file is used.


Utilizing SvnBridge

The first approach I take for doing project conversion for CodePlex, which has a success rate of 85%, is using the hosted SvnBridge. From the command line type the following

hg convert --authors auth.txt https://xunitcontrib.svn.codeplex.com/svn xunitcontrib

If the conversion works, your project will be successfully converted to Mercurial. It is recommended that you view the log history to ensure everything is in order. Should anyone continue to check in sources, just re-run the conversion on the already-converted Mercurial repository, and it will convert anything new.


Utilizing A Suite of Tools

Should the hosted SvnBridge not work, or you have TFS hosted elsewhere, the process is not nearly as straight forward and requires the use of several tools. Download and install the following

  • tfs2svn - This converts TFS history into Subversion history
  • VisualSVN Server - Used as the intermediary Subversion repository store
  • TortoiseSVN - Used for storing the Subversion credentials as well as admin tasks

After you have installed all of these tools (and probably rebooted your machine), follow these steps.


Step 1: Create an empty repository in VisualSVN Server. This is where the history from TFS will be migrated to as an intermediary step. Make sure that you do not select Create default structure (trunk, branches, tags). If you accidently check this option your import via tfs2svn will fail because it is expecting an empty repository.


Step 2: On the newly created repository open the Manage Hooks dialog by Right Clicking Repository -> All Tasks -> Manage Hooks... Edit the Pre-revision property change hook by entering a single carriage return. This is necessary for enabling hook which allows tfs2svn to rewrite the history in Subversion. If the hook is enabled correctly, it will become bolded in the list of available hooks. Click OK to discard the dialog, and restart the VisualSVN server by Right Clicking VisualSVN Server -> Restart


Step 3: You now need to add a user account to VisualSVN server so that tfs2svn can authenticate and import the history. In VisualSVN, right click Users and select Create User. Type in a easy-to-remember username and password and click OK. How tfs2svn operates, is that it uses cached Subversion credentials for the import. The easiest way of caching your credentials, is checking out your new Subversion repository. When prompted for credentials, ensure the checkbox Save Authentication is checked.


Step 4: Launch tfs2svn and start filling in the connection information to your TFS and Subversion servers. Once the information is correctly filled out, click the Convert button and wait while it starts extracting the history from TFS and importing it into your Subversion repository.



Step 5: Once the tfs2svn process has been completed, you can view the history of the Subversion repository. One thing you'll notice, is that tfs2svn prefixes all commit messages with "[TFS Changeset #12345]". There are also some instances where tfs2svn will add "tfs2svn: " as a commit message as well. If you don't care if your Mercurial repository will have these messages, skip to step 6 - otherwise continue on.



To remove the erroneous commit messages, you will need to rewrite the Subversion log using administrative tools. The process follows

  1. Get an XML log of the repository history
  2. For each logentry in the XML log get the msg content and remove the messages
  3. Create a temporary file writing the updated log message to it
  4. Call the setlog command on the executable svnadmin passing in the path to the temporary file in step 3.

As you can see, this is a tedious process - so I automated it with a Powershell script (downloadable here). The script assumes that your VisualSVN server can be accessed at https://localhost/svn/<RepositoryName> as well as the physical path of your VisualSVN repositories are located at C:\Repositories. Once you have updated the script, place it in %My Documents%\WindowsPowerShell, open a Powershell command prompt and type rewrite-log RepositoryName

Note: If you view the Subversion log again, you may notice that the message for each revision still contains the content we were trying to strip off. If this is the case, don't worry - the history has been rewritten, and you may be viewing a cached version of the repository.


Step 6: Now that you have your repository migrated to Subversion and you have rewritten the log , you can now convert it into a Mercurial repository. Using the same syntax as SvnBridge conversion earlier, type the following on you command line

hg convert --authors auth.txt https://localhost/svn/xunitcontrib xunitcontrib



Once completed, you will have a full history of your TFS repository converted to Mercurial! You can now start using your local repository immediately or push the history to a central repository for others to start using.


kick it on DotNetKicks.com

VisualHG: A Mercurial Plugin for Visual Studio

March 15, 2010 12:03 by matthaw
In addition to blogging, I'm also using Twitter. Follow me @matthawley


imageMercurial is quickly gaining momentum in the open source world, and the need for great tooling to make developers lives easier is always essential.  Most developers using Mercurial know of the the explorer shell plugin, TortoiseHg, but what many don't know about is VisualHG. In summary, VisualHG is

  • A source control plugin for Visual Studio (works with 2005, 2008 and 2010)
  • It sits on top TortoiseHg exposing common commands in Visual Studio
  • It tracks file status changes automatically indicating the state in Visual Studio
  • There's absolutely no SCC bindings!
  • It's absolutely free!

An occasional gripe we hear from CodePlex users, is that when they download and open a project from CodePlex that contains SCC bindings, they quickly get annoyed by the Visual Studio warnings of working temporarily uncontrolled. This is why the second to last bullet point is even listed, it's that important! The mere fact that it's free is just a bonus :)


To get started working with VisualHG is very simple.

  1. Download and install TortoiseHg and VisualHG.
  2. Open Visual Studio and go to Tools -> Options.
  3. In the options tree view, select Source Control. (You may need to click the Show all Settings checkbox)
  4. Select VisualHG from the drop down list, and click OK.
  5. Open your Mercurial based solution to see the plugin installed and determining your files' statuses.

So, why am I writing this post? Well, I wanted to highlight and recommend a great Mercurial based project hosted on CodePlex that we, the CodePlex team, use every day. Don't get me wrong, a few of us still use Mercurial from the command line (myself included), but I wouldn't even go without having VisualHG installed for simply tracking file status changes (necessary when doing lots of refactoring).


kick it on DotNetKicks.com

WikiPlex v1.3 Released

March 11, 2010 17:53 by matthaw
In addition to blogging, I'm also using Twitter. Follow me @matthawley


It's been a many months since the last release of WikiPlex, but its only because there hasn't been a lot of churn recently.  I've very happy where WikiPlex is at, and it continues to be a very integral part of the CodePlex website!


Here's what you'll find in WikiPlex v1.3:

  1. Documentation - This new documentation includes
    1. Full Markup Guide with Examples
    2. Articles on Extending WikiPlex
    3. API Documentation
  2. Video Macro - This macro was updated to support Channel9 Videos.
  3. Syntax Highlight Support - One more language has been included:
    1. {code:powershell} ... Your Powershell Code ... {code:powershell

This time I did what I promised two releases ago, provided some good documentation. I even went so far as creating another open-source project called WikiMaml which will take wiki syntax and convert it to Sandcastle MAML output. The project isn't full proof, and not where I want it to end up, but it is working great within WikiPlex to generate all of the non-API documentation. As always, if you have any ideas for new macros or extensibility points, please post them in the issue tracker and I'll make sure to look at them!

Now, go and download WikiPlex v1.3!

Extending WikiPlex with Scope Augmenters

March 10, 2010 10:40 by matthaw
In addition to blogging, I'm also using Twitter. Follow me @matthawley


This post is long overdue, but as I'm preparing the v1.3 release of WikiPlex and working on documentation (yes, I did say documentation) I realized that another extension point with WikiPlex was not covered. This last, important (but rarely used) part is extending with Scope Augmenters. Scope Augmenters allow you to post process the collection of scopes to further augment, or insert/remove, new scopes prior to being rendered. WikiPlex comes with 3 out-of-the-box Scope Augmenters that it uses for indentation, tables, and lists. For reference, I'll be explaining how and why the IndentationScopeAugmenter was created.


Why Its Needed

The IndentationMacro allows for an arbitrary indentation level indicated by the number of colons that are placed a the beginning of the line. Let's take a look at the primary macro rule:

new MacroRule(@"(^:+\s)[^\r\n]+((?:\r\n)?)$",
              new Dictionary<int, string>
                 {1, ScopeName.IndentationBegin},
                 {2, ScopeName.IndentationEnd}

As you can see, we're capturing any number of colons at the beginning, but our ending scope knows nothing of the how many defined levels there are. If you can imagine, knowing nothing about your the beginning scope when rendering to correctly render the ending is not a trivial task without context. This is the exact reason a Scope Augmenter is used, it has that context.


Ultimately, we would like the input

:: I am content

to be rendered as

<blockquote><blockquote>I am content</blockquote></blockquote>


Create a Scope Augmenter

Scope Augmenters can be as easy as you need to make it, but can also be fairly difficult - point of example, the supplied ListScopeAugmenter requires a complex algorithm to correctly identify levels, nested levels, and determining when to start new lists. When creating a Scope Augmenter, it will take in the associated macro, the current list of scopes, and the current content returning a new list of scopes to be rendered. In your solution, create a class called IndentationScopeAugmenter and extend it from WikIPlex.Parsing.IScopeAugmenter. You'll then implement the Augment method.

   1:  public IList<Scope> Augment(IMacro macro, IList<Scope> capturedScopes, string content)
   2:  {
   3:      var augmentedScopes = new List<Scope>(capturedScopes.Count);
   5:      int insertAt = 0;
   6:      for (int i = 0; i < capturedScopes.Count; i = i + 2)
   7:      {
   8:          Scope begin = capturedScopes[i];
   9:          Scope end = capturedScopes[i + 1];
  11:          string beginContent = content.Substring(begin.Index, begin.Length);
  12:          int depth = Utility.CountChars(':', beginContent);
  14:          for (int j = 0; j < depth; j++)
  15:          {
  16:              augmentedScopes.Insert(insertAt, begin);
  17:              augmentedScopes.Add(end);
  18:          }
  20:          insertAt = augmentedScopes.Count;
  21:      }
  23:      return augmentedScopes;
  24:  }

The Indentation begin / end scopes always come in a sequential pair, which is why I'm able to grab the begin and end scope in lines 8 and 9. Next, you'll see that we need to determine the depth to indent, so we grab the beginning content (which ultimately will be a string containing only colons). In line 12, we count the number of colons there are, which gives us our depth count. Lines 14 - 18 are adding the N-1 listing of IndentationBegin and IndentationEnd scopes. The method then returns this, newly augmented, list of scopes. Basically the augmentation goes from






Registering a Scope Augmenter

Just as registering macros and renderers, there is (only) a static endpoint. Since augmenters should not rely on runtime dependencies, there is no runtime equivalent of using scope augmenters. When you register a Scope Augmenter, it is always associated with a single macro type, and during parsing, the WikiPlex parser will query for the Scope Augmenter that is associated with the current macro being used. To register your Scope Augmenter, have the following code in your application startup method

ScopeAugmenters.Register<IndentationMacro, IndentationScopeMacro>();

When you call the WikiEngine.Render("content"), it will automatically pick up all registered Scope Augmenters and use them during parsing.



You now have a fully functioning macro / augmenter / renderer that will take an arbitrary depth and have it render correctly. As previously stated, WikiPlex also ships two other Scope Augmenters, ListScopeAugmenter and TableScopeAugmenter, which have a bit more logic associated with them. While Scope Augmenters allow you to manipulate the list of scopes prior to rendering, they should only be used in certain situations in which you cannot capture the correct set of conditions or are unable to contextually determine rendering based on separate scopes.


Download WikiPlex now!

Copyright © 2000 - 2014 , Excentrics World