TeamCity 4.0.1 is released

Christmas in Russia is celebrated in January, so this gives us an excellent opportunity to work and even release TeamCity in the middle of Christmas holidays in US and Europe.

Here you are. TeamCity 4.0.1 is out.

Our tracker can let you know the full fixed issues list.

Merry Christmas and Happy New Year!

4.0.1 EAP, build 8111

Here is a new build to open the EAP program for 4.0.1 bug fix update.

Search results highlighting, build queue and notification calculation speedups, possible (but rare) deadlock fix - these are some of the changes in the build.

Currently we do not make any major changes focusing on important or small changes in the code, so the build should be quite stable.

BTW, did you notice what's special in the build number?

And the winner is...

Build number 8080!
Congratulations, you've just been promoted to the official TeamCity 4.0 release build.

The most advanced continuous integration for Ruby on Rails with TeamCity 4.0

TeamCity 4.0 has a dedicated rake runner, with support for Rake tasks, Test::Unit, and RSpec tests.

Our special thanks goes to the guys behind RubyMine and Ruby plugin for IntelliJ IDEA, who created this runner for TeamCity.

Here is a short (3.5 min, 7.3Mb) demo of some unique TeamCity's features and how they work for a Ruby on Rails project.

This demo includes:

• setting up Rake runner build configuration
• running a build, with progress estimate for subsequent builds
• ongoing test reporting while build is running
• viewing changes in version control, which are included into the build (web diff)
• "first failed in"/"already fixed in" features for failed tests
• full sortable test list for a build
• a graph for test duration time

View demo

Read more about TeamCity features and what's new in 4.0 or download it now and see for yourself!

Regards,

KIR

A chance to try almost TeamCity 4.0 before the official release

We've published build 8018 which is separated from the official release build by only a couple of build number increments.

Don't miss your chance to try it and be fast to let us know if you spot any critical issues with it.

How ReSharper found a bug in IDEA with the help of TeamCity

A history of one bug.

Several days ago, I received a message from one of ReSharper developers:

The link opened our internal TeamCity installation with the ReSharper build failed on sources checkout. The error was produced by SVN unable to update a directory:

svn: Failed to add directory 'test/assemblies': a versioned directory of the same name already exists

After some digging it turned out that the directory was deleted from the repository some time ago but was not deleted form the build agent. Then it was added anew and the build failed trying to add a directory that already existed on disk. The workaround was to invoke "Enforce clean checkout" from the TeamCity UI for the build configuration.

So ReSharper team can continue their current development effort on performance optimization, but why the directory was not deleted in the first place?

TeamCity uses SVNKit library to talk to Subversion repositories and luckily the developer of the library happened to be just a room away these days. Introduced into the issue he reached for the code to check and soon found a bug in the library that might result in not deleted directory in the working copy. It could only reproduce under certain optimization options that only two products are actually using, one of them being TeamCity and the other ... IntelliJ IDEA. But wait, IDEA is preparing for the 8.0 release these days!

No worries. As I am writing this, the library is already fixed, the latest build downloaded from SVNKit TeamCity installation and as I've just checked on our TeamCity server, the fix is already checked into IDEA release branch.

Actually, the bug is a rare one since it can reproduce only under certain circumstances, it was introduced not so long ago and was only included into EAP releases of IDEA and TeamCity.

Calcutta EAP, build 7888

A new EAP build was published on Tuesday.

The main changes are described in our EAP space, but there were many others and I will point out some:

• Connected agents now display the time of last activity rather then time of registration on the server;
• Presentation of Test Details page has been improved;
• "Triggered by" field of the build now displays the build configuration name if triggered by same-sources dependency;
• Storing of duplicate code finder results has been improved. The results are now valid for all the builds. Previously, the duplicate results were only valid for the most recent build and older builds could show incorrect data;
• If you write a build script or tool and need to detect whether it is run under TeamCity or not, you can now use TEAMCITY_VERSION environment variable that holds TeamCity version as shown in the web UI footer;
• IVY library has been updated. We use it to resolve artifact dependencies on agents. The updated version properly handles spaces in the file names, at last;
• Logging of VCS-related errors into the TeamCity logs has been improved to be less verbose but note more details;
• Windows build agent installer no longer defines JAVA_HOME to point to bundled JRE. This will make the administrators make an explicit choice if their builds use JAVA;
• Build stopping logic has been improved under Windows to eliminate wrong process tree building when process ID was reused by the OS;

Grab you copy of the fixes on the download page.

As always, your comments are welcome!

Agents maintenance

If you read this blog from the beginning you probably know that in JetBrains we use quite a big farm of agents. The agents produce a large amount of work. According to our statistics usage of the most of our 45 agents is around 40-50% (there are no builds at night time).

Moreover some of our builds produce a large number of small files on disk. The files are then deleted by build itself or by agent but this increases fragmentation of the file system. Not surprisingly these builds started to run slower, and we decided that we need some kind of agents maintenance policy. Ideally we need to defragment agent disk and probably reboot agent once per week. Well, since we already have agents installed on these PCs why not do maintenance tasks with еру help of the agents themselves?

The solution

Note: we did not experience performance problems with Unix/Linux based agents so all the scripts below are for Windows only.

First of all we created a project in TeamCity and set necessary access rights. Authorization is important since maintenance tasks are potentially harmful and only experienced users should be able to configure and run them.

Then we created three build configurations:
1) Clean & Defragment
2) Reboot
3) Run all

"Clean & Defragment" build configuration removes temporary files from the user home directory and starts disk defragmentation. Currently it is Ant based, and here is the script we use:

<project name="AgentsMaintenance" default="defragment" basedir=".">

  <target name="defragment">
    <echo message="##teamcity[progressMessage 'Defragmenting ...']"/>
    <exec executable="defrag">
      <arg value="C:"/>
      <arg value="-v"/>
    </exec>
  </target>

  <property environment="env"/>
  <property name="userTemp" location="%env.USERPROFILE%/Local Settings/Temp"/>

  <target name="clean">
    <echo message="##teamcity[progressMessage 'Cleaning ${userTemp} directory...']"/>
    <delete quiet="yes" failonerror="false" includeemptydirs="true">
      <fileset dir="${userTemp}" includes="**/*"/>
    </delete>
    <mkdir dir="${userTemp}"/>
  </target>

</project>

"Reboot" build configuration reboots an agent. It uses command line runner and simply invokes the following command:

shutdown -r -f -t 20 -c "Reboot build agent"

And finally the main step is build configuration "Run all". Here is the Ant script:

<project name="AgentsMaintenance" default="runAll" basedir=".">

  <target name="runAll">
    <property name="user" value="username"/>
    <property name="pwd" value="password"/>

    <get src="http://teamcity.server.com/httpAuth/action.html?add2Queue=<clean&defrag conf id>&amp;agentId=allEnabledCompatible"
         dest="response.txt" verbose="yes" username="${user}" password="${pwd}"/>

    <get src="http://teamcity.server.com/httpAuth/action.html?add2Queue=<reboot conf id>&amp;agentId=allEnabledCompatible"
         dest="response.txt" verbose="yes" username="${user}" password="${pwd}"/>
  </target>
</project>

Where <clean&defrag conf id> and <reboot conf id> are identifiers of the corresponding build configurations (buildTypeId from the TeamCity URLs).

"Run all" build configuration is triggered by TeamCity scheduling trigger at Saturday night. The build of this configuration adds "Clean&Defragment" and "Reboot" builds in the queue. It uses not yet released feature of TeamCity allowing to start a build on all compatible and enabled agents (note agentId=allEnabledCompatible in the URL). So if you do not want "Reboot" builds to run on some agent just make this agent incompatible with "Reboot" configuration! Pretty simple, isn't it?

So far the solution works nice. And if your builds started to run slower try to setup something like this, probably the reason is not in the builds itself. BTW the new feature to run build on all compatible agents will be available in the next EAP build.

TeamCity in Google Chrome

We try things here at JetBrains, so of course we tried Google Chrome on the day it was released. Some of us even switched to Chrome as their main browser.

Apart from just trying the new browser we were also curious whether TeamCity will work OK in Chrome. No surprises here, we did not spot anything broken (since TeamCity works OK in Safari/WebKit).

One of the cool features of Chrome is making web application out of a page and we tried that with TeamCity. It worked out of the box, but we decided to make use of Google Gears API to provide default text for the Create Shortcut dialog and also provide several sizes for the TeamCity favicons so that Chrome can put on the desktop the right size instead of scaling 16x16 default.

Several meta and link tags more and we're in. The latest EAP already has the changes.
If you have it installed, you can get the different-sized icons via URLs:

your_teamcity_server/img/icons/TeamCity16.png
your_teamcity_server/img/icons/TeamCity32.png
your_teamcity_server/img/icons/TeamCity48.png
your_teamcity_server/img/icons/TeamCity128.png
your_teamcity_server/img/icons/TeamCity512.png

Huh? 512? Yes. Just in case someone will need it that big :)

Here is also a related post on DZone.

CITCON Amsterdam 2008

Two of our team: Pavel Sher and Yegor Yarko attended Continuous Integration and Testing Conference (CITCON) whose European meeting was held in Amsterdam this year.

It was the first full-time OpenSpace conference I've attended and it was fun to see how it works. It's really amazing how a non-managed process can work out so well.

It was a pleasure to meet TeamCity users, developers from other CI tools and other insightful people. Here is a small CITCON photo set.

Shall we meet at the next CITCON Europe?

Calcutta EAP, build 7757

On Thursday we've released a new EAP build, but I was a bit in a hurry leaving for a flight to Amsterdam to post the announcement here. But better late then never :)

This build introduces enhancements in the builds dependencies, namely: option to reuse the builds if they have no changes rather then rebuilding them each time for an upstream build; ability to configure artifact dependency to use the dependency built on the same sources; support for the builds sequences in the personal builds and display of the dependencies for a running/finished build.
If you try the feature, let us now how it suits your building process and what is still missing to fulfill your build dependencies requirements.

Other additions in the build are individual test details page with the test runs history and ability to commit a pre-tested commit if no new tests failed.
And yes, we've also added support for .Net code inspections tool: FxCop form Microsoft.

One of the features we started to work on is integration with externals source browsing tools. Some initial functionality is already baked in this EAP release.

Here is how to try it: Create a new file named change-viewers.properties in the .BuildServer/config directory with the content like:

;external change viewers config example
;all entries are optional
;general format is: rootId=external tool link template
*=http://some.tool/view/?changelist=${changeSetId}
;all roots of type 'svn'
svn=http://fisheye/changelog/REPO1/?cs=${changeSetId}
;specific root id
107=http://fisheye/changelog/REPO2/?cs=${changeSetId}

Then look for a link "Open in external change viewer" in the change description on the build's Changes page.
Do you like the feature? What tools do you need integration with?

Browse the full EAP release change log and download the build.

Your comments are welcome!

Calcutta EAP, build 7601

We've just uploaded a new EAP and you can start downloading it :)

The Change Log page pretty much says it all.

A couple of "undocumented" notes, though.

The build has enabled "Agents Statistics Table" tab. Look for one more tab under Agents. It is designed to help TeamCity administrators understand how the load is spread across agents and build configurations.

Actually, the tab was included but turned off for quite a while already. It still needs a lot of polishing, but hopefully we will be able to improve it till the release. Is it helpful? What are your thoughts?

Also, the new search can in fact search for more information that is configured by default. For those of you brave enough, you can try to add test names into the search index by adding -Dtc.search.index.tests=true JVM option to the TeamCity server. If you have many builds/tests and the database is not quick enough, this can cause 100% CPU load for quite a long time after the server start, so use with caution. You can track progress of the indexing on the separate search results page.

As always, your feedback is highly appreciated.

Upgrade button

You probably have seen this illustration before. This is Baron Münchhausen pulling himself and his horse up from the swamp by his own hair. I think this image is the best illustration of our own TeamCity server upgrade process.

What I mean is that TeamCity server we are using at JetBrains to build our projects now is able to upgrade itself... We even have a button in the TeamCity UI which starts the upgrade process. Interested? Well, you can do the same in your testing environment.

First of all we've installed build agent on the same server where TeamCity is running. For security purposes this agent is configured to run one build configuration only (let's call it Upgrade).

Artifact dependencies in the Upgrade configuration were configured to download the artifacts produced by the build that assembles TeamCity installer.

Then we created a build.xml script which unpacks these artifacts, stops the server, backups the previous installation and installs new files. The script is very simple since it is running on the same PC where the server is installed. Finally, the script starts our TeamCity server again. Since all of this is done by the build agent running in a separate process the server can be safely restarted. Agent waits till the server is up again, sends the produced build log and finishes the build.

And now to upgrade our server we just need to click on the "Run" button:



However there is still one significant limitation: we can't choose which build to install. To do that we have to change the build configuration settings (artifact dependencies) every time we want to upgrade, which is not convenient.

Fortunately there is a solution. If you are using TeamCity 4.0 EAP, you are probably aware of the new "Run custom build" feature. Since the last EAP we've improved it a lot. In brief this feature allows you to specify values of build parameters when you click "Run" button. To solve our problem we decided to add one small improvement: we allowed to specify build parameters references in the artifact dependencies build number:



And here is our "Run custom build" dialog where this parameter value can be specified:



Do you remember that TeamCity allows to publish artifacts in parallel with a build itself? This means that in this dialog you can specify build number of a running build too. Think about it, Münchhausen didn't lie...

TeamCity 3.1.2 Release is out

We've uploaded 3.1.2 release on our official download site at jetbrains.com.

This release build number is 6881. We've uploaded 6878 build yesterday, but quickly after that (thanks to one of our attentive users) a problem has been found with connections to Subversion with "svn+ssh" protocol. It turned out we forget to add one of the jars of SVNKit library that we use to access Subversion repositories. A new build with the problem fixed has been uploaded just several minutes ago.

Enjoy!

P.S. If you are already use one of Calcutta EAP builds, just stay tuned. In a week or a bit more we will release a next EAP build with more features included!

Mercurial SCM Support in TeamCity

Among new stable release we prepared another good plugin for you. This plugin adds support for wonderful SCM system - Mercurial. The plugin supports:
* changes collecting
* checkout on server mode
* obtaining files content (allows to view diffs on files)
* checkout rules

Plugin does not support:
* remote run (requires corresponding support in IDE)
* checkout on agent (this feature is in progress)

The plugin works with 3.1.x and 4 EAP releases.

You can get the latest version of the plugin from teamcity.jetbrains.com

There you will find two zip files:
* mercurial-server.zip
* mercurial-src.zip

First zip contains jar files that should be moved to the WEB-INF/lib directory of the TeamCity 3.1.x server. If you need to install plugin under the TeamCity 4.x you can put the whole zip into the .BuildServer/plugins folder. After that restart your server.

Second zip contains full source code of the plugin available under the terms of Apache license 2.0 (you can also obtain source code of the plugin from the mercurial repository: http://hg.jetbrains.org/hg/mercurial/ where it actually reside).

We hope that source code will help us to catch bugs in the plugin faster and will be a good reference for our plugin writers.

Happy building!

Update: Current information on the plugin can be found on the plugin page.

TeamCity 3.1.2 Release Candidate

Since the 3.1.1 release, thanks to our users, we discovered and fixed some bugs in the code. The most critical ones were integrated into 3.1.x branch and now it's time to release the fixes as 3.1.2 release.

Before the release we would want to check that the fixes really work and no new ones were introduced, hence the 3.1.2 release candidate.

In this release we have included:

• reworked clean checkout for VSS which now uses native client instead of glitchy VSS API;
• reworked build stopping logic under Linux platforms: we no longer use JNI (to find out the PID of the current process), but some command-line magic;
• reworked database migration tool. OutOfMemory error on large databases is (hopefully) fixed now, as well as data truncation errors. Also, detailed progress reporting is implemented which is really useful when migrating large setups.

  Nevertheless, for a production use please consider migrating to one of supported external databases as soon as possible;

• fixed security issue when a user watching "All Projects" may receive notification for a project he/she does not have access to;
• miscellaneous fixes in CVS and StarTeam support, other issues.

So, if you are still using TeamCity 3.1.1 or earlier and did not upgrade to Calcutta EAP, please try this release candidate build.

Those using the 4.x branch EAP does not have the option to try this build because the TeamCity data directory was already upgraded to a newer format and it cannot be downgraded. On the other hand, most of the fixes were already available in the latest Calcutta EAP and the most recent ones will be available in the next.

PS
While preparing the release notes for this release candidate I approached the team developing our home-grown issue tracker with a reminder that the tracker still has no way to make an external link to a shared folder like "3.1.2 fixes". 30 minutes later we already had the new version of the tracker deployed with the feature added. "Wow", said I and grabbed the link for the 3.1.2 fixes.
Not having the feature beforehand is the cost, but ability to have a feature added in the matter of minutes is the benefit of using in-house developed solutions.

TeamCity and Mono

Do you use Mono with TeamCity. Please describe some use cases or cases you wish to implement with TeamCity and Mono.

Please vote for Mono support issue in our tracker at http://www.jetbrains.net/tracker/issue/TW-5425

Calcutta EAP, build 7385

I also think that two months is a too large period between EAPs.

But here it is, to your pleasure.

• personal builds can be targeted to a specific build agent
  
• Ability to redefine certain build parameters on manual build run
  
• Risk tests reordering support for NUnit
  
• IntelliJ IDEA 8 (Diana) is used to collect inspections and find duplicates
  
• IntelliJ IDEA 8 (Diana) project file format is supported in Ipr runner, Inspections and Duplicate runners.
  
• Reworked plugin packaging (plugin writers, take a look!)
  
• Ability to configure charts on Project home page (Statistics tab)
  
• Custom artifacts-based tab on the Project page
  
• more features and details (with screenshots)
  

Much of the work done in this EAP build is not visible in UI, since we're working on composed builds/history builds feature and didn't manage to make it testable by the public. Yet.

Download and don't forget to backup before installing. Would you help us test it?

Kind regards,
KIR

TeamCity scalability: database and disk space

This post is inspired by recent "guidance on planning database space" questions in our community forums and support mail. It will also approach scalability and performance from the point of handling huge amounts of data that may be stored inside of TeamCity.

Information given here is obtained by observing JetBrains internal production server that is used for building most of our products (IDEA, Resharper, TC itself and others).

TeamCity database size depends on number and type of builds that are held in history. This number is increased with each build run and can be reduced during cleanup if cleanup policy has been set up. By default all information is kept forever and database will grow only.

On our server used database space size is typically growing during the day (proportionally to number of builds run), than dropping almost to the previous level during cleanup. Average is relatively slowly growing due to builds pinned and small amount of statistical data being accumulated.

After setting up new project or configuration (i.e. release branching) we usually notice rapid growth of DB size until reaching next plateau then new data is balanced by corresponding cleanup policy.

Typical policy for an active project is "Delete artifacts and history for builds older than 14 days since last build". Note that all pinned builds and all builds that have their artifacts used by other builds are unaffected by clenup.

We have about 8000 builds in history at the moment. The history is quite "dense" for last month - 50-100 builds per day for major projects. After some point in past most builds are cleaned and history is quite "sparse" and its "density" gradually decreases with descending into past. Deepest entries in our history are more than 3 years old. Build statistics on our server is kept forever (It's now about a year since it was first introduced).

Although in an ongoing development process everything is going smooth, there are conditions possible that can lead to huge spikes in space usage. We had actual case of build that was setup to be re-run after failure and one day it was broken in a way that made it fail almost immediately. In no time we had +15000 builds in history. That actually didn't lead to any problems because very little information were produced and stored for each run. But in other cases such builds can quickly use all your database quota or artifacts and build logs disk space.

Our production server runs under 32bit Windows XP and has quite an average hardware: 3.2GHz Pendium D, 3.25 gigs of RAM with swap disabled. We have separate (big) hard drive for TeamCity data directory (holding build logs and artifacts).

We are using MySQL 5.0.4x database as backing storage that is running on the same box. (Configuration: inno_db/file-per-table, caches and buffers increased, uses about 600M of RAM). It currently takes slightly more than 2 gigs on the system partition.

About 50% of space is used by passed/failed test data (our typical build has 3 000 - 11 000 unit tests). Next top consumers are Inspection builds, compiler output and VCS changes information - taking about 7% each. Other features use about 1% per data domain, with build history and statistics data totaling to 2%.

These numbers can be used to planning and predicting database space usage. However judging from user feedback disk space problems (due to amount of artifacts stored) are encountered much earlier than any problems with database.

Although we have integration builds for all supported databases (Oracle 10+, PosgreSQL 8+, MS SQL 2005+) run over each commit and development instances using these databases we'd like to obtain more information about different real-world setups.

Feedback is very appreciated!

Visual Studio Feed

I know, many people uses Microsoft Visual Studio 2005 or 2008. There is the very first page called 'Start Page'. If you have a look to options, you'll notice that it is possible to replace the standard news feed in the center of screen with your custom one.

Today I gonna describe how to put TeamCity feed there.

TeamCity is able to create feeds for notifying one about selected build configurations. It is easy to construct the feed you like. For doing that you may simply click on any feed icons on the Web UI, or build you own feed using feed builder page in 'My Settings & Tools' tab.

First of all you need to switch TeamCity feed generator to use RSS 0.93. We have posted an issue in our tracker for making this easier. But now, follow the instructions:

On the server open <teamcity data directory>/config/default-feed-item-template.ftl. Replace there '<#global feedType="atom_1.0">' with '<#global feedType="rss_0.93">'. You can also modify template as you like. Save the file to <teamcity data directory>/config/feed-item-template.ftl. You do not need to restart the server. All changes will be loaded automatically.

Now go to Visual Studio. Open Tools\Options\Environment\Startup. Write the link to TeamCity feed in the 'Start Page news channel'.

Enjoy looking to TeamCity news in you VS start page.

I'd like to say thanks to Jonathan Harley for good idea written in forums.

NAnt and MSBuild 3.5

Today I have just upgraded TeamCity Visual Studio plugin sources to use new VS 2008. So I had an issue updating my NAnt script to support MSBuild 3.5. We use task from NAnt contrib project.

As you may know, TeamCity NAnt runner replaces standard msbuild task with it's own
one. This is done specially to provide better logging for MSBuild process.

To switch to MSBuild 3.5 one can do one of the following tricks:
- set target framework to net-3.5 (if it is NAnt 0.86 beta 1);
- define property 'teamcity_dotnet_use_msbuild_v35' in the msbuild task and set it to 'true'. This will work only when run under TeamCity.


So I have selected the second approach. So my NAnt script now looks like:

<msbuild project="${base}\addin\addin.sln" target="Rebuild">
  <property name="Configuration" value="${config}" />
  <property name="RegisterOutputPackage" value="false" />
  <!-- Switch on msbuild 3.5 for msbuild task -->
  <property name="teamcity_dotnet_use_msbuild_v35" value="true" />
</msbuild>

I tried that as a pre-tested commit from Visual Studio 2008, it passed, and was auto-committed into TFS.

Gallio, MBUnit and others

Just curious weather and how MBUnit/Gallio/MSTest/xUnit/... are used with TeamCity. Who can share she experiences or/and plugins for that.

TeamCity NUnit support features

TeamCity supports .NET as well as Java. For .NET there is couple of build runners. To run a build it is possible to use NAnt, MSBuild, Sln2003, Sln2005, Sln2008 build runner.

Actually, there is no matter if you use NUnit or something else. TeamCity is able to track NUnit tests and report all tests information on-the-fly. For that feature you do not need to change the build.

TeamCity automatically replaces NUnit calls from NAnt with <nunit2> task or MSBuild with <nunit> task from MSBuild Community tasks. For Visual Studio solutions there are UI options to enable unit tests. In anyway if you want to run NUnit tests you may use mentioned tasks in you build scripts or even call an .exe to run the tests.

TeamCity supports NUnit 2.2.10, 2.4.1, 2.4.6, 2.4.7. Actually, there is really a small difference what NUnit test runner to use. There is no strong dependency between NUnit test runner and NUnit.Framework.dll that is linked to the application.

Tests may require to be started under exact NET 1.1 (or .NET 2.0). In case .NET 2.0 there is an option x64 or x86. Under TeamCity it is easy to specify it.

For NAnt you have to specify target framework to .NET 1.1 and that would start your tests under .NET 1.1

If you use command line for your tests it would look like this:

$(teamcity.dotnet.nunitlauncher) v1.1 MSIL NUnit-2.4.7 <list of assemblies>

To select .NET 2.0 x86 or .NET 2.0 x64. For NAnt you will have to define property named 'teamcity.dotnet.nant.nunit2.platform'. Values could be x64,x86,MSIL. For command line runner it would look like: 

  $(teamcity.dotnet.nunitlauncher) v2.0 x86 NUnit-2.4.7 <list of assemblies>

All mentioned settings are available on Sln2005/Sln2008 runners from TeamCity Web interface.

For MSBuild TeamCity defines task called NUnitTeamCity. It is possible to define any behavior of the tests under TeamCity. Using Platform attribute it is easy to select platform (x64 or x86) to run your tests. NUnitVersion attribute allows to select NUnit test runner version.

And finally, if some test has failed you will be notified at the exact time of that failure. You do not need to wait for all tests to finish.

TeamCity Plugins Repository is Emerging

We are starting to build TeamCity plugin repository. Well, "repository" is too big a word for it at this time, but more entries are under way.

The idea is not only to list a plugin, but provide plugin documentation space, host plugin repository and bug tracker, and (the most interesting part) to build the plugin against the latest released/current trunk TeamCity versions.

We've set this up for several our plugins (build with TeamCity trunk version is a pending task), and plan to provide the same services to the third-party plugins.

So if you want to write a plugin, or already have one that can be of interest to other users, just let us know.

Getting test results faster

Have you ever paid attention on how fast you can get results of building your changes from TeamCity? I mean if a test failed you do not need to wait while the whole build finishes, TeamCity will notify you immediately. You can say "So what? It's quite natural to expect this feature from an automated build system". I agree, but show me any other continuous integration server which offers similar functionality. Most of them (if not all) require you to specify paths to test results files (usually in JUnit format) and in turn it means that you have to wait while build finishes (what if tests in your build run 40 minutes?). Not to mention that sometimes you have to modify your build script to produce test results in required format.

This is a great feature and we are going to improve it further. First of all we are going to force your build to run recently failed tests before any other tests. This should help you if you are fixing broken tests. TeamCity will try to do this transparently, but in some cases (for example, if you have your own test runner) it can't be done. For these cases we'll provide list of tests via property or via file and you will be able to modify your test runner to support this feature too.

At the moment this feature is already working on our internal server. Currently it is supported for Ant and IPR runners and for JUnit and TestNG frameworks. We hope to add support for other frameworks and runners before the first EAP.

A further improvement would be to run new and modified tests first. This should be especially useful for personal builds. Since TeamCity tracks changes for every build we can determine whether modified files are test cases or not. We do not have a working code for this feature yet but we hope to add it to the first EAP too.

By the way we are going to release first EAP of Calcutta (TeamCity 4.0 codename) at the next week. I hope you'll find time to install it and try these new features.

The fools day TeamCity plugin

We've worked hard to create a new very cool plugin for you (even the 3.1.1 release had not taken so much efforts ;). If you have administrative rights on your TeamCity server then you have unique ability to make fun of your colleagues! Just install this plugin and enable it on the server configuration page. I do not want to describe its abilities here, all I can tell is that "jokes mode" is enabled when browser losts focus... Also if you understand JavaScript take a look at joke.js file, it contains some notes about plugin abilities.

Feel free to extend and improve it :)

Personal builds: practical experience

TeamCity has two interesting features: pre-tested commit and remote run. Remote run feature allows you to send your changes to TeamCity server and run a build with them without performing actual commit to a version control system. Pre-tested commit feature just adds automated commit to version control system if build was successful. In TeamCity such builds are called personal builds. There are some things that user of these features should be aware of.

Under the hood

First of all TeamCity currently does not allow you to run personal builds with your changes only. When a build starts your changes will always be integrated with current trunk. Sometimes this causes build failures when you and other developers modified the same files, but your colleague has already committed his changes. However from my experience this rarely causes problems.

Second you can safely modify your files while personal build is running. You can even modify same files that you just sent to TeamCity via remote run. Automated commit will check-in those versions of files that were sent to TeamCity when your personal build started, not the last changes you've made. Moreover starting from TeamCity 3.0 you can manually check-in these versions when personal build finishes. This works because before starting a personal build content of modified files corresponding to this build stored on disk elsewhere.

Third, automated commit actually is performed from IDE. So if you start a personal build with automated commit and then close IDE the commit will not be done until you start your IDE again.

Returning to practical usage...

So when these features are useful? Below I tried to summarize our practical experience of working with these features.

Personal builds are useful when you have a lot of functional tests (which are usually slow), or when you need to run your changes on several platforms with different software. In other words, they are useful when you are not able to run full suite of tests on your developer PC. With help of TeamCity build agents you can run your personal builds on different platforms in parallel and thus save a lot of time.

Personal builds are very useful for testing your code during refactoring. Often refactorings can be split in parts. For example, you've rewritten some component, your code compiles again and you feel that change should work fine. This is a good time for personal build. Usually in this case I am starting a personal build and continue my refactoring which could change the same component again. TeamCity IDE plugin will notify me when build finishes or when a first failure occures in the build (I prefer to have such notification for builds with my changes). And do not forget that TeamCity works hard to provide me with information about failures at the same time when they occur, so if I've broken something I'll soon be informed about that. If results are looking good I may decide to commit changes that were run in this personal build. Personal builds allow you to keep good quality in your code base during long refactoring tasks.

Not all of the functionality can be covered with tests. Sometimes you need to test your fix manually. Or you need to send a patched version of your distribution to a client. Personal builds can be useful here in case if your distribution cannot be created on your PC. For example, TeamCity contains Java / .NET and C++ code. I work mostly on Java components and I do not have Visual Studio installed so I cannot easily build a TeamCity distribution on my PC. In this case I would prefer to start a personal build. There were several cases when a personal build of TeamCity was installed to our internal server (where all Jetbrains projects are built) to check whether performance improvement or other critical fix we've made actually works.

Usually project build scripts and make files should be modified very carefully since a little error in these files can brake all of your builds. Personal builds are good choice when you need to check modifications in such files.

TeamCity does not have functionality explicitly designed for code review, but it is still possible to do a pre-commit code review with help of personal builds. For example, a developer could send you a link to a successfully finished personal build. First of all build results can tell you much about the change. Second you can review changes with help of TeamCity diff window and then make a final decision about it.

For those who never used TeamCity I hope this summary helps you better understand how personal builds work and in which cases they can be useful. If you are using TeamCity for day-to-day work feel free to share your experience with these features.

Puzzle Answer

See the previous post for the puzzle. Here comes the answer.

So, what's the strange part in the screenshot? A-ha, you must've noticed that the build number of the server (the one in the footer of the page) is the same as... one of the running builds! How that can be? Is it a fake? Is it a bug?

Nope, thanks to the two TeamCity features: during-the-build artifacts publishing and not loosing the running builds on server restart.


One of the features of TeamCity that was present since 1.0, but is not always noticed at first glance is on-the-fly tests reporting. With it you do not have to wait till the build finishes to know what tests have passed and what have failed - you simply get the information in the real time. (The feature on its own deserves a separate post or two.)

Getting used to this real-time support users kept asking for more: why not get an artifact as soon as it is ready, not waiting for the build to finish? There were some speculations on how this can be implemented: Should we watch for the files and start uploading them as soon as they are ready? But how do we know the files are ready and written fully? Under Windows, it seems we can detect file is open for writing, but what about other platforms?
For now we ended up with a simple but working solution: build script writer should notify TeamCity that the artifacts are ready to be published.

In 3.1 we introduced TeamCity "service messages": specially formatted strings that, when printed during the build, will be detected and parsed by TeamCity to perform an action. And one of messages implemented in the first place was publishArtifacts. You just print ##teamcity[publishArtifacts 'myValuedFile.zip'] into standard output and voilà: myValuedFile.zip is being uploaded to the server in parallel to the main build process. Once we deployed the build with the feature to our TeamCity server we configured our main TeamCity build to upload installers as soon as they are ready (we are creating packages before running the tests: that proved to be useful on a number of occasions).

That's the first part of the story: we can get TeamCity distribution package while the build is still on its way. What we do next is just upgrade the server (shutdown, place the new application files, startup). And here comes another great feature: the server sate is fully restored as if there were no server restart at all. This is way more easy to describe then it was to implement, but without the feature each server restart would be a pain, which we certainly strive to eliminate.

So, here we are: a TeamCity build that builds itself. Isn't it cool? :)

A Puzzle

Here is a real screenshot. Can you spot what's not usual in it?



Depicted is our JetBrains TeamCity installation with Benares3.1::MasterBuild build configuration which is the main build for TeamCity 3.1 branch: it creates installers and runs a set of tests.

See the next post for the answer.

New feature: search tests by name

The upcoming version of TeamCity 3.1 will contain a possibility to search tests with a particular name in build results.

It allows to find out if a particular test has been run in the build. If you have a project with several thousand tests, using a filter is much more convenient than loading the full test list.

Right now, you can see ordinal number of the test, its status, and duration (with possibility to view a graph of the test duration history). All this is described in the corresponding part of the TeamCity UI documentation.

In the future, we plan to add a possibility to view the output of a particular test right from this interface (watch/vote for TW-3553).

Incidental Field Test: Email Notifications Necessity

Yesterday we noticed an error sending email notification in the logs of our internal TeamCity installation. A quick investigation revealed that the server email sending settings are misconfigured. What was more interesting (and bitter) is that the settings were spoiled 2 weeks ago. Two weeks. All that time nobody was receiving email notifications from our internal TeamCity server! That might seem a disaster, but wait: why we had no complaints about it?

Of course we have other notifiers like Jabber, Windows tray, IDE and RSS, but still email should be the most popular. So why?

My guess is that advanced TeamCity users (who JetBrainers certainly are) do not want to be notified, they want to monitor. Web UI gives you a continuous, real-time feedback on what is going on. There are the whole bunch of additional data like the changes, commits, build running steps, own changes status and more. People may open TeamCity's web in the morning and leave it open for the whole day.

So most people either turned off the notifications at all or just did not notice them missing.

I remember our CruiseControl days when almost the only interface to the server was email. Web UI did not actually produce much more information and after all it was unbearably slow because of the build running on the same machine. Breaking email notifications in those days would mean no CI results - everyone would notice the problem in a couple of hours (or minutes, depending on the length of the builds). But not now. Things change.

Good notifications should not be underestimated. And we do improve notifications and will continue to do so. But who knows, with the tools advances, in a couple of years we may look down on email notifications like we do now on snail mail.

Getting down to earth, what we can do to prevent silly misconfiguration mistakes from going unnoticed? A possible solution is to display a warning to administrators in the web UI if any alike server error occur. Filed as TW-4552.

500 build hours per day

You might not know but here at Jetbrains we are using our own TeamCity server to build almost all of our projects (including such giants as IntelliJ IDEA and Resharper). This is a part of our company strategy proudly named "eat your own dog food". And it works quite well.

Big projects usually require much time to build and run tests, for example, to compile IntelliJ IDEA sources you will need about 16min on a relatively modern PC. While to run a full test suite you'll need almost two hours. The same picture is in Resharper project: the whole build with compilation and tests takes more than 2 hours to complete. Given such a long builds it is critical to be able to run them in parallel. That is why TeamCity has such feature as build grid from the beginning, otherwise we won't be able to use TeamCity at all.

But how much agents will be enough for us? Well, when we released TeamCity 1.0 in October 2006 we had a pool of about 20 agents:




Things did not change much when we released 2.0 version in April 2007 (except that now we had a so-called "glass" indicating current server load :)






Then as more Jetbrains projects moved to TeamCity we started to realize that we need more power. In December 2007 when TeamCity 3.0 released we had more than 30 agents:




Interestingly even with 33 agents we have almost as many builds waiting in the queue :)

So did we stop? Nope! Thanks to our selfless administrators we increased our pool again:



Now we are able to build more than 700 builds per day (all of them took >500 hours, or about 20 build days) and publish more than 10Gb of artifacts. I think it is serious power. BTW our server is installed on a PC with dual core CPU 3.2GHz / 4Gb RAM of that TeamCity uses 750Mb only.

Do you think so much agents will be enough for us? Will see :)