Upgrade button

You probably have seen this illustration before. This is Baron Münchhausen pulling himself and his horse up from the swamp by his own hair. I think this image is the best illustration of our own TeamCity server upgrade process.

What I mean is that TeamCity server we are using at JetBrains to build our projects now is able to upgrade itself... We even have a button in the TeamCity UI which starts the upgrade process. Interested? Well, you can do the same in your testing environment.

First of all we've installed build agent on the same server where TeamCity is running. For security purposes this agent is configured to run one build configuration only (let's call it Upgrade).

Artifact dependencies in the Upgrade configuration were configured to download the artifacts produced by the build that assembles TeamCity installer.

Then we created a build.xml script which unpacks these artifacts, stops the server, backups the previous installation and installs new files. The script is very simple since it is running on the same PC where the server is installed. Finally, the script starts our TeamCity server again. Since all of this is done by the build agent running in a separate process the server can be safely restarted. Agent waits till the server is up again, sends the produced build log and finishes the build.

And now to upgrade our server we just need to click on the "Run" button:



However there is still one significant limitation: we can't choose which build to install. To do that we have to change the build configuration settings (artifact dependencies) every time we want to upgrade, which is not convenient.

Fortunately there is a solution. If you are using TeamCity 4.0 EAP, you are probably aware of the new "Run custom build" feature. Since the last EAP we've improved it a lot. In brief this feature allows you to specify values of build parameters when you click "Run" button. To solve our problem we decided to add one small improvement: we allowed to specify build parameters references in the artifact dependencies build number:



And here is our "Run custom build" dialog where this parameter value can be specified:



Do you remember that TeamCity allows to publish artifacts in parallel with a build itself? This means that in this dialog you can specify build number of a running build too. Think about it, Münchhausen didn't lie...

TeamCity 3.1.2 Release is out

We've uploaded 3.1.2 release on our official download site at jetbrains.com.

This release build number is 6881. We've uploaded 6878 build yesterday, but quickly after that (thanks to one of our attentive users) a problem has been found with connections to Subversion with "svn+ssh" protocol. It turned out we forget to add one of the jars of SVNKit library that we use to access Subversion repositories. A new build with the problem fixed has been uploaded just several minutes ago.

Enjoy!

P.S. If you are already use one of Calcutta EAP builds, just stay tuned. In a week or a bit more we will release a next EAP build with more features included!

Mercurial SCM Support in TeamCity

Among new stable release we prepared another good plugin for you. This plugin adds support for wonderful SCM system - Mercurial. The plugin supports:
* changes collecting
* checkout on server mode
* obtaining files content (allows to view diffs on files)
* checkout rules

Plugin does not support:
* remote run (requires corresponding support in IDE)
* checkout on agent (this feature is in progress)

The plugin works with 3.1.x and 4 EAP releases.

You can get the latest version of the plugin from teamcity.jetbrains.com

There you will find two zip files:
* mercurial-server.zip
* mercurial-src.zip

First zip contains jar files that should be moved to the WEB-INF/lib directory of the TeamCity 3.1.x server. If you need to install plugin under the TeamCity 4.x you can put the whole zip into the .BuildServer/plugins folder. After that restart your server.

Second zip contains full source code of the plugin available under the terms of Apache license 2.0 (you can also obtain source code of the plugin from the mercurial repository: http://hg.jetbrains.org/hg/mercurial/ where it actually reside).

We hope that source code will help us to catch bugs in the plugin faster and will be a good reference for our plugin writers.

Happy building!

Update: Current information on the plugin can be found on the plugin page.

TeamCity 3.1.2 Release Candidate

Since the 3.1.1 release, thanks to our users, we discovered and fixed some bugs in the code. The most critical ones were integrated into 3.1.x branch and now it's time to release the fixes as 3.1.2 release.

Before the release we would want to check that the fixes really work and no new ones were introduced, hence the 3.1.2 release candidate.

In this release we have included:

• reworked clean checkout for VSS which now uses native client instead of glitchy VSS API;
• reworked build stopping logic under Linux platforms: we no longer use JNI (to find out the PID of the current process), but some command-line magic;
• reworked database migration tool. OutOfMemory error on large databases is (hopefully) fixed now, as well as data truncation errors. Also, detailed progress reporting is implemented which is really useful when migrating large setups.

  Nevertheless, for a production use please consider migrating to one of supported external databases as soon as possible;

• fixed security issue when a user watching "All Projects" may receive notification for a project he/she does not have access to;
• miscellaneous fixes in CVS and StarTeam support, other issues.

So, if you are still using TeamCity 3.1.1 or earlier and did not upgrade to Calcutta EAP, please try this release candidate build.

Those using the 4.x branch EAP does not have the option to try this build because the TeamCity data directory was already upgraded to a newer format and it cannot be downgraded. On the other hand, most of the fixes were already available in the latest Calcutta EAP and the most recent ones will be available in the next.

PS
While preparing the release notes for this release candidate I approached the team developing our home-grown issue tracker with a reminder that the tracker still has no way to make an external link to a shared folder like "3.1.2 fixes". 30 minutes later we already had the new version of the tracker deployed with the feature added. "Wow", said I and grabbed the link for the 3.1.2 fixes.
Not having the feature beforehand is the cost, but ability to have a feature added in the matter of minutes is the benefit of using in-house developed solutions.

TeamCity and Mono

Do you use Mono with TeamCity. Please describe some use cases or cases you wish to implement with TeamCity and Mono.

Please vote for Mono support issue in our tracker at http://www.jetbrains.net/tracker/issue/TW-5425

Calcutta EAP, build 7385

I also think that two months is a too large period between EAPs.

But here it is, to your pleasure.

• personal builds can be targeted to a specific build agent
  
• Ability to redefine certain build parameters on manual build run
  
• Risk tests reordering support for NUnit
  
• IntelliJ IDEA 8 (Diana) is used to collect inspections and find duplicates
  
• IntelliJ IDEA 8 (Diana) project file format is supported in Ipr runner, Inspections and Duplicate runners.
  
• Reworked plugin packaging (plugin writers, take a look!)
  
• Ability to configure charts on Project home page (Statistics tab)
  
• Custom artifacts-based tab on the Project page
  
• more features and details (with screenshots)
  

Much of the work done in this EAP build is not visible in UI, since we're working on composed builds/history builds feature and didn't manage to make it testable by the public. Yet.

Download and don't forget to backup before installing. Would you help us test it?

Kind regards,
KIR

TeamCity scalability: database and disk space

This post is inspired by recent "guidance on planning database space" questions in our community forums and support mail. It will also approach scalability and performance from the point of handling huge amounts of data that may be stored inside of TeamCity.

Information given here is obtained by observing JetBrains internal production server that is used for building most of our products (IDEA, Resharper, TC itself and others).

TeamCity database size depends on number and type of builds that are held in history. This number is increased with each build run and can be reduced during cleanup if cleanup policy has been set up. By default all information is kept forever and database will grow only.

On our server used database space size is typically growing during the day (proportionally to number of builds run), than dropping almost to the previous level during cleanup. Average is relatively slowly growing due to builds pinned and small amount of statistical data being accumulated.

After setting up new project or configuration (i.e. release branching) we usually notice rapid growth of DB size until reaching next plateau then new data is balanced by corresponding cleanup policy.

Typical policy for an active project is "Delete artifacts and history for builds older than 14 days since last build". Note that all pinned builds and all builds that have their artifacts used by other builds are unaffected by clenup.

We have about 8000 builds in history at the moment. The history is quite "dense" for last month - 50-100 builds per day for major projects. After some point in past most builds are cleaned and history is quite "sparse" and its "density" gradually decreases with descending into past. Deepest entries in our history are more than 3 years old. Build statistics on our server is kept forever (It's now about a year since it was first introduced).

Although in an ongoing development process everything is going smooth, there are conditions possible that can lead to huge spikes in space usage. We had actual case of build that was setup to be re-run after failure and one day it was broken in a way that made it fail almost immediately. In no time we had +15000 builds in history. That actually didn't lead to any problems because very little information were produced and stored for each run. But in other cases such builds can quickly use all your database quota or artifacts and build logs disk space.

Our production server runs under 32bit Windows XP and has quite an average hardware: 3.2GHz Pendium D, 3.25 gigs of RAM with swap disabled. We have separate (big) hard drive for TeamCity data directory (holding build logs and artifacts).

We are using MySQL 5.0.4x database as backing storage that is running on the same box. (Configuration: inno_db/file-per-table, caches and buffers increased, uses about 600M of RAM). It currently takes slightly more than 2 gigs on the system partition.

About 50% of space is used by passed/failed test data (our typical build has 3 000 - 11 000 unit tests). Next top consumers are Inspection builds, compiler output and VCS changes information - taking about 7% each. Other features use about 1% per data domain, with build history and statistics data totaling to 2%.

These numbers can be used to planning and predicting database space usage. However judging from user feedback disk space problems (due to amount of artifacts stored) are encountered much earlier than any problems with database.

Although we have integration builds for all supported databases (Oracle 10+, PosgreSQL 8+, MS SQL 2005+) run over each commit and development instances using these databases we'd like to obtain more information about different real-world setups.

Feedback is very appreciated!