TeamCity scalability: database and disk space

This post is inspired by recent "guidance on planning database space" questions in our community forums and support mail. It will also approach scalability and performance from the point of handling huge amounts of data that may be stored inside of TeamCity.

Information given here is obtained by observing JetBrains internal production server that is used for building most of our products (IDEA, Resharper, TC itself and others).

TeamCity database size depends on number and type of builds that are held in history. This number is increased with each build run and can be reduced during cleanup if cleanup policy has been set up. By default all information is kept forever and database will grow only.

On our server used database space size is typically growing during the day (proportionally to number of builds run), than dropping almost to the previous level during cleanup. Average is relatively slowly growing due to builds pinned and small amount of statistical data being accumulated.

After setting up new project or configuration (i.e. release branching) we usually notice rapid growth of DB size until reaching next plateau then new data is balanced by corresponding cleanup policy.

Typical policy for an active project is "Delete artifacts and history for builds older than 14 days since last build". Note that all pinned builds and all builds that have their artifacts used by other builds are unaffected by clenup.

We have about 8000 builds in history at the moment. The history is quite "dense" for last month - 50-100 builds per day for major projects. After some point in past most builds are cleaned and history is quite "sparse" and its "density" gradually decreases with descending into past. Deepest entries in our history are more than 3 years old. Build statistics on our server is kept forever (It's now about a year since it was first introduced).

Although in an ongoing development process everything is going smooth, there are conditions possible that can lead to huge spikes in space usage. We had actual case of build that was setup to be re-run after failure and one day it was broken in a way that made it fail almost immediately. In no time we had +15000 builds in history. That actually didn't lead to any problems because very little information were produced and stored for each run. But in other cases such builds can quickly use all your database quota or artifacts and build logs disk space.

Our production server runs under 32bit Windows XP and has quite an average hardware: 3.2GHz Pendium D, 3.25 gigs of RAM with swap disabled. We have separate (big) hard drive for TeamCity data directory (holding build logs and artifacts).

We are using MySQL 5.0.4x database as backing storage that is running on the same box. (Configuration: inno_db/file-per-table, caches and buffers increased, uses about 600M of RAM). It currently takes slightly more than 2 gigs on the system partition.

About 50% of space is used by passed/failed test data (our typical build has 3 000 - 11 000 unit tests). Next top consumers are Inspection builds, compiler output and VCS changes information - taking about 7% each. Other features use about 1% per data domain, with build history and statistics data totaling to 2%.

These numbers can be used to planning and predicting database space usage. However judging from user feedback disk space problems (due to amount of artifacts stored) are encountered much earlier than any problems with database.

Although we have integration builds for all supported databases (Oracle 10+, PosgreSQL 8+, MS SQL 2005+) run over each commit and development instances using these databases we'd like to obtain more information about different real-world setups.

Feedback is very appreciated!

Visual Studio Feed

I know, many people uses Microsoft Visual Studio 2005 or 2008. There is the very first page called 'Start Page'. If you have a look to options, you'll notice that it is possible to replace the standard news feed in the center of screen with your custom one.

Today I gonna describe how to put TeamCity feed there.

TeamCity is able to create feeds for notifying one about selected build configurations. It is easy to construct the feed you like. For doing that you may simply click on any feed icons on the Web UI, or build you own feed using feed builder page in 'My Settings & Tools' tab.

First of all you need to switch TeamCity feed generator to use RSS 0.93. We have posted an issue in our tracker for making this easier. But now, follow the instructions:

On the server open <teamcity data directory>/config/default-feed-item-template.ftl. Replace there '<#global feedType="atom_1.0">' with '<#global feedType="rss_0.93">'. You can also modify template as you like. Save the file to <teamcity data directory>/config/feed-item-template.ftl. You do not need to restart the server. All changes will be loaded automatically.

Now go to Visual Studio. Open Tools\Options\Environment\Startup. Write the link to TeamCity feed in the 'Start Page news channel'.

Enjoy looking to TeamCity news in you VS start page.

I'd like to say thanks to Jonathan Harley for good idea written in forums.

NAnt and MSBuild 3.5

Today I have just upgraded TeamCity Visual Studio plugin sources to use new VS 2008. So I had an issue updating my NAnt script to support MSBuild 3.5. We use task from NAnt contrib project.

As you may know, TeamCity NAnt runner replaces standard msbuild task with it's own
one. This is done specially to provide better logging for MSBuild process.

To switch to MSBuild 3.5 one can do one of the following tricks:
- set target framework to net-3.5 (if it is NAnt 0.86 beta 1);
- define property 'teamcity_dotnet_use_msbuild_v35' in the msbuild task and set it to 'true'. This will work only when run under TeamCity.


So I have selected the second approach. So my NAnt script now looks like:

<msbuild project="${base}\addin\addin.sln" target="Rebuild">
  <property name="Configuration" value="${config}" />
  <property name="RegisterOutputPackage" value="false" />
  <!-- Switch on msbuild 3.5 for msbuild task -->
  <property name="teamcity_dotnet_use_msbuild_v35" value="true" />
</msbuild>

I tried that as a pre-tested commit from Visual Studio 2008, it passed, and was auto-committed into TFS.

Gallio, MBUnit and others

Just curious weather and how MBUnit/Gallio/MSTest/xUnit/... are used with TeamCity. Who can share she experiences or/and plugins for that.