Sunday, October 26, 2008

Agents maintenance

If you read this blog from the beginning you probably know that in JetBrains we use quite a big farm of agents. The agents produce a large amount of work. According to our statistics usage of the most of our 45 agents is around 40-50% (there are no builds at night time).

Moreover some of our builds produce a large number of small files on disk. The files are then deleted by build itself or by agent but this increases fragmentation of the file system. Not surprisingly these builds started to run slower, and we decided that we need some kind of agents maintenance policy. Ideally we need to defragment agent disk and probably reboot agent once per week. Well, since we already have agents installed on these PCs why not do maintenance tasks with еру help of the agents themselves?

The solution


Note: we did not experience performance problems with Unix/Linux based agents so all the scripts below are for Windows only.

First of all we created a project in TeamCity and set necessary access rights. Authorization is important since maintenance tasks are potentially harmful and only experienced users should be able to configure and run them.

Then we created three build configurations:
1) Clean & Defragment
2) Reboot
3) Run all

"Clean & Defragment" build configuration removes temporary files from the user home directory and starts disk defragmentation. Currently it is Ant based, and here is the script we use:
<project name="AgentsMaintenance" default="defragment" basedir=".">

  <target name="defragment">
    <echo message="##teamcity[progressMessage 'Defragmenting ...']"/>
    <exec executable="defrag">
      <arg value="C:"/>
      <arg value="-v"/>
    </exec>
  </target>

  <property environment="env"/>
  <property name="userTemp" location="%env.USERPROFILE%/Local Settings/Temp"/>

  <target name="clean">
    <echo message="##teamcity[progressMessage 'Cleaning ${userTemp} directory...']"/>
    <delete quiet="yes" failonerror="false" includeemptydirs="true">
      <fileset dir="${userTemp}" includes="**/*"/>
    </delete>
    <mkdir dir="${userTemp}"/>
  </target>

</project>


"Reboot" build configuration reboots an agent. It uses command line runner and simply invokes the following command:
shutdown -r -f -t 20 -c "Reboot build agent"


And finally the main step is build configuration "Run all". Here is the Ant script:
<project name="AgentsMaintenance" default="runAll" basedir=".">

  <target name="runAll">
    <property name="user" value="username"/>
    <property name="pwd" value="password"/>

    <get src="http://teamcity.server.com/httpAuth/action.html?add2Queue=<clean&defrag conf id>&amp;agentId=allEnabledCompatible"
         dest="response.txt" verbose="yes" username="${user}" password="${pwd}"/>

    <get src="http://teamcity.server.com/httpAuth/action.html?add2Queue=<reboot conf id>&amp;agentId=allEnabledCompatible"
         dest="response.txt" verbose="yes" username="${user}" password="${pwd}"/>
  </target>
</project>


Where <clean&defrag conf id> and <reboot conf id> are identifiers of the corresponding build configurations (buildTypeId from the TeamCity URLs).

"Run all" build configuration is triggered by TeamCity scheduling trigger at Saturday night. The build of this configuration adds "Clean&Defragment" and "Reboot" builds in the queue. It uses not yet released feature of TeamCity allowing to start a build on all compatible and enabled agents (note agentId=allEnabledCompatible in the URL). So if you do not want "Reboot" builds to run on some agent just make this agent incompatible with "Reboot" configuration! Pretty simple, isn't it?

So far the solution works nice. And if your builds started to run slower try to setup something like this, probably the reason is not in the builds itself. BTW the new feature to run build on all compatible agents will be available in the next EAP build.

Wednesday, October 8, 2008

TeamCity in Google Chrome

We try things here at JetBrains, so of course we tried Google Chrome on the day it was released. Some of us even switched to Chrome as their main browser.

Apart from just trying the new browser we were also curious whether TeamCity will work OK in Chrome. No surprises here, we did not spot anything broken (since TeamCity works OK in Safari/WebKit).

One of the cool features of Chrome is making web application out of a page and we tried that with TeamCity. It worked out of the box, but we decided to make use of Google Gears API to provide default text for the Create Shortcut dialog and also provide several sizes for the TeamCity favicons so that Chrome can put on the desktop the right size instead of scaling 16x16 default.

Several meta and link tags more and we're in. The latest EAP already has the changes.
If you have it installed, you can get the different-sized icons via URLs:

your_teamcity_server/img/icons/TeamCity16.png
your_teamcity_server/img/icons/TeamCity32.png
your_teamcity_server/img/icons/TeamCity48.png
your_teamcity_server/img/icons/TeamCity128.png
your_teamcity_server/img/icons/TeamCity512.png

Huh? 512? Yes. Just in case someone will need it that big :)

Here is also a related post on DZone.

CITCON Amsterdam 2008

Two of our team: Pavel Sher and Yegor Yarko attended Continuous Integration and Testing Conference (CITCON) whose European meeting was held in Amsterdam this year.

It was the first full-time OpenSpace conference I've attended and it was fun to see how it works. It's really amazing how a non-managed process can work out so well.

It was a pleasure to meet TeamCity users, developers from other CI tools and other insightful people. Here is a small CITCON photo set.

Shall we meet at the next CITCON Europe?

Calcutta EAP, build 7757

On Thursday we've released a new EAP build, but I was a bit in a hurry leaving for a flight to Amsterdam to post the announcement here. But better late then never :)

This build introduces enhancements in the builds dependencies, namely: option to reuse the builds if they have no changes rather then rebuilding them each time for an upstream build; ability to configure artifact dependency to use the dependency built on the same sources; support for the builds sequences in the personal builds and display of the dependencies for a running/finished build.
If you try the feature, let us now how it suits your building process and what is still missing to fulfill your build dependencies requirements.

Other additions in the build are individual test details page with the test runs history and ability to commit a pre-tested commit if no new tests failed.
And yes, we've also added support for .Net code inspections tool: FxCop form Microsoft.

One of the features we started to work on is integration with externals source browsing tools. Some initial functionality is already baked in this EAP release.

Here is how to try it: Create a new file named change-viewers.properties in the .BuildServer/config directory with the content like:

;external change viewers config example
;all entries are optional
;general format is: rootId=external tool link template
*=http://some.tool/view/?changelist=${changeSetId}
;all roots of type 'svn'
svn=http://fisheye/changelog/REPO1/?cs=${changeSetId}
;specific root id
107=http://fisheye/changelog/REPO2/?cs=${changeSetId}

Then look for a link "Open in external change viewer" in the change description on the build's Changes page.
Do you like the feature? What tools do you need integration with?

Browse the full EAP release change log and download the build.

Your comments are welcome!