Agents maintenance

If you read this blog from the beginning you probably know that in JetBrains we use quite a big farm of agents. The agents produce a large amount of work. According to our statistics usage of the most of our 45 agents is around 40-50% (there are no builds at night time).

Moreover some of our builds produce a large number of small files on disk. The files are then deleted by build itself or by agent but this increases fragmentation of the file system. Not surprisingly these builds started to run slower, and we decided that we need some kind of agents maintenance policy. Ideally we need to defragment agent disk and probably reboot agent once per week. Well, since we already have agents installed on these PCs why not do maintenance tasks with еру help of the agents themselves?

The solution

Note: we did not experience performance problems with Unix/Linux based agents so all the scripts below are for Windows only.

First of all we created a project in TeamCity and set necessary access rights. Authorization is important since maintenance tasks are potentially harmful and only experienced users should be able to configure and run them.

Then we created three build configurations:
1) Clean & Defragment
2) Reboot
3) Run all

"Clean & Defragment" build configuration removes temporary files from the user home directory and starts disk defragmentation. Currently it is Ant based, and here is the script we use:

<project name="AgentsMaintenance" default="defragment" basedir=".">

  <target name="defragment">
    <echo message="##teamcity[progressMessage 'Defragmenting ...']"/>
    <exec executable="defrag">
      <arg value="C:"/>
      <arg value="-v"/>
    </exec>
  </target>

  <property environment="env"/>
  <property name="userTemp" location="%env.USERPROFILE%/Local Settings/Temp"/>

  <target name="clean">
    <echo message="##teamcity[progressMessage 'Cleaning ${userTemp} directory...']"/>
    <delete quiet="yes" failonerror="false" includeemptydirs="true">
      <fileset dir="${userTemp}" includes="**/*"/>
    </delete>
    <mkdir dir="${userTemp}"/>
  </target>

</project>

"Reboot" build configuration reboots an agent. It uses command line runner and simply invokes the following command:

shutdown -r -f -t 20 -c "Reboot build agent"

And finally the main step is build configuration "Run all". Here is the Ant script:

<project name="AgentsMaintenance" default="runAll" basedir=".">

  <target name="runAll">
    <property name="user" value="username"/>
    <property name="pwd" value="password"/>

    <get src="http://teamcity.server.com/httpAuth/action.html?add2Queue=<clean&defrag conf id>&amp;agentId=allEnabledCompatible"
         dest="response.txt" verbose="yes" username="${user}" password="${pwd}"/>

    <get src="http://teamcity.server.com/httpAuth/action.html?add2Queue=<reboot conf id>&amp;agentId=allEnabledCompatible"
         dest="response.txt" verbose="yes" username="${user}" password="${pwd}"/>
  </target>
</project>

Where <clean&defrag conf id> and <reboot conf id> are identifiers of the corresponding build configurations (buildTypeId from the TeamCity URLs).

"Run all" build configuration is triggered by TeamCity scheduling trigger at Saturday night. The build of this configuration adds "Clean&Defragment" and "Reboot" builds in the queue. It uses not yet released feature of TeamCity allowing to start a build on all compatible and enabled agents (note agentId=allEnabledCompatible in the URL). So if you do not want "Reboot" builds to run on some agent just make this agent incompatible with "Reboot" configuration! Pretty simple, isn't it?

So far the solution works nice. And if your builds started to run slower try to setup something like this, probably the reason is not in the builds itself. BTW the new feature to run build on all compatible agents will be available in the next EAP build.

500 build hours per day

You might not know but here at Jetbrains we are using our own TeamCity server to build almost all of our projects (including such giants as IntelliJ IDEA and Resharper). This is a part of our company strategy proudly named "eat your own dog food". And it works quite well.

Big projects usually require much time to build and run tests, for example, to compile IntelliJ IDEA sources you will need about 16min on a relatively modern PC. While to run a full test suite you'll need almost two hours. The same picture is in Resharper project: the whole build with compilation and tests takes more than 2 hours to complete. Given such a long builds it is critical to be able to run them in parallel. That is why TeamCity has such feature as build grid from the beginning, otherwise we won't be able to use TeamCity at all.

But how much agents will be enough for us? Well, when we released TeamCity 1.0 in October 2006 we had a pool of about 20 agents:




Things did not change much when we released 2.0 version in April 2007 (except that now we had a so-called "glass" indicating current server load :)






Then as more Jetbrains projects moved to TeamCity we started to realize that we need more power. In December 2007 when TeamCity 3.0 released we had more than 30 agents:




Interestingly even with 33 agents we have almost as many builds waiting in the queue :)

So did we stop? Nope! Thanks to our selfless administrators we increased our pool again:



Now we are able to build more than 700 builds per day (all of them took >500 hours, or about 20 build days) and publish more than 10Gb of artifacts. I think it is serious power. BTW our server is installed on a PC with dual core CPU 3.2GHz / 4Gb RAM of that TeamCity uses 750Mb only.

Do you think so much agents will be enough for us? Will see :)