Dependency Cache and Sphinx Support on Codeship

Codeship News

Reading Time: 3 minutes

This week we are happy to announce a big change that we released two weeks ago on Codeship: Our new Dependency Cache.

Why cache dependencies

For every build we provide a clean virtual machine. The advantage is that our customers can be sure that their tests run under the same conditions every time. Any change done during the build is automatically removed from our system after the build finishes. This makes installing dependencies slow as we need to install them for every build.

We pre-installed thousands of Ruby Gems in the past, but the dependency installation still took a considerable amount of time for every build. This was not just a bad experience for all of our users, but also used a lot of our server’s resources. Furthermore we wanted to speed up dependency installation for other languages we support.

Sign up for a free Codeship Account

Whenever a build finishes now, we store a tar archive of all dependencies. For the next build we extract that same archive back into the virtual machine. This happens completely transparent. Only when you add a dependency it needs to be installed. The next time it will be used from the cache.

The difference is enormous. The average bundle install to install Ruby Gems took 220 seconds on average in the week before we released the cache. It went down to 10 seconds on average.

Installing pip packages for python went down from 97 seconds on average to 12 seconds. And the same goes for other languages as well. For our customers this means up to double the speed for their build and they are loving it.


Customer statements about our new Dependency Cache

Mixcloud
“FYI loving the dependency cache, so so so much better!”
Mat Clayton – Co Founder & CTO

Ambassador
“We’re amazed at how fast our builds are now. Fetching dependencies used to take minutes and slowed us down as we waited for feedback on our commits. Thanks to Codeship’s dependency caching it takes seconds, and we can get right back to work.”
Chase Lee – CTO


How the Dependency Cache works

We store the tar archives on S3. We make sure they aren’t broken by checking their MD5 sums whenever we upload or download an archive. If the download fails, we retry. If we cannot restore the cache several times, we run the build without the cache. This will slow down the build, but make sure that it can proceed without interruption.

The cache was in beta for a couple of weeks. We pushed it into production two weeks ago and it works very well.


Language support

Ruby

We overwrite the default path bundler installs gems into and cache that folder.

Python

All dependencies installed through pip install are cached.

Node.js

All packages installed through npm install are cached.

PHP

PHP is still giving us headaches on how to cache dependencies properly. We have been trying out various ways, but so far we haven’t found a method that works for all of our customers. We are working on this and will ship an update soon.

Java

We are currently caching the ~/.m2/repository folder for dependencies installed through Maven.


Sphinx full text search engine integration

Sphinx 2.0.4 runs for every build on the default port.


ZeroMQ support

Libraries for ZeroMQ 2.2.2 are now installed on our system.


Conclusions

With our new Dependency Cache you will get even faster feedback for your builds. Give it a try and see how Codeship can speed up your build and deploy times.

Subscribe via Email

Over 60,000 people from companies like Netflix, Apple, Spotify and O'Reilly are reading our articles.
Subscribe to receive a weekly newsletter with articles around Continuous Integration, Docker, and software development best practices.



We promise that we won't spam you. You can unsubscribe any time.

Join the Discussion

Leave us some comments on what you think about this topic or if you like to add something.

  • Edgars Lazdins

    Well done! Eagerly awaiting PHP support for this. :)

    • Codeship

      Thank you Edgars!

      • Aaron

        Cache things installed with composer?

        • beanieboi

          Hey Aaron,

          currently we don’t support composer since there are some issues regarding a unique cache path and restoring composer installed extensions reliabliy.

          best,
          ben

  • A.T. Fouty

    We’ve definitely seen an improvement in build speed (Django/python) and it’s much appreciated! Huge fans of codeship!

    • Codeship

      Thanks for your comment. Yes, we love it as well! Glad to have you on the ship :)

  • Gustavo Hingel Morada

    How can I clear the cache? I have to uninstall some dependencies

    • Currently there is no way to do that as a user. Could you send an In-app message and we will remove your current cache.

      • We now have built in support for clearing the dependency cache via a (appropriately named) button on the bottom of the build details page. No in-app message necessary any more! ;)

  • Zoltan Altfatter

    Using Gradle, would be also nice to cache ~/.gradle folder in order to cache the gradle downloaded by the gradle wrapper. The default gradle version (1.10) is quite old.

    • Ahoy, you can by adding the following snippet to your project settings

      export GRADLE_USER_HOME=”${HOME}/cache/gradle/

  • Low Kian Seong

    Erlang support …!

  • Primož Cigler

    What about bower dependencies?

    And +1 for PHP composer.

    • Primož Cigler

      Also, does dependency cache also work if the deps are installed inside a shell script, not directly in the Setup Commands?

  • Xander

    PHP support there yet? Composer install is slow to watch

  • Configurable paths to cache would be nice… we use miniconda3!