What I Learned from Running Rundeck

• 2018/12/24

Background

We use Rundeck at work to manage jobs. It is handy, but it can also be a pain: the server CPU suddenly spikes, memory usage explodes, backups are tedious, and so on. Because I used to be a Java engineer, have touched Groovy, and know the basics of Rails, I decided to read Rundeck’s source code and dig into the architecture. Here is a summary for anyone who still manages a mountain of cron jobs by hand.

What is Rundeck?

• Rundeck is a job scheduler that, like cron, lets you run programs on a schedule.
• It gives you a visual way to manage jobs that would otherwise be hard to read inside crontab, and it helps tame a large number of jobs by grouping them by project and by target node so that you can fan out to multiple servers easily.

Benefits of using Rundeck

• Compared with plain cron, you gain visibility, can organize jobs per project, and can keep target host information in node definitions to orchestrate multiple servers.
• SSH keys can be stored and managed on the Rundeck server itself.
• Plenty of plugins exist—for example, to send Slack notifications when a job fails—so you do not have to script every integration by hand.

Downsides

• Even though Rundeck is open source, you may still see sudden CPU and memory spikes. (Countermeasures later.)
• If you need to run jobs against multiple clients, you need a dedicated Rundeck server.
• The GUI can feel sluggish.

Architecture highlights

• Because Rundeck is open source, the only operating cost is the server and database.
• The app is built with Grails (a Groovy framework that ultimately sits on Spring MVC), so if you understand Spring MVC you can follow the code path.
• The default database is H2, the embedded DB that ships with Java.
• Rundeck stores all job definitions in the database (login accounts live in files on the server). Embedded H2 adds load to the app server, so it is better to run a separate DB such as MySQL or PostgreSQL.

Finer-grained architecture notes

• Grails uses GSP (Groovy Server Pages) for the view layer.
• You get the g taglib so you can generate forms, anchors, and more dynamically.
• Because the foundation is Spring MVC, it relies on a DI container and creates the database the first time the app boots (see applicationContext.xml).
• grails.xml defines the DB schema.
• The grails-app directory contains the Grails application proper.
• Database access is implemented with Hibernate, the familiar Java ORM, so you will not find raw SQL in the codebase.
• Running EXPLAIN on the DB console showed no query design issues.
• Controllers call services, and the services hit the DB. If you are on MySQL, enable slow query logs and compare them with what Rundeck is doing.

Troubleshooting tips

• Slow DB access
  • Add indexes to heavily accessed columns.
• Slow list pages
  • Split large projects into smaller ones. The source code reveals that sample views perform full table scans, so reducing the number of rows per project shortens render time.
• High CPU usage
  • Configure job timeouts.
  • Consider whether the DB is simply underpowered. When memory runs out, the system starts paging to disk and CPU shoots up. Make sure the server specs match your workload (e.g., resize the instance class on AWS).
• Jobs piling up
  • As with the CPU issue, tune the connection pool size if needed.
  • On MySQL, enable slow query logging to find the bottleneck queries. In my experience, few queries are intrinsically bad because the app uses JPA-style constructs, so adding the right index usually fixes it.

Final note

If you know Java, read the source—it pays off.