Adventures with escaping quotes in Jenkins pipelines

It seems inevitable that if you deal with writing pipelines in Jenkins, you’ll run into issues dealing with quotation marks in commands. Escaping quotes is a special nightmare in shell scripts generally, but Jenkins adds yet another layer of confusion on top of that. I’m going to cut through that layer and find out what Jenkins is actually executing.

Today I was trying to format a shell command that included a user argument that could contain special characters. The safest way to handle this is usually to single quote the argument:

mycommand --input 'this string could be anything'

In a Jenkins pipeline, this whole thing has to be put into a String, meaning it has to be wrapped in quotes itself. At first glance, we can just wrap it in double quotes so single quotes don’t need to be escaped, and we get string interpolation for the input:

sh "mycommand --input '${input}'"

When you run this, however, you may notice that the console log for the job actually spits out something different, depending on the value of input:

String input = 'oneword'
sh "mycommand --input '${input}'"
// output in log: + mycommand --input oneword

String input = 'multiple words'
sh "mycommand --input '${intput}'"
// output in log: + mycommand --input 'multiple words'

It appears as if Jenkins is dropped the quotes when they aren’t needed. What?

A little googling quickly turns up this horrifying snippet on github: https://gist.github.com/Faheetah/e11bd0315c34ed32e681616e41279ef4

If you’re anything like me, you’ll quickly go cross-eyed trying to parse through all those quotes and slashes, and more questions come up than answers. Does the behaviour change if you’re using single or double quotes to wrap the Groovy string, for example? It does not help that these examples use echo as the command to demonstrate that, since that alone can appear to take liberties with quotations:

echo hello world
> hello world
echo 'hello world'
> hello world
echo \'hello world\'
> 'hello world'

The difference between those first two is how many arguments shell “sees”: the first has two arguments, “hello” and “world”, while the second only has one argument: the single string “hello world”. Quotes around shell arguments aren’t taken to be part of the argument, unless they’re escaped.

In fact, where it looked like Jenkins was dropped quotes in the log, it was actually shell itself. If you run sh -x, you can see the same behaviour:

sh-3.2$ echo 'oneword'
+ echo oneword
oneword
sh-3.2$ echo 'multiple words'
+ echo 'multiple words'
multiple words

The lines that start with + are shell telling you what shell is executing. That’s all that Jenkins is showing you in its log. Shell drops those single quotes when they aren’t needed, not Jenkins.

So, back to my problem. When trying to debug a shell script where quotes or special characters need to be escaped, it can be hard to tell where an error might be coming from if you can’t see what was actually executed. Did I escape them properly and they just don’t show up in the log, or did they not get through to shell at all? I did not want to start adding backslashes until it worked. At one point today, just to replace a single quote in a string with \' I was doing this:

String input = "this isn't easy"
String escaped = input.replaceAll("'", "\\\\'")
println escaped // output: this isn\'t easy

Four blackslahes? What? No wonder that github gist above is a mess. Don’t even get me started on “slashy strings” or “dollar slashy strings”.

Whatever mess you find yourself in, the question comes back to: What is Jenkins actually passing to shell? This error message was my clue:

/var/jenkins_home/jobs/My_Job/workspace@tmp/durable-cb025511/script.sh: 4: /var/jenkins_home/jobs/My_Job/workspace@tmp/durable-cb025511/script.sh: Syntax error: Unterminated quoted 

Jenkins actually writes the script from the pipeline file to a script.sh file somewhere on disk. The file only exists briefly, in a directory with a random name, but it’s enough that if I time things right (or slow things down enough), we can peek at that file and make sure that the actual honest-to-goodness shell script—no longer mediated through Groovy or Jenkins—is what we expect it to be:

find /var/jenkins_home/jobs/My_Job/workspace@tmp/ \
  -name 'script.sh' \
  -exec cat '{}' \;

This looks for any file named script.sh anywhere in the job’s workspace tmp files and prints it to the terminal. (Locally I run Jenkins in a docker image called jenkins-dev, so I prefaced this command with docker exec jenkins-dev). I wondered if I might be able to get the script to print itself to the log automatically, but I wasn’t sure if I would trust the log even if I could.

This confirmed for me that what appears in the build’s console log is not actually what Jenkins ran, because of the way shell modifies things when it echos commands. Could I have just echo’d the constructed command?

String command = "some complicated stuff"
echo command // for debugging
sh command

It turns out, yeah probably. But again, when you’re in the headspace of “I can’t trust this log file”, the fewer variables between you and what the shell receives, the better.

Finally, how did I end up escaping single quotes that user input string? For this I fell back on an old bash trick of quintuple-quoting: stop the string you’re in, wrap the single quote in double quotes, and then start a new string for everything after the quote. It looks a mess:

echo 'this ain'"'"'t easy'
// this ain't easy

but it’s just breaking it down into three strings:

1. this ain
2. '
3. t easy

and concatenating them together.

I started with very little hope that I’d be able to create this string in Groovy given the adventures with backslashes we had before, but this time slashy-strings actually made it simple:

String input = "this ain't easy"
String escaped = "'" + input.replaceAll(/'/, /'"'"'/) + "'"
println escaped
// 'this ain'"'"'t easy'

No backslash-escaping required! Phew. Now I can pass that into shell, and it gets interpreted correctly:

sh "mycommand --input ${escaped}"