Meet a zulily developer: Bala

IMG_4180 

Who are you, and what do you do at zulily?

I am Bala, a Software Engineer on the EMS (Event Management Systems) Team. We create and operate the tools that help other teams launch and manage sales and deals. Our goal is to minimize the time it takes to get deals on zulily by providing efficient and easy to use tools.

When did you join zulily?

I started in October of 2013, so it has been a year.

What was it like in the early days?  Tell us a crazy story.

Although I officially joined at the end of October, I did not come to work for a week after my hire date. I had asked for a vacation for a week and I got one 🙂 I had very little work experience prior to zulily and was scared about the work culture in a start-up (which zulily was at that time). The first impression I have of zulily: this is a company I want to work for who cares about the employees.

On my actual first day of work at zulily, my manager greeted me and introduced me to the team. Luckily on the very first day we had “All Hands” meeting. It was a small auditorium and everyone was gathered there and the host started calling out people who joined that week. All the new hires gathered on the dais and there was a grand welcome for us. They asked us a question: “Which Christmas Carol movie do you like the best?” Everyone was answering the question with their most liked movie name. People were shouting and clapping all the time. When it was my turn, they asked my what mine was. I wanted to steal others’ ideas as I have not seen many Christmas Carol movies. So, I made up my mind and said “I have not seen any Christmas Carol movies but I like Iron Man.” I thought people would be laughing, but it was another round of applause and screams. I loved the energy of people at zulily. This is small but something I cherish working here.

One week later zulily went public and we celebrated that day in the Auditorium. I never had a chance before to experience the vibe when the company you work for goes public. It was epic. There was a countdown and people were cheering and later we heard Darrell’s (the CEO) live speech from the NASDAQ office. What a day it was!

Later I was involved in talking to a lot of people from the business. I was told about the fast-paced environment at zulily and “zulily time.” I didn’t really understand it until I released a new feature for vendors called “Vendor Inventory Update Automation” in my first few weeks at zulily. After that I never turned back….

What is different now?

zulily has grown a lot. But zulily still moves very fast and is very aggressive. The tech team has doubled in size, there are more people you would be able to work with and the development vision has changed from “Get it out now. We can think of the future later.” to “We need to do it right and make it useful for the future.” We also have PMs to help us to define the priorities and let us code more and attend fewer meetings.

What’s a typical day like for you?

I get into the office around 10am. By the time I come in most folks are here. I check my mail, look at my calendar and plan my day. I will be so engrossed in coding that I forget to eat sometimes. I generally keep reminders for that! I keep coding and attend meetings. I go back home when I feel that I have completed something concrete. I bike and bus to the office and I take the time on the bike to catch-up with what’s going on in this world. I go back home and spend some time with my family and then if I get some time I read something or else I go to bed… repeat.

What gets you excited about working about working at zulily?

I agree with everyone about how much of an impact your changes and work have on the routine of the people at zulily. As I work on internal tools for zulily employees I get a chance to meet my customers directly, talk to them, get accurate requirements and build tools that cater to their needs. This kind of customer interaction is something I love about working at zulily. Also, I own what I build and I support it, which motivates me to code better. Most importantly: all the people I work with are awesome.

Expect: How a 20-Year-Old Tool Saved My Project

I joined zulily in August of 2010, and at the time the company as a whole consisted of only 35 people. One of my first projects involved integrating one of our systems with a system owned by a much larger, more established partner company. The details of what these systems did aren’t relevant, except that the integration mechanism was for our system to drop XML files on an SFTP server that the partner company owned and operated.

At the time we were a 100% PHP shop (this has since changed), so I implemented our side of the integration in PHP, and used an open source PHP library called phpseclib to handle the actual SFTP data transfer. The partner company didn’t have any clients who used PHP, and didn’t officially support this library, but it worked great throughout development and integration testing. The integration test phase of the project took approximately 3 months, and not once during that time did we ever have even the slightest hiccup when transferring data between systems.

However, once we went to full production, we started seeing file corruption — specifically, sometimes files we transferred to the partner’s SFTP server would be truncated. There was no discernible pattern to the truncation; it happened at different points every time, and often a file that failed once would work fine when it was retried.

Naturally, this caused some consternation, as our code hadn’t changed, and it had been working for months without fail. When I pointed the exact same code at their test server, and sent the exact same file content, everything worked flawlessly. When I pointed it back to their production server, the files were truncated.

Clearly, to my mind at least, the problem was on their end. After a couple of days of frenzied troubleshooting, we discovered that the version of the SFTP server software running on their test machines was newer than what they ran in production. Presumably, we were hitting some bug in that software that was fixed between the two versions.

Since they were a very large company with many other clients using these servers, they were unwilling to upgrade their SFTP software on our behalf. Also, given that we were already well into the go-live phase of the integration, rewriting our system in another language wasn’t an appealing option, especially since there was no guarantee that the new implementation would work any better.

One option that the parent company did officially support, though, was the sftp client included with OpenSSL. I tried manually transferring a few files this way…and the file truncation issue disappeared.

There were problems with this approach, though: for one, the SFTP server required authentication, and the partner company was unwilling to set up SSH keys for us to do non-interactive authentication. OpenSSL’s sftp client doesn’t support setting the authentication credentials via command line parameters, leaving us stuck with authenticating interactively. This obviously wasn’t an acceptable long-term solution, since these systems needed to communicate with each other without human intervention.

I don’t recall exactly when I stumbled across it, but somewhere in the midst of searching for a solution I came across Expect.

Quoting the introduction on the Expect homepage:

Expect is a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc. Expect really makes this stuff trivial.

This sounded exactly like what I needed! I bought a copy of “Exploring Expect“, read enough of it on the train ride home to get started, and after a couple of hours I had whipped up a PHP script that did the following:

  1. Dump the XML data we wanted to transfer into a local temp file.
  2. Build an Expect script in memory by doing some variable interpolation in a PHP string.
  3. Write the resulting Expect script to another local temp file.
  4. Exec() the Expect script, capturing the process return code and anything the script wrote to stdout or stderr.
  5. Write the process output to our application log for later troubleshooting if anything went wrong.
  6. Finally, if the process return code was zero (indicating success), delete the two temp files. Otherwise, send an email to our notification alias so I could investigate.

Here’s the meat of the Expect script generation in all its glory:

$expectContents =
 '#!/usr/bin/expect' . "\n"
 . 'set timeout ' . $this->m_timeout . "\n"
 . 'spawn sftp -o Port=' . $this->m_port . ' ' . $this->m_user . '@' . $this->m_host . "\n"
 . 'expect {' . "\n"
 . ' default {exit 1}' . "\n"
 . ' "Connecting to ' . $this->m_host . '..."' . "\n"
 . '}' . "\n"
 . 'expect {' . "\n"
 . ' default {exit 2}' . "\n"
 . ' "continue connecting (yes/no)?" {send "yes\n"; exp_continue}' . "\n"
 . ' "ssword"' . "\n"
 . '}' . "\n"
 . 'send "' . $this->m_pass . '\n"' . "\n"
 . 'expect {' . "\n"
 . ' default {exit 3}' . "\n"
 . ' "ermission denied" {exit 4}' . "\n"
 . ' "sftp>"' . "\n"
 . '}' . "\n"
 . 'send "cd /inbound\n"' . "\n"
 . 'expect {' . "\n"
 . ' default {exit 5}' . "\n"
 . ' "not found" {exit 6}' . "\n"
 . ' "No such file" {exit 7}' . "\n"
 . ' "sftp>"' . "\n"
 . '}' . "\n"
 . 'send "put ' . $filename . '\n"' . "\n"
 . 'expect {' . "\n"
 . ' default {exit 8}' . "\n"
 . ' "not found" {exit 9}' . "\n"
 . ' "sftp>"' . "\n"
 . '}' . "\n"
 . 'send "quit\n"' . "\n"
 . 'exit 0' . "\n";

Yes, I’m aware that there are more elegant ways to interpolate strings in PHP than this, but at the time I was still fairly new to PHP and under a ton of pressure to get something — anything — working.

Coming from a background in statically typed compiled languages, this just felt wrong somehow. Even my stints in the land of Perl didn’t feel this hacky. It was an ugly, nasty, weird abomination…but it worked.

And it kept working, without fail, for the entire lifespan of this system. Over time, we encountered bugs in many other areas, as is inevitable with any complex system, but this little corner of the codebase never had a single issue.

What this taught me is that sometimes you should go slowly, be methodical, and take the time necessary to create a simple, elegant solution to your problem.

And sometimes you just need to break out the duct tape.