Running Scheduled Perl Scripts in Azure Using Strawberry Perl and Quartz.Net

Background (or Why would I even think to do this?): I inherited some useful scripts that were written in perl. When I started working with them, it was far easier to refresh my decade-old knowledge of perl than to rewrite in some other language. The scripts update data in a database, and since my application is hosted in Azure, I decided to have these scripts write to SQLAzure. This is no problem in perl. It’s pretty much the same as connecting to any MSSQL db. Since the data that these scripts generate is time sensitive,  I really need to schedule them to run. (Most of them need to be run daily, and a couple of them more often.) Since my main app, which uses this data, runs in Azure already, and my database is SQL Azure, I decided that I should look into running these scripts from Azure itself. So now, I have these scripts running under my Azure Web Role deployment, scheduled with Quartz.Net.

The Project:

  • It includes a web role that includes a startup task. The startup task uses a powershell script to download Strawberry Perl from blob storage and then unzips it with 7zip. Lastly, the startup task installs required CPAN modules.
  • In the OnStart() method of the web role, a quartz.net perl job is scheduled to run  periodically (1 minute in the sample project, below).
  • When the quartz job is triggered to run, the perl job starts the perl process and executes the perl script. It captures standard output and standard error, and writes them to the trace logs.

That’s really all there is to it, though it’s trickier to set up than I had thought. I’ve posted the solution to github, to hopefully make this easier for the next person.

If you need to do this or something similar to this, I highly recommend the following articles/blog posts. They were incredibly valuable to me:

The Visual Studio 2010 solution is available on github https://github.com/squdgy/RunningPerlInAzure

After downloading, follow these steps to get a working solution:

  • use nuget to retrieve Quartz.net
  • download 7za.exe, the 7-Zip Command Line Version, available here: http://www.7-zip.org/download.html; place this in the root of WebRole1 and set “Copy to Output Directory” to “Copy always”
  • set the Diagnostics Connection String in ServiceConfiguration.Cloud.cscfg
  • set the url to the strawberry perl zip file in downloadPerl.ps1

This should run in the development environment and in Azure. You can browse to the startup log files in \approot\bin\startuplogs. In the development environment, during debugging, this would be under [Your Cloud Project]\csx\Debug\roles\WebRoot1. In the cloud I found this under an E:\ or F:\ drive (while connecting via RDP).

Some gotchas (or maybe tips):

  • Multiple instances of this role will each execute the perl scripts on its own schedule. Therefore, the scripts need to be idempotent. In the long run I don’t want this behavior, so the next step for this project is to use blob leases in the perl job. I plan to deploy the script changes via blob, so at the start of a job, the job will try to acquire a blob lease on the script. If the job can acquire a lease it will download the script, execute it, and release the lease. If the job cannot acquire a lease it won’t do anything. The scripts already contain logic to make sure the data was not already generated by a different run of the script. In this case a script will start, but it will notice that there’s nothing to do and end.
  • Because I used a feature of powershell 2, I needed to specify osFamily=”2”, so my role would run on Server 2008 R2. (ServiceConfiguration.Cloud.cscfg)
  • I initially used powershell and shell.application to unzip the perl zip file. This worked quite well in the development environment, but failed in Azure. After being unsuccessful at tracking down the cause, I switched to 7zip which just worked. I would have preferrred not having this dependency.
  • I made sure to log/trace almost every step of the startup. It was invaluable to be able to browse to the startup logs in the development environment, and later in Azure (via RDP) to see what was going on.
  • I had to explicitly add a DiagnosticMonitorTraceListener to my WebRole OnStart, so that standard output and standard error from my perl job execution would also be logged. (See Neil MacKenzie’s article)
Advertisements
This entry was posted in ASP.Net, Azure, Perl, Web Development. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s