Integrating a CDN into your Sitecore Solution

This blog post will take you through how we have implemented functionality to integrate our Sitecore solution to a CDN. We will talk about Media Providers, Event Handlers, and more! Read on! 🙂

Why use a CDN

Integrating a CDN with your website is a good idea. First it allows static content such as images to be downloaded from a geographic friendly location, this decreases latency for your visitor. This also means your web server does not need to handle these requests.

Another benefit of using a CDN is that the browser only downloads 2 requests from a domain at a time (this is actually incorrect, most modern browsers now do up to 6 requests, thanks to Jesper Balle for the updated information!), therefore if you increase the amount of domains, the more content you can download at one time.

How to integrate a CDN

Some say that integrating a CDN into an already existing website can be a difficult task. During the development phase of the new GoCollette.com we had time in the schedule that called for a CDN to be implemented. Like many projects we ran up against the clock and needed to implement the CDN post deployment (about 6-weeks after deployment to be exact). The good thing is, I didn’t find it very difficult. Let me show you how I went about it! 🙂

Step 1: Modifying Sitecore Templates

In order for the Sitecore to come up with the correct URL for the media item (MediaManager.GetMediaUrl), it needs to know whether the media has been flagged as added to the CDN.  We are flagging the media items simply by adding a checkbox to the versioned and unversioned File template (/sitecore/templates/System/Media/Unversioned/File, /sitecore/templates/System/Media/Versioned/File).  This identifies that media item has been uploaded to the CDN if the “Pushed to CDN” field is checked.

mediaTemplate

 

Step 2: Uploading and Setting the Pushed to CDN Flag

As you are probably familiar, Sitecore has an extensive set of pipelines and events which allow you to perform all kinds of custom actions.  As shown below we are tapping into the item:saved event and adding our own custom handler, PushMediaToCDN. This allows us to have our logic executed whenever an item is saved.

<events>
	<event name="item:saved">
		<handler type="Collette.Library.CustomSitecore.Events.Saved.PushMediaToCDN, Collette.Library" method="OnItemSaved"></handler>
	</event>
</events>

In our custom handler, PushMediaToCDN, the OnItemSaved method has one basic responsibility, verify the item that is being saved needs to have its media pushed up to the CDN. To determine this we run through a variety of checks, if any of them do not meet the criteria, we abort.

  • Verify the item passed in the event args is not null (always check for nulls!)
  • Verify the item is a media item
  • Verify the item is being saved in the master database(no need to run on web, media should only be uploaded/modified on the master database).
  • We also check to see if this media has been previously FTP’d to the CDN by looking at our flag we created in Step 1.
protected void OnItemSaved(object sender, EventArgs args)
{
	if (args == null)
	{
		return;
	}

	//take item from args
	SitecoreEventArgs sitecoreArgs = args as SitecoreEventArgs;
	Assert.IsNotNull(sitecoreArgs, "eventArgs");

	SitecoreEventArgs eventArgs = args as SitecoreEventArgs;
	Assert.IsNotNull(eventArgs, "eventArgs");

	Item item = eventArgs.Parameters[0] as Item;
	if (item == null
              || item.Database.Name.ToLower() != "master"
              || !item.Paths.IsMediaItem)
	{
		return;
	}

	MediaItem mediaItem = item;
	if (mediaItem.InnerItem["Pushed to CDN"] == "1")
	{
		return;
	}

	_mediaItem = mediaItem;
	if (string.IsNullOrEmpty(mediaItem.Extension))
	{
		return;
	}

	Stream mediaStream = mediaItem.GetMediaStream();
	if (mediaStream == null || mediaStream.Length == 0)
	{
		return;
	}

	if (JobManager.IsJobRunning("CdnMaintenance" + mediaItem.ID))
	{
		return;
	}

	BeginUpload(mediaItem, mediaStream);
}

If all the conditions pass, we call the method BeginUpload method which is responsible for:

  • Creating and mimicking the Sitecore folder structure in the CDN
  • Retrieving the media into a binary stream.
  • Uploading the binary media to the CDN
  • Setting the “Pushed to CDN” field if the upload was successful
private void BeginUpload(MediaItem mediaItem, Stream mediaStream)
{
	Sitecore.Context.Job.Status.State = JobState.Running;

	//get parts-folders
	List<string> pathParts = GetPathParts(mediaItem);

	//verify and build folder structure
	string folderPath = VerifyParts(pathParts);

	//get filename without .ashx extension
	string fileName = string.Format("{0}.{1}", mediaItem.Name, mediaItem.Extension).ToLower();

	//final directory and filename
	string directory = string.Format("{0}{1}/{2}", FtpUrl, folderPath, fileName);

	FtpWebRequest ftpClient = (FtpWebRequest)FtpWebRequest.Create(directory);
	ftpClient.Credentials = new NetworkCredential(FtpUsername, FtpPassword);
	ftpClient.Method = WebRequestMethods.Ftp.UploadFile;
	ftpClient.UseBinary = true;
	ftpClient.KeepAlive = false;
	ftpClient.ContentLength = mediaStream.Length;

	byte[] buffer = new byte[4097];
	int bytes = 0;
	int total_bytes = (int)mediaStream.Length;

	Stream rs = ftpClient.GetRequestStream();
	while (total_bytes > 0)
	{
		bytes = mediaStream.Read(buffer, 0, buffer.Length);
		rs.Write(buffer, 0, bytes);
		total_bytes = total_bytes - bytes;
	}

	mediaStream.Close();
	rs.Close();

	try
	{
		//upload to CDN
		FtpWebResponse uploadResponse = (FtpWebResponse)ftpClient.GetResponse();
		string value = uploadResponse.StatusDescription;
		uploadResponse.Close();

		//if we did successfully upload the file to the cdn, set the media item checkbox
		using (new SecurityDisabler())
		{
			using (new EditContext(mediaItem.InnerItem))
			{
				mediaItem.InnerItem["Pushed to CDN"] = "1";
			}
		}
	}
	catch (Exception ex)
	{
		Logger.Error("Push Media to CDN: Unable to push media to CDN.");
		Logger.Error(ex.Message);
	}

	Sitecore.Context.Job.Status.State = JobState.Finished;
}

Step 3: Tapping into the Media Manager

Retrieving a url for a media item in Sitecore is very easy. We can call the static method GetMediaUrl provided by Sitecore’s MediaManager. Another provider that is highly configurable within the configuration files. If you dig deep enough the MediaManager ends up calling the MediaProvider and this is where we can start to customize again 🙂

In our custom configuration file, we have added a hook for our own media provider and setup a couple of attributes that allow us to manage the base url to the CDN as we well as which sites we want this functionality to be run against. Border-line overkill but I don’t like to hardcode things.

<hooks>
	<hook type="Collette.Library.CustomSitecore.Media.MediaProvider, Collette.Library">
		<originPrefix>http://i.gocollette.com</originPrefix>
		<sites>collette-us|collette-uk|collette-ca|thomasCook|aaa</sites>
	</hook>
</hooks>

To make your own MediaProvider class you need to inherit from Sitecore’s own MediaProvider and because we are using a hook within the configuration, we need to implement the IHook interface. For more on hooks, read John West’s blog post on hooks.

Conceptually our custom provider is very simple. There are two methods we are overriding (shown below), in both methods we call the base method to have Sitecore give us the media url, then call our own GetMediaUrl. Our method is responsible for making sure:

  • Verify we have a base url to the CDN as defined in the configuration
  • That the current site is one of the allowable sites listed in the configuration
  • Verify that the url returned from Sitecore has not been expanded to a full url (http://www.somewebsite.com/~/media/somemedia.ashx). This happens depending on your configuration settings and/or if you are using Sitecore’s Email Campaign Manager (emails need to have the full url embedded into them).
  • Verify that our custom field, Pushed to CDN, has been flagged.

Once these checks have been done, we do a little cleanup on the url itself and return the media url to the requestor. Let’s take a look at the before and after:

Before:
/~/media/images/cont/eu/it/trevifountain_41733867_fotoliarf_3176_960x380.ashx
After:
http://i.gocollette.com/images/cont/eu/it/trevifountain_41733867_fotoliarf_3176_960x380.jpg

public class MediaProvider : Sitecore.Resources.Media.MediaProvider, IHook
{
	public void Initialize()
	{
		MediaManager.Provider = this;
	}

	public override string GetMediaUrl(MediaItem item)
	{
		string mediaUrl = base.GetMediaUrl(item);
		return GetMediaUrl(mediaUrl, item);
	}

	public override string GetMediaUrl(MediaItem item, MediaUrlOptions options)
	{
		string mediaUrl = base.GetMediaUrl(item, options);
		return GetMediaUrl(mediaUrl, item);
	}

	/// <summary>
	/// Determines if we should be pulling from the CDN or not
	/// </summary>
	/// <param name="mediaUrl"></param>
	/// <param name="item"></param>
	/// <returns></returns>
	public string GetMediaUrl(string mediaUrl, MediaItem item)
	{
		//verify the domain was set in the config
		if (string.IsNullOrEmpty(OriginPrefix))
		{
			return mediaUrl;
		}

		//make sure this is one of the allowed sites defined in the config
		if (!AllowedSites.Contains(Sitecore.Context.GetSiteName()))
		{
			return mediaUrl;
		}

		//if the url is already expanded than ignore
		//(this could happen with emails)
		if (mediaUrl.Contains("http"))
		{
			return mediaUrl;
		}

		//verify the item has been pushed to the cdn
		if (item.InnerItem["Pushed to CDN"] != "1")
		{
			return mediaUrl;
		}

		//clean the url
		mediaUrl = mediaUrl.Replace("/~/media/", "/");

		//this happens while indexing unless the proper site is set
		mediaUrl = mediaUrl.Replace("/sitecore/shell/", "/");

		//reference the file in the cdn by the actual extension
		mediaUrl = mediaUrl.Replace(".ashx", "." + item.Extension);

		mediaUrl = string.Format("{0}{1}", OriginPrefix, mediaUrl);

		if (HttpContext.Current != null && HttpContext.Current.Request.IsSecureConnection)
		{
			//if we are on a secure connection, make sure we are making an https url over
			//to the cdn
			mediaUrl = mediaUrl.Replace("http://", "https://");
		}

		return mediaUrl;
	}

	/// <summary>
	/// Property defined in the config
	/// </summary>
	public string OriginPrefix { get; set; }

	/// <summary>
	/// Property defined in the config
	/// </summary>
	public string Sites { get; set; }

	/// <summary>
	/// Sites that are allows to use the CDN Media Provider
	/// </summary>
	public List<string> AllowedSites
	{
		get
		{
			if (string.IsNullOrEmpty(Sites))
			{
				return new List<string>();
			}

			return Sites.Split('|').Where(x => !string.IsNullOrEmpty(x)).ToList();
		}
	}
}

The functionality we just went over is great if you want to maintain a CDN and feel confident that everything in Sitecore is in the CDN. You might be asking: How did you get all the images there in the first place? That’s a very good question and something I didn’t touch on above. Once the above structure was working, I created a simple util page that went through all the media items and called the save method on them. This triggered our custom event handler, which went through some checks and pushed the binary media stream to the CDN using FTP. That was it, it took a little bit of time for the util page to run but since we implemented the functionality it has been working like a charm.

If you have any questions or if you feel I may have left something out, feel free to reach out. Happy to help! 🙂
Enjoy!
Tim

4 thoughts on “Integrating a CDN into your Sitecore Solution

  1. Jesper Balle

    I think that the 2 simultaneously requests per domain is an old number. As far I know all/most modern browsers do 6 requests or similar. From IE8 I think it was 6 if you were on broadband, but IE7 is 2

    Reply
  2. Pingback: A Sitecore CDN Integration Approach | Microsoft Enterprise Technologies

  3. Pingback: Sitecore Media Library integration with Azure CDN using origin pull | Brian Pedersen's Sitecore and .NET Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s