MyDownloader: A Multi-thread C# Segmented Download Manager

• Download demo project – 155.4 KB
• Download source – 248.2 KB

Introduction

MyDownloader is an open source application written in C# that is almost a complete download manager. MyDownloader has many features to manage downloads:

• Segmented downloads from HTTP and FTP
  • With smart segments: when one segment ends, starts another segment to help to terminate another segment more fast
• Allow downloads to be paused and resumed
• (NEW) New friendly “New Video Download” window to download videos from: (you need save the file as *.flv)
  • You Tube
  • (NEW) Google Video
  • (NEW) Break
  • (NEW) PutFile
  • (NEW) Meta Cafe
• (NEW) Speed Limit — to avoid to use all your bandwidth
• (NEW) Download Scheduler
  • (NEW) Download files only on allowed times
  • (NEW) Limit the number of simultaneous downloads
  • (NEW) When one download ends, starts another automatically
• (NEW) Support for FTP site that requires authentication
• (NEW) Support for Mirrors
• (NEW) Download from HTTPS
• Notification download completion with sounds and XP balloon
• Anti-virus integration
• Automatic retry when a segment or download fails
• Batch downloads (enter a generic URL such as http://server/file(*).zip and MyDownloader generates a set of URLs with numbers or letters)

How a Segmented Download Works

Downloads can be segmented because both HTTP and FTP protocols allow the client to specify the start position of the stream. First, MyDownloader performs a request to the server to discover the file size. After that, MyDownloader calculates the segment size as follows:

segment size = min( (file size / number of segments),
    minimum allowed segment size )

With the segment size, MyDownloader creates another request specifying the start position of the stream. In this way, we can have multi-requests for the same files running in parallel using multi-threading techniques. This technique speeds up the transfer rate even more if you are using mirrors.

Using the Code: MyDownloader API

To start a segmented download using the MyDownloader API is very simple. Check the code below, extracted from the MyDownloader source code. When the download is finished, an XP balloon is displayed near the windows clock:

Collapse

// starts to listen to the event 'DownloadEnded' from DownloadManager

DownloadManager.Instance.DownloadEnded +=
new EventHandler<DownloaderEventArgs>(Instance_DownloadEnded);

// indicates that download should start immediately

bool startNow = true;

Downloader download = DownloadManager.Instance.Add(
    "http://jogos.download.uol.com.br/videos/pc/thewitcher12.wmv",
    @"c:\temp\thewitcher12.wmv",
    3,          // Three segments

    startNow    // Start download now

    );  

static void Instance_DownloadEnded(object sender, DownloaderEventArgs e)
{
    if (Settings.Default.ShowBallon && 

AppManager.Instance.Application.NotifyIcon.Visible)
    {
        // Display the XP Balloon

        AppManager.Instance.Application.NotifyIcon.ShowBalloonTip(
            Settings.Default.BallonTimeout,
            AppManager.Instance.Application.MainForm.Text,
            String.Format("Download finished: {0}", e.Downloader.LocalFile),
            ToolTipIcon.Info);
     }
}

Plug-in Architecture

Many features from MyDownloader are implemented using the concept of extensibility. Because the most important classes in MyDownloader offer a lot of events, extensions can listen to those events to change the application behavior. Another nice thing is that each extension has its own settings. Therefore the Options dialog needs to be created based on extensions. If you open Options at design time, you will only see an empty Panel.

Below, you can see how we load settings from the extension to populate the tree view:

for (int i = 0; i < App.Instance.Extensions.Count; i++)
{
    IExtension extension = App.Instance.Extensions[i];
    IUIExtension uiExtension = extension.UIExtension;

    Control[] options = uiExtension.CreateSettingsView();

    TreeNode node = new TreeNode(extension.Name);
    node.Tag = extension;

    for (int j = 0; j < options.Length; j++)
    {
        TreeNode optioNd = new TreeNode(options[j].Text);
        optioNd.Tag = options[j];
        node.Nodes.Add(optioNd);
    }

    treeOptions.Nodes.Add(node);
}

The DownloadManager that I showed in the beginning of this article also doesn’t know anything about HTTP or FTP. DownloadManager accepts protocols registered on ProtocolProviderFactory, and the HTTP and FTP protocols are registered by an extension. Check the HTTP/FTP download extension:

public class HttpFtpProtocolExtension: IExtension
{
    #region IExtension Members

    public string Name
    {
        get { return "HTTP/FTP"; }
    }

    public IUIExtension UIExtension
    {
        get { return new HttpFtpProtocolUIExtension(); }
    }

    public HttpFtpProtocolExtension()
    {
        ProtocolProviderFactory.RegisterProtocolHandler("http",
            typeof(HttpProtocolProvider));
        ProtocolProviderFactory.RegisterProtocolHandler("https",
            typeof(HttpProtocolProvider));
        ProtocolProviderFactory.RegisterProtocolHandler("ftp",
            typeof(FtpProtocolProvider));
    }

    #endregion
}

When we think of an HTTP download, what are the settings that an HTTP downloader would require? Proxy is one of the answers. Many users are behind an HTTP proxy and connecting directly to an HTTP server is not allowed in most companies.

So, to expose the settings for our HttpFtpProtocolExtension, we need to create an IUIExtension and return it through UIExtension property of IExtension. On this class we implement the method CreateSettingsView, that returns all settings that will be displayed on Options dialog.

public class HttpFtpProtocolUIExtension : IUIExtension
{
    public System.Windows.Forms.Control[] CreateSettingsView()
    {
        // create the Proxy user control an return it.

        return new Control[] { new Proxy() };
    }

    public void PersistSettings(System.Windows.Forms.Control[] settingsView)
    {
        ...
    }

    ...
}

The HttpFtpProtocolUIExtension class provides a factory method named CreateSettingsView. This creates an array of Controls that are the visualization of the extension settings. The Options dialog uses this array to populate the TreeView of options and display the setting on the right panel.

Protocol Abstraction

On previous versions of MyDownloader, the protocols support was implemented by classes that inhererited from Downloader. This was because the previous version didn’t support Mirrors, so at the time, a single download could only come from one source. But now, with Mirrors features, we can have one piece of a download coming from HTTP and another piece coming from an FTP server.

For that reason, I have refactored the code and now all supported protocols (HTTP, FTP, HTTPS) are implemented by classes that implement IProtocolProvider. The concrete instance of IProtocolProvider is created by ProtocolProviderFactory, protocols providers classes are implemented in a different class hierarchy from the Downloader class. This is done to address the restriction of using a single protocol for the download.

To make it easier to retrive the correct IProtocolProvider, the ResourceLocation class has a factory method. This method is used by the Downloader class.

Video Downloads Abstraction

Like many MyDownloader features, video downloads is just another extension. The secret is at VideoDownloadExtension and the “New Video Download” window. All URLs in MyDownloader are represented by the ResourceLocation class — this class has the method GetProtocolProvider which returns the apropriated instance of IProtocolProvider interface — the only thing that we need to do (at “New Video Download”) is to force the correct protocol provider type by setting the property ProtocolProviderType of ResourceLocation.

Setting this property, when ResourceLocation class calls GetProtocolProvider, the created protocol provider will be the type stored in ProtocolProviderType, and not the provider registed on ProtocolProviderFactory. In this way we can replace the default protocol provider, and avoid that the HTML content be saved, and force to download the video from web site.

The first step is register the Video protocol providers on VideoDownloadExtension:

public VideoDownloadExtension()
{
   handlers = new List<VideoDownloadHandler>();
   handlers.Add(new VideoDownloadHandler(YouTubeDownloader.SiteName,
           YouTubeDownloader.UrlPattern, typeof(YouTubeDownloader)));
   handlers.Add(new VideoDownloadHandler(GoogleVideoDownloader.SiteName,
           GoogleVideoDownloader.UrlPattern, typeof(GoogleVideoDownloader)));
   // ... register other sites here ...

}

After registering, we need to discover which video handler we need to use and also, set the corret protocol provider on the ProtocolProviderType property of ResourceLocation. This is done at “New Video Download” window, check bellow :

VideoDownloadExtension extension;
...
extension = (VideoDownloadExtension)App.Instance.GetExtensionByType(
    typeof(VideoDownloadExtension));
...
handler = extension.GetHandlerByURL(txtURL.Text);
...
ResourceLocation rl = ResourceLocation.FromURL(txtURL.Text);
rl.ProtocolProviderType = handler.Type.AssemblyQualifiedName;

How a Download from YouTube Video Site Works

Basically, a video download from any site has three steps:

• Download the HTML page from the video site
• Parse the HTML to discover the video URL
• Download the video URL

All common things are on BaseVideoDownloader class. This class retrieves the HTML and starts to download the video. The inherited classes (YouTubeDownloader, GoogleVideoDownloader) are responsible to parse the HTML text and return the video URL to the base class. Check the code:

Collapse

public abstract class BaseVideoDownloader : HttpProtocolProvider
{
   private ResourceLocation mappedUrl;

   protected abstract ResourceLocation ResolveVideoURL(string url,
      string pageData);

   public override Stream CreateStream(ResourceLocation rl,
      long initialPosition, long endPosition)
   {
      return base.CreateStream(mappedUrl ?? rl, initialPosition, endPosition);
   }

   public override RemoteFileInfo GetFileInfo(ResourceLocation rl,
      out Stream stream)
   {
      stream = null;

      mappedUrl = null;
      String pageData;

      using (StreamReader sr = new StreamReader(this.CreateStream(rl, 0, 0)))
      {
         pageData = sr.ReadToEnd();
      }

      mappedUrl = ResolveVideoURL(rl.URL, pageData);

      WebRequest request = this.GetRequest(mappedUrl);

      using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
      {
         RemoteFileInfo result = new RemoteFileInfo();
         result.FileSize = response.ContentLength;
         result.AcceptRanges = String.Compare(
            response.Headers["Accept-Ranges"], "bytes", true) == 0;

         if (response.ResponseUri != null)
         {
            mappedUrl.URL = response.ResponseUri.OriginalString;
         }
         else
         {
            stream = response.GetResponseStream();
         }

         return result;
      }
   }
}

public class YouTubeDownloader: BaseVideoDownloader
{
   public const string SiteName = "You Tube";

   //http://www.youtube.com/watch?v=5zOevLN3Tic

   public const string UrlPattern =
      @"(?:[Yy][Oo][Uu][Tt][Uu][Ee]\.[Cc][Oo][Mm]/watch\?v=)(\w[\w|-]*)";

   protected override ResourceLocation ResolveVideoURL(string url,
      string pageData)
   {
      return ResourceLocation.FromURL(
      String.Format("{0}/get_video?video_id={1}&t={2}",
      TextUtil.GetDomain(url), TextUtil.JustAfter(url, "v=", "&"),
      TextUtil.JustAfter(pageData, "t:'", "',sk:")));
   }
}

Download Scheduler

The Download Scheduler is activated (or deactivated) thought the “two arrows” button in MyDownloader toolbar. When this feature is enabled, MyDownloader starts to work as a batch downloader, accomplishing each download on download queue.

The maximum number of downloads is configured in the “Options” dialog. Another nice thing is that the user is able to choose at which times the “Download Scheduler” will work. This is done easily by selecting the “time grid”:

The Download Scheduler, works using events (DownloadAdded, DownloadEnded, DownloadEnded) from DownloadManager. When some of these events were raised, the extension starts the download respecting the maximum number of simultaneous downloads:

using (DownloadManager.Instance.LockDownloadList())
{
   int count = GetActiveJobsCount();
   int maxJobs = Settings.Default.MaxJobs;

   if (count < maxJobs)
   {
      for (int i = 0;
         i < DownloadManager.Instance.Downloads.Count && (count < maxJobs);
         i++)
      {
         if (DownloadManager.Instance.Downloads[i].State !=
            DownloaderState.Ended &&
            ! DownloadManager.Instance.Downloads[i].IsWorking())
         {
            DownloadManager.Instance.Downloads[i].Start();
            count ++;
         }
      }
   }
}

Future Ideas

This kind of project is “infinite,” so below I have listed some ideas for future implementations. As any open source project, it would be very nice if you wish to contribute.

• Add and remove segments while downloading
• Download only required files from a ZIP file (see this article)
• Improve download rate limit
  • Option to disable the speed limit while screen saver is running
• Integrate with FireFox and Internet Explorer
• Improve mirrors feature by choosing the fasters mirrors sites
• Support authentication on HTTP sites
• Support MMS protocol
• Create downloads category and allow downloads to be labeled
• XY graph to show the bandwidth usage
• Up Arrow / Down Arrow to allow the use to reorganize the download queue
• Auto shutdown after download end
• Hang-Up internet connection after download end
• Support metalink
• Video downloads:
  • Create a media monitor integrated with IE and FF that allows the user to download videos from any site
  • Options to convert videos from .FLV to .AVI, .MPG and .MP3
• Clipboard Monitor, so when a URL is copied into the clipboard, the application prompts the user to download the file using the SetClipboardViewer API

I hope you enjoyed the code! If you have any questions or feedback, feel free to contact me.

License

This article has no explicit license attached to it but may contain usage terms in the article text or the download files themselves. If in doubt please contact the author via the discussion board below.

A list of licenses authors may to use can be found here

About the Author

Guilherme Labigalini

Occupation: Web Developer Location: Brazil