MyDownloader: A Multi-thread C# Segmented Download Manager

• Download demo project - 2.74 MB
• Download source - 470.28 KB

Introduction

MyDownloader is an open source application written in C# that is almost a complete download manager. MyDownloader has many features to manage downloads:

• Segmented downloads from HTTP and FTP
• With smart segments: when one segment ends, starts another segment to help to terminate another segment more fast
• Automatic retry when a segment or download fails
• Allow downloads to be paused and resumed
• Video Downloads
• Support to download videos from:
• YouTube
• Google Video
• Break
• PutFile
• Meta Cafe
• (NEW) Support to convert downloaded videos to MPEG, AVI and MP3 (using ffmpeg)
• (NEW) Video file name suggestion based on video title
• Speed Limit — to avoid to use all your bandwidth
• Support for Auto-Downloads
• (NEW) Limit the bandwidth at specific times
• (NEW) Possibility to enable "Auto-downloads" at startup, allowing the downloads to start automaticaly at application startup
• Download files only on allowed times
• Limit the number of simultaneous downloads
• When one download ends, starts another automatically
• Support for FTP site that requires authentication
• Support for Mirrors
• Download from HTTPS
• (NEW) Download from authenticated HTTP urls
• Notification download completion with sounds and XP balloon
• Anti-virus integration
• Batch downloads (enter a generic URL such as http://server/file(*).zip and MyDownloader generates a set of URLs with numbers or letters)
• (NEW) Move up / Move down button to change the order of download on download queue
• (NEW) Bug fixes and improvements
• (NEW) Web Spider (Web Crawler)
• (NEW) Download all files from an specific page
• (NEW) Download all images from an specific page
• (NEW) Allow to filter URLs by extension or by name
• (NEW) Support to convert downloaded videos to MPEG, AVI and MP3 (using ffmpeg)
• (NEW) Video file name suggestion based on video title
• (NEW) Clipboard Monitor
• (NEW) Internet Explorer Integration
• (NEW) Download links when they are clicked and the user is holding the ALT key
• (NEW) When navigating tough an video site (YouTube, Google Video, etc), enable the video button to download the video with MyDownloader
• (NEW) Button to launch MyDownloader
• (NEW) Import URLs from file
• (NEW) From a local text file
• (NEW) From a local html file

How a Segmented Download Works

Downloads can be segmented because both HTTP and FTP protocols allow the client to specify the start position of the stream. First, MyDownloader performs a request to the server to discover the file size. After that, MyDownloader calculates the segment size as follows:

segment size = min( (file size / number of segments), 
    minimum allowed segment size )

With the segment size, MyDownloader creates another request specifying the start position of the stream. In this way, we can have multi-requests for the same files running in parallel using multi-threading techniques. This technique speeds up the transfer rate even more if you are using mirrors.

Using the Code: MyDownloader API

To start a segmented download using the MyDownloader API is very simple. Check the code below, extracted from the MyDownloader source code. When the download is finished, an XP balloon is displayed near the windows clock:

// starts to listen to the event 'DownloadEnded' from DownloadManager
DownloadManager.Instance.DownloadEnded += 

new EventHandler<DownloaderEventArgs>(Instance_DownloadEnded);

// indicates that download should start immediately
bool startNow = true;

Downloader download = DownloadManager.Instance.Add(
    "http://jogos.download.uol.com.br/videos/pc/thewitcher12.wmv",
    @"c:\temp\thewitcher12.wmv",
    3,          // Three segments 

    startNow    // Start download now
    );  

void Instance_DownloadEnded(object sender, DownloaderEventArgs e)
{
    if (Settings.Default.ShowBallon && 
    AppManager.Instance.Application.NotifyIcon.Visible)
    {
        // Display the XP Balloon 

  }
finally
{
    DownloadManager.Instance.OnEndAddBatchDownloads();
}
      AppManager.Instance.Application.NotifyIcon.ShowBalloonTip(
            Settings.Default.BallonTimeout,
            AppManager.Instance.Application.MainForm.Text,
            String.Format("Download finished: {0}", e.Downloader.LocalFile),
            ToolTipIcon.Info);
     }
}

Protocol Abstraction

On previous versions of MyDownloader, the protocols support was implemented by classes that inhererited from Downloader. This was because the previous version didn't support Mirrors, so at the time, a single download could only come from one source. But now, with Mirrors features, we can have one piece of a download coming from HTTP and another piece coming from an FTP server.

For that reason, I have refactored the code and now all supported protocols (HTTP, FTP, HTTPS) are implemented by classes that implement IProtocolProvider. The concrete instance of IProtocolProvider is created by ProtocolProviderFactory, protocols providers classes are implemented in a different class hierarchy from the Downloader class. This is done to address the restriction of using a single protocol for the download.

To make it easier to retrieve the correct IProtocolProvider, the ResourceLocation class has a factory method. This method is used by the Downloader class.

Plug-in Architecture

Many features from MyDownloader are implemented using the concept of extensibility. Because the most important classes in MyDownloader offer a lot of events, extensions can listen to those events to change the application behavior. Another nice thing is that each extension has its own settings. Therefore the Options dialog needs to be created based on extensions. If you open Options at design time, you will only see an empty Panel.

Below, you can see how we load settings from the extension to populate the tree view:

for (int i = 0; i < App.Instance.Extensions.Count; i++)
{
    IExtension extension = App.Instance.Extensions[i];
    IUIExtension uiExtension = extension.UIExtension;
    
    Control[] options = uiExtension.CreateSettingsView();
    
    TreeNode node = new TreeNode(extension.Name);
    node.Tag = extension;

    for (int j = 0; j < options.Length; j++)
    {
        TreeNode optioNd = new TreeNode(options[j].Text);
        optioNd.Tag = options[j];
        node.Nodes.Add(optioNd);
    }

    treeOptions.Nodes.Add(node);
}

The DownloadManager that I showed in the beginning of this article also doesn't know anything about HTTP or FTP. DownloadManager accepts protocols registered on ProtocolProviderFactory, and the HTTP and FTP protocols are registered by an extension. Check the HTTP/FTP download extension:

public class HttpFtpProtocolExtension: IExtension
{
    #region IExtension Members

    public string Name
    {
        get { return "HTTP/FTP"; }
    }

    public IUIExtension UIExtension
    {
        get { return new HttpFtpProtocolUIExtension(); }
    }

    public HttpFtpProtocolExtension()
    {
        ProtocolProviderFactory.RegisterProtocolHandler("http", 
            typeof(HttpProtocolProvider));
        ProtocolProviderFactory.RegisterProtocolHandler("https", 
            typeof(HttpProtocolProvider));
        ProtocolProviderFactory.RegisterProtocolHandler("ftp", 
            typeof(FtpProtocolProvider));
    }

    #endregion

}

When we think of an HTTP download, what are the settings that an HTTP downloader would require? Proxy is one of the answers. Many users are behind an HTTP proxy and connecting directly to an HTTP server is not allowed in most companies.

So, to expose the settings for our HttpFtpProtocolExtension, we need to create an IUIExtension and return it through UIExtension property of IExtension. On this class we implement the method CreateSettingsView, that returns all settings that will be displayed on Options dialog.

public class HttpFtpProtocolUIExtension : IUIExtension
{
    public System.Windows.Forms.Control[] CreateSettingsView()
    {
        // create the Proxy user control an return it.

        return new Control[] { new Proxy() };
    }

    public void PersistSettings(System.Windows.Forms.Control[] settingsView)
    {
        ... 
    }
    
    ...
}

The HttpFtpProtocolUIExtension class provides a factory method named CreateSettingsView. This creates an array of Controls that are the visualization of the extension settings. The Options dialog uses this array to populate the TreeView of options and display the setting on the right panel.

Web Spider

Web Spider works over MyDownloader API, the only secret on the spider is to parse the HTML pages using regular expressions. Below we can see a screenshot of Web Spider:

When an download of an file is complete (download state is changed to DownloaderState.Ended), the spider checks if it's an HTML document (comparing the mime type) and then lookup for all references such hyperlinks, images, frames and iframes. The following code is executed to add all page references to the download list:

...
if (download.RemoteFileInfo.MimeType.IndexOf("text/html",
    StringComparison.OrdinalIgnoreCase) < 0)
{
    return;
}
...
try
{
    DownloadManager.Instance.OnBeginAddBatchDownloads();

    using (Stream htmlStream = File.OpenRead(localFile))
    {
        using (HtmlParser parser = new HtmlParser(htmlStream))
        {
            AddUrls(parser.GetHrefs(context.BaseLocation), UrlType.Href);
            AddUrls(parser.GetImages(context.BaseLocation), UrlType.Img);
            AddUrls(parser.GetFrames(context.BaseLocation), UrlType.Frame);
            AddUrls(parser.GetIFrames(context.BaseLocation), UrlType.IFrame);
        }
    }
}
finally
{
    DownloadManager.Instance.OnEndAddBatchDownloads();
}

Video Downloads from YouTube, Google Video (and etc) with Conversion

Like many MyDownloader features, video downloads is just another extension. The secret is at VideoDownloadExtension and the "New Video Download" window. All URLs in MyDownloader are represented by the ResourceLocation class — this class has the method GetProtocolProvider which returns the apropriated instance of IProtocolProvider interface — the only thing that we need to do (at "New Video Download") is to force the correct protocol provider type by setting the property ProtocolProviderType of ResourceLocation.

Setting this property, when ResourceLocation class calls GetProtocolProvider, the created protocol provider will be the type stored in ProtocolProviderType, and not the provider registed on ProtocolProviderFactory. In this way we can replace the default protocol provider, and avoid that the HTML content be saved, and force to download the video from web site.

The first step is register the Video protocol providers on VideoDownloadExtension:

public VideoDownloadExtension()
{
   handlers = new List<VideoDownloadHandler>();
   handlers.Add(new VideoDownloadHandler(YouTubeDownloader.SiteName, 
           YouTubeDownloader.UrlPattern, typeof(YouTubeDownloader)));
   handlers.Add(new VideoDownloadHandler(GoogleVideoDownloader.SiteName, 
           GoogleVideoDownloader.UrlPattern, typeof(GoogleVideoDownloader)));
   // ... register other sites here ...


}

After registering, we need to discover which video handler we need to use and also, set the correct protocol provider on the ProtocolProviderType property of ResourceLocation. This is done at "New Video Download" window, check Below :

VideoDownloadExtension extension;
...
extension = (VideoDownloadExtension)App.Instance.GetExtensionByType(
    typeof(VideoDownloadExtension));
...
handler = extension.GetHandlerByURL(txtURL.Text);
...
ResourceLocation rl = ResourceLocation.FromURL(txtURL.Text);
rl.ProtocolProviderType = handler.Type.AssemblyQualifiedName;

Basically, all video site handlers only need to parse the HTML page and return the URL of the FLV. This process have three main steps:

• Download the HTML page from the video site
• Parse the HTML to discover the video URL
• Return the video URL

All common things are on BaseVideoDownloader class. This class retrieves the HTML and starts to download the video. The inherited classes (YouTubeDownloader, GoogleVideoDownloader) are responsible to parse the HTML text and return the video URL to the base class. Below we can see how to get the URL from a FLV file on a YouTube page:

public class YouTubeDownloader: BaseVideoDownloader
{
   public const string SiteName = "You Tube";

   //http://www.youtube.com/watch?v=5zOevLN3Tic


   public const string UrlPattern = 
      @"(?:[Yy][Oo][Uu][Tt][Uu][Ee]\.[Cc][Oo][Mm]/watch\?v=)(\w[\w|-]*)";

   protected override ResourceLocation ResolveVideoURL(string url, string pageData, 
         out string videoTitle)
   {
      videoTitle = TextUtil.JustAfter(pageData,
          "< meta name=\"title\" content=\"", "\">"); 
      
      return ResourceLocation.FromURL(String.Format("{0}/get_video?video_id={1}&t={2}", 
       TextUtil.GetDomain(url), TextUtil.JustAfter(url, "v=", "&"), 
       TextUtil.JustAfter(pageData, "&t=", "&hl=")));
   }
}

After downloadeding, the video can be converted to MPEG, AVI or MP3 (audio only), this process in done using an external open source tool: ffmpeg. This tool, which is a command line tool, is called by MyDownloader with the FLV filename and conversion arguments. If you want to see details about the arguments that were send to ffmpeg, I suggest you to download the code / demo project of this article.

Selecting Files inside a Remote ZIP File

This is another very cool feature of MyDownloder. Sometimes, you need to download an big ZIP file just because you want a single file inside the ZIP, on New Download window, if user checks the option "Choose files inside ZIP", MyDownloader will enumerate the files inside ZIP and allow user to select only that files that we want to download.

The feature is based on the article Extracting files from a remote ZIP archive and the updated version by Unruled Boy (see comments on the end of the article). Below we can the how New Download window displays the ZIP file and allow user to choose the files inside ZIP:

Auto-Downloads

The Auto-Downloads is activated (or deactivated) through the "two arrows" button in MyDownloader toolbar. When this feature is enabled, MyDownloader starts to work as a batch downloader, accomplishing each download on download queue.

The maximum number of downloads is configured in the "Options" dialog. Another nice thing is that the user is able to choose at which times the "Auto-Downloads" will work and is also possible to limit the bandwidth usage at specific times. This is done easily by selecting the "time grid":

The Auto-Downloads, works using events (DownloadAdded, DownloadEnded) from DownloadManager. When some of these events were raised, the extension starts the download respecting the maximum number of simultaneous downloads:

using (DownloadManager.Instance.LockDownloadList(false))
{
   int count = GetActiveJobsCount();
   int maxJobs = Settings.Default.MaxJobs;

   if (count < maxJobs)
   {
      for (int i = 0; 
         i < DownloadManager.Instance.Downloads.Count && (count < maxJobs); 
         i++)
      {
         if (DownloadManager.Instance.Downloads[i].State != 
            DownloaderState.Ended &&

            ! DownloadManager.Instance.Downloads[i].IsWorking())
         {
            DownloadManager.Instance.Downloads[i].Start();
            count ++;
         }
      }
   }
}

Internet Explorer Integration

Browser integration is a critical feature for any download manager. This new version of MyDownloader introduces a very simple Internet Explorer (IE) integration. The IE integration is a IE toolbar, which is build on top of BandObjectLib, that has three main features:

• Shortcut button that be enabled when user is navigation on a video site that allow user to download the video
• Replace IE download window when user is holding Alt key
• Shortcut to lauch MyDownloader

Below we can see the IE displaying an empty page and perceive that the download button is disabled, the second image shows IE displaying a YouTube video and the download button became enabled:

To enable the video download button, we need to listen to the AfterNavigate event from IE and then check if property LocationURL is an URL from a video site:

void AfterNavigate(object iDisp, ref object URL)
{
   SHDocVw.WebBrowser IEDocument = GetIEDocument();            

   btnDownload.Enabled = videoSites.IsVideoSite(IEDocument.LocationURL);
}

To replace the IE download window (only when Alt is pressed), the FileDownload event is used:

void FileDownload(bool ActiveDocument, ref bool Cancel)
{
   if (!ActiveDocument)
   {
      if ((Control.ModifierKeys & Keys.Alt) == Keys.Alt)
      {
         Cancel = true;

         if ((DateTime.Now - lastDownload).TotalSeconds >= 1.9)
         {
            ThreadPool.QueueUserWorkItem(
               delegate(object state)
               {
                  DownloadURL(lastUrl);
               });

            lastDownload = DateTime.Now;
         }
      }
   }
}

Import URLs from Files

Other new feature of MyDownloader is "Import URLs from files" window, which allows the user to import the URLs from a text file or from an HTML file. The text files must to have one URL each line. For HTMLs, the URLs will be extracted using the same HTML parser used on Web Spider.

All URLs that were found in the file will be added to the download list. "Import URLs from files" window also have a shortcut to enable the "Auto-downloads", and to setup the maximum number of simultaneous downloads.

Future Ideas

This kind of project is "infinite," so below I have listed some ideas for future implementations. As any open source project, it would be very nice if you wish to contribute.

• Add and remove segments while downloading
• Option to disable the speed limit while screen saver is running
• Integrate with FireFox and improve Internet Explorer integration
• Improve mirrors feature by choosing the fasters mirrors sites
• Support MMS protocol
• Create downloads category and allow downloads to be labeled
• XY graph to show the bandwidth usage
• Auto shutdown after download end
• Hang-Up internet connection after download end
• Support metalink
• Video downloads:
• Create a media monitor integrated with IE and FF that allows the user to download videos from any site

I hope you enjoyed the code! If you have any questions or feedback, feel free to contact me.