The C# Webscraping Library

IronWebScraper provides a powerful framework to extract data and files from websites using C# code.

  1. Install IronWebScraper to your Project using NuGet
  2. Create a Class Extending WebScraper
  3. Create an Init Method that uses the Request Method to parse at least one URL.
  4. Create a Parse method to process the requests, and indeed Request More pages. Use response.Css to work with HTML elements using jQuery-style CSS selectors.
  5. In your application, please create an instance of your web scraping class and call the Start(); method
REQUEST QUOTE
CONTACT US

C# Web Scraper

  
using IronWebScraper;
public class Program
{
    private static void Main(string[] args)
    {
        var ScrapeJob = new BlogScraper();
        ScrapeJob.Start();
    }
}
public class BlogScraper : WebScraper
{
    public override void Init()
    {
        LoggingLevel = LogLevel.All;
        Request("https://www.zyte.com/blog/", Parse);
    }
    public override void Parse(Response response)
    {
        foreach (HtmlNode title_link in response.Css(".oxy-post-title"))
        {
            string strTitle = title_link.TextContentClean;
            Scrape(new ScrapedData() { { "Title", strTitle } });
        }
        if (response.CssExists("div.oxy-easy-posts-pages > a[href]"))
        {
            string next_page = response.Css("div.oxy-easy-posts-pages > a[href]")[0].Attributes["href"];
            Request(next_page, Parse);
        }
    }
}
Explore All 1 Code Samples

Cross Platform Support

Designed for C#, F#, & VB.NET running on .NET 8, 7, 6, Core, Standard, or Framework

powerful-scraping-engine

Powerful Scraping Engine Under Your Control

Just write a single C# web-scraper class to scrape thousands or millions of web pages into C# Class Instances, JSON, or Downloaded Files. IronWebScraper allows you to code concise, linear workflows simulating human browsing behavior. IronWebScraper will run your code as a swarm of virtual web browsers, massivelyparallel, polit,e and fault tolerant.

simple-flexible-logic

Simple, Flexible Logic

IronWebScraper must be programmed to know how to handle each “type” of page it encounters. This is achieved in a very concise manner using CSS Selectors or XPath expressions and can be fully customized in C#. This freedom allows you to decide which pages to scrape within a website, and what to do with the data extracted. Each method can be debugged and watched neatly in Visual Studio.

fast-and-polite-behavior

Fast and Polite Behavior

IronWebScraper deals with multithreading and web-requests to allow for hundreds of concurrent threads without the developer needing to manage them. Politeness can be set to throttle requests, so reducing risk of excessive load on target web servers.

create-virtual-user-identities

Create virtual user Identities

IronWebScraper can use one or multiple “identities” – sessions that simulate real world human requests. Each request may programmatically or randomly assign its own Identity, User Agent, Cookies, Logins and even IP addresses. Requests are set as auto-unique with a combination of URL, parse method and post variables.

action-replay

Powerful Scraping Engine Under Your Control

Just write a single C# web-scraper class to scrape thousands or millions of web pages into C# Class Instances, JSON, or Downloaded Files. IronWebScraper allows you to code concise, linear workflows simulating human browsing behavior. IronWebScraper will run your code as a swarm of virtual web browsers, massivelyparallel, polit,e and fault tolerant.

rapid-img

Rapid Installation with Microsoft Visual Studio

IronWebScraper puts Web Scraping tools in your own hands quickly with a Visual Studio installer. Whether installing directly from NuGet within visual studio or downloading the DLL, you’ll be setup in no time. Just one DLL and no dependencies.