# Webscraping with F# **URL:** \ **Category:** Uncategorized\ **Created:** [February 12, 2019, 6:43am UTC](https://forums.fsharp.org/t/webscraping-with-f/674 "2019-02-12T06:43:12Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![wad](https://avatars.discourse-cdn.com/v4/letter/w/d78d45/32.png) [@wad](https://forums.fsharp.org/u/wad)\ **Post date:** [February 12, 2019, 6:43am UTC](https://forums.fsharp.org/t/webscraping-with-f/674/1 "2019-02-12T06:43:12Z")
Hi, does anyone knows some resources where i will learn web scraping with f#. ---
**Author:** ![FoggyFinder](https://yyz2.discourse-cdn.com/flex030/user_avatar/forums.fsharp.org/foggyfinder/32/136_2.png) [@FoggyFinder](https://forums.fsharp.org/u/FoggyFinder)\ **Post date:** [February 13, 2019, 9:16am UTC](https://forums.fsharp.org/t/webscraping-with-f/674/2 "2019-02-13T09:16:51Z")
If you are searching for a silver bullet then, alas, there is no any. So, I suggest you asking more specific questions. One by one. Step by step. Here is some libraries, which you probably find useful: [F# Data: Library for Data Access](http://fsharp.github.io/FSharp.Data/) [OpenScraping HTML Structured Data Extraction C# Library](https://github.com/Microsoft/openscraping-lib-csharp) [AngleSharp](https://github.com/AngleSharp/AngleSharp) I used only F# Data & AngleSharp. I had chosen AngleSharp when F# Data wasn’t able to parse pages correctly. But for simple cases also worked fine. ---
**Author:** ![StachuDotNet](https://yyz2.discourse-cdn.com/flex030/user_avatar/forums.fsharp.org/stachudotnet/32/13_2.png) [@StachuDotNet](https://forums.fsharp.org/u/StachuDotNet)\ **Post date:** [February 13, 2019, 1:04pm UTC](https://forums.fsharp.org/t/webscraping-with-f/674/3 "2019-02-13T13:04:20Z")
You may also want to look into Canopy: [https://lefthandedgoat.github.io/canopy/](https://lefthandedgoat.github.io/canopy/) This is a browser testing framework, a wrapper around Selenium, but there’s probably a good opportunity to hack this into doing what you’d like 🙂 ---
**Author:** ![wad](https://avatars.discourse-cdn.com/v4/letter/w/d78d45/32.png) [@wad](https://forums.fsharp.org/u/wad)\ **Post date:** [February 14, 2019, 6:19am UTC](https://forums.fsharp.org/t/webscraping-with-f/674/4 "2019-02-14T06:19:18Z")
Thanks. Anyone who wants build framework for web scraping 😄 ---
**Author:** ![bruinbrown](https://yyz2.discourse-cdn.com/flex030/user_avatar/forums.fsharp.org/bruinbrown/32/343_2.png) [@bruinbrown](https://forums.fsharp.org/u/bruinbrown)\ **Post date:** [February 14, 2019, 10:07am UTC](https://forums.fsharp.org/t/webscraping-with-f/674/5 "2019-02-14T10:07:22Z")
I gave a talk at F# London a few years ago where I talked about building a simple search engine which featured a little bit of web scraping [https://skillsmatter.com/skillscasts/8901-f-sharpunctional-londoners-meetup](https://skillsmatter.com/skillscasts/8901-f-sharpunctional-londoners-meetup) The code for it is still on my GitHub but it’s a private repo because of keys etc but I’ve attached the code for scraping a webpage using HtmlAgilityPack. I also have a robots.txt parser which I should probably open source at some point. ``` module Scraper open System open HtmlAgilityPack open System.Text let scrapePage (uriSink:string -> Async) (indexDocument:Indexer.HtmlDocument -> unit) (storeLink:LinkPair -> unit) (getDocument:string -> Async>) (resolveUri:string -> string -> string) (url:string) = async { let! content = getDocument url match content with | Some content -> let document = HtmlAgilityPack.HtmlDocument() document.LoadHtml(content) do! document.DocumentNode.SelectNodes("//a[@href]") |> Seq.map (fun t -> t.GetAttributeValue("href", url)) |> Seq.map (resolveUri url) |> Seq.filter (fun t -> t <> url) |> Seq.filter (fun t -> try let uri = Uri(t) uri.Scheme.StartsWith("http") with | _ -> false) |> Seq.map (fun t -> { SourceUrl = url; DestinationUrl = t }) |> Seq.map (fun t -> async { do storeLink t //do! uriSink t.DestinationUrl }) |> Async.Parallel |> Async.Ignore let pageTitle = let title = document.DocumentNode.SelectSingleNode("//title") if title = null then "" else title.InnerText let text = document.DocumentNode.SelectSingleNode("//body").Descendants() |> Seq.filter (fun t -> t.NodeType = HtmlNodeType.Text && t.ParentNode.Name <> "script" && t.ParentNode.Name <> "style") |> Seq.map (fun t -> t.InnerText) |> Seq.filter (String.IsNullOrWhiteSpace >> not) |> String.concat Environment.NewLine let domain, path = let uri = Uri(url) uri.Host, uri.AbsolutePath let doc = { documentId = Convert.ToBase64String(Encoding.UTF8.GetBytes(url)) domain = domain path = path pageTitle = pageTitle textContent = text pageRankScore = 0.0 } indexDocument doc return () | None -> return () } ``` ---
**Author:** ![wad](https://avatars.discourse-cdn.com/v4/letter/w/d78d45/32.png) [@wad](https://forums.fsharp.org/u/wad)\ **Post date:** [February 15, 2019, 6:11am UTC](https://forums.fsharp.org/t/webscraping-with-f/674/6 "2019-02-15T06:11:52Z")
> [@bruinbrown](#): > > sync\ **Author:** ![gus](https://yyz2.discourse-cdn.com/flex030/user_avatar/forums.fsharp.org/gus/32/405_2.png) [@gus](https://forums.fsharp.org/u/gus)\ **Post date:** [February 15, 2019, 8:56pm UTC](https://forums.fsharp.org/t/webscraping-with-f/674/7 "2019-02-15T20:56:13Z") I started experimenting with monadic web scraping: [https://github.com/gusty/ScrapeM](https://github.com/gusty/ScrapeM) But I didn’t go further, because there were no anonymous types available. Now I should continue it with F# 4.6 ---
**Author:** ![gsomix](https://yyz2.discourse-cdn.com/flex030/user_avatar/forums.fsharp.org/gsomix/32/484_2.png) [@gsomix](https://forums.fsharp.org/u/gsomix)\ **Post date:** [February 21, 2019, 3:45pm UTC](https://forums.fsharp.org/t/webscraping-with-f/674/8 "2019-02-21T15:45:29Z")
- [Why F# is the best language for web scraping](https://biarity.gitlab.io/2016/11/23/why-f-is-the-best-langauge-for-web-scraping/) - [Html Provider + Canopy](http://www.colinbull.net/2016/11/29/HtmlProvider-Canopy/)