SCRAPERWIKIPRODUCTSINDUSTRIESCONSULTINGBLOGABOUTDATABAKER QuickCode is the new name for the original ScraperWiki product. We renamed it, as it isn’t a wiki or just for scraping any more. It’s a Python and R data analysis environment, ideal for economists, statisticians and data managers who are new to coding. Go to QuickCode website. The Sensible Code Company is the new name for our company.

ACADEMIC RESEARCH

Scraperwiki is a partner on two major EU research projects, NewsReader and TIMON.We specialise in the exploitation and dissemination of results (particularly through the use of Hack Days), bringing professional software engineering standards to projects, and data ingestion and cleaning. SCRAPING GUIDES: EXCEL SPREADSHEETS Scraping guides: Excel spreadsheets. by Francis Irving; on September 14, 2011; under Developer • Comments Off on Scraping guides: Excel spreadsheets Following on from the CSV scraping guide, we’ve now added one about scraping Excel spreadsheets. You can get to them from the documentation page.. The Excel scraping guide is available in Ruby, Python and PHP. . Just as with all HOW TO SCRAPE AND PARSE WIKIPEDIA All ready for parsing. I’ve written a nice complicated recursive template parser that I use in wikipedia_utils, which makes it easy to extract all the templates from the page in the following way: import scraperwiki wikipedia_utils = scraperwiki.swimport ("wikipedia_utils") title = "Aquamole Pot" val = wikipedia_utils.GetWikipediaPage (title HOW TO GET ALONG WITH AN ASP WEBPAGE How to get along with an ASP webpage. by Julian Todd; on November 9, 2011; under Developer • 7 Comments Fingal County Council of Ireland recently published a number of sets of Open Data, in nice clean CSV, XML and KML formats.. Unfortunately, the one set of Open Data that was difficult to obtain, was the list of sets of open data. YAHOO!FINANCE TO TABLEAU VIA SCRAPERWIKI Yahoo!Finance to Tableau via ScraperWiki. by Ian Hopkinson; on April 17, 2014; under Products • Comments Off on Yahoo!Finance to Tableau via ScraperWiki Our recently announced OData connector gives Tableau users access to a world of unstructured and semi-structured data.. In this post I’d like to demonstrate the power of a Python library, Pandas, and the Code in a Browser tool to get PDF TABLE EXTRACTION OF PAGENATED TABLE The Isle of Man aircraft registry (in PDF form) has long been a target of mine waiting for the appropriate PDF parsing technology. The scraper is here.. Setting aside the GetPDF() function, which deals with copying out each new pdf file as it is updated and backing it up into the database as a base64 encoded binary blob for quicker access, let’s have a look at the what the PDF itself looks like. THE HISTORY OF PIVOT TABLE The history of Pivot table. by Sophie Buckley; on July 16, 2014; under Data Science • 1 Comment A pivot table is a spreadsheet feature that allows data tables to be rearranged in many ways for different views of the same data (pivot from one view to another).. Pivot Tables have become ubiquitous amongst power users of Excel, even being listed as a skill in CVs and a “desirable” in job ‘DOCUMENTATION IS LIKE SEX: WHEN IT IS GOOD, IT IS VERY ‘Documentation is like sex: when it is good, it is very, very good; and when it is bad, it is better than nothing’ by Francis Irving; on May 25, 2011; under Developer • 4 Comments You may have noticed that the design of the ScraperWiki site has changed substantially. AND SUDDENLY I COULD CONVERT MY BANK STATEMENT FROM PDF TO .and suddenly I could convert my bank statement from PDF to Excel by Aine McGuire; on August 5, 2015; under Front page, Products • Comments Off on .and suddenly I could convert my bank statement from PDF to Excel Do you ever: Need an old bank statement only to find out that the bank has archived it, and want to charge you to get it back? SCRAPERWIKIPRODUCTSINDUSTRIESCONSULTINGBLOGABOUTDATABAKER QuickCode is the new name for the original ScraperWiki product. We renamed it, as it isn’t a wiki or just for scraping any more. It’s a Python and R data analysis environment, ideal for economists, statisticians and data managers who are new to coding. Go to QuickCode website. The Sensible Code Company is the new name for our company.

ACADEMIC RESEARCH

Scraperwiki is a partner on two major EU research projects, NewsReader and TIMON.We specialise in the exploitation and dissemination of results (particularly through the use of Hack Days), bringing professional software engineering standards to projects, and data ingestion and cleaning. SCRAPING GUIDES: EXCEL SPREADSHEETS Scraping guides: Excel spreadsheets. by Francis Irving; on September 14, 2011; under Developer • Comments Off on Scraping guides: Excel spreadsheets Following on from the CSV scraping guide, we’ve now added one about scraping Excel spreadsheets. You can get to them from the documentation page.. The Excel scraping guide is available in Ruby, Python and PHP. . Just as with all HOW TO SCRAPE AND PARSE WIKIPEDIA All ready for parsing. I’ve written a nice complicated recursive template parser that I use in wikipedia_utils, which makes it easy to extract all the templates from the page in the following way: import scraperwiki wikipedia_utils = scraperwiki.swimport ("wikipedia_utils") title = "Aquamole Pot" val = wikipedia_utils.GetWikipediaPage (title HOW TO GET ALONG WITH AN ASP WEBPAGE How to get along with an ASP webpage. by Julian Todd; on November 9, 2011; under Developer • 7 Comments Fingal County Council of Ireland recently published a number of sets of Open Data, in nice clean CSV, XML and KML formats.. Unfortunately, the one set of Open Data that was difficult to obtain, was the list of sets of open data. YAHOO!FINANCE TO TABLEAU VIA SCRAPERWIKI Yahoo!Finance to Tableau via ScraperWiki. by Ian Hopkinson; on April 17, 2014; under Products • Comments Off on Yahoo!Finance to Tableau via ScraperWiki Our recently announced OData connector gives Tableau users access to a world of unstructured and semi-structured data.. In this post I’d like to demonstrate the power of a Python library, Pandas, and the Code in a Browser tool to get PDF TABLE EXTRACTION OF PAGENATED TABLE The Isle of Man aircraft registry (in PDF form) has long been a target of mine waiting for the appropriate PDF parsing technology. The scraper is here.. Setting aside the GetPDF() function, which deals with copying out each new pdf file as it is updated and backing it up into the database as a base64 encoded binary blob for quicker access, let’s have a look at the what the PDF itself looks like. THE HISTORY OF PIVOT TABLE The history of Pivot table. by Sophie Buckley; on July 16, 2014; under Data Science • 1 Comment A pivot table is a spreadsheet feature that allows data tables to be rearranged in many ways for different views of the same data (pivot from one view to another).. Pivot Tables have become ubiquitous amongst power users of Excel, even being listed as a skill in CVs and a “desirable” in job ‘DOCUMENTATION IS LIKE SEX: WHEN IT IS GOOD, IT IS VERY ‘Documentation is like sex: when it is good, it is very, very good; and when it is bad, it is better than nothing’ by Francis Irving; on May 25, 2011; under Developer • 4 Comments You may have noticed that the design of the ScraperWiki site has changed substantially. AND SUDDENLY I COULD CONVERT MY BANK STATEMENT FROM PDF TO .and suddenly I could convert my bank statement from PDF to Excel by Aine McGuire; on August 5, 2015; under Front page, Products • Comments Off on .and suddenly I could convert my bank statement from PDF to Excel Do you ever: Need an old bank statement only to find out that the bank has archived it, and want to charge you to get it back? ABOUT US | SCRAPERWIKI ScraperWiki was founded in 2009 by Julian Todd and Aidan McGuire. We received initial funding from British TV station Channel 4. After fostering an active community of open data coders and data journalists, ScraperWiki won the Knight News Challenge in 2011. In 2012 ScraperWiki closed a million dollar round of investment led by

EV, to improve

CONSULTING | SCRAPERWIKI We build data rich web applications. We’re experienced in building websites which front large collections of data. This gives the user the power to explore and understand the data, to reuse and repurpose it and to visualise it in imaginative ways. DOCUMENTATION / SCRAPERWIKI LIBRARY In addition to all the standard Python libraries for downloading and parsing pages from the web, ScraperWiki provides the scraperwiki Python library.. Access like this: import scraperwiki. The source code that implements these functions can be found in our bitbucket

repository.

BLOG | SCRAPERWIKI

The Sensible Code Company is our new name. by Francis Irving; on August 9, 2016; under Uncategorized • Leave a comment For a few years now, people have said “but you don’t just do scraping, and you’re not a wiki, why are you called that?” HOW TO GET ALONG WITH AN ASP WEBPAGE How to get along with an ASP webpage. by Julian Todd; on November 9, 2011; under Developer • 7 Comments Fingal County Council of Ireland recently published a number of sets of Open Data, in nice clean CSV, XML and KML formats.. Unfortunately, the one set of Open Data that was difficult to obtain, was the list of sets of open data. PDF TABLE EXTRACTION OF PAGENATED TABLE The Isle of Man aircraft registry (in PDF form) has long been a target of mine waiting for the appropriate PDF parsing technology. The scraper is here.. Setting aside the GetPDF() function, which deals with copying out each new pdf file as it is updated and backing it up into the database as a base64 encoded binary blob for quicker access, let’s have a look at the what the PDF itself looks like. BRANDED AND GENERIC MEDICATION COMPARED Branded and Generic medication compared. by Leonisha Barley; on August 26, 2015; under Case Studies • 5 Comments According to the Office of Health Economics for the Association of the British Pharmaceutical Industry (ABPI), the total medicines bill in the UK was £13.6 billion in 2011 and £10.8 billion of this was spent on branded medication.Prescribers such as GPs are encouraged to DATABAKER – MAKING SPREADSHEETS MACHINE-READABLE DataBaker – making spreadsheets machine-readable. by David McKee; on March 26, 2015; under Case Studies, Data Science, Front page • Comments Off on DataBaker – making spreadsheets machine-readable Spreadsheets are often the way of choice for publishing data. They look great, are understandable by people who don’t use databases, and with judicious use of formatting you can

IS SCRAPING LEGAL?

Very interesting post. In my opinion, if data is publicly viewable / indexed by search engines, expect it to be scraped. There are ways to prevent scraping from happening, and if one really wants scraping of data to be stopped, they should implement various methods to within their website/service. HOW TO TEST SHELL SCRIPTS Urchin doesn’t help you at all with outputs, but it makes testing side-effects easier. In urchin, you can nest tests inside of directories; to test a side-effect, you make a subdirectory, put the command of interest in the setup_dir file and then test your side effects in your test files. Urchin is SCRAPERWIKIPRODUCTSINDUSTRIESCONSULTINGBLOGABOUTDATABAKER QuickCode is the new name for the original ScraperWiki product. We renamed it, as it isn’t a wiki or just for scraping any more. It’s a Python and R data analysis environment, ideal for economists, statisticians and data managers who are new to coding.

ACADEMIC RESEARCH

Scraperwiki is a partner on two major EU research projects, NewsReader and TIMON.We specialise in the exploitation and dissemination of results (particularly through the use of Hack Days), bringing professional software engineering standards to projects, and data ingestion and cleaning. DOCUMENTATION / SCRAPERWIKI LIBRARY In addition to all the standard Python libraries for downloading and parsing pages from the web, ScraperWiki provides the scraperwiki Python library.. Access like this: import scraperwiki. The source code that implements these functions can be found in our bitbucket

repository.

BLOG | SCRAPERWIKI

The Sensible Code Company is our new name. by Francis Irving; on August 9, 2016; under Uncategorized • Leave a comment For a few years now, people have said “but you don’t just do scraping, and you’re not a wiki, why are you called that?” SCRAPING GUIDES: EXCEL SPREADSHEETS Scraping guides: Excel spreadsheets. by Francis Irving; on September 14, 2011; under Developer • Comments Off on Scraping guides: Excel spreadsheets Following on from the CSV scraping guide, we’ve now added one about scraping Excel spreadsheets. You can get to them from the documentation page.. The Excel scraping guide is available in Ruby, Python and PHP. . Just as with all YAHOO!FINANCE TO TABLEAU VIA SCRAPERWIKI Yahoo!Finance to Tableau via ScraperWiki. by Ian Hopkinson; on April 17, 2014; under Products • Comments Off on Yahoo!Finance to Tableau via ScraperWiki Our recently announced OData connector gives Tableau users access to a world of unstructured and semi-structured data.. In this post I’d like to demonstrate the power of a Python library, Pandas, and the Code in a Browser tool to get BRANDED AND GENERIC MEDICATION COMPARED Branded and Generic medication compared. by Leonisha Barley; on August 26, 2015; under Case Studies • 5 Comments According to the Office of Health Economics for the Association of the British Pharmaceutical Industry (ABPI), the total medicines bill in the UK was £13.6 billion in 2011 and £10.8 billion of this was spent on branded medication.Prescribers such as GPs are encouraged to SCRAPE ANYONE’S TWITTER FOLLOWERS You may think this is clever, but it is an invasion of people’s privacy and goes against every principle of every privacy legislation. This is the kind of behavior that hurts an entire industry. ‘DOCUMENTATION IS LIKE SEX: WHEN IT IS GOOD, IT IS VERY ‘Documentation is like sex: when it is good, it is very, very good; and when it is bad, it is better than nothing’ by Francis Irving; on May 25, 2011; under Developer • 4 Comments You may have noticed that the design of the ScraperWiki site has changed substantially. AND SUDDENLY I COULD CONVERT MY BANK STATEMENT FROM PDF TO .and suddenly I could convert my bank statement from PDF to Excel by Aine McGuire; on August 5, 2015; under Front page, Products • Comments Off on .and suddenly I could convert my bank statement from PDF to Excel Do you ever: Need an old bank statement only to find out that the bank has archived it, and want to charge you to get it back? SCRAPERWIKIPRODUCTSINDUSTRIESCONSULTINGBLOGABOUTDATABAKER QuickCode is the new name for the original ScraperWiki product. We renamed it, as it isn’t a wiki or just for scraping any more. It’s a Python and R data analysis environment, ideal for economists, statisticians and data managers who are new to coding.

ACADEMIC RESEARCH

Scraperwiki is a partner on two major EU research projects, NewsReader and TIMON.We specialise in the exploitation and dissemination of results (particularly through the use of Hack Days), bringing professional software engineering standards to projects, and data ingestion and cleaning. DOCUMENTATION / SCRAPERWIKI LIBRARY In addition to all the standard Python libraries for downloading and parsing pages from the web, ScraperWiki provides the scraperwiki Python library.. Access like this: import scraperwiki. The source code that implements these functions can be found in our bitbucket

repository.

BLOG | SCRAPERWIKI

The Sensible Code Company is our new name. by Francis Irving; on August 9, 2016; under Uncategorized • Leave a comment For a few years now, people have said “but you don’t just do scraping, and you’re not a wiki, why are you called that?” SCRAPING GUIDES: EXCEL SPREADSHEETS Scraping guides: Excel spreadsheets. by Francis Irving; on September 14, 2011; under Developer • Comments Off on Scraping guides: Excel spreadsheets Following on from the CSV scraping guide, we’ve now added one about scraping Excel spreadsheets. You can get to them from the documentation page.. The Excel scraping guide is available in Ruby, Python and PHP. . Just as with all YAHOO!FINANCE TO TABLEAU VIA SCRAPERWIKI Yahoo!Finance to Tableau via ScraperWiki. by Ian Hopkinson; on April 17, 2014; under Products • Comments Off on Yahoo!Finance to Tableau via ScraperWiki Our recently announced OData connector gives Tableau users access to a world of unstructured and semi-structured data.. In this post I’d like to demonstrate the power of a Python library, Pandas, and the Code in a Browser tool to get BRANDED AND GENERIC MEDICATION COMPARED Branded and Generic medication compared. by Leonisha Barley; on August 26, 2015; under Case Studies • 5 Comments According to the Office of Health Economics for the Association of the British Pharmaceutical Industry (ABPI), the total medicines bill in the UK was £13.6 billion in 2011 and £10.8 billion of this was spent on branded medication.Prescribers such as GPs are encouraged to SCRAPE ANYONE’S TWITTER FOLLOWERS You may think this is clever, but it is an invasion of people’s privacy and goes against every principle of every privacy legislation. This is the kind of behavior that hurts an entire industry. ‘DOCUMENTATION IS LIKE SEX: WHEN IT IS GOOD, IT IS VERY ‘Documentation is like sex: when it is good, it is very, very good; and when it is bad, it is better than nothing’ by Francis Irving; on May 25, 2011; under Developer • 4 Comments You may have noticed that the design of the ScraperWiki site has changed substantially. AND SUDDENLY I COULD CONVERT MY BANK STATEMENT FROM PDF TO .and suddenly I could convert my bank statement from PDF to Excel by Aine McGuire; on August 5, 2015; under Front page, Products • Comments Off on .and suddenly I could convert my bank statement from PDF to Excel Do you ever: Need an old bank statement only to find out that the bank has archived it, and want to charge you to get it back? ABOUT US | SCRAPERWIKI Aine McGuire Head of Business Development. Aine began her career selling HP computers and software to large enterprise customers in the finance, pharmaceutical and mining sectors. She created Pygmalion, Microsoft’s first UK Gold Certified Partner and developed business relationships with the City of London’s top 20 leading financial

institutions.

CONSULTING | SCRAPERWIKI We build data rich web applications. We’re experienced in building websites which front large collections of data. This gives the user the power to explore and understand the data, to reuse and repurpose it and to visualise it in imaginative ways. DOCUMENTATION / SCRAPERWIKI LIBRARY In addition to all the standard Python libraries for downloading and parsing pages from the web, ScraperWiki provides the scraperwiki Python library.. Access like this: import scraperwiki. The source code that implements these functions can be found in our bitbucket

repository.

BLOG | SCRAPERWIKI

The Sensible Code Company is our new name. by Francis Irving; on August 9, 2016; under Uncategorized • Leave a comment For a few years now, people have said “but you don’t just do scraping, and you’re not a wiki, why are you called that?”

PRICING | QUICKCODE

We offer Enterprise data hubs with professional services at prices from $2000 / month. For more information read our Corporate FAQ, and if you are interested get

DIGITAL GOVERNMENT

Governments everywhere are making their services easier to use, making full use of the web and mobile. This reduces hassle for citizens and businesses, and saves money for Government. HOW TO SCRAPE AND PARSE WIKIPEDIA How to scrape and parse Wikipedia. by Julian Todd; on December 7, 2011; under Developer • 5 Comments Today’s exercise is to create a list of the longest and deepest caves in DATABAKER – MAKING SPREADSHEETS MACHINE-READABLE DataBaker – making spreadsheets machine-readable. by David McKee; on March 26, 2015; under Case Studies, Data Science, Front page • Comments Off on DataBaker – making spreadsheets machine-readable Spreadsheets are often the way of choice for publishing data. They look great, are understandable by people who don’t use databases, and with judicious use of formatting you can PDF TABLE EXTRACTION OF PAGENATED TABLE The Isle of Man aircraft registry (in PDF form) has long been a target of mine waiting for the appropriate PDF parsing technology. The scraper is here.. Setting aside the GetPDF() function, which deals with copying out each new pdf file as it is updated and backing it up into the database as a base64 encoded binary blob for quicker access, let’s have a look at the what the PDF itself looks like.

IS SCRAPING LEGAL?

Reblogged this on Media law and ethics and commented: . ScraperWiki is a Liverpool-based data tools service and community I did some work for in 2010/11 and a winner of the Knight News Challenge 2011. SCRAPERWIKIPRODUCTSINDUSTRIESCONSULTINGBLOGABOUTDATABAKER QuickCode is the new name for the original ScraperWiki product. We renamed it, as it isn’t a wiki or just for scraping any more. It’s a Python and R data analysis environment, ideal for economists, statisticians and data managers who are new to coding. Go to QuickCode website. The Sensible Code Company is the new name for our company. DOCUMENTATION / SCRAPERWIKI LIBRARY In addition to all the standard Python libraries for downloading and parsing pages from the web, ScraperWiki provides the scraperwiki Python library.. Access like this: import scraperwiki. The source code that implements these functions can be found in our bitbucket

repository.

CONSULTING | SCRAPERWIKI We build data rich web applications. We’re experienced in building websites which front large collections of data. This gives the user the power to explore and understand the data, to reuse and repurpose it and to visualise it in imaginative ways.

BLOG | SCRAPERWIKI

The Sensible Code Company is our new name. by Francis Irving; on August 9, 2016; under Uncategorized • Leave a comment For a few years now, people have said “but you don’t just do scraping, and you’re not a wiki, why are you called that?” SCRAPING GUIDES: EXCEL SPREADSHEETS Scraping guides: Excel spreadsheets. by Francis Irving; on September 14, 2011; under Developer • Comments Off on Scraping guides: Excel spreadsheets Following on from the CSV scraping guide, we’ve now added one about scraping Excel spreadsheets. You can get to them from the documentation page.. The Excel scraping guide is available in Ruby, Python and PHP. . Just as with all BRANDED AND GENERIC MEDICATION COMPARED Branded and Generic medication compared. by Leonisha Barley; on August 26, 2015; under Case Studies • 5 Comments According to the Office of Health Economics for the Association of the British Pharmaceutical Industry (ABPI), the total medicines bill in the UK was £13.6 billion in 2011 and £10.8 billion of this was spent on branded medication.Prescribers such as GPs are encouraged to YAHOO!FINANCE TO TABLEAU VIA SCRAPERWIKI Yahoo!Finance to Tableau via ScraperWiki. by Ian Hopkinson; on April 17, 2014; under Products • Comments Off on Yahoo!Finance to Tableau via ScraperWiki Our recently announced OData connector gives Tableau users access to a world of unstructured and semi-structured data.. In this post I’d like to demonstrate the power of a Python library, Pandas, and the Code in a Browser tool to get SCRAPE ANYONE’S TWITTER FOLLOWERS You may think this is clever, but it is an invasion of people’s privacy and goes against every principle of every privacy legislation. This is the kind of behavior that hurts an entire industry. ‘DOCUMENTATION IS LIKE SEX: WHEN IT IS GOOD, IT IS VERY ‘Documentation is like sex: when it is good, it is very, very good; and when it is bad, it is better than nothing’ by Francis Irving; on May 25, 2011; under Developer • 4 Comments You may have noticed that the design of the ScraperWiki site has changed substantially. AND SUDDENLY I COULD CONVERT MY BANK STATEMENT FROM PDF TOCONVERT BANK STATEMENT TO EXCELBANK STATEMENT PDF TO EXCELEXPORT BANK STATEMENT TO EXCELCONVERT BANK STATEMENT TO CSVCONVERT BANK STATEMENTS TO EXCELTRANSFER BANK STATEMENT TO EXCEL .and suddenly I could convert my bank statement from PDF to Excel by Aine McGuire; on August 5, 2015; under Front page, Products • Comments Off on .and suddenly I could convert my bank statement from PDF to Excel Do you ever: Need an old bank statement only to find out that the bank has archived it, and want to charge you to get it back? SCRAPERWIKIPRODUCTSINDUSTRIESCONSULTINGBLOGABOUTDATABAKER QuickCode is the new name for the original ScraperWiki product. We renamed it, as it isn’t a wiki or just for scraping any more. It’s a Python and R data analysis environment, ideal for economists, statisticians and data managers who are new to coding. Go to QuickCode website. The Sensible Code Company is the new name for our company. DOCUMENTATION / SCRAPERWIKI LIBRARY In addition to all the standard Python libraries for downloading and parsing pages from the web, ScraperWiki provides the scraperwiki Python library.. Access like this: import scraperwiki. The source code that implements these functions can be found in our bitbucket

repository.

CONSULTING | SCRAPERWIKI We build data rich web applications. We’re experienced in building websites which front large collections of data. This gives the user the power to explore and understand the data, to reuse and repurpose it and to visualise it in imaginative ways.

BLOG | SCRAPERWIKI

The Sensible Code Company is our new name. by Francis Irving; on August 9, 2016; under Uncategorized • Leave a comment For a few years now, people have said “but you don’t just do scraping, and you’re not a wiki, why are you called that?” SCRAPING GUIDES: EXCEL SPREADSHEETS Scraping guides: Excel spreadsheets. by Francis Irving; on September 14, 2011; under Developer • Comments Off on Scraping guides: Excel spreadsheets Following on from the CSV scraping guide, we’ve now added one about scraping Excel spreadsheets. You can get to them from the documentation page.. The Excel scraping guide is available in Ruby, Python and PHP. . Just as with all BRANDED AND GENERIC MEDICATION COMPARED Branded and Generic medication compared. by Leonisha Barley; on August 26, 2015; under Case Studies • 5 Comments According to the Office of Health Economics for the Association of the British Pharmaceutical Industry (ABPI), the total medicines bill in the UK was £13.6 billion in 2011 and £10.8 billion of this was spent on branded medication.Prescribers such as GPs are encouraged to YAHOO!FINANCE TO TABLEAU VIA SCRAPERWIKI Yahoo!Finance to Tableau via ScraperWiki. by Ian Hopkinson; on April 17, 2014; under Products • Comments Off on Yahoo!Finance to Tableau via ScraperWiki Our recently announced OData connector gives Tableau users access to a world of unstructured and semi-structured data.. In this post I’d like to demonstrate the power of a Python library, Pandas, and the Code in a Browser tool to get SCRAPE ANYONE’S TWITTER FOLLOWERS You may think this is clever, but it is an invasion of people’s privacy and goes against every principle of every privacy legislation. This is the kind of behavior that hurts an entire industry. ‘DOCUMENTATION IS LIKE SEX: WHEN IT IS GOOD, IT IS VERY ‘Documentation is like sex: when it is good, it is very, very good; and when it is bad, it is better than nothing’ by Francis Irving; on May 25, 2011; under Developer • 4 Comments You may have noticed that the design of the ScraperWiki site has changed substantially. AND SUDDENLY I COULD CONVERT MY BANK STATEMENT FROM PDF TOCONVERT BANK STATEMENT TO EXCELBANK STATEMENT PDF TO EXCELEXPORT BANK STATEMENT TO EXCELCONVERT BANK STATEMENT TO CSVCONVERT BANK STATEMENTS TO EXCELTRANSFER BANK STATEMENT TO EXCEL .and suddenly I could convert my bank statement from PDF to Excel by Aine McGuire; on August 5, 2015; under Front page, Products • Comments Off on .and suddenly I could convert my bank statement from PDF to Excel Do you ever: Need an old bank statement only to find out that the bank has archived it, and want to charge you to get it back? CONSULTING | SCRAPERWIKI We build data rich web applications. We’re experienced in building websites which front large collections of data. This gives the user the power to explore and understand the data, to reuse and repurpose it and to visualise it in imaginative ways. DOCUMENTATION / SCRAPERWIKI LIBRARY In addition to all the standard Python libraries for downloading and parsing pages from the web, ScraperWiki provides the scraperwiki Python library.. Access like this: import scraperwiki. The source code that implements these functions can be found in our bitbucket

repository.

DIGITAL GOVERNMENT

The Government Digital Service (GDS) is tasked with transforming UK government IT services into world-class products and experiences with a strong user focus. In 2010, they came to ScraperWiki, to migrate content from dozens of government websites and portals into one single domain: www.gov.uk. During the initial stage, GDS programmers MARKETING & LEAD GENERATION It’s the era of Big Data. To thrive, marketeers need to get and use data from many places. With ScraperWiki, the open web, social media and customer databases can be sliced and diced to discover and categorise leads, and target campaigns more effectively.

PRICING | QUICKCODE

We offer Enterprise data hubs with professional services at prices from $2000 / month. For more information read our Corporate FAQ, and if you are interested get DATABAKER | SCRAPERWIKI The information can be nested, and the columns broken. This means a machine process doesn’t get a clean run of data by simple parsing. DataBaker solves this problem, making it quick and easy to create recipes which convert spreadsheets into data. DataBaker can be combined with our PDFTables.com technology, which converts PDFs to

spreadsheets.

PDF TABLE EXTRACTION OF PAGENATED TABLE The Isle of Man aircraft registry (in PDF form) has long been a target of mine waiting for the appropriate PDF parsing technology. The scraper is here.. Setting aside the GetPDF() function, which deals with copying out each new pdf file as it is updated and backing it up into the database as a base64 encoded binary blob for quicker access, let’s have a look at the what the PDF itself looks like. A PLACE IN THE COUNTRY A place in the country. by Ian Hopkinson; on October 23, 2013; under Data Science • Comments Off on A place in the country Recently Shelter came to us asking for data on house prices across the UK to help them with some research in support of campaign on housing affordability.. This is a challenge we’re well suited to address, in fact a large fraction of the ScraperWiki team have scraped

IS SCRAPING LEGAL?

Very interesting post. In my opinion, if data is publicly viewable / indexed by search engines, expect it to be scraped. There are ways to prevent scraping from happening, and if one really wants scraping of data to be stopped, they should implement various methods to within their website/service. THE BIG LOTTERY DATA The total awarded is £5,277,058,180 over nearly 10 years. It’s going to 81,386 different organisations. The sizes of grants vary enormously; the biggest, £214,340,846, going to the Big Local Trust, which is an umbrella organisation. Other big recipients include the Royal Society of Wildlife Trusts, who received £59,842,400 for the

Local

SCRAPERWIKIPRODUCTSINDUSTRIESCONSULTINGBLOGABOUTDATABAKER QuickCode is the new name for the original ScraperWiki product. We renamed it, as it isn’t a wiki or just for scraping any more. It’s a Python and R data analysis environment, ideal for economists, statisticians and data managers who are new to coding. CONSULTING | SCRAPERWIKI We build data rich web applications. We’re experienced in building websites which front large collections of data. This gives the user the power to explore and understand the data, to reuse and repurpose it and to visualise it in imaginative ways. DOCUMENTATION / SCRAPERWIKI LIBRARY In addition to all the standard Python libraries for downloading and parsing pages from the web, ScraperWiki provides the scraperwiki Python library.. Access like this: import scraperwiki. The source code that implements these functions can be found in our bitbucket

repository.

BLOG | SCRAPERWIKI

The Sensible Code Company is our new name. by Francis Irving; on August 9, 2016; under Uncategorized • Leave a comment For a few years now, people have said “but you don’t just do scraping, and you’re not a wiki, why are you called that?” SCRAPING GUIDES: EXCEL SPREADSHEETS Scraping guides: Excel spreadsheets. by Francis Irving; on September 14, 2011; under Developer • Comments Off on Scraping guides: Excel spreadsheets Following on from the CSV scraping guide, we’ve now added one about scraping Excel spreadsheets. You can get to them from the documentation page.. The Excel scraping guide is available in Ruby, Python and PHP. . Just as with all BRANDED AND GENERIC MEDICATION COMPARED Branded and Generic medication compared. by Leonisha Barley; on August 26, 2015; under Case Studies • 5 Comments According to the Office of Health Economics for the Association of the British Pharmaceutical Industry (ABPI), the total medicines bill in the UK was £13.6 billion in 2011 and £10.8 billion of this was spent on branded medication.Prescribers such as GPs are encouraged to YAHOO!FINANCE TO TABLEAU VIA SCRAPERWIKI Yahoo!Finance to Tableau via ScraperWiki. by Ian Hopkinson; on April 17, 2014; under Products • Comments Off on Yahoo!Finance to Tableau via ScraperWiki Our recently announced OData connector gives Tableau users access to a world of unstructured and semi-structured data.. In this post I’d like to demonstrate the power of a Python library, Pandas, and the Code in a Browser tool to get SCRAPE ANYONE’S TWITTER FOLLOWERS You may think this is clever, but it is an invasion of people’s privacy and goes against every principle of every privacy legislation. This is the kind of behavior that hurts an entire industry. ‘DOCUMENTATION IS LIKE SEX: WHEN IT IS GOOD, IT IS VERY ‘Documentation is like sex: when it is good, it is very, very good; and when it is bad, it is better than nothing’ by Francis Irving; on May 25, 2011; under Developer • 4 Comments You may have noticed that the design of the ScraperWiki site has changed substantially. AND SUDDENLY I COULD CONVERT MY BANK STATEMENT FROM PDF TOCONVERT BANK STATEMENT TO EXCELBANK STATEMENT PDF TO EXCELEXPORT BANK STATEMENT TO EXCELCONVERT BANK STATEMENT TO CSVCONVERT BANK STATEMENTS TO EXCELTRANSFER BANK STATEMENT TO EXCEL .and suddenly I could convert my bank statement from PDF to Excel by Aine McGuire; on August 5, 2015; under Front page, Products • Comments Off on .and suddenly I could convert my bank statement from PDF to Excel Do you ever: Need an old bank statement only to find out that the bank has archived it, and want to charge you to get it back? SCRAPERWIKIPRODUCTSINDUSTRIESCONSULTINGBLOGABOUTDATABAKER QuickCode is the new name for the original ScraperWiki product. We renamed it, as it isn’t a wiki or just for scraping any more. It’s a Python and R data analysis environment, ideal for economists, statisticians and data managers who are new to coding. CONSULTING | SCRAPERWIKI We build data rich web applications. We’re experienced in building websites which front large collections of data. This gives the user the power to explore and understand the data, to reuse and repurpose it and to visualise it in imaginative ways. DOCUMENTATION / SCRAPERWIKI LIBRARY In addition to all the standard Python libraries for downloading and parsing pages from the web, ScraperWiki provides the scraperwiki Python library.. Access like this: import scraperwiki. The source code that implements these functions can be found in our bitbucket

repository.

BLOG | SCRAPERWIKI

The Sensible Code Company is our new name. by Francis Irving; on August 9, 2016; under Uncategorized • Leave a comment For a few years now, people have said “but you don’t just do scraping, and you’re not a wiki, why are you called that?” SCRAPING GUIDES: EXCEL SPREADSHEETS Scraping guides: Excel spreadsheets. by Francis Irving; on September 14, 2011; under Developer • Comments Off on Scraping guides: Excel spreadsheets Following on from the CSV scraping guide, we’ve now added one about scraping Excel spreadsheets. You can get to them from the documentation page.. The Excel scraping guide is available in Ruby, Python and PHP. . Just as with all BRANDED AND GENERIC MEDICATION COMPARED Branded and Generic medication compared. by Leonisha Barley; on August 26, 2015; under Case Studies • 5 Comments According to the Office of Health Economics for the Association of the British Pharmaceutical Industry (ABPI), the total medicines bill in the UK was £13.6 billion in 2011 and £10.8 billion of this was spent on branded medication.Prescribers such as GPs are encouraged to YAHOO!FINANCE TO TABLEAU VIA SCRAPERWIKI Yahoo!Finance to Tableau via ScraperWiki. by Ian Hopkinson; on April 17, 2014; under Products • Comments Off on Yahoo!Finance to Tableau via ScraperWiki Our recently announced OData connector gives Tableau users access to a world of unstructured and semi-structured data.. In this post I’d like to demonstrate the power of a Python library, Pandas, and the Code in a Browser tool to get SCRAPE ANYONE’S TWITTER FOLLOWERS You may think this is clever, but it is an invasion of people’s privacy and goes against every principle of every privacy legislation. This is the kind of behavior that hurts an entire industry. ‘DOCUMENTATION IS LIKE SEX: WHEN IT IS GOOD, IT IS VERY ‘Documentation is like sex: when it is good, it is very, very good; and when it is bad, it is better than nothing’ by Francis Irving; on May 25, 2011; under Developer • 4 Comments You may have noticed that the design of the ScraperWiki site has changed substantially. AND SUDDENLY I COULD CONVERT MY BANK STATEMENT FROM PDF TOCONVERT BANK STATEMENT TO EXCELBANK STATEMENT PDF TO EXCELEXPORT BANK STATEMENT TO EXCELCONVERT BANK STATEMENT TO CSVCONVERT BANK STATEMENTS TO EXCELTRANSFER BANK STATEMENT TO EXCEL .and suddenly I could convert my bank statement from PDF to Excel by Aine McGuire; on August 5, 2015; under Front page, Products • Comments Off on .and suddenly I could convert my bank statement from PDF to Excel Do you ever: Need an old bank statement only to find out that the bank has archived it, and want to charge you to get it back? DOCUMENTATION / SCRAPERWIKI LIBRARY In addition to all the standard Python libraries for downloading and parsing pages from the web, ScraperWiki provides the scraperwiki Python library.. Access like this: import scraperwiki. The source code that implements these functions can be found in our bitbucket

repository.

DIGITAL GOVERNMENT

Governments everywhere are making their services easier to use, making full use of the web and mobile. This reduces hassle for citizens and businesses, and saves money for Government.

PRICING | QUICKCODE

We offer Enterprise data hubs with professional services at prices from $2000 / month. For more information read our Corporate FAQ, and if you are interested get MARKETING & LEAD GENERATION It’s the era of Big Data. To thrive, marketeers need to get and use data from many places. With ScraperWiki, the open web, social media and customer databases can be sliced and diced to discover and categorise leads, and target campaigns more effectively. DATABAKER | SCRAPERWIKI Excel spreadsheets are often the method of choice for sharing data. They look great, they’re understandable by people who don’t use

databases, and they

PDF TABLE EXTRACTION OF PAGENATED TABLE The Isle of Man aircraft registry (in PDF form) has long been a target of mine waiting for the appropriate PDF parsing technology. The scraper is here.. Setting aside the GetPDF() function, which deals with copying out each new pdf file as it is updated and backing it up into the database as a base64 encoded binary blob for quicker access, let’s have a look at the what the PDF itself looks like. A PLACE IN THE COUNTRY A place in the country. by Ian Hopkinson; on October 23, 2013; under Data Science • Comments Off on A place in the country Recently Shelter came to us asking for data on house prices across the UK to help them with some research in support of campaign on housing affordability.. This is a challenge we’re well suited to address, in fact a large fraction of the ScraperWiki team have scraped

IS SCRAPING LEGAL?

Very interesting post. In my opinion, if data is publicly viewable / indexed by search engines, expect it to be scraped. There are ways to prevent scraping from happening, and if one really wants scraping of data to be stopped, they should implement various methods to within their website/service. THE BIG LOTTERY DATA The BIG Lottery Data. by Ian Hopkinson; on December 31, 2013; under Data Science • Comments Off on The BIG Lottery Data The UK’s BIG Lottery Fund recently released its grant data since 2004 as a set of lovely CSV files: You can get it yourself here or here.I found it a great opportunity to try out some new tricks with Tableau, and have a bit of a poke around another largish dataset from PDFTABLES – A PYTHON LIBRARY FOR GETTING TABLES OUT OF PDF pdfminer brings additional functionality over pdftohtml, hence the switch – the fact it is Python based is convenient but not essential. We’ve used Abby in the past, and if we go down the commercial application route we’d probably stick with them.

Navigation

SCRAPERWIKI

EXTRACT TABLES FROM PDFS AND SCRAPE THE WEB

* Products

* PDFTables.com

* DataBaker

* QuickCode

* Industries

* Academic Research

* Digital Government * Intergovernmental Organizations * Business Publishing

* Data Journalism

* Marketing & Lead Gen

* Consulting

* Blog

* About

HOME

SCRAPERWIKI HAS TWO NEW NAMES! ONE FOR THE PRODUCT AND ONE FOR THE COMPANY: QuickCode is the new name for the original ScraperWiki product. We renamed it, as it isn’t a wiki or just for scraping any more. It’s a Python and R data analysis environment, ideal for economists, statisticians and data managers who are new to coding. Go to QuickCode website The Sensible Code Company is the new name for our company. We design and sell products that turn messy information into valuable

data.

Go to Sensible Code website You might also like our other product Get tables from PDFs ScraperWiki © 2016. All Rights Reserved. — Admin login

* About

* Consulting

* Contact us

* Blog

* Newsletter

* Jobs

Powered by WordPress . Designed by

We're hiring!