“The past” and “the future” battle live in QueryPic web

Our wondrous Query Harvester can extract the data you need from Trove’s newspaper holdings to track trends in detail. But before embarking on a major mining exercise it is sometimes useful to survey the terrain. QueryPic creates a quick snapshot of your search query, displaying the total number of articles your query matches over a span of years.

QueryPic comes in two varieties. The classic QueryPic gives you full control, but demands a certain familiarity with devices such as the command-line. It is provided as a Python script to be downloaded and run upon your own machine.

If your needs are simple and your patience is thin then you might prefer the speed and excitement of our latest QueryPic in-browser engine. No downloads! No terminals! Just go to QueryPic enter a word or phrase and… whoosh!

As an extra added bonus, the web incarnation of QueryPic includes access to the the wondrous collection of New Zealand newspaper made available through Papers Past.

QueryPic web is able to accomplish this magic by utilising the APIs generously provided by DigitalNZ and Trove.

Both varieties of QueryPic present the results of your query in graphical form. Additional queries are easily added to existing graphs for the purposes of detailed comparison. But what, you may wonder, lies beneath these peaks and troughs — investigate further by simply clicking on a point to reveal the first twenty matching results. More than just a fetching illustration, this is a tool for determined inquiry!

QueryPic web

 QueryPic classic



The new API-powered future

The long-awaited Application Programming Interface (API) to Trove is now available. Huzzah! The frustrations of broken screen-scrapers will soon be a thing of the past.

But this glorious new future will not come without cost. WraggeLabs will henceforth be embarking on a process of code consolidation and renewal as we overhaul all our Trove products to take advantage of this machine-readable goodness.

The first fruits of this process can be enjoyed in the new fully-web-driven version of QueryPic. Instant graphs! No downloads!

More such wonders are to come, but bear with us dear client, as it make take some weeks or months to fully update our wares. In the meantime, it is unlikely that any development of pre-API versions will be undertaken.

We should also announce the imminent retirement of our Unofficial API. It has served us well, but it’s time has now come. Farewell, old friend.


More Trove scraper drama

Once again a minor change to the Trove newspapers code (from <i> and <b> to <em> and <strong>) broke my scraper and the tools that depend on it. Fortunately @erochest was quickly on the job and has submitted a fix. If you’re having problems, please update scrape.py from the repository.

I will update the unofficial API shortly.


Trove Tools calamity

I am sad to report that due to a minor change in the Trove website (a <strong> tag was changed a <h1>!) most of my Trove tools are experiencing difficulties.

The good news is that the fix required is small and I’ve updated the scraper that powers most of the tools.

If you’re using the command line version of the Query Harvester you can download the latest code and be on your way. The Search Summariser wasn’t affected, but there’s a new version available for your perusal including some extra features (these will be documented anon).

The Zotero translator for the newspapers database has been updated and submitted to the Zotero repository. Once it has been approved the translator will be upgraded automatically.

I believe that the Unofficial Trove API is unaffected. If you believe otherwise, please let me know.

And now the bad news…

The Query Harvester GUI will need to be rebuilt. I will try and do this as soon as possible, but I can’t make any promises. Bribery and flattery might help.

My apologies for any discomfort or anxiety. I always say that screen scrapers are inherently fragile. It’s part of the game. But it’s still extremely annoying when something like this happens. :(

Let’s just look forward to an official Trove API which should ease our pain considerably.

Trackbacks / Pings

Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>