LAMP to be integrated into Jisc’s Learner and Business Analytics R&D activities

Until now, the LAMP project has run in parallel with related activities such as the joint Jisc/HESA/HESPA Business Intelligence and Effective Learner Analytics initiatives.   We are pleased to announce that LAMP will now move forward as an integral part of the overarching learning analytics R&D efforts of Jisc. Specifically, the LAMP project objectives will be combined with those of the Learning Analytics challenge, which is developing a basic student attainment and retention dashboard for all universities and colleges, developing support around navigating the ethics of using analytics about students and providing guidance to help people engage with using learning analytics. A key component of this dashboard will be concerned with the use cases identified by LAMP, and particularly the ability to view library resource usage by subject, course, social demographics and level of student attainment.

This integration with wider analytics efforts requires a more robust and scaleable approach to technical development, and Jisc has started an EU procurement exercise to identify a number of technology providers who can work with us to develop the dashboard. This procurement will also include effort to help identify universities and colleges who are in a good position to act as early adopters for the dashboard, and among these will be institutions that have already contributed to the LAMP project.

For further information and updates on LAMP and Jisc analytics R&D activity,  please now go to: http://analytics.jiscinvolve.org/wp/

 

A Library Analytics and Metrics Service? Moving into the next phase of work

Although we’ve been sharing the work of the LAMP project at the UKSG conference, Jisc Digifest, and the SCONUL conference over the last few months, we realise it’s been a few months since we’ve posted on our progress and intended next steps.

Our work has amassed a lot of interest over the last 6 months, with the leadership in Jisc pointing to it as one of our exemplar projects: responding to clear demand, developed in close collaboration with the stakeholder community, and demonstrating our capability to innovate and develop services in strategically vital areas.

The project is now officially entering into its second stage, aiming to move this exploratory project forward into a fully fledged service. To make this happen, we’re going to be focusing on several areas of work:

Creating a user interface prototype that is easy and pleasurable to use.  We have already developed what we’ve affectionately called the ‘ugly’ prototype, which has allowed us to play with the data and explore the potential for the tools.

This has thrown up all sorts of questions around what level of functionality a data visualisation of this nature should incorporate, and also broader questions over data literacy, and what ‘data analysis’ takes place within the system and what is undertaken by the user herself.  After consultation with our Community Advisory and Planning Group, we have developed a set of wireframes that we feel will support users in analysing their data in different ways, supporting them to view and experiment with the data in different ways, but within a supported environment. We are presently undertaking the technical work to produce v0.1 of LAMP, which will be released in November to the seven partner institutions that have supplied their data: University of Manchester, University of Salford, University of Huddersfield, Wolverhampton University, University of Exeter, De Montfort University, and Lancaster University.

Testing and evaluating the tools. Once the user interface (UI) is released to the institutions, we will be undertaking extensive evaluation of the tools, assessing the usability of the UI, identifying data issues or opportunities, and working to get a better understanding of how tools such as these might fit within library workflows — the benefits they may help deliver, and their overall value. We are also looking at creating a UI with dummy data so that users outside the seven pilot institutions can access and meaningfully experience and experiment with the tools. Outcomes from this work will feed into future versions of the prototype, as well as the overarching business case for the service. We’ll need to understand the value and impact of the service to ensure it’s validity and sustainability.

Beyond v0.1. Bringing in NSS data, institutional profiling and other functions.
The version released in November won’t include institutional profiling (formerly referred to as benchmarking) features, National Student Satisfaction (NSS) data views,  the statistical significance layer, or the ability to look at item level data around individual or batches of resources.  These are all areas identified as priority developments by our Community Advisory and Planning Group and other stakeholders, and we’ll be exploring further how to take them forward over the next few months. Ellen Collins from the Research information Network (RIN) taking the lead and developing specifications where it is feasible to do so, for example, we need to investigate whether NSS data can interoperate with the UCAS data contributed by institutions before we can say we can easily integrate it into the final service. However, our aim is to integrate into the tools:

  • the ability to know whether data is revealing a statistically significant trend or not, i.e. is the disparity between male and female usage on a particular course of significance, or is it merely reflective of the course make up as a whole?)
  • the ability to view resource usage against NSS data, i.e. enabling users to examine the correlation between departmental/subject area usage of resources and NSS scores.
  • the ability to view item level data, so that we users can view overall usage of items or groups of items, and also dig deeper to see who is using those items (which departments, courses, and so on).
  • the ability to view usage of your institution’s resources compared to others using the system, a.k.a. institutional profiling.

 

Supporting data-driven decision-making — the need for community engagement

We know that our testing of the tools with real users on top of real data will reveal how the tools might be useful. But we also know from our engagement with librarians and bodies such as SCONUL and RLUK over the last year that we’re simultaneously opening up a range of broader questions about the role of data and visualisations in supporting library and institution’s decision-making, the skill-sets and confidence of librarians in working with data in these new ways, and the need to share stories and best practice with the broader community.  We will be developing these case studies as the tools develop, and producing guidance materials based on real use cases, and launching these in spring of 2015.  We recognise there is a need to build a community around Jisc library support and analytics tools, and are in the early stages of planning a wider event around these issues in April 2015. Here we will share the progress of the LAMP work along with similar initiatives, and promote discussion and exploration of the issues surrounding analytics and data-driven decision-making in libraries today.

Beyond measuring loans and logins. Capturing eResource data trails.

Although we can capture eResource logins from many institutions, and tie this to anonymised identifiers that enable us to view the level of eResource usage of particular cohorts, what we can’t tell is what specific eResources, databases, or articles are being viewed by those cohorts. This is an result of the current approach of the UK Access Management Federation, configured to ensure Data Protection and privacy.  However, there are questions over whether it would be feasible to gather and leverage this data in secure ways to support LAMP use cases as well as others, including Learning Analytics.

Indeed, how viable is a service like LAMP if it can only meaningfully track activity around physical items? Jisc and other stakeholders have indicated a strong interest in revisiting this territory so we can identify the opportunities and barriers, and Ben Showers and I look forward to taking this forward in behalf of the LAMP team over the next few months.

 

 

 

 

 

 

 

 

Screen flows, wireframes, prototypes and guts.

Following on from my last post where we were looking at gathering user research and requirements, here’s an update on the recent  team workshop where we focussed on the structure and skeleton of the LAMP.

1. The 20 second “Gut” test

We kicked off the session with a 20 Second “Gut” Test, which is technique used to clarify preferences, and better understand the team’s views on the aesthetics of visual design. The test showed a screen capture of 20 different analytics dashboards / user interfaces / visual elements, for 20 seconds only. Each participant has to score their gut reaction to the slide 1-5 (5 being the highest) and make any notes.

This was a really enlightening exercise, which really helped the team to articulate what they did and didn’t like. Looking at our top /bottom 5 there were very apparent themes running between them:

  • Clean and simple style, with space to let the content breath
  • Informative charts with the right balance of detail
  • Modular blocks of content held within frames
  • Restricted colour palette
  • Visualisations of the data were honest and unadorned.

2. Developing the screen flows and navigation model

The navigation model is the big picture, or the “birds eye view” of the system. It considers where users start, how they get from here to there, and what all of the major elements will be. This can then be summarised as a flow diagram that the models the user journeys.

Storyboarding the user experience

  • We started off as a group using our understanding of the tool to build a prototype screen flow.
  • We then validated this against the real life use cases / job stories.
  • We then tried to break the system we’d created. What were the extreme limitations of the system, and had these been taken into account.

LAMP-Screen-flow

This followed an iterative design process: Sketch > Prototype > Present > Validate > Repeat until we had exhausted our time. Collectively  we had formulated a solid idea that, that has been validated against our user research. This can now be taken away and explored in more detail by the UX team, modelled and then presented to the CAB for feedback.

3. Getting into the details

Low fidelity prototyping

Based on the screen flows and navigation model we had created it was clear that there were 2 key areas of action to focus on; the chart creation screen and the Dashboard area where charts are stored. We wanted to start understanding these in more detail and begin to wireframe the user journey, the interactions and functions of these screens. We had generated loads of ideas for these screens and we needed to capture them. We’re not intending hammer down every detail but rather to create a consensus that can be refined outside of the workshop.

The eight guiding principles of prototyping

  1. Understand your audience and intent
  2. Plan a little – prototype the rest
  3. Set expectations
  4. You can sketch
  5. It’s a prototype — Not the Mona Lisa!
  6. If you cant make it, fake it
  7. Prototype only what you need
  8. Reduce risk prototype early and often.

6-8-5 Design studio.

The first iteration of wireframing follows a 6-8-5 rule – do 6-8 sketches, on an 8-up grid in 5 minutes. The sketches can be different versions of a particular aspect you’re working on or a storyboard workflow (before, during and after login) or mix and match! Keep it high level, and get just enough detail down, to convey your concept. When the 5 min is up each person presents their ideas and the group critiques the ideas.

Quantity trumps quality at first.

The idea here is to get a large quantity of ideas rather than quality.  Here’s a short example to illustrate what we mean by this.

“A ceramics teacher announced he was dividing his class into two groups. All those on the left side of the studio would be graded solely on the quantity of work they produced, all those on the right graded solely on its quality.

His procedure was simple: on the final day of class he would weigh the work of the “quantity” group: 50 pounds of pots rated an A, 40 pounds a B, and so on. Those being graded on “quality”, however, needed to produce only one pot – albeit a perfect one – to get an A.

Well, come grading time and a curious fact emerged: the works of highest quality were all produced by the group being graded for quantity!

It seems that while the “quantity” group was busily churning out piles of work – and learning from their mistakes – the “quality” group had sat theorizing about perfection, and in the end had little more to show for their efforts than grandiose theories and a pile of dead clay.”

http://www.jeffgothelf.com/blog/quantity-trumps-quality/

This story perfectly articulates one of the fundamental Lean UX principles: prioritize making over analysis. Instead of sitting around, debating ad nauseam which direction to go in, what features make sense, which colors perfectly reflect your brand values or which words will get your customers to convert, just make something. It won’t be perfect. It won’t work as well as you’d hoped at first but it will teach you something. You’ll get some feedback, some insight on how building your product can be better and you’ll do a better job the second time around.

A lot of the methods and ideas we’ve used in this workshop have been taken from ‘Prototyping – A practitioner’s guide’  by Todd Zaki Warfel (http://rosenfeldmedia.com/books/prototyping/). In his book he talks about the value of prototyping, the value of show, tell and experience.

Prototyping reduces misinterpretation

Take a 60 page requirements document. Bring in 15 people into a room. Hand it out. Let them read it all. Now ask them what you’re building. You’re going to get 15 different answers. Prototypes are a more concrete and tactile representation of the system you’re building. They provide tangible experiences.

He then goes on to say moving from  requirements-dependant process to a prototype-dependant process has increased consensus on interpretation to 60-80% to +90%. It also requires far less effort and time for everyone involved. Taking this user centred design approach is essential for LAMP as the system is still being explored, designed and interpreted. Manifesting the development work in a physical form helps to generate hundreds of ideas, some will be great, some will be not so. But even these not so great ideas can be the catalyst for great solutions.

The ideas that were generated will be explored in more detail by the UX team, modelled, and validated. We are then meeting again for another wire framing session with the LAMP team to work through more of these details ready to presented to the CAB for feedback.

Sketch > Prototype > Present > Validate > Repeat

 

So what do we mean when we say ‘Analytics’?

This is a guest post by David Kay of Sero Consulting who describes some of the project’s work to develop user stories and enable a better understanding of the kinds of functionality any shared analytics service would need to have

Analytics has become quite a buzzword over the past couple of years. Awareness has been promoted by general talk of ‘big data’ as well as by increasing emphasis in the sector on the student experience and success factors (linked to ‘Learning Analytics’) and on resource optimisation driven by economic constraints.

Furthermore EDUCAUSE and the Gates Foundation in the US and Jisc in the UK have enabled notable exploratory work.

And now every new generation library systems offering needs the ‘Analytics’ badge.

But what does analytics mean to library professionals UK Higher Education? Is analytics ‘all things to all men’ or simply ‘the emperor’s new clothes’ (formerly known as management reporting or the director’s dashboard)?

So in Autumn 2013 the LAMP project set out to discover what library teams really have on their minds. Whilst LAMP is specifically focussed on the opportunities for shared services around library analytics, we stepped back to the underlying question of ‘What do libraries want to achieve with analytics?’ regardless of whether a shared service can help (our second order question as a project being to identify the cases where LAMP might help).

A total of eleven libraries working with the LAMP project agreed to develop a set of User Stories describing what they would like to achieve with analytics. We agreed to a two-step process whereby seven libraries were interviewed to source a set of stories and then the wider group (the original seven and four more) voted on the relevance of the stories (around 90) from their local perspective.

Thanks go to the library teams at the universities of Birmingham, De Montfort, Exeter, Huddersfield, Hull, Manchester, Warwick, Wolverhampton, York, the London Business School and the Open University.

About User Stories

User Stories are recognised as a valuable tool for capturing clearly focused requirements to inform software development. For the purpose of this investigation, a user story was taken to be a statement in the form of:

As a (role),
I want (a thing)
in order to (achieve an outcome)

For example

As a (late riser),
I want (to get my breakfast quickly)
in order (to catch the train)

We’d consider that to be an ‘epic’ story, from which a number of more detailed stories about real requirements might be teased out; for example

As a (late riser),
I want (a four-slice toaster)
in order (to make breakfast quicker)

and
I want (a folding bike)
in order (to get to the station quicker)

The stories we collected from library teams fell in to both these categories – epic stories that described the mission to which analytics might contribute and lower level descriptions of how particular analytic activities might deliver or contribute to key outcomes; for example, the mission might be

As a (library manager)
I want (more comprehensive activity data)
in order (to improve student satisfaction)

That mission might be unpacked in to

I want (front desk enquiry analysis)
in order (to improve first level resolution)

and

I want (short loan turn away data)
in order (to expand the collection to meet demand)

What‘s analytics about? Our Library Stories

So what did our 11 libraries consider the most important contributions to be made by analytics?

As described above, we collected around 90 stories and then put them to the vote! Our voting system allowed a library to allocate 2 points for any story they regarded as ‘important’ and 1 point for a ‘useful’ story. Therefore a story regarded as ‘important’ by everyone could gain 22 points (11 libraries x 2 points). The 49 stories that gained over one third of the maximum points (i.e. 8/22) are listed here.

We classified 19 stories of those 49 as ‘epic’ or ‘mission’ stories – very interesting because they indicate the management and strategic purposes that library analytics need to serve. They are as follows:

We classified 30 of the 49 as ‘activity’ stories – more detailed things that librarians want to do with analytics. They are as follows:

Some reflections

You’ll see from the listings above that we categorised each statement in terms of its broad intent:

  • Mission – High level ‘mission’ statements that are ‘epic’ user stories
  • Data – Stories about the range of data available for analysis
  • Collection – Use of analytics for collection management
  • Service – Use of analytics for service improvement, including enquiries
  • Teaching & Learning – Use of analytics to enhance the learning experience and student success
  • Recommendation – Use of analytics to provide recommender services

It is important to observe that the principal focus of the ‘mission’ stories is collection management (AN) and its contribution to each of value (M), satisfaction (D) and impact (C). There is also strong recognition of analytics as a tool in:

  • Supporting dialogue with faculty (K)
  • Evidencing and positioning library business cases (A, F)
  • Proactively enabling support activity such as skills development to be better designed and targeted (V, AB, AS, AD)

Whilst the ‘activity’ stories mainly speak for themselves, the challenge for libraries and for systems providers is to identify what data is required to support these requirements and how it might feasibly be collected within and across systems.

  • The focus on e-resources emphasises this challenge as represented in two of the top three activity stories (38, 4, also 19) – especially linking e-resource activity to users just as we are accustomed to doing with print.
  • There is a persistent recognition that insightful analytics need to combine data from more than just a single vendor system (2, 29, 32, 1).
  • More firmly within grasp is the use of analytics to respond more effectively to differentiations in terms of faculty (14, 9) and user demographics (33).
  • Analytics relating to enquiry management and related service improvement is an important dimension (29, 48, 54)
  • Whilst clearly recognised as an opportunity (61, 62, 34), there is less emphasis on using analytics for recommendation, surfacing reading options for users as popularised by such as Amazon.
  • Last but not least, we shouldn’t underestimate that presentation is a critical part of the challenge (8, 9)

There is much food for thought here, hopefully informing how services might be developed to exploit the data pool in which ‘no system is an island’!

Whilst JUSP and LAMP are partnering with UK academic libraries to develop responses in specific areas, it is clear from our User Stories that library analytics will demand considerable thought and may reveal even greater potential as our understanding and practices mature.

Personas, job stories and simple planes: wireframing a LAMP interface

Let me start by introducing myself – my name is Benjamin Perry  and I am the Creative Design Coordinator at Mimas. My role on this project is alongside Leigh Morris as the User Experience (UX) Team who are responsible for designing the website itself. As part of this team it’s crucial that I not only understand but am involved with the Information Architecture and the User Interaction Design (the Skeleton Plane – more about planes below), However my particular focus is the visual appearance of the website itself (the Surface Plane).

Having recently read the seminal work of Jesse James Garrett on The Elements of User Experience , it’s clear that where I join this project it has already been through some good User Centered Design (UCD) processes. In Garrett’s book he talks about UX design consisting of 5 layers; the Strategy Plane, the Scope Plane, the Structure Plane, the Skeleton Plane and finally the Surface Plane. “These five planes provide a conceptual framework for talking about user experience problems and the tools we use to solve them”

 

Jesse James Garrett's simple planes http://www.jjg.net/elements/pdf/elements_simpleplanes.pdf Jesse James Garrett’s simple planes http://www.jjg.net/elements/pdf/elements_simpleplanes.pdf

As you work through each of the planes the issues that you have to deal with move from being abstract to more concrete. Each of these planes is dependent on the ones below it, but this does not mean that each plane needs to be finalized before the next can be started. It’s much better to be flexible; sometimes decisions made on an upper plane may force a reevaluation (or an evaluation for the first time) of an issue from a lower plane.

As I said before my dealings are mainly with the Skeleton and Surface planes, so I’m very much at the sharp end of the process. My work requires the planes before these to have been clearly defined (but not finalized!). As much as I’d like it, I don’t ever expect to be handed all the project brief and documentation then sail through the work without having any questions and get sign off first go – It’s never going to happen. User Centered Design is core to our business at Mimas so the UX team need to be the ones preaching this gospel the loudest. If we can enlighten people to the process we use and give them the tools to help work through the issues that they are faced with, then we will not only build better products but our jobs will be made much easier too.

So lets take a step back and look at what’s happened so far:

David Kay has been looking at the Epic Level narrative with the User Stories. These have been essential to help provide a clear business case for this project and define its strategy – the Strategy Plane.

Bethan Ruddock  has then used these to create some Personas, which have been used to create workflows. These are step-by-step guides that detail how a user would potentially perform a tasks based around of the features of the website (we’ve made these available). These have been used to start to flesh out the Scope Plane and also start to inform the Structure Plane.

The UX team has then taken these workflows along with some early prototype wireframes and visuals to create interactive walkthroughs of how these tasks might be achieved in the UI (we used InVision to do this – its fantastic!) – Structure, Skeleton and Surface Planes.

These have been essential to help generate discussion not only within the team but the wider LAMP CAP group. Seeing something physical in front of you is very powerful and it certainly highlighted some of the issues on the lower planes that needed to be readdressed or hadn’t yet been addressed.

In discussing these issues I was reminded of this blog post by Alan Klement  that Leigh had found which introduced the idea of Job Stories. In his post he summarises –

“… the problem with user stories is that it’s too many assumptions and doesn’t acknowledge causality. When a task is put in the format of a user story (As a [type of user], I want [some action], so that [outcome] ) there’s no room to ask ‘why’ – you’re essentially locked into a particular sequence with no context.”

Klement says with mature defined projects his team were able to talk through and understand proposed changes however “because our canvas is blank, we are having trouble getting on the same page when it comes to customer motivations, events and expectations. But today, things have turned around. I’ve come across a great way to use the jobs to be done philosophy to help define features. I call them Job Stories”

This immediately drew parallels with our project and seemed a natural solution for us for us to gather more information to inform our UCD process. We needed to get a more definitive idea of what people want to do with LAMP but also wrap that in real world context and expectations. With the CAP group full of future users of the site we thought this would be the perfect opportunity to introduce this idea and get them to tell us about their own Job Stories. So we gave the group blank forms to fill in following Klement’s process: (situation) When _____ , (Motivation) I want to _____ , (Expected Outcome) so I can _____ .

Getting the CAP group to think in this way was really positive. The information we collected is helping to define the features for this first phase of the project as well as generating discussion around future enhancements.

As a first outing using Job Stories we found this to be very successful. I’m not sure we did this in the same way that Alan Klement did it with his team, but it certainly generated lots of discussion, highlighted issues and gave us much more information to work with than we had before. What’s more, this information is not based on assumptions, as with the personas, but based on real life users, which is what we set out to achieve. You can see the job stories the CAP created.

We’ll be using the information and insights from these job stories as we work on the next stage of developing the LAMP interface.

It’s time to talk about standards

Look, this is a library project. You knew the s-word was going to come up at some point.

One of LAMP’s most important attributes is that it’s bigger than a single institution. While we want individual universities to be able to upload and interrogate their own data through the platform, we also want to offer them somewhere that they can aggregate with and benchmark against their peers. The tools that we build have to meet the needs of a lot of different people.

We’ve written before about some of the tricky decisions we’re taking about how we standardise and reclassify the data that we get, in order to make sure that it can work with LAMP’s systems, and can be aggregated across institutions. But a recent conference call with the team who are managing Wollongong University’s Library Cube service reminded us that there’s another way to do this: looking at the way we ask that information to be provided in the first place and creating clear standards which help institutions to collect their data the way that we want them to.

A bit of background.

The Library Cube is a pretty well-established initiative from Wollongong University Library which seeks to collect and analyse data from a number of systems to understand how libraries add value. Wollongong have been working on this service for several years and the scope is now extending beyond assessing library value to thinking about real-time data and service development. We’ve been aware of their work through the links they had made with the Huddersfield Library Impact Data Project and the opportunity came up to share progress on their project and on LAMP.

Now, previous work we’ve done on normalisation has tended to be about how we might aggregate groups that are classified differently in different organisations. Subjects are particularly tricky for this, as every university has its own way of organising courses and departments. These decision are taken locally, and it’s improbable that a university’s academic departments will be completely reorganised to meet the needs of a project on library analytics (well, we can dream!).

But the conversation with Wollongong highlighted some areas where we might have a bit more control, and could think about asserting standards and/or best practice about how data are collected and supplied. Take, for example, e-resource logins. These datasets are huge, recording every login from every student over the course of a year. To simplify our analysis for the LIDP at Huddersfield and subsequently with LAMP, we looked at how many times a student had logged in during a given hour over the course of a year, for each of the 24 hours in the day. Wollongong did the same, but their time period was ten minutes.

This means that comparing our data isn’t straightforward. There’s no intrinsic reason that we picked an hour, and that they picked ten minutes; both have advantages and disadvantages. The ten-minute data will give a more nuanced analysis, while the hour-by-hour data will be easier to process. Both choices are valid. But because we made them separately and individually, we didn’t necessarily think about the wider ramifications of our eventual decisions.

Of course, doing a project such as LAMP will begin to set some informal standards, simply because we’re asking for data in particular formats. But, as our conversation with Wollongong made clear, it’s important that we don’t allow those informal standards to evolve into more widely-accepted ones without interrogating and testing them. LAMP isn’t happening in isolation; there’s a wider set of projects especially in Australia and the US which are looking at library analytics and measurement.

Over the next few months, we hope to start talking about the best ways to collect and share data, building on our experiences and that of others, to ensure that LAMP’s collaborative ethos extends to some bigger conversations about library data and capturing library value.

Diagnosing uncertainty: A premortem for LAMP

A few weeks ago (okay, it was early July!), LAMP had the second meeting of its community advisory and planning (CAP) group.

The meeting started with an update on the work of the project so far, and sort advice and input from the group on a number of challenges.

A number of these challenges have already been blogged about, including the technical architecture; the use of unique identifiers associated with the data; data normalisation and statistical analysis, and; designing the database.

Importantly, the project has also drafted Terms and Conditions for institutions submitting data to the project. This is a small, but critical part of the project being able to get data from institutions.

There is a lot happening with LAMP at the moment (as these posts highlight); but what of the future?

The LAMP Premortem

In the afternoon the group undertook a premortem of the project, facilitated by Andy McGregor (of Jisc, but also a member of the CAP group).

The premortem imagines a future where the project has been a failure, and participants must work backwards to understand what contributed to the project’s failure.

 

 

 

 

 

Despite the slightly gloomy sounding description, the exercise is actually a huge amount of fun, and generated some really useful insights and ideas for the project team to take away.

What follows is a brief outline of some of the main themes that emerged during the premortem and specific ideas for the project team (and CAP group) to work on.

Technical

It was clear that the technical side of things could result in a number of significant risks. The majority of the technical risks actually related to the expectations libraries, our potential users, may have of the prototype service.

It was therefore clear that the project would need to be careful to not over-sell the service; making it clear this project is about collaboration and a large amount of learning as we progress (both the project and the libraries).Some of the possible ways to address these challenges included:

  • Expect some failure in certain areas – a complex project like this may mean not everything will work as expected;
  • Logging and learning as we go, and seek help from institutions/CAP group.
  • Guest blog posts from the community group (maybe around each of the categories identified).

Usability

The project will need to expend considerable energy on understanding user requirements; testing the prototypes with different user groups (librarians, reistrars etc).

This also means we need to be able to show users the prototype when it’s still rough and messy, so they have no qualms about providing critical and immediate feedback.

Fortunately we have our Community group to help us test the prototypes and to constantly challenge our assumptions and ideas.

Legal and Ethical

Legal and ethical issues were another significant concern that emerged during the premortem.

Many of the issues revolved around being able to reassure institutional registrars and CIOs about the way the data will be used, and ensure there is no possibility of damage to institutional reputations.

In many ways this is a subtle problem, requiring the project to deal with legal, ethical and reputational issues.

Some possible ways to address these problems included:

  • Use Jisc Legal: Discuss potential issues associated with the project and develop some pre-emptive resources and guidance for institutions;
  •  Produce a legal ‘toolkit’ for institutions and libraries – this might include advice and guidance as well as best practice.

Finally, there was a suggestion that the project, or rather the prototype service, provide the ability for institutions to ‘opt-out‘. This might be an out out clause in any agreement, that also makes it clear how libraries can disengage from the service and what happens to their data – how it is given back to them.

This is an interesting issue, and reminds me of the ‘right to be forgotten’ debate, and is critical legal and ethical issue for the project to consider.

Commercial

This particular concern is not about things like competitive advantage (the project is very clear that it is meeting a need that falls outside the ability of commercial vendors to meet – an explicit principle of the project is to not duplicate existing product functionality).

Rather, the project needs to ensure it is aware of vendor developments for reasons of interoperability and the possibility of additional functionality for existing systems.

It will be important that LAMP’s API can feed into commercial vendor products. 

Cost and Complexity

This is a critical issue for institutions: The benefits of the service must outway the costs of participation.

Initially, as the prototype is developed the balance of benefits may be outweighed by the challenges of providing the project with data: The complexities of engaging are largely borne by the institutions.

But this will have to rapidly evolve, so that the service is able to absorb much of this complexity and make institutional engagement simple and worthwhile.

Ways the project can start to address this concern includes:

  • Develop some best practice and guidance for participating institutions. Make it clear what they need to do and how (a LAMP manual!);
  • Tools for making the submission of data simple – the service should do the heavy-lifting for institutions;
  • Where possible, link to other institutional systems and data services, or enable these links to be made as easily as possible;
  • Clearly articulate the benefits for the participating institutions – almost a service level agreement (almost!). This might also be done through case-studies with some of the early adopter institutions.

Sustainability

This was a popular challenge for the project – unsurprisingly!

However, in a clever and possibly illegal move, we simply parked it with the following rationale:

Such a risk/challenge is almost always inherited by a project; it’s not simply going to go away. We can park this issue for now, and focus on those risks that are likely to blind-side us.

Of course, that’s not to say it’s not a critical issue that needs addressing. But we can keep in mind that this phase of the project is about demonstrating the feasibility of the prototype. Indeed, this feasibility phase may not succeed – which will require us to think carefully about how the project might be wrapped up or changed.

Doomsday

This is just a very brief overview of the issues and risks that surfaced during the premortem. The exercise was incredibly useful in providing the project with both the key challenges it needs to address, but also an opportunity to crowd-source some of the potential solutions and actions to address those issues.

What, at first glance, appears to be a slightly pessimistic and gloomy activity turned out to be a vibrant session with some useful concrete outcomes.

Having said that, there were one or two ‘doomsday’ scenarios described, including:

  • The Internet ‘goes down’ and there’s no way to get access to the service.

Fingers crossed this won’t happen – but it makes it clear we should double-check on our disaster planning protocols.

—————————————————————————————————————————————————————————————————————-

Two of the CAP group members also blogged about the meeting and the premortem exercise:

Paul Stainthorp (Lincoln): LAMP project: A lets pretend post-it note post-mortem

Richard Nurse (OU): The Pre-mortem

 

All about data normalisation

I missed the last CAP meeting and I’m really disappointed because it sounded great. I’m also sorry because it seems that there were a few questions about the approach we’re taking to data categorisation and normalisation. Since I’m the difficult person who is setting the parameters and making the demands on this, I thought perhaps I ought to explain myself in a blog post!

One of the big challenges for this project is reconciling the big set of things that institutions might want to do with their data, and the much smaller subset of things that are statistically acceptable. That’s not to say that institutions won’t be able to make their own decisions about how they analyse their data – quite the reverse, in fact – but we need to make sure that if they are doing statistical analysis it is done right. We have to pay attention to statistical principles, and this has big repercussions for the way that data is structured within the project.

Whenever you run a statistical test, you are basically trying to understand the relationship between two or more variables – for example, discipline and book borrowing, or country of origin and e-resource usage. Now, because we’re working with samples (in our case, a single year of students, rather than every student who has ever studied at a university, ever!) we have to accept that there might be a random blip in the data which means the relationship only exists in the sample (the year group), not the wider population (all students ever at the university). Significance testing allows you to say how confident you are that any relationships you find do exist within the wider population, not just within your particular sample. It does this by calculating the probability of you finding your result if the relationship did not exist in the wider population and spits out a number between 0 and 1. You compare this number to your ‘critical value’ – usually .05 in the social sciences – and if your result is smaller, your finding is statistically significant.

‘This is all very interesting’, you may be thinking, ‘but what on earth does it have to do with data normalisation?’ Allow me to explain!

Some of the tests that we are using need data that has been separated into groups: for example, to understand the relationship between discipline and library usage, you need to group your students into different disciplines. You can then look and see whether these groups have different levels of library usage. Let’s take a hypothetical example where you have three groups, A, B and C, and you want to see whether these groups have significantly different levels of e-resource use.

The first thing that you do is run the Kruskal-Wallis test, which tells you whether any of the groups have a level of e-resource use that is significantly different from any of the other groups. Crucially, it only tells you that this difference exists somewhere: it doesn’t tell you whether the difference is between A&B, A&C, or B&C – or, indeed, any combination of the above. So that’s not especially helpful, if you want to use the information to decide which of your groups needs support in learning how to use e-resources.

If you find that your Kruskal-Wallis test is significant, you then need to go on and run Mann-Whitney tests on the same data. You take each pair – A&B, A&C and B&C – and run three separate tests, to see whether there is a difference between the two groups in each pair. For reasons that I’m not even going to try to explain, when you run lots of Mann Whitney tests on the same set of data, you increase your risk of a Type I error, which is statistical jargon for thinking a relationship exists where it actually doesn’t. In this example, it would result in libraries spending a lot of time educating a group of people in e-resource use where, in fact, the group is already perfectly competent. Again, not particularly helpful!

To avoid Type I errors you apply a Bonferroni correction, which basically means dividing your critical value –.05 – by the number of tests you’re running, to give you a new critical value. So, in our A, B, C example, you would divide .05 by 3, giving you a critical value of .017. For your Mann-Whitney test on a single pair to be statistically significant, it now needs to spit out a number which is smaller than .017. If you had four groups and you wanted to compare them all, your critical value would be .008. That’s pretty small, and makes it much less likely that you’ll find a statistically significant relationship.

So this is where – finally! – we get to data categorisation. With LAMP, we want to maximise the chances of telling universities something useful about their datasets. So we need to keep our number of groups as small as possible. You can minimise the number of tests that you run by taking a control group for each variable – the biggest group, let’s say – and comparing all the others to it with Mann-Whitney tests, without comparing them to each other. But if you have six groups, this will still mean running five tests and therefore working with a critical value of .01. So we really want to keep the numbers of groups down.

In short, then: if we have too many groups, we starkly reduce our chances of finding a statistically significant relationship between variables. This would make LAMP much less useful to institutions which want to build an evidence base in order to develop their library services. So we need to be a bit prescriptive.

The situation is further complicated by the fact that one aim of LAMP is to aggregate data across institutions. If we’re going to do this, obviously we need everybody to be working with the same set of definitions: it’s no use one university using groups A, B and C if another is using D, E and F and another is using red, blue and green!

In principle, there’s no reason that universities shouldn’t run a separate analysis on their own data using different groupings, if that makes more sense for them. But they’ll still have to think about the number of groups they include if they want to get statistically significant results.

Another option we’re thinking about is allowing people to compare sub-groups within each group: so, for example, within group A you might have subgroups 1, 2 and 3, and within group B you might have subgroups 4, 5 and 6. You can use the same Kruskal-Wallis/Mann-Whitney procedure to compare groups 1, 2 and 3 and groups 4, 5 and 6: but – crucially – you can’t compare 1, 4 and 6, and you can’t compare all six groups with each other. This should be helpful with something like discipline.

I hope that clears things up a bit! If not, let me know in the comments and I’ll do my best to answer any questions…

LAMP Principles

As the project begins to engage with institutions and existing library systems vendors and services it’s important that we make it very clear what we plan to do with the data, and more broadly how the project will undertake its work.

With this in mind the team have come up with a set of principles by which the project will operate.

Let us know what you think and how we could improve them…

The following project principles are designed to ensure that the work of LAMP and its partners, contributors and contractors is aligned to all relevant legal and ethical frameworks.

These principles will help ensure:
• An understanding of the status of data provided by contributors and third parties to the project;
• Legal and ethical guidelines for the project, partners and contributors;
• Clarity on issues of competition and market differentiation.

Data Protection

The ownership of any and all raw data supplied to the project will remain under the ownership of the originating institution or organisation. Data can be taken down or removed at anytime upon request from the owner. At the completion of the project all data will be returned to the owner, or deleted by the project.

The project will ensure protection of data and confidentiality to persons and organisations through appropriate measures (such as anonymisation of records linked to individuals) in accordance with the Data Protection Act.

Commercial Confidentiality

In order to protect any commercially confidential data or information the project will seek to use other sources of openly available data, or ensure that this information and/or data is not made publicly available.

Access to the service will be via UK Access Management Federation, ensuring confidential data cannot be accessed outside of an institution.

Licensing and standards

The project will ensure its reports and technical specifications are licensed under an appropriate open license (such as Creative Commons) to encourage standardisation and reuse. All reports will be made available via the project website.

LAMP will look to adopt and implement existing technical standards and make use of structured data principles to facilitate interoperability with other systems.

Ethics

Wherever the project handles data that pertains to the analysis of learning and research, it will act in accordance with ethical principles that treat the wellbeing and interests of the individual as paramount and as the basis for the good reputation of the sector and its institutions. [Link to Legal and Ethical framework for the project].

Development

The ongoing development of LAMP will be driven by engagement with the UK library community.

The prototype service will look to add value to existing institutional systems and services through the possibilities of data aggregation and benchmarking. It will not look to duplicate the functionality of existing systems or services where the market is functioning and healthy.

Participation

LAMP has convened a Community Advisory and Planning Group to ensure the project meets the requirements, values and aspirations of the UK academic library community. The project also has a website and blog which will regularly be updated with new information.

The project is also directly working with six institutions who are supplying data to the project. A full list of the participating institutions can be found here.

As LAMP progresses and prototypes are developed, the project will plan ways of engaging wider input and feedback from the wider library community, including: International libraries and commercial suppliers.

The prototype service will be available to the UK academic community upon its release in December 2013.

Library Analytics – Community Survey Results

The team is currently prepping for our first Community Advisory Board (CAB) meeting for the Jisc LAMP project. There’s a great deal to discuss, not least the use case ideas we have been drafting for feedback.  Ben Showers and I met last week to talk about setting the context for the meeting, and we agreed that it would be useful to more broadly share the findings of the survey we ran back in November 2012.  With the support and input of RLUK and SCONUL, Mimas worked with Jisc to run a community wide survey. We wanted to gauge the potential demand for data analytics services that could an enhance business intelligence at the institutional level and so support strategic decision-making within libraries and more broadly.  Below is a summary of the results available through slideshare.

Library Analytics – Community Survey Results (Nov 2012) from joypalmer
We wanted to get a better handle on how important analytics will be to academic libraries now and in the future, and what demand might be for a service in this area, for example, a shared service that centrally ingests and processes raw usage data and data visualisations back to local institutions (and this, of course, is what LAMP is exploring further in more practical detail).  We had response from 66 UK HE institutions, and asked a good number of questions. For example, we asked whether the following functions might be potentially useful:
  • Automated provision of analytics demonstrating the relationship between student attainment and resource/library usage within institutions
  • Automated provision of analytics demonstrating e-resource and collections (e.g. monographs) usage according to demographics (e.g. discipline, year, age, nationality, grade)
  • Resource recommendation functions for discovery services

Perhaps not surprisingly, the overwhelming response was positive – these tools would be valuable, yes (over 90 % ‘yes’ rate each time). But we also asked respondents to indicate which strategic drivers were informing their responses, i.e. supporting research excellence, enhancing the student experience, collection management, creating business efficiencies, demonstrating value for money, and others. What we found (based on our sample) was that the dominant driver was ‘enhancing the student experience,’ closely followed by the ability to demonstrate value for money, and then to support research excellence.

We also asked whether institutions would find the ability to compare and benchmark against other institutions would be of value. Whilst there was general consensus that this would be useful, respondents also indicated a strong preference to share data to be used as a benchmark for other institutions if it were anonymised and made available by a category such as Jisc Band (91%) (This compared to a 47% ‘yes’ rate when asked if they would, in principle, be willing to make this data available where users could see the source institution’s name).  So, there is appears to be a strong willingness to share business intelligence data with the wider community, so long as this is done in a carefully managed way that does not potentially expose too much about individual institutions. In addition, there was far more hesitation over sharing UCAS and student data than other forms of transactional data (again, not surprising).

Are analytics a current strategic priority for institutions?  Only nine respondents said yes it was a top priority at the present moment, with 39 stating that it was important but not essential. However, when asked whether it would become a strategic priority in the next five years, 40 respondents indicated it would become a ‘top priority.’

However, the question of where the decision-making in this area would reside evoked a wide range of different responses, indicating the organisational complexities we’d be dealing with here. Clearly the situation at each institution is complex and highly variable. Overall Library Directors and IT Directors are seen as the key decision-makers, but respondents also referenced Vice Chancellors, Registrars, Deputy Vice Chancellors. At certain individual institutions, the University Planning Office would need to be involved, or at another, the Director of Finance.

Other potential barriers to sharing include concerns over data privacy and sharing business intelligence, and our results revealed a mixed picture in terms of concerns over data quality, lack of technical expertise, and the fact that there are strong competing demands at the institutional level.

The LAMP project is now working to build on these findings and develop live prototypes to fully test out these use cases, working with data from several volunteer institutions.  Our major challenge will be to ascertain to what extent the data available can help us support these functions, and that’s very much what the next six months is going to be focused on.