• Follow us


How to build an agile data pipeline

Agility and data are two of the most overused buzzwords of the business community – and for good reason.

Every business wants to be agile, to be responsive to the changing environment, to survive and thrive. Likewise, forward-thinking businesses are majorly focused on data as a route to greater insights, creativity and efficiency. It seems buzzword squared to put these two concepts together, but rather than being a technology to hype, it refers to a smarter way of managing with what enterprises already have, or with readily acquired skills.

An agile data pipeline is what data-centric organisations are putting in place in order to make the best use out of their data investments and ensure that the business can incorporate data-led analytical decision-making in a healthy and sustainable way.

As with any business process, building an agile pipeline involves several stages and should properly encompass a range of appropriate stakeholders within the business. As it is, that’s not always the case as many organisations tend to develop their analytics functions in a higgledy-piggledy manner.

It’s no surprise that the data estate of a business can quickly grow out of control – the four Vs of big data, as defined by IBM are the variety, velocity, volume, and veracity of big data and show that data is no monolithic thing. It’s a living, changing entity. So fluid in fact, that in 2017 Experian built on this format and added two more Vs: Vulnerability and value.

So how do you corral and harness the bucking bronco of data and put it behind the corporate plough, to turn up the nuggets of true insight?

A data catalogue makes storing, finding and using data a much more seamless experience. It’s an organised solution that allows business users to explore data sources and understand them. It saves the user time and can stop them recreating new data if they might have failed to find what they wanted in a non-catalogued state. It’s a great resource to keep the analytical process ticking over at speed, without slowing down the work of data scientists or ‘line of business’ analysts.

A faultless data catalogue doesn’t arrive fully formed, and the history of data governance integrations is littered with solutions that have failed to achieve a critical adoption in an organisation. To truly deliver on a data catalogue the business must also focus on the people and the process, not just the technology. Analytic leaders must build a culture that enables users to succeed with data.

Discover together

Data discovery can be fun, but it’s a hygiene factor that the analyst needs to get through before they can do the job they want to: Analytics, insights, and adding value to the business. Really, the organisation wants to unite all of the data workers with the data and analytic assets they could possibly (but legitimately!) need in a controlled and secure way. It’s important to take steps to make data both searchable and trackable. A platform will offer this and event data lineage, offering more visibility for better governance. When data discovery and data security are breathtakingly easy, there’s no room for data governance missteps. It’s a great first step before an enterprise can create a culture of collaboration, sharing, and innovation by extending formally tribal knowledge across the organisation.

Culture the data culture

The data catalogue is the starting point for most analytical activities. Searching and finding content, understanding context and gaining trust in the results through community feedback and interaction – it’s a great resource when it’s used correctly, saving time and energy, and greatly aiding productivity.

The success of the catalogue is tied into the success of the organisation. Track and reward the most active contributors who add value to the analytic process, understand the assets that are creating the most impactful results, and promote those users to ensure that information assets are well curated and maintained.

The right data culture is socially engaging. It empowers users to impart and share knowledge, and is supported by technology that supports the different ways that users bring their experience together to solve problems. This includes creating and annotating definitions, discussing quality and purpose in conversation threads, and even simple social gestures like sharing a link or giving a 'thumbs up' reinforce the value of the underlying asset and make it richer and easier to find for future users.

Collaborate or die!

It might be that during the course of the pre-data-focused days others in the organisation have already collected the same information or performed a similar analysis, but different analysts have no good way of finding it. Data assets and resulting information proliferate, thus compounding the problem and creating inefficiencies and delays in answering critical business questions.

Taking a cue from social media and wiki techniques, social interactions can help users share and utilise organisational tribal knowledge easily. And everything in the analytic process: Data, analytic apps, workflows, macros, visualisations, and dashboards, should be sharable. When everything is seamlessly shareable and it is fast work to identify trusted information assets as well as insights into how they are used and lineage, it’s very simple to make more impactful business decisions.

One of the most important pieces to this is closing the gap not only around finding the right data but around the roles within an organisation: Between IT, business analysts, data scientists, everyday ‘citizen data scientists’, and onwards to all who use data. Sharing across an organisation is the grease to the wheels of innovation.

Define the best working practices

From the moment you embark on analytics project you stand at a base camp with the peak of expectations staring at you from across the chasm of ignorance. Building a social repository of all the organisation's data sources, reports, workflows, terminology, and more (potentially thousands of lifetimes of accumulated knowledge) is as daunting as climbing Mount Everest. So, don't.

Start small, but think big. Tackle smaller challenges to get some early victories and build momentum from there.

Pick a single department or project. Perhaps start with a handful of critical datasetsDocument expertise while reports and data sources are being created, before the skills and the knowledge leaves the project (or the company!) Ensure that new people can understand the function of dashboards, reports other datasetsFollow your business strategy: Document and socialise the assets associated with key strategic projects, and use the catalogue as a means to change the culture towards greater collaborationTo ensure adoption, it’s vital that users find the information always up-to-date. Without timeliness, the catalogue immediately loses trust and credibility and the pipeline starts to leakA business glossary is a critical component of your data strategy. A glossary can take many forms: definitions, concepts, subject areas, etc. It captures the unique language of your organisation in a central location, and then connects that meaning with the contents of the catalogueA proper analytics pipeline lives-or-dies on whether users find value in the information within. There is no-one central to the organisation, not even BI and IT teams that have a 100 per cent understanding of all those data sources, data sets, and reports and other types of assets. This expertise and 'know-how' is in the heads of staff: Business teams, analysts, knowledge workers, analytics groups, and more. It's pervasive and waiting to be harnessedTrusting data

It’s one thing to have data, it’s another to trust it and use it properly. Famously executives relied on their experience, their ‘gut’, when making decisions, and sometimes, that’s not a necessarily a bad idea. Where data is not cleaned, rated and trusted, it might not be worth the time to review. But where the right steps are in place the data can tell a very honest and trustworthy story. It is a better resource than the thoughts and opinions of an executive who may not have access to all the facts, the long-term trends, or the powerful analytical ability to correlate all their contents appropriately.

So to stock the data pipeline put in place some simple best practices, encourage your people with good processes and give them the technology that makes this all easy. We’re not in the days of needing to know how code to operate analytical tools, and end-to-end platforms take out the sting of finding, moving, prepping and using data. In fact, stocking the analytics pipeline should be a breeze, exhilarating, process, the opening stages in a virtuoso performance by a data maestro.

Nick Jewell, Director of Product Strategy, AlteryxImage source: Shutterstock/alexskopje

Read More

Leave A Comment

More News

Latest ITProPortal news

Ryuk ransomware "still going strong" 2019-02-20 11:00:19Multiple groups still using Ryuk to extort money from companies.

Keep your business centre operations running 24/7 with 2019-02-20 08:00:40Reboot to restore solutions help IT admins take a preventive approach to computer management at business centres, thus enhancing the availability and

Microsoft uncovers major hacking attempts against EU organisations 2019-02-20 07:30:44Firms across Europe were hit in the attacks.

Qualcomm unveils most powerful 5G modem 2019-02-20 07:00:06Second-generation X55 modem will hopefully power the first 5G smartphones.

12 billion devices will be internet-connected by 2022 2019-02-20 06:30:28Up to four billion IoT devices will be online soon, Cisco estimates.

UK companies still worried about cyber risks 2019-02-20 06:00:38They fear 5G, but they're willing to invest.

Don’t let the tech takeover: Time rich, mindfulness 2019-02-20 06:00:22With today’s data-driven on-demand economy, we are winning back some of that precious time. But are we getting the most out of it?

The technology trust gap that’s hurting sales efforts 2019-02-20 05:30:02Here are my five key steps to get salespeople onboard with technology projects:

Why hackers love mainframe passwords – and what 2019-02-20 05:00:37Why are IBM’s mainframe customers seemingly reluctant to upgrade their security by incorporating multi-factor authentication?

Reflecting on data privacy for 2019 – Why 2019-02-20 04:30:11Below, six industry experts give their take on why data security needs to be at the heart of operations, and their opinions on what can be done to ens

Shipping on the cusp of a digital wave 2019-02-20 04:00:42Despite its significance, the industry still remains largely untouched by digital transformation and efficiencies it can bring.

Microsoft Surface Go review 2019-02-19 12:19:33An ideal pocket-sized budget work companion, but don't expect anything earth-shattering.

TechRadar: Internet news

These are the top 3 deals to go New! 2019-02-22 13:39:22Make sure you've chosen the right tariffs for your new Samsung Galaxy S10 deals - these are the top 3 pre-orders we've found.

Google to launch .dev domains New! 2019-02-22 13:18:29Google has launched a new TLD for developers looking for a new domain for their website.

Game Boy Advance: why it's the best way New! 2019-02-22 13:13:04We take a look at why the Game Boy Advance is becoming a popular way to play retro Nintendo titles.

Facebook shutters Onavo VPN app New! 2019-02-22 11:59:16Onavo VPN removed from Play Store after report Facebook used the app to spy on users.

Xbox Two: what we want to see out New! 2019-02-22 11:39:11When will we see Xbox Scarlett? What games will it have? We've done some digging and here's what we found.

The best cheap US TV deals and sale 2019-02-22 11:25:29We've scoured the net to compare prices and bring you the finest selection of cheap TV deals.

Best free and public DNS servers of 2019 2019-02-22 11:25:15Using an alternative DNS can have many benefits, particularly if you pick a good one.

Next Xbox alleged specs point to 2020 release 2019-02-22 11:18:45Leaked information on the next-generation Xbox consoles suggests they will be revealed at E3 2019.

Best robot vacuums 2019: the best robot vacuum 2019-02-22 11:03:21Tired of doing all the housework? Let these top-notch robot vacuums do all the dirty work for you.

A Japanese startup is set to go hunting 2019-02-22 10:46:05The Japan Airlines-backed venture is planning an Earth-Moon transport system starting next year.

Best Apple Watch screen protectors: our top picks 2019-02-22 10:17:54Is a thin film tough enough to protect your Apple Watch, or should you enclose it in a rugged case?

The high costs of storing data locally in 2019-02-22 10:00:25Blancco’s Fredrik Forslund explains why some businesses are reluctant to embrace the cloud over local storage.

TechCrunch » Enterprise

Mixmax brings LinkedIn integration and better task automation 2019-02-20 12:10:20Mixmax today introduced version 2.0 of its Gmail-based tool and plugin for Chrome that promises to make your daily communications chores a bit easier

Google’s hybrid cloud platform is now in beta 2019-02-20 12:00:41Last July, at its Cloud Next conference, Google announced the Cloud Services Platform, its first real foray into bringing its own cloud services into

New conflict evidence surfaces in JEDI cloud contract 2019-02-20 11:00:21For months, the drama has been steady in the Pentagon’s decade-long, $10 billion JEDI cloud contract procurement process. This week the plot thi

Arm expands its push into the cloud and 2019-02-20 09:00:43For the longest time, Arm was basically synonymous with chip designs for smartphones and very low-end devices. But more recently, the company lau

Xage brings role-based single sign-on to industrial devices 2019-02-20 09:00:27Traditional industries like oil and gas and manufacturing often use equipment that was created in a time when remote access wasn’t a gleam in an

Why Daimler moved its big data platform to 2019-02-20 06:00:50Like virtually every big enterprise company, a few years ago, the German auto giant Daimler decided to invest in its own on-premises data centers. And

Google acquires cloud migration platform Alooma 2019-02-19 12:18:48Google today announced its intention to acquire Alooma, a company that allows enterprises to combine all of their data sources into services like Goog

Slack off — send videos instead with $11M-funded 2019-02-19 09:44:52If a picture is worth a thousand words, how many emails can you replace with a video? As offices fragment into remote teams, work becomes more visual

GN acquires Altia Systems for $125M to add 2019-02-19 08:01:49Some interesting M&A is afoot in the world of hardware and software that’s aiming to improve the quality of audio and video communications o

Redis Labs raises a $60M Series E round 2019-02-19 08:00:35Redis Labs, a startup that offers commercial services around the Redis in-memory data store (and which counts Redis creator and lead developer Salvato

Senseon raises $6.4M to tackle cybersecurity threats with 2019-02-19 06:49:18Darktrace helped pave the way for using artificial intelligence to combat malicious hacking and enterprise security breaches. Now a new U.K.

As GE and Amazon move on, Google expands 2019-02-15 13:06:48NYC and Boston were handed huge setbacks this week when Amazon and GE decided to bail on their commitments to build headquarters in the respectiv

Digital Trends

Metro Exodus update brings DLSS improvements to Nvidia New! 2019-02-22 13:00:39Having issues in Metro Exodus? A February 21 update for the title recently delivered enhancements to Nvidia’s deep learning supersampling f

Motorola’s Moto G7 range offers compelling phones that New! 2019-02-22 12:39:07After a number of leaks and rumors, the Motorola Moto G7, Moto G7 Play, and Moto G7 Power are finally here. The devices represent quite a spec bump ov

The rumors were true. Nvidia’s 1660 Ti GPU, New! 2019-02-22 12:39:04Nvidia has officially launched the GTX 1660 Ti, its next-generation, Turing-based GPU. It promises to deliver all the performance and efficiency for a

Apple may be making noise-canceling headphones safer to New! 2019-02-22 12:32:34Over-the-ear headphones are massively popular, but they block a lot of outside sounds, which increases the risk of accidents. A new Apple invention hi

Destiny 2: Where to find Xur for the New! 2019-02-22 12:28:25The weekly vendor in Destiny 2: Forsaken always brings Exotic weapons and armor, some of the toughest loot to find in the game. Here's everything you

Kia is bringing a bionic-looking electric concept car New! 2019-02-22 12:15:34Kia wants to hog the spotlight at the 2019 Geneva Auto Show by revealing a head-turning electric concept car. The yet-unnamed model reaffirms the bran

The best guns in PUBG New! 2019-02-22 12:00:26Which weapons in PUBG are worth the time to scout out and fit with attachments? Which are going to help you become the last player standing? We've go

Waymo’s self-driving prototype obeys a traffic cop’s hand New! 2019-02-22 11:54:53One of Waymo's self-driving prototypes successfully navigated a situation that leaves even some human drivers confused: An intersection whose traffic

LeBron James’ Space Jam 2 gets an official New! 2019-02-22 11:46:56LeBron James has brought on Black Panther director Ryan Coogler to produce his upcoming Space Jam sequel, with Terence Nance attached to direct the fi

Everything you need to know about Nintendo Switch New! 2019-02-22 11:41:58If you want to play online multiplayer on Switch, you'll need a Nintendo Switch Online subscription. Here's what you need to know about Nintendo Swi

How to factory reset an Xbox One 2019-02-22 11:22:09Whether you're upgrading to a One X and giving your old console to a friend, or troubleshooting a technical issue, sometimes your Xbox One needs a cl

Kickstarter campaign aims to help make 3D-printed space 2019-02-22 11:06:04Mars X-House is an ambitious project that's intended to create a prototype future Mars habitat using 3D printing. And, thanks to a new Kickstarter ca

Disclaimer and Notice:WorldProNews.com is not responsible of these news or any information published on this website.