DH: Can you take a second and describe some of your roles over your career?
DL: For the last 25+ years my career has focused on data. I have done everything from developing data warehouse for telemarketing, to creating visualizations to identify cyber attacks on networks, to developing 10 year data strategies that would provide organizations the means to effectively use their data to achieve their mission. I have worked with many of the major tools at one point or another in areas of ETL, data stores, data quality, data governance, reporting, analytics and visualization. Of course, the technical side is easy. The most difficult role, and the most crucial is the person who can translate the business problem to the data. I have had the luxury of working with a variety of industries including telecommunication, public safety, judiciary, financial, healthcare, and agriculture. Each of these industries have a different perspective on data and what data is important.
DH: What are some of the challenges you have seen as it relates to data and data management?
DL: Let’s start with the basic - data quality. If you can't get complete, current, consistent, compliant and clean data, you do more harm than good.
The Lemony infographic on data quality identifies the startling statistics below:
The industry is making headway. Techniques and tools for Master Data Management, Quality Rules Engines, Data Cleansing, Data Governance and more are helping to improve the overall quality of data and reduce the impacts identified above.
DH: Are there industries that you have worked in that seems to have a greater challenges with data & data management as to others?
DL: I would use the term opportunities rather than challenges. Right now the two industries ripe for exploiting data to meet their missions is Healthcare and Finance. Both have tremendous amounts of data. Both have very high standards for the quality of data at the system of record (transactional system). There are hints of whats to come - Watson's use in Healthcare, the concept of population health, the use of data mining tools to prevent Fraud, Waste and Abuse, and the use of data to predict money laundering, etc.
DH: Most recently you worked for a large hospital group, what was your role and what was some of your biggest challenges?
DL: Hospitals are interesting from a data perspective.The introduction of the Electronic Medical Record has really forced healthcare providers to become data focused. Tactical and operational decisions are based on evidence from observations, monitors, lab results and radiology results. The challenge is to extend data to the strategic realm and using it to predict provide resolutions to problems.
Moving the use of data to that strategic realm, means incorporating different data together than was ever conceived before. Internal data (i.e., finance, workforce planning, supply planning, regulatory info,) used in conjunction with patient information can help improve patient care by focusing resources, identifying high risk patients and ensuring safety of both staff and patients.
Hospitals find many challenges in moving to that strategic realm. There are ethical challenges - how can/should data be used. There are security challenges - who should see the data. There are challenges in obtaining external data. There are challenges in integrating data from diverse systems with no apparent key. And of course, there are challenges of time and resources to do this.
DH: Have you seen in your experience upper management take a more active role in data & data management decisions?
DL: Some people get data and some don’t. That is also true with upper management. However, as it is becoming more and more obvious that data is a true asset to every organization, more upper management are trying to take an active role.
I have worked with some amazing leaders that truly did get it. These people understood the need for data driven decision making and pushed that down through the organization. As a result, their workforce was lean but effective, their vendor payments were on time and correct, their contract management was consistent; and they were able to achieve their mission more effectively.
DH: What are your thoughts around the promise of Hadoop?
DL: Hadoop is now in its second decade, and the promise is starting to be revealed. Although not the silver bullet to data, Hadoop has been the key to many data problems.
I find the data lake concept the most intriguing. The data lake is a holding area for all the data an organization collects. It is a solution for organizations that need to maintain lots of data for legal purposes; organizations that have data that may be needed in the future; organizations that need an initial “staging area” for data, but don’t want to lose the history. The use of Hadoop provides a write no structure/read structure is the perfect data lake.
DH: What do you think of the NO SQL database market? Can it replace the relational database market?
DL: No SQL and NonSQL databases are a very diverse. They include content databases intended for document storage, columnar databases intended for high performance computing, grid databases for network analysis and much more. At this point in time, I would not see the relation database market being replaced.
I look forward to moving into the NoSQL and NonSQL world. But, at this point in time, I would not see the relation database market being replaced. Basic reporting and analysis assumes data stored in a relational format. Many of the resulting analysis from No SQL databases are then moved to relational format so that reporting, slice and dice analysis and drill down analysis can be done. This redundance may not be needed for long as reporting tools (like Splunk) become more efficient at reporting on NoSQL and relational databases.