Big data processing and analytics

Today’s processing power allows for the analysis of big data to find hidden correlations, consumer information and other business-relevant insights. With developments in cloud services, the capabilities of data analysis are increasing at the same time as progress in the field of artificial intelligence such as deep learning (The Economist, 2016). A number of technologies make big data processing possible:
1.  Cloud computing —provides significant computing scale without huge capital investment.
2.  Hadoop – a framework for running applications on a large cluster of commodity software.
3.  MapReduce – runs on Hadoop and reduces large applications into smaller elements of work.
4.  Logical data warehouse – data can be stored in many disparate places; shopping data may be separate from purchasing data. The logical data warehouse would see and treat these data sites as one, making analysis much easier.
5.  NoSQL –provides an excellent database structure for big data processing.
6.  In-memory databases: To achieve maximum performance, database processing is best done with data already in the computer’s memory, which cuts out the time it takes the computer to read data off a disk. Machine learning allows computers to crunch vast amounts of data, recognise patterns – including real time image recognition of shared photos – and get better at what they do with experience. This has the potential to disrupt many aspects of consumer purchases.
Companies across the various travel and transportation industry segments, such as airlines, airports, railways, freight logistics, hospitality and others, have been handling large amounts of data for years, but until recently have not always had the capability to turn that raw data into insights.
In today's instrumented and interconnected world, unprecedented amounts of data are captured from almost every kind of system or event - and much of it is unstructured data. From passenger name records, transaction history and pricing data to customer feedback surveys, call centre logs
and Twitter feeds, travel and transportation companies now have access to a lot of data on all aspects of their business operations and customer interactions.
At the same time, consumers are becoming smarter shoppers with a variety of choices via various channels - phone, web, kiosk, counter, 3rd party agency, etc. Travellers are becoming more demanding in the quality and variety of services available, as a result of rewards systems or loyalty programmes. The "empowered," or "always connected," consumer has particularly profound implications for the way that the travel and transportation industry manages travellers, requiring new approaches in responding to greater expectations of (for example) being kept informed and able to amend travel plans at short notice.
In order for travel and transportation organizations to capitalize on these and other challenges in the industry, they need ways to collect, manage and analyse a tremendous volume, variety, velocity and veracity of data. Organizations who can tackle the big data challenge will differentiate from competitors, gain market share and increase revenue and profits with innovative new services.
Currently, the travel and transportation industry as a whole is perceived as generally lagging behind other sectors in terms of how data is used, according to a recent IBM Center for Applied Insights study [1]. According to the study, for example, travel and transportation companies are surprisingly behind in predictive modelling, simulations and next best action modelling – all areas necessary to run operations efficiently and target customers effectively. According to an article [2] in Engineering News Record, a survey by a big data analytics company specializing in transportation planning underscored this picture of sluggish adoption of technology and age bias towards the potential of new technology.
This represents a huge opportunity for travel and transportation companies to derive more insights from their data, according to a recent Forbes article [3]. The article presents some examples of ways in which the travel and transportation industry can create value from big data and analytics are shown below.
For Customer analytics and loyalty marketing, data analytics has the potential to help companies create a comprehensive 360-degree view of the customer, dramatically improving customer interaction at every touch point and across the end-to-end passenger or traveller experience. This enables greater personalization and relevance to increase marketing effectiveness, improve customer service and drive loyalty.
The ability to analyse more historical information with higher frequency — in near real time — allows for more dynamic and smarter pricing actions, optimized capacity planning and effective yield management.
Predictive, proactive and sensor-based analytics can help companies achieve operational efficiencies and improve performance outcomes. By capturing and analysing more complete operational data, big data and analytics can help organizations manage and maintain their assets to improve safety, performance and equipment life. This enables asset optimization to shift from reactive repairs to preventative maintenance. Total cost of asset ownership is reduced, asset life and operational capacity is increased, and on-time performance is improved.
The travel and transportation industry is now seeking to emulate other sectors in deriving greater insights from the newly-available and manageable data it now has access to. In an era in which it is possible to predict a new retail trend or customer retention patterns for cellular phones, similar types of analytics are providing insights that can predict flight delays and better understand customer decisions and behaviours. For example, as reported in another recent Forbes article 96], IBM is working with Lufthansa on a research project to develop automated systems to learn and interact naturally with airline operators so they can more precisely respond to weather conditions, minimizing the ripple effect of flight disruptions. The goal is a more proactive flight scheduling system to help boost operational efficiency and customer service. It could also help avoid costly downtime, and reduce maintenance and services costs for the airline.
In summary, areas where big data analysis could apply to travel distribution include:
1.  Providing real-time transparency of operational data: enabling companies to see look-to-book ratios, responses to new promotions, reaction to new user interfaces, performance against plan, individual agent conversion rates, online abandonment, call center talk time, volume of help calls referred to a higher level, user satisfaction, and response to crisis events.
2.  Determining relevant content to be displayed based on stage of the trip, location, device, psychographic profile, purchase and conversion histories (of a specific traveller or ones with similar profiles), and purpose of the trip.
3.  Leveraging location based services: Location based services can be used for route optimization, understanding traveller behavior, developing new transportation facilities and tourist venues, and reducing gas consumption. 
4.  Defining the next generation of travel products and services based on behaviors throughout the travel distribution value chain. 
5.  Providing real-time operational traveller support services throughout the integration of the Internet of Things– real-time sensors that monitor the environment, transportation facilities, hotel rooms, etc.
6.  Improving operations management: correlating service and preventative and restorative maintenance to reduce cost and increase efficiency, particularly in aircraft maintenance.

[1] Inside the Mind of Generation D: What it means to be data-rich and analytically driven. IBM Center for Applied Insights, 2014. 
[2] Drawing Transportation Planning Inferences From Big Data, by Tom Sawyer. ENR: Engineering News-Record; 9/22/2014, Vol. 273 Issue 8, p1 
[3] Why Big Data Means Big Opportunity For The Travel Industry, by Shannon Adelman. – Forbes, 9 July 2015