The Real Reason Hadoop Is Such A Big Deal In Big Data

Hadoop is the poster child for Big Data, so much so that the open source data platform has become practically synonymous with the wildly popular term for storing and analyzing huge sets of information.

While Hadoop is not the only Big Data game in town, the software has had a remarkable impact. But exactly why has Hadoop been such a major force in Big Data? What makes this software so damn special - and so important?

Sometimes the reasons behind something success can be staring you right in the face. For Hadoop, the biggest motivator in the market is simple: Before Hadoop, data storage was expensive. 

Hadoop, however, lets you store as much data as you want in whatever form you need, simply by adding more servers to a Hadoop cluster. Each new server (which can be commodity x86 machines with relatively small price tags) adds more storage and more processing power to the overall cluster. This makes data storage with Hadoop far less costly than prior methods of data storage.

(See also Hadoop: What It Is And How It Works.)

Spendy Storage Created The Need For Hadoop

We're not talking about data storage in terms of archiving… that's just putting data onto tape. Companies need to store increasingly large amounts of data and be able to easily get to it for a wide variety of purposes. That kind of data storage was, in the days before Hadoop, pricey.

(See also Hadoop Adoption Accelerates, But Not For Data Analytics.)

And, oh what data there is to store. Enterprises and smaller businesses are trying to track a slew of data sets: emails, search results, sales data, inventory data, customer data, click-throughs on websites… all of this and more is coming in faster than ever before, and trying to manage it all in a relational database management system (RDBMS) is a very expensive proposition.

Historically, organizations trying to manage costs would sample that data down to a smaller subset. This down-sampled data would automatically carry certain assumptions, number one being that some data is more important than other data. For example, a company depending on e-commerce data might prioritize its data on the (reasonable) assumption that credit card data is more important than product data, which in turn would be more important than click-through data.

Assumptions Can Change

That's fine if your business is based on a single set of assumptions. But what what happens if the assumptions change? Any new business scenarios would have to use the down-sampled data still in storage, the data retained based on the original assumptions. The raw data would be long gone, because it was too expensive to keep around. That's why it was down-sampled in the first place.

Expensive RDBMS-based storage also led to data being siloed within an organization. Sales had its data, marketing had its data, accounting had its own data and so on. Worse, each department may have down-sampled its data based on its own assumptions. That can make it very difficult (and misleading) to use the data for company-wide decisions.

Hadoop: Breaking Down The Silos

Hadoop's storage method uses a distributed filesystem that maps data wherever it sits in a cluster on Hadoop servers. The tools to process that data are also distributed, often located on the same servers where the data is housed, which makes for faster data processing.

Hadoop, then, allows companies to store data much more cheaply. How much more cheaply? In 2012, Rainstor estimated that running a 75-node, 300TB Hadoop cluster would cost $1.05 million over three years. In 2008, Oracle sold a database with a little over half the storage (168TB) for $2.33 million - and that's not including operating costs. Throw in the salary of an Oracle admin at around $95,000 per year, and you're talking an operational cost of $2.62 million over three years - 2.5 times the cost, for just over half of the storage capacity.

This kind of price savings mean Hadoop lets companies afford to hold all of their data, not just the down-sampled portions. Fixed assumptions don't need to be made in advance. All data becomes equal and equally available, so business scenarios can be run with raw data at any time as needed, without limitation or assumption. This is a very big deal, because if no data needs to be thrown away, any data model a company might want to try becomes fair game.

That scenario is the next step in Hadoop use, explained Doug Cutting, Chief Architect ofCloudera and an early Hadoop pioneer. "Now businesses can add more data sets to their collection," Cutting said. "They can break down the silos in their organization."

More Hadoop Benefits

Hadoop also lets companies store data as it comes in - structured or unstructured - so you don't have to spend money and time configuring data for relational databases and their rigid tables. Since Hadoop can scale so easily, it can also be the perfect platform to catch all the data coming from multiple sources at once.

Hadoop's most touted benefit is its ability to store data much more cheaply than can be done with RDBMS software. But that's only the first part of the story. The capability to catch and hold so much data so cheaply means businesses can use all of their data to make more informed decisions. 

BRIC nations lag in cloud computing: Study

Brazil, Russia, India and China still lag far behind developed countries in policies considered critical for the future of cloud computing, but each made some progress over the past year, a US industry group said.

The Business Software Alliance, which represents US industry heavyweights such as Microsoft, said the BRIC nations all came in at the bottom half of 24 countries surveyed in its second annual cloud computing report.

Brazil moved from final position to 22nd with a tally of 44.1 out of a possible 100 points.

China, India and Russia each also rose two slots with scores of 51.5, 53.1 and 59.1, respectively.

Cloud computing refers to providing software, storage, computing power and other services to customers from remote data centers over the Web.

Demand for cloud-based software is rising rapidly because the approach allows companies to start using new programs faster and at lower cost than traditional products that are installed at a customer's own data center.

"The cloud is really the hot sector of IT right now," and US companies have a big interest in countries harmonizing policies instead of chopping the cloud into pieces, said Robert Holleyman, president of the Business Software Alliance.

At the same time, the aggregation of massive amounts of data in large data centers "creates new and highly tempting targets" for cyber attacks, making it vital that both law enforcement officials and cloud providers have adequate tools to fight the intrusions, the BSA report said.

"Australia, France, Germany, and Japan score extremely highly in the cybercrime section. Canada, China, (South) Korea, Russia, and Vietnam score poorly. The country that shows the most improvement is Brazil, which finally passed cybercrime laws after a long campaign," the report said.

The 24 countries included in the survey represent 80 per cent of the global information and communications technology industry. They were assessed in seven areas, including data privacy, security, free trade, intellectual property protection, infrastructure and support for industry-led standards to promote smooth data flows.

China got a small boost in this year's rating for introducing new data privacy laws, while Russia got credit for reforms made as a result of its entry into the World Trade Organization. India's improved score reflects changes to its copyright laws to bring them in line with international standards, the report said.

Japan came in first again with 84.1 points. It was followed closely by other developed countries, including Australia, the United States, Germany, Singapore, France, Britain and South Korea, which all scored in the upper 70s.

Singapore jumped to fifth place, from tenth last year, after it passed a new data privacy law praised by BSA for its "light touch" and balanced approach.

"They are really taking on digital trade as another way of putting a stake in the ground and to say they are going to be global hub of business," Holleyman said.

The United States finished second in the survey, up from third in the inaugural report, while Germany, France and Britain each slipped a notch and Italy fell four spots.

Holleyman said the European Union was working on data protection regulations that could potentially make it harder to move data across its borders.

"If that happens I think you can continue to see further sliding by the major European countries," Holleyman said.

Talks on a US-EU free trade agreement are expected to start by June, he said.

Cross-border data flows are already a focus in talks on the Trans-Pacific Partnership (TPP), a proposed regional free trade agreement between the United States and ten other countries in the Asia-Pacific slated for conclusion this year.

One of the TPP countries, Vietnam, finished last in this year's cloud computing scorecard, with a tally of 40.1 points.

Vietnam, Indonesia, China and India have pursued policies that threaten to divide the cloud, either by trying "to wall themselves off or by imposing local requirements that are antithetical to the very underpinning of cloud computing," Holleyman said.




Source- http://timesofindia.indiatimes.com/tech/enterprise-it/infrastructure/BRIC-nations-lag-in-cloud-computing-Study/articleshow/18845086.cms

Govt allows IT SEZs to set up backup centres anywhere in India

The government has allowed IT and ITeS special economic zones to set up disaster recovery centres outside their limits at any part of the country, meeting the long-pending demand of the industry.

Issuing the guidelines for setting up of disaster recovery centres (DRC) and business continuity plan (BCP) for IT/ITeS special economic zones, the Commerce and Industry Ministry said the locations for such facilities will be approved by the respective development commissioner.

"The DRC/BCP location will be approved by the development commissioner (DC) on an application made by the SEZ unit. Such approval will allow the unit to relocate its operations, data and employees to the DRC/BCP location upon the occurrence of a disaster," it said.

However, it said that as this activity is envisaged as a purely internal exercise to be carried out across branches of the same SEZ entity to ensure that business continuity, there will be no commercial activity involved and accordingly, no commercial invoice will be raised in such movement of data, operations and employees.

"It was a long pending demand of the industry. IT/ITeS SEZs need such facilities at the time of any type of disasters. It will certainly help the sector," an official said.

Prevention and creating data back up is an integral part of the sectors' DR/BCP strategy.

"The data are regularly backed up at locations which are isolated from the main business centres to prevent loss in the event of a disaster. This would entail movement of data from SEZ to a DR/BCP location outside the SEZ and movement of storage media back into the zone," it said.

It also said that movement of data from outside the zone would not be treated as exports besides a record of movement of magnetic, storage tapes and devices would be maintained at the tax free enclave.

However, the unit would have to pay necessary duty on the tapes and storage tapes on which the data is being moved.

It said that the back up location where the "devices are moved could be a location under another SEZ or export oriented units i.e. a bonded secured location".

Further, the guidelines have comprehensively defined the term 'disaster' and has classified it into categories - natural and manmade. The manmade disasters include hazardous material spills, infrastructure failure or bio-terrorism.

It has also provided norms for setting up of these centres by a third party client.

The move assumes significance as out of over 160 operational SEZs, about half of them relate to IT/ITeS SEZs.




Source- http://timesofindia.indiatimes.com/tech/enterprise-it/infrastructure/Govt-allows-IT-SEZs-to-set-up-backup-centres-anywhere-in-India/articleshow/18672137.cms

Mobile industry to employ 10 million globally: Report

 The mobile industry will invest $1.1 trillion by 2017 and the ecosystem around it is expected to employ 10 million people globally, said a report released by global industry body GSM Association said.

"For the period through 2017, the mobile industry will invest $$2.6 trillion to public funding. Importantly, in 2017, companies across the ecosystem will employ nearly 10 million people globally," 'The Mobile Economy 2013' report prepared Developed by GSMA and consulting major AT Kearney said.

The report said revenue from total mobile ecosystem revenues reached $1.6 trillion -- around 2.2 per cent of the global Gross Domestic Product ( GDP).

"To fully realise this future and to enable the mobile industry to maximise its investments, it is essential that we establish a light-touch regulatory environment, based predominantly on competition, and develop new business models that will allow all ecosystem participants to benefit from the mobile economy," GSMA Director General Anne Bouverot said.

The report said it expects a further 700 million subscribers will be added by 2017 and the 4 billion-subscriber milestone will be reached in 2018 across the globe.

At the end of 2012, there were 6.8 billion mobile connections worldwide and the study expects it to grow to 9.7 billion by the end of 2017.

High speed internet on mobile phone accounted for 1.6 billion of these connections in 2012, increasing to 5.1 billion in 2017, including 920 million LTE connections, the report said.

Mobile subscriber penetration globally stood at 45 per cent while mobile connection penetration is currently 94 per cent.

As per GSM Association Wireless Intelligence, the variance between the number of mobile subscribers and the number of mobile connections is related to multiple sim ownership as well as inactive sims.



Source-http://timesofindia.indiatimes.com/tech/careers/job-trends/Mobile-industry-to-employ-10-million-globally-Report/articleshow/18676659.cms

HP's webOS operating system to power LG TVs

Hewlett-Packard Co said it will sell the webOS operating system to South Korea's LG Electronics Inc, unloading the smartphone software it acquired through a $1.2 billion acquisition of Palm in 2010.

LG will use the operating software, used in now-defunct Palm smartphones years ago, for its "smart" or Internet-connected TVs. The Asian electronics company had worked with HP on WebOS before offering to buy it outright.

Under the terms of their agreement, LG acquires the operating software's source code, associated documentation, engineering talent, various associated websites, and licenses under HP's intellectual property including patents covering fundamental operating system and user interface technology.

HP will retain the patents and all the technology relating to the cloud service of webOS, HP Chief Operating Officer Bill Veghte said in an interview.

"As we looked at it, we saw a very compelling IP that was very unique in the marketplace," he said, adding that HP has already had a partnership with LG on webOS before the deal was announced.

"As a result of this collaboration, LG offered to acquire the webOS operating system technology," Veghte said.

Skott Ahn, President and CTO, LG Electronics, said the company will incorporate the operating system in the Smart TV line-up first "and then hopefully all the other devices in the future."

Both companies declined to reveal the terms of the deal.

LG will keep the WebOS team in Silicon Valley and, for now, will continue to be based out of HP offices, Ahn said.

HP opened its webOS mobile operating system to developers and companies in 2012 after trying to figure out how to recoup its investment in Palm, one of the pioneers of the smartphone industry.

The company had tried to build products based on webOS with the now-defunct TouchPad tablet its flagship product.

HP launched and discontinued the TouchPad in 2010, a little over a month after it hit store shelves with costly fanfare after it saw poor demand for a tablet priced on par with Apple's dominant iPad.

WebOS is widely viewed as a strong mobile platform, but has been assailed for its paucity of applications, an important consideration while choosing a mobile device.




Source-http://timesofindia.indiatimes.com/tech/tech-news/hardware/HPs-webOS-operating-system-to-power-LG-TVs/articleshow/18686985.cms

Indian IT must cross Japan hurdle for next $100 billion: Experts

As the Indian IT industry crosses $100 billion (about 5.4 lakh crore) in revenues and aims for the next $100 billion, it cannot afford to ignore Japan, the world's largest IT market after the US. Most large Indian information technology providers have been present in Japan for close to 20 years but success has been slow in coming.

Of the $125-billion Japanese IT services market, Indian service providers get only $500 million. Embedded services contribute another $500 million, according to technology researcher Gartner. In all, Japan contributes less than 2% of India's software exports.

"If we fix the language and culture issues, growth will happen," said N Chandrasekaran, Nasscom chairman and chief executive of Tata Consultancy Services.

For instance, India's fifth-largest software exporter HCL Technologies trains all its employees working in Japan to speak and understand Japanese.

But even bigger is the cultural barrier. Unlike their western counterparts, companies in Japan do not do big bang outsourcing. They initially look for proof of concept, and if that works and they are comfortable with the vendor, then the relationship progresses to the long term, said Sameer Kishore, corporate vice president who heads the Japan business unit for HCL Technologies.

"From my experience of working in this market for a fair bit of time now, organisations here value relationships," said V Sriram, senior vice president and head of Japan business for Infosys.

Attitudes in Japan are also changing, forced by the rapid pace of technology changes. "The battlefield is shifting to software," said Nobuhiko Hidaka, president of Gartner Japan. Companies, which previously followed a 'rice-farmer culture of doing everything same every year' and were more inward-looking, are now changing as they globalise.

Hidaka said most applications used to be custom-built, but as Japanese companies go global, custom-built applications are being replaced by more standard packages. "Because attitudes are changing and Indian providers are winning in the global IT market, the door is open for the first time for India." In addition, the CEO is getting younger and chief information officers more westernised.

US IT providers, which have a 14% share of the Japanese IT services pie, are moving in to tap this opportunity. Firms like IBM, Accenture and HP-EDS are well-entrenched compared with the Indian providers. However, most of their operations are staffed by Japanese locals and done out of Japan.


Source- http://timesofindia.indiatimes.com/tech/tech-news/software-services/Indian-IT-must-cross-Japan-hurdle-for-next-100-billion-Experts/articleshow/18670455.cms

Infosys’ new platform pulls big data 40% faster

 Infosys on Wednesday formally launched what it says is one of the most comprehensive solutions in the big data space.

The solution, called BigDataEdge, allows enterprises to easily bring together not just the organized or structured data, but also a vast variety of unstructured data (information contained in, say, emails, document files, contracts with customers or vendors, blogs, social media, call centre voice records, videos). It then enables them to glean insights from all of this data, and take appropriate action.

One major element of the solution is a patent-pending connector framework, which automatically connects to internal and external data sources, including any new source that emerges, and which then makes pulling data together very easy.

"We have been able to reduce the time to discover and aggregate data by up to 40%," says Vishnu Bhat, head of Infosys' cloud and big data business. He says that in the case of a financial service provider, the solution was able to uncover hidden exposures in 43% of their accounts by going through all the written contracts. "Earlier, this would at best be a manual process that took many months. Now you can do it in days or weeks," he says.

The solution can convert voice calls into text to find necessary information. It uses facial recognition and similar technologies to extract information from videos.

Enterprises can then use built-in algorithms (there are some 250 of them) to obtain the insight required from a desired set of data sources, and visualize the insight using some 50 customizable dashboards. "We are able to generate insights eight times faster than is normal for enterprises," Bhat says. The solution also includes a collaboration tool that allows users across functions and regions to interact on the insights and take decisions in real time.

Bhat says the solution can even be used for specific requirements such as fraud detection or predicting network failures with its ability to match current records with historical records. For instance, people have a certain pattern of usage of their bank account. If there is a change in that pattern (because of an online fraudster), the solution quickly recognizes that and can send an alert or temporarily lock the account.

BigDataEdge is the latest in Infosys' Edge series of platforms that also includes, among others, WalletEdge, the mobile payments platform, and BrandEdge, which addresses marketing needs.


Source-http://timesofindia.indiatimes.com/tech/enterprise-it/infrastructure/Infosys-new-platform-pulls-big-data-40-faster/articleshow/18603556.cms