Google Dremel vs Apache Hadoop: Big Data Analytics Tools and Techniques

Google Dremel vs Apache Hadoop: Big Data Analytics Tools and Techniques

This is the article about some useful and popular real time big data analytics tools and techniques. Some of the big names are Apache Hadoop and Google Dremel. There are also other open source big data analytics tools and techniques like storm and apache S4. We will see the difference between Google Dremel and Apache Hadoop.

What is Google Dremel?

Google Dremel is a scalable, interactive ad-hoc query system for analysis of read-only nested data. By combining multi-level execution trees and columnar data layout, it is capable of running aggregation queries over trillion-row tables in seconds. The system scales to thousands of CPUs and petabytes of data, and has thousands of users at Google.

Apache Hadoop vs Google Dremel: Difference between Apache Hadoop and Google Dremel

Dremel is a data analysis tool designed to quickly run queries on massive, structured datasets (such as log or event files). It supports a SQL-like syntax, but apart from table appends, it is read-only. It doesn't support update or create functions, nor does it feature table indexes. Data is organized in a "columnar" format, which contributes to very fast query speed. Google's BigQuery product is an implementation of Dremel accessible via RESTful API.

Hadoop (an open source implementation of MapReduce) in conjunction with the "Hive" data warehouse software, also allows data analysis for massive datasets using a SQL-style syntax. Hive essentially turns queries into MapReduce functions. In contrast to using a ColumIO format, Hive attempts to make queries quick by using techniques such as table indexing.

Hadoop is for batch processing, meaning that queries are run on a set of data that you already have. Streaming engines process data as it comes in. The terms “streaming” and “real time” are often used interchangeably, which could lead to some confusion about Dremel/Drill since they are also referred to as real time.

It should be noted that Google is intending Dremel as a complement, not a replacement, for MapReduce and Hadoop. According to the paper, Dremel is frequently used to analyze MapReduce results or serve as a test run for large scale computations. Dremel can execute many queries over such data that would ordinarily require a sequence of MapReduce, but at a fraction of the execution time. As noted before, Dremel experimentally surpassed MapReduce by orders of magnitude.

Google Dremel vs Apache Drill

Apache Drill is an attempt to build an open source version of Google Dremel. There’s another project in the works to create an open source version of Dremel called OpenDremel. Other projects working on speedy queries for big data include Apache CouchDB and the Cloudant backed variant BigCouch.

Other open source Big Data Analytics Tools and Techniques

1. Storm, which was developed at Backtype and open sourced by Twitter.

2. Apache S4, which was open sourced by Yahoo.

 

Difference between Dremel and other real-time big data systems such as Storm and S4 is that these are streaming engines, while Dremel is designed for ad-hoc querying, ie really fast search results.

Apache Hadoop Big Data Analytics Tool and Technology: Defination, Advantages, Disadvantages

Apache Hadoop Big Data Analytics Tool and Technology: Defination, Advantages, Disadvantages

In this article, we will discuss some point about Apache Hadoop Big Data Tool and Technology available to us for managing the real time big data. We will also look on advantages and disadvantages of Apache Hadoop Big Data Analytics.

What is Hadoop Big Data Analytics Tool and Technology

Hadoop is the best tool available today for processing and storing herculean amounts of big data . Hadoop throws hundreds or thousands of computers at the big data problem, rather than using single computer.

Hadoop makes data mining, analytics, and processing of big data cheap and fast. Hadoop can take most of your big data problems and unlock the answers, because you can keep all your data, including all of your historical data, and get an answer before your children graduate college.

Apache Hadoop is an open-source project inspired by research of Google. Since you were wondering, Hadoop is named after the stuffed toy elephant of the lead programmer's son. This explains the preponderance of pachyderms wherever Hadoop is mentioned.

In Hadoop parlance, the group of coordinated computers is called a cluster, and the individual computers in the cluster are called nodes.

Advantages of Hadoop Big Data Analytics Tool and Technology

1. Hadoop is cheap. Hadoop is an open-source Apache project, which means anybody is free to use it. Hadoop runs on commodity hardware (i.e. normal everyday computers), so you don't have to buy million-dollar specialized database machines.

2. Hadoop is fast. Hadoop can deal with terabytes of data in minutes, and with petabytes in hours. Hadoop is the only way that companies with gigantic amounts of data like Facebook, Twitter, Yahoo, eBay, and Amazon can cost-effectively and quickly make decisions.

3. Hadoop scales to large amounts of big data storage. Need to add more space? Just add more hard drives to a node, or even add more nodes to your cluster. You never shut down Hadoop.

4. Hadoop scales to large amounts of big data computation. Is your cluster slow? Just add more nodes to spread out the computation. Hadoop scales almost linearly in many cases - this means you can halve the time it takes to do a job by doubling the number of compute nodes.

5. Hadoop is flexible with types of big data. Are you dealing with structured data? Great. Do you have semi-structured or unstructured (document-oriented) data? Lovely. Hadoop stores and processes any kind of data.

6. Hadoop is flexible with programming languages. Hadoop is natively written in Java, but you can access your data in a SQL-inspired language called Apache Hive. If you want a more procedural language for analysis, there is Apache Pig. If you want to get deep into the framework, you can custom-analyse your data by writing code in Java, C/C++, Ruby, Python, C#, QBASIC or anything else.

Disadvantages of Hadoop Big Data Analytics Tool and Technology

1. Plain Hadoop is hard to to set up. Have you tried setting up this thing? Your best bet may be to kidnap some professors and press them into your service.

2. Plain Hadoop is hard to manage. How do you do anything? Where is the graphical user interface? Oh, there is none.

3. Plain Hadoop is hard to keep alive. Hadoop has various single points of failure. When Hadoop collapses, you lose data and you lose time. That hurts.

4. Plain Hadoop is hard to use. Seriously, this is not a joke. Even adding up a list of numbers is painful.

5. Plain Hadoop is not secure. Your files are not secure and users can easily corrupt or steal data. I hope you trust everybody.

6. Plain Hadoop is not optimized for your hardware. Hadoop does not run at full capacity for your hardware, which is like being stuck in second gear.

Real Time Big Data: Defination, Need, Uses, Examples, Challenges, Technologies and Analytics

Real Time Big Data: Defination, Need, Uses, Examples, Challenges, Technologies and Analytics

This article is based on real time big data. We will discuss what is big data? What is the need of big data? We will also provide some uses and examples of big data which will let you think the actual need of real time big data. We will also discuss the various challenges in the field of real time big data and how can we face these big data challenges. Then we have some big data analytics with us.

Defination of Real Time Big Data

Big data usually includes data sets with sizes beyond the ability of commonly-used software tools to capture, curate, manage, and process the data within a tolerable elapsed time.

Big data is a popular term used to describe the exponential growth, availability and use of information, both structured and unstructured.

Need of Real Time Big Data

Many factors are there which increase the need of real time big data like Volume of data, Variety of data and Velocity of data, Variability and Complexity of data. These factors are discussed below in detail:

1. Volume: Many factors contribute to the increase in data volume – transaction-based data stored through the years, text data constantly streaming in from social media, increasing amounts of sensor data being collected, etc. In the past, excessive data volume created a storage issue. But with today's decreasing storage costs, other issues emerge, including how to determine relevance amidst the large volumes of data and how to create value from data that is relevant.

2. Variety: Data today comes in all types of formats – from traditional databases to hierarchical data stores created by end users and OLAP systems, to text documents, email, meter-collected data, video, audio, stock ticker data and financial transactions. By some estimates, 80 percent of an organization's data is not numeric! But it still must be included in analyses and decision making.

3. Velocity: According to Gartner, velocity "means both how fast data is being produced and how fast the data must be processed to meet demand." RFID tags and smart metering are driving an increasing need to deal with torrents of data in near-real time. Reacting quickly enough to deal with velocity is a challenge to most organizations.

4. Variability: In addition to the increasing velocities and varieties of data, data flows can be highly inconsistent with periodic peaks. Is something big trending in the social media? Perhaps there is a high-profile IPO looming. Maybe swimming with pigs in the Bahamas is suddenly the must-do vacation activity. Daily, seasonal and event-triggered peak data loads can be challenging to manage – especially with social media involved.

5. Complexity: When you deal with huge volumes of data, it comes from multiple sources. It is quite an undertaking to link, match, cleanse and transform data across systems. However, it is necessary to connect and correlate relationships, hierarchies and multiple data linkages or your data can quickly spiral out of control. Data governance can help you determine how disparate data relates to common definitions and how to systematically integrate structured and unstructured data assets to produce high-quality information that is useful, appropriate and up-to-date.

Uses of Real Time Big Data

So the real issue is not that you are acquiring large amounts of data (because we are clearly already in the era of big data). It's what you do with your big data that matters. The hopeful vision for big data is that organizations will be able to harness relevant data and use it to make the best decisions.

Technologies today not only support the collection and storage of large amounts of data, they provide the ability to understand and take advantage of its full value, which helps organizations run more efficiently and profitably. For instance, with big data and big data analytics, it is possible to:

1. Analyze millions of SKUs to determine optimal prices that maximize profit and clear inventory.

2. Recalculate entire risk portfolios in minutes and understand future possibilities to mitigate risk.

3. Mine customer data for insights that drive new strategies for customer acquisition, retention, campaign optimization and next best offers.

4. Quickly identify customers who matter the most.

5. Generate retail coupons at the point of sale based on the customer's current and past purchases, ensuring a higher redemption rate.

6. Send tailored recommendations to mobile devices at just the right time, while customers are in the right location to take advantage of offers.

7. Analyze data from social media to detect new market trends and changes in demand.

8. Use clickstream analysis and data mining to detect fraudulent behavior.

9. Determine root causes of failures, issues and defects by investigating user sessions, network logs and machine sensors.

 

Examples of Real Time Big Data

1. RFID (Radio Frequency ID) systems generate up to 1,000 times the data of conventional bar code systems.

2. 10,000 payment card transactions are made every second around the world.

3. Walmart handles more than 1 million customer transactions an hour.

4. 340 million tweets are sent per day. That's nearly 4,000 tweets per second.

5. Facebook has more than 901 million active users generating social interaction data.

6. More than 5 billion people are calling, texting, tweeting and browsing websites on mobile phones.

Challenges of Real Time Big Data

Many organizations are concerned that the amount of amassed data is becoming so large that it is difficult to find the most valuable pieces of information.

What if your data volume gets so large and varied you don't know how to deal with it?

1. Do you store all your data?
2. Do you analyze it all?
3. How can you find out which data points are really important?
4. How can you use it to your best advantage?

 

Until recently, organizations have been limited to using subsets of their data, or they were constrained to simplistic analyses because the sheer volumes of data overwhelmed their processing platforms. What is the point of collecting and storing terabytes of data if you can't analyze it in full context, or if you have to wait hours or days to get results? On the other hand, not all business questions are better answered by bigger data.

You now have two choices:

Incorporate massive data volumes in analysis. If the answers you are seeking will be better provided by analyzing all of your data, go for it. The game-changing technologies that extract true value from big data – all of it – are here today. One approach is to apply high-performance analytics to analyze the massive amounts of data using technologies such as grid computing, in-database processing and in-memory analytics.

Determine upfront which big data is relevant. Traditionally, the trend has been to store everything (some call it data hoarding) and only when you query the data do you discover what is relevant. We now have the ability to apply analytics on the front end to determine data relevance based on context. This analysis can be used to determine which data should be included in analytical processes and which can be placed in low-cost storage for later availability if needed.

Technologies of Real Time Big Data

A number of recent technology advancements are enabling organizations to make the most of big data and big data analytics:

1. Cheap, abundant storage and server processing capacity.

2. Faster processors.

3. Affordable large-memory capabilities, such as Hadoop.

4. New storage and processing technologies designed specifically for large data volumes, including unstructured data.

5. Parallel processing, clustering, MPP, virtualization, large grid environments, high connectivity and high throughputs.

6. Cloud computing and other flexible resource allocation arrangements.

Big data technologies not only support the ability to collect large amounts of data, they provide the ability to understand it and take advantage of its value. The goal of all organizations with access to large data collections should be to harness the most relevant data and use it for optimized decision making.

It is very important to understand that not all of your data will be relevant or useful. But how can you find the data points that matter most? It is a problem that is widely acknowledged.

What is an EXE (Executable) File?

What is an EXE (Executable) File?

An exe file (executable file) is a computer file that ends with the extension ".exe" known as an executable file. When one clicks on an exe file, a built-in routine automatically executes code that can set several functions into motion. Exe files are used to install and run programs and routines.

An executable file is a file that is used to perform various functions or operations on a computer. Unlike a data file, an executable file cannot be read because it has been compiled.

EXE File Format

An exe file is just one of several file format types that are recognized by various operating systems. Text files, which are files that do not generate code but simply display text, end in txt. Microsoft Word saves files with the doc extension, short for document. Another common file type is the compressed or zipped file, which uses the zip extension.

On an IBM compatible computer, common executable files are .BAT, .COM, .EXE, and .BIN. Depending on the operating system and its setup, there can also be several other executable files.

Precautions while downloading and opening an exe file

The exe file is one of the most useful types of files precisely because it runs programs; however, this also makes it potentially harmful. It can be used as a delivery system for viruses or other malicious routines. Outwardly, the infected exe file might seem quite benign. Clicking on it might appear to launch nothing more than an animated cartoon or simple arcade game. However, unseen code can be running in the background, infecting or compromising the computer.

If one clicks on a malicious exe file, over the next several hours or days the computer might become unstable. Often the user does not relate the problem back to the cause and attributes it to other factors, such as the need to defragment or replace a disk. Personal information or password keystrokes might be handed over to an unknown website without knowledge of the user, or the machine might delete its own files and crash altogether. Any of these scenarios and more can be set in motion by simply clicking on a 'bad' exe file.

Because of the potential harm, when downloading any exe file it's a good idea to scan it with a reputable virus scanner before clicking on it. To this end, security experts recommend setting browsers so that websites cannot load programs automatically. This gives the surfer the chance to monitor which programs get passed to his or her computer.

Exe files are generally not meant to be edited, and changing an exe's file size will render it inoperable. Since the exe file is most often a program, it is routinely protected by copyright laws, per the linked license agreement issued by its author. Hacking an exe file is illegal in this case. The exception is public domain software. Public domain software belongs to the public and can legally be altered by anyone.

As it is safest to err on the side of caution, the best defense is a good offense. Experts recommend keeping virus checkers up to date and deleting email from unknown sources. If you receive an exe file from someone you trust, scan it first before clicking on it. Even files that appear to be coming from reputable sources can be maligned by third parties without the source's knowledge.

Online Professional IT Certification Courses

Online Professional IT Certification Courses

There are many online professional IT certifications courses available. Most popular online IT certification courses include Cisco CCNA and Microsoft certifications. You can get online certification training and can give online certification exams. Beside these, there are also testing and ethical hacking online certification courses available. Here is the list of online professional IT certification courses.

1. Online Professional Cisco CCNA (Cisco Certified Network Associate) Certification Courses

A) CCNA Discovery

CCNA Discovery provides an overview of general networking theory and opportunities for practical experience, career exploration, and soft-skills development.

B) Cisco CCNA

CCNA Exploration provides a comprehensive overview of foundational to advanced networking concepts, with an emphasis on theory and practical application.

C) CCNA Security 

CCNA Security introduces the core security concepts and skills needed to install, troubleshoot, and monitor a network to maintain the integrity, confidentiality, and availability of data and devices. 

D) CCNP 

CCNP teaches the advanced skills needed to install, configure, monitor, and troubleshoot enterprise-sized networks and manage wireless, security, and voice applications.

2. Online Professional Microsoft Certification Courses

Microsoft Certification has many benefits and enables you to keep your skills relevant, applicable, and competitive. In addition, Microsoft Certification is an industry standard that is recognized worldwide—which helps open doors to a lot of career options. After you earn your Microsoft Certification, you have access to a number of benefits, which can be found on the MCP, MCT, or MOS member site.

A) The Microsoft Certified Technology Specialist (MCTS)

An MCTS certification proves your ability to successfully implement, build on, troubleshoot, and debug a particular Microsoft technology, such as a Windows operating system, Microsoft Exchange Server, Microsoft SQL Server, and Microsoft Visual Studio. MCTS credential enables professionals to target specific technologies, and is generally the first step toward the Professional-level certifications.

B) Microsoft Certified IT Professional (MCITP)

The Microsoft Certified IT Professional (MCITP) certification helps validate that an individual has the comprehensive set of skills necessary to perform a particular role, such as database administrator or enterprise messaging administrator. MCITP certifications build on the technical proficiency measured in the Microsoft Certified Technology Specialist (MCTS) certifications. Therefore, you will earn one or more MCTS certifications on your way to earning a MCITP certification. MCITP certified professionals end up as IT support engineer, Windows server System administrator, or Senior Database Administrator.

C) Microsoft Certified Architect Program (MCA)

If you have an experience of more than ten years in IT or more than five years in a specific architecture, then MCA can give your career that much needed boost. If you are a technology architect designing IT solutions for enterprises along with the skills to execute a project from start until the end, earning a MCA might just be the right thing to do. The Microsoft Certified Architect Program is necessary for all IT professionals seeking a specialist degree to validate their expertise in the IT architecture.

3. Certified Information Systems Security Professional (CISSP)

If you plan to build a career in information security, one of today's most visible professions, then the CISSP credential should be your next career goal. It is the credential for professionals who develop policies and procedures in information security. This certification can easily fetch you the post of security managers and security professionals.

4. IT Infrastructure Library (ITIL foundation certificate)

ITIL certifications provide demonstration of knowledge and skills involved in management positions in IT services. There are three certification levels: Foundation, Practitioner, and Manager. The Manager level certification requires completion of a two-week training program, and candidates must have the Foundation certification and five years of IT management experience. Then, they must pass two three-hour exams consisting of essay questions. ITIL is intended to assist organizations in developing a framework for IT service management.

5. International Software Testing Qualifications Board (ISTQB)

ISTQB is software testing qualification certification organization. It is responsible for the international qualification scheme called "ISTQB Certified Tester". As of March 2011, it has issued over 180 certifications in more than 70 countries. The qualifications are based on a syllabus, and there is a hierarchy of qualifications and guidelines for accreditation and examination. It is the ISTQB's role to support a single, universally accepted, international qualification scheme, aimed at software and system testing professionals, by providing the core syllabi and by setting guidelines for accreditation and examination for national boards.

6. Apple Certified Technical Coordinator (ACTC)

For those interested in working with the Mac OS X, this is one rare and necessary certification. It targets entry level system administrators, who provide support and maintain Mac OS X client server environments.

7. Project Management Professional (PMP)

PMI's Project Management Professional (PMP) credential is the most important industry-recognized certification for project managers. Globally recognized and demanded, the PMP demonstrates that you have the experience, education and competency to successfully lead and direct projects. This recognition is seen through increased marketability to employers and higher salary.

8. Ethical Hacker Certification (EHC)

The EHC Program certifies individuals in the specific network security discipline of Ethical Hacking from a vendor-neutral perspective. The Certified Ethical Hacker certification builds up the application knowledge of security officers, auditors, security professionals, site administrators, and anyone who is concerned about the integrity of the network infrastructure. A Certified Ethical Hacker is a skilled professional who understands and knows how to look for the weaknesses and vulnerabilities in target systems and uses the same knowledge and tools as a malicious hacker.