SMBUD - Systems and Methods for Big and Unstructured Data

Collect Widely different data sources: Logs IoT Social Media Ecommerce Traditional businness da...

Definition of the four Vs Volume (Data at scale): terabytes to hexabyte of data cumulated on ch...

Using data decisions no longer have to be made in the dark or based on gut instinct; they can be ...

Foundations An entity–relationship model (or ER model) describes interrelated things of interest ...

Keys Superkey: a set of attributes K is a superkey for a relation r if r does not contain two dis...

Exercise 1 Design an ER Model for a car rental system that manages the customers, the cars and th...

Data ingestion Data ingestion is the first and fundamental step of any Data Analysis Pipeline. Th...

Is a standardized resource based way of designing API. The RESTful API uses the available HTTP ve...

Web crawling, data crawling, and web scraping are all names to define the process of data extrac-...

Differences with the traditional data model Schema Less Approach This new types of data model re...

Transaction is SQL, ACID In the relational world we are used to having the concept of a transacti...

MultiValue databases at TRW in 1965. DBM is released by AT&T in 1979. Lotus Domino released in 1...

Key-Value Store A key that refers to a payload (actual content / data). E.g. MemcacheDB, Azure T...

Basic definitions The Graph is a data structure that was first used in 1736 to represent a city a...

Motivations The table based structure of relational databases makes it hard to represent relation...

The most popular graph database is Neo4J, implemented in Java by Neo Technologies, and it has the...

2021 06 22 Q1 (8 points) A dedicated online review and social networking system tracks the activi...

Document databases deviate from the entity-based denormalized data model of relational dbs and pr...

MongoDB is a document-oriented database that stores data within: Documents: consist of key-valu...

Create Create a database: use database_name Create a collection: db.createCollection(name, option...

2021 06 22 Q2 (6 points) A dedicated online review and social networking system tracks the activi...

In many applications performance is an essential priority, and often a small delay in response ti...

Redis is an advanced key-value store, where keys can contain data structures such as strings, has...

Memcache is a free & open source, high-performance, distributed memory object caching system that...

2021 02 04 Q1 A maintenance and service management company supports public administrations in the...

In the recent years there has been a ever growing need for technologies capable of handling large...

Apache Cassandra is a highly scalable, high-performance distributed database designed to handle l...

HBase Table: Split it into multiple regions: replicated across servers. One Store per ColumnFa...

To query the data stored within Cassandra, a dedicated query language named Cassandra Query Langu...

Kibana: Visualize and Manage Elasticsearch: Store, Search and Analyze Logstash + Beats: Inges...

Elasticsearch stores data structures in JSON documents which are distributed and can be accessed ...

Creating and index: PUT /index_name Define a mapping: PUT /my_index/_mapping { "properties": { ...

Working with Beats Beats focus on data collection and shipping while Logstash focuses on processi...

Software platform to easily process vast amounts of data. The main features are: Scalable: It c...

HDFS Architecture NameNode Manages File System Namespace Maps a file name to a set of blocks M...

Shell Commands There are two types of shell commands: User Commands hdfs dfs – runs filesyste...

YARN Splits up the two major functions of JobTracker Global Resource Manager - Cluster resourc...

Introduction MapReduce is a programming model and an associated implementation for processing an...

Input Splitter Is responsible for splitting your input into multiple chunks (default is 64MB). Th...

By default, Hadoop uses FIFO to schedule jobs. Alternate scheduler options: capacity and fair Cap...

HBase is a key-valued row/column store modeled on Google’s Bigtable providing Bigtable-like capab...

Apache Pig is a high-level platform for creating programs that run on Apache Hadoop. The language...

Apache Hive is a data warehouse software project built on top of Apache Hadoop for providing data...

Apache Impala is an open source massively parallel processing (MPP) SQL query engine for data sto...

Apache Storm is a distributed stream processing computation framework. Storm provides realtime co...

Apache Flume is a distributed, reliable, and available software for efficiently collecting, aggre...

Sqoop is a command-line interface application for transferring data between relational databases ...

The step after Data Acquisition and before Analysis in the data management flow is Data Wrangling...

Data Wrangling is the process of transforming “raw” data into data that can be analyzed to gener...