Use the right terms to tell apart the systems that store data, the methods that search it for patterns, and the approach of putting results to use in decisions.
Explain the three characteristics of big data and the steps for analyzing text data.
Explain why using data is not a one-off job but a flow that leads on to the next question.
BI is the umbrella term for the idea of employees and executives analyzing a company's stored data themselves to inform business and operational decisions, along with the tools that support it.
BI stands for Business Intelligence, and charting monthly store visits to compare them and plan the next move is one example. Software that bundles data search and analysis functions is called a BI tool (BIツール).
AnalogyIt works like a car's dashboard. Speed and fuel level are visible at a glance, but the driver still decides what to do.
How to rememberBesides “smarts,” intelligence also means “information,” as in military intelligence, so read BI as turning information into decision-making power.
Exam points
ExamIn BI, the analysis is done by the very employees and executives who will use the results in their decisions.
ExamThe aim of BI tools (BIツール) is to support decision-making, which is why they provide functions for searching and analyzing stored data.
A data warehouse (データウェアハウス) is a database that integrates data scattered across departmental business systems and accumulates it over time for analysis.
← Scroll the figure sideways →
It is abbreviated DWH, and a warehouse is simply a storehouse. Gathering records from separate systems in one place makes it possible to compare company-wide activity over long periods. An operational database that handles daily orders prioritizes keeping its data current, whereas a data warehouse keeps past records for analysis.
AnalogyThink of a head-office archive where copies of every branch's ledgers are filed by year. Older ledgers are kept too, so you can compare across years.
Exam points
ExamA data warehouse (データウェアハウス) gives a cross-departmental view of the company's past records and helps in working out management strategy.
ExamData in a data warehouse is organized by analysis purpose and retained as a time series (時系列).
Mix-upA data warehouse is the base that stores the raw material; BI is the idea of analyzing that material and using it to make decisions.
Data mining (データマイニング) is a method of analyzing large amounts of data with statistics, machine learning, and similar techniques to uncover patterns and relationships no one had noticed.
Aggregation
Data mining
Form of the question
How much is this item?
What is related to what?
What you get
Totals or averages of chosen items
Patterns no one expected
Rather than only testing a set hypothesis, it starts from the data and looks for relationships. For example, if library lending records show that people who borrow books in one field tend to borrow from another, that suggests a way to rearrange the shelves. A relationship found this way does not prove cause and effect, so check it against business knowledge before acting on it.
AnalogyIt is like panning river sand for gold. You cannot tell what will turn up until you sift.
How to rememberMine means both a pit and the act of digging one, which fits the image of digging through a mountain of data.
Exam points
ExamA classic example of data mining (データマイニング) is finding products bought together, known as market basket analysis (バスケット分析). The rules it finds also feed product recommendations tailored to each customer.
ExamWorking out the value of fixed items, such as totals and averages, is aggregation (集計), not data mining.
Mix-upData mining is a method of finding patterns in data, and it is a different thing from a data warehouse (データウェアハウス), which integrates and stores data.
Big data (ビッグデータ) is the general term for data so large in volume, so varied in format, and generated so continuously that conventional databases and methods struggle to handle it.
Element
Meaning
Example
Volume
Large quantity
Several years of register records from stores nationwide
Variety
Many formats
Product photos and call recordings
Velocity
Generated and updated quickly
Measurements sent every second by factory equipment
One common framing takes the initials of the three English words and calls them the 3Vs. Big data is not limited to tidy tables of numbers; it also includes text, images, audio, and equipment logs. The specialists who analyze such data and draw out insights a business can use are called data scientists (データサイエンティスト).
AnalogyLess like water from a tap and more like a great river, with things of every shape and size pouring in without end.
How to rememberPut quantity, type, and speed into English words and each one begins with V.
Exam points
ExamOne view defines the three elements of big data (ビッグデータ) as Volume, Variety, and Velocity, known collectively as the 3Vs.
ExamData in formats other than text can be analyzed too. Combining statistical methods with machine learning (機械学習) also helps predict what will happen next.
ExamThe job of a data scientist (データサイエンティスト) is to gain new insights through statistical and machine-learning analysis, which differs from the role of building database infrastructure.
Text mining (テキストマイニング) is a method of breaking text data, such as free-form survey answers and social media posts, into words and reading trends from how often they appear and how they relate.
Collect text
Split into words
Count frequency and links
Read the trends
Read common topics and changes from the counts.
4 / 4
Unlike a table of numbers, text cannot be totaled or averaged as it stands. Cutting sentences into words and counting them is what first makes it possible to treat text as quantities. For example, records from a support desk can reveal which complaints are on the rise.
AnalogyIt is like handing a machine the job of a person who reads piles of comments and notices that a certain topic keeps coming up lately.
Exam points
Mix-upText mining (テキストマイニング) is not a separate field from data mining (データマイニング) but an application of it that aims the digging at text.
Data Mining vs. Text Mining
Aspect
Data Mining
Text Mining
Key differenceTarget
Large amounts of data of all kinds, such as sales records
Co-occurrence keywords (共起キーワード) are sets of words that tend to appear together in the same sentence or document.
Word pair
Reviews containing both (fictional example)
What it suggests
Breakfast + variety
120
Interest in the breakfast selection
Room + small
85
Complaints about room size
“Co-occurrence” means happening together, and the focus is on how readily words appear in combination. A word's count alone tells you at most what the topic is, while the words that appear with it show how that topic is being discussed. Text mining (テキストマイニング) uses this result as a clue to context.
AnalogyIt is like the way hearing “rain” brings “umbrella” to mind.
Exam points
Mix-upCo-occurrence keywords (共起キーワード) refer to combinations of words, not to a single word that appears many times.
The data science cycle (データサイエンスのサイクル) is a repeating flow of setting a question, collecting and processing data, using the analysis results, and moving on to the next question.
Set the question
Get and process data
Analyze and visualize
Share and use results
Share the results with those involved, put them to use, and carry new questions back to the first stage.
4 / 4
Sources divide the stages differently, but one example is the four stages in the figure. Without a question set at the start, neither the data to collect nor the analysis method can be decided. Using the results raises new questions, so one lap is never the end.
AnalogyIt resembles testing a recipe. You decide on the flavor, gather ingredients and cook, then use the tasters' reactions in the next attempt.
Key points
BI is the umbrella term for the idea and tools that let the people who use stored data analyze it themselves and apply it to decisions.
A data warehouse (データウェアハウス) is an analytical database that combines each department's data and keeps it over time.
Data mining (データマイニング) is a method of searching for unknown patterns, applied in cases such as finding products bought together and recommending products.
One view calls the three elements of big data (ビッグデータ) the 3Vs, and what can be analyzed extends beyond text to other formats.
Text mining (テキストマイニング) reads trends from word frequency and co-occurrence keywords (共起キーワード). Data use as a whole runs as a cycle in which each round leads back into a new question.