目錄
1. Introduction 引言… ………………………………………………………………………………1
1.1 Towards Intelligent Machines 走向智能機(jī)器 ………………………………………………1
1.2 Well-Posed Machine Learning Problems 適定的機(jī)器學(xué)習(xí)問題 ……………………………5
1.3 Examples of Applications in Diverse Fields 不同領(lǐng)域的應(yīng)用實(shí)例 …………………………7
1.4 Data Representation 數(shù)據(jù)表示 ………………………………………………………………12
1.4.1 Time Series Forecasting 時(shí)間序列預(yù)測 ………………………………………………15
1.4.2 Datasets for Toy (Unreastically Simple) and Realistic Problems
初級問題和現(xiàn)實(shí)問題數(shù)據(jù)集 …………………………………………………………17
1.5 Domain Knowledge for Productive use of Machine Learning
使機(jī)器學(xué)習(xí)有效應(yīng)用的領(lǐng)域知識(shí) ……………………………………………………………18
1.6 Diversity of Data: Structured/Unstructured 數(shù)據(jù)多樣性：結(jié)構(gòu)化/非結(jié)構(gòu)化 ……………20
1.7 Forms of Learning 學(xué)習(xí)形式 …………………………………………………………………21
1.7.1 Supervised/Directed Learning 監(jiān)督/指導(dǎo)學(xué)習(xí) ……………………………………21
1.7.2 Unsupervised/Undirected Learning 非監(jiān)督/無指導(dǎo)學(xué)習(xí) …………………………22
1.7.3 Reinforcement Learning 強(qiáng)化學(xué)習(xí) …………………………………………………22
1.7.4 Learning Based on Natural Processes: Evolution, Swarming, and Immune Systems
基于自然處理的學(xué)習(xí)：進(jìn)化、集群和免疫系統(tǒng)………………………………………23
1.8 Machine Learning and Data Mining 機(jī)器學(xué)習(xí)和數(shù)據(jù)挖掘 …………………………………25
1.9 Basic Linear Algebra in Machine Learning Techniques
機(jī)器學(xué)習(xí)技術(shù)中的基礎(chǔ)線性代數(shù) ……………………………………………………………26
1.10 Relevant Resources for Machine Learning 機(jī)器學(xué)習(xí)的相關(guān)資源 …………………………34
2. Supervised Learning: Rationale and Basics 監(jiān)督學(xué)習(xí)：基本原理和基礎(chǔ)… ………………36
2.1 Learning from Observations 從觀測中學(xué)習(xí) …………………………………………………36
2.2 Bias and Variance 偏差和方差 ………………………………………………………………42
2.3 Why Learning Works: Computational Learning Theory
學(xué)習(xí)為什么有效：計(jì)算學(xué)習(xí)理論 ……………………………………………………………46
2.4 Occam’s Razor Principle and Overfitting Avoidance
奧卡姆剃刀原理和防止過擬合 ………………………………………………………………49
2.5 Heuristic Search in Inductive Learning 歸納學(xué)習(xí)中的啟發(fā)式搜索 …………………………51
2.5.1 Search through Hypothesis Space 假設(shè)空間搜索 ……………………………………52
2.5.2 Ensemble Learning 集成學(xué)習(xí) ………………………………………………………53
2.5.3 Evaluation of a Learning System 學(xué)習(xí)系統(tǒng)的評價(jià) …………………………………55
2.6 Estimating Generalization Errors 估計(jì)泛化誤差 ……………………………………………56
2.6.1 Holdout Method and Random Subsampling 留出法和隨機(jī)下采樣 …………………56
2.6.2 Cross-validation 交叉驗(yàn)證 ……………………………………………………………57
2.6.3 Bootstrapping 自助法 …………………………………………………………………58
2.7 Metrics for Assessing Regression (Numeric Prediction) Accuracy
評價(jià)回歸（數(shù)值預(yù)測）精度的指標(biāo) …………………………………………………………59
2.7.1 Mean Square Error 均方誤差 …………………………………………………………60
2.7.2 Mean Absolute Error 平均絕對誤差 …………………………………………………60
2.8 Metrics for Assessing Classification (Pattern Recognition) Accuracy
評價(jià)分類（模式識(shí)別）精度的指標(biāo) …………………………………………………………61
2.8.1 Misclassification Error 誤分類誤差 …………………………………………………61
2.8.2 Confusion Matrix 混淆矩陣 …………………………………………………………62
2.8.3 Comparing Classifiers Based on ROC Curves 基于ROC曲線的分類器比較 ……66
2.9 An Overview of the Design Cycle and Issues in Machine Learning
機(jī)器學(xué)習(xí)中的設(shè)計(jì)周期和問題概述 …………………………………………………………68
3. Statistical Learning 統(tǒng)計(jì)學(xué)習(xí)… …………………………………………………………………73
3.1 Machine Learning and Inferential Statistical Analysis 機(jī)器學(xué)習(xí)與推斷統(tǒng)計(jì)分析 …………73
3.2 Descriptive Statistics in Learning Techniques 學(xué)習(xí)技術(shù)中的描述統(tǒng)計(jì)學(xué) …………………74
3.2.1 Representing Uncertainties in Data: Probability Distributions
表示數(shù)據(jù)中的不確定性：概率分布 …………………………………………………75
3.2.2 Descriptive Measures of Probability Distributions 概率分布的描述方法 …………80
3.2.3 Descriptive Measures from Data Sample 數(shù)據(jù)樣本的描述方法 ……………………83
3.2.4 Normal Distributions 正態(tài)分布 ………………………………………………………84
3.2.5 Data Similarity 數(shù)據(jù)相似性 …………………………………………………………85
3.3 Bayesian Reasoning: A Probabilistic Approach to Inference
貝葉斯推理：推斷的概率方法 ………………………………………………………………87
3.3.1 Bayes Theorem 貝葉斯定理 …………………………………………………………88
3.3.2 Naive Bayes Classifier 樸素貝葉斯分類器 …………………………………………93
3.3.3 Bayesian Belief Networks 貝葉斯信念網(wǎng)絡(luò) …………………………………………98
3.4 k-Nearest Neighbor (k-NN) Classifier k近鄰（k-NN）分類器 …………………………102
3.5 Discriminant Functions and Regression Functions 判別函數(shù)和回歸函數(shù) ………………106
3.5.1 Classification and Discriminant Functions 分類和判別函數(shù) ……………………107
3.5.2 Numeric Prediction and Regression Functions 數(shù)值預(yù)測和回歸函數(shù) ……………108
3.5.3 Practical Hypothesis Functions 實(shí)踐應(yīng)用中的假設(shè)函數(shù) …………………………109
3.6 Linear Regression with Least Square Error Criterion
基于最小二乘誤差準(zhǔn)則的線性回歸法 ……………………………………………………112
3.6.1 Minimal Sum-of-Error-Squares and the Pseudoinverse 最小誤差平方和與偽逆 113
3.6.2 Gradient Descent Optimization Schemes 梯度下降法優(yōu)化方案 …………………115
3.6.3 Least Mean Square (LMS) Algorithm 最小均方（LMS）算法 …………………115
3.7 Logistic Regression for Classification Tasks 分類任務(wù)的邏輯回歸法 ……………………116
3.8 Fisher’s Linear Discriminant and Thresholding for Classification
Fisher線性判別和分類閾值…………………………………………………………………120
3.8.1 Fisher’s Linear Discriminant Fisher線性判別式 …………………………………120
3.8.2 Thresholding 閾值 …………………………………………………………………125
3.9 Minimum Description Length Principle 最小描述長度原理 ……………………………126
3.9.1 Bayesian Perspective 貝葉斯角度 …………………………………………………127
3.9.2 Entropy and Information 熵和信息 ………………………………………………128
4. Learning With Support Vector Machines (SVM) 利用支持向量機(jī)（SVM）學(xué)習(xí)…………130
4.1 Introduction 簡介 …………………………………………………………………………130
4.2 Linear Discriminant Functions for Binary Classification
二元分類的線性判別函數(shù) …………………………………………………………………132
4.3 Perceptron Algorithm 感知機(jī)算法 …………………………………………………………136
4.4 Linear Maximal Margin Classifier for Linearly Separable Data
線性可分?jǐn)?shù)據(jù)的最大邊緣線性分類器 ……………………………………………………141
4.5 Linear Soft Margin Classifier for Overlapping Classes
重疊類的軟邊緣線性分類器 ………………………………………………………………152
4.6 Kernel-Induced Feature Spaces 核函數(shù)引導(dǎo)的特征空間 …………………………………158
4.7 Nonlinear Classifier 非線性分類器 ………………………………………………………162
4.8 Regression by Support Vector Machines 支持向量機(jī)回歸 ………………………………167
4.8.1 Linear Regression 線性回歸 ………………………………………………………169
4.8.2 Nonlinear Regression 非線性回歸 …………………………………………………172
4.9 Decomposing Multiclass Classification Problem Into Binary Classification Tasks
將多類分類問題分解為二元分類任務(wù) ……………………………………………………174
4.9.1 One-Against-All (OAA) 一對所有（OAA） ……………………………………175
4.9.2 One-Against-One (OAO) 一對一（OAO） ………………………………………176
4.10 Variants of Basic SVM Techniques 基礎(chǔ)支持向量機(jī)技術(shù)的變體 ………………………177
5. Learning With Neural Networks (NN) 利用神經(jīng)網(wǎng)絡(luò)（NN）學(xué)習(xí)…………………………181
5.1 Towards Cognitive Machine 走向認(rèn)知機(jī)器 ………………………………………………181
5.1.1 From Perceptrons to Deep Networks 從感知機(jī)到深度網(wǎng)絡(luò) ………………………182
5.2 Neuron Models 神經(jīng)元模型 ………………………………………………………………184
5.2.1 Biological Neuron 生物神經(jīng)元 ……………………………………………………184
5.2.2 Artificial Neuron 人工神經(jīng)元 ……………………………………………………186
5.2.3 Mathmatical Model 數(shù)學(xué)模型 ……………………………………………………190
5.3 Network Architectures 網(wǎng)絡(luò)架構(gòu) …………………………………………………………193
5.3.1 Feedforward Networks 前饋網(wǎng)絡(luò) …………………………………………………194
5.3.2 Recurrent Networks 循環(huán)網(wǎng)絡(luò) ……………………………………………………199
5.4 Perceptrons 感知機(jī) …………………………………………………………………………200
5.4.1 Limitations of Perceptron Algorithm for Linear Classification Tasks
線性分類任務(wù)采用感知機(jī)算法的局限性 …………………………………………201
5.4.2 Linear Classification using Regression Techniques
使用回歸技術(shù)的線性分類 …………………………………………………………201
5.4.3 Standard Gradient Descent Optimization Scheme: Steepest Descent
標(biāo)準(zhǔn)梯度下降法優(yōu)化方案：最速下降法 …………………………………………203
5.5 Linear Neuron and the Widrow-Hoff Learning Rule
線性神經(jīng)元與Widrow-Hoff學(xué)習(xí)規(guī)則 ……………………………………………………206
5.5.1 Stochastic Gradient Descent 隨機(jī)梯度下降法 ……………………………………207
5.6 The Error-Correction Delta Rule 糾錯(cuò)Delta法則 …………………………………………208
5.6.1 Sigmoid Unit: Soft-Limiting Perceptron Sigmoid單元：軟限制感知機(jī) …………211
5.7 Multi-Layer Perceptron (MLP) Networks and the Error-Backpropagation Algorithm
多層感知機(jī)網(wǎng)絡(luò)及誤差反向傳播算法 ……………………………………………………213
5.7.1 The Generalized Delta Rule 廣義Delta法則 ……………………………………216
5.7.2 Convergence and Local Minima 收斂性和局部極小值 …………………………226
5.7.3 Adding Momentum to Gradient Descent 為梯度下降法加入動(dòng)量 ………………227
5.7.4 Heuristic Aspects of the Error-backpropagation Algorithm
誤差反向傳播算法的啟發(fā)性 ………………………………………………………228
5.8 Multi-Class Discrimination with MLP Networks 使用MLP網(wǎng)絡(luò)的多類判別 …………232
5.9 Radial Basis Functions (RBF) Networks 徑向基函數(shù)（RBF）網(wǎng)絡(luò) ……………………235
5.9.1 Training the RBF Network 訓(xùn)練RBF網(wǎng)絡(luò) ………………………………………239
5.10 Genetic-Neural Systems 遺傳-神經(jīng)系統(tǒng) …………………………………………………241
6. Fuzzy Inference Systems 模糊推理系統(tǒng)… ……………………………………………………245
6.1 Introduction 簡介 …………………………………………………………………………245
6.2 Cognitive Uncertainty and Fuzzy Rule-Base 認(rèn)知不確定性和模糊規(guī)則庫 ………………248
6.3 Fuzzy Quantification of Knowledge 知識(shí)的模糊量化 ……………………………………253
6.3.1 Fuzzy Logic 模糊邏輯 ……………………………………………………………253
6.3.2 Fuzzy Sets 模糊集 …………………………………………………………………257
6.3.3 Fuzzy Set Operations 模糊集運(yùn)算 …………………………………………………267
6.3.4 Fuzzy Relations 模糊關(guān)系 …………………………………………………………268
6.4 Fuzzy Rule-Base and Approximate Reasoning 模糊規(guī)則庫與近似推理 …………………277
6.4.1 Quantification of Rules via Fuzzy Relations 通過模糊關(guān)系量化規(guī)則 ……………281
6.4.2 Fuzzification of Input 輸入的模糊化 ………………………………………………283
6.4.3 Inference Mechanism 推理機(jī)制 ……………………………………………………284
6.4.4 Defuzzification of Inferred Fuzzy Set 推理模糊集的解模糊 ……………………298
6.5 Mamdani Model for Fuzzy Inference Systems 模糊推理系統(tǒng)的Mamdani模型 …………301
6.5.1 Mobile Robot Navigation Among Moving Obstacles
移動(dòng)障礙物環(huán)境下的移動(dòng)機(jī)器人導(dǎo)航 ……………………………………………303
6.5.2 Mortgage Loan Assessment 抵押貸款評估 ………………………………………308
6.6 Takagi-Sugeno Fuzzy Model Takagi-Sugeno模糊模型 …………………………………311
6.7 Neuro-Fuzzy Inference Systems 神經(jīng)模糊推理系統(tǒng) ……………………………………317
6.7.1 ANFIS Architecture ANFIS結(jié)構(gòu) …………………………………………………318
6.7.2 How Does an ANFIS Learn? ANFIS如何學(xué)習(xí) ……………………………………320
6.8 Gentic-Fuzzy Systems 遺傳模糊系統(tǒng) ……………………………………………………324
7. Data Clustering and Data Transformations 數(shù)據(jù)聚類和數(shù)據(jù)變換… ………………………328
7.1 Unsupervised Learning 非監(jiān)督學(xué)習(xí) ………………………………………………………328
7.1.1 Clustering 聚類 ……………………………………………………………………329
7.2 Engineering the Data 數(shù)據(jù)工程 ……………………………………………………………331
7.2.1 Exploratory Data Analysis: Learning about What is in the Data
探索數(shù)據(jù)分析：了解數(shù)據(jù)中的內(nèi)容 ………………………………………………333
7.2.2 Cluster Analysis: Finding Similarities in the Data
聚類分析：發(fā)現(xiàn)數(shù)據(jù)間的相似性 …………………………………………………334
7.2.3 Data Transformations: Enhancing the Information Content of the Data
數(shù)據(jù)變換：提高數(shù)據(jù)的信息量 ……………………………………………………339
7.3 Overview of Basic Clustering Methods 基本聚類方法概述 ………………………………341
7.3.1 Partitional Clustering 劃分聚類 ……………………………………………………341
7.3.2 Hierarchical Clustering 層次聚類 …………………………………………………344
7.3.3 Spectral Clustering 譜聚類 …………………………………………………………345
7.3.4 Clustering using Self-Organizing Maps 自組織映射聚類 ………………………349
7.4 K-Means Clustering K均值聚類 …………………………………………………………352
7.5 Fuzzy K-Means Clustering 模糊K均值聚類 ……………………………………………356
7.6 Expectation-Maximization (EM) Algorithm and Gaussian Mixtures Clustering
期望最大化（EM）算法和高斯混合聚類 …………………………………………………362
7.6.1 EM Algorithm EM算法 ……………………………………………………………362
7.6.2 Gaussian Mixture Models 高斯混合模型 …………………………………………365
7.7 Some Useful Data Transformations 一些有用的數(shù)據(jù)變換 ………………………………372
7.7.1 Data Cleansing 數(shù)據(jù)清洗 …………………………………………………………372
7.7.2 Derived Attributes 衍生屬性 ………………………………………………………374
7.7.3 Discretizing Numeric Attributes 離散化數(shù)值屬性 ………………………………375
7.7.4 Attribute Reduction Techniques 屬性約簡技術(shù) ……………………………………377
7.8 Entropy-Based Method for Attribute Discretization 基于熵的屬性離散化方法 …………377
7.9 Principal Components Analysis (PCA) for Attribute Reduction
用于屬性約簡的主成分分析（PCA） ……………………………………………………382
7.10 Rough Sets-Based Methods for Attribute Reduction 基于粗糙集的屬性約簡方法 ……390
7.10.1 Rough Set Preliminaries 粗糙集準(zhǔn)備 ……………………………………………392
7.10.2 Analysis of Relevance of Attributes 屬性相關(guān)性分析 ……………………………397
7.10.3 Reduction of Attributes 約簡屬性 …………………………………………………399
8. Decision Tree Learning 決策樹學(xué)習(xí)……………………………………………………………404
8.1 Introduction 簡介 …………………………………………………………………………404
8.2 Example of a Classification Decision Tree 分類決策樹示例 ……………………………406
8.3 Measures of Impurity for Evaluating Splits in Decision Trees
評價(jià)決策樹分裂的不純度度量 ……………………………………………………………411
8.3.1 Information Gain/Entropy reduction 信息增益/熵減少 …………………………411
8.3.2 Gain Ratio 增益率 …………………………………………………………………416
8.3.3 Gini Index 基尼指數(shù) ………………………………………………………………417
8.4 ID3, C4.5, and CART Decision Trees ID3, C4.5與CART決策樹 ………………………418
8.5 Pruning the Tree 剪枝 ………………………………………………………………………427
8.6 Strengths and Weaknesses of Decision-Tree Approach 決策樹方法的優(yōu)缺點(diǎn) …………429
8.7 Fuzzy Decision Trees 模糊決策樹 …………………………………………………………433
9. Business Intelligence and Data Mining: Techniques and Applications
商業(yè)智能和數(shù)據(jù)挖掘：技術(shù)與應(yīng)用… …………………………………………………………445
9.1 An Introduction to Analytics 分析簡介 ……………………………………………………445
9.1.1 Machine Learning, Data Mining, and Predictive Analytics
機(jī)器學(xué)習(xí)、數(shù)據(jù)挖掘和預(yù)測分析 …………………………………………………448
9.1.2 Basic Analytics Techniques 基本分析技術(shù) ………………………………………449
9.2 The CRISP-DM (Cross Industry Standard Process for Data Mining) Model
CRISP-DM（數(shù)據(jù)挖掘跨行業(yè)標(biāo)準(zhǔn)流程）模型 ……………………………………………451
9.3 Data Warehousing and Online Analytical Processing 數(shù)據(jù)倉庫和聯(lián)機(jī)分析處理 ………456
9.3.1 Basic Concepts 基本概念 …………………………………………………………456
9.3.2 Databases 數(shù)據(jù)庫 …………………………………………………………………458
9.3.3 Data Warehousing: A General Architecture, and OLAP Operations
數(shù)據(jù)倉庫：通用架構(gòu)和OLAP操作 ………………………………………………461
9.3.4 Data Mining in the Data Warehouse Environment
數(shù)據(jù)倉庫環(huán)境中的數(shù)據(jù)挖掘 ………………………………………………………466
9.4 Mining Frequent Patterns and Association Rules 挖掘頻繁模式和關(guān)聯(lián)規(guī)則 ……………467
9.4.1 Basic Concepts 基本概念 …………………………………………………………469
9.4.2 Measures of Strength of Frequent Patterns and Association Rules
頻繁模式和關(guān)聯(lián)規(guī)則的強(qiáng)度的度量 ………………………………………………471
9.4.3 Frequent Item Set Mining Methods 頻繁項(xiàng)集挖掘方法 …………………………473
9.4.4 Generating Association Rules from Frequent Itemsets
從頻繁項(xiàng)集生成關(guān)聯(lián)規(guī)則 …………………………………………………………477
9.5 Intelligent Information Retrieval Systems 智能信息檢索系統(tǒng) ……………………………479
9.5.1 Text Retrieval 文本檢索 ……………………………………………………………483
9.5.2 Image Retrieval 圖像檢索 …………………………………………………………486
9.5.3 Audio Retrieval 音頻檢索 …………………………………………………………488
9.6 Applications and Trends 應(yīng)用和趨勢 ………………………………………………………490
9.6.1 Data Mining Applications 數(shù)據(jù)挖掘應(yīng)用 …………………………………………490
9.6.2 Data Mining Trends 數(shù)據(jù)挖掘趨勢 ………………………………………………495
9.7 Technologies for Big Data 大數(shù)據(jù)技術(shù) ……………………………………………………498
9.7.1 Emerging Analytic Methods 新興分析方法 ………………………………………500
9.7.2 Emerging Technologies for Higher Levels of Scalability
用于更高層次可擴(kuò)展性的新興技術(shù) ………………………………………………503
Appendix A Genetic Algorithm (GA) For Search Optimization
搜索優(yōu)化的遺傳算法（GA）… …………………………………………………… 508
A.1 A Simple Overview of Genetics 遺傳學(xué)的簡單概述 ……………………………………510
A.2 Genetics on Computers 計(jì)算機(jī)遺傳學(xué) ……………………………………………………511
A.3 The Basic Genetic Algorithm 基本遺傳算法 ……………………………………………515
A.4 Beyond the Basic Genetic Algorithm 超越基本遺傳算法 ………………………………524
Appendix B Reinforcement Learning (RL) 強(qiáng)化學(xué)習(xí)（RL）… ……………………………527
B.1 Introduction 簡介 …………………………………………………………………………527
B.2 Elements of Reinforcement Learning 強(qiáng)化學(xué)習(xí)的要素 …………………………………530
B.3 Basics of Dynamic Programming 動(dòng)態(tài)編程基礎(chǔ) …………………………………………535
B.3.1 Finding Optimal Policies 尋找最優(yōu)策略 …………………………………………538
B.3.2 Value Iteration 值迭代 ……………………………………………………………539
B.3.3 Policy Iteration 策略迭代 …………………………………………………………540
B.4 Temporal Difference Learning 時(shí)間差分學(xué)習(xí) ……………………………………………542
B.4.1 Q-learning Q-學(xué)習(xí) …………………………………………………………………544
B.4.2 Generalization 歸納 ………………………………………………………………546
B.4.3 Sarsa-learning SARSA學(xué)習(xí) ………………………………………………………548
Datasets from Real-Life Applications for Machine Learning Experiments
機(jī)器學(xué)習(xí)實(shí)驗(yàn)的實(shí)際應(yīng)用數(shù)據(jù)集………………………………………………………………… 549
Problems 習(xí)題… …………………………………………………………………………………567
References 參考文獻(xiàn)………………………………………………………………………………613
Index 索引… ………………………………………………………………………………………623