wordcount

First wordcount

写在前面

  • 写作不易,如果感觉有用,请打赏作者一杯咖啡~
  • 转载请联系作者,或者注明作者,以及原文链接!

引言

承接上篇,Hadoop 伪分布式搭建
apache YARN 是Hadoop 的集群资源管理系统,YARN 的引入最初也是为了改善 mapreduce 的实现,
但他具有足够的通用性,同样支持其他的分布式计算模式。本节将介绍yarn的基本配置,以及第一个
wordcount 案例。

yarn 的配置

  1. 修改hadoop/etc/hadoop/mapred-site.xml文件
    指定mapreduce计算模型运行在yarn上。

    1
    2
    3
    4
    <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
    </property>
  2. 修改hadoop/etc/hadoop/yarn-site.xml文件
    指定启动运行mapreduce上的nodemanager的运行服务

    1
    2
    3
    4
    <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
    </property>
  3. 指定resourcemanager主节点机器,可选项,不一定要配置,默认是本机,但是 指定了之后,在其他机器上启动,就会报错

1
2
3
4
<property>
<name>yarn.resourcemanager.hostname</name>
<value>bigdata-4</value>
</property>

wordcount 案例

  1. 启动yarn

    1
    2
    sbin/yarn-daemon.sh start resourcemanager
    sbin/yarn-daemon.sh start nodemanager
  2. 查看yarn外部web界面bigdata-4或者IP地址 跟上8088端口号,
    外部通信http

  3. 测试环境,运行一个mapreduce,wordcount单词统计案例

1
2
3
4
5
6
7
一个mapreduce分为五个阶段
input -> map() -> shuffle -> reduce() -> output
步骤:将mapreduce运行在yarn上,需要打jar包
新建一个数据文件,用于测试mapreduce
将数据文件从本地上传到HDFS
bin/hdfs dfs -put word.txt /user/wxt/wxt_test/input
使用官方提供的示例jar包:share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar
  1. 运行

    1
    bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar wordcount /user/wxt/wxt_test/input/word.txt /output/word.txt
  2. 成功运行

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    17/12/08 19:55:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    [wxt@bigdata-4 hadoop-2.5.0]$ bin/yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.0.jar wordcount /user/wxt/wxt_test/input/word.txt /output/word.txt
    17/12/08 19:59:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    17/12/08 19:59:11 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
    17/12/08 19:59:12 INFO input.FileInputFormat: Total input paths to process : 1
    17/12/08 19:59:13 INFO mapreduce.JobSubmitter: number of splits:1
    17/12/08 19:59:13 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1512719236321_0001
    17/12/08 19:59:14 INFO impl.YarnClientImpl: Submitted application application_1512719236321_0001
    17/12/08 19:59:14 INFO mapreduce.Job: The url to track the job: http://bigdata-4:8088/proxy/application_1512719236321_0001/
    17/12/08 19:59:14 INFO mapreduce.Job: Running job: job_1512719236321_0001
    17/12/08 19:59:32 INFO mapreduce.Job: Job job_1512719236321_0001 running in uber mode : false
    17/12/08 19:59:32 INFO mapreduce.Job: map 0% reduce 0%
    17/12/08 19:59:52 INFO mapreduce.Job: map 100% reduce 0%
    17/12/08 20:00:12 INFO mapreduce.Job: map 100% reduce 100%
    17/12/08 20:00:13 INFO mapreduce.Job: Job job_1512719236321_0001 completed successfully
    17/12/08 20:00:13 INFO mapreduce.Job: Counters: 49
    File System Counters
    FILE: Number of bytes read=552
    FILE: Number of bytes written=195039
    FILE: Number of read operations=0
    FILE: Number of large read operations=0
    FILE: Number of write operations=0
    HDFS: Number of bytes read=431
    HDFS: Number of bytes written=354
    HDFS: Number of read operations=6
    HDFS: Number of large read operations=0
    HDFS: Number of write operations=2
    Job Counters
    Launched map tasks=1
    Launched reduce tasks=1
    Data-local map tasks=1
    Total time spent by all maps in occupied slots (ms)=17464
    Total time spent by all reduces in occupied slots (ms)=17724
    Total time spent by all map tasks (ms)=17464
    Total time spent by all reduce tasks (ms)=17724
    Total vcore-seconds taken by all map tasks=17464
    Total vcore-seconds taken by all reduce tasks=17724
    Total megabyte-seconds taken by all map tasks=17883136
    Total megabyte-seconds taken by all reduce tasks=18149376
    Map-Reduce Framework
    Map input records=1
    Map output records=61
    Map output bytes=557
    Map output materialized bytes=552
    Input split bytes=119
    Combine input records=61
    Combine output records=48
    Reduce input groups=48
    Reduce shuffle bytes=552
    Reduce input records=48
    Reduce output records=48
    Spilled Records=96
    Shuffled Maps =1
    Failed Shuffles=0
    Merged Map outputs=1
    GC time elapsed (ms)=175
    CPU time spent (ms)=2240
    Physical memory (bytes) snapshot=309456896
    Virtual memory (bytes) snapshot=1680064512
    Total committed heap usage (bytes)=136450048
    Shuffle Errors
    BAD_ID=0
    CONNECTION=0
    IO_ERROR=0
    WRONG_LENGTH=0
    WRONG_MAP=0
    WRONG_REDUCE=0
    File Input Format Counters
    Bytes Read=312
    File Output Format Counters
    Bytes Written=354

致此,First WordCount 运行结束。

坚持原创技术分享,您的支持将鼓励我继续创作!