Unix and Linux Stack Exchange
Unix and Linux Stack Exchange

Questions tagged [hdfs]

10 questions
2
votes
0 answers

ssh: connect to host localhost port 22: Connection refused

I have installed hadoop and ssh. hadoop was working fine, then today I am getting the error below when I run the command sbin/start-dfs.sh: Starting namenodes on [localhost] localhost: ssh: connect to host localhost port 22: Connection…
Sanaya
  • 31
  • 2
1
vote
0 answers

Clear RAM Memory Cache and buffer on production Hadoop cluster with HDFS filesystem

we have Hadoop cluster with 265 Linux RHEL machines. from total 265 machines, we have 230 data nodes machines with HDFS filesystem. total memory on each data-node is 128G and we run many spark applications on these machines. last month we added…
yael
  • 12,598
  • 51
  • 169
  • 303
1
vote
2 answers

Convert list of hdf5 files to netcdf files with same name using shell scripting

I have a list of datasets containing satellite data arranged in monthly folders as follows: 01 02 03 04 05 06 07 08 09 10 11 12 These folder are further divided into daily data folder for example for first month 01, daily files are…
Mala Pokhrel
  • 13
  • 3
1
vote
0 answers

master: ssh: connect to host master port 22: Connection refused

i am trying to start my hadoop cluster using the command "start-dfs.sh" but am getting errors as shown below Starting namenodes on [master] master: ssh: connect to host master port 22: Connection refused Starting datanodes master: ssh: connect to…
Sanaya
  • 31
  • 2
1
vote
1 answer

how to find the owner of user and group from user HDFS

we can grant the permissions as hdfs user for hive as the following su hdfs $ hdfs dfs -chown hive:2098 but how to do the opposite way? in order to verify the owner of hive and hive group?
yael
  • 12,598
  • 51
  • 169
  • 303
0
votes
0 answers

Avoid cumulative sum by Unix shell script -hdfs file copy

We are copying Hdfs directory (contains files) from 1 location to other however there is only 1 Target directory. For 1st iteration it gives correct count of file size and counts of files as in 1st time target directory is not having any previous…
user2429695
  • 1
  • 2
0
votes
1 answer

CPU LOAD AVRG + how to deal process with D state

we can see from our RHEL 7.6 server ( kernel version - 3.10.0-957.el7.x86_64 ) that following process are with D state ( they runs from HDFS user ) Note - D state code means that process is in uninterruptible sleep ps -eo s,user,cmd | grep ^[RD] D…
yael
  • 12,598
  • 51
  • 169
  • 303
0
votes
0 answers

How to move the last n files in hdfs

I have a folder in HDFS contains 830000 files, and I want to move the last "8797" files enter code here to another folder in HDFS? I tried using xargs but didn't work fine. Any other ideas? Here is the exact split point between all files. I want to…
Omar AlSaghier
  • 101
  • 2
0
votes
0 answers

how to run script from user hdfs without password

we create the following script on rhel 7.6 /home/run_tasks and in visudo we configured %sudo ALL=(ALL:ALL) ALL root ALL=(ALL) ALL hdfs ALL = (ALL) ALL hdfs ALL= (root) NOPASSWD: /home/run_tasks and ls -ltr /home/run_tasks -rwxrwxrwx …
yael
  • 12,598
  • 51
  • 169
  • 303
-2
votes
1 answer

Hadoop cluster + designing number of disks on data node machine and min requirements

we are using HDP version - 2.6.5 , and HDFS Block replication is 3 we are try to understand data nodes disks min requirements for production mode and according to the fact that Block replication=3 since we are talking about production cluster and…
yael
  • 12,598
  • 51
  • 169
  • 303