Grep and Kill process on linux. Kill process in linux matching a string / Grep

nuke – Named keeping in mind that it could kill process with matches all the process given by a string. This is a perl program with take input a  string which you are trying to grep and kill.

 


#!/usr/bin/perl

 

use strict;
use Term::ANSIColor qw(:constants);
use Shell;

my (@procs, @all, @pid_list, @proc_name);
sub ps;

die ("Usage: kill.pl <string_to_match>\n") if (scalar(@ARGV) == 0);

@all = ps('-ef');

my $str = $ARGV[0];
my @procz = grep (/$str/, @all);

foreach (@procz){
if(/vim/ or /emacs/ or /xterm/){
next;
}
push(@procs, $_);
}
@pid_list = map {(split( /\s+/, $_))[1]} @procs;

@proc_name = ();

foreach(@procs){
my (@a, $b, @c);
@c = ();
@a = split(/\s/, $_);
foreach $b (@a){
chomp($b);
next if length($b) == 0;
@c = (@c, $b);
}
@proc_name = (@proc_name, $c[7]);
}
if(scalar(@procs) > 1){
printf("%-10s %-65s %-15s\n", "PID", "PROCESS", "STATUS");
printf("%-10s %-65s %-15s\n\n", "-" x 10, "-" x 65, "-" x 15);
}

for(my $i = 0; $i < scalar(@pid_list); $i++){
chomp($pid_list[$i]);
if(!defined($pid_list[$i]) or (length("$pid_list[$i]") == 0) or $pid_list[$i] == $$){

print (RESET);
next;
}
chomp($proc_name[$i]);
my $pname = $proc_name[$i];
my $pidx = -1;

$pidx = rindex($pname, "/");
if($pidx > 0){
$pidx -= 1;
foreach (0..4){
$pidx = rindex($pname, "/", $pidx);
last if ($pidx == -1);
$pidx -= 1;
}
if($pidx > 0){
$pidx += 1;
$pname = substr($pname, $pidx);
$pname = "...$pname";
$proc_name[$i] = $pname;
}
}

print(YELLOW);
printf("%-10s %-65s ", $pid_list[$i], $proc_name[$i]);
my $count = kill ("KILL", $pid_list[$i]);
if($count == 1){
print(RED);
printf ("%-15s\n", "[ KILLED ]");
}
else{
print(BLUE);
printf ("%-15s\n", "[ ALIVE ]");
}
print(RESET);
}
if(scalar(@procs) > 1){
printf("%-10s %-65s %-15s\n", "-" x 10, "-" x 65, "-" x 15);
}

Saving dataFrame to single file in Spark Java

If you are trying to verify your spark application, and you want to data to be saved to single file on HDFS or Local file system you can achieve that using method

coalesce(1)

What this method actually does is to force spark to create a single partition to write the data to the underlying file system.


df.coalesce(1).write().format("com.databricks.spark.csv").option("inferSchema", "false").option("header", headerFlag)
.option("charset", "UTF-8").option("escape", escape).option("delimiter", ",").option("quote", "\'").mode(SaveMode.Overwrite).save("hdfs://localhost:9002/data/data.csv");

&nbsp;

// End of code

 

 

Writing DataFrame to CSV file using Spark Java

Many times we want to save our spark dataframe to a file in a CSV file so that we can persist it. CSV is the very popular form which can be read as DataFrame back with CSV datasource support. Similar to reading, writing to CSV also possible with same com.databricks.spark.csv datasource package. Following is example code

 


df.write().format("com.databricks.spark.csv").option("inferSchema", "false").option("header", headerFlag)
.option("charset", "UTF-8").option("escape", escape).option("delimiter", ",").option("quote", "\'").mode(SaveMode.Overwrite).save("hdfs://localhost:9002/data/data.csv");

Docker hangs without any response on RHEL or OEL linux

Docker commands including docker version are hanging for a long time without any response, following commands might help you to recover.

Note: Make sure you do not this instance.


# Stop the docker service&nbsp;
sudo /sbin/service docker stop
#Remove docker runtime files&nbsp;
sudo rm -rf /var/run/docker&nbsp;
#Remove lock files&nbsp;
sudo rm /var/run/docker.*
#Start docker
sudo /sbin/service docker start

Starting spark-shell with proxy configuration

When you try to run import packages to spark-shell you normally use option –package

spark-shell --packages com.databricks:spark-csv_2.11:1.5.0 

This works fine if you are in open network. If you are inside a firewall protected by a proxy it fails with an exception.

To avoid this issue, proxy configuration should be passed. Example:

spark-shell --conf "spark.driver.extraJavaOptions=-Dhttp.proxyHost=myproxy.host.com -Dhttp.proxyPort=80 -Dhttps.proxyHost=myproxy.host.com -Dhttps.proxyPort=80"   --packages com.databricks:spark-csv_2.11:1.5.0 

Creating a dataframe using CSV file in Apache Spark Java

Add following maven artifact to your project.

 <dependency>
<groupId>com.databricks</groupId>
<artifactId>spark-csv_2.10</artifactId>
<version>1.5.0</version>
</dependency>
Now CSV data source is available in your classpath, you should able to use it to read CSV as a dataframe.
SparkConf sparkConf = new SparkConf().setAppName("CSVRead").setMaster("local[*]");
jsc = new JavaSparkContext(sparkConf);
SQLContext sc = new SQLContext(jsc);
DataFrame df = sc.read().format("com.databricks.spark.csv").option("inferSchema", "false").option("header", "true") .option("charset", "UTF-8").option("escape", escape).option("delimiter", ",").option("mode", "PERMISSIVE").option("quote", "\'").load("hdfs://localhost:9002/data/data.csv);
df.show();

Example java program connecting to Oracle Cloud Storage

import java.io.File;
import java.io.FileInputStream;
import oracle.cloud.storage.CloudStorage;
import oracle.cloud.storage.CloudStorageConfig;
import oracle.cloud.storage.CloudStorageFactory;

public class ExampleOracleCloudStorage {

public static void main(String[] args) {
try {
// Creating connection
String ocsUserName = "bacd.xyz@oracle.com";
String ocsPasswd = "mypassword";
CloudStorageConfig OCSConfig = new CloudStorageConfig();
OCSConfig.setServiceName("storagetrial12121-inoracletrial1000")
.setUsername(ocsUserName)
.setPassword(ocsPasswd.toCharArray())
.setServiceUrl("https://storage.us2.oraclecloud.com/");
CloudStorage OCSConnection = CloudStorageFactory.getStorage(OCSConfig);
System.out.println("Creating tmp container ");

// Create a container
OCSConnection.createContainer("NEW_Container");
FileInputStream inStream = new FileInputStream(new File("/home/user/data.txt"));

// Store a file to container
OCSConnection.storeObject("NEW_Container", "hello-world.txt", "text/plain", inStream);
inStream.close();

} catch (Exception e) {
e.printStackTrace();
}
}
}

Java Program connecting to AWS S3 – Using proxy

import java.io.IOException;
import java.io.InputStream;
import java.util.List;

import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.model.Bucket;
import com.amazonaws.services.s3.model.S3Object;
import com.amazonaws.util.StringUtils;

public class ASWS3Example {

public static void main(String[] arga) throws IOException {
// Generate access key and access password from the Amazon Control
BasicAWSCredentials credentials = new BasicAWSCredentials("AKIAIYTI5SSSVPCYWRVNXKA",
"jvYf5daIZ/FznnwSSFShsZQ2Q+IZcafddXpb134qyeE9il7");

ClientConfiguration clientConfig = new ClientConfiguration();
clientConfig.setProtocol(Protocol.HTTP);

// clientConfig.setProxyHost("yourproxy.com");
// clientConfig.setProxyPort(80);

System.out.println("Connecting to s3.amazonaws.com ..............");
AmazonS3 connection = new AmazonS3Client(credentials, clientConfig);
connection.setEndpoint("s3.amazonaws.com");

List buckets = connection.listBuckets();
for (Bucket bucket : buckets) {
System.out.println(bucket.getName() + "\t"
+ StringUtils.fromDate(bucket.getCreationDate())); 
}

S3Object object = connection.getObject("testbucket", "hello.txt");
InputStream in = object.getObjectContent(); 
}
}