org.apache.gobblin.util.hadoop

Class TokenUtils

java.lang.Object

org.apache.gobblin.util.hadoop.TokenUtils

public class TokenUtils
extends Object

A utility class for obtain Hadoop tokens and Hive metastore tokens for Azkaban jobs.

This class is compatible with Hadoop 2.

Field Summary

Fields

Modifier and Type	Field and Description
static String	OTHER_NAMENODES
static String	TOKEN_RENEWER

Constructor Summary

Constructors

Constructor and Description
TokenUtils()

Method Summary

Modifier and Type	Method and Description
static void	getAllFSTokens(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.security.Credentials cred, String renewer, com.google.common.base.Optional<String> userToProxy, List<String> remoteFSURIList)
static void	getAllFSTokensImpl(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.security.Credentials cred, String renewer, List<String> remoteFSURIList)
static org.apache.hadoop.security.UserGroupInformation	getHadoopAndHiveTokensForProxyUser(State state, com.google.common.base.Optional<File> tokenFile, org.apache.hadoop.security.UserGroupInformation ugi, org.apache.hadoop.hive.metastore.IMetaStoreClient client, String targetUser) Get Hadoop tokens (tokens for job history server, job tracker, hive and HDFS) using Kerberos keytab, on behalf on a proxy user, embed tokens into a UserGroupInformation as returned result, persist in-memory credentials if tokenFile specified Note that when a super-user is fetching tokens for other users, fetchHcatToken(String, HiveConf, String, IMetaStoreClient) getDelegationToken} explicitly contains a string parameter indicating proxy user, while other hadoop services require impersonation first.
static void	getHadoopFSTokens(State state, com.google.common.base.Optional<File> tokenFile, org.apache.hadoop.security.Credentials cred, String renewer)
static void	getHadoopTokens(State state, com.google.common.base.Optional<File> tokenFile, org.apache.hadoop.security.Credentials cred) Get Hadoop tokens (tokens for job history server, job tracker and HDFS) using Kerberos keytab.
static void	getHiveToken(State state, org.apache.hadoop.hive.metastore.IMetaStoreClient hiveClient, org.apache.hadoop.security.Credentials cred, String userToProxy, org.apache.hadoop.security.UserGroupInformation ugi)
static void	getLocalFSToken(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.security.Credentials cred, String renewer)
static org.apache.hadoop.io.Text	getMRTokenRenewerInternal(org.apache.hadoop.mapred.JobConf jobConf)
static void	getRemoteFSTokenFromURI(org.apache.hadoop.conf.Configuration conf, org.apache.hadoop.security.Credentials cred, String renewer, List<String> remoteNamenodesList)
static String	obtainKerberosPrincipal(State state) Obtain kerberos principal in a dynamic way, where the instance's value is determined by the hostname of the machine that the job is currently running on.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

OTHER_NAMENODES

public static final String OTHER_NAMENODES

See Also:

Constant Field Values

TOKEN_RENEWER

public static final String TOKEN_RENEWER

See Also:

Constant Field Values

Constructor Detail

TokenUtils

public TokenUtils()

Method Detail

getHadoopAndHiveTokensForProxyUser

public static org.apache.hadoop.security.UserGroupInformation getHadoopAndHiveTokensForProxyUser(State state,
                                                                                                 com.google.common.base.Optional<File> tokenFile,
                                                                                                 org.apache.hadoop.security.UserGroupInformation ugi,
                                                                                                 org.apache.hadoop.hive.metastore.IMetaStoreClient client,
                                                                                                 String targetUser)
                                                                                          throws IOException,
                                                                                                 InterruptedException

Get Hadoop tokens (tokens for job history server, job tracker, hive and HDFS) using Kerberos keytab, on behalf on a proxy user, embed tokens into a UserGroupInformation as returned result, persist in-memory credentials if tokenFile specified Note that when a super-user is fetching tokens for other users, fetchHcatToken(String, HiveConf, String, IMetaStoreClient) getDelegationToken} explicitly contains a string parameter indicating proxy user, while other hadoop services require impersonation first.

Parameters:

state - A State object that should contain properties.

tokenFile - If present, the file will store materialized credentials.

ugi - The UserGroupInformation that used to impersonate into the proxy user by a "doAs block".

targetUser - The user to be impersonated as, for fetching hadoop tokens.

Returns:

A UserGroupInformation containing negotiated credentials.

Throws:

IOException

InterruptedException

getHadoopFSTokens

public static void getHadoopFSTokens(State state,
                                     com.google.common.base.Optional<File> tokenFile,
                                     org.apache.hadoop.security.Credentials cred,
                                     String renewer)
                              throws IOException,
                                     InterruptedException

Throws:

IOException

InterruptedException

getHadoopTokens

public static void getHadoopTokens(State state,
                                   com.google.common.base.Optional<File> tokenFile,
                                   org.apache.hadoop.security.Credentials cred)
                            throws IOException,
                                   InterruptedException

Get Hadoop tokens (tokens for job history server, job tracker and HDFS) using Kerberos keytab.

Parameters:

state - A State object that should contain property USER_TO_PROXY, KEYTAB_USER and KEYTAB_LOCATION. To obtain tokens for other namenodes, use property OTHER_NAMENODES with comma separated HDFS URIs.

tokenFile - If present, the file will store materialized credentials.

cred - A im-memory representation of credentials.

Throws:

IOException

InterruptedException

obtainKerberosPrincipal

public static String obtainKerberosPrincipal(State state)
                                      throws UnknownHostException

Obtain kerberos principal in a dynamic way, where the instance's value is determined by the hostname of the machine that the job is currently running on. It will be invoked when KEYTAB_USER is not following pattern specified in KEYTAB_USER_PATTERN.

Throws:

UnknownHostException

getHiveToken

public static void getHiveToken(State state,
                                org.apache.hadoop.hive.metastore.IMetaStoreClient hiveClient,
                                org.apache.hadoop.security.Credentials cred,
                                String userToProxy,
                                org.apache.hadoop.security.UserGroupInformation ugi)

Parameters:

userToProxy - The user that hiveClient is impersonating as to fetch the delegation tokens.

ugi - The UserGroupInformation that to be added with negotiated credentials.

getAllFSTokens

public static void getAllFSTokens(org.apache.hadoop.conf.Configuration conf,
                                  org.apache.hadoop.security.Credentials cred,
                                  String renewer,
                                  com.google.common.base.Optional<String> userToProxy,
                                  List<String> remoteFSURIList)
                           throws IOException,
                                  InterruptedException

Throws:

IOException

InterruptedException

getAllFSTokensImpl

public static void getAllFSTokensImpl(org.apache.hadoop.conf.Configuration conf,
                                      org.apache.hadoop.security.Credentials cred,
                                      String renewer,
                                      List<String> remoteFSURIList)

getLocalFSToken

public static void getLocalFSToken(org.apache.hadoop.conf.Configuration conf,
                                   org.apache.hadoop.security.Credentials cred,
                                   String renewer)
                            throws IOException

Throws:

IOException

getRemoteFSTokenFromURI

public static void getRemoteFSTokenFromURI(org.apache.hadoop.conf.Configuration conf,
                                           org.apache.hadoop.security.Credentials cred,
                                           String renewer,
                                           List<String> remoteNamenodesList)
                                    throws IOException

Throws:

IOException

getMRTokenRenewerInternal

public static org.apache.hadoop.io.Text getMRTokenRenewerInternal(org.apache.hadoop.mapred.JobConf jobConf)
                                                           throws IOException

Throws:

IOException