Using Polaris
Defining a Catalog
In Polaris, the catalog is the top-level entity that objects like tables and views are organized under. With a Polaris service running, you can create a catalog like so:
cd ~/polaris
./polaris \
--client-id ${CLIENT_ID} \
--client-secret ${CLIENT_SECRET} \
catalogs \
create \
--storage-type s3 \
--default-base-location ${DEFAULT_BASE_LOCATION} \
--role-arn ${ROLE_ARN} \
quickstart_catalog
This will create a new catalog called quickstart_catalog. If you are using one of the Getting Started locally-built Docker images, we have already created a catalog named quickstart_catalog for you.
The DEFAULT_BASE_LOCATION you provide will be the default location that objects in this catalog should be stored in, and the ROLE_ARN you provide should be a Role ARN with access to read and write data in that location. These credentials will be provided to engines reading data from the catalog once they have authenticated with Polaris using credentials that have access to those resources.
If you’re using a storage type other than S3, such as Azure, you’ll provide a different type of credential than a Role ARN. For more details on supported storage types, see the docs.
Additionally, if Polaris is running somewhere other than localhost:8181, you can specify the correct hostname and port by providing --host and --port flags. For the full set of options supported by the CLI, please refer to the docs.
Creating a Principal and Assigning it Privileges
With a catalog created, we can create a principal that has access to manage that catalog. For details on how to configure the Polaris CLI, see the section above or refer to the docs.
./polaris \
--client-id ${CLIENT_ID} \
--client-secret ${CLIENT_SECRET} \
principals \
create \
quickstart_user
./polaris \
--client-id ${CLIENT_ID} \
--client-secret ${CLIENT_SECRET} \
principal-roles \
create \
quickstart_user_role
./polaris \
--client-id ${CLIENT_ID} \
--client-secret ${CLIENT_SECRET} \
catalog-roles \
create \
--catalog quickstart_catalog \
quickstart_catalog_role
Be sure to provide the necessary credentials, hostname, and port as before.
When the principals create command completes successfully, it will return the credentials for this new principal. Be sure to note these down for later. For example:
./polaris ... principals create example
{"clientId": "XXXX", "clientSecret": "YYYY"}
Now, we grant the principal the principal role we created, and grant the catalog role the principal role we created. For more information on these entities, please refer to the linked documentation.
./polaris \
--client-id ${CLIENT_ID} \
--client-secret ${CLIENT_SECRET} \
principal-roles \
grant \
--principal quickstart_user \
quickstart_user_role
./polaris \
--client-id ${CLIENT_ID} \
--client-secret ${CLIENT_SECRET} \
catalog-roles \
grant \
--catalog quickstart_catalog \
--principal-role quickstart_user_role \
quickstart_catalog_role
Now, we’ve linked our principal to the catalog via roles like so:
In order to give this principal the ability to interact with the catalog, we must assign some privileges. For the time being, we will give this principal the ability to fully manage content in our new catalog. We can do this with the CLI like so:
./polaris \
--client-id ${CLIENT_ID} \
--client-secret ${CLIENT_SECRET} \
privileges \
catalog \
grant \
--catalog quickstart_catalog \
--catalog-role quickstart_catalog_role \
CATALOG_MANAGE_CONTENT
This grants the catalog privileges CATALOG_MANAGE_CONTENT to our catalog role, linking everything together like so:
CATALOG_MANAGE_CONTENT has create/list/read/write privileges on all entities within the catalog. The same privilege could be granted to a namespace, in which case the principal could create/list/read/write any entity under that namespace.
Using Iceberg & Polaris
At this point, we’ve created a principal and granted it the ability to manage a catalog. We can now use an external engine to assume that principal, access our catalog, and store data in that catalog using Apache Iceberg. Polaris is compatible with any Apache Iceberg client that supports the REST API. Depending on the client you plan to use, refer to the respective examples below.
Connecting with Spark
Using a Local Build of Spark
To use a Polaris-managed catalog in Apache Spark, we can configure Spark to use the Iceberg catalog REST API.
This guide uses Apache Spark 3.5, but be sure to find the appropriate iceberg-spark package for your Spark version. From a local Spark clone on the branch-3.5 branch we can run the following:
Note: the credentials provided here are those for our principal, not the root credentials.
bin/spark-sql \
--packages org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.7.1,org.apache.hadoop:hadoop-aws:3.4.0 \
--conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
--conf spark.sql.catalog.quickstart_catalog.warehouse=quickstart_catalog \
--conf spark.sql.catalog.quickstart_catalog.header.X-Iceberg-Access-Delegation=vended-credentials \
--conf spark.sql.catalog.quickstart_catalog=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.quickstart_catalog.catalog-impl=org.apache.iceberg.rest.RESTCatalog \
--conf spark.sql.catalog.quickstart_catalog.uri=http://localhost:8181/api/catalog \
--conf spark.sql.catalog.quickstart_catalog.credential='XXXX:YYYY' \
--conf spark.sql.catalog.quickstart_catalog.scope='PRINCIPAL_ROLE:ALL' \
--conf spark.sql.catalog.quickstart_catalog.token-refresh-enabled=true \
--conf spark.sql.catalog.quickstart_catalog.client.region=us-west-2
Replace XXXX and YYYY with the client ID and client secret generated when you created the quickstart_user principal.
Similar to the CLI commands above, this configures Spark to use the Polaris running at localhost:8181. If your Polaris server is running elsewhere, but sure to update the configuration appropriately.
Finally, note that we include the hadoop-aws package here. If your table is using a different filesystem, be sure to include the appropriate dependency.
Using Spark SQL from a Docker container
Replace the credentials used in the Docker container using the following code:
USER_CLIENT_ID="XXXX"
USER_CLIENT_SECRET="YYYY"
sed -i "s/^\(.*spark\.sql\.catalog\.polaris\.credential=\).*/\1${USER_CLIENT_ID}:${USER_CLIENT_SECRET}\",/" getting-started/eclipselink/docker-compose.yml
docker compose -f getting-started/eclipselink/docker-compose.yml up -d
Attach to the running spark-sql container:
docker attach $(docker ps -q --filter name=spark-sql)
Sample Commands
Once the Spark session starts, we can create a namespace and table within the catalog:
USE polaris;
CREATE NAMESPACE IF NOT EXISTS quickstart_namespace;
CREATE NAMESPACE IF NOT EXISTS quickstart_namespace.schema;
USE NAMESPACE quickstart_namespace.schema;
CREATE TABLE IF NOT EXISTS quickstart_table (id BIGINT, data STRING) USING ICEBERG;
We can now use this table like any other:
INSERT INTO quickstart_table VALUES (1, 'some data');
SELECT * FROM quickstart_table;
. . .
+---+---------+
|id |data |
+---+---------+
|1 |some data|
+---+---------+
If at any time access is revoked…
./polaris \
--client-id ${CLIENT_ID} \
--client-secret ${CLIENT_SECRET} \
privileges \
catalog \
revoke \
--catalog quickstart_catalog \
--catalog-role quickstart_catalog_role \
CATALOG_MANAGE_CONTENT
Spark will lose access to the table:
INSERT INTO quickstart_table VALUES (1, 'some data');
org.apache.iceberg.exceptions.ForbiddenException: Forbidden: Principal 'quickstart_user' with activated PrincipalRoles '[]' and activated grants via '[quickstart_catalog_role, quickstart_user_role]' is not authorized for op LOAD_TABLE_WITH_READ_DELEGATION
Connecting with Trino
Replace the credentials used in the Docker container using the following code:
USER_CLIENT_ID="XXXX"
USER_CLIENT_SECRET="YYYY"
sed -i "s/^\(iceberg\.rest-catalog\.oauth2\.credential=\).*/\1${USER_CLIENT_ID}:${USER_CLIENT_SECRET}/" getting-started/eclipselink/trino-config/catalog/iceberg.properties
docker compose -f getting-started/eclipselink/docker-compose.yml down trino
docker compose -f getting-started/eclipselink/docker-compose.yml up -d
Attach to the running Trino container:
docker exec -it eclipselink-trino-1 trino
You may not see Trino’s prompt immediately, type ENTER to see it. A few commands that you can try:
SHOW CATALOGS;
SHOW SCHEMAS FROM iceberg;
CREATE SCHEMA iceberg.quickstart_schema;
CREATE TABLE iceberg.quickstart_schema.quickstart_table AS SELECT 1 x;
SELECT * FROM iceberg.quickstart_schema.quickstart_table;
If at any time access is revoked…
./polaris \
--client-id ${CLIENT_ID} \
--client-secret ${CLIENT_SECRET} \
privileges \
catalog \
revoke \
--catalog quickstart_catalog \
--catalog-role quickstart_catalog_role \
CATALOG_MANAGE_CONTENT
Trino will lose access to the table:
SELECT * FROM iceberg.quickstart_schema.quickstart_table;
org.apache.iceberg.exceptions.ForbiddenException: Forbidden: Principal 'quickstart_user' with activated PrincipalRoles '[]' and activated grants via '[quickstart_catalog_role, quickstart_user_role]' is not authorized for op LOAD_TABLE_WITH_READ_DELEGATION
Connecting Using REST APIs
To access Polaris from the host machine, first request an access token:
export POLARIS_TOKEN=$(curl -s http://polaris:8181/api/catalog/v1/oauth/tokens \
--resolve polaris:8181:127.0.0.1 \
--user ${CLIENT_ID}:${CLIENT_SECRET} \
-d 'grant_type=client_credentials' \
-d 'scope=PRINCIPAL_ROLE:ALL' | jq -r .access_token)
Then, use the access token in the Authorization header when accessing Polaris:
curl -v http://127.0.0.1:8181/api/management/v1/principal-roles -H "Authorization: Bearer $POLARIS_TOKEN"
curl -v http://127.0.0.1:8181/api/management/v1/catalogs/quickstart_catalog -H "Authorization: Bearer $POLARIS_TOKEN"