Snowpark Container Services: Quick Tutorial Running Metabase
Don’t bring your data to your apps, bring your apps to your data!
One of the most anticipated features from Snowflake this year is Snowpark Container Services, which will allow developers to deploy containerized apps directly in their Snowflake account. Instead of deploying on Kubernetes and managing permissions to an application in some bespoke way, you can now deploy an application directly in Snowflake and manage access to the app using Snowflake roles. It leverages a Kubernetes like-framework, but in true snowflake fashion, the complexity is abstracted away. You simply define the compute pool and how much it should scale, Snowflake handles the rest.
What excitement do we have to look forward to?
Gen AI
Snowpark promises a seamless integration of sophisticated generative AI and full-stack applications directly within Snowflake, simplifying development, deployment, management, and scaling of containerized workloads.
Simple Governance
Since your data never leaves Snowflake, it reduces the governance burden and enhances security and compliance.
Programming Flexibility
Snowpark Container Services heralds a new era of programming language and hardware flexibility, enabling developers to use any programming language and leverage configurable hardware options, including GPUs. This flexibility caters to a wide array of workloads, from machine learning and AI to full-stack app development, all while ensuring that data does not need to be moved outside of Snowflake's secure environment.
Simplified Life Cycle
Additionally, the service is designed to simplify the container lifecycle management by providing a fully managed and unified experience. It eliminates the complexities associated with container orchestration and management, allowing developers to focus on innovation rather than infrastructure. The service supports a variety of execution options, including jobs, service functions, and services, each designed to cater to different application needs.
Third Party Apps Coming to Snowflake!
A significant advantage of Snowpark Container Services is its facilitation of sophisticated third-party software and apps within the Snowflake environment. This enables developers to distribute and consumers to securely install and run cutting-edge applications entirely within their Snowflake account. Partnerships with industry leaders like Nvidia, Alteryx, Astronomer, Dataiku, Hex, and SAS further enrich the ecosystem, offering a range of powerful tools and applications that leverage Snowpark Container Services for enhanced performance and efficiency.
Snowpark Container Services Tech Partners
Bring Your Own Apps
As mentioned, one exciting feature of SCS is installing your own apps. In this tutorial, we will be covering how you can install Metabase inside your Snowflake account.
Tutorial: Running Metabase in Snowpark Container Services
This tutorial is very simple, and can be easily be completed by following the repository readme found here:
https://github.com/jeff-skoldberg-gmds/snowpark-metabase-demo
Here is a video walkthrough of the tutorial for those video learners out there:
Let’s step through the process:
setup.sql
Housekeeping ddl (optional), so you can easily drop these objects at the end
In the first step, we create some basic Snowflake objects and execute grants. This requires no explanation.
USE ROLE ACCOUNTADMIN;
CREATE ROLE test_role;
CREATE DATABASE IF NOT EXISTS tutorial_db;
GRANT OWNERSHIP ON DATABASE tutorial_db TO ROLE test_role;
CREATE OR REPLACE WAREHOUSE tutorial_warehouse WITH
WAREHOUSE_SIZE='X-SMALL'
WAREHOUSE_TYPE = 'STANDARD'
AUTO_SUSPEND = 60
AUTO_RESUME = TRUE
MIN_CLUSTER_COUNT = 1
MAX_CLUSTER_COUNT = 1
SCALING_POLICY = 'STANDARD'
INITIALLY_SUSPENDED = TRUE
COMMENT = 'Compute WH for tutorial.';
GRANT USAGE ON WAREHOUSE tutorial_warehouse TO ROLE test_role;
Create a security integration
CREATE SECURITY INTEGRATION IF NOT EXISTS snowservices_ingress_oauth
TYPE=oauth
OAUTH_CLIENT=snowservices_ingress
ENABLED=true;
This SQL command is about setting up a security integration within Snowflake for OAuth authentication, a method widely used for secure authorization. By executing this command, you instruct Snowflake to create a new security integration, provided it doesn't already exist, specifically for OAuth.
Bind Service Endpoint
GRANT BIND SERVICE ENDPOINT ON ACCOUNT TO ROLE test_role;
Enables the ability to create a service that supports public endpoints. For more information about public endpoints, see Ingress: Using a service from outside Snowflake.
Create the compute pool - the new secret sauce
Okay, now things are getting cool. The new object type in Snowflake is providing the compute resources for the application. In a sense, you can think of this simple command as replacing your Kubernetes cluster.
CREATE COMPUTE POOL tutorial_compute_pool
MIN_NODES = 1
MAX_NODES = 1
INSTANCE_FAMILY = CPU_X64_XS
AUTO_RESUME = TRUE
INITIALLY_SUSPENDED = FALSE
AUTO_SUSPEND_SECS = 900;
COMMENT = 'Compute pool for tutorial.';
GRANT USAGE, MONITOR ON COMPUTE POOL tutorial_compute_pool TO ROLE test_role;
Finish basic setup, verify objects:
Not too much explanation required here. directory = (enabled = true) allows us to stage our image later.
CREATE SCHEMA IF NOT EXISTS data_schema;
CREATE IMAGE REPOSITORY IF NOT EXISTS tutorial_repository;
CREATE STAGE IF NOT EXISTS tutorial_stage
DIRECTORY = ( ENABLE = true );
Create The Dockerfile:
# Dockerfile # Use Metabase as base image FROM metabase/metabase:latest # Expose the Metabase port EXPOSE 3000 # Start Metabase and keep it running CMD ["/bin/bash", "-c", "/app/run_metabase.sh && tail -f /dev/null"]
Build and push to Snowflake managed Container Repository:
Replace <repository_url> with the url from the `show image repositories` command
- Building from Dockerfile:
docker build --rm --platform linux/amd64 -t <repository_url>/metabase_service_image:latest . - Authenticate Docker with the Snowflake registry:
docker login rctdbyr-fe41115.registry.snowflakecomputing.com -u <user> - Push image:
docker push <repository_url>/<image_name>
This is a big deal! Your image has been pushed to your Snowflake account.
Create a Service (endpoint)
The final step before we can login:
USE ROLE test_role;
USE DATABASE tutorial_db;
USE SCHEMA data_schema;
USE WAREHOUSE tutorial_warehouse;
DESCRIBE COMPUTE POOL tutorial_compute_pool; -- STATE MUST BE "ACTIVE" OR "IDLE"
DROP SERVICE IF EXISTS metabase_service;
CREATE SERVICE metabase_service
IN COMPUTE POOL tutorial_compute_pool
FROM SPECIFICATION $$
spec:
containers:
- name: echo
image: /tutorial_db/data_schema/tutorial_repository/metabase_service_image:latest
endpoints:
- name: echoendpoint
port: 3000
public: true
$$
MIN_INSTANCES=1
MAX_INSTANCES=1;
SHOW SERVICES; -- List services in your account
DESCRIBE SERVICE metabase_service;
-- Get info:
SELECT SYSTEM$GET_SERVICE_STATUS('metabase_service');
SELECT SYSTEM$GET_SERVICE_LOGS('metabase_service', 0, 'echo');
SELECT SYSTEM$GET_SERVICE_LOGS('metabase_service', 0, 'echo', 10); -- Get last ten, modify accordingly.
-- Get link to access:
SHOW ENDPOINTS IN SERVICE metabase_service;
Here, the `create service` command is creating a new service in the compute pool we previously created. We can easily add max_instances if we want our service to scale.
This step also creates the endpoint url, which can be found by running `show endpoints in service metabase_service` command.
Run your app!
Copy paste your endpoint URL into your web browser. This should present the Snowflake login screen. Once logged in, enjoy Metabase!
Clean up when you are done playing
-- Stop all services and jobs on the compute pool
ALTER COMPUTE POOL tutorial_compute_pool STOP ALL;
DROP COMPUTE POOL tutorial_compute_pool;
-- Clean up the image registry (remove all images) and the internal stage (remove specifications)
DROP IMAGE REPOSITORY tutorial_repository;
DROP STAGE tutorial_stage;
Just a Proof of Concept
It is important to understand this tutorial is simply a proof of concept. It should not be used for production scenarios for several reasons. But most importantly, the application database is not persisted with Snowflake, so you would have problems using this in Production.
Wrapping up
Snowpark Container Services significantly transforms cloud computing and data management. It enhances the Snowflake ecosystem by simplifying the deployment of containerized apps, offering developers a more efficient way to innovate. The Metabase deployment tutorial in Snowflake via SCS highlights this ease and efficiency. Snowpark Container Services is a key advancement, enabling faster and simpler innovation by integrating data and app development seamlessly. I’m excited to see what the future brings for Snowflake and Snowpark!
